R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
Authors: Zefan Cai
Wen Xiao
Hanshi Sun
Cheng Luo
Yikai Zhang
Ke Wan
Yucheng Li
Yeyang Zhou
Li-Wen Chang
Jiuxiang Gu
Zhen Dong
Anima Anandkumar
Abedelkadir Asi
Junjie Hu
Venue: NeurIPS 2025 poster ()
Keywords: KV Cache Compression
Inference Acceleration
Reasoning Models
Primary area: deep_learning

Abstract: Reasoning models have demonstrated impressive performance in self-reflection and chain-of-thought reasoning. However, they often produce excessively long outputs, leading to prohibitively large key-value (KV) caches during inference. While chain-of-thought inference significantly improves performance on complex reasoning tasks, it can also lead to reasoning failures when deployed with existing KV cache compression approaches. To address this, we propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a novel method specifically targeting redundant tokens in reasoning models. Our method preserves nearly 100% of the full KV cache performance using only 10% of the KV cache, substantially outperforming existing KV cache baselines, which reach only 60% of the performance. Remarkably, R-KV even achieves 105% of full KV cache performance with 38% of the KV cache. This KV-cache reduction also leads to a 50% memory saving and a 2x speedup over standard chain-of-thought reasoning inference. Experimental results show that R-KV consistently outperforms existing KV cache compression baselines across two mathematical reasoning datasets.

R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
Zefan Cai Wen Xiao 2M Hanshi Sun Cheng Luo", Yikai Zhang', Ke Wan', Yucheng Li Yeyang Zhou', Li-Wen Chang, Jiuxiang Gu?, Zhen Dong*, Anima Anandkumar',
Abedelkadir Asi?, Junjie Hu _s
University of Wisconsin Madison Miicossoft CCannegie Mellon University
CCalifornia Institute of Technology Universitty of California San Diego University of Surrey
7 Adobe University of California Berkeley
https hpnpa://zefaa-eee cai gi io KV. KVgee
https: bttps://github.com Zef Cowpa//aooooo/eeeeaaaaaaaa/a
Abstract
Reasoning models have demonstrated impressive performance in self-reflection and chain-of-thon -ofthhuttttth reasoning. However, they often produce excessively long outputs, leading prohibitively large key-value (KV) caches during inference. While chain-of-thought inference significantly improves performance on complex reasoning tasks, it can also lead to reasoning failures when deployed with existing KV cache compression approaches. To address this, we propose Redundancy- aware KV Cache Compression for Reasoning models (R-KV), novel method specifically targeting redundant tokens in reasoning models. Our method preserves nearly 100% of the full KV cache performance using only 10% of the KV cache, substantially outperforming existing KV cache baselines, which reaches only 60% of the performance. Remarkably, R-KV cache performance with 16% of the KV cache. This KV-cache reduction also leads to 90% memory saving and 6.6x throughput over standard chain-of- thought reasoning inference. Experimental results show that R-KV consistently outperforms existing KV cache compression baselines across two mathematical reasoning datasets.
achieves 105% of full KV
Introduction
Recent advancements in large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning and self-reflection. However, reasoning models (c.g., DeepSeek-RI (I) exhibit critical deployment challenge: their tendency to produce excessively lengthy and redundant reasoning traces results unsustainable memory demands [2]. primarily due to the rapid growth of the key- value (KV) cache during autoregressive generation. For instance, a DeepSeek- R -isttiilll-Laama--- 8B model may generate 32K tokens solve a complex math problem, consuming 15.5GB of memory to load the model weight and 4. IGB of memory to store the KV cache. This paradigm of long chain-of-thought (CoT) reasoning generation necessitates the development of KV cache compression.
Outputs from current reasoning models, especially during complex chain-of-thought generation, are fundamentally marked by pervasive redundancy. This inherent characteristic means they are often filled with superfluous content, including unnecessary reflections, iterative re-evaluations. and verbose self-dialogue, all of which add little new semantic value while significantly inflating the
B Corresponding to Zefan Cai wwwfcaiigmail.ccoomomo com, Wen Xiao wxiao@micoosoft. com and Junjie Hu
junjie hwwwisc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Decoding- Compression
KV Cache Selection
Key Cache
Query Cache
Redundency Estimnation
Importance Scoring
Decoding
Cosine Similarity
Attention
Before KV Cache Compression
13
R- Cache Compression
Joint Selection Strategy
64
81
KV Compression
Let
Decoding
KV Compression
Key Cache
Vatue Cache
Figure I: R-KV: (1) Decoding- Time Compression (.1;;
(2) KV Cache Selection with Importance and Redundancy Estimation ($3.2 $3.3 (3) KV Cache Compression by joint selection ($3.4)
length of the generation beyond what is needed for concise, effective reasoning. Our analysis ( 2. shows that over half of the tokens in RI's reasoning chains contribute minimally task performance, indicating that repetitive self- seff-verification steps or intermediate calculations could be substantially condensed by KV cache compression methods without compromising reasoning accuracy.
However, existing KV cache compression works 5 7 primarily handle long input prompts but do explore extensively for long generation outputs. Furthermore, based standard KV-cache compression methods that rely on simple attention-base importance filtering often fail because the repetitive sections generate high attention signals for themselves. Naively pruning tokens with "low attention weight" may remove crucial but scattered bits of reasoning, or over-retain duplicative self-reflections that appear to have high attention. This observation motivates our exploration of redundancy-avee compression strategies, which selectively retain "important and non-repetitive context" during decoding to preserve the model's 's critical reasoning ability.
observation ( (2.22
In this work, we propose Redundancy-aware KV cache compression for reasoning models (i.c., R-KV). Our approach consists of three key components: (I) attention-basee based importance scoring mechanism that selects critical tokens for retention, (2) dynamic redundancy scoring mechanism that identifies repetitive tokens through real-time analysis of key vectors, and (3) joint eviction mechanism that balances both redundancy and importance to optimize cache efficiency.
In our experiments popular math reasoning benchmarks ($4), by selectively retaining only 10-34% % of the original KV cache, R-KV achieves comparable performance parity with the uncompressed reasoning model, outperforming state-of-the-art compression baselines with only 60% of the perfor- mance. Remarkably, R-KV even achieves 105% accuracy of the full KV baseline with around 16 % of the KV cache using DeepSeck-RI -istsek-----iiii---a-laaaa Distill--Llam--BB the AIME-24 dataset.
This advancement addresses fundamental tension deploying state-of-the-art ALMM-balanccie reasoning capabilities with practical memory constraints. Our contributions extend beyond technical optimization: we provide systematic evidence that redundancy CoT generation can be strategically compressed without compromising reasoning abilities. As a training- free and R-KV can be used in the rollout process in reinforcement learning (RL) and LLM serving.
method,
2 Observation
2.1 Redundancy in Reasoning Models
As noted in 22] reasoning models often generate detailed chain of thoughts and multiple reflec- tion steps, resulting in significantly longer responses than standard models. |Figure 2) shows that both reasoning models (i.e., DeepSeek-RR--Disill-LLmmme generate more than 8x longer generation output compared to the ground truth on two popular math reasoning datasets. However, not all the additional tokens contribute meaningful content, as much of the decoded context is dominated by repetition. Figure
DeepSeek-RI-Distll-OOeeee I-Distill-Qwen-77 and

---

Kath-500 -Average (okerss
Math-500 1-gram Frequency
Math-500- 2-gram frequercy
29028
230
3937
153
199
1255
162-
120
sa
200.7
15
67
Qwinn 140
Jan 43
Owen
Qcin 140
Quinn 10
Qean
Uamo 66
Q689 T8
Average Length
L.gra Average Frequency
2-gram Average Frequency
Langge (ickarnl
11392.
190
$6.1
11.7
01.3
9.0
50
7000
5000
2500
67
Owin
U'mn
Ground Iruhh
Soctl-11 E-lnu-Qannnn
Dessbeet-h 2at--tt--------oo DatB-Qncn-Te
Figure 2: Comparison of generation length and average 1-12-gram frequency for reasoning models and ground truth of MATH-500 [8] and AIME 2024 (9] Reasoning models generate substantially longer responses with 8- 14x more tokens, and show higher word repetition with 5-7x higher frequency.
also shows that the average frequency 1- to 2-grams consistently higher in the generation output of reasoning models than in ground truth, indicating greater repetitions in the generated outputs of reasoning models.
2.2 Failure of Existing KV Compression Methods to Handle Redundancy
Most existing KV cache compression methods prioritize token selection based primarily tokens' contextual impor- tance, typically measured through atten- tion scores between key and query to- kens [ 4 While this approach effec- tively retains critical context, it fails to count for
You given problem. Problem: Mr. Roper's class 30 students, [Question and Instruction 102 words]
problem says
total
[Think- 203 words]
[Reflection for 13 times and 581 words in totai]]
But wait, the So, 10% of 30 So students are leavinng early.
[Thikk 36 words]
prob- lematic in reasoning models. In such models, we find that repetitive content often receives disproportionately high at- tention scores, as it closely mirrors pre- viously generated repetitive text. As result, redundant tokens are excessively retained, unnecessarily inflating the KV cache size without providing additional meaningful new information.
But in the initial problem...Sd students leaving early.
[Think 42 words]
1.30 Sd students leaving
Think words]
But in the Initial
problem, the... So students leaving earty.
[Think 42 words]
But wait, the early?
leaving early.
the
So, 10% 30 3. students are leaving early.
[Tink words]
But wait, the user wrote: 10.1 30 So, students are leaving early.
words]
But in the initial
30
Therefore, students are leaving earty.
Fig- ure 3 we visualize the cached tokens (inside red boxes) selected by a popular attention-based KV cache method (i.e., SnapKV), showing many repetitions re- lated to self-reflection and conclusion to the final answer.
Think words]
think that's all. The calculation is straightforward: 10% of 30 is
[Conclusion- words]
Figure 3: KV selected by SnapK SnapK suffers from
redundancy in reasoning models. Black tokens are not se- lected by SnapKV; brighter colors reflect higher attention scores. Blue tokens omitted output.
3 Redundancy-aware KV Cache Compression (R-KV)
To address the redundant thinking issue, we propose a redundancy-aware decoding-time KV cache compression method (R-KV) that explicitly targets the compression of redundant tokens in reasoning models. Our approach balances importance and non-redundancy in token selection, ensuring that KV cache storage is allocated to both highly informative and diverse content. By incorporating
3

---

redundancy estimation into the selection process, our method effectively mitigates unnecessary KV cache growth while preserving the model's reasoning capabilities.
Specifically, R-KV consists of three key components: (1) an importance scoring mechanism ($..)
leveraging attention weights, (2) redundancy estimation mechanism ($3.3) based on semantic
similarity of key vectors, and (3) a joint selection strategy ($3.4) that optimizes cache efficiency by balancing redundancy and importance.
3.1 Decoding-time Compression
Different from existing KV cache compression methods 5 that focus on the prefilling stage to manage long-context inputs. our R-KV focuses the decoding stage for reasoning models distinctive setting where the generated output is significantly longer than the prompt.
Specifically, R-KV allocates memory for two components: cache of budget size Buuiged to store retained KV tokens, and buffer of size Buuller for newly generated text tokens. The total memory requirement is thus Buol Bradget + Bruffer- After the model generates each fixed-length text segment in the buffer, R-KV performs KV cache compression. At the end of each text segment, the last tokens are always retained in the cache as observation tokens, following prior work []. Next, we concatenate the existing Brudger tokens in the cache with the first Buutfrr a tokens the buffer, resulting in Buudget + Buuffer score (3.4), and we select the top k = Brudget a tokens to in the rest of the cache budget, in addition to the observation tokens. This process compresses the KV cache while preserving critical context, enabling efficient memory utilization during autoregressive decoding.
candidate KV tokens. Each candidate is assigned selection
3.2 Importance Scoring via Attention Weights
Following attention-baset methods (e.g.. SnapKV 13, PyramidKV (l) R-KV estimates token importance using attention weights, leveraging the intuition that tokens receiving higher attention contribute more to decoding and are thus more critical for preserving model performance. Specifically, we compute each key token's 's attention scores received from the last a observation tokens during decoding. In addition to the standard multi-head attention mainly adopted by the prior works (3), we also propose the importance score estimation using the grouped-query attention. Below, we detail the estimation top of these two popular attention mechanisms used by current LLMs.
Multi-Head Attention (MHA). Given the last a observation tokens as query Qh E Raxd and key states Kh Rn for each attention head h, the attention scores A Roxa are computed as:
A softmax @ ((KK() (VV).
(1)
Brouped-Quers Attention (GQA). GQA, each key/value head is shared among a group of G distinct query heads indexed by g E [0, G). Correspondingly, we denote the shared key/value states as Kh, V E Raxd, and the G query states as Qh. ..... Qh.G-1 E Raxd within the head
group indexed by h, where is the number of key/value states, d is the head hidden dimension. The attention score for each of the G query heads within the group is computed as:
A hgg group
= Qh (K*) T l/V E Rax" for g
- -
(2)
These G individual matrices are then aggregated into single consolidated matrix A for the head group h using a max-pon max-pooling operation across the group dimension. The final attention weight A" for the head group h is then obtained by renormalizing Ab group along the key token dimension.
A hh gmup
maxpool ([A
0,0 group
b.G-1]) group
R A softmax (Apr) R
Stabilization and Importance Estimation. We use A hereafter to denote the attention weights calculated using either MHA or GQA. Note that the per-token importance scores derived from A" may contain outliers with excessively high values, resulting in unstable estimation importance scores. To mitigate this influence, we follow the prior work
3 and apply max-pooling operation to these per-token importance scores over a sliding window size 2W across recent tokens. Specifically, we denote A as the attention score from the j-th query the i-th key in A We obtain the stabilized

---

attention score A by computing its () entry, and finally obtain the importance score of retaining the i-th token in the KV cache as I for each attention head h, as shown below:
A max (Aj.-w,,... AJ.... Afi 4 = Σ
- Aie
ER.
(4)
2=0
3.3 Redundancy Estimation via Semantic Similarity
To identify redundant tokens, we measure the semantic similarity between key states using cosine similarity. Tokens with high similarity to others are considered potentially redundant and can be selectively removed to optimize KV cache memory.
Cosine Similarity between Key Tokens: Given the key tokens K Ra for specific head h,
We first normalize each key vector K, Vi E [0, 1) into K;, and then compute the cosine similarity matrix Shh using the normalized key vectors.
KP E Rdd s ="=K"(K") E R"**, St + O, E [0,n). K 12
(5)
where || ||2 is the L.2 norm and e is a small constant (e.g., 10-8) for numerical stability. To prevent tokens from being marked as redundant with themselves, we zero out diagonal elements S
Enforce Retention of Recent Tokens. While redundant, such tokens may still carry meaningful information. Thus, naively removing all redundant tokens can impair model performance. To address this, we retain only the 3 most recently generated tokens among those exhibiting high similarity, as these later tokens tend to better support the model's decoding than earlier ones. To enforce this, we further zero out the similarity scores in Sh corresponding to these 3 most recent similar tokens. Formally, for each token E [0, n)), we identify the set of indices of highly similar tokens: I ( S > T.j [0,n)); where T a fixed hyperparameter for similarity threshold. For this set, we extract the subject ZZ,6 C Ih,, containing up to the 3 largest indices-i.e.. the 3 most recent similar tokens to token i, or fewer if not enough such tokens exist. We then suppress their influence by zeroing out their similarity scores with token in Shh i..., Sh. O, Vj E: I This modification effectively nullifies the direct similarity links from token to its 3 most recent highly similar tokens.
Redundancy Score Estimation: Finally, we compute normalized redundancy scores for all key tokens in Eq. (6). First, for each key token E [0, n) in each head h, we compute its average similarity score S Intuitively, Shhh measures how similar token i is, on average, to all other key tokens in the sequence. A high 5 indicates that the semantic content of token is largely shared with other tokens, suggesting potential redundancy. Next, to obtain per-token redundancy scores R within a fixed numerical range for each head h, we normalize 5th using a softmax operation. The resulting score R reflects the redundancy of token i for head h, with higher values indicating greater redundancy.
S
S R = (softmax (1Sb ................................ Stt -h-1)),;
.IS.....,
3.4 Joint Selection Strategy for KV Cache Retention
To efficiently manage KV cache storage while retaining essential context, employ joint selection strategy that integrates both importance and redundancy scores. Given a predefined token budget Buudget per attention head, our goal is to retain tokens that maximize information diversity while minimizing redundancy. The final selection score Z for each token in head h is computed as:
Z = A1 (1 R
where the importance score 1tth and the redundancy score R are computed in Eq. (4) and Eq. (6) respectively. A higher If indicates that a token is more important and should idcally be retained, while higher RR suggests higher token redundancy. The hyperparameter À controls the trade-off
5

---

DepSSeekRRL--Diill-Laaa---- MATH-500
DeepSeek -lama-88 AIME24
Performance (Acc.)
50
X*
70
40
Performance
50
R-KV
20
RKV
SnapKV
10
SnapKV
FullKV
Fu
0
500
1000
1500
2000
1000
3000
52%
69%
2% 3% 5%
7%
10%
13%
16%
20%
KV Cache Budget
KV Cache Budget
DeepSeek Distill Qwen 14B
AIME24
MATH-500
Performance (Acc.)
60
Performance
40
R-KV
40
SnapKV FullKV
SnapKV
KV
500
1000
1500
2000
2000
3000
4000
599%
18%
27% 36%
54%
72%
2%66%
8%
12% 17%
25%
33%
KV Cache Budget
KV Cache Budget
Figure 4: Results of R-KV compared with SnapKV and FullKV the MATH-500 and AIME24 datasets for R R1-Llama-8B (top) and RI-Qwen-14B (bottom). Results are reported as pass @ based on 64 generated responses per question.
between prioritizing important tokens and reducing redundant tokens. We discuss the rationale for choosing À through a sensitivity analysis in $5.1 This strategy ensures that the KV cache prioritizes storing tokens that are both important and semantically diverse, thereby improving memory efficiency without compromising model performance.
4 Experiment
4.1
Experimental Setup
Models and Datasets In experiments, variants of the DeepSeek-RI distilled model: DeepSeek-RI --DistillLLlama-88, and
[]] which we refer to as R1-Llama-8B and R1 R1-Qwen-14B, 14B, respectively, for brevity throughout the paper.
We evaluate the models' mathematical reasoning capabilities using three benchmarks: MATH-500 and AIME 2024 [9].
g
Hyperparameters
We set Boutter 128, a and 0.1, with an analysis of in $.1
Baselines We compare method against SnapKV 3. originally designed for long prefilling.
To adapt it for decoding, we apply the same compression interval as our method, i.e., compressing the KV cache every 128 decoding steps using identical Buudger and Brufter- Our approach focuses on improving KV cache eviction through a hybrid strategy, and we therefore restrict comparison to state-of-the- thetthe-rrt iitentiionbased eviction methods. Budget allocation techniques (c.g.. head-level 6 and layer-level (5)) are orthogonal to our work and not included. We also report results for FullKV, which retains the full KV cache and serves the gold standard for decoding quality.
Evaluation Setup We set the maximum generation length to 16,384 tokens for MATH-500 and 32,768 tokens for AIME 2024 and AIME 2025, because further increasing generation length has shown no improvement that using greedy decoding to evaluate long-output reasoning models results in significant variability across different setups. Following existing works [I] we utilize pass @ & evaluation [10] and report
model performance on these datasets from our attempts. We find

---

pass I using non-zero temperature. We use the recommended sampling temperature and top-p value for each model. i.e., sampling temperature of 0.6 and top-p value of 0.95 for DeepSeek- R1 Distilled models. We generate 64 responses for each question. Pass@ is then calculated as @ 1 = k Σ -i=1 Pi. where p; denotes the correctness of the i-th response. This method provides more reliable performance estimates.
4.2 Results
The accuracy performance of R-KV compared with all baselines is shown in Figure with detailed accuracy numbers in Appendix B.2 The KV cache budget ratio is calculated based on the KV cache budget and the average generation length of tokens, i.e., R1-Llama-88h 8B: 2, 979.1 on MATH-500 and 15, 535.8 on AIME24; R RI-Qwen-14e 14B: 2, 833. 04 on MATH-500 and 12, 402 on AIME24. Our method significantly outperforms the baseline SnapK V, achieving up to 40% Acc. improvement. We provide two KV cache budget and performance analysis. Fixed budget analysis is more practical because when the model outputs longer (i.e., from 2, 979.1 on MATH-500 to 15, 535.8 on AIME24), the KV cache budget needed for lossless compression increases less (i.e., 512). In the KV cache budget ratio perspective, the changes of lossless compression ratio is dominated by generation length.
Ratio Budget For RI RI-Llama-8B, R-KV achieves lossless compression with 34% KV cache budget on the MATH-500 dataset and with 10% KV cache budget on the AIME-2024 dataset. Given 16% KV cache budget, our method even surpasses the FullKV baseline, reaching 105% of its accuracy. Similarly, for R RI-Qwen- DeB, R-KV achieves lossless compression with 54% KV cache budget on the MATH-500 dataset and with 25% KV cache budget on the AIME-2024 dataset. Given 33% KV cache budget, our method achieves 105% of FullK accuracy.
Fixed Budget For RI-Llama- 8B, R-KV achieves lossless compression with 1024 KV cache budget on the MATH-500 dataset and with 1536 KV cache budget the AIME-2024 dataset. For RI-Llama- 8B, R-KV achieves lossless compression with 1536 KV cache budget on the MATH-500 dataset and with 3072 KV cache budget on AIME-2024.
Discussion
5.1 How to Choose X?
Figure 5) shows the distributions of the Importance Score (I') and Redundancy Estimation (R*) for head = 0 at the top layer (Niyye = 31). The figure reveals that I' sparse and dominated by a few outlier values, while the similarity distributions (which inform R²) are relatively dense. When == 0, the token retention strategy is overned entirely by Redundancy Estimation (R"). As shown in Figure 5] the initial four tokens are not guaranteed be preserved. As highlighted by prior work (7), evicting these initial tokens can severely impair the generative capabilities of LLMs. Therefore, it is crucial to select À value that starts from at least 0.01. On the other hand, À increases beyond 0.1, the selection metric becomes increasingly dominated by attention scores. These observations suggest that an optimal lies within the range of 0.01 X < 0. effectively balancing the contributions of Importance Score and Redundancy Estimation.
presents the accuracy (Acc.) performance of R-KV the DeepSeek- Disttll-DiiiIIII -1----ma---aaa model using the MATH-500 dataset. The results further guide the choice of À for optimal performance. The figure demonstrates that À = 0.1 yields the highest accuracy. In contrast, strategies relying solely on redundancy (A 0) or solely on attention (A = 1) exhibit the poorest performance, underscoring the complementary nature of these two metrics and the importance of balanced approach. Thus, based on this finding, we select a = 0.1 for all evaluations detailed in Figure 4
Figure 6
5.2 Failure of Attention-Bann Based Methods to Capture Redundancy
To thoroughly investigate the advantages of R-KV's hybrid selection metrics (combining attention and redundancy) over pure attention -aaed importance metrics, we compared the tokens selected by R-KV against those chosen by a pure attention-base based method (SnapKV). We present case where R-KV correctly completes the task while the comparison method fails. As illustrated in Figure 7
7

---

Performance Comparison
Score pariarison Attention Metric Redundancy only Metric R-KY Different
80
Redunsancy
Score valua toymlog)
(om0.03)
1.10.11
20.4
Ret. (o.04)
RRV 1a=0.21
Tuffommncca
60
Actontion only lo=1)
50-
30
con Indox
200
420
500
503
3000
KV Cache Budget
Figure 5: KV sekection score comparison of atly metric v.s. redundency-only metric v.s. R-KV with different A. When À 2 0.1, the selection score starts to be dominated by attention score.
Figure 6: Performance Com- parison of the same methods Figure 5
(Questtion Instruction- words]
(bccodd(
[Queation
given
[Think- words] Okay, need to find out how many students
Roper's class students, 20% class football players, Out remaining clase, 20% the students cheerleaders part bond. These groups students need
are footbol playorr and band members. 1..12 #2atudents lsaving earty.
(Refeection-
1067 words] Walt.
problem
earty tocky invel owny gaime How many sudents are loaving early? need solve problom step step. First, you need provide chain of-thoughh, then provice final anower Provide the final aroose
"You glvan a diass students
many studants are laaving early?
Wait Wiill.. 20% 35%
feaving sony How many students ond leaving Wrly in 12 stude nes leave borry, But
12 Paudamts
famms Final
bbowdD.
War.... So. 20% 30 10 In uceeet are leoving early But wait. So 20% of stucients.
[Think- 107 words] Okay, to havve this math prostem here from Mr. Roper n class. out step by step. Finnt, protlem says thene onn- students in total in the olass Out these, 20% are footboll players. Hmm, okay, to need to tind out how mony stucents ahotis To find 20% of 30, con cakculate 0.20 mulisdiad by 50. Let do that 0.20 50 equala Bo, ( wae football player in the claso. Now, the problum says that out of the remaining class, 25% are cheerleaders part the haed.
30 is No inudemss lnaving early But "You are gheen class
Let me try fig
Wart Wolly -So 5o [Suddonts are loaving
anty But in the users a first metsago. ...
Sos Sodddtdddett and berving marly Bol wall,
20 studerta
leneing melly But in
the ITD prodeem
You
class of 30 musonts, How mam students are
lnoxingearl earlyr So gtudents are lerving early But walt, the und Uiikes So,
tore in So intenent onn leising enny, But in the initiad pritiem... 3
Studento baving carly But walt, You ore given cats of 30studenta How
[Reflextion- 292 words] Walt, so aftrr takking out the footboll players, how many students am lon? Let me submract the football players from the toto dlass. Total studerta da 30 Football playera 6 Remaining stuentts 30 24 Bo, there are
pludentg pre Bar rig sorly? So, 10% or 30 163, So soudenn 1reten
early, Butin the initial problem. the wotee "You given class of 50
(ctuentss] 10%6 of them leaving) por moww patinns leaving
students left who are not footboll players, Of these 24, 25% ard choutlacders of part the band. Inced to find 25% of 24, Caloukzing 25% of 24, 0.25 24 6, So, of the band. The problem askas how many the football playen, cheerleaders, and adding them Football players (8) Cheorteaders/Be /aad (8) students, Walt, let double check 20% indeed
16% of 30 So, datudents
lasving
[Su wait, the user weate: YYou
given closs 30 doadens,
Bre 1esyingg How manyy dudents
studiets wither cheeriesdes
eorly" So. 30 is So.
students insvingartt These
initial problem,
"You
30 [iprudents
band
are leaving earty Hoo many
early""
S.
Therefore
leaving merly,
given
remaining students (20 247 football
doesn' Than 25%5 right. Alternatvvely maybel players
Wall,
class students 10% of them leaving early (andeFudeett) 10% of Therfooe, I[suadents leawing ear
preteawing early?
leaving S Yuuh,
user mentioned thet answer
box and using LoTox So,
saye
remaining class?
to write the final answer as (boxod[3). Incorreat Answer
remaining are cheerteaders band.
loaving: of 30
[Conclusion words] Ithink that's all. The coloutation straightforwatt: 10%
then 25% of is
Conrect
of 10 is Puru attention- based metric selects redundant& unessential tokens
[Canelusion 13 words] Yeah, that makes sunse. think that's cornect. the answer is 12 students leaving early.
R-VV selects more diverse tokens
(a) SnapKV
(b) R-KV
Figure 7: Comparison of selected key-value (KV) tokens for an example between SnapKV (left) and R-KV (right). Grey tokens are unselected, while the gradient from light to dark red indicates the number of attention heads selecting each token (darker more heads). R-KV selects more diverse and broadly distributed set of tokens, capturing richer contextual information.
grey tokens represent unselected tokens, while the gradient from light orange to red indicates the number of heads selecting each token, with darker red signifying selection by more heads.
When considering the tokens selected by all heads, we observe that R-KV selects a more diverse set of tokens that cover a broader range and contain more effective information. These selections are more evenly distributed throughout the decoded output, capturing a more comprehensive context representation. In contrast, SnapK selected tokens exhibit more limited coverage. It tends to favor tokens positioned close to the query token, which are often selected multiple times by various heads, indicating a concentration of attention in localized areas. Furthermore, SnapKV also selects tokens that are not in close proximity to the query but still constitute largely redundant and unimportant segments (i.e.. "3 students are leaving early." and "But in the initial").
5.3 Efficiency Analysis
Memory Saving R-KV achieves improved memory efficiency by allocating fixed-size buffers for both the retained KV cache and newly generated tokens. Unlike FullK which scales memory linearly with sequence length, R-KV memory footprint remains constant, enabling substantial savings during long-form generation. Detailed memory accounting is provided in Appendix C.I
8

---

Gen. Length
Method
Budget
Saving
Throughput (tok/s)
(8)
FullKV
75.44
8094
107.30
62 (max)
849.13
501 828
590,99
Fixed 1024
87.50
80,46
3 8094
100.60
8K
Fixed- 1024
87,50
402 (max)
3251.52
0000.0
R-KV
Fixed 1536
81.25
287 (max)
2525.75
6 6546972
919.72
Ratio 10% 8819
90.00
479 (max)
3809.15
3 3
0117.82
Ratio 34% -2785
66,00
167 (max
6008
3 698
840.61
Ratio
46.00
(max)
257..8
849
FullKV
69.41
16.86
234.65
30 (max)
347.03
488 S8U
407.89
Fixed 1024
93.75
80,95
162266
201.18
16K
Fixed 1024
93.75
402 (max)
188.82
6.446972
2053.10 10
R-KV
Fixed 1536
90.63
287 (max)
2447.61
4 46744082
909.65
Ratio 10% 1638
90.00
271 (max)
2,300.28
4 4413
918.68
Ratio 34% -5570
66,00
82 (max)
797.43
1 1.335 452
674,70
Ratio
46.00
(max)
584.77
749 156
Table 1: Memory saving, throughput, and decoding-tinne comparison for Llama3-8B under various generation length and KV cache compression budget settings.
Computation Overhead While R-KV introduces additional computation for importance and redundancy scoring, the total overhead modest and often outweighed by the reduced attention cost over a compressed KV cache. This trade-off becomes increasingly favorable as sequence length grows. Complexity comparisons can be found in Appendix C.I
We present the real-time analysis of memory saving and end-to-end throughput improvement in Table I When the batch size is I, R-KV exhibits a slight throughput advantage over FullKV. This suggests that the acceleration achieved by R-KV through reduced attention computation outweighs computational overhead of R-KV. However, this direct speedup constitutes a minor portion of the overall benefit. The primary throughput improvement from R-KV stems from enabling significantly larger inference batch sizes due to KV cache compression.
Real-time analysis
We evaluate end-to-end throughput under both ratio-based and fixed KV cache budgets. R-KV consistently enables much larger batch sizes and higher throughput than FullK with benefits becoming more pronounced at longer sequence lengths. For example, at sequence length of 16K, R-KV achieves up to 9x larger batch sizes and over 6.6x higher throughput under 10% compression ratio, and 13.4x larger batch sizes with 9.2x throughput under fixed budget of 1024. Detailed analysis are provided in Appendix
Related Work
The optimization of KV cache memory efficiency in LLMs has garnered increasing attention model sizes and context windows expand. Existing approaches primarily fall into three categories: dynamic token eviction
KV Cache Compression
[2] quantization [4] [5], merging [6] [7]
and low-rank decomposition 19 20 21]. Previous eviction methods like VV). PyramidKV[5I Ada-KV' KV22), HeadK ( dynamically prune tokens based on attention scores, but mainly focus on evieting tokens for prefilling stage. StreamingL.LM(7 and H20
proposed for decoding. However, these general-purpose techniques often struggle with reasoning-intents
tasks, where aggressive eviction risks disrupting critical intermediate steps in CoT, and suffers from reasoning models' inherent redundency.
Efficient Reasoning Recent works efficient reasoning focus on training the model generate use RL optimization with length penalty rewards to encourage models to produce more concise chains-of- -f-thought (CoT). [7] employs variable-length CoT datasets to supervised tune-tne (SFT) the LLM to reduce token usage while preserving reasoning correctness. Both RL and SFT methods require additional training. (27 28 29)
less CoT without sacrificing performance. 23
24
25
use test-time prompting to reduce generation length, but these methods may hurt the performance. As KV cache compression work for reasoning models, R-KV able to achieve lossless compression without extensive training and prompting.

---

7 Conclusion
We introduced R-KV, novel decoding-time KV cache compression method tailored the challenges of complex reasoning in large language models (LLMs). Reasoning models often generate long. redundant outputs that impose substantial memory and computational burdens during inference. R-KV addresses this by jointly scoring token importance and redundancy, enabling the retention of essential reasoning content while discarding repetitive or uninformative tokens. This dynamic and attention-guided strategy allows R-KV to preserve nearly full model performance using only 10-34% of the original KV substanlial outperforming prior compression methods.
Extensive throughput and efficiency analysis demonstrate that R-KV enables up to 13x larger batch sizes and over 9x speedup long-sequence generation scenarios compared to FullKV, with particularly strong gains under constrained memory budgets. With its training-free and model- agnostic design, R-KV provides a scalable and deployment-ready solution for reasoning LLMs, especially in streamlining the rollout phase of reinforcement learning workflows.
8 Acknowledgement
Research reported in this publication was partially supported by the National Science Foundation under Award Number IIS-2449766 The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Science Foundation.
10

---

References
[1] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin
Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan. Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingehao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng. Yuchen Zhu, Yunxian Ma. Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability lims via reinforcement learning, 2025.
[2] Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning, 2025.
[3] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, T sianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation, 2024.
[4] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o Heavy-hitter oracle for efficient generative inference of large language models, 2023.
[5] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.0200 2024.
[6] Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning, 2024.
[7] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.
[8] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprini
2021.
[9] MAA. American invitational mathematies examination aime. In American Invitational Mathematics Examination AIME 2024, February 2024.
11

---

[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman. Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, has/2107.03374 2021.
[11] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023.
[12] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of impor- tance hypothesis for I1m cache compression at test time. Advances Neural Information Processing Systems, (6652342--53366 2023.
[13] Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. Advances in Neural Information Processing Systems, 17:1270-1303, 2024.
[14] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750. 2024.
[15] Yuxuan Yue, Zhihang Yuan, Haojie Duammu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more, 2024.
[16] Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong. Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for memory-efficien llms inference. In Forty-first International Conference on Machine Learning, 2024.
[17] Jang- Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. arXiv preprint arXiv:2312.03414, 2023.
[18] Piotr Nawrot, Adrian Lańcuuki, Marcin Chochowski, David Tarjan, and Edoardo M Ponti. Dynamic memory compression: Retrofitting llms for accelerated inference. arXiv preprint arXiv:2403.096 09636 2024.
[19] Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, long-context
Yuejic Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for IIm inference, 2025.
[20] Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention low-rank space for kv cache compression. 2024.
[21] Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang. Hao Cheng, Chao Zhang, and Yelong Shen. Lore: Low-rank compression for llms kv cache with progressive compression strategy, 2024.
[22] Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient IIm inference. arXiv preprint arXiv:2407.11550 2024.
[23] Chen Li, Nazhou Liu, and Kai Yang. Adaptive group policy optimization: Towards stable training and tofen--fficient reasoning, 2025.
12

---

[24] Junjie Yang, Ke Lin, and Xing Yu. Think when you need: Self-adaptive chain-of of-thought learning, 2025.
[25] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang. Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng. Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li. Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1lms, 2025.
[26] Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, Suhang Wang, Yue Xing, Jiliang Tang, and Qi He. Stepwise cerplexity-guidee refinement for efficient chain-of-thought reasoning in large language models, 2025.
[27] Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. sukken-budge--awwer Ilm reasoning, 2025.
[28] Yule Liu, Jingyi Zheng, Zhen Sun, Zifan Peng, Wenhan Dong, Zeyang Sha, Shiwen Cui, Weiqiang Wang, and Xinlei He. Thought manipulation: External thought can be efficient for large reasoning models, 2025.
[29] Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking, 2025.
[30] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovie, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala. Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao
13

---

Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojmic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hos- seini, Sahana Chennahasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta. Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet. Xiaodong Wang. Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen. Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani. Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf. Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi. Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le. Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le. Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun. Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand. Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena. Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg. Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu. Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo. Nicolas Usunier, Nikhil Mehta. Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parckh. Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto. Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe. Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez,
14

---

Tamar Glaser, Tamara Best, Thilo Koehler. Thomas Robinson, Tianhe Li. Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary VeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Ilama herd of models, 2024.
[31] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024.
[32] Joshua Ainslie, James Lee- Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895-4901, Singapore, December 2023. Association for Computational Linguistics.
A Method
A.1 Algorithm
The pseudo-code of the method is shown in Algorithm
A.2
Implementation Details
Max Pooling of Attention Weights Latest open-source LLMs 30 31 have widely adopted Grouped-Quee Querr-qeery Attention (GQA) [2), where multiple query heads share common pair of key- value heads to substantially reduce memory access overhead during inference. In key-value (KV) cache eviction strategies, it's thus often necessary to downscale attention scores from (Q head, seq_len, seq_len) to (KV_head, len, seq_len). While previous works such as SnapKV (3] have predominantly employed mean pooling to aggregate attention scores across query head groups, we hypothesize that max pooling could better preserve the most critical tokens for each query head. Our empirical results demonstrate that max pooling leads to improved performance, and we adopt it for all main experiments.
Calibration of SnapKV's Observation Window Mask The official implementation of SnapKV applied an upper triangular attention mask to the attention weights matrix to enforce causality. The attention weights matrix is then processed with softmax, slicing, and summation to obtain observation window scores for each prefix token.
They adopted an upper triangle to prevent tokens in the observation window from seeing tokens after them, and then applied softmax and summation. However, their implementation does not account for the fact that tokens within the observation window absorb part of the attention weight originally assigned to prefix tokens, thereby disrupting the normalization property.
In our implementation, address this issue by first slicing the attention weights before applying softmax. This approach ensures proper normalization and leads to significantly better scores in our tests.
if not fix_obs__ wibdwww mask torch.full. full(
15

---

Algorithm R-KV: Qate are query states a observation tokens, Krull, Via are the full KV cache states of length Liall
1: procedure R-KV((Kull Viui), Liun, Lundeet Qobs: a, Budgets Beatter T, B, X, e, H, dj)
2: if Trall - Trbudget < Bbutter then
D Check if compression is triggered
3: return full; Vian)
4: end
5: (Kobs: Vots) t last a tokens of (uulll, Viun))
6:
(Keand, Veand) + first (Leill a) tokens of (Kal,,, Vian)
7: No Liuil
Number of candidate tokens
8:
if No S Brodget then
9: return (Kumn ViaI)
> Not enough candidates to prune beyond budget
10: end if
for each head h 0... H do
12:
Compute attention matrix A E BaxN using Voss and Kh cand b Handles MHA/GQA as
per Eqs. (3-33 from text
13:
for k Ii - Lx= (A )4t
O... No 1 do
P For each candidate token k: > observation token, k: candidate token
14:
15:
end for NNN-1 ID-Pooling (4, k== 1 {
17:
end for
18:
for each head h ........ do
K norm R.Nx For k 0.. N. & Cosine Similarity Matrix Computation, similarity matrix
K4
norn,& Kh // ( cand, xll +
20:
5hh + Kh norm Kh )
Shh E RNN
21:
for l: = O.... N. do
b Prevent Self-Redundancy
22:
(S lkk t
23:
end for
24:
Bh f ((Sh) uv > T:1: : 0) for u, u {O...., 1} Identify Highly Similar Pairs
25:
for
O... N₂ 1 do
Enforce Retention of Recent Tokens
26:
Th +- {u uv = 1, v E [O,...,Ne- -1)
27:
Th. f- subset of Th with up to 3 largest indices v.
28:
for r' E Th
T do
(Sh)a,'' 00
end for end for
5 is now modified
31:
Let Sh No where (S*)a + N. IN--1( Juv
for 0 N₂ do
34:
R., + (softmax(**)
end for end for
36:
37:
for each head h 0.. H do
38:
for k = O...... .. do
39:
Scorer,A + Alen - (1 R)
40:
end for end for
41:
42:
Let AggScore E R.N.
43:
for 0..... No - do AggScore, means (ScoreR,1) end for
44:
Aggregate scores across heads
45:
46:
Id <- indices of Bap-Buudeet tokens from (0, ...,, Na 1} based AggScore (0...., K Keand + Kanaa [ddx,et); Veand set t- Veand [dr K Kcomp concatenate( K sean Kobs)
47:
48:
Order might vary
49:
Voomp +- concatenate Veand Vabs)
50:
Lipre_coot comp Brudget
Update length for next cycle
51:
return Keomp, Voomp)
52:
end procedure
16

---

Model
Benchmark
Method
128
256
512
768
1024
1536
2048
2350
3072
4 499
FulIKV
82.38
82.38
82.38
82.38
82.38
82.38
82.38
MATH
R-KV
67.39
80.21
81.34
82.34
82.65
SnapKV
32.53
50.07
64.03
70.81
74.43
78.43
80.50
Llama3-8B
FulKKV
49.79
49.79
49,79
49.79
49.79
49.79
49.79
49.79
49.79
AIME24
R-KV
0.42
10:21
29,48
40.31
45.26
51.56
52.29
53.85
53.13
SnapKV
0.16
0.94
4.53
11.20
15.73
26.04
32.76
39.43
41.93
FulKKV
94.58
94,58
94,58
94,
94.58
94.58
94,58
MATH
R-KV
56.21
73.33
84.77
88.79
90.72
92.72
93.62
SnapKV
26.32
43.93
77.93
82.52
86.63
90.86
92.73
Qwen-14B
FulIKV
65.68
65.68
65.68
65.68
65.68
65.68
65.68
65.68
65.68
AIME24
R-KV
0.57
7.92
24.53
36.25
42.66
55,00
56.09
64.32
67.45
SnapKV
0.26
2.86
12.86
16.30
25.00
36.41
46.56
52.86
54.32
Table Accuracy (%) of Llama3-8B and Qwen- on the MATH and AIME24 benchmarks
under different memory-optimizztion methods across context lengths. "_" denotes configurations that were not evaluated.
(self.window window_size self www...iidoww... ssidoo_siie), torch. finnffftttttttneeee weightssswwweess.... .eeigghtsdtype) min, device-attn_weights device,
6
7
mask_cond torch arange (mask. size (-1), aericceratt_weighhe device)
mask masked_fillllllllldl fill__maskk_______aeaaaaaaaaaaaa cond < (mask_cond 1) view (mask (-z) 1), 0) mask mask. to weights device)
attention_mask mask None, None, ;, :] Sttn_weightss
:, :, -self. wwwlffwwndde... window_size seef window_size ] += attention_mast
attn_weights nn functional. .offriinnlsssfta((( softmax attn weights, dim=-1, dtype=torch. float32
16
). to statteyyssttsst.e stapyy
(ttn_weights_st attn_ weights
:, :, -self window_size :, -self window_size
] mean(dim=-2)
else:
tttt_swights.sd
nn wwu.tiinnnnlll... softmax(
25
attn_weights(:; -self. window_sizo -self windov_sizel dim=-1, dtype-torch. float32,
mean (dim=-2) to -query_states.dyee
30
B Experiment
B.1 Devices
use NVIDIA A100 80G to finish all the experiments.
B.2 Main Results
See Table 2
17

---

C
Efficiency
C.1 Complexity Analysis of Memory and Computation
Memory Saving As discussed in $3.1 we need to allocate memory for the KV cache budget Musget 1BxxBuss Nugar* Nexx to retain Buddget KV cache tokens, and for the buffer Muufter Rax Brimi x X xd to store Buutter newly generated KV cache tokens during the generation of a text segment. Here, b is the batch size, Nuyer is the number of Transformer layers, Nhad is the number of attention heads, and d is the dimension of attention heads. In addition, we also need to allocate memory for the model weight Mj. During decoding, the previous query states are typically discarded by default, so we use a query cache store the last a tokens in the query state, consuming memory of M, E Rexxxx NNaast x In summary, R-KV requires memory of M Mg + Mudget Muuteer + M,, during generation. In comparison FullKV without KV cache compression, generating Bral tokens requires memory of Miil E R Noya Nuse Divot to retain Brn KV tokens, and memory of the model weight Mo. Therefore, the memory saved by our Mbutter Ma.
method w.r.t. V is: M Maaving Muull
Moudget
Computation Overhead The computational complexity of importance scoring (See $3.2) is Oudgg)))) while redundancy estimation (see $3.3) has complexity O OBBuugge) Thus, the total overhead incurred during each generation segment is O Brudgee) The generation com- plexity without KV cache compression is (Bru D(BrunButtee Boutter) whereas the complexity with KV cache compression is D(Bbaulee + Buutter) ). For reasoning models, Biu tends to be large because of the long generation length, and using a relatively small Brudget value can efficiently reduce com- putation cost. The effectiveness of this approach depends on depends on whether the speedup gained by attending over a reduced KV cache outweighs the overhead of computing the compression scores-ie.. the combined cost of importance and redundancy scores, (O(a + ( Buugge)))
C.2 Detailed Analysis of Throughput Results
We analyze the end-to-end throughput from two perspectives: ratio budget and fixed budget.
section 4.2 indicates that for DeepSeek-Rl-Disee -i---ll-------sssi---aaa------- lossless compression (i.e., model performance equivalent to KV compression) is achievable when the KV budget ratio, relative to the output length, is between 10% and 34%. For DeepSeck-RI- ---isiill--wen- range for lossless compression is 25% to 54% of the output length. Consequentlywe investigated the maximum achievable batch size and corresponding throughput for R-KV at compression ratios of 10%, 34%, and 54%, comparing these against the maximum batch size and throughput of FullKV using DeepSeek--RR RR -----ittlll-Llamaa 8BB In 8K sequence length setting, at a 54% compression ratio, R-KV allows for batch size 1.7 x larger than FullKV, resulting in 1.5 x the throughput. At a 10% compression ratio, K-KV achieves a 7.7 X increase in batch size and a 4.5 x increase in throughput compared to FullKV. For a 16K sequence length setting, at 54% compression, the batch size is 1.5 x that of FullK and the throughput is 1.7 x higher. At 10% compression, R-KV supports a 9 x larger batch size, delivering 6.6 x the throughput. We observe that for smaller batch sizes (e.g.. less than 128), throughput scales nearly linearly with increasing batch size. However, for larger batch sizes this linear scaling diminishes as inference on the NVIDIA A100 GPU becomes compute-boun bound.
Ratio Budget:
this
We also conducted an analysis under fixed KV cache budget. With an output length of 8K and fixed budget Brudget 1024, R-KV enables a batch size 6.48 x larger than VullKV, yielding 3.8 x the throughput. At Brudges 1536, the batch size is 4.6 x larger, and throughput is 3 x that of VullKV. For an output length of 16K and Brudger = 1024, R-KV achieves a 13.4 x increase in batch size and a 9.19 x increase in throughput. With Buddge = 1536, the batch size is 9.6 x larger, and throughput is 7.1 x higher. In the fixed budget scenario, the advantage of R-KV becomes more pronounced with longer generation lengths. This is because the KV cache size for R-KV under fixed budget does not increase with the sequence length, unlike FullKV where the memory footprint grows lincarly with the generation length, thus more severely limiting its maximum batch size.
Fixed Budget:
18

---

Length
Method
Budget
Saving (%)
Throughput (tok/)
(s)
75.44
8094
107.30
FullKY
(max)
849.
Fixed 1024
87.50
81.26
8094
99.60
1024
87,50
402 (max)
253.93
999,96
Fixed 1536
81.25
287 (max)
2,525.22
2.3229988
919.90
SnupKV
(max)
Ratio 10% - 819
90.00
479 (max)
3 808.81
3 026
1117.9
Ratio 34% 2
66.00
167
625,46
3
831.58
Ratio 54% 4423
46.00
105 (max)
269
849 870
669.36
Fixed
87.50
80.46
Fixed 1024
87,50
402 (max)
3 251.52
3233788 788
000.
Fixed 1536
81.25
287 (max)
2. 2.525.75
6546977
919.72
R-VV
Fixed 3072
62.50
1550 (max)
520.99
1214
798.23
Ratio 10% - 819
90.00
479 (max)
3 809.15
3 877 026
017.82
Ratio 34% 2 785
66.00
167 (max)
608.01
1351
840.61
Ratio 54% 4.233
46.00
105 (max)
257.83
849 870
675.66
FulKKY
69.41
166
234.65
30 (max)
347.03
488 580
407 89
Fixed 1024
87.50
81.03
16.286
200.99
Fixed 1024
87.50
402 (max)
3 202.17
6746972
044.55
Fixed 1536
81.25
287 (max)
2449.00
4674-0822
908
S
Fixed 3072
81.25
150 (max)
1413.84
2 2.442 900
727 84
Ratio
(max)
2.306.26
16K
Ratio 34% 55570
66.00
82 (max)
798.42
333 452
672.66
Ratio 54% 88847
46.00
(max)
$86.43
749 156
277.488
Fixed 1024
93.75
80.95
16:286
201.18
(max)
Fixed 1536
90.63
287 (max)
2-447.61
4674088
909 65
R-KV
Fixed 3072
81.25
150 (max)
1.06.26
2.442.00
737.13
Ratio 10% 638
90.00
(max)
2,300.28
4413 506
918.68
Ratio 34% 5:570
66.00
82 (max)
797.44
1
674.700
Ratio 54% 847
46.00
46 (max)
584.77
749 156
281.12
Table 3: Memory-saving throughput, and decoding time comparison for LLAMA3-8B under various generation lengths and KV-cache compression budgets.
C.3
Results
Full results could found at Table While R-KV incurs a minor computational overhead for redundancy estimation compared with SnapKV, this results in throughput that is only slightly lower, with negligible difference of less than 1%
D Limitations
One limitation of our proposed KV cache compression method is its current compatibility with certain advanced attention mechanisms, such as paged attention. Adapting our compression technique to seamlessly integrate with such mechanisms presents non-trivial challenge and may require further investigation. Additionally, the implementation of KV cache compression within existing serving frameworks encounter practical difficulties, particularly if these frameworks lack native support or flexible interfaces for KV cache compression. In serving frameworks that do not offer specialized KV cache compression interfaces, the performance benefits of method might be less pronounced. Without such interfaces, implementing KV cache compression may necessitate reallocating memory store the compressed KV cache and subsequently deallocating the memory used for the original, uncompressed cache. This process of memory reallocation can introduce significant overhead, potentially offsetting some of the acceleration gains. In contrast, serving frameworks equipped with dedicated KV compression interfaces can handle these operations much more efficiently, avoiding such costly memory management tasks.
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: abstract and introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitation
20

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: 5.2 Failure Attention- -Based Methods to Capture Redundancy
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: code open-source
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 7/nips. /ttppp.cc/pbllc/guudeeee http://oeeuuiiiiiioooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: MATH-500 and AIME24
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
Run experiments for 64 times and calculate averaged Pass 1,
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: A100
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Conclusion: As a training-free free and model-agnostic solution, R-KV offers practical path deploy advanced reasoning LLMs more efficiently and scalably.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
24

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
25

---

Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy http:e/r/mecrppppecc//oooeereeee for what should or should not be described.
26


Reviews:
[{"review_id": "1UpistQzCG", "reviewer": "Reviewer_evCh", "summary": "This paper introduces Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a method that computes and combines importance and redundancy scores for generated tokensthat guide their compression. The focus of R-KV is on decoding rather than prefilling stage with immediate application on the excessively long outputs of reasoning models. The core idea is managing two fixed size buffers for retained (budget buffer) and newly generated tokens and compressing between fixed-length text generation steps. In particular, a number of most recently generated tokens in the text are retained (observation tokens) and the rest tokens to fit in the budget buffer are selected by assigning scores to the remaining generated tokens and previously retained ones (candidate tokens). A token score is a linear combination of its importance (attention) score (where observation tokens serve as queries and a sliding window around a candidate token stabilizes the computation of the attention it receives) and its redundancy score (computed by averaging its key similarities to all other candidate tokens). \n\nIn the empirical evaluation of R-KV, authors typically report the accuracy on MATH500 and AIME 2024 datasets for various KV-cache budgets. Full KV-cache setup serves as the bare minimum baseline and SnapKV adapted to the same compression intervals R-KV is another (KV-cache compression) baseline; in most setups, models are Deepseek-R1 distilled ones (8B and 14B parameters). Reported results are impressive: in some configurations, R-KV can maintain the full KV-cache performance by using only 10% of the KV-cache, outperform it (still utilizing a fraction of the KV-cache) or achieve up to 90% memory savings and x4.9 speedup. Interestingly, the authors include a discussion on choosing the coefficients in the linear combination of the two scores and a nice illustration of the excessively hight scores other methods can put on very recent tokens or repetitive token subsets, however of minimal importance for eviction decisions.", "questions": "Can you elaborate a bit more on why choosing SnapKV as the only KV-cache compression method for comparison? There are many KV-cache compression methods, it is true that most focus on prefilling stage and need adaptation to be used in this context, but it would be nice to have some additional arguments. For example, where some other methods like H2O tried and they were found too unfit for this setup?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "+ Results are impressive and demonstrate that R-KV can be highly important in practice.\n\n+ This work is timely, given that (output) reasoning traces are extremely lengthy and there is a pressing need for their KV-cache to compressed. Ideas from compressing the KV-cache of (input) long contexts can certainly be leveraged, but here the setup is more dynamic.\n\n- There is room for improvement in the presentation: Basically some minor typos (e.g. in line 186 out of the three datasets mentioned only two are listed, in line 223-224 the reference should be to the 14B model) and most importantly the use of different symbols (alpha and lambda) for the coefficient in the combination of the two scores both in Figures and text (e.g. lines 240 and 243).", "was_revised": true, "final_justification": "This work is timely, reported results are impressive. Also authors provided satisfactory clarifications and additional empirical evidence during rebuttal as requested. Assuming that presentation improvements will be integrated in the final version, I am maintaining my original positive score (5: accept).", "rebuttal": "We thank the reviewer for the insightful feedback, and we would like to address a few concerns:\n\n> **Q1**: There is room for improvement in the presentation.\n\n**A1**: We appreciate the feedback and will improve the presentation in the revised revision.\n\n> **Q2**: Can you elaborate a bit more on why choosing SnapKV as the only KV-cache compression method for comparison? There are many KV-cache compression methods, it is true that most focus on prefilling stage and need adaptation to be used in this context, but it would be nice to have some additional arguments. For example, where some other methods like H2O tried and they were found too unfit for this setup?\n\n**A2**: We chose SnapKV as the primary baseline because it directly uses attention scores for KV cache eviction. Since our method builds upon this paradigm by introducing a hybrid selection metric that combines attention with redundancy signals, SnapKV—representing the strongest attention-based KV cache eviction strategy—is the most appropriate point of comparison. As discussed in Section 4.1, our improvements specifically target and enhance this class of methods, making SnapKV a natural and focused baseline.\n\nWe tried using StreamingLLM and H2O as baselines, but their performance was too poor and fell short of expectations. However, we agree that including comparisons with other recent decoding-time KV pruning methods—such as StreamingLLM, and H2O—would provide a broader empirical perspective. To address this, we have conducted additional experiments comparing R-KV with several of these recent baselines. Results show that R-KV consistently outperforms them across our budgets, highlighting its effectiveness in balancing memory reduction and performance preservation. We will include these results and their discussion in the revised version to strengthen the empirical scope of our work.\n\n**Deepseek-Distilled-Llama-8B - Math500**\n\n| Method                  | 128   | 256   | 512   | 1024  |\n|-------------------------|-------|-------|-------|--------|\n| StreamingLLM           | 21.8  | 29.4  | 42.8  | 54.4   |\n| H2O                     | 29.6  | 45.0  | 56.6  | 67.6   |\n| SnapKV        | 32.53 | 50.07 | 64.03 | 74.43  |\n| R-KV          | 51.08 | 67.39 | 76.92 | 81.34  |"}, {"review_id": "fU9ETELlLH", "reviewer": "Reviewer_cogD", "summary": "This paper addresses a critical challenge in deploying large language models (LLMs) for reasoning tasks: the excessive GPU memory consumption caused by long chain-of-thought (CoT) outputs. These verbose generations often lead to large key-value (KV) caches during autoregressive decoding. While prior work has attempted to reduce memory usage by selecting important tokens based on attention scores, this paper claims that this is insufficient.\n\nTheir key insight is that even among high-attention tokens, there is substantial redundancy. Many tokens are semantically repetitive and contribute little to final task performance. Retaining such redundant tokens crowds out more diverse and informative context, ultimately hurting reasoning accuracy.\n\nTo address this, the paper proposes **R-KV**, a **redundancy-aware KV cache compression method** that considers both:\n\n1. **Token importance**, estimated via attention scores;\n2. **Token redundancy**, measured through semantic similarity (cosine similarity of key vectors);\n3. A **joint selection strategy**, which balances these two aspects using a tunable parameter λ, to retain a subset of KV tokens that are both important and diverse.\n\nR-KV operates during the decoding stage---where output grows rapidly---and performs cache compression at fixed intervals. It maintains two memory buffers: a cache of size $B_{\\text{budget}}$ to store selected key-value tokens, and a buffer of size $B_{\\text{buffer}}$ to temporarily hold newly generated tokens. \n\nAfter generating each $B_{\\text{buffer}}$ tokens, the algorithm first preserves the most recent $\\alpha$ tokens (observation tokens) to ensure continuity. Then, from the combined pool of $B_{\\text{budget}} + B_{\\text{buffer}} - \\alpha$ candidate tokens, it selects the top $B_{\\text{budget}} - \\alpha$ tokens according to the joint importance--redundancy score. These tokens, along with the $\\alpha$ most recent ones, are retained for the next decoding step, effectively compressing the KV cache while preserving useful and diverse context.\n\nExperiments are conducted on two mathematical reasoning benchmarks: **MATH-500** and **AIME 2024**, using **DeepSeek-R1-Distill-Llama-8B** and **Qwen-14B** models. Results show that:\n\n- R-KV achieves near-parity or even **105% of full KV accuracy** using only **10–34%** of the original KV cache.\n- It **outperforms the prior state-of-the-art method SnapKV by up to 40%** in accuracy.\n- It enables **up to 90% memory savings**, **13× larger batch sizes**, and **9× higher throughput**, significantly improving inference efficiency under constrained memory settings..\n\nOverall, R-KV is a training-free and model-agnostic method that demonstrates strong effectiveness in both accuracy and throughput.", "questions": "Questions are included in the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. **The core idea makes a lot of sense**. The paper points out that just looking at attention scores isn't enough in reasoning tasks—many tokens that look \"important\" are actually just repetitive. By explicitly addressing both importance and redundancy, the method targets a real gap in existing work.\n2. **The method is clearly explained**. The paper does a good job of breaking down the algorithm into importance scoring, redundancy estimation, and joint selection. The decoding-time workflow—how tokens are kept or discarded—is easy to follow.\n3. **The results are strong**. R-KV consistently outperforms SnapKV in both accuracy and efficiency. Getting close to or even beating full KV cache performance while saving up to 90% memory and boosting throughput is impressive, especially without any model retraining.\n\nWeaknesses:\n\n1. **The baseline comparison is a bit narrow**. The paper only compares against SnapKV, while other recent methods like StreamingLLM, H2O, QUEST etc. work also perform token pruning during decoding. Although these are discussed in the related work section, it would be more convincing to include at least some of them in the experiments. Given that they share similar motivations (e.g., reducing KV growth during decoding), experimental comparisons would help clarify where R-KV stands among these alternatives.\n2. **All the experiments are on DeepSeek distilled models**. It’s unclear how well R-KV would generalize to other reasoning models like Qwen or QwQ. These models might generate differently, and redundancy patterns could vary.\n3. **Efficiency analysis setup isn’t fully transparent**. It’s not clear what attention backend is used, and since the method doesn’t currently support things like paged attention in vLLM, it’s hard to know how useful R-KV is in production setups where those are standard. A discussion or test under those conditions would help.", "was_revised": true, "final_justification": "I have read the authors' response, and I keep my score unchanged. The method, which combines importance and redundancy scores for pruning KV states, is practically effective. I give a positive score because we need more research into understanding which parts of the computations can be reused to reduce computational costs. I cannot give a higher score because the work is still quite empirical due to limited task and model evaluations. While it sounds intuitive, the idea of leveraging reasoning redundancy is quite limited in practice because, as LLMs get stronger, they seldom produce repetitive words, leaving limited room for improvement. This can be seen in the authors' response to evaluating a different model, where on QwQ, the method achieved diminishing gains as the budget increased.", "rebuttal": "We thank the reviewer for the insightful feedback, and we would like to answer a few questions:\n\n> **Q1**: The baseline comparison is a bit narrow\n\n**A1**: As noted in Section 4.1, we chose SnapKV as the primary baseline because it directly uses attention scores for KV cache eviction. Since our method builds upon this paradigm by introducing a hybrid selection metric that combines attention with redundancy signals, SnapKV—representing the strongest pure attention-based KV cache eviction strategy—is the most appropriate point of comparison. As discussed in Section 4.1, our improvements specifically target and enhance this class of methods, making SnapKV a natural and focused baseline.\n\nWe agree that including comparisons with other recent decoding-time KV pruning methods—such as StreamingLLM, and H2O—would provide a broader empirical perspective. To address this, we have conducted additional experiments comparing R-KV with several of these recent baselines. Results show that R-KV consistently outperforms them across budgets, highlighting its effectiveness in balancing memory reduction and performance preservation. We will include these results and their discussion in the revised version to strengthen the empirical scope of our work.\n\n**Deepseek-Distilled-Llama-8B - Math500**\n\n| Method                  | 128   | 256   | 512   | 1024  |\n|-------------------------|-------|-------|-------|--------|\n| StreamingLLM           | 21.8  | 29.4  | 42.8  | 54.4   |\n| H2O                     | 29.6  | 45.0  | 56.6  | 67.6   |\n|  SnapKV        | 32.53 | 50.07 | 64.03 | 74.43  |\n| R-KV          | 51.08 | 67.39 | 76.92 | 81.34  |\n\n> **Q2**: All the experiments are on DeepSeek distilled models\n\n**A2**: We tested R-KV on mathematical reasoning benchmarks using a different family of reasoning models—QwQ. R-KV consistently showed strong performance and effectiveness across these diverse reasoning tasks, highlighting its robustness and adaptability. Results are shown as below. We would include the results of the extended models in the revised version of our work.\n\n**QwQ**\n\n| - | MATH Pass@1 | - | AIME24 | - | AIME25 Pass@1 | - |\n|--------|------------------|--------------------|---------------------|-----------------------|---------------------|-----------------------|\n| budget | R-KV | SnapKV | R-KV | SnapKV | R-KV | SnapKV |\n| 128    | 0.4812           | 0.1522             | 0.0215              | 0.0021                | 0.0052              | 0.0011                |\n| 256    | 0.6476           | 0.334              | 0.0734              | 0.0063                | 0.0151              | 0.0031                |\n| 512    | 0.8031           | 0.5581             | 0.1479              | 0.0308                | 0.0802              | 0.0142                |\n| 768    | 0.8585           | 0.7476             | 0.2251              | 0.0604                | 0.162               | 0.0355                |\n| 1024   | 0.8897           | 0.8359             | 0.3062              | 0.1742                | 0.213               | 0.1088                |\n| 1536   | 0.9323           | 0.9032             | 0.4229              | 0.2833                | 0.2845              | 0.1891                |\n| 2048   | 0.9457           | 0.938              | 0.5354              | 0.4646                | 0.3396              | 0.2567                |\n| 2560   | 0.9559           | 0.9549             | 0.5729              | 0.5458                | 0.3718              | 0.3124                |\n| 3072   | 0.9623           | 0.9583             | 0.6014              | 0.5735                | 0.4033              | 0.3402                |\n\n> **Q3**: Efficiency analysis setup isn’t fully transparent\n\n**A3**: We used Flash-Infer and Flash-Attn as our attention backend during evaluation. In the revised version of our paper, we will also report the performance of R-KV when deployed in vLLM."}, {"review_id": "SIalqMhl0t", "reviewer": "Reviewer_vf9G", "summary": "The paper introduces R-KV(Redundancy-aware KV Cache Compression), a training-free KV cache compression method for reasoning models that produce long, redundant outputs. R-KV works by jointly considering token importance (via attention) and redundancy (via cosine similarity) to selectively prune the KV cache. Evaluated on two distilled models across two mathematical reasoning benchmarks, the method achieves performance comparable to the full KV cache with a significantly smaller memory footprint, leading to substantial memory savings and inference speedups.", "questions": "**Questions:**\n1. Unexplained Performance Boost. The paper reports that R-KV can surpass the full-cache baseline but does not provide a deep analysis of why this regularization-like effect occurs.\n2. Discrepancy in Baseline Performance. The reported baseline scores (FullKV) appear lower than\npublicly available benchmarks. For example, the paper reports 82.38% for Llama3-8B on MATH-\n500 and 65.68% for Qwen-14B on AIME24, while official or other reported scores are higher (e.g.,\n~89.1% and ~69.7%, respectively). This gap raises questions about the experimental setup and\ncould overstate the relative gains of R-KV.\n\n**Suggestions:**\n1. Provide stronger evidence for R‑KV’s redundancy elimination. The manuscript currently includes only a single case study to illustrate how R‑KV removes redundancy in long-chain-of-thought (CoT) reasoning, which may invite skepticism in the absence of further examples. We recommend that the authors bolster their claims with more compelling statistical analyses—such as an n‑gram redundancy analysis similar to that shown in Figure 2—to quantitatively demonstrate the method’s effectiveness.\n2. Typo. In Section 5.1 (line 231), the word \"overned\" appears to be a typo and should likely be \"governed\".\n3. Figure Inconsistency. The diagram in Figure 1 states \"Let λ = 0.5\". However, the paper's\nexperimental setup establish λ = 0.1 as the default value. It is suggested to update the figure to\nreflect λ = 0.1 for consistency with the main text.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Clear Problem Definition and Motivation. The paper precisely identifies and motivates a critical\nbottleneck in deploying reasoning models—the long, redundant outputs generated during the\ndecoding phase.\n2. Strong Empirical Performance. The method demonstrates significant effectiveness, achieving\nperformance comparable to, and in some cases surpassing, the uncompressed baseline while\nusing only a small fraction of the KV cache.\n3. Simple yet principled algorithm. The method is training-free and model-agnostic, making it an\neasy-to-deploy solution for improving inference efficiency without retraining.\n\n**Weaknesses**\n1. Limited Scope of Models and Tasks. The evaluation is limited to two distilled models on math\ntasks. Given the method is training-free and computationally light, its generalizability should be\nvalidated on a broader range of architectures and reasoning domains to strengthen its claims.\n2. Some parameter settings have not been specified in the paper. For example, neither β nor T is mentioned in the hyperparameter configuration.", "was_revised": true, "final_justification": "- Strengths: Problem is important, solution is simple, training-free, and now shown to transfer across two additional settings with compelling memory-/speed-to-accuracy trade-offs.\n- Weaknesses: Breadth of evaluation and analysis depth remain limited; baseline score gap introduces residual uncertainty about the true magnitude of gains.\n\nGiven the added evidence and clarifications, the paper’s contributions outweigh its shortcomings, but only narrowly. I therefore maintain a **borderline accept** recommendation. Strengthening the baseline reproduction and providing the promised redundancy analysis would move the work into clear-accept territory.", "rebuttal": "We thank the reviewer for the insightful feedback, and we would like to clarify a few points:\n\n> **Q1**: Limited Scope of Models and Tasks. \n\n**A1**: We conduct additional evaluations of R-KV on the GPQA benchmark, which targets scientific knowledge-based question answering. Despite the increased complexity of this domain, R-KV consistently outperforms the baseline method. Moreover, we extended our evaluation to a different family of reasoning models—QwQ—on mathematical reasoning benchmarks, further validating the robustness of R-KV across architectures. In all cases, R-KV demonstrates strong and reliable performance, underscoring its adaptability to diverse reasoning domains. The consistent superiority of R-KV across varied KV budgets and datasets further confirms its robustness. We will include these extended results in the revised version of the paper.\n\n**Deepseek-Distilled-Llama-8B - GPQA**:\n\n| Method-Budget | Overall Accuracy | Physics | Chemistry | Biology |\n|-------|------------------|---------|-----------|--------|\n| rkv-128 | 3.03% | 4.65% | 1.08% | 5.26% |\n| snapkv-128 | 0.00% | 0.00% | 0.00% | 0.00% |\n| rkv-256 | 17.17% | 22.09% | 9.68% | 31.58% |\n| snapkv-256 | 0.00% | 0.00% | 0.00% | 0.00% |\n| rkv-512 | 23.23% | 33.72% | 8.60% | 47.37% |\n| snapkv-512 | 1.52% | 3.49% | 0.00% | 0.00% |\n\n**QwQ**:\n\n| - | MATH Pass@1 | - | AIME24 | - | AIME25 Pass@1 | - |\n|--------|------------------|--------------------|---------------------|-----------------------|---------------------|-----------------------|\n| budget | R-KV | SnapKV | R-KV | SnapKV | R-KV | SnapKV |\n| 128    | 0.4812           | 0.1522             | 0.0215              | 0.0021                | 0.0052              | 0.0011                |\n| 256    | 0.6476           | 0.334              | 0.0734              | 0.0063                | 0.0151              | 0.0031                |\n| 512    | 0.8031           | 0.5581             | 0.1479              | 0.0308                | 0.0802              | 0.0142                |\n| 768    | 0.8585           | 0.7476             | 0.2251              | 0.0604                | 0.162               | 0.0355                |\n| 1024   | 0.8897           | 0.8359             | 0.3062              | 0.1742                | 0.213               | 0.1088                |\n| 1536   | 0.9323           | 0.9032             | 0.4229              | 0.2833                | 0.2845              | 0.1891                |\n| 2048   | 0.9457           | 0.938              | 0.5354              | 0.4646                | 0.3396              | 0.2567                |\n| 2560   | 0.9559           | 0.9549             | 0.5729              | 0.5458                | 0.3718              | 0.3124                |\n| 3072   | 0.9623           | 0.9583             | 0.6014              | 0.5735                | 0.4033              | 0.3402                |\n\n> **Q3**: Some parameter settings have not been specified in the paper.\n\n**A3**: We have used β = 8 and temperature T = 0.5 in all experiments. These settings will be explicitly stated in the revised version.\n\n> **Q4**: Discrepancy in Baseline Performance\n\n**A4**: In all evaluations (including FullKV), we followed the official reports by using the same sampling hyperparameters: temperature = 0.6 and top-p = 0.95, and reported the average pass@1 over 64 samples. However, we were unable to exactly reproduce the official results, likely due to unreported details such as prompt formatting, sampling seed control, or other subtle differences in the inference pipeline. Although we could not replicate the official results precisely, we emphasize that all comparisons in our work were made under a controlled and consistent inference setup. Importantly, all our experiments—including both R-KV and baseline comparisons—were conducted under the same hyperparameters and experiment setup, ensuring that the relative performance differences reported in our paper remain valid and meaningful.\n\n> **Q5**: Provide stronger evidence for R‑KV’s redundancy elimination\n\n**A5**: We agree that stronger empirical evidence would enhance the claims. In the revised version, we will include a detailed statistical analysis comparing the retained KV tokens between baseline methods and R-KV. Specifically, we will perform an n-gram redundancy analysis—similar to the one presented in Figure 2—to quantitatively evaluate how R-KV reduces repetitive information in the KV cache. This analysis will further substantiate the effectiveness of our redundancy-aware selection strategy.\n\n> **Q6 and Q7**: Typo and Figure Inconsistency\n\n**A6**: We acknowledge the inconsistencies and will ensure they are corrected in the revised manuscript to improve clarity and presentation quality."}, {"review_id": "OkzoPc2xuD", "reviewer": "Reviewer_Dks4", "summary": "This paper tackles the problem of excessive redundant information in the KV cache during long chain‑of‑thought reasoning in large language models by introducing R‑KV, a training‑free, decoding‑time plugin. R‑KV computes, for each cached token, an importance score based on attention weights and a redundancy score based on semantic similarity, then employs a tunable trade‑off coefficient to retain only the most informative and diverse tokens. As a result, it achieves reasoning accuracy comparable to—or even exceeding—that of the full KV cache using substantially less cache space, while significantly reducing memory footprint and accelerating inference.", "questions": "1. In Figure 5, we observe that the primary difference between the λ = 0.1 configuration and the Redundancy‑only variant lies in the preservation of Attention Sink tokens and those at the tail positions. Could one instead explicitly designate certain positional tokens to retain, while applying the redundancy metric exclusively to all other tokens?\n\n2. Since different backbone models exhibit varying degrees of inherent redundancy across datasets, how does this variability influence the efficacy and gains of the proposed method? Are there any deeper insights into how a model’s intrinsic redundancy profile interacts with R‑KV’s compression strategy?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- **Novel redundancy‑aware view of KV eviction.**\nBy introducing a combined importance‑and‑redundancy scoring mechanism, the paper overcomes the limitation of prior methods that rely solely on attention weights and thus fail to eliminate repetitious self‑reflections in the KV cache. This new perspective directly targets semantic duplication and yields a more information‑dense cache\n.\n\n- **Comprehensive and transparent experimental analysis.**\nThe authors evaluate R‑KV across multiple layers, λ‑settings and cache budgets, and directly compare against both vanilla attention‑based pruning and recent compression baselines. Clear plots (e.g. accuracy vs. cache size) and ablations (importance only vs. redundancy only vs. joint scoring) convincingly demonstrate the complementary benefits of each component\n.\n\n- **Lightweight, training‑free deployment.**\nR‑KV operates entirely at decoding time and requires neither re‑training nor architectural changes, making it immediately pluggable into existing Transformer‑based inference pipelines with minimal engineering effort and no model‑specific adjustments\n.\n\n**Weaknesses**\n\n- **Limited domain generalization.**\nExperiments are confined to mathematical reasoning benchmarks (MATH‑500, AIME‑24) and a single distilled model family (DeepSeek‑R1). The absence of evaluations on other reasoning tasks leaves open questions about R‑KV’s effectiveness beyond purely math domains.\n\n- **Unexamined hyperparameter sensitivity.**\nThe method depends on several tunable factors—redundancy threshold  T, recent‑token retention β—yet the paper lacks a systematic study of how robust performance is to these settings or guidance for selecting them in new contexts.\n\n- **Figure annotation errors.**\nSome plot legends and symbols differ from those used in the text, and certain figures contain outright errors. For example, in Figure 1 the “12” below “decoding” should read “14,” and in Figures 5 and 6 the legend erroneously spells λ as α.\n\n- **Compatibility with other techniques.**\nWhile R‑KV offers substantial speed and memory gains, its interaction with other techniques (e.g. quantization approaches) is not discussed. Potential conflicts or synergies with these established techniques warrant further exploration.", "was_revised": true, "final_justification": "**Resolved Issue**\n- The rebuttal convincingly strengthens the paper on generalization and usability while leaving only moderate, non-blocking gaps in hyper-parameter coverage and technique interplay.\n- Given the novelty of the redundancy-aware perspective, the clear empirical benefits, and the now-broader evidence base, the merits outweigh the remaining weaknesses.\n\n**Remaining Issues**\n- Pending empirical validation of interoperability with quantization.\n\nBalancing these factors, I raise my rating from 3 (weak reject) to **4 (borderline accept)**.", "rebuttal": "We thank the reviewer for the insightful feedback, and we would like to address a few concerns:\n\n> **Q1**: Limited domain generalization\n\n**A1**: To address generalization beyond the original evaluation scope, we conducted additional experiments on the GPQA benchmark, which emphasizes scientific knowledge-based question answering. R-KV substantially outperformed the baseline method. These results clearly indicate that R-KV consistently outperforms SnapKV across both scientific QA and mathematical reasoning tasks. We also evaluated R-KV on a mathematical reasoning benchmark using a different family of reasoning models—QwQ—and observed consistently strong performance. These results demonstrate that R-KV is robust and effective across diverse reasoning domains and model families. We will include these extended evaluations in the revised version of the paper.\n\n**Deepseek-Distilled-Llama-8B - GPQA**:\n\n| Method-Budget | Overall Accuracy | Physics | Chemistry | Biology |\n|-------|------------------|---------|-----------|--------|\n| rkv-128 | 3.03% | 4.65% | 1.08% | 5.26% |\n| snapkv-128 | 0.00% | 0.00% | 0.00% | 0.00% |\n| rkv-256 | 17.17% | 22.09% | 9.68% | 31.58% |\n| snapkv-256 | 0.00% | 0.00% | 0.00% | 0.00% |\n| rkv-512 | 23.23% | 33.72% | 8.60% | 47.37% |\n| snapkv-512 | 1.52% | 3.49% | 0.00% | 0.00% |\n\n**QwQ**:\n\n| - | MATH Pass@1 | - | AIME24 | - | AIME25 Pass@1 | - |\n|--------|------------------|--------------------|---------------------|-----------------------|---------------------|-----------------------|\n| budget | R-KV | SnapKV | R-KV | SnapKV | R-KV | SnapKV |\n| 128    | 0.4812           | 0.1522             | 0.0215              | 0.0021                | 0.0052              | 0.0011                |\n| 256    | 0.6476           | 0.334              | 0.0734              | 0.0063                | 0.0151              | 0.0031                |\n| 512    | 0.8031           | 0.5581             | 0.1479              | 0.0308                | 0.0802              | 0.0142                |\n| 768    | 0.8585           | 0.7476             | 0.2251              | 0.0604                | 0.162               | 0.0355                |\n| 1024   | 0.8897           | 0.8359             | 0.3062              | 0.1742                | 0.213               | 0.1088                |\n| 1536   | 0.9323           | 0.9032             | 0.4229              | 0.2833                | 0.2845              | 0.1891                |\n| 2048   | 0.9457           | 0.938              | 0.5354              | 0.4646                | 0.3396              | 0.2567                |\n| 2560   | 0.9559           | 0.9549             | 0.5729              | 0.5458                | 0.3718              | 0.3124                |\n| 3072   | 0.9623           | 0.9583             | 0.6014              | 0.5735                | 0.4033              | 0.3402                |\n\n> **Q2**: Unexamined hyperparameter sensitivity\n\n**A2**: We conducted additional experiments to examine the sensitivity of our method to the recent-token retention parameter β. Our findings indicate that performance is indeed sensitive to β, with the optimal setting typically around 12.5% of the KV budget. We will include a more comprehensive analysis of this and other key hyperparameters in the revised version, along with practical guidance for selecting them.\n\n| R-KV Budget | Window Size | Window/Budget (%) | Value |\n|-------------|-------------|--------------------|--------|\n| 128         | 8           | 6.3%               | 47.0   |\n| 128         | 16          | 12.5%              | 48.0   |\n| 128         | 32          | 25.0%              | 46.4   |\n| 128         | 64          | 50.0%              | 42.0   |\n| 256         | 16          | 6.3%               | 64.6   |\n| 256         | 32          | 12.5%              | 65.4   |\n| 256         | 64          | 25.0%              | 64.4   |\n| 256         | 128         | 50.0%              | 62.0   |\n\n> **Q3:** Figure annotation errors\n\n**A3**: We appreciate the reviewer pointing this out and will correct the annotation errors in the revised version.\n\n> **Q4**: Compatibility with other techniques\n\n**A4:** We will include experiments and analysis of R-KV's compatibility with other techniques including quantization in the revised version.\n\n> **Q5**: Design based on Figure 5\n\n**A5**: We agree that explicitly designating certain positional tokens to retain—such as attention sink tokens or tail-position tokens—while applying the redundancy metric to others could potentially yield good results for specific models. However, the optimal number and position of tokens to preserve may vary significantly across different models and tasks. Additionally, tokens with large attention scores can also emerge at intermediate positions, depending on the model and input characteristics. R-KV offers a flexible and elegant alternative by eliminating the need for manual specification of such positions. Instead, it adopts a unified strategy based on real-time analysis during inference that consistently applies across all tokens, thereby avoiding model-specific or position-specific heuristics. This generality enables R-KV to adapt more broadly and reduces the need for task-specific tuning.\n\n> **Q6**: How does the model's intrinsic redundancy influence R-KV's performance\n\n**A6**: To clarify, by models’ inherent redundancy across datasets, we refer to the degree of output redundancy exhibited by a model when applied to different datasets. In our observations, a reasoning-oriented model typically shows a consistently high level of redundancy across datasets, whereas non-reasoning models exhibit little to no such redundancy. This suggests that redundancy is more tightly tied to the model parameters, rather than the dataset alone. We have used both a reasoning model and a non-reasoning model on the Math dataset, evaluating performance under different values of the redundancy threshold λ. The results indicate that for non-reasoning models, introducing redundancy control yields minimal to no performance improvement. In contrast, the reasoning model benefits significantly from R-KV, showing large gains under the same settings. These findings suggest that R-KV is particularly effective when applied to models with higher intrinsic redundancy. This aligns with our hypothesis that the more redundant the model’s output is, the greater the compression and performance benefits R-KV can provide. We will include this discussion and the supporting results in the revised version."}]

Author rebuttal:

Meta-review: **summary**
This paper introduces R-KV, a training-free method for compressing the Key-Value (KV) cache in large language models during inference on reasoning tasks. The core idea is to prune the cache by jointly considering both token importance (from attention scores) and token redundancy (from semantic similarity of key vectors). This dual-criterion approach aims to create a more information-dense cache, improving inference efficiency without sacrificing accuracy.

**strengths**
- The method's core idea of targeting redundancy in addition to importance is novel and addresses a clear limitation of prior attention-only eviction strategies.
- The empirical results are strong, demonstrating that R-KV can match or even exceed the performance of a full KV cache while using only a small fraction of the memory, leading to significant speedups.
- The approach is lightweight, training-free, and model-agnostic, making it a practical and easily deployable solution for accelerating LLM inference.

**weaknesses**
- The initial evaluation was limited in scope, focusing on a specific family of models and mathematical reasoning tasks, which raised questions about generalizability.
- The paper's initial baseline comparison was narrow, primarily focusing on SnapKV while omitting other relevant methods like StreamingLLM or H2O.
- The manuscript lacked a thorough analysis of key hyperparameters and their sensitivity, which is crucial for reproducibility and practical application.

**final descision**
The paper presents a novel, practical, and effective method for a significant problem. The strong empirical results, combined with a successful rebuttal that broadened the experimental scope, justify its acceptance.
Decision: Accept (poster)