ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
Authors: Hanshi Sun
Li-Wen Chang
Wenlei Bao
Size Zheng
Ningxin Zheng
Xin Liu
Harry Dong
Yuejie Chi
Beidi Chen
Venue: ICML 2025 spotlightposter (2025)
Keywords: Long-Context LLMs
Inference
KV Cache Optimization
Primary area: deep_learning->large_language_models
TL;DR: High-Throughput Long-Context LLM Inference System

Abstract: With the widespread deployment of long-context large language models (LLMs), there has been a growing demand for efficient support of high-throughput inference. However, as the key-value (KV) cache expands with the sequence length, the increasing memory footprint and the need to access it for decoding both result in low throughput when serving long-context LLMs. While various dynamic sparse attention methods have been proposed to accelerate inference while maintaining generation quality, they either fail to sufficiently reduce GPU memory usage or introduce significant decoding latency by offloading the KV cache to the CPU. We present ShadowKV, a high-throughput long-context LLM inference system that stores the low-rank key cache and offloads the value cache to reduce the memory footprint for larger batch sizes and longer sequences. To minimize decoding latency, ShadowKV employs an accurate KV selection strategy that reconstructs minimal sparse KV pairs on-the-fly. By evaluating ShadowKV on benchmarks like RULER, LongBench, and models such as Llama-3.1-8B and GLM-4-9B-1M, we demonstrate that it achieves up to 6$\times$ larger batch sizes and 3.04$\times$ higher throughput on an A100 GPU without sacrificing accuracy, even surpassing the performance achievable with infinite batch size under the assumption of infinite GPU memory.

SHADOWKV: KV Cache in Shadows for -igh-Throoue Throughput Long-Context LLM Inference
Hanshi Sun 12
Li-Wen Chang
Wenlei Bao'
Size Zheng' Ningxin Zheng'
Xin Liu
Harry Dong?
Yuejie Chi
Beidi Chen
Abstract
Theoretical Equivalent Bandwidth for ShadowKV
With the widespread deployment of long-context large language models (LLMss, there has been growing demand for efficient support of high- throughput inference. However, as the key-value (KV) cache expands with the sequence length, the increasing memory footprint and the need to ac- cess it for decoding both result in low throughput when serving long-context LLMs. While vari- dynamic sparse attention methods have been proposed to accelerate inference while maintain- ing generation quality, they either fail to suffi- ciently reduce GPU memory usage or introduce significant decoding latency by offloading the KV cache to the CPU. We present SHADOWK KVV, a high-throughput long-context LLM inference sys- tem that stores the low-rank key cache and of- floads the value cache to reduce the memory foot- print for larger batch sizes and longer sequences. To minimize decoding latency, SHADOW K V em- ploys an accurate KV selection strategy that re- constructs minimal sparse KV pairs on-the-fly. By evaluating SHADOWKV on benchmarks like RULER, LongBench, and models such as Llama- 3.B-8B and GLM-4-9B- IM, we demonstrate that it achieves up to 6x larger batch sizes and 3.04 higher throughput on an A100 GPU without sacri- ficing accuracy, even surpassing the performance achievable with infinite batch size under the as- sumption of infinite GPU memory.
31.5 GB/s (PCle)
TB/s (A100)
7 TB//
Acc. on 128K Mulli-turn NIAH
0.8
0.6
0.4
ShadowKV
Dynamic Cache
Static Cache
100.0
6.25
1.56
0.39
Sparse KV Cache Budget (%)
Figure I. SHADOWKV effectively utilizes limited KV budget achieve high accuracy, theoretically reaching over TB/s equiva- lent bandwidth A100 GPU.
(Achiam et al., 2023; Team et al., 2023; Microsoft, 2024; Liu et al., 2024a), enabling them to tackle complex tasks like multi-document question answering and information re- trieval from extensive contexts of up to IM tokens (Achiam et al., 2023; Wang et al., 2024b). However, efficiently serv- ing these long-context LLMs presents challenges related to the key-value (KV) cache (Ge al., 2023; Liu al., 2024b), which stores previous key-value activations avoid re-computation. As KV cache scales with sequence length, its growing memory footprint and the need to access it for each token generation lead to low throughput during long- context LLM inference. To address these, KV eviction or sparse attention methods have been widely explored.
1. Introduction
Large language models (LLMs) have increasingly demon- strated their ability to scale and handle long contexts
However, existing methods face three primary limitations: accuracy degradation, inadequate memory reduction, and significant decoding latency overhead. KV cache eviction strategies (Zhang et al., 2024d:c) aim to reduce the memory footprint by discarding KV pairs based on specific poli- cies, but they often result information loss and accuracy degradation in tasks such as multi-turn conversations (Yang
ByyteDance Seed "Carnegie Mellon University. Correspon- Beidi Chen
dence Hanshi Sun srunshi.s@brtedances <beidic mnnrrw.emm.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SHADOWKV: KV Cache in Shadows for High- High-Throuughut Long-Context LLM Inference
10
1.00
7000
FFN
Altentiio
Relaavv Singular Value
Sulspato Simllarty
0.95
6000
percenttage
10"
5.000
0.90
Time (m))
S0D 00 conte
10
0.85
3000
X
0.80
Context
2000
(ore-RoPPE) K (oost-RoPE)
Wk
Extended Context
1000
Wy
0.75
Inter-context
400 Component
800
1000
0-
64K
120K
152K
25
384K
Layer Index
Sequence Length
Figure 2. Left: For sample from PG-19 (Rae al., 2019) fed into Llama-3.1-88 the pre-RoPE keys are the most low-rank, as indicated by the sharpest decay in singular values. Middle: Average similarities, defined in Section 3.1, between rank-256 truncated SVD projections of pre-RoPE keys from PG-19 sequences using Llama-3.1-8P Similarity is measured between length 16K "Context" and either 16K+2K continuation on "Context" ("Extended context") or new length 16K sequence ("Inter-context") Pre-RoPE keys share low-rank subspaces within sequences but differ across sequences. Right: The relative overhead of singular value decomposition (SVD) decreases sequence length scales for the pre-filling stage.
al., 2024b; Tang et al., 2024a). Dynamic sparse attention methods (Tang et al., 2024b) preserve all KV pairs the GPU and accelerate inference by computing attention with selected KV pairs. However, this line of work does not mitigate the memory footprint, thereby limiting the batch size and preventing accommodation of extremely long con- texts (e.g.. 1M tokens). A naive solution based on sparse attention involves offloading the KV cache to the CPU to reduce memory usage (Lee et al., 2024a; He & Zhai, 2024). Nonetheless, as shown in Figure 4, this approach incurs sig- nificant overhead due to the latency of fetching the selected sparse KV pairs from the CPU during decoding.
of low-rank decomposition during pre-filling negligible, as illustrated in Figure 2'. To reduce memory footprint, retain the low-rank pre-RoPE key cache on the GPU and offload the value cache to the CPU since the value cache does not exhibit low-rank properties, minimizing memory footprint without sacrificing accuracy. During decoding with sparse attention, we employ CUDA multi-streams to overlap the recovery of the selected key cache with the fetching of the corresponding value cache. This approach conceals key cache reconstruction and reduces data fetching overhead by 2 compared to the naive offloading strategy, thereby decreasing the decoding latency of sparse attention.
Consequently, an ideal system for long-context LLM infer- ence with sparse attention should: (i) reduce GPU memory usage, (ii) minimize inference latency, and (iii) maintain ac- curacy within limited sparse KV cache budgets. Fortunately, we can potentially overcome these challenges by leveraging our discovery that pre- Rotary Position Embedding (Su et al., 2024) (RoPE) keys are exceptionally low-rank compared to the layer inputs, Rost-RoPE keys, values, key weight matrix, and value weight matrix, as indicated in Figure 2. Unlike prior works that apply low-rank approximations to weights (Chang al., 2024; Lee al., 2024b), we directly compress pre-RoPE key cache, achieving higher accuracy. Our analy- sis reveals that Rre-RoPE keys lack significant similarities in low-rank subspaces across different sequences, while a se- quence and its continuation tend to strongly share low-rank subspaces, enabling high compression rates within each se- quence. Motivated by these findings, we developed two key insights that pave the way for the design of an applicable system, detailed in Section 3.
Accurate KV Selection for Fast Decoding: To further re- duce decoding latency in sparse attention, we propose an accurate KV selection method that maintains accuracy with minimal number of selected tokens (i.c., the K of TopK), which we refer to as sparse budgets (1.5 (1.56%). Our analysis reveals that most Rost-RoPE keys exhibit high cosine simi- larity with adjacent tokens, enabling chunk-level approxi- mations for selecting important tokens. A minimal number of outlier chunks (0.3%), which are more challenging to ap- proximate (Figure 5), are stored static cache on the GPU to preserve accuracy. As shown in Figure 1, method outperforms the naive sparse attention approach (Tang et al., 2024b) and achieves higher sparsity, accelerating decoding.
Building on these insights, we present SHADOWKV in Sec- tion 4, depicted Figure 3, a high-throoughut system for long-context LLM inference. Specifically, during pre-filling offload the value cache to the CPU, retaining only the low-rank pre-Rot RoPE keys, along with compressed landmarks of the key cache and detected outliers for larger batch sizes.
Low-rank Keys and Offloaded Values for Storage: In long- context LLM inference, the quadratic scaling of attention computation with sequence length makes the linear cost
"In practical scenarios, the key cache can be offloaded to the CPU perform SVD asynchronously or precomputed and stored as part the prefix cache (Juravsky et al., 2024).
2

---

SHADOWKV: KV Cache in Shadows for High- High-TThroughut Long-Context LLM Inference
Pre-filling
Decoding
SVD
Low-rank Key Cache (Cached)
Landmarks
Low-rank Key Cache
Outliers
Pre-RoPE Key Cache
Coche
RoPE Reduce
GPU
Landmarks (Cached)
KV Q Sel.
Cache Hit/Miss
Low-rank Key Cache Reconstruction RoPE
Sparse Attention
Find Outliers
Outliers (Cached)
CPU
Offlood
Value Cache
Chunk IDs
Selected Missed
Value Cache Fetching
Figure 3. SHADOWKV enhances long-context LLM inference throughput by offloading the value cache to the CPU while maintaining low-rank key cache, landmarks, and outliers on the GPU. During decoding, it employs landmarks for efficient sparse attention, reducing computation and data movement.
During decoding, landmarks used to select chunk in- dices for key cache recovery and value cache fetching. We perform accurate sparse attention computation with selected KV pairs and static outliers to achieve high throughput.
(a) Prefetch Full KV
time
KV Fotching
(b) Fetch Sparse KV (e.g.
afflsading))
Sel.
Adtn
FFN
time
Empirically, we conduct extensive experiments and abla- tion studies demonstrate the effectiveness and efficiency of SHADOWKV, V. In Section 5.1, evaluate across vari- long-context LLMs, such as Llama-3-8B-IM (Gradi- ent., 2024), Llama-3. 1-8B (Meta AI, 2024), GLM-4-9B-IN (GLM et al., 2024), Yi-9B- B-0KK (AI et al., 2024), Phi-3- Mini-128K (Abdin et al., 2024) and Qwen2-7B-1288 (Yang et al., 2024a) using benchmarks including RULER (Hsich et al., 2024), LongBench (Bai et al., 2023), and Needle In A Haystack (Kamradt, 2023) with contexts up IM.
KV Fetching
Prefetch Sparse KV (e.g. InfiniGen)
Attn
Attin
timp
KV Fotching
(d) ShadowKV (Ours)
Reduction
Sel.
Recon.
FFN
time
Fetching
Figure 4. SHADOWKV outperforms prior works (Tang et al., 2024b; Lee al., 2024b) and scales better larger KV.
In Section 5.2, we demonstrate that SHADOWKY KV can sup- port 6x larger batch sizes and boost throughput by up to 3,04x compared to small batches on an A100 using Llama- 3.1-8B. We also present results across different models and context lengths, increasing throughput up to 2.97x for LlaMa-3-8B-IM, 2.56x for GLM-4-9B-IM, and 2.66x for Yi-9B-200K, even surpassing infinite batch size under the assumption of infinite GPU memory. The code is avail- able at https:/ /gttthb..... ctt. ShadowKV.
they suffer from performance degradation and information loss since the evicted tokens will never be recovered.
Dynamic Sparse Attention. This line of work retains all KV cache but performs dynamic sparse attention within selected KV pairs to reduce inference latency. SparQ (Ribar et al., 2023) uses the norm of the query to decide an impor- tant subset of the key cache's ' channels to calculate a metric to select relevant tokens. Quest (Tang et al., 2024b) seg- ments tokens into pages and selects pages by approximating the highest attention within each page. Loki (Singhania et al., 2024) performs principal component analysis key caches using calibration dataset, selecting tokens based on attention scores computed in low-dimensional space. Tri- Force (Sun et al., 2024a) combines sparse attention with speculative decoding (Leviathan et al., 2023) for lossless acceleration. InfiniGen (Lee et al., 2024a) offloads the entire KV cache the CPU and prefetches essential entries using predefined projections via SVD for KV selection. In con- trast, SHADOW KV employs online, prompt-depended SVD for key cache compression, reducing data fetching. As shown in Figure 4, SHADOWKV scales better, as overlap- ping KV fetching and computation becomes challenging with larger KV caches. Additionally, InfiniGen suffers per- formance drops due to inaccurate prefetching KV selection.
2. Related Works
Token Eviction. To reduce memory footprint, eviction- based strategies keep a fixed size of KV cache to store the critical token KV pairs and discard unnecessary tokens. Streaming (Xiao et al., 2023b) addresses the limita- tions of window attention by retaining attention sinks and recent KV pairs. H₂O (Zhang et al., 2024d) introduces a low-cost eviction policy, updating the KV cache based on cumulative attention scores. LESS (Dong al., 2024b) accumulates evicted token information by constant-sized low-rank cache, which allows partial access evicted in- formation, along with tokens maintained by sparse policy. SnapKV (Li et al., 2024) uses the local window of prompts to select important tokens for future generations. However,
3

---

SHADOWKV: KV Cache in Shadows for High- Sigh-Thrrughhut Long-Context LLM Inference
1.0
1.0
0.9
Mirimum Sinilarity within Chunk
0.8
0.8
0.8
0.6
KV Cache Hi Rate
0.7
Accuras
0.6
0.6
0.4
0.5
0.2
Value Cache
0.0
Layer-0, Head-1
0.4
Layer-5, KV Head-O
Post-RoPE Key Cache
Layer-16, Head-7
Layer-15, KV Head-3
Pre-RoPE Key Cache
-0.2
Outlier Chunks
0.3
Layer-25, KV Head-7
0.0
256
512
768
1024
4096
8192
12288
16384
50
100
150
200
250
SVD Rank
Chunk Index (Chunk Size=B)
Generated Token Index
Figure 5. Left: Accuracy on the needle retrieval across various ranks shows that the pre-RoPE key cache can be compressed by over 6 times without drop in accuracy. Middle: The number of notable outlier chunks is small, taking only 0.2-0.3%. Right: The KV cache has a high hit rate, reducing computations and data movements by over 60% for each decoding step.
Quantization. Several methods have been introduced to optimize KV cache quantization (Hooper et al., 2024; Yue et al., 2024; Xiao et al., 2023a), reducing memory con- sumption while retaining accuracy. KIVI (Liu et al., 2024c) applies different quantization strategies for keys and values, quantizing the keys per-channel and the values per-token to 2-bit. Palu (Chang et al., 2024) decomposes KV weight ma- trices offline, caching low-rank KV projections to achieve higher compression rate. Quantization methods reduce the KV cache bit width, which is orthogonal to our approach.
rank subspaces of two rank-r projection matrices, H1 and H2, where (5.) is the Frobenius inner product". In our case with truncated SVDs of pre-RoPE keys, let K1, ,, K2 E Radd have rank-r truncated SVDs, & V and Φ Eg 2 respectively, where Φ1 E Rexr, En Rexr, V
Rdxx, and similarly for 42, Σz, and 222 Then, D I 12 tween the low-rank subspaces of the two right singular matrices. Depicted in Figure 2, pre-RoPE keys between sequences do not strongly share similar low-rank subspaces, but extensions of the same sequence do. Thus, applying low- rank approximations weights alone degrades performance.
measure the similarity be-
3. Observations
We present two key insights of long-context LLMs that inspire SIAADOWKV's design, as follows.
Insights. Our observation of the low-rank nature in the pre-RoPE keys indicates that storing the low-rank projec- tions is sufficient for each sequence. By keeping the low- rank key cache on the GPU and offloading the value cache to the CPU since it not low-rank, we can largely reduce the memory footprint. During decoding, selected KV pairs can be reconstructed on-the-fly for computation.
3.1. Low-Rank Keys and Offloaded Values for Storage
To reduce memory footprint, the low-rank nature of the KV cache has been explored by recent studies (DeepSeek-AI, 2024; Xu et al., 2024; Chang et al., 2024; Saxena et al., 2024; Yu et al.. 2024; Lin et al., 2024). However, these methods focus on data-independent decomposition, either requiring training or achieving limited compression rates.
3.2. Accurate KV Selection for Fast Decoding
To further reduce the latency overhead in sparse attention, including fetching the selected value cache from the CPU and reconstructing the corresponding key cache, an accurate KV selection method is needed to minimize the sparse KV cache budget while maintaining the accuracy.
Observation. In our study, we visualize the relative sin- gular value distributions of the model weights Wa. We, the input X, the pre-/post-RoPE key cache, and the value cache of Llama-3.1-8B by conducting SVD, as shown in Figure 2. In Figure 5, we further analyze the accuracy impact. We ob- serve that Pre-RoPE keys exhibit the lowest rank, allowing for 6x compression without performance degradation.
Observation.
We found most Rost-RoPE key cache ex- hibits spatial locality, with high cosine similarity to adjacent
"Since Hi and H are projection matrices, their squared Frobenius
the of their singular values which consist of and d-r O's, i.c. ||H 1 (D(H,, Hz)l < 1. Additionally, D(H,, H2) 2 0 by the cyclic property of trace and positive semidefiniteness of projection ma- trices. Together, this shows D 1. H2) E [0. 1], maximized or minimized when the projection matrices project onto identical orthogonal subspaces, respectively.
We also identify striking dynamic and static behaviors in low-rank keys between and within sequences, inspired by a related investigation in FFN layers (Dong et al., 2024a). Analogous to cosine similarity, we define D(H,, H2) =
Thus, by Cauchy- Schwarz,
(H,, H2)/r r to be the similarity metric between low-

---

SHADOWKV: KV Cache in Shadows for High- Sighohhhrughhut Long-Context LLM Inference
Algorithm 1 SHADOWKV Pre-filling
Algorithm 2 SHADOWKV Decoding
Input: K, K KRROPE V Rbxhooxeed, SVD rank r,
Input: A. B, L, VCPU Q Rbxhgxx,xd Koullee
chunk size e, number of outlier chunks o
V Vaulier, K, V.V. RDbhhuxxaxd number of chunks ne-
b. Store low-rank projection of pre-RoPE key cache
number of selected chunk budget k
A E Rbxsxr, -B E IRbxbeuXXe xrxd + SVD(K)
> Commute chunk attention score
b. Segment post-RoPE key cache into chunks and compute
P E Rbxh, Xeahh,e8xx0 XxgxXe t- MatMul QQ L
the mean of each chunk
S E abxhyxxyXn + Softmax (P/vd)
C E B xafexd t Reduce (Kricee(RRan)) R
S E Rbxh,xn + (S, dim -2)
Compute cosine similarity within each chunk
So E Rbxhauxne t MaXXxy_group (S1)
S E Roxh Xs/exe + CosineSimilarity
K-OPEE)
Select top-k chunks for each KV head
Find lowest cosine similarity outliers
I E Bbxhi. + ArgTopK ())
I E Rixhau Xo + ArgTopK (MMin( SS dim = - 0)
> Gather value cache from the CPU
K Kouutier Voutier <- Gather( K:RoPE V.I)
Vsparse + Gather VCCUU : ,I) V <- -Voouliar, V V]
b Offload the rest of values to the CPU and store the non-outlier chunks' mean as landmarks
D Reconstruct key cache from low-rank projection
K sparse MatMul(Gathed A, I), B)
VCU VV Vouller, L t C Gather(C,,))
K
K Koulieer RoPE K sparse ): KK
tokens, except for few outliers. To quantify this, we con- ducted inference experiments on 128K contexts. We divided the Post-RoPE keys into chunks of eight tokens and visu- alized the minimum cosine similarity between the chunk's 's mean and its key cache. The results indicate that, apart from few outliers, there is generally high cosine similar- ity, suggesting the mean values can serve as landmarks to approximate attention well within normal chunks.
volves low-rank decomposition of the post-RoPE key cache, offloading the value cache, and constructing landmarks to facilitate subsequent high-throughput decoding. The de- coding phase includes accurate KV selection and efficient sparse KV cache reconstruction.
Pre-filling. During the pre-filling phase, we optimize GPU memory usage by performing low-rank compression on the key cache of each layer and offloading values to the CPU, Specifically, as demonstrated in Algorithm 1, we apply SVD on the pre-RoPE key cache and store only the low-rank representations for each layer. Post-RoPE key cache is seg- mented into chunks, with the mean of each chunk computed as landmarks. By computing the cosine similarity within these chunks, we identify poorly approximated tokens as outliers. This small set of outliers is gathered and stored on the GPU as the static cache, while the remaining key cache is maintained as compact landmarks, with the corresponding values offloaded to the CPU memory.
Analysis. This finding suggests that for the majority of chunks, we can maintain the mean value as compressed landmarks to select minimal important KV pairs (1.56%) accurately during decoding. Outlier chunks, which may contain dense or critical information and are difficult to approximate, are retained to ensure accuracy, Given their relatively small number (0.2-0.3%), storing them the GPU is feasible without affecting memory capacity. Fur- thermore, as shown in Figure 5, considering the temporal locality of the KV cache-meaning that the KV pairs se- lected by the queries of two adjacent decoding steps have a high repetition rate, a cache policy (Zhang et al., 2024a) can be leveraged to further reduce the latency overhead by 60% during decoding with optimized CUDA kernels.
tighh-throughut Decoding. For incoming queries, we first compute the approximate attention scores using the landmarks. As detailed in Algorithm 2, by identifying the top-k scoring chunk indices, the corresponding values retrieved from the CPU, and the key cache is simultane- ously reconstructed from low-rank projections, effectively concealing the construction of the key cache. Based on the insight that the KV cache has temporal locality, we conduct an index sean to detect the missed chunks and only rebuild the necessary KV pairs on-the-fly, reducing computation and data fetching by 60% with optimized CUDA kernels.
4. SHADOWKV
In this section, we introduce SHADOWKV, a novel high- throughput long-context LLM inference system. We first elaborate our algorithm in Section 4.1, covering both the pre- filling and decoding phases. Subsequently, in Section 4.2, we discuss the concept of theoretical equívalent bandwidth to illustrate the benefits of our approach.
Based on our observations in Section 3. future pre- Re-R keys within a sequence reside in shared low-rank subspace with the context. As a result, an extension of our algorithm would be to store generated tokens as low-rank states using the same low-rank projections obtained from pre-filling to
4.1. Algorithm
The algorithm of SHADOWKV is divided into two main phases: pre-filling and decoding. The pre-filling phase in-
5

---

SHADOWKV: KV Cache in Shadows for High-Throughhe Long-Context LLM Inference
Table 1. Performance comparison of different models and methods RULER (Ieft side of the tahle) and LongBench (right side of the table). SHADOWKV outperforms other methods and maintains the accuracy.
Methods
$1
$2
MK1
MK2
MQ
MV
QA-1
QA-2
VT FWE|
Avg.
NQA
MQA
IIQA
MQue
DRead
GRep
SAM
PRetr
LCC
Avg.
Llama-3 3.80.18. I.
100.00
100.00
98.
96
98.96
98.96
95.57
75.00
48.96
78.54
71.85
86.68
18.98
41.84
36.79
21.47
31.93
34.18
35.96
81.50
56.07
39.86
Loki
18.75
1.04
2.08
0.00
1.56
0.78
4.17
13:54
26.04
25.35
9.33
2.26
10.19
5,48
3.16
12.17
28.97
7.84
40.52
31.44
15.78
Loki (V)
41.67
6.25
37.50
1.04
8.07
30.73
10:42
19.79
51.67
37.50
24.46
3.20
21.01
12.41
3.86
17.07
31.24
16.23
52.57
38.10
21.74
InfiniGen
100.00
98.96
84.38
53.13
63.28
54.95
63.63
48.96
81.67
50,35
70.13
14.39
31.46
31.63
17.94
26.65
27.38
21.97
74.30
38.58
31.81
InfiniGen (V)
100,00
98.96
96.88
76.04
81.25
77.08
67.71
50,00
81,67
53.47
78.31
17.83
36.08
35.28
19.64
28.39
29.28
28.12
74.85
45.53
35.00
Quest
100.00
100.00
98,96
77.08
97.65
93.49
60.42
50.00
77.08
65.63
82.03
20.13
36.63
35.00
18.14
24.55
27.11
35.63
79.00
53.64
36.65
Quest (V)
100.00
100,00
98.96
85,42
97.92
95.49
70.83
46.88
78.75
65.63
83.99
17.26
39.51
36.78
18.71
26.41
29.49
79.50
60.05
38.17
SHADOWKV
100.00
100.00
97.92
98.96
96.88
95.83
72.92
52.08
81.67
72.57
86.88
17.17
39.73
38.29
21.08
31.77
31.62
35.87
80,00
63.93
39,94
GLMM-498-11
100.00
100.00
87.50
93.75
67.71
97.29
72.22
86.82
25.44
51.09
32.04
29.97
40.31
48.24
Loki
71.88
27.08
22.92
2,08
9.90
11.46
28. 13
27.08
31.04
54.17
28.57
5.82
30.60
22.73
9.20
30.09
30.35
22.70
98,92
40.77
32.35
Loki (V)
96.88
55.21
56.25
18.75
51.04
50.52
45
39.58
72.71
59.72
54.65
10.89
44.97
45.44
23.51
32.07
30.56
35.34
99.50
50.27
41.39
InfiniGen
100.00
93.75
82:29
0.00
79.43
60.16
57.29
53.13
92.71
57.29
67.60
23.67
46.31
55.69
33.91
27.49
25.44
33.48
91.83
36.96
41.64
InfiniGen (V)
100.00
96.88
87.50
7.29
95.31
75.26
56 25
54.17
95.63
60.76
72.91
25.63
48.44
57.23
36.94
29.77
26.67
36.64
93.58
46.69
44.62
Quest
100.00
95.83
90.62
54.17
76.30
51.21
52.08
95.88
64.58
77.86
23.81
44.53
56.41
35,49
23.54
21.73
37.39
87.00
80
41.52
Quest (V)
100.00
96.88
93.75
72.92
95.83
83.07
56. 25.25
53.13
96.88
65.97
8 81.47
26.00
46.32
57.54
36.42
24.58
24.52
37.71
93.50
46.52
43.68
SHADOW KV
100.00
100.00
95.83
83.33
98.70
87.76
69.79
55.21
97.50
68.06
85.62
26.50
51.31
59.09
38.87
32,92
28.54
38.70
96.50
58.55
47.89
1.mo-3.1-8B
100.00
100.00
98.96
91.67
98.96
95.31
82
47.92
68.96
71.18
85.53
31.56
55,10
57.65
29.46
35. 35.26
34.45
29.84
100.00
67.31
48.96
Loki
68.75
32.29
32.29
20.83
42.71
28.65
41.67
33.33
24.79
29.86
35.52
2.31
18.89
10.64
5.47
19.30
31.16
15.91
94.88
44.60
27.012
Loki (V)
95.83
36.46
57.29
62.50
77.86
70.83
69.79
39.58
35.21
37.50
58.29
3.93
38.59
22.85
12.96
27.43
32,22
26.43
98.25
56.11
35.42
InfiniGenn
100.00
77.08
78.13
13.54
58.07
47.40
63.63
41.67
60.83
50.35
59,27
27.23
52.72
53.89
26.81
27.72
29.61
24.42
98.93
56.89
44.25
InfiniGen (V)
100.00
88.54
87.50
26.04
79.43
77.08
72.92
43.75
57.08
55,21
68.76
29.73
53.47
55.11
28.72
28.55
31,42
26.76
99.17
46.18
Quest
100,00
98.96
97.92
34.38
93.49
88.54
70.83
44.79
65.63
68.40
76.29
29.70
49.04
53.96
27.18
27.16
30.43
29.85
98.50
57.35
44.80
Quest (V)
100.00
98.96
98.96
56.25
95.83
90.63
76.04
46.88
66.25
67.36
79.72
30.02
53.97
56.39
27.06
29.06
31.65
30.23
99.00
63.89
46.81
SHADOWKY
100.00
100.00
100.00
83,33
97.92
92.19
81.25
48.96
67.08
64.93
83.57
30.93
55,20
57.32
29.13
31.85
32.79
30.40
99.50
66.03
48.13
reduce the memory usage for the future generations We evaluate it and include the results in Appendix A.
which is 3.6x higher than A100 memory bandwidth. This result indicates that SILADOW KV V theoretically achieves a high equivalent bandwidth to accelerate attention computa- tion. System implementation is detailed in Appendix B.
4.2. Theoretical Equivalent Bandwidth
The benefit of SHADOWKV V in terms of increasing through- put can be analyzed through the concept of equivalent band- width. Consider each K or V vector as being M bytes in size, with a sequence length of S, a chunk size of C, selected chunk budget of K, O outliers, and hit rate a. During KV selection, SHAADWWV loads M x S/C bytes using the GPU memory bandwidth Boru. For value cache fetching, it loads M x K x C' bytes using the PCIe band- width Bpcie (Sheng et al., 2023). Since value movement and key cache reconstruction can be overlapped, we do not need to count key cache reconstruction here. Following this, SHADOWKV performs standard attention computa- tion for the top-k chunks and predefined outliers, requiring 2M x (K + O) x C' bytes. The equivalent bandwidth of SHADDWWV is defined as below and the GPU memory savings is detailed in Appendix A.2.
5. Empirical Evaluation
In this section, we showcase the effectiveness and efficiency of SHADOWKV. Specificalll..
In Section 5.1, we show that SHADOWKV can reduce the GPU memory footprint of the KV cache by over 6x without accuracy degradation on a wide range of models and evaluation benchmarks.
In Section 5.2, we demonstrate SHIADOWYV supports 6x larger batch sizes and increase the inference throughput by up to 3.04 x without compromising model quality.
In Section 5.3, we present ablation studies that validate the effectiveness of each component of SHADOWKV V in optimizing GPU memory usage and performance.
BS
5.1. Accuracy Evaluation
B= 2/C+ + +OC + (1 (1- KCBCC/ Bpcre
We demonstrate that SHADOWKY V can reduce GPU memory usage of the KV cache by 6x while maintaining accuracy on a range of tasks with a minimal sparse KV budget.
For example, assuming C-8, S=128K, K=256, O=48, B GB/s, and BGr==2 TB/s for A100, the equiva- lent bandwidth of SHADOWKV is calculated as 7.2 TB/s,
Setup. We choose four widely used long-context mod- els for our evaluation: Llama-3-8B- -B- (Gradient., 2024), GLM-4-9B-IXX (GLM et al., 2024), Llama-3.1-8B (Meta AI, 2024), and Yi-9B-200K (AI et al., 2024). We evalu-
'If v E Bddx is the right singular matrix calculated from the SVD of pre-RoPE context keys K E Rx new Rre- keys K' Xd can be stored K' and projected hack up with when needed.
6

---

SHADOWKV: KV Cache in Shadows for HighhTTroouhhut Long-Context LLM Inference
ate our approach on three challenging long-context bench- marks: RULER (Hsieh et al., 2024). LongBench (Bai et al., 2023), and Needle In A Haystack (NIAH) (Kamradt, 2023), covering QA, multi-hop. reasoning, summarization, code completion". We set the chunk size to 8, the rank to 160, and the number of outliers to 48 for SHADOWKV.
range of models can be found in Appendix B.3.
Integrate with Efficient Pre-filling Methods. We also combined SHADOWKV with a state-of-the-arr efficient pre- filling method MInference (Jiang et al., 2024). As shown in Table 2, following the setting of MInference, we tested it on RULER with contexts scaling from 8K to 256K. This demonstrates that our method is compatible with pre-filling acceleration techniques. For some certain context length settings, we even see slight performance improvement.
Baselines. We include three dynamic sparse attention methods as baselines: Quest (Tang et al., 2024b), Loki (Singhania et al., 2024), and InfiniGen (Lee et al., 2024b). For all methods, we retain exact pre-filling and perform dynamic sparse attention during decoding, where the com- putation cost is set to 1/16 of full attention for selecting sparse KV pairs. We include two variants for each baseline: one where all the KV cache is offloaded, and another where only the value cache offloaded. The former has similar latency to SHADOW V but smaller sparse budget since SHADOWKY KV only needs to fetch the value cache from the CPU. The latter aligns with the same sparse KV cache bud- get but significantly increases GPU memory usage. The latter one is marked as "(V)" in the table.
Table 2. Performance of different methods on RULER using MIn- ference (Jiang et al., 2024) in the pre-filling stage.
Methods
8K
16K
32K
64K
128K
256K|
Avg.
Llamu-3-8B-IM
89.92
88.02
82.81
78.45
78.12
74.57
81.98
SHADOWKV
90.47
88.12
83.28
77.71
78.32
74.31
82.04
Multi-tura Capability. To simulate multi-turn conversations, we chal- lenged SHADOWKV with
Accuracy on Multi-NIlH
1.0
0.8
As shown in Table 1, KV demon- strates excellent performance on 128K contexts. With a fixed sparse budget of 1.56%, other methods experience performance degradation. In contrast, SHADOWKV is more robust and even outperforms original full attention on cer- tain tasks, such variable tracking. For complex tasks like multi-document QA or multi-key needle retrieval, other methods suffer from significant performance degradation while SHADOWKV does not.
RULER.
Full Attandonn
Snapky
StreamingL ML
multi-turn needle re- trieval task (Multi-turn NIAH). We also test two eviction-based methods
ShadowkY (Curs)
0.2
0.0
N2
#3
84
45
HB
Conversation Tum
in Figure 7, including SnapKV V (Li et al., 2024) and StreamingLLMM (Xiao et al., 2023b). The performance of SnapKV drops significantly from the second round due to the required context information being different from the first round. Since SnapKV inevitably evicted tokens based on the first-turn conversation, it cannot successfully retrieve related information for future queries. In contrast, SHAD- ow can maintain accuracy in the multi-turn conversation setting.
Figure 7. Multi-turn NIAH.
LongBench. On LongBench, we evaluate our method with a range of realistic scenarios, including single-/multi- document QA, document summarization, code completion, information retrieval, etc. We only test on samples longer than 4K and set the sparse KV cache budget to 256 for this benchmark since it has shorter inputs compared to RULER. As shown in Table 1, SHADOW KV outperforms other meth- ods consistently and maintains the performance.
5.2. Efficiency Evaluation
To demonstrate the efficiency of SHADOWKV, we deploy it into real-world large batch serving scenarios. By measuring the throughput during decoding across different models on A A100, we show that SHADOWKV can support up to 6x larger batch sizes and boost throughput by up to 3.04x.
NIAH. On NIAH
Depiin Percent (%)
NIAH LI8ma-3-88-1M w/ ShadowKV
St # stize
dataset, as shown in Figure 6, SHAD-
OWKY shows the ability to process in- formation at differ- ent positions across various context win- dows, ranging from 16K to 1M tokens. More experiments
Context Length
Baselines. The baseline selects the largest batch size that can fit entirely on the GPU with full attention. We also in- clude results for the same batch size of SHADOWKV and the infinite batch size, assuming infinite GPU memory capabili- ties We set the sparse budget to 1.56% for SHIADOWYV
Figure 6. Needle In A Haystack.
*We include results for Yi-9B-200K and other models (c.g., Llama-3-70B-IMM in Appendix A. Needle In A Haystack is also tested Phi-3-Mini-123 -28K (Ahdin al., 2024) and Qwen2-7B- 128K (Yang et al., 2024a).
'For the equivalent SHADOWKV batch size, we evaluate a single Transformer block with FlashAttention and then project the
7

---

SHADOWKV: KV Cache in Shadows for High-Throughtt Long-Contex LLM Inference
Mult-koys NINH, Liama
Multi keys MAH GB
Limo
1.0
0.75
0.0
0.00
0.03
06
0.50
Full (1284)
(129K)
0.03
Full (120K)
0.50
(11(122K)
0.5
ShidoeeK
Quest
0.3
ShadowiV
Guest
BradowWV
Quest
D46
ShatooWVV
Quest
0.45
192
112 273
234 1:35 156 17 378 0.35 022
100
3.2 2.25 254 16 16-1113 12 0.26 30 020
100
5.12 75 2.34 1.46 1.59 117 0.78 2.33 0.20
130 52 223 2.34 196- 56 177 0.28 1039 5.00 Sparso Cache Budget
Sppase Budget (%)
Sparso KV Cacho Budget (76)
Sparso Cache Budgel (56
Varable Tracking Lama-3-99-111
Vartable Tracking
Frequent
Limma3-08--11
Frequent
GLM-499B-1M
0.05
0.0
0.04
0.752
0.02
0.20
0.725
8.00
0.702
0.70
0.3
Full (128K)
ShadowKY
Quest
0:1
0.22
0.80
ShadowiV
Full (129K)
0.902
0.525
0.572
Full (128K)
ShadowWV
Guest
0.62
0:04
ShadowKV
Full (128()))))
312 373 234 1:45 155
17 378 0.34 0:20
100
3:2 2:23 2:34 18 150.1 17 0.26 0.39 0:20
100 312 777 3:34 1.85 1.50 0.78 0.20
3:19 223 2:3 145 16 177 633 0.20 Sparse KV Cache Burget
Sporse Budget (%)
Sparse KV Cache Budget (N)
Sparce KV Cache Budget (%)
Figure& 8 Comparison results between different models on a range of long-cone congect tasks with full cache, our SHADOWKY V, and Quest. SHADOWKV consistently surpasses Quest under the same sparse budgets and achieves higher throughput.
Table 3. Generation throughput (tokens/s) on an A100. The gray text in brackets denotes batch size.
Table 4. Generation throughput (tokens/s) under varying batch
sizes and sequence lengths on Llama-3-8B-I1
Model
Contest
Hull Attn SHADOW KV
Gain
Pall Artn (Inf)
Comest
8.
12
16
32
48
3-S-----8-IM
60K
160.62 (8)
455.14 (48) x
168.72 (48) (48) 273.07 (Inf)
Full KV
(8 heads)
122K
239.51 (24)
2.97
83:05 (24) L34.30
24K
40.37 (2)
119.01 (12) 2,95 x
$2.00 (12)/ 67.15 (Inl)
60K
19.19
111.44
126.73
142.62
147 16062 OOM
UOM
DOM
OUM
GOM
122K
15.12
75.16
30.77
0OM
OOM OOM COM
0OM
DOM
00M
OOM
Llama-3.1 3,1-88
60K
160.93 (8)
472.77 (48)
x
168.22 (48) 2773077 (Inl)
241K
40.37
GOM
0OM
0OM
OUM
OOM 0OM
00M
(8 heads)
245.90 (24)
3.01
83.05 (24) 134.30 (Inl)
OUM
0OM
0OM
OUM
OUM
OUM
0OM
0OM
OOM
COM
SHADOW KV
GLM-4-98-IM
60K
241.05 (12)
615.89 (50) 2.56x
266.24
(50) /436.91 (lnf)
(4 KV heads)
122K
122.67 (6)
2933.40 (25)
x,39
158.83
(25) /214.87 (Inf)
60K
89.60
126.61
159.41
184.92
205.41
244.20
306.09
346.34
599.65
428.63
455.14
61.13 (3)
136.51 (12) 2.23
78.84 (12) 107.44 (Inf)
122K
94,25
115.01
143.77
166.72
196.73
OOM
264K
45.29
65.95
78.92
87.83
94.07
104.73 119.01
0OM
0OM
OOM
Y--9--20KK
60K
204.81 (10)
544.36 (42) 2.66
271.21 (42))334.00 (Inf)
485K
29.82
41.01
47.13
50.85
$3.46
OOM
OOM
00M
0OM
00M
OOM
(4 KV heads)
122K
101.44 (5)
260.00 (21) x,56x
133.53 (21) 179.06 (Iol)
244K
46.74 (2)
118.55 (10) x.54x
66.79 (10) 89.53 (Inf)
and remains funetional for extremely longer sequences. Ad- ditionally, we provide an efficiency comparison with Quest under 1M contexts in Appendix A.6, demonstrating that SHADOWWV significantly enhances throughput.
Results. As shown in Table 3, SHADOWKV demonstrates significant throughput improvements for various models on an A100, surpassing even those with infinite GPU memory. Notably, SHAADOWV supports batch sizes up to 6x larger and enhances throughput by up to 3.04x compared to full attention, even surpassing infinite batch size assuming infi- nite GPU memory. While the gains for GLM-4-9B- 1M and Yi-9B-200K are slightly lower, the improvements still reach up to 2.56x and 2.66x respectively, highlighting SHAD- OWKV's adaptability even with fewer KV heads. To further illustrate scalability, we provide detailed throughput mea- surements for Llama-3-8B-IM under varying batch sizes and context lengths in Table 4. As context length increases, full attention quickly runs out of memory, while SHAD- OwKV V continues scale up batch size 48 at 60K context
5.3. Ablation Results
In this section, we present extensive ablation studies of SHADOWKV, focusing on three key points: (1) sparse KV cache budget variations, (2) chunk size selections, (3) pre- RoPE key cache rank choices, (4) latency breakdown, and (5) accuracy contribution of outlier KV Cache. Additional ablations, including precision sensitivity analysis, are pro- vided in Appendix A.
Sparse KV Cache Budget. We examine SHADOWKV's performance across various tasks with different sparse bud- gets, as illustrated in Figure 8. SHADOWKV consistently surpasses Quest under the same sparse budgets and achieves higher throughput. On most tasks, it maintains accuracy
number to the entire model. For the infinite batch size, we lever- age A 100's theoretical memory bandwidth (2 TB/s) for attention computations.
8

---

SHADOWKV: KV KV Cache in Shadows for High- High-Throoughut Long-Context LLM Inference
1100
1.0
Batih Size Scaling Factor
Table 6. Latency breakdown (ms) of Transformer block of Llama-
0.9
Accuray RU RULERNIH
3-8B- IM during decoding.
-07
0.88
Agqurayy
0.8
RULERINIAHS
Context
G Soltmax
Max
Topk
Recompute (Overlapped)
Fetch
Attention
FFN
QKV
0.4
RULERINAHMK
-0.5
0.6
0.2
RULERIFWE
RULERINIAH-O
12x
48x61K
128K
0.56
0.65
0.07
0.07
0.07
0.16
0.14
1.25
1.19
1.36
1.84
1.75
0.21
0.19
0.23
0.33
0.25
0.05
0.05
0.05
0.0
RULERIQA
6x512K
0.71
0.07
0.17
1,51
1.69
0.18
0.23
0.05
0-
32
500
Table 7. Performance across different number of outlier chunks on
RULER (Hsieh et al.. 2024) evaluated at length of 128K.
Figure 9. Left: Impact of chunk size on batch size and accuracy.
Outliers
N1
N
MK1
N MMK
N MQQ
N MV
QA
QA 2
VI
FWE]
Avg.
Right: Accuracy trends across different ranks.
0000000
100.00
100.00
85.42
73.18
70.83
43.75
50.58
73.54
$7.20
74.05
00.000
100.00
98.96
94.79
K5.01
2:0.012
100:00
100.00
07:93
98.95
95.
95:57
70.83
$1.04
72.08
70.49
65.25
4 16) 100.00
100,00
97.92
98.96
95.83
95.57
71.88
51.04
74.38
71.18
R3.68
Table Latency breakdown (ms) of Transformer block of Llama-
$ S:
100.00
100.00
97.92
98.96
95. 55
95.05
72.92
$1.04
78.13
72.57
96.22
3-8B-1M MM during prefilling.
16 (0.18
32
100.00
100.00 100.00
100:00
97.92
97.97
98.96
98.96
96.09
06. 35
95:57
U5.31
72.92
72.92
51.34
53.08
81
90.42
71.53
72.23
96.42
96.71
43 10.29
100.00
100,00
96
83
95.83
72.92
52.08
71.57
96.83
Context
Attcation
FFN
SVD Reduce CosineSimilarity TopK
Gather|
Cost
Quest (Ref.) 100.00 100.00
95.96
77.08
97.65
93.49
60.42
50,00
77.08
66.63
62.00
64K
186.23
96.47
6.65%
Aail (8af.)
100.00 100:00
96.96
98.96
98.96
95.57
75.00
48.96
78.54
71,85
96,65
128K
721.13
193.32
26.62
0.20
2.77
0.14
0.02
3.25%
256K
2880:21
50.56
L.75%
512K
11720.30
789.23
108.38
0.84
12.19
0.15
0.06
0.97%
for Llama-3-8B-IM on the RULER benchmark with 128K context length. As presented in Table 7, our findings indi- cate that outliers play a crucial role. For instance, the first chunk, a significant outlier, has previously been shown to act as an attention sink (Xiao et al., 2023b), underscoring its importance in maintaining model accuracy. Remarkably, with just 8 outliers (0.049%), SHADOWKV outperforms the Quest baseline and nearly matches the accuracy achieved by full attention. However, when outliers are not adequately managed, the performance of the mean-based landmarks in SHADOWKY may fall below the min-max approach used by Quest, underscoring the importance of handling outliers properly.
with just a 1.56% sparse budget compared to full attention and even improves slightly on some tasks.
Chunk Size. As shown in Figure 9, increasing the chunk size allows for larger batch sizes. However, accuracy de- clines when the chunk size exceeds eight. Meanwhile, the chunk size choice has minimal impact on the chunk hit rate, which remains around 60%.
Rank of Pre-RoPE Keys. We assess SHADOWKV's per- formance across various tasks using different ranks for pre- RoPE keys. As illustrated in Figure 9, accuracy increases with the rank up approximately 160, after which it stabi- lizes near full-rank performance. Interestingly, the trends vary across tasks, and in some cases, low-rank approxima- tions achieve better performance.
6. Conclusion
We present SHADOWKV, a high-throughput inference sys- tem for long-context LLM inference. SHADOWKV opti- mizes GPU memory usage through the low-rank key cache and offloaded value cache, allowing for larger batch sizes. It reduces decoding overhead by accurate sparse attention, boosting throughput while maintaining accuracy. Our pirical experiments demonstrate SHIADOOY V can support up to 6x larger batch sizes and enhance throughput by up to 3.04 x on an A 100 across various long-context models, including Llama-3. 1-BB, Llama-3-8B-INe GLM-4-9B- IM, and Yi-9B-200K. SIIADOWK V holds great promise for im- proving cont-co LLM inference.
Scalability for Longer Sequences. As shown in Table 5, the overhead of SVD, reduce, cosine similarity, topK, and gather computing is very low and tends to decrease as the sequence scales, proving that SHADOWK K V's scalability to longer sequences.
Overlapping Operations for Latency Reduction. In Ta- ble 6, we demonstrate how overlapping the recomputation of the key cache with value cache fetching from the CPU significantly reduces decoding latency. This concurrent processing approach ensures that SHADOWKV minimizes overhead when handling long-context models.
Acknowledgment
We thank Zhuoming Chen, Xinyu Yang, and Yang Zhou for their helpful discussions and feedback on early drafts of the paper. This work was partially supported by Li-Auto, Amazon, Intel, Moffet AI, Cylab seed.
Accuracy Contribution of Outlier KV Cache. We con- duct experiments using different numbers of outlier chunks

---

SHADOWKV: KV Cache in Shadows for High- High-Thrroughut Long-Context LLM Inference
Impact Statement
Wang, P-S.,, Huang. N.-C., Ceze, L.,, and Wu, K.-C. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118 2024.
The deployment of long-context LLMs is often constrained by their significant memory and computational demands, limiting accessibility for real-world applications that require handling extended contexts. This work introduces SHAD- OWKV, high-throughput inference system that addresses these constraints by optimizing the KV cache. By leverag- ing low-rank key compression and efficient value offloading, SHADOWKV reduces GPU memory usage and inference latency while supporting larger batch sizes and longer se- quences. These innovations enable more efficient use of existing hardware, reducing the resource barriers associated with deploying LLMs.
Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.088 230.009677,, 2023.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process-
ing Systems, 35:16344-16359, 2022.
DeepSeck-AL Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
The societal implications of SI are multifaceted. By making it feasible to deploy LLMs on constrained hard- ware, this work democratizes access to advanced AI capabil- ities, empowering smaller organizations and researchers in resource-limited settings. Furthermore, its ability to main- tain accuracy while reducing computational overhead con- tributes to the broader sustainability goals of AI by lowering energy consumption in large-scale deployments.
Dong, H., Chen, B., and Chi, Y. Prompt-prompeee adaptive structured pruning for efficient Ilm generation. In First Conference on Language Modeling, 2024a.
Dong, H., Yang. X., Zhang, Z., Wang. Z., Chi, Y., and Chen, B. Get more with less: Synthesizing recurrence with kv cache compression for efficient Ilm inference. arXiv preprint arXiv:2402.0920
2024b.
No specific ethical concerns or societal risks are associated with the proposed method. Instead, it provides path to- ward equitable and responsible AI deployment, ensuring that advancements in LLM capabilities accessible and sustainable across diverse sectors.
Fu, J., Jiang, Y., Chen, J., Fan, J., Geng, X., and Yang, X. Speculative ensemble: Fast large language model ensem- ble via speculation. arXiv preprint arXiv:2502.01662, 2025.
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:23 10.01801, 2023.
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
GLM, T., Zeng, A., Xu, B., Wang. B., Zhang, C., Yin, D., Rojas, D.,, Feng, G., Zhao, H., Lai, H., Yu, H., Wang, H., Sun, J., Zhang, J., Cheng, J., Gui, J., Tang, J., Zhang, J., Li, Zhao, L., Wu, L.,, Zhong, L., Liu, M., Huang, M., Zhang, P., Zheng, Q., Lu, R., Duan, S., Zhang, S., Cao. S., Yang, S., Tam, W. L., Zhao, W., Liu, X., Xia, X., Zhang, X., Gu, X., Lv, X., Liu, X., Liu, X., Yang, X., Song, X., Zhang, X., An, Y., Xu, Y., Niu, Y., Yang, Y., Li, Y., Bai, Y., Dong, Y., Qi, Z., Wang, Z., Yang. Z., Du, Z., Hou, Z., and Wang, Z. Chatglm: A family of large language models from glm- glm-130b to glm-4 all tools, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 0rXiv:2303.087// 2023.
:, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang. S., Yu, T., Xie, W., Huang, W. Hu, X., Ren, X., NIu, X., Nie, P. Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by 01. ai, 2024.
Gradient. Llama- --w-n--------sssss 8b-instruct gradient 4194k (v0.1), 2024. URL ht Li Lilma-33 3- 8B Inst:rutt-rrrdieen------------ee
He, J. and Zhai, J. Fastdecode: High-throughput gpu- efficient Ilm serving using heterogeneous pipelines. arXiv preprint arXiv: 11411, 2024.
Bai, Y., Lv, X.. Zhang, J., Lyu, H., Tang, J.. Huang, Z., Du, Z.,, Liu, X., Zeng, A., Hou, L.,, Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprini 2308v:2308. 14508, 2023.
Hong, K., Dai, G., Xu, J., Mao, Q. Li, X., Liu, J., Chen, K., Dong, H., and Wang, Y. Flashdecoding++: Faster large language model inference on gpus. arXiv preprint W:Xiv:231..0/255 2023.
Chang, C.-C., Lin, W.C.,, Lin, C.Y.,, Chen, Y., Hu, Y.-F,
10

---

SHADOWKV: KV Cache in Shadows for HlghhTThouuhhpu Long-Context LLM Inference
Hooper, C., Kim, S., Mohammazzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length Ilm in- ference with kv cache quantization. arXiv preprint arXiv:2401. 18079, 2024.
Lin, B., Zeng, Z., Xiao, Z., Kou, S., Hou, T., Gao, X., Zhang, H., and Deng. Z. Matryoshkakv: Adaptive kv compression via trainable orthogonal projection. arXiv preprint arXiv:2410. 14731, 2024.
Liu, H., Yan, W., Zaharia, M., and Abbeel, World model on million-lengee video and language with ringattention. arXiv preprint arXiv:2402.0822 2024a.
Hsich, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprint orXiv:2404.0666 2024.
Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyril- lidis, A., and Shrivastava, A. Seissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024b.
Jiang, H., Li, Y., Zhang, C., Wu, Q. Luo, X., Ahn, S., Han, Z., Abdi, H., Li, D.. Lin, C.-Y., et al. Minference 1.0: Accelerating pre-filling for long-context Ilms via dynamic sparse attention. arXiv preprint arXiv:2407.02499 2024.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuming-fre free asym- metric 2hit quantization for kv cache. arXiv preprint arXhv:2402.02750 2024c.
Juravsky, J., Brown, B., Ehrlich, R., Fu, D. Y., Ré, C., and Mirhoseini, A. Hydragen: high-throughhut Ilm inference with shared prefixes. arXiv preprint arXiv:2402.05099 2024.
Meta AL. Introducing Llama 3.1, 2024. URL https: / //ai metaa com/ woomblogg/ee cessed: 2024-08-21.
1lama-l 1-mm-3--222 3-1// Ac-
Kamradt, G. Needle in a haystack pressure testing llms. 2023.
Microsoft. Microsoft bingchat, 2024. URL https:// www. bi com/ chat.
Kwon, W. Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Lee, W., Lee, J., Je,, and Sim, J. InfiniGen: Efficient gen- erative inference of large language models with dynamic kv cache management. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), 2024a.
Peng, Y., Zhang, G., Zhang, M., You, Z., Liu, J., Zhu, Q.. Yang, K., Xu, X., Geng, X., and Yang. X. Lmm-rl: Em- powering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025.
Lee, w. Lee, J., Seo, J., and Sim, J. [InfiniGen]: Effi- cient generative inference of large language models with dynamic (KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implemen- fation (OSDI 24), pp. 155-172, 2024b.
QwenTeam. Qwen2.5: A party of foundation models, September 2024. URL https: //wen/mm github. io/blog/qwen2. 5/.
Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Inter- national Conference on Machine Learning, pp. 19274- 19286. PMLR, 2023.
2019.
Ribar, L., Chelombiev, I., Mudlass-Galley, L., Blake, C., Luschi, C., and Orr, D. Sparq attention: Bandwidth- efficient Ilm inference. arXiv preprint arXiv:2312.0449 049955,, 2023.
Li, Y., Dong, B., Lin, C., and Guerin, F. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201, 2023.
Saxena, U., Saha, G., Choudhary, S., and Roy, K. Eigen attention: Attention in low-rank space for kv cache com- pression. arXiv preprint ar Xiv: 406.066466 2024.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint 2404. 14469, 2024.
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I,, and Zhang, C. Flexgen:

---

SHADOWKV: KV Cache in Shadows for High- Throohhhruuhhut Long-Context LLM Inference
High-throughput generative inference of large language models with single gpu. In International Conference on Machine Learning, pp. 31094-31116, PMLR, 2023.
Wolf, T. Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint arXiv: 1rXiv:9900003771, 2019.
Singhania, P., Singh, S., He, S., Feizi, S., and Bhatele, A. Loki: Low-rank keys for efficient sparse attention. arXiv preprint ar (tt.:2406.02442. 2024.
Xiao, G., Lin, J., Seznee, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023a.
Song, Z., Yuan, J., and Yang, H. Fmint: Bridging human designed and data pretrained models for differential equa- tion foundation model. arXiv preprint 24XY:2400414688. 2024.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. In The Twelfth International Conference Learning Representations, 2023b.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. 2024.
Xu, Y., Jie, Z., Dong, H., Wang, L.,, Lu, X., Zhou, A., Saha, A., Xiong, C., and Sahoo, D. Think: Thinner key cache by query- pruning. arXiv preprint 20711:2407.21018, 2024.
Sun, H., Chen, Z.,, Yang. X., Tian, Y., and Chen, B. Tri- force: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404. 2024a.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint ar arXiv:2407.10671 2024a.
Sun, H., Haider, M., Zhang, R., Yang. H., Qiu, J., Yin, M.. Wang. M., Bartlett, P., and Zanette, A. Fast best- of-n decoding via speculative rejection. arXiv preprint arXiv:2410.20290. 2024b.
Yang. J. Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S. J., and Lee, D. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint 2rXvv:2402.180099 2024b.
Tang, H., Lin, Y., Lin, J., Han, Q. Hong, S., Yao, Y., and Wang, G. Razorattention: Efficient kv cache compression through retrieval heads. arXiv preprint arXiv:2407.1589. 2024a.
Ye, Z., Lai, R., Lu, B.-R., Chien- Yu, L.,, Zheng, S., Chen, L., Chen, T., and Ceze, L. Cascade inference: Mem- ory bandwidth efficient shared prefix batch decoding, 2024. URL
a.t..ii/2220 02/ 02/ het/caccaae-infeeeeeee hhml. Accessed: 2024- 09-25.
Tang, J., Zhao, Y., Thu, K., Xiao, G., Kasikei, B., and Han, S. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint arXiv:2406.1077 2024b.
Yu, H., Yang. Z., S., Li, Y., and Wu, Effectively com- press kv heads for 11m. arXiv preprint arXiv:2406.07056 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 231:2231..1805 2023.
Yuan, Y., Huang, Y., Ma. Y. Li, X., Li, Z., Shi, Y., and Zhou, H. Rhyme-aware chinese lyric generator based on gpt. arXiv preprint arXiv:2408. 10130, 2024.
Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E.,, Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M.. Merrill, D., Blasig, D., Qiao, F,, Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, January 2023. URL https: /ttp//ggtttt/t///// com Com/NV:10 /t/.//VIIIAA/AA cass.
Yue, Y., Yuan, Z., Duanmu, H., Zhou, S., Wu, J., and Nie, L. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402. 12.40..22066 12065, 2024.
Wang, H., Yang, X., Chang, J., Jin, D., Sun, J., Zhang, S., Luo, X., and Tian, Q. Parameter-efficient tuning of large- scale multimodal foundation model. Advances in Neural Information Processing Systems, 36, 2024a.
Zhang, H., Ji, X., Chen, Y., Fu, F., Miao, X., Nie, X., Chen, W., and Cui, B. Pqcache: Product quantization based kveache for long context Ilm inference. arXiv preprint arXiv:2407.12220 2024a.
Wang, M., Chen, L., Fu, C., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L., Luo, R., et al. Leave no document behind: Benchmarking long-context 1mms with extended multi-doc qa. arXiv preprint arXiv:2406. /7419, 2024b.
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L., Wang, S., Liu, Z., and Sun, M. ocbench: Extending long context evaluation beyond 100k tokens, 2024b.
12

---

SHADOWKV: KV Cache in Shadows for HlgghTTrooughpt Long-Context LLM Inference
Zhang, Y., Gao, B., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., Xiao, W., et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406. 02069, 2024c.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L.,, Cai, R., Song, 7.,, Tian, Y., Ré, C.,, Barrett, C., al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024d.
13

---

SHADOWKV: KV Cache in Shadows for High-Throughtn Long-Context LLM Inference
A. Additional Experiment Results
In this section, we present additional experiments experiments not covered in the main text, including the handling of newly generated tokens (as discussed in Section 4.1), scalability analysis for larger models and longer sequences (mentioned in Section 5.1), additional ablation studies (referenced in Section 5.3), and etc.
A.1. Handling of Newly Generated Tokens
To address the handling of newly generated tokens, we project these tokens' key cache into a low-rank space using the same projections applied during the prefilling phase. This approach preserves the benefits of reduced GPU memory usage, particularly for long output sequences.
As shown in Table 8 and Table 9, we refer to this extension as SHADOWKV Our evaluation across various models demonstrates that SHADOWKV effectively maintains accuracy while optimizing memory usage.
Table Performance of SHADOWKV and SHADOWKY K V across different models on RULER (Hsieh et al., 2024) evaluated at length of 128K.
Methods
N-S1
N-S2
N-MKI
N-MK2
N-MQ
N-MV
QA-1
QA-2
VT
FWE
Avg.
Llama-3-8B 3-M
100.00
100.00
98.96
98.96
98.96
95.57
75.00
48.96
78.54
71.85
86.68
SHADOWKY
100.00
100.00
97.92
98.96
96.88
95.83
72.92
52.08
81.67
72.57
86.88
SHADOWKV V+
100.00
100.00
98.96
100.00
95.83
93.49
71.88
50.00
80.21
71.88
86.23
GLM-4-9B- IM
100.00
100.00
94.79
87.50
99.74
93.75
67.71
55.21
97.29
72.22
86.82
SHADOWKY
100.00
100.00
95.83
83.33
98.70
87.76
69.79
55.21
97.50
68.06
85.62
SHADOW +
100.00
100.00
95.83
85.42
98.17
85.16
69.79
56.25
97.92
67.71
85.63
Llama-3.1 -.1-BB
100.00
100.00
98.96
91.67
98.96
95.31
82.29
47.92
68.96
71.18
85.53
SHADOWKV
100.00
100,00
100.00
83.33
97.92
92.19
81.25
48.96
67.08
64.93
83.57
SHADOWK V
100.00
100.00
100.00
84.38
96.88
91.67
81.25
52.08
65.63
62.85
83.47
Yi-9B-200K
100.00
100.00
86.46
62.50
64.58
32.55
44.79
39.58
36.87
89.93
65.73
SHADOWKY
100.00
100.00
82.29
67.71
63.28
31.51
43.75
38.54
56.04
72.22
65.53
SHADOWKV+
100.00
100.00
81.25
67.71
61.72
31.51
46.88
38.54
53.54
72.92
65.41
Table Performance of SHADOWKV and SHADOWKV V++ different models with LongBench (Bai al., 2023) samples exceeding 4K
tokens.
Methods
NarratQA MultiFQA HotpotQA MuSiQue DuRead GovRep SAMSum PassRetr I.CC| Avg.
Alama-3-88-11 IR-IM
18.98
41.84
36.79
21.47
31.93
34.18
35.96
81.50 56.07 39.86
SHADOWKY KV
17.17
39.73
38.29
21.08
31.77
31.62
35.87
80.00 63.93 39.94
SHADOWKV+
20.42
41.16
37.22
21.03
31.77
31.98
35.80
80.00 63.89 40.36
GLM-4-9B-1T 1M
25.44
51.09
58.67
39.61
32.04
29.97
40.31
99.00 58 02 48.24
SHADOWKV
26.50
51.31
59.09
38.87
32.92
28.54
38.70
96.50
58.55 47.89
SHADOWK V+
27.59
51.31
59.17
38.34
33.55
31.25
39.46
96.50 55 86
48.11
Llama-3. 3.10-...--8 18B
31,56
55.10
57.65
29.46
35.26
34.45
29.84
100.00 67.3 1 48.96
SHADOWKV
30.93
55.20
57.32
29.13
31.85
32.79
30.40
99.50
66.01
48.13
SHADOWKV+
32.25
54.29
57.75
28.37
31.07
32.89
28.73
98.75 67.59
47.97
Yi-9B-200K
13.88
30.02
52.46
28.20
22.29
30.25
19.08
67.00
73.50
37.41
SHADOWKS V
12.44
30.82
52.43
27.73
20,79
29.83
20.73
64.00
72.89
36.85
SHADOW V+
14.08
30.94
51.16
27.00
19.50
29.34
21.16
66.00 73.47
36.96
14

---

SHADOWKV: KV Cache in Shadows for Hligh-Throughput Long-Context LLM Inference
A.2. Quantitative Analysis of GPU Memory Savings
The GPU memory savings provided by SHADOWWV can be quantitatively analyzed follows. Let each K or V vector have size of M bytes, with sequence length S, chunk size C, selected chunk budget K, O outliers, and pre-RoPE key cache rank The GPU memory savings of SHADOWKY KV can then be expressed as:
25M
Memory Savings
SM/C- 2(K + + rM
For example, assuming M 1024, -,5 128K, calculated 7.08x. This result demonstrates that SHADOWKV V can theoretically reduce the KV cache memory footprint on the GPU by 7.08x for longer sequences and larger batch sizes.
256, O 48, = 160, the memory savings of SHADOWKV
A.3. Accuracy Results for Yi-9B-200K
We present accuracy results for Yi-9B-200K (AI et al., 2024) on RULER (Hsich al., 2024) and LongBench (Bai et al., 2023), highlighting SHADOWKV's superior performance across diverse tasks compared to other methods.
Table 10. Performance of Vi-9B-200k with different methods on RULER (Isieh et al., 2024) evaluated at length of 128K. SHADOWKY outperforms other methods with a 1.56% sparse budget.
Methods
N-S1
N-S2
N-MKI
N-MK2
N-MQ
N-MV
QA-1
QA-2
VT
FWE
Avg.
YI-9B-200K
100.00
100.00
86.46
62.50
64.58
32.55
44.79
39.58
36.87
89.93
65.73
Loki
34.38
2.08
2.08
0.00
0.00
0.52
22.92
21.88
0.00
25.00
10.89
Loki (V only)
59.38
11.46
18.75
5.21
4.43
2.08
22.92
31.25
0.00
35.07
19.06
InfiniGen
100.00
94.79
77.08
1.04
40.10
20.57
37.50
34.38
41.46
46.18
49.31
InfiniGen (V only)
100.00
98.96
78.13
2.08
58.33
24.48
40.63
35.42
52.92
55.90
54.69
Quest
100.00
98.96
79.17
26.04
56.51
31.77
32.29
31.25
51.04
71.88
57.89
Quest (V only)
100.00
100.00
80,21
45.83
59.37
31.90
36.45
34.37
53.54
71.88
61.36
SHADOWKV
100.00
100.00
82.29
67.71
63.28
31.51
43.75
38.54
56.04
72.22
65.53
Table 11. Performance of Yi-9B-200K with LongBench (Bai al., 2023) samples exceeding 4K tokens. SHADOWKV outperforms other methods and maintains the accuracy.
Methods
NanQA MultiFQA HotpotQA MuSiQue DuRead GovRep SAMSum PassRetr I.CC) Avg.
Yi-9B-200x
13.88
30.02
52.46
28.20
22.29
30.25
19.08
67.00
37.41
Loki
1.63
2.73
16.21
4.87
4.75
2.13
4.95
0.00 38.72 8.44
Loki (V only)
1.96
10.39
21.31
7.36
6.78
9.15
10.02
4.00 58.75 14.41
InfiniGen
10.01
23.61
50.47
25.91
15.11
27.96
18.97
30.00 56.46 28.72
InfiniGen (V only)
11.31
26.46
51.13
26.77
16.09
28.67
19.33
34.00 62.07 30.65
Quest
10.57
25.83
46.06
23.04
17.09
17.11
20.59
50.50 67.70 30,94
Quest (V only)
14.56
25.73
48.73
24.73
18.44
20.83
20.08
57.50 71.13 33.53
KV
12.44
30.82
52.43
27.73
20.79
29.83
20.73
64.00 72.89 36.85
A.4. Precision Sensitivity
In the main experiments, we used BF16 for both model weights and KV cache. To further investigate the impact of precision on SHADOW KV's performance, we conducted additional experiments using FP8 precision (torch. float 8 __e 5m2). These tests aim to determine whether SHADOWKY V can retain its accuracy at this lower precision, addressing concerns about precision sensitivity, particularly SVD computations.
As detailed in Table 12 and Table 13, SHADOWKV and baseline methods were evaluated using FP8. Results show that SHADOW KV maintains accuracy and achieves consistently high performance even with FP8 precision. This robustness, despite FP8's reduced numerical range, confirms that SIIADOWKV can continue to deliver efficiency gains without compromising accuracy.
15

---

SHADOWKV: KV Cache in Shadows for High-Throughhe Long-Context LLM Inference
Table 12. Performance comparison of SHADOWKV and baseline methods the RULER (Hsich al., 2024) using FP8 precision,
evaluated at sequence length of 128K.
Methods
N-S1
N-S2
N-MK1
N-MK2
N-MQ
N-MV
QA-1
QA-2
FWE
Avg.
Llama-3-8B-IM
100.00
100.00
98.96
95.83
97.40
95.57
63.54
48,96
75.83
73.26
84.94
Loki
5.21
1.04
0.00
0.00
0.78
0.26
5.21
13.54
28.33
28.82
8.32
Loki (V only)
36.46
9.38
31.25
0.00
6.25
21.09
11.46
15.63
57.08
35.76
22.44
Quest
100.00
98.96
98.96
71.88
96.61
93.49
63.54
45.83
78.13
67.01
81.44
Quest (V only)
100.00
100.00
98.96
85.42
97.40
93.49
70.83
48.96
78.13
65.63
83.88
SHADOWKV
100.00
100.00
97.92
94.79
95.31
93.49
75.00
48.96
80.42
73.61
85.95
Table Evaluation of SHADOWKV and baseline methods on LongBench (Bai et al., 2023) with sequence lengths exceeding 4K tokens, using FP8 precision.
Methods
NarratQA MultiFQA HotpotQA MuSiQue DuRead GovRep SAMSum PassRetr LCC|
Avg.
Llama-3-8B-IM.
18.69
41.21
35.76
21.59
31.81
33.77
35.29
80.50 56.77 39.49
Loki
2.21
11.12
5.70
1.84
15.42
28.59
11.41
41.91 33.99
16.91
Loki (V only)
2.68
22.33
12.69
3.35
21.43
30.57
16.32
47.68 36.64
21.52
Quest
19.41
38.92
34.02
19.64
23.13
26.40
28.04
78.50 49.81
35.32
Quest (V only)
16.19
36.73
36.64
19.59
25.57
29.46
27.14
79.50
60.05
36.76
SHADOWKV
18.29
39.39
36.06
21.04
30.47
31.87
35.56
78.50 62.11 39.25
A.5. Scalability Analysis for Larger Models and Longer Sequences
Neadle Haystack Liama-3-708 Instruct- Shadow
To demonstrate the scalability of SHADOWKV, we present experi- ments with Llama-3-8B- IM on IM contexts and Llamm-3-70B-I1
10.
20-
30-
on 512K contexts, using the RULER benchmark (Hsich et al., 2024). Additionally, we evaluate Llama-3-70B-11 1M on the Needle In Haystack dataset, testing context lengths ranging from 16K 1M tokens.
Degth Percent
0.6
50.
0.4
80
As shown in Figure 10 and Table 14, STIADOWKY maintains robust performance across increasing context lengths and model sizes, demonstrating its scalability in handling large-scale inputs. This scalability allows SHIADOWKV to process extensive contexts with high accuracy, making it valuable solution for real-world applications requiring extensive sequences.
187
2078
720M
Book
Context Length
Figure 10. Needle In A Haystack.
Table 14, Performance of different methods on RULER (Hsich al., 2024) evaluated length of MM. The Llama-3-BB-IM is evaluated on 1M contexts while the Llama-3770BIIM is evaluated on 512K contexts.
Methods
N-SI
N-S2
N-MKI
N-MK2
N-MQ
N-MV
QA-1
QA-2
VT
FWE
Avg.
Llama-3-70B-11
100.00
82,29
90.63
54.17
85.16
96.61
69.79
35.42
68.75
69.44
75.23
Loki
100.00
1.04
0.00
0.00
0.00
0.00
13.54
11.46
34.30
22.92
18.33
Loki (V only)
100.00
15.63
26.04
0.00
0.00
0.00
25.00
19.79
40.00
31.94
25.84
Quest
100.00
76.04
78.13
35.42
85.47
92.19
53.21
34.38
38.33
58.33
65.15
Quest (V only)
100.00
77.08
79.17
36.49
86.19
95.31
54.17
36.58
47.70
58.68
67.14
SHADOWK KV
100.00
82.29
88.54
53.04
88.02
94.79
67.71
37.50
68.54
68.25
74.87
Llama-3-88 (aama----BBJJ1
3-8B-IM
96.88
100.00
96.88
69.79
91.15
85.68
64.58
42.71
25,00
56.25
72.89
Loki
9.38
1.04
10.42
0.00
2.60
4.43
38.54
11.46
1.67
0.69
8.02
Loki (V only)
68.75
29.17
60.42
1.04
26.56
43.23
59.38
15.63
6.46
0.69
31.13
Quest
94.79
92.71
80.21
4.17
76.30
69.27
57.29
28.13
25.67
30.56
55.91
Quest (V only)
94.79
93.75
81.25
4.17
79.69
69.27
62.50
31.25
26.00
32.99
57.57
SHADOW V
96.88
100,00
96.88
65.63
89.38
83.16
69.79
42.71
26.04
59,38
72.98
16

---

SHADOWKV: KV Cache in Shadows for High-Thrruughut Long-Context LLM Inference
A.6. Efficiency Comparison with Quest
We present efficiency comparison with Quest, particularly under long contexts or high batch sizes where the GPU memory alone cannot accommodate the KV cache. In such cases, both Full Attention and Quest must offload the KV cache the CPU. As shown in Table 15, SHADOWKV significantly outperforms both Full Attention and Quest under the same sparse budget.
The efficiency advantage of SHADOWK V over Quest is due to two key factors: (1) SHADOWKV V only fetches the value cache from the CPU, rather than the entire KV pair, minimizing data transfer and reducing latency, and (2) SHADOW integrates cache mechanism that leverages the temporal locality of the KV cache.
Table 15. Efficiency comparsion with Quest.
Context
Full Attention
Full Attention (CPU)
Quest
Quest (CPU)
SHADOWKV
3x
OOM
0.21 tokens/s
OOM
9.34 tokens/s
45.32 tokens/s
A.7. Detailed Comparison with InfiniGen
We provide further clarification on the key distinctions and conduct additional experiments between SHADOWKV and InfiniGen. These experiments show that SHADOWKV significantly outperforms InfiniGen across a wide range of downstream tasks.
Differences in SVD Usage. Infinigen performs an offline SVD to get a projection matrix, which is applied to post-RoPE key and query states for KV selection, while SHADOWKV applies an online, prompt-dependent SVD directly to the pre-RoPE key cache for compression, not for KV selection.
Methodological Differences. While InfiniGen uses SVD for KV selection, it requires fetching selected, exact KV pairs from the CPU. In contrast, SHADOWKV only fetches the value cache from the CPU, reconstructing the key cache from its low-rank storage the GPU. By overlapping these processes, SHADOWKV reduces data-fetch overhead and achieves improved efficiency in KV cache management.
Accuracy Comparison. To empirically validate SHADOW V advantages, we conducted accuracy evaluations. Results confirm SHADOWK V's effectiveness maintaining accuracy while optimizing memory usage. Although InfiniGen performs well on simpler tasks like RULER-N-SI, shows significant accuracy drops on more complex tasks, such as GULER-N-MKS RULER-FWE. LongBench--CCe LCC, and others, where SHADOWK KV maintains consistently high accuracy.
17

---

SHADOWKV: KV Cache in Shadows for High- Thghuhhhoughhut Long-Context LLM Inference
B. Experiment Details
In this section, our goal is to provide the details of the system implementation (mentioned Section 4.2), experiment settings, and additional experiments on InfiniteBench (Zhang et al., 2024b) and Needle A Haystack (Kamradt, 2023).
B.1. System Implementation.
We implement the framework based on PyTorch (Paszke et al., 2019; Wolf, 2019) and dedicated kernels (Thakkar et al., 2023). FlashAttention (Dao et al., 2022; Dao, 2023; Hong al., 2023) used for attention computation and some efficient fused kernels in Flashinfer (Ye al., 2024) and vLLM (Kwon et al., 2023) are used, including layer norm. To reduce memory movement and kernel launch overhead, key cache low-rank reconstruction, value cache fetching, cache mechanism, etc. We leverage multi-streams to overlap the reconstruction of key cache and value cache fetching. We set the rank of pre-RoPE key cache to 160, chunk size to 8, and sparse KV cache budget to 1.56% for most cases.
fuse some operations into CUDA kernels, including attention approximation,
B.2. Dataset Details
LLMs are widely used in various fields (Sun et al., 2024b; Peng et al., 2025; Fu et al., 2025; Li et al., 2023; Yuan et al., 2024; QwenTeam, 2024; Wang et al., 2024a; Song et al., 2024), and we select three long-context benchmarks, detailed below.
RULER (Hsich et al., 2024) consists of 13 complex tasks and supports adjustable context lengths, including retrieval, multi-hop tracking, aggregation, and QA tasks. For the with MInference (Jiang et al., 2024), we set up test sets scaling from 8K 256K for evaluation.
LongBench (Bai et al., 2023) is challenging long-context benchmark that assesses the performance of LLMs in extended contexts. Featuring Chinese and English languages, LongBench encompasses main categories and 21 diverse tasks, evaluating LLM capabilities across crucial long-text applications like hingle/multi-docunn QA, summarization, code completion, etc.
Needle In A Haystack (Kamradt, 2023) long-context retrieval benchmark testing LLM's performance with context window scales up to IM tokens where information placed at various positions. We tested the retrieval capabilities of six long-context LLMs based on their context length.
B.3. Needle In A Haystack
In addition to the Needle In A Haystack results for (-lama3--B--il IM shown in Figure 6, also present results for GLM- 4-9B- 1M, Llama-3.1 3.1-8BB Yi-9B-2000 200K, 1in-3-Mini-128K, and Qwen2-7B. Dwen2-7B-128K, shown in Figure 11. Compared to full attention, using SHADOWKV V has minimal impact on the ability to understand semantic information across different context windows and needle depths. There is even a slight performance improvement for Yi-9B-200K.
B.4. InfiniteRench
InfiniteBench (Zhang et al., 2024b) is challenging long-context benchmark that consists of 10 tasks, including QA, coding, dialogue, summarization, and retrieval, with an average length of 214K.
Table Accuracy different methods on InfiniteBench (Zhang al., 2024b).
Methods
SumSum En.QA
En.MC En. Zh.QA Code. Math.Find Retr. PassKyy Retr.Num
Jaama-3-8B-IM
23.05
18.14
65.06
10.50
12.47
24.36
37.14
100.00
100.00
SHADOWKV
21.50
17.73
64.63
10.50
12.45
23.86
37.43
100.00
100.00
GLM-4-9B-IM
28.61
9.25
68.12
39.50
11.77
30.20
40.00
100.00
100.00
SHADOWKV
23.22
8.48
68.56
32.50
11.27
30.46
40.00
100.00
100.00
Llama-3.1 1-8-
26.42
14.48
66.38
16.00
12.92
21.07
34.00
100.00
99.66
SHADOWKV
24.23
13.83
66.38
16.50
12.76
21.07
34.00
100.00
94.41
Yi-B--000
8.88
10.61
61.57
5.50
13.88
21.57
23.71
100.00
99.66
SHADOWK K.
8.92
10.06
59.39
6.00
13.89
20.56
24.29
100.00
99.83
18

---

SHADOWKV: KV Cache in Shadows for High- High-Throoughut Long-Context LLM Inference
Needle in Haystack GLM-4-98-1M
Needle in A Haystack GLM-4-98-1M w/ ShadowKV
1.0
1.0
11
Depth Percent (96)
22
33
Depth Percent
33
0.4
0.4
78-
89
100
15
227x
297k
305x
7kie
B30c
16c
87x
157x
227h
300k
5000
8500
230K
Context Length
Context Length
Needle Haystack Llama 3.1--...-8 Instruct
Needle in Haystack Liama-32 3.1-86 1.88 Instruct Insstuut
ShadowKV
0.8
0.8
Degtn Percent
Deptr Percent
33
0.6
56
0.2
022
100
100
16k
1.200
1208
100
240
1200
Context Length
Context Length
Needle Haystack YI-98-20OK
Needle A Haystack YI-98-200K w/ ShadowKV
1.0
1.0
11.
11-
Deoth Percent (%)
0.6
0.6
56
0.4
67
78
90x
387x
200K
0.0
360r
187x
200x
0.0
Context Length
Context Length
Needle in Haystack
Needle in A Haystack Phi-3-Mini-128kk 12ntttittkkkttttttttcccc w/ ShadowitV
1.0
1.0
0.8
0.8
22
Depth Percent
33
Depth Percent
33
0.4
0.4
78-
89
89
100
100
16c
11ar
120
120k
16k
SPC
Sroue
220e
Context Length
Context Length
Needle Haystack -n-struutt
Needle in Haystack Qwen2-78 ---itruttntt ShadowkV
0.8
Death Percent
33
0.6
0.6
44
96
56
0.2
0.2
100
100
10k
24h
Stcc
10MK
1.2%
1208
1286
406
2236
1200
2386
Length
Length
Figure 11. Needle In A Haystack (Kamradt, 2023) results using GLM-4-9B-I IM (GLM et al., 2024), Llama-3. 1-nn--sttBB-In (Meta Al,
2024), Yi-9B-200K (Al al., 2024), Phi-3-Mini- 128K (Abdin et al., 2024), and Qwen2-7B- ---1--7--18K (Yang al., 2024a).
19


Reviews:
[{"review_id": "v5mitq8OCf", "reviewer": "Reviewer_Pdg2", "summary": "Large language models (LLMs) excel at handling extended contexts but face challenges with key-value (KV) cache scaling, increasing memory usage and reducing inference throughput. Existing methods like KV eviction and sparse attention either degrade accuracy or inadequately optimize memory use and latency. This paper introduces SHADOWKV, leveraging two key insights: (1) pre-RoPE keys are inherently low-rank and highly compressible, and (2) value caches, being less compressible, can be efficiently offloaded to CPU storage. SHADOWKV integrates low-rank key compression, value cache offloading, and precise sparse KV selection to significantly reduce memory usage while preserving high accuracy and low latency. Comprehensive experiments demonstrate SHADOWKV achieves 6× larger batch sizes and throughput gains up to 3× compared to baseline methods across various LLMs and tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and your positive assessment of our work. We appreciate your recognition of the novelty of ShadowKV, particularly its system-level insights and the strength of accuracy evaluations. We address your concerns with additional clarifications and experiments as detailed below.\n\n---\n\n### **Q1: Throughput evaluation is somewhat limited. Memory savings do not always translate directly into performance gains.**\n\nWe appreciate the suggestion. To further support our claim, we include a table below, showing throughput (tokens/s) under varying batch sizes and contexts for both Full KV and ShadowKV with Llama3-8B-1M. \n\n|(seq,method)|bsz=1|bsz=2|bsz=3|bsz=4|bsz=5|bsz=6|bsz=8|bsz=12|bsz=16|bsz=24|bsz=32|bsz=48|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|60K,Full|58.62|89.19|111.44|126.73|142.62|147.40|**160.62**|OOM|OOM|OOM|OOM|OOM|\n|122K,Full|45.59|65.12|75.16|**80.77**|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|\n|244K,Full|32.22|**40.37**|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|\n|488K,Full|**20.15**|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|OOM|\n|60K,ShadowKV|47.06|89.69|126.61|159.41|184.92|205.41|244.20|306.09|346.34|399.65|428.63|**455.14**|\n|122K,ShadowKV|36.57|65.61|94.28|115.01|132.23|143.77|166.72|196.73|217.24|**239.51**|OOM|OOM|\n|244K,ShadowKV|27.63|48.39|65.95|78.92|87.83|94.07|104.73|**119.01**|OOM|OOM|OOM|OOM|\n|488K,ShadowKV|17.27|29.82|41.01|47.13|50.85|**53.46**|OOM|OOM|OOM|OOM|OOM|OOM|\n\n- ShadowKV enables significantly larger batch sizes at long context lengths, where Full KV quickly runs out of memory.\n- At small batch sizes (e.g., bsz = 1), ShadowKV may be slightly slower due to reconstruction and fetching overhead. However, as batch size increases, the benefit of memory savings and increased parallelism translate directly into throughput gains.\n\nThese additional results help clarify that ShadowKV not only reduces memory usage but also yields consistent throughput improvements. We will include it in the final version and appreciate the reviewer's suggestion, which helped us strengthen this aspect of the paper.\n\n---\n\n### **Q2: What is the time breakdown for low-rank key cache reconstruction in a typical scenario?**\n\nAs detailed in Appendix A.6, we provide a full component-wise decoding latency breakdown (in milliseconds) across various batch sizes and context lengths on an A100 GPU, which includes the time cost of both key reconstruction (Recompute K) and value cache fetching (Fetch V). Our system is designed to leverage CUDA multi-streams to overlap these two operations.\n\n|Context|GEMM+Softmax|Max|TopK|Recompute K (Overlapped)|Fetch V|Attention|FFN|QKV|\n|-|-|-|-|-|-|-|-|-|\n|48$\\times$64K|0.56|0.07|0.14|1.25|1.84|0.23|0.33|0.05|\n|24$\\times$128K|0.58|0.07|0.15|1.36|1.66|0.21|0.29|0.05|\n|12$\\times$256K|0.65|0.07|0.16|1.49|1.75|0.19|0.25|0.05|\n|6$\\times$512K|0.71|0.07|0.17|1.51|1.69|0.18|0.23|0.05|\n\n---\n\n### **Q3: How effectively can value cache fetching overlap with computation, and under what conditions?**\n\nAs discussed in Appendix A.6, we leverage CUDA multi-streams to overlap CPU-GPU data transfers (Fetch V) with GPU-side compute (Recompute K), ensuring high GPU utilization during decoding. This overlap is particularly effective in long-context settings where attention computation dominates latency.\n\nAs shown in the table, Fetch V and Recompute K exhibit comparable latency. Since they are executed on separate CUDA streams, their durations can be effectively overlapped—meaning only Fetch V contributes to the critical path. \n\nFurthermore, our measured end-to-end throughput gains validate that these design choices translate into practical system-level improvements.\n\n---\n\n### **Q4: Experimental settings for Table 3—are they based on standard vLLM?**\n\nOur baseline implementation leverages open-source efficient CUDA kernels, including those from both vLLM and FlashInfer [1]. Our inference engine is optimized specifically for large-batch long-context decoding.\n\nThe table below presents a decoding latency (tokens/s) benchmark, demonstrating that our baseline achieves comparable or slightly better efficiency than vLLM in full-attention mode. This shows that our baseline is a valid and fair reference point.\n\n|Context (Full Attention)|vLLM |Our Baseline|\n|-|-|-|\n|1$\\times$30K|64.19|69.38|\n|1$\\times$60K|55.30|58.62|\n|1$\\times$120K|43.41|46.60|\n|1$\\times$240K|30.51|32.78|\n|1$\\times$480K|19.04|20.74|\n\n---\n\n### **Q5: Limitations for integrating SHADOWKV into vLLM and SGLang?**\n\nShadowKV employs custom CUDA kernels for low-rank reconstruction, RoPE fusion, and efficient CPU data prefetching with overlap , built on top of CUTLASS [2]. Integrating ShadowKV into vLLM and SGLang would require backend modifications to support these kernels and its memory pool management strategy. While this integration is non-trivial, it is technically feasible, and we are actively working toward supporting popular serving engines in the future.\n\n[1] https://github.com/flashinfer-ai/flashinfer\n\n[2] https://github.com/NVIDIA/cutlass", "claims_and_evidence": "SHADOWKV claims significant improvements in accuracy and system performance through its combination of low-rank compression and optimized memory management. The evidence supporting accuracy improvements is robust, detailed, and quantitatively convincing. The paper also provides adequate quantitative analysis regarding the potential benefits to system performance, though further detailed experiments on throughput could strengthen these claims.", "methods_and_evaluation_criteria": "The paper evaluates two primary metrics: accuracy and throughput. Accuracy evaluation is extensive, clearly demonstrating the effectiveness of the proposed compression techniques. Throughput evaluations, while less thorough, sufficiently illustrate efficiency improvements. Expanding throughput analysis, particularly across more diverse settings, would further strengthen the evaluation.", "theoretical_claims": "The theoretical foundation for low-rank key compression is plausible and well-articulated, though I am less confident in fully assessing its theoretical rigor. Nevertheless, quantitative system performance analyses and experimental results effectively support these theoretical claims.", "experimental_designs_or_analyses": "The experimental design, primarily focused on accuracy and system performance metrics, is solid. Accuracy evaluations are especially thorough and convincing. However, throughput experiments could benefit from more comprehensive scenarios or additional benchmarks to better understand the practical limits and generalizability of the proposed approach.", "relation_to_broader_scientific_literature": "The paper situates itself within existing literature on KV cache management strategies, such as compression and offloading techniques, highlighting its contribution with the insight on compressing RoPE keys and reconstruct values.", "essential_references_not_discussed": "None explicitly identified.", "other_strengths_and_weaknesses": "A notable strength of the paper is its insightful design, particularly in efficiently compressing pre-RoPE key caches and significantly reducing CPU-GPU communication overhead, demonstrating substantial potential for long-context scenarios.\n\nHowever, the evaluation of throughput is somewhat limited. Memory savings do not always translate directly into performance gains; thus, expanding this evaluation with additional context or detailed performance metrics could enhance the credibility of system-related claims.", "other_comments_or_suggestions": "Overall, this is a well-written and carefully prepared paper.", "questions_for_authors": "1. In a typical scenario, what is the time breakdown for low-rank key cache reconstruction?\n\n2. How effectively can value cache fetching overlap with computation, and under what conditions?\n\n3. Could you clarify the experimental settings used for generating Table 3? Specifically, is this based on standard vLLM?\n\n4. What are the primary limitations that might prevent integrating SHADOWKV into modern LLM serving engines like vLLM and SGLang?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MUcnG1XZsO", "reviewer": "Reviewer_vsSH", "summary": "This paper finds a novel method of KV cache management that takes advantage of partial offloading to CPU and matrix decomposition to obtain impressive KV reduction without affecting accuracy and reducing latency significantly. The paper finds two important properties of LLMs that it takes advantage of: (i) the Key cache in KV caches, before applying positional embeddings (this rotates each token/vector in K by an amount determined by its position so that the model can learn positions of tokens), has very low rank and thus high compressibility. (ii) A majority of K tokens after positional embeddings have been applied, show high cosine similarity to adjacent tokens, with few outliers. \nWhat this paper does it takes advantage of property (i) by offloading the V (Value) cache to CPU memory and compress K (key) using Singular Value Decompsition (SVD) cache in GPU memory, thus during inference when KV values are needed, while V is being fetched from CPU simultaneously K is being uncompressed and has positional embeddings applied to it. Furthermore, it takes advantage of (ii) to perform a ‘chunk level approximation’, where a small number of K tokens (1.56%) are retained that can be used to approximate most of the other tokens while the outliers are retained in GPU memory. This leads to further compression. The results show 3x improvement in throughput while supporting 6x larger inputs or batches, this performs better with regards to throughput than a theoretical infinite KV cache memory on GPU, due to minimizing total data movement.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for detailed review and valuable feedback. We appreciate the reviewer's recognition of the novelty and effectiveness of our method. Below, we address the raised concerns below and will incorporate clarifications into the revised version. We hope the reviewer can consider raising your score in light of our response. \n\n---\n\n### **Q1: Clarification on Figure 8 and the meaning of \"Sparse KV Cache Budget\".**\n\nThe x-axis in Figure 8 refers to the percentage of KV pairs selected at each decoding step for attention computation—i.e., the active sparse KV budget per step, rather than the overall GPU memory usage for KV cache storage. In contrast, the 6$\\times$ memory compression refers to the total reduction in GPU memory footprint for the KV cache, achieved through a combination of low-rank key compression and value offloading. The extremely small sparse budget (e.g., 1.5%) is made possible by our accurate TopK selection mechanism. We will clarify this distinction explicitly in the figure caption in the future version.\n\n---\n\n### **Q2: Undefined terminology such as KV cache hit rate and $D(H_1, H_2)$.**\n\nThank you for the valuable suggestion. We will add clarifications below in the future version.\n\n- The KV cache hit rate refers to the proportion of selected KV chunks at decoding step $t$ that are also selected at step $t+1$, due to temporal locality in generation. This metric reflects the effectiveness of caching selected sparse KV pairs across decoding steps to reduce redundant key cache reconstruction and value cache fetching.\n- As for $D(H_1, H_2)$, it is defined in Section 3.1 (Line 204) as $D(H_1, H_2) = \\langle H_1, H_2 \\rangle / r$, where $H_1$ and $H_2$ are rank-$r$ projection matrices and $\\langle \\cdot, \\cdot \\rangle$ denotes the Frobenius inner product. We agree the definition could be made more prominent, and we will revise the presentation to improve clarity and visibility.\n\n---\n\n### **Q3: GPU memory usage during inference — will it spike due to K reconstruction and V fetching?**\n\nIn ShadowKV, GPU memory usage remains bounded and does not spike during inference. This is due to two key factors:\n- At each decoding step, only a sparse subset of KV pairs (e.g., 1.5%) is selected for reconstruction and fetching, not the full sequence.\n- These operations are performed layer-by-layer, and the buffers are reused across layers, so peak memory does not accumulate across the entire model.\n\nFor example, in Table 3 (e.g., 122K context, batch size of 24), we reconstruct 1.56% of the full KV cache during each decoding step, resulting in <200MB peak additional memory usage—negligible compared to the full KV size. We will add a memory profiling figure to better illustrate this in the future version.\n\n---\n\n### **Q4: The paper currently does not mention limitations.**\n\nThank you for this suggestion. We plan to add a limitations paragraph, covering the following points:\n\n1. **Dependency on RoPE positional encoding.** Our low-rank key compression relies on the pre-RoPE key representation being compressible. While RoPE is used in most state-of-the-art LLMs, the method may require adaptation for models using absolute or learned positional embeddings. We are exploring this generalization as future work.\n\n2. **Linear-time sparse KV selection.** Our current KV selection requires an $O(N)$ complexity. We view the design of $O(\\log N)$-time sparse search methods as a promising direction.\n\n3. **SVD-based compression on low-precision hardware.** ShadowKV relies on SVD for key cache compression, which is highly efficient on modern GPUs. However, on edge devices with only low-precision compute units (e.g., INT8-only cores), efficient SVD implementation may be challenging. Exploring lightweight alternatives to SVD could improve portability to constrained hardware.\n\n---\n\n### **Q5: Generalization to non-RoPE models.**\n\nWe focus on RoPE-based models because RoPE is the dominant positional encoding scheme in state-of-the-art LLMs (e.g., LLaMA, Qwen, Yi, GLM, etc.). We acknowledge that generalization to other positional embeddings is an interesting direction, and we are currently exploring variants such as ALiBi in ongoing work.\n\n---\n\n### **Q6: What if CPU-to-GPU bandwidth is low, or the CPU is slow?**\n\nShadowKV performs well under typical bandwidth conditions and is designed to keep transfer sizes small and overlappable. We clarify that:\n\n- Value cache fetching is PCIe bandwidth-bound. The CPU merely serves as a memory pool for the value cache and is not involved in computation. Hence, CPU speed has negligible impact on performance.\n- In our implementation, we use PCIe Gen4 with 32 GB/s bandwidth, which is standard in modern server-grade inference deployments. In practice, the PCIe bandwidth is typically well-matched to the GPU performance. For instance, H200 GPUs are commonly paired with PCIe Gen5 (128 GB/s). Therefore, it is rare to find a high-performance GPU configured with a low-performance PCIe.", "claims_and_evidence": "The paper makes modest claims and back them up through empirical evidence.", "methods_and_evaluation_criteria": "The evaluation criteria are explained reasonably well.", "theoretical_claims": "The paper does not have a theoretical claim.", "experimental_designs_or_analyses": "The experiments are designed and executed well and the results are analyzed in sufficient depth", "relation_to_broader_scientific_literature": "The paper does a good job in comparing the proposed idea against existing works.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths: \n•\tPaper discovers two separate important properties of LLMs and exploits them each, to obtain very impressive compression (6x reduction in GPU memory) and throughput increases (3x higher).\n•\tBenchmarks are comprehensive, with 3 different benchmarks used, along with 3 different LLM models. Behaviors of different parameters are also shown in the paper. Comparisons to several other state of the art KV methods are also done.\n\nWeaknesses:\n•\tIn Figure 8, x axis needs clarification. The paper reports 6x larger batch sizes, yet the figures here show x axis to have ‘sparse budgets’ down to 0.2% while maintaining performance, which us much more than a 6x compression. It looks like sparse budgets here is referring to something other than KV cache GPU memory usage, this should be clarified in the caption.\n•\tSome undefined terminology: KV cache hit rate is not defined in text or in the figures; the ‘D’ in D(H1, H2) in observation section is not defined, along with a few other mathematical notations in that section.\n•\tAuthors do not mention how GPU memory usage increases during inference, as the K cache is reconstructed and V cache is prefetched. It seems like the memory usage should spike up during inference in this case.\n•\tIt would be good if the paper briefly addresses the limitations of this approach. Currently it does not.", "other_comments_or_suggestions": "This paper proposes a method of KV cache compression and latency improvements obtaining very impressive results with two complementary approaches. It introduces a method of obtaining a very high compression ratio on K caches, and offloading V to CPU. During inference fetching V from CPU memory and K reconstruction are overlapped.\nSome things can be addressed to clarify some parts. In figure 8 it looks like the x axis shows KV compression as it is labelled as ‘Sparse KV Cache Budget’ and shows up to 0.20% KV budget while retaining accuracy. This implies up to 500x compression, however the introduction states there is a 6x compression. It seems that ‘Sparse KV Cache Budget’ is referring to something other than total KV cache memory usage and should be clarified. \nThe authors test the method on 3 models, but this raises the question whether this method can generalize to other models and positional embeddings other than RoPE. This should be discussed in the paper.\nAnother possible limitation that should be addressed is how the CPU side affects the performance. The authors implicitly assume that fetching V from CPU memory and reconstructing K in GPU memory have around the same time, but what is the CPUs are slow, or CPU to GPU bandwidth is low. This should also be addressed.\nA final possible limitation of this paper is how GPU memory usage changes during inference. Outside of inference the authors report 6x compression of GPU memory, but if the KV cache is reconstructed during inference, then it seems that the GPU memory will spike back up to roughly baseline KV cache size. This needs to be addressed in the paper.", "questions_for_authors": "Can you please address the concerns in the weaknesses part above?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5cMoPdQGHy", "reviewer": "Reviewer_QmhP", "summary": "This paper presents several interesting findings, including the observation that pre-ROPE keys exhibit low-rank properties, and that post-ROPE keys show high similarity with neighboring tokens. \nBased on these insights, the authors propose two main techniques: Low-Rank Keys and Offloaded Values for Storage, and Accurate KV Selection for Fast Decoding. \nThe designs are well-reasoned and supported by comprehensive experiments. \nThe proposed methods demonstrate strong compression performance, achieving impressive results across multiple models and long-text benchmarks.\n\n## update after rebuttal\nThe rebuttal partially addresses my concerns, so I will maintain my rating of 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and thorough review. We truly appreciate your recognition of our findings and experiments. We have thoroughly addressed each of your questions and hope our responses will lead you to consider raising your score.\n\n---\n\n### **Q1: Concern about unaccounted latency from key reconstruction and value fetching in the Theoretical Equivalent Bandwidth analysis.**\n\nAs detailed in Appendix A.6, we provide a full component-wise decoding latency breakdown (in milliseconds) across various batch sizes and context lengths on an A100 GPU, which includes the time cost of both key reconstruction (Recompute K) and value cache fetching (Fetch V). Our system is designed to leverage CUDA multi-streams to overlap these two operations.\n\n|Context|GEMM+Softmax|Max|TopK|Recompute K (Overlapped)|Fetch V|Attention|FFN|QKV|\n|-|-|-|-|-|-|-|-|-|\n|48$\\times$64K|0.56|0.07|0.14|1.25|1.84|0.23|0.33|0.05|\n|24$\\times$128K|0.58|0.07|0.15|1.36|1.66|0.21|0.29|0.05|\n|12$\\times$256K|0.65|0.07|0.16|1.49|1.75|0.19|0.25|0.05|\n|6$\\times$512K|0.71|0.07|0.17|1.51|1.69|0.18|0.23|0.05|\n\nAs shown in the table, Fetch V and Recompute K exhibit comparable latency. Since they are executed on separate CUDA streams, their durations can be effectively overlapped—meaning only Fetch V contributes to the critical path.\n\nFurthermore, our measured end-to-end throughput gains validate that these design choices translate into practical system-level improvements.\n\n---\n\n### **Q2: Some references not discussed.**\n\nAs stated in Section 3.1 (Line 188), prior low-rank approaches primarily focus on **data-independent, offline weight decomposition**, i.e., performing low-rank factorization on the **model weight matrices** using calibration data or during training. These methods either **require training or achieve limited compression**. The cited works (EigenAttention, HeadsKV, and MatryoshkaKV) belong to this category. Specifically:\n\n- EigenAttention generates lower-rank weight matrices, yielding at most 40% KV cache reduction.\n- HeadsKV converts MHA into GQA via low-rank decomposition of weights, which requires training and is only applicable to MHA-based models.\n- MatryoshkaKV trains orthogonal projection matrices, achieving around 60% average KV cache compression.\n\nIn contrast, ShadowKV **identifies the dynamic, sequence-dependent low-rank structure in pre-RoPE keys**, and performs **online SVD directly on the KV cache** during inference. This approach is training-free, adaptive to each sequence, and achieves significantly higher compression (over 6$\\times$, 15.6% of the original memory).\n\nThank you for giving us the chance to elaborate on this point. We will add these mentioned references to the future version.\n\n---\n\n### **Q3: Core innovation of the work.**\n\nAs discussed in Q2, unlike existing low-rank methods that compress weights offline, ShadowKV performs online activation-level SVD on the KV cache in a sequence-adaptive manner, and achieves substantially better compression while preserving accuracy. Prior methods rely on static weight decomposition and often require fine-tuning or model modification.\n\nMoreover, our approach is tightly integrated with a system-level design. We carefully co-design algorithm and system by enabling low-rank key cache reconstruction and CPU-to-GPU value fetching to overlap through CUDA multi-streams. Additionally, we utilize an accurate sparse KV selection mechanism that reduces attention computation and memory movement. This holistic integration ensures end-to-end throughput gains.\n\n---\n\n### **Q4: Outlier tokens — are there interpretable patterns? And does outlier count correlate with layers/heads?**\n\nThank you for the insightful question. We clarify that we selected a subset of layers rather than showing all layers in Figure 5c for visualization purposes to avoid clutter. We performed additional analysis to better understand the nature of outlier tokens and their distribution.\n\nWe found that certain tokens, such as attention sinks [1], are consistently identified as outliers, with their similarity scores even being negative, highlighting their distinctiveness in the distribution. Across heads, the sets of outliers tend to differ. We did not observe a clear trend or correlation between outlier frequency and specific layers or head indices.\n\nWe absolutely agree that this is a rich direction for future exploration, and we plan to conduct deeper investigations into the interpretability and structure of outliers in future work.\n\n[1] Efficient Streaming Language Models with Attention Sinks", "claims_and_evidence": "The paper mentions that the reconstruction of the low-rank keys and the movement of values from the CPU to the GPU are synchronized. \nAs a result, the authors omit this part of the process in their calculation of the Theoretical Equivalent Bandwidth. \nHowever, it would be important to experimentally consider the time overhead of this step. \nSpecifically, the reconstruction involves an additional matrix multiplication, followed by ROPE which has an O(L) complexity, this might have a significant impact on performance that should be accounted for.", "methods_and_evaluation_criteria": "The authors propose two main techniques: Low-Rank Keys and Offloaded Values for Storage, which effectively reduce the KV cache footprint for long-text sequences, and Accurate KV Selection for Fast Decoding, which improves accuracy. \nThe proposed methods are well-designed and supported by comprehensive experiments. \nThe authors use relevant benchmarks, such as RULER and Longbench, which are highly suitable for the problem at hand, effectively demonstrating the superiority of their approach across multiple models and long-text datasets.", "theoretical_claims": "The theoretical proofs provided in Section 4.2 are generally sound. \nHowever, I have some concerns about one of the underlying assumptions: that the time required for low-rank key states reconstruction can be disregarded. \nAs I mentioned in the \"Claims and Evidence\" section, I find this assumption somewhat questionable. \nGiven that the reconstruction involves an additional matrix multiplication, and ROPE has an O(L) complexity, the potential time cost should be considered more thoroughly.\n\nThe provided theoretical proofs in Appendix A.2 were reviewed and appear correct and carefully derived, and no immediate issues were identified.", "experimental_designs_or_analyses": "I reviewed the experimental evidence supporting the two key findings, and I find them reasonable. \nThe selection of RULER and Longbench, which are both long-text datasets, is appropriate for the experimental requirements. However, I believe that the theoretical claims discussed earlier would benefit from some experimental validation to strengthen the argument.", "relation_to_broader_scientific_literature": "This paper’s findings, such as pre-Rope keys having the lowest rank and post-Rope keys showing high similarity between adjacent tokens, offer valuable insights for KV cache compression. These results extend prior work on transformer optimizations by highlighting key structural patterns, which can lead to more efficient compression techniques and improved memory and computational performance in large models.", "essential_references_not_discussed": "Previous works like Eigen-attention[1], HeadsKV[2], MatyroshkaKV[3] has also explored the low-rank properties of the KV cache and utilized corresponding methods, such as SVD/PCA, to reduce the KV cache footprint.\n\n[1] Eigen Attention: Attention in Low-Rank Space for KV Cache Compression\n[2] Effectively Compress KV Heads for LLM \n[3] MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection", "other_strengths_and_weaknesses": "**Strengths**\nThe paper presents several interesting findings, such as the pre-ROPE key states have a low-rank property compared to other parameters, and that most of the key states post-ROPE exhibit high similarity with neighboring tokens. \nThese contributions offer valuable perspectives in the field. \nAdditionally, the proposed methodology is well thought out and highly reasonable. \nThe detailed results demonstrate strong performance, confirming the effectiveness of the approach.\n\n\n**Weaknesses**\n- The amount of work done in this paper is impressive, particularly the discovery mentioned above. However, the core innovation of the work is not entirely clear. To my knowledge, both the use of SVD decomposition on the KV cache to store low-rank matrices and the application of landmarks for estimating attention scores for selection have been explored in prior work.  What novel adaptations or insights does this paper offer beyond simply combining these existing techniques?", "other_comments_or_suggestions": "The overall writing of the paper is well-structured and adheres to proper conventions, with no immediate issues identified.", "questions_for_authors": "I find your findings very intriguing! \nHowever, I have one curiosity regarding the outlier tokens. \nDo these outlier tokens correspond to any specific patterns in the original text? \nIs there a way to identify these tokens as outliers based on the original text itself? \nAdditionally, I noticed that you only presented results from a subset of the layers. \nHave you analyzed whether the number of outlier tokens correlates with the number of layers and heads in the transformer model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qbG0TdBk0L", "reviewer": "Reviewer_1RzE", "summary": "This paper presents ShadowKV, a system for long-context LLM inference that optimizes memory usage and throughput with negligible impact on output quality.\n\nShadowKV consists of two key techniques:\n1. In GPU memory, it only stores the SVD decomposition of pre-ROPE key caches to reduce memory usage per request. The value cache will be stored in CPU memory and be loaded back to GPU during the inference.\n2. During the inference, it selects a subset of the tokens' KV cache to run the attention, which further increases the batch size and the inference throughput.\n\nThe evaluation results show that Shadow KV achieves similar quality to the SOTA KV cache eviction baselines and can improve the inference throughput compared to full KV cache computation.\n\n## update after rebuttal\n\nThanks to the authors for answering my questions. The rebuttal has addressed my concerns and I will keep my score of \"accept\".", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive comments and for recognizing the novelty of our method and the thorough evaluations. We hope our detailed clarifications below address the remaining concerns.\n\n---\n\n### **Q1: Missing clarification on which dataset was used for observations**\n\nWe appreciate the reviewer's careful reading. The observations regarding the low-rank structure of pre-RoPE keys, the high similarity of post-RoPE keys, and the KV cache hit rate plots (Figures 2a, 2b, 5b, and 5c) are based on sequences drawn from PG-19 [1], a real-world long-context dataset. We chose PG-19 to ensure that our structural analyses reflect natural language characteristics rather than synthetic artifacts.\n\nFor Figure 5a, the results are based on sequences sampled from RULER-NIAH, consistent with our downstream evaluation tasks.\n\n---\n\n### **Q2: How does the token selection interact with state-of-the-art attention frameworks like FlashAttention? Does it require kernel modifications?**\n\nThank you for the insightful question. In our implementation, token selection is applied prior to the attention computation. At each decoding step, we identify the most relevant KV tokens based on chunk-level attention scores. For efficiency, the chunk-level attention score computation (GEMM + Softmax) is fused into a single kernel. The selected KV entries are then gathered into a compact buffer, which is passed directly to the FlashAttention interface.\n\nSince we only modify the input tensors to attention and not the attention logic itself, no modifications to the FlashAttention kernel are required. This makes ShadowKV easy to integrate into modern inference stacks that already support FlashAttention.\n\n[1] Compressive Transformers for Long-Range Sequence Modelling", "claims_and_evidence": "The first key claim in this paper is that the Pre-ROPE key tensor is much sparser than other intermediate catchable results during LLM inference. The author provides clear and easy-to-understand evidence in the form of figures to support this claim.\n\nThe second key claim in the paper is that the attention scores of the adjacent tokens are usually very similar (with a small amount of outliers). The authors also used real experimental results to support the claim.\n\nHowever, the authors did not clarify which dataset they used to produce the experimental result, whether the dataset is real or synthetic, and whether it matches the target use case of this paper. This limits the credibility of the claims.", "methods_and_evaluation_criteria": "The evaluation section consists of 3 main parts:\n\n1. evaluating the impact on the LLM generation quality.\n2. evaluating the improvement of the runtime efficiency.\n3. evaluating the impact of different configurations in the system\n\nIn all three parts, the baselines and the setup are clear and easy to understand. The selected datasets also fit the target use case of this paper. \n\nOverall, the evaluation of the paper is solid.", "theoretical_claims": "The theoretical analysis and algorithm design (Section 3.1 and Section 4) make sense.", "experimental_designs_or_analyses": "The experimental design is clear and comprehensive.", "relation_to_broader_scientific_literature": "This paper provides a new way of compressing and storing the KV cache. The SVD technique will not only be useful to save the memory usage of the KV cache but also useful in reducing the cost for the KV cache storage use cases.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- Well-designed system with clear technical details, covering most aspects during the inference.\n- Solid experimental results with SOTA baselines and a wide range of datasets.\n- Good baseline selection in performance evaluation (used FlashAttention)", "other_comments_or_suggestions": "I really enjoyed reading this paper and liked the idea of SVD decomposition of pre-ROPE key cache.", "questions_for_authors": "It would be great if you could give some details about how the token selection works with the state-of-the-art attention frameworks like FlashAttention. Does it require modifying the kernel?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to reduce the memory overhead of long-context LLM inference. It discovers a low-rank property of key vectors before the positional encodings are applied, which leads to a novel offloading algorithm. Evaluations across various models and tasks are comprehensive and well-executed.

All reviewers agree that the paper novel, well-motivated, and well-exacted. Most concerns have been addressed.
Decision: Accept (spotlight poster)