ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

## Abstract

Abstract With the widespread deployment of long-context large language models (LLMs), there has been a growing demand for efficient support of high-throughput inference. However, as the key-value (KV) cache expands with the sequence length, the increasing memory footprint and the need to access it for each token generation both result in low throughput when serving long-context LLMs. While various dynamic sparse attention methods have been proposed to speed up inference while maintaining generation quality, they either fail to sufficiently reduce GPU memory consumption or introduce significant decoding latency by offloading the KV cache to the CPU. We present ShadowKV , a high-throughput long-context LLM inference system that stores the low-rank key cache and offloads the value cache to reduce the memory footprint for larger batch sizes and longer sequences. To minimize decoding latency, ShadowKV employs an accurate KV selection strategy that reconstructs minimal sparse KV pairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks, including RULER, LongBench, and Needle In A Haystack, and models like Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, and Qwen2-7B-128K, we demonstrate that it can support up to 6 × \times × larger batch sizes and boost throughput by up to 3.04 × \times × on an A100 GPU without sacrificing accuracy, even surpassing the performance achievable with infinite batch size under the assumption of infinite GPU memory. The code is available at https://github.com/bytedance/ShadowKV .

## 1 Introduction

Large language models (LLMs) have increasingly demonstrated their ability to scale and handle long contexts [, , , ], enabling them to tackle complex tasks like multi-document question answering and information retrieval from extensive contexts of up to 1M tokens [, ]. However, efficiently serving these long-context LLMs presents challenges related to the key-value (KV) cache [, ], which stores previous key-value activations to avoid re-computation. As the KV cache scales with sequence length, its growing memory footprint and the need to access it for each token generation lead to low throughput during long-context LLM inference. To address these issues, KV cache eviction or sparse attention methods have been widely explored.

However, existing methods face three primary limitations: accuracy degradation, inadequate memory reduction, and significant decoding latency overhead. KV cache eviction strategies [, ] aim to reduce the memory footprint by discarding KV pairs based on specific policies, but they often result in information loss and accuracy degradation in tasks such as multi-turn conversations [, ]. Dynamic sparse attention methods [] preserve all KV pairs on the GPU and accelerate inference by computing attention with selected KV pairs. However, this line of work does not mitigate the memory footprint, thereby limiting the batch size and preventing accommodation of extremely long contexts (e.g., 1M tokens). A naive solution based on sparse attention involves offloading the KV cache to the CPU to reduce memory usage [, ]. Nonetheless, this approach incurs significant overhead due to the latency of fetching the selected sparse KV pairs from the CPU during decoding.

Consequently, an ideal effective system for long-context LLM inference with sparse attention should: (i) reduce GPU memory usage, (ii) minimize inference latency, and (iii) maintain accuracy within limited sparse KV cache budgets. Fortunately, we can potentially overcome these challenges by leveraging our discovery that pre-Rotary Position Embedding [] (RoPE) keys are exceptionally low-rank compared to the layer inputs, post-RoPE keys, values, key weight matrix, and value weight matrix, as indicated in Figure 1(a). Furthermore, our analysis in Figure 1(b) reveals that pre-RoPE keys lack significant similarities in low-rank subspaces across different sequences, while a sequence and its continuation tend to strongly share low-rank subspaces, enabling high compression rates within each sequence. Motivated by these findings, we have developed two key insights that pave the way for the design of an applicable system, detailed in Section 3.

Figure: (a)
Refer to caption: x2.png

Low-rank Keys and Offloaded Values for Storage: In long-context LLM inference, the quadratic scaling of attention computation with sequence length makes the linear cost of low-rank decomposition during pre-filling negligible, as illustrated in Figure 1(c)(^1^11In practical scenarios, the key cache can be offloaded to the CPU to perform SVD asynchronously or precomputed and stored as part of the prefix cache [].). To reduce memory footprint, we retain the low-rank pre-RoPE key cache on the GPU and offload the value cache to the CPU since the value cache does not exhibit low-rank properties, minimizing memory footprint without sacrificing accuracy. During decoding with sparse attention, we employ CUDA multi-streams to overlap the recovery of the selected key cache with the fetching of the corresponding value cache. This approach conceals key cache reconstruction and reduces data fetching overhead by $2\times$ compared to the naive offloading strategy, thereby decreasing the latency of sparse attention during decoding.

Accurate KV Selection for Fast Decoding: To further reduce decoding latency in sparse attention, we propose an accurate KV selection method that maintains accuracy with minimal sparse budgets (1.56%). Our analysis reveals that most post-RoPE keys exhibit high cosine similarity with adjacent tokens, enabling chunk-level approximations for selecting important tokens. A minimal number of outlier chunks (0.3%), which are more challenging to approximate (Figure 3(b)), are stored as static cache on the GPU to preserve accuracy. As shown in Figure 2, our method outperforms the naive sparse attention approach [] and achieves higher sparsity, accelerating decoding.

Building on these insights, we present ShadowKV in Section 4, depicted in Figure 2, a high-throughput system for long-context LLM inference. Specifically, during pre-filling, we offload the value cache to the CPU, retaining only the low-rank pre-RoPE keys, along with compressed landmarks of the key cache and detected outliers for larger batch sizes. During decoding, landmarks are used to select chunk indices for key cache recovery and value cache fetching. We perform accurate sparse attention computation with selected KV pairs and static outliers to achieve high throughput.

Figure: Figure 2: Left: ShadowKV enhances long-context LLM inference throughput by offloading the value cache to the CPU while maintaining a low-rank key cache, landmarks, and outliers on the GPU. During decoding, it employs landmarks for efficient sparse attention, reducing computation and data movement. Right: ShadowKV effectively utilizes a limited KV budget to achieve high accuracy, theoretically reaching over 7 TB/s equivalent bandwidth on an A100, and empirically boosts generation throughput by 3.04$\times$ for Llama-3.1-8B with on a batch of 122K contexts.
Refer to caption: x5.png

Empirically, we conduct extensive experiments and ablation studies to demonstrate the effectiveness and efficiency of ShadowKV. In Section 5.1, we evaluate across various long-context LLMs, such as Llama-3-8B-1M [], Llama-3.1-8B [], GLM-4-9B-1M [], Yi-9B-200K [], Phi-3-Mini-128K [] and Qwen2-7B-128K [] using benchmarks including RULER [], LongBench [], and Needle In A Haystack [] with contexts up to 1M.

In Section 5.2, we demonstrate that ShadowKV can support 6$\times$ larger batch sizes and boost throughput by 3.04$\times$ compared to small batches on an A100 using Llama-3.1-8B, with each sample having a context length of 122K. We also present results across different models and context lengths, increasing throughput up to 2.97$\times$ for Llama-3-8B-1M, 2.56$\times$ for GLM-4-9B-1M, and 2.66$\times$ for Yi-9B-200K, even surpassing infinite batch size under the assumption of infinite GPU memory.

## 2 Related Works

#### Token Eviction.

To reduce memory footprint, eviction-based strategies keep a fixed size of KV cache to store the critical token KV pairs and discard unnecessary tokens. StreamingLLM [] addresses the limitations of window attention by retaining attention sinks and recent KV pairs. H2O [] introduces a low-cost eviction policy, updating the KV cache based on cumulative attention scores. LESS [] accumulates evicted token information by a constant-sized low-rank cache, which allows partial access to previously evicted information, along with tokens maintained by a sparse policy. SnapKV [] uses the local window of prompts to select important tokens for future generations. However, they suffer from performance degradation and information loss since the evicted tokens will never be recovered.

#### Dynamic Sparse Attention.

This line of work retains all KV cache but performs dynamic sparse attention within selected KV pairs to reduce inference latency. SparQ [] uses the norm of the query to decide an important subset of the key cache’s channels to calculate a metric to select relevant tokens. Quest [] segments tokens into pages and selects pages by approximating the highest attention within each page. Loki [] performs principal component analysis on the key cache using a calibration dataset, selecting tokens based on attention scores computed in low-dimensional space. TriForce [] combines sparse attention with speculative decoding [] for lossless acceleration. InfiniGen [] offloads all KV cache to the CPU and prefetches essential entries during decoding.

#### Quantization.

Several methods have been introduced to optimize KV cache quantization [, , ], reducing memory consumption while retaining accuracy. KIVI [] applies different quantization strategies for keys and values, quantizing the keys per-channel and the values per-token to 2-bit. Palu [] decomposes KV weight matrices offline, caching low-rank KV projections to achieve a higher compression rate. Quantization methods reduce the KV cache bit width, which is orthogonal to our approach.

## 3 Observations

We present two key insights of long-context LLMs that inspire ShadowKV’s design, as follows.

Figure: (a)
Refer to caption: x6.png

### 3.1 Low-Rank Keys and Offloaded Values for Storage

To reduce memory footprint, the low-rank nature of the KV cache has been explored by recent studies [, , ]. However, these methods focus on data-independent decomposition, either requiring training or achieving limited compression rates.

#### Observation.

In our study, by conducting SVD on the model weights $W_{k}$, $W_{v}$, the input $X$, the pre-/post-RoPE key cache, and the value cache of Llama-3.1-8B, we visualize the relative singular value distributions in Figure 1(a) together with the accuracy in Figure 3(a). As we observed, pre-RoPE keys have the lowest rank and can be compressed by 6$\times$ without performance degradation.

We also identify striking dynamic and static behaviors in low-rank keys between and within sequences, inspired by a related investigation in FFN layers []. Analogous to cosine similarity, we define $\mathcal{D}(\boldsymbol{H}_{1},\boldsymbol{H}_{2})=\langle\boldsymbol{H}_{1},
\boldsymbol{H}_{2}\rangle/r$ to be the similarity metric between low-rank subspaces of two rank-$r$ projection matrices, $\boldsymbol{H}_{1}$ and $\boldsymbol{H}_{2}$, where $\langle\cdot,\cdot\rangle$ is the Frobenius inner product(^2^22Since $\boldsymbol{H}_{1}$ and $\boldsymbol{H}_{2}$ are projection matrices, their squared Frobenius norms are the sum of their singular values which consist of $r$ 1’s and $d-r$ 0’s, i.e., $\|\boldsymbol{H}_{1}\|^{2}_{F}=r$. Thus, by Cauchy-Schwarz, $|\mathcal{D}(\boldsymbol{H}_{1},\boldsymbol{H}_{2})|\leq 1$. Additionally, $\mathcal{D}(\boldsymbol{H}_{1},\boldsymbol{H}_{2})\geq 0$ by the cyclic property of trace and positive semidefiniteness of projection matrices. Together, this shows $\mathcal{D}(\boldsymbol{H}_{1},\boldsymbol{H}_{2})\in[0,1]$, maximized or minimized when the projection matrices project onto identical or orthogonal subspaces, respectively.). In our case with truncated SVDs of pre-RoPE keys, let $\boldsymbol{K}_{1},\boldsymbol{K}_{2}\in\mathbb{R}^{n\times d}$ have rank-$r$ truncated SVDs, $\boldsymbol{\Phi}_{1}\boldsymbol{\Sigma}_{1}\boldsymbol{\Psi}^{\top}_{1}$ and $\boldsymbol{\Phi}_{2}\boldsymbol{\Sigma}_{2}\boldsymbol{\Psi}^{\top}_{2}$, respectively, where $\boldsymbol{\Phi}_{1}\in\mathbb{R}^{n\times r},\boldsymbol{\Sigma}_{1}\in
\mathbb{R}^{r\times r},\boldsymbol{\Psi}_{1}\in\mathbb{R}^{d\times r}$, and similarly for $\boldsymbol{\Phi}_{2}$, $\boldsymbol{\Sigma}_{2}$, and $\boldsymbol{\Psi}_{2}$. Then, $\mathcal{D}(\boldsymbol{\Psi}_{1}\boldsymbol{\Psi}_{1}^{\top},\boldsymbol{\Psi
}_{2}\boldsymbol{\Psi}_{2}^{\top})$ can measure the similarity between the low-rank subspaces of the two right singular matrices. Depicted in Figure 1(b), pre-RoPE keys between sequences do not strongly share similar low-rank subspaces, but extensions of the same sequence do.

#### Insights.

Our observation of the low-rank nature in the pre-RoPE keys indicates that storing the low-rank projections is sufficient for each sequence. By keeping the low-rank key cache on the GPU and offloading the value cache to the CPU since it is not low-rank, we can largely reduce the memory footprint. During decoding, selected KV pairs can be reconstructed on-the-fly for computation.

### 3.2 Accurate KV Selection for Fast Decoding

To further reduce the latency overhead in sparse attention, including fetching the selected value cache from the CPU and reconstructing the corresponding key cache, an accurate KV selection method is needed to minimize the sparse KV cache budget while maintaining the accuracy.

#### Observation.

We found most post-RoPE key cache exhibits spatial locality, with high cosine similarity to adjacent tokens, except for a few outliers. To quantify this, we conducted inference experiments on 128K contexts. We divided the post-RoPE keys into chunks of eight tokens and visualized the minimum cosine similarity between the chunk’s mean and its key cache, as shown in Figure 3(b). The results indicate that, apart from a few outliers, there is generally high cosine similarity, suggesting the mean values can serve as landmarks to approximate attention well within normal chunks.

#### Analysis.

This finding suggests that for the majority of chunks, we can maintain the mean value as compressed landmarks to select minimal important KV pairs (1.56%) accurately during decoding. Outlier chunks, which may contain dense or critical information and are difficult to approximate, are retained to ensure accuracy. Given their relatively small number (0.2–0.3%), storing them on the GPU is feasible without affecting memory capacity. Furthermore, as shown in Figure 3(c), considering the temporal locality of the KV cache, a cache policy [] can be leveraged to further reduce the latency overhead by 60% during decoding with optimized kernels [].

## 4 ShadowKV

In this section, we introduce ShadowKV, a high-throughput long-context LLM inference system. We first elaborate our algorithm in Section 4.1, covering both the pre-filling and decoding phases. Subsequently, in Section 4.2, we discuss the concept of theoretical equivalent bandwidth to illustrate the benefits of our approach.

Figure: Algorithm 1 ShadowKV Pre-filling
Refer to caption: x9.png

### 4.1 Algorithm

The algorithm of ShadowKV is divided into two main phases: pre-filling and decoding. The pre-filling phase involves low-rank decomposition of the post-RoPE key cache, offloading the value cache, and constructing landmarks to facilitate subsequent high-throughput decoding. The decoding phase includes accurate KV selection and efficient sparse KV cache reconstruction.

#### Pre-filling.

During the pre-filling phase, we optimize GPU memory usage by performing low-rank compression on the key cache of each layer and offloading values to the CPU. Specifically, as demonstrated in Algorithm 1 and Figure 4, we apply SVD on the pre-RoPE key cache and store only the low-rank representations for each layer. Post-RoPE key cache is segmented into chunks, with the mean of each chunk computed as landmarks. By computing the cosine similarity within these chunks, we identify poorly approximated tokens as outliers. This small set of outliers is gathered and stored on the GPU as the static cache, while the remaining key cache is maintained as compact landmarks, with the corresponding values offloaded to the CPU memory.

#### High-throughput Decoding.

For incoming queries, we first compute the approximate attention scores using the landmarks. As detailed in Algorithm 2, by identifying the top-k scoring chunk indices, the corresponding values are retrieved from the CPU, and the key cache is simultaneously reconstructed from low-rank projections, effectively concealing the construction of the key cache. Based on the insight that the KV cache has temporal locality, we build cache-aware CUDA kernels, reducing computation and value fetching by 60%. As shown in Figure 5, we conduct an index scan to detect the missed chunks and only rebuild the necessary KV pairs on-the-fly.

Figure: Algorithm 2 ShadowKV Decoding
Refer to caption: x10.png

Based on our observations in Section 3.1, future pre-RoPE keys within a sequence reside in a shared low-rank subspace with the context. As a result, an extension of our algorithm would be to store generated tokens as low-rank states using the same projections obtained from pre-filling to reduce the memory usage for future generations(^3^33If $\boldsymbol{\Psi}\in\mathbb{R}^{d\times r}$ is the right singular matrix calculated from the SVD of pre-RoPE context keys $\boldsymbol{K}\in\mathbb{R}^{s\times d}$, new pre-RoPE keys $\boldsymbol{K}^{\prime}\in\mathbb{R}^{s_{q}\times d}$ can be stored as $\boldsymbol{K}^{\prime}\boldsymbol{\Psi}$ and projected back up with $\boldsymbol{\Psi}^{\top}$ when needed.). For simplicity, we exclude it from the implementation.

### 4.2 Theoretical Equivalent Bandwidth

The benefit of ShadowKV in terms of increasing throughput can be analyzed through the concept of equivalent bandwidth. Consider each K or V vector as being $M$ bytes in size, with a sequence length of $S$, a chunk size of $C$, a selected chunk budget of $K$, $O$ outliers, and hit rate $\alpha$. During KV selection, ShadowKV loads approximately $M\times S/C$ bytes using the GPU memory bandwidth $B_{\text{GPU}}$. For value cache fetching, it loads $M\times K\times C$ bytes using the PCIe bandwidth $B_{\text{PCIe}}$ []. Since value movement and key cache reconstruction can be overlapped, we do not need to count key cache reconstruction here. Following this, ShadowKV performs standard attention computation for the top-k chunks and predefined outliers, requiring $2M\times(K+O)\times C$ bytes. The equivalent bandwidth of ShadowKV can be defined as:

$$ $B_{\text{equivalent}}=\frac{2SB_{\text{GPU}}}{S/C+2(K+O)C+(1-\alpha)KCB_{\text {GPU}}/B_{\text{PCIe}}}$ $$

For example, assuming C=8, S=128K, K=256, O=48, $B_{\text{PCIe}}$=31.5 GB/s, and $B_{\text{GPU}}$=2 TB/s for A100, the equivalent bandwidth of ShadowKV is calculated as 7.2 TB/s, which is 3.6$\times$ higher than A100 memory bandwidth. This result indicates that ShadowKV theoretically achieves a high equivalent bandwidth to accelerate attention computation. System implementation is detailed in Section A.1.

**Table 1: Performance of different models and different methods on RULER [] evaluated at length of 128K. ShadowKV outperforms other methods with a 1.56% sparse budget.**
| Methods | N-S1 | N-S2 | N-MK1 | N-MK2 | N-MQ | N-MV | QA-1 | QA-2 | VT | FWE | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B-1M | 100.00 | 100.00 | 98.96 | 98.96 | 98.96 | 95.57 | 75.00 | 48.96 | 78.54 | 71.85 | 86.68 |
| Loki | 18.75 | 1.04 | 2.08 | 0.00 | 1.56 | 0.78 | 4.17 | 13.54 | 26.04 | 25.35 | 9.33 |
| Loki (V only) | 41.67 | 6.25 | 37.50 | 1.04 | 8.07 | 30.73 | 10.42 | 19.79 | 51.67 | 37.50 | 24.46 |
| Quest | 100.00 | 100.00 | 98.96 | 77.08 | 97.65 | 93.49 | 60.42 | 50.00 | 77.08 | 65.63 | 82.03 |
| Quest (V only) | 100.00 | 100.00 | 98.96 | 85.42 | 97.92 | 95.49 | 70.83 | 46.88 | 78.75 | 65.63 | 83.99 |
| ShadowKV | 100.00 | 100.00 | 97.92 | 98.96 | 96.88 | 95.83 | 72.92 | 52.08 | 81.67 | 72.57 | 86.88 |
| GLM-4-9B-1M | 100.00 | 100.00 | 94.79 | 87.50 | 99.74 | 93.75 | 67.71 | 55.21 | 97.29 | 72.22 | 86.82 |
| Loki | 71.88 | 27.08 | 22.92 | 2.08 | 9.90 | 11.46 | 28.13 | 27.08 | 31.04 | 54.17 | 28.57 |
| Loki (V only) | 96.88 | 55.21 | 56.25 | 18.75 | 51.04 | 50.52 | 45.83 | 39.58 | 72.71 | 59.72 | 54.65 |
| Quest | 100.00 | 95.83 | 90.62 | 54.17 | 94.01 | 76.30 | 55.21 | 52.08 | 95.83 | 64.58 | 77.86 |
| Quest (V only) | 100.00 | 96.88 | 93.75 | 72.92 | 95.83 | 83.07 | 56.25 | 53.13 | 96.88 | 65.97 | 81.47 |
| ShadowKV | 100.00 | 100.00 | 95.83 | 83.33 | 98.70 | 87.76 | 69.79 | 55.21 | 97.50 | 68.06 | 85.62 |
| Llama-3.1-8B | 100.00 | 100.00 | 98.96 | 91.67 | 98.96 | 95.31 | 82.29 | 47.92 | 68.96 | 71.18 | 85.53 |
| Loki | 68.75 | 32.29 | 32.29 | 20.83 | 42.71 | 28.65 | 41.67 | 33.33 | 24.79 | 29.86 | 35.52 |
| Loki (V only) | 95.83 | 36.46 | 57.29 | 62.50 | 77.86 | 70.83 | 69.79 | 39.58 | 35.21 | 37.50 | 58.29 |
| Quest | 100.00 | 98.96 | 97.92 | 34.38 | 93.49 | 88.54 | 70.83 | 44.79 | 65.63 | 68.40 | 76.29 |
| Quest (V only) | 100.00 | 98.96 | 98.96 | 56.25 | 95.83 | 90.63 | 76.04 | 46.88 | 66.25 | 67.36 | 79.72 |
| ShadowKV | 100.00 | 100.00 | 100.00 | 83.33 | 97.92 | 92.19 | 81.25 | 48.96 | 67.08 | 64.93 | 83.57 |
| Yi-9B-200K | 100.00 | 100.00 | 86.46 | 62.50 | 64.58 | 32.55 | 44.79 | 39.58 | 36.87 | 89.93 | 65.73 |
| Loki | 34.38 | 2.08 | 2.08 | 0.00 | 0.00 | 0.52 | 22.92 | 21.88 | 0.00 | 25.00 | 10.89 |
| Loki (V only) | 59.38 | 11.46 | 18.75 | 5.21 | 4.43 | 2.08 | 22.92 | 31.25 | 0.00 | 35.07 | 19.06 |
| Quest | 100.00 | 98.96 | 79.17 | 26.04 | 56.51 | 31.77 | 32.29 | 31.25 | 51.04 | 71.88 | 57.89 |
| Quest (V only) | 100.00 | 100.00 | 80.21 | 45.83 | 59.37 | 31.90 | 36.45 | 34.37 | 53.54 | 71.88 | 61.36 |
| ShadowKV | 100.00 | 100.00 | 82.29 | 67.71 | 63.28 | 31.51 | 43.75 | 38.54 | 56.04 | 72.22 | 65.53 |

## 5 Empirical Evaluation

In this section, we showcase the effectiveness and efficiency of ShadowKV. Specifically,

- •
In Section 5.1, we show that ShadowKV can reduce the GPU memory footprint of the KV cache by over $6\times$ without accuracy degradation on a wide range of models and evaluation benchmarks.
- •
In Section 5.2, we demonstrate ShadowKV can support up to $6\times$ larger batch sizes and increase the inference throughput by up to $3.04\times$ without compromising model quality.
- •
In Section 5.3, we present extensive ablation studies that validate the effectiveness of each component of ShadowKV in optimizing GPU memory usage and enhancing performance.

All details (hyperparameters, datasets, etc.), along with additional experiments, are in Appendix A.

### 5.1 Accuracy Evaluation

We demonstrate that ShadowKV can reduce the GPU memory usage of the KV cache by $6\times$ while maintaining accuracy on a range of long-context tasks with a minimal sparse KV cache budget.

#### Setup.

We choose four widely used long-context models for our evaluation: Llama-3-8B-1M [], GLM-4-9B-1M [], Llama-3.1-8B [], and Yi-9B-200K []. We evaluate our approach on three challenging long-context benchmarks: RULER [], LongBench [], and Needle In A Haystack [], covering QA, multi-hop, reasoning, summarization, code completion. Needle In A Haystack is also tested on Phi-3-Mini-128K [] and Qwen2-7B-128K []. We set the chunk size to 8, the rank to 160, and the number of outliers to 48 for ShadowKV.

#### Baselines.

We include two dynamic sparse attention methods as baselines: Quest [] and Loki []. For all methods, we retain exact pre-filling and perform dynamic sparse attention during decoding, where the computation cost is set to 1/16 of full attention for selecting sparse KV pairs. We include two variants for each baseline: one where all the KV cache is offloaded, and another where only the value cache is offloaded. The former has similar latency to ShadowKV but a smaller sparse budget since ShadowKV only needs to fetch the value cache from the CPU. The latter aligns with the same sparse KV cache budget but significantly increases GPU memory usage. The latter one is marked as “V only” in the table.

**Table 2: Performance of various methods on different models with LongBench [] samples exceeding 4K tokens. ShadowKV outperforms other methods and maintains the accuracy.**
| Methods | NarratQA | MultiFQA | HotpotQA | MuSiQue | DuRead | GovRep | SAMSum | PassRetr | LCC | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B-1M | 18.98 | 41.84 | 36.79 | 21.47 | 31.93 | 34.18 | 35.96 | 81.50 | 56.07 | 39.86 |
| Loki | 2.26 | 10.19 | 5.48 | 3.16 | 12.17 | 28.97 | 7.84 | 40.52 | 31.44 | 15.78 |
| Loki (V only) | 3.20 | 21.01 | 12.41 | 3.86 | 17.07 | 31.24 | 16.23 | 52.57 | 38.10 | 21.74 |
| Quest | 20.13 | 36.63 | 35.00 | 18.14 | 24.55 | 27.11 | 35.63 | 79.00 | 53.64 | 36.65 |
| Quest (V only) | 17.26 | 39.51 | 36.78 | 18.71 | 26.41 | 29.49 | 35.80 | 79.50 | 60.05 | 38.17 |
| ShadowKV | 17.17 | 39.73 | 38.29 | 21.08 | 31.77 | 31.62 | 35.87 | 80.00 | 63.93 | 39.94 |
| GLM-4-9B-1M | 25.44 | 51.09 | 58.67 | 39.61 | 32.04 | 29.97 | 40.31 | 99.00 | 58.02 | 48.24 |
| Loki | 5.82 | 30.60 | 22.73 | 9.20 | 30.09 | 30.35 | 22.70 | 98.92 | 40.77 | 32.35 |
| Loki (V only) | 10.89 | 44.97 | 45.44 | 23.51 | 32.07 | 30.56 | 35.34 | 99.50 | 50.27 | 41.39 |
| Quest | 23.81 | 44.53 | 56.41 | 35.49 | 23.54 | 21.73 | 37.39 | 87.00 | 43.80 | 41.52 |
| Quest (V only) | 26.00 | 46.32 | 57.54 | 36.42 | 24.58 | 24.52 | 37.71 | 93.50 | 46.52 | 43.68 |
| ShadowKV | 26.50 | 51.31 | 59.09 | 38.87 | 32.92 | 28.54 | 38.70 | 96.50 | 58.55 | 47.89 |
| Llama-3.1-8B | 31.56 | 55.10 | 57.65 | 29.46 | 35.26 | 34.45 | 29.84 | 100.00 | 67.31 | 48.96 |
| Loki | 2.31 | 18.89 | 10.64 | 5.47 | 19.30 | 31.16 | 15.91 | 94.88 | 44.60 | 27.02 |
| Loki (V only) | 3.93 | 38.59 | 22.85 | 12.96 | 27.43 | 32.22 | 26.43 | 98.25 | 56.11 | 35.42 |
| Quest | 29.70 | 49.04 | 53.96 | 27.18 | 27.16 | 30.43 | 29.85 | 98.50 | 57.35 | 44.80 |
| Quest (V only) | 30.02 | 53.97 | 56.39 | 27.06 | 29.06 | 31.65 | 30.23 | 99.00 | 63.89 | 46.81 |
| ShadowKV | 30.93 | 55.20 | 57.32 | 29.13 | 31.85 | 32.79 | 30.40 | 99.50 | 66.03 | 48.13 |
| Yi-9B-200K | 13.88 | 30.02 | 52.46 | 28.20 | 22.29 | 30.25 | 19.08 | 67.00 | 73.50 | 37.41 |
| Loki | 1.63 | 2.73 | 16.21 | 4.87 | 4.75 | 2.13 | 4.95 | 0.00 | 38.72 | 8.44 |
| Loki (V only) | 1.96 | 10.39 | 21.31 | 7.36 | 6.78 | 9.15 | 10.02 | 4.00 | 58.75 | 14.41 |
| Quest | 10.57 | 25.83 | 46.06 | 23.04 | 17.09 | 17.11 | 20.59 | 50.50 | 67.70 | 30.94 |
| Quest (V only) | 14.56 | 25.73 | 48.73 | 24.73 | 18.44 | 20.83 | 20.08 | 57.50 | 71.13 | 33.53 |
| ShadowKV | 12.44 | 30.82 | 52.43 | 27.73 | 20.79 | 29.83 | 20.73 | 64.00 | 72.89 | 36.85 |

#### RULER.

As shown in Table 1, ShadowKV demonstrates excellent performance on 128K contexts. With a fixed sparse budget of 1.56%, Loki and Quest experience performance degradation. In contrast, ShadowKV is more robust and even outperforms original full attention on certain tasks, such as variable tracking. For complex tasks like multi-document QA or multi-key needle retrieval, other methods suffer from significant performance degradation while ShadowKV does not.

#### LongBench.

On LongBench, we evaluate our method with a range of realistic scenarios, including single-/multi-document question-answering, document summarization, code completion, information retrieval, etc. We only test on samples longer than 4K and set the sparse KV cache budget to 256 for this benchmark since it has shorter inputs compared to RULER. As shown in Table 2, ShadowKV outperforms other methods consistently and maintains the performance.

Figure: Figure 6: Needle In A Haystack.
Refer to caption: x11.png

#### Needle In A Haystack.

On the Needle In A Haystack dataset, as shown in Figure 6, ShadowKV shows the ability to process information at different positions across various context windows, ranging from 16K to 1M tokens. More experiments on a range of models can be found in Section A.3.

#### Integrate with Efficient Pre-filling Methods.

We also combined ShadowKV with a state-of-the-art efficient pre-filling method MInference []. As shown in Table 3, following the setting of MInference, we tested it on RULER with contexts scaling from 8K to 256K. This demonstrates that our method is compatible with pre-filling acceleration techniques. For some certain context length settings, we even see a slight performance improvement.

**Table 3: Performance of different methods on RULER [] using MInference [] in the pre-filling stage. ShadowKV is compatible with MInference.**
| Methods | 8K | 16K | 32K | 64K | 128K | 256K | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B-1M w/ MInference | 89.92 | 88.02 | 82.81 | 78.45 | 78.12 | 74.57 | 81.98 |
| ShadowKV w/ MInference | 90.47 | 88.12 | 83.28 | 77.71 | 78.32 | 74.31 | 82.04 |

Figure: Figure 7: Multi-turn NIAH.
Refer to caption: x12.png

#### Multi-turn Conversation Capability.

As the input context length scales, pre-filling becomes quite costly due to the quadratic growing computation time for attention, which means multi-turn conversation capability is important. To simulate multi-turn conversations, we challenged ShadowKV with a multi-turn needle retrieval task (Multi-turn NIAH). We also test two eviction-based methods in Figure 7, including SnapKV [] and StreamingLLM []. The performance of SnapKV drops significantly from the second round due to the required context information being different from the first round. Since SnapKV inevitably evicted tokens based on the first-turn conversation, it cannot successfully retrieve related information for future queries. In contrast, ShadowKV can maintain accuracy in the multi-turn conversation setting.

### 5.2 Efficiency Evaluation

To demonstrate the efficiency of ShadowKV, we deploy it into real-world large batch serving scenarios. By measuring the throughput during decoding across different models on A100, we show that ShadowKV can support up to $6\times$ larger batch sizes and boost throughput by up to 3.04$\times$.

#### Baselines.

The baseline selects the largest batch size that can fit entirely on the GPU with full attention. We also include results for the same batch size of ShadowKV and the infinite batch size, assuming infinite GPU memory capabilities(^4^44For the equivalent ShadowKV batch size, we evaluate a single Transformer block with FlashAttention and then project the number to the entire model. For the infinite batch size, we leverage A100’s theoretical memory bandwidth (2 TB/s) for attention computations.). We set the sparse budget to 1.56% for ShadowKV.

#### Results.

As shown in Table 4, ShadowKV demonstrates significant throughput improvements for various models on an A100, surpassing even those with infinite GPU memory. Notably, ShadowKV supports batch sizes up to $6\times$ larger and enhances throughput by up to $3.04\times$ compared to full attention, even surpassing infinite batch size assuming infinite GPU memory. While the gains for GLM-4-9B-1M and Yi-9B-200K are slightly lower, the improvements still reach up to $2.56\times$ and $2.66\times$ respectively, highlighting ShadowKV’s adaptability even with fewer KV heads.

**Table 4: Generation throughput (tokens/s) on an A100. The gray text in brackets denotes batch size.**
| Model | Context | Full Attention | ShadowKV | Gain | Full Attention (Inf) |
| --- | --- | --- | --- | --- | --- |
| Llama-3-8B-1M | 60K | 160.62 (8) | 455.14 (48) | 2.83$\times$ | 168.72 (48) / 273.07 (Inf) |
| (8 KV heads) | 122K | 80.77 (4) | 239.51 (24) | 2.97$\times$ | 83.05 (24) / 134.30 (Inf) |
|  | 244K | 40.37 (2) | 119.01 (12) | 2.95$\times$ | 52.00 (12) / 67.15 (Inf) |
| Llama-3.1-8B | 60K | 160.93 (8) | 472.77 (48) | 2.94$\times$ | 168.72 (48) / 273.07 (Inf) |
| (8 KV heads) | 122K | 80.78 (4) | 245.90 (24) | 3.04$\times$ | 83.05 (24) / 134.30 (Inf) |
| GLM-4-9B-1M | 60K | 241.05 (12) | 615.89 (50) | 2.56$\times$ | 266.24 (50) / 436.91 (Inf) |
| (4 KV heads) | 122K | 122.67 (6) | 293.40 (25) | 2.39$\times$ | 158.83 (25) / 214.87 (Inf) |
|  | 244K | 61.13 (3) | 136.51 (12) | 2.23$\times$ | 78.84 (12) / 107.44 (Inf) |
| Yi-9B-200K | 60K | 204.81 (10) | 544.36 (42) | 2.66$\times$ | 271.21 (42) / 364.09 (Inf) |
| (4 KV heads) | 122K | 101.44 (5) | 260.03 (21) | 2.56$\times$ | 133.53 (21) / 179.06 (Inf) |
|  | 244K | 46.74 (2) | 118.55 (10) | 2.54$\times$ | 65.79 (10) / 89.53 (Inf) |

### 5.3 Ablation Results

We present extensive ablation studies of ShadowKV, focusing on three key points: (1) sparse KV cache budget variations, (2) chunk size selections, and (3) pre-RoPE key cache rank choices.

Figure: Figure 8: Comparison results between the models with full cache, our ShadowKV, and Quest.
Refer to caption: x13.png

Figure: (a)
Refer to caption: x14.png

#### Sparse KV Cache Budget.

We examine ShadowKV’s performance across various tasks with different sparse budgets, as illustrated in Figure 8. ShadowKV consistently surpasses Quest under the same sparse budgets and achieves higher throughput. On most tasks, it maintains accuracy with just a 1.56% sparse budget compared to full attention and even improves slightly on some tasks.

#### Chunk Size.

As shown in Figure 9(a), increasing the chunk size allows for larger batch sizes. However, accuracy declines when the chunk size exceeds eight. Meanwhile, the chunk size choice has minimal impact on the chunk hit rate, which remains around 60%, as illustrated in Figure 9(b).

#### Rank of Pre-RoPE Keys.

We assess ShadowKV’s performance across various tasks using different ranks for pre-RoPE keys. As illustrated in Figure 9(c), accuracy increases with the rank up to approximately 160, after which it stabilizes near full-rank performance. Interestingly, the trends vary across tasks, and in some cases, low-rank approximations achieve better performance.

## 6 Conclusion

We present ShadowKV, a high-throughput inference system for long-context LLM inference. ShadowKV optimizes GPU memory usage through the low-rank key cache and offloaded value cache, allowing for larger batch sizes. It reduces decoding overhead by accurate sparse attention, boosting throughput while maintaining accuracy. Our empirical experiments demonstrate ShadowKV can support up to $6\times$ larger batch sizes and enhance throughput by up to $3.04\times$ on an A100 across various long-context models, including Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, and Yi-9B-200K. ShadowKV holds great promise for improving long-context LLM inference.

## Appendix A Experiment Details

In this section, our goal is to provide the details of the system implementation (mentioned in Section 4.2), experiment settings, and additional experiments (mentioned in Section 5).

### A.1 System Implementation.

We implement the framework based on PyTorch [, ] and dedicated kernels []. FlashAttention [, , ] is used for attention computation and some efficient fused kernels in Flashinfer [] and vLLM [] are used, including layer norm. To reduce memory movement and kernel launch overhead, we fuse some operations into CUDA kernels, including attention approximation, key cache low-rank reconstruction, value cache fetching, cache mechanism, etc. We leverage multi-streams to overlap the reconstruction of key cache and value cache fetching. We set the rank of pre-RoPE key cache to 160, chunk size to 8, and sparse KV cache budget to 1.56% for most cases.

### A.2 Dataset Details

LLMs are widely used in various fields [, , , , , , , , , , , , ], and we select three long-context benchmarks, detailed below.

- •
RULER [] consists of 13 complex tasks and supports adjustable context lengths, including retrieval, multi-hop tracking, aggregation, and QA tasks. For the test with MInference [], we set up test sets scaling from 8K to 256K for evaluation.
- •
LongBench [] is a challenging long-context benchmark that assesses the performance of LLMs in extended contexts. Featuring Chinese and English languages, LongBench encompasses 6 main categories and 21 diverse tasks, evaluating LLM capabilities across crucial long-text applications like single-/multi-document QA, summarization, code completion, etc.
- •
Needle In A Haystack [] is a long-context retrieval benchmark testing LLM’s performance with context window scales up to 1M tokens where information placed at various positions. We tested the retrieval capabilities of six long-context LLMs based on their context length.

### A.3 Needle In A Haystack

In addition to the Needle In A Haystack results for Llama-3-8B-1M shown in Figure 6, we also present results for GLM-4-9B-1M, Llama-3.1-8B, Yi-9B-200K, Phi-3-Mini-128K, and Qwen2-7B-128K, shown in Figure 10. Compared to full attention, using ShadowKV has minimal impact on the ability to understand semantic information across different context windows and needle depths. There is even a slight performance improvement for Yi-9B-200K.

Figure: (a) GLM-4-9B-1M
Refer to caption: x17.png

### A.4 InfiniteBench

InfiniteBench [] is a challenging long-context benchmark that consists of 10 tasks, including QA, coding, dialogue, summarization, and retrieval, with an average length of 214K.

**Table 5: Accuracy of different methods with different models on InfiniteBench [].**
| Methods | En.Sum | En.QA | En.MC | En.Dia | Zh.QA | Code.Debug | Math.Find | Retr.PassKey | Retr.Num |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B-1M | 23.05 | 18.14 | 65.06 | 10.50 | 12.47 | 24.36 | 37.14 | 100.00 | 100.00 |
| ShadowKV | 21.50 | 17.73 | 64.63 | 10.50 | 12.45 | 23.86 | 37.43 | 100.00 | 100.00 |
| GLM-4-9B-1M | 28.61 | 9.25 | 68.12 | 39.50 | 11.77 | 30.20 | 40.00 | 100.00 | 100.00 |
| ShadowKV | 23.22 | 8.48 | 68.56 | 32.50 | 11.27 | 30.46 | 40.00 | 100.00 | 100.00 |
| Llama-3.1-8B | 26.42 | 14.48 | 66.38 | 16.00 | 12.92 | 21.07 | 34.00 | 100.00 | 99.66 |
| ShadowKV | 24.23 | 13.83 | 66.38 | 16.50 | 12.76 | 21.07 | 34.00 | 100.00 | 94.41 |
| Yi-9B-200K | 8.88 | 10.61 | 61.57 | 5.50 | 13.88 | 21.57 | 23.71 | 100.00 | 99.66 |
| ShadowKV | 8.92 | 10.06 | 59.39 | 6.00 | 13.89 | 20.56 | 24.29 | 100.00 | 99.83 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: As listed in the weakness part.
soundness: 3
strengths: This paper is well-organized and clearly written. The proposed method is well-motivated, addressing relevant challenges, and is supported by thorough analysis. The evaluation is comprehensive and robust, effectively substantiating the claims and demonstrating thoughtful considerations. Overall, this is a strong submission for ICLR.
summary: This paper presents a high throughput LLM inference system for long sentence length. It proposes a compression method to leverage the low rank property of key cache and improve the sparse attention method by accurate KV selection.
weaknesses: 1. As shown in Fig. 8, some downstream tasks, such as 'Frequent Words Extraction,' perform significantly worse with sparse KV enabled. A brief analysis of why this approach underperforms for these types of tasks would be helpful, as well as any potential solutions to address these limitations.
2. The proposed solution is currently evaluated on an 8B model with a 128K sequence length. It would strengthen the paper to include an analysis of whether this approach scales effectively for larger models, such as a 70B model with an extremely long sequence length of 1M.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: My questions are listed in the weakness section.
soundness: 2
strengths: + This work presents a very interesting observation on the KV cache: pre-RoPE key cache is exceptionally low-rank compared to post-RoPE key cache, value cache and KV projection weights. Built on this observation, the proposed SHADOWKV significantly reduces the memory footprint of the Key cache.
+ The work also improves the previous sparse attention work including QUEST by introducing outlier KV cache.
+ This work also implements the inference system and shows actual throughput improvement on the real world A100 GPUs.
summary: The increasing KV-cache poses great challenges to long-context LLM inference. This work presents a long-context LLM inference system, SHADOWKV. It decreases the memory footprint by storing the low-rank key cache and offloading value cache to CPU, and reduces the decoding latency by reconstructing the sparse KV pairs on-the-fly. Evaluations show that SHADOWKV supports up to 6x larger batch sizes and improves throughput by up to 3X on A100 GPU while maintaining the accuracy.
weaknesses: - The proposed method is complex, including low-rank K cache, CPU offloaded V cache, outlier KV cache, and dynamic sparse attention. However,  the ablation study on each component is missing.
   - In terms of model accuracy: 
      - it is unclear how much accuracy improvement an extra outlier KV cache will bring.
      - previous work Quest uses Min-Max as landmark cache, ShadowKV adopts Mean as landmark cache. It is unclear how much accuracy improvement this change will bring.
   - In terms of efficiency:
      - Authors only show a rough prefiling latency breakdown in Figure 1(c). It is unclear unclear how long it takes for computing the outlier cache (i.e., reduce, cosine-similarity, top-k, gather) in the profiling stage, how long it takes for KV cache chunk selection (i.e, MatMul, Softmax, Max, TopK) in the decoding stage, how long it takes for recomputing the K cache from low-rank cache. These overheads seem to increase linearly with the context length. It would be better to see the efficiency breakdown on every part of the system under different context lengths (e.g., 128K, 256K, 512K).
      - it is unclear how SHADOWKV performs for extremely long context. For example, the authors evaluated on Llama-3-8B-1M but only with up to 128K context length.
     - this work lacks efficiency comparison against previous work LoKi, Quest and MInference.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 1
questions: * What exactly is the “sparse budget”?

* How does SHADOWKV leverage the temporal locality of the KV cache?

* What are the exact GPU memory savings of SHADOWKV? Including a quantitative discussion on GPU memory savings in the paper would be helpful.

* How can SHADOWKV handle the KV cache for newly generated tokens, and what would be the impact of this?
soundness: 2
strengths: * In-depth analysis on low-rank nature of key cache in comparison to value cache as well as weights.
* Leveraging spatial locality of post-ROPE key cache for dynamic sparse attention appears both novel and effective.
* Empirical results are impressive.
summary: This paper introduces SHADOWKV, a CPU-offloading-based system for long-context LLM inference. SHADOWKV addresses GPU memory constraints by leveraging the low-rank property of the key cache, storing a low-rank pre-ROPE key cache on the GPU while offloading the value cache to the CPU. Additionally, SHADOWKV stores landmarks—mean values of post-ROPE keys for adjacent tokens—along with low-rank pre-ROPE key cache in GPU memory. During decoding, SHADOWKV utilizes these landmark keys to identify significant tokens, selectively recovering their keys and retrieving their values from the CPU. Evaluations on long-context benchmarks demonstrate that SHADOWKV maintains high accuracy while significantly reducing GPU memory consumption and inference latency.
weaknesses: * Incomplete Descriptions

The term "sparse budget" is not clearly defined in the paper, which may lead to confusion. Additionally, while SHADOWKV claims to leverage the temporal locality of the KV cache to reduce computation and communication (by approximately 60%), it lacks any detailed explanations on what that feature is.

* Handling Newly Generated Tokens

While the paper says that it excludes the handling of newly generated tokens for simplicity, this issue is quite significant and should not be ignored. If not addressed, the KV cache for newly generated tokens could negate SHADOWKV’s key benefits of reduced GPU memory usage and lower inference latency, especially with long output sequences. Incorporating mechanisms to handle these tokens within SHADOWKV is essential, and the authors should evaluate and report on its impact on accuracy.

* Lack of Comparison with Infinigen

The paper does not sufficiently compare SHADOWKV with Infinigen [1], a closely related work that similarly stores low-rank key cache in GPU memory, offloads the value cache to the CPU, and selectively fetches important values based on approximate attention scores. Although the paper briefly discusses Infinigen, given the significant similarities, a more in-depth comparison with Infinigen should be made in order to highlight the main differentiator of SHADOWKV.

[1] Lee et al., "InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management", OSDI'24

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: The paper tackles a significant problem in LLM inference by reducing GPU memory usage through a hybrid CPU-GPU cache system, enabling long-context LLMs to operate more efficiently without sacrificing accuracy.

The system shows impressive throughput gains, handling up to six times larger batch sizes compared to baseline methods, which could substantially impact real-world LLM deployment.

SHADOWKV is tested on multiple models and benchmarks (e.g., Llama, GLM) across various tasks, demonstrating consistent performance improvements across all settings.
summary: The paper introduces SHADOWKV, a novel system that improves inference throughput for long-context LLMs. The key challenge addressed is the increasing memory footprint of KV caches as sequence length increases, which slows down inference. SHADOWKV offers a solution by offloading the value cache to the CPU while keeping a low-rank key cache on the GPU, reducing memory consumption without sacrificing performance. By employing a method of KV selection for sparse attention, it boosts throughput and supports larger batch sizes, showing improvements of up to 3.04× throughput on the A100 GPU.
weaknesses: The main issue is that, as far as I know, SVD is precision sensitive, but I didn't find any discussion about precision in the paper. My main question is what precision is used for ShadowKV and baselines. If you are using precision like FP16/FP32, my question is how does ShadowKV work on FP8/(FP8 & FP16) precision? If the precision is FP8, how does ShadowKV survive from precision-sensitive SVD?

For the speed evaluation, I only found the throughput (tokens/s), are there any experiments for the time of each operation separately?

Meta-review: After reviewing the entire discussion and considering input from all reviewers, I recommend the rejection of this paper. While the submission has merits, the unresolved concerns outweigh the strengths.  

**Strengths**:  
- The paper presents a novel observation about efficiently compressing pre-RoPE Key caches, reducing CPU-GPU communication costs. This is a potentially useful insight that has practical implications for offloading methods.  
- The revised methodology demonstrates improved scalability for long sequences while maintaining accuracy.  

**Weaknesses**:  
1. **Incremental Contribution**:  
   The proposed work builds on existing methods of CPU offloading and low-rank approximation. However, the enhancements, while effective, appear to be incremental rather than groundbreaking.   

2. **Presentation and Positioning**:  
   The paper struggles with positioning itself clearly within the existing literature. The unique contributions remain insufficiently distinguished from prior work, leaving ambiguity about its originality. This issue undermines the perceived value of the paper's contributions.  


Given these mixed reviews, the lack of consensus among reviewers, and the remaining issues of novelty and presentation, I believe the paper does not meet the threshold for acceptance in its current form. While the work demonstrates promise, a more substantial contribution and clearer articulation of its impact are necessary to justify publication.

Decision: Reject