TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
Authors: Dingyu Yao, Bowen Shen, Zheng Lin, Wei Liu, Jian Luan, Bin Wang, Weiping Wang
Accepted at: acl25

TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

Dingyu Yao1,2*, Bowen Shen1,2, Zheng Lin1,2†, Wei Liu3, Jian Luan3, Bin Wang3, Weiping Wang1
1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
3MiLM Plus, Xiaomi Inc, Beijing, China
{yaodingyu, shenbowen, linzheng, wangweiping}@iie.ac.cn
{liuwei40, luanjian, wangbin11}@xiaomi.com

Abstract
The Key-Value (KV) cache in generative large language models (LLMs) introduces substantial memory overhead. Existing works mitigate this burden by offloading or compressing the KV cache. However, loading the entire cache incurs significant latency due to PCIe bandwidth bottlenecks in CPU-GPU communication, while aggressive compression causes notable performance degradation. We identify that certain layers in the LLM need to maintain global information and are unsuitable for selective loading. In contrast, other layers primarily focus on a few tokens with dominant activations that potentially incur substantial quantization error. This observation leads to a key insight that loading dominant tokens and quantizing all tokens can complement each other. Building on this insight, we propose a hybrid compression method, TailorKV, which seamlessly integrates quantization and offloading. TailorKV develops an inference framework along with a hardware-friendly implementation that leverages these complementary characteristics. Extensive long-context evaluations exhibit that TailorKV achieves nearly lossless performance under aggressive compression settings, outperforming the state-of-the-art. Particularly, the Llama-3.1-8B with 128k context can be served within a single RTX 3090 GPU, reaching 82 ms per token during decoding1.

1 Introduction
Large language models (LLMs) have demonstrated exceptional performance in tasks such as multi-turn dialogues (Chiang et al., 2023) and multi-document understanding (Bai et al., 2024). In response to the growing complexity of tasks, recent LLMs have expanded their context windows to over 128k tokens, e.g., GPT-4 (Achiam et al., 2023) and DeepSeek V3 (Liu et al., 2024a). Typically, the inference of LLMs is auto-regressive, with the Key-Value (KV) cache stored in memory to avoid recomputation. However, the size of KV cache grows linearly with sequence length, leading to much higher GPU memory consumption and inference latency.

Recent studies have proposed sparse attention mechanisms to reduce KV cache usage. These methods fall into two categories: irreversible eviction and recallable selection. Irreversible eviction methods (Li et al., 2024; Zhang et al., 2023; Xiao et al., 2024b) suffer from accuracy degradation due to permanently discarding tokens that may later become crucial, particularly in multi-turn dialogues. Recallable selection methods adopt a different approach by maintaining the entire KV cache while selecting only a subset of tokens for processing. However, methods like Quest (Tang et al., 2024) and SparQ (Ribar et al., 2024) encounter memory limitations when attempting to store all tokens on the GPU. Although CPU offloading mitigates GPU memory limitations, existing approaches (Xiao et al., 2024a; Zhang et al., 2024a) still require retrieving a substantial portion of tokens (around 20%), introducing significant decoding latency overheads due to slow data transfer between CPU RAM and GPU RAM.

To optimize accuracy, memory, and latency simultaneously, we first analyze the compression preferences for the KV cache based on layer characteristics. Prior researches (Feng et al., 2024; Cai et al., 2024) applied different sparsity rates to different layers under the same compression strategy. However, our analyses demonstrate that performance degradation primarily stems from the application of unsuitable compression at the layer level (Section 3). Therefore, we suggest that shallow layers, which exhibit dense attention patterns and emphasize global information (Wan et al., 2025), are better suited for uniform compression like quantization. Conversely, layers with a few dominant
tokens and largely redundant information are well-suited for sparsity, as performance can be maintained by retrieving only the dominant tokens.

Building upon these insights, we propose a novel framework, TailorKV, which employs hybrid compression techniques to reduce GPU memory usage. We introduce an identification metric to classify Transformer layers into two distinct types: (i) quantization-friendly layers, which preserve global information from a macro perspective, and (ii) sparsity-friendly layers, which capture crucial information from a micro perspective. This design enables quantization-friendly layers to employ static quantization, achieving a high compression ratio (1-bit per floating-point number) while maintaining model quality. Meanwhile, for sparsity-friendly layers, the system offloads the KV cache to CPU memory during prefilling and dynamically retrieves the Top-K tokens during decoding. By aligning compression strategies with the characteristics of each layer, this tailored approach significantly reduces overall memory consumption.

The accuracy and efficiency of TailorKV are evaluated on various backbone LLMs using long-context benchmarks. The results demonstrate that TailorKV drastically reduces memory usage by quantizing 1 to 2 layers to 1-bit precision and loading only 1% to 3% of the tokens for the remaining layers while maintaining nearly lossless performance. Specifically, TailorKV achieves a decoding latency of 82 ms for Llama-3.1-8B with a 128k-context on a single RTX 3090 (PCIe 1.0)\footnote{We combine TailorKV with 4-bit weight-only quantization (Lin et al., 2024) for prefill phase memory allocation.}, yielding a 53.7% reduction in peak GPU memory usage. The key contributions are summarized as follows.

• We identify layer-specific compression preferences and develop an identification metric to determine optimal compression strategies for different layers in the model.

• We present TailorKV, a hybrid KV cache compression framework that combines quantization and offloading techniques through an algorithm-system co-design, preserving both model accuracy and execution efficiency.

• Extensive experiments on long-context benchmarks demonstrate the nearly lossless performance of TailorKV with minimal GPU memory consumption and acceptable latency.

2 Preliminaries

2.1 Attention and KV Cache
LLM inference consists of two stages: *prefill* and *decode*. During prefilling, the entire prompt is used to generate the first token. Consider the prompt embedding \( \mathbf{X} \in \mathbb{R}^{n \times d} \) along with the weight matrices \( \mathbf{W}_i^q, \mathbf{W}_i^k, \mathbf{W}_i^v \in \mathbb{R}^{d \times d_h} \) for head \( i \in [1, h] \), where \( n \) is the sequence length, \( d \) is the hidden dimension and \( d_h \) is the head dimension. The keys and values for head \( i \) are computed and cached, as follows:

\[
\mathbf{K}_i = \mathbf{X} \mathbf{W}_i^k, \quad \mathbf{V}_i = \mathbf{X} \mathbf{W}_i^v.
\]

During decoding, the new token embedding \( \mathbf{x} \in \mathbb{R}^{1 \times d} \) is computed iteratively to produce the query, key, and value vectors. The cache is updated and the output \( o \) of each attention head is computed as:

\[
\mathbf{K}_i \leftarrow \mathrm{Cat}[\mathbf{K}_i, \mathbf{x} \mathbf{W}_i^k], \quad \mathbf{V}_i \leftarrow \mathrm{Cat}[\mathbf{V}_i, \mathbf{x} \mathbf{W}_i^v],
\]
\[
\mathbf{a}_i = \mathrm{Softmax}(\mathbf{q}_i \mathbf{K}_i^\top / \sqrt{d_h}), \quad \mathbf{o}_i = \mathbf{a}_i \mathbf{V}_i,
\]
where \( \mathbf{q}_i = \mathbf{x} \mathbf{W}_i^q \), and the attention outputs from all heads are concatenated and sent to the FFN.

2.2 Quantization of KV Cache
Quantization converts continuous or high-precision values into lower-precision discrete representations. Given a tensor \( \mathbf{X} \) in high precision, the typical uniform quantization process can be expressed as:

\[
\mathbf{X}_Q = \mathrm{Quant}_b(\mathbf{X}, s, z)
= \mathrm{clamp}(\lfloor \frac{\mathbf{X} - z}{s} \rfloor, 0, 2^b - 1),
\]

where \( \mathbf{X}_Q \) represents the quantized tensor in \( b \)-bit precision, with \( z = \min \mathbf{X} \) as the zero-point and \( s = \frac{\max \mathbf{X} - \min \mathbf{X}}{2^b - 1} \) as the scaler. The clamp function restricts values to the \( b \)-bit integer range and \( \lfloor \cdot \rfloor \) denotes the rounding function.

2.3 GPU-CPU Co-execution
As the sequence length increases, the size of the KV cache grows, significantly raising the demand for GPU resources. For example, with a sequence length of 512k, Llama-2-7B (Touvron et al., 2023) requires up to 256GB of memory for the KV cache. Current LLM serving systems (Kwon et al., 2023; Qin et al., 2025) employ an offloading strategy that stores the KV cache in cost-effective CPU memory and loads it onto the GPU during inference. However, I/O transfer latency becomes the bottleneck in inference due to the low-bandwidth PCIe
Figure 1: Observations on attention. (a) Attention weights on Llama-2-7B-32K-Instruct. Detailed visualizations are in Appendix I. (b) Sparse error of different models on the 2WikiMQA dataset, with only the top 5% of attention scores retained. (c) Sparse error on different datasets, with only the top 5% of attention scores retained.

<table>
  <tr>
    <th>Strategy</th>
    <th>RB-P</th>
    <th>LCC</th>
    <th>GovReport</th>
    <th>TriviaQA</th>
  </tr>
  <tr>
    <td>16-bit</td>
    <td>56.7</td>
    <td>63.4</td>
    <td>34.9</td>
    <td>91.6</td>
  </tr>
  <tr>
    <td>1-bit (<i>KIVI</i>)</td>
    <td>24.4</td>
    <td>26.2</td>
    <td>8.3</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>1-bit (\( \mathbb{L} = \{0\} \))</td>
    <td><b>57.1</b></td>
    <td><b>62.6</b></td>
    <td><b>34.9</b></td>
    <td><b>92.1</b></td>
  </tr>
  <tr>
    <td>1-bit (\( \mathbb{L} = \{2\} \))</td>
    <td>53.0</td>
    <td>59.3</td>
    <td>32.6</td>
    <td>91.9</td>
  </tr>
  <tr>
    <td>1-bit (\( \mathbb{L} = \{10\} \))</td>
    <td>52.3</td>
    <td>59.3</td>
    <td>31.2</td>
    <td>90.7</td>
  </tr>
  <tr>
    <td>1-bit (\( \mathbb{L} = \{18\} \))</td>
    <td>53.7</td>
    <td>60.0</td>
    <td>34.9</td>
    <td>89.4</td>
  </tr>
</table>

Table 1: Results of 1-bit quantization on different layers, using Llama-3.1-8B. \( \mathbb{L} \) denotes the quantized layer. <i>KIVI</i> (Liu et al., 2024d) is an advanced KV cache quantization algorithm.

interface (Zhang et al., 2024b). For instance, transferring the KV cache of a single layer (\( \approx 8GB \)) from the CPU memory to the RTX 3090 GPU via PCIe 1.0 link (4GB/s) takes around 2s, while the attention computation for a single layer on the RTX 3090 GPU only takes around 10ms. Thus, on-demand fetching is currently the most common approach to reduce GPU idle time.

3 Motivations and Observations

Layers have compression preferences. In contrast to previous belief (Li et al., 2024; Zhang et al., 2023), we propose that not all layers are suitable for sparsity. To quantify the sparsity challenges during decoding, we define the sparse error \( \mathcal{E} \). Let \( \mathbf{a} \in \mathbb{R}^{1 \times n} \) represents the attention weight as defined in Equation 3, and let \( \mathcal{M} \in \{0, 1\}^n \) denote a binary mask that selects the top \( k \) elements of \( \mathbf{a} \). The sparse error \( \mathcal{E} \) for each head is defined as:

\[
\hat{\mathbf{a}} = \mathbf{a} \odot \mathcal{M}, \quad \mathcal{E} = 1 - \sum_{i=1}^n \hat{a}_i.
\]

As shown in Figure 1a, layers with dense attention distributions exhibit higher sparse errors compared to those with sparse distributions. Additionally, we observe sparse error patterns across models and datasets. Figure 1b shows that sparse errors are similar across models, with higher sparse errors in shallower layers (e.g., 0, 1). Figure 1c shows that the distribution of sparse errors remains consistent across various datasets for the same model.

Similarly, not all layers are suitable for quantization. As shown in Table 1, quantizing the KV cache to 1-bit leads to significant performance degradation. This degradation is primarily caused by layers with sparse distributions, which are more sensitive to quantization. In contrast, quantizing the dense layer (e.g., 0th) incurs no performance loss.

These findings highlight the need for a tailored KV cache compression strategy. We regard layers with dense distributions as <b>quantization-friendly</b>, which focus on global information, and layers with sparse distributions as <b>sparsity-friendly</b>, which prioritize crucial information.

Attention scores correlate with outliers. Each channel in the key and query contributes to the attention scores through their dot product, as ex-
Figure 3: System overview of TailorKV. Offline identification categorizes the layers into quantization-friendly and sparsity-friendly. For quantization-friendly layers, we employ aggressive static quantization. For sparsity-friendly layers, we dynamically retrieve Top-K tokens. Critical current query and critical key cache represent the outliers in the query and key cache, respectively.

pressed by the formula \( \mathbf{q} \mathbf{K}^T \). Figure 2 (Top) illustrates that some channels have large magnitudes in the query and key. It follows from the dot product formula that attention scores correlate with these outliers. A recent method (Yang et al., 2024b) focuses on static channel sparsity, utilizing offline calibration technique to identify high-magnitude channels. However, we find that the sparsity of query and key channels is dynamic rather than static. As shown in Figure 2 (Bottom), outliers in the query and key do not consistently appear in fixed positions; instead, they may appear in any position. Furthermore, dynamically selecting high-magnitude channels improves the recall of dominant tokens compared to using a static offline strategy. This claim is empirically validated in Section 5.4.

4 Methodology

4.1 Offline Identification

Empirical observation in Section 3 suggests that some layers benefit more from quantization, while others are better suited for sparsity. To avoid disrupting the standard inference, we apply an offline strategy to identify the compression preference of each layer. In this phase, we introduce a metric—dense preference score \( \mathcal{P} \)—to assess whether each attention layer favors quantization or sparsity. Given a prompt length \( n \), we first use the most recent \( n_q \) query vectors \( \mathbf{Q}_{\text{last}, q} \in \mathbb{R}^{n_q \times d_h} \) and the key vectors \( \mathbf{K} \in \mathbb{R}^{n \times d_h} \) to compute the attention score matrix \( \hat{\mathbf{A}} \) for each head during prefilling:
\[
\hat{\mathbf{A}} = \operatorname{Softmax}(\mathbf{Q}_{\text{last}, q} \mathbf{K}^T / \sqrt{d_h}).
\]
Next, we select the top \( k \) indices from \( \hat{\mathbf{A}} \) and sum the top \( k \) elements in order to compute the dense preference score \( \mathcal{P} \):
\[
\hat{\mathcal{I}} = \left\{ (i, j) \mid \operatorname{Top}_k(\hat{\mathbf{A}}_{i,:}, k) \right\}_{i=1}^{n_q},
\]
\[
\mathcal{P} = n_q - \sum_{(i,j) \in \hat{\mathcal{I}}} \hat{\mathbf{A}}_{i,j}.
\]
If the dense preference score \( \mathcal{P}_l \) of layer \( l \) exceeds the threshold \( \tau \), the layer is regarded as quantization-friendly; otherwise, it is deemed sparsity-friendly. This can be formalized as:
\[
C(l) = \begin{cases}
\text{quantization-Friendly}, & \text{if } \mathcal{P}_l > \tau, \\
\text{Sparsity-Friendly}, & \text{otherwise}.
\end{cases}
\]
The threshold \( \tau \) is a predefined hyperparameter, and its optimal value is determined through experimentation on the synthetic Longbench task. The metric \( \mathcal{P} \) consistently assesses the same model across various datasets (for details, see Appendix C). After layer-level identification, we apply dynamic retrieval for sparsity-friendly layers and static quantization for quantization-friendly layers. The overall workflow is shown in Figure 3.

4.2 Dynamic Retrieval

For sparsity-friendly layers, we propose a dynamic retrieval algorithm with an asynchronous system design. Figure 3 shows the management framework of the CPU memory pool and GPU memory buffer. To facilitate LLM inference on memory-limited devices, we offload the KV cache to lower-cost CPU memory layer by layer during prefilling. Subsequently, we retrieve the Top-K tokens on demand during decoding, thus minimizing communication overhead. The core design is illustrated in Figure 4.

As explained in Section 3, attention scores correlate with outliers in the query and key. To more
accurately assess token importance, we approximate attention scores prior to original operation based on this insight. We first estimate the critical channels to identify outliers in the query and key cache, referred to as the critical current query and critical key cache. Since the critical key cache resides in the CPU, we employ prefetching to load it in advance. We leverage inter-layer similarity to predict the critical channels ahead of time (for a detailed explanation, see Appendix B). The similarity between adjacent layers arises from the residual connection, as validated in prior research (Lee et al., 2024). At layer \( l - 1 \), we estimate the query \( \hat{q} \) for layer \( l \), using the weight matrix from layer \( l \) and the hidden state from layer \( l - 1 \). The contribution of the \( i \)-th channel to the attention scores is computed via element-wise multiplication of \( \hat{q} \) and \( K_i \):

\[
s_i = |\hat{q}_i| \cdot \max(|K_i|), \quad i = 1, 2, ..., d_h.
\]

Next, we prefetch the \( l \)-th layer’s critical key cache based on \( s \), using double buffering—one buffer for writing and the other for reading—to enable concurrent execution. Then, we retrieve the Top-K tokens by approximating the attention scores at \( l \)-th layer based on the critical current query and the critical key cache, followed by fetching the Top-K tokens. Figure 5 outlines the computation and communication during decoding. The only non-overlapable operation is fetching Top-K tokens, as it depends on the current layer’s query. TailorKV demonstrates how a heterogeneous design overcomes resource constraints by leveraging CPU-GPU co-execution.

Figure 4: Two-stage dynamic retrieval process: Stage 1 estimates critical channels at layer \( l - 1 \) and prefetches critical key cache for layer \( l \). Stage 2 approximates attention scores and selects Top-K tokens at layer \( l \).

Figure 5: Timeline of dynamic retrieval. Blue signifies computation and pink signifies communication.

4.3 Static Quantization

Unlike traditional quantization methods (Liu et al., 2024d; Yang et al., 2024a; He et al., 2024), TailorKV focuses on ensuring that each layer "plays its role," thus enabling more aggressive compression scheme, such as 1-bit quantization. As illustrated in Figure 2, outliers are present in the key cache along the channel dimension, while the value cache contains no outliers. For quantization-friendly layers, we apply static per-token quantization to the value cache and static per-channel quantization to the key cache (Liu et al., 2024d). As shown in Equation 4, we introduce a 1-bit quantization kernel and also implement FP16×INT1 GEMV to improve hardware performance under aggressive compression.

4.4 Memory Footprint Analysis

Let the number of layers be \( L \), the number of heads be \( h \), the sequence length be \( n \), and the head dimensions be \( d_h \). All input tokens are represented in FP16. We present a comparison of GPU memory usage across different methods in Table 2. TailorKV mainly manages a quantized KV cache buffer in quantization-friendly layers and a critical key buffer in sparsity-friendly layers. The quantization zero-point and scaler are stored in FP16 format.

<table>
  <tr>
    <th>Method</th>
    <th>Memory</th>
    <th>Parameters</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>2Lnhd_h</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>2aLnhd_h</td>
    <td>budget: \( \alpha \)</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>2Lnhd_h(1 + \( \frac{1}{2} \))</td>
    <td>page size: \( \beta \)</td>
  </tr>
  <tr>
    <td>Ours (Q)</td>
    <td>2l_qnhd_h(\( \frac{1}{16} + \frac{2}{9} \))</td>
    <td>bit size: 1, group size: \( g \), num layers of Q: \( l_q \)</td>
  </tr>
  <tr>
    <td>Ours (S)</td>
    <td>2nhds</td>
    <td>num critical channel: \( d_s \)</td>
  </tr>
</table>

Table 2: Comparison of memory usage among different methods. The symbols Q and S denote the quantization-friendly layer and sparsity-friendly layer, respectively.
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="7">LongBench</th>
    <th colspan="7">InfiniteBench</th>
  </tr>
  <tr>
    <th>Tokens</th>
    <th>SD.QA</th>
    <th>MD.QA</th>
    <th>Summ</th>
    <th>FS.L</th>
    <th>Code</th>
    <th>Synth</th>
    <th>Avg.</th>
    <th>Tokens</th>
    <th>Retr</th>
    <th>Dia</th>
    <th>Novel</th>
    <th>Math</th>
    <th>Code</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>128k</td>
    <td>49.6</td>
    <td>50.9</td>
    <td>31.2</td>
    <td>69.4</td>
    <td>60.0</td>
    <td>53.5</td>
    <td>53.8</td>
    <td>128k</td>
    <td>99.6</td>
    <td>19.0</td>
    <td>30.2</td>
    <td>34.0</td>
    <td>22.8</td>
    <td>44.0</td>
  </tr>
  <tr>
    <td>StreamLLM</td>
    <td>192</td>
    <td>26.3</td>
    <td>42.7</td>
    <td>17.9</td>
    <td>50.0</td>
    <td>48.2</td>
    <td>53.5</td>
    <td>40.6</td>
    <td>1024</td>
    <td>3.2</td>
    <td>7.0</td>
    <td>23.7</td>
    <td>34.0</td>
    <td>22.8</td>
    <td>18.3</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>192</td>
    <td>35.2</td>
    <td>48.1</td>
    <td>20.2</td>
    <td>56.5</td>
    <td>52.8</td>
    <td>52.5</td>
    <td>45.2</td>
    <td>1024</td>
    <td><b>96.6</b></td>
    <td>9.5</td>
    <td>27.4</td>
    <td>34.0</td>
    <td>22.8</td>
    <td>41.0</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>192</td>
    <td>40.1</td>
    <td>46.9</td>
    <td>20.7</td>
    <td>61.6</td>
    <td>48.0</td>
    <td>52.4</td>
    <td>46.2</td>
    <td>1024</td>
    <td>64.4</td>
    <td>14.0</td>
    <td>25.7</td>
    <td>34.0</td>
    <td><b>25.1</b></td>
    <td>33.8</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>192</td>
    <td>48.4</td>
    <td>49.5</td>
    <td>27.0</td>
    <td>67.3</td>
    <td>56.3</td>
    <td>53.6</td>
    <td>51.7</td>
    <td>1024</td>
    <td>5.5</td>
    <td>15.0</td>
    <td>27.5</td>
    <td>34.0</td>
    <td>23.3</td>
    <td>21.5</td>
  </tr>
  <tr>
    <td>TailorKV-1</td>
    <td>64(+128)</td>
    <td>48.2</td>
    <td>50.9</td>
    <td>29.2</td>
    <td>68.1</td>
    <td><b>58.3</b></td>
    <td>53.4</td>
    <td>52.6</td>
    <td>128(+896)</td>
    <td>86.5</td>
    <td>18.0</td>
    <td>28.9</td>
    <td><b>34.0</b></td>
    <td>22.8</td>
    <td>40.4</td>
  </tr>
  <tr>
    <td>TailorKV-2</td>
    <td>64(+128)</td>
    <td><b>49.3</b></td>
    <td>50.5</td>
    <td><b>29.4</b></td>
    <td><b>68.7</b></td>
    <td>58.1</td>
    <td>53.3</td>
    <td><b>52.9</b></td>
    <td>128(+896)</td>
    <td>94.8</td>
    <td><b>18.5</b></td>
    <td><b>30.0</b></td>
    <td><b>34.0</b></td>
    <td>22.8</td>
    <td><b>42.6</b></td>
  </tr>
  <tr>
    <td>Yi-9B</td>
    <td>200k</td>
    <td>36.6</td>
    <td>44.7</td>
    <td>28.8</td>
    <td>60.6</td>
    <td>69.6</td>
    <td>35.0</td>
    <td>47.0</td>
    <td>200k</td>
    <td>100.0</td>
    <td>2.5</td>
    <td>25.2</td>
    <td>23.4</td>
    <td>26.3</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>StreamLLM</td>
    <td>192</td>
    <td>21.3</td>
    <td>33.6</td>
    <td>11.0</td>
    <td>44.1</td>
    <td>51.8</td>
    <td>14.7</td>
    <td>30.6</td>
    <td>1024</td>
    <td>1.5</td>
    <td>2.5</td>
    <td>24.2</td>
    <td>23.7</td>
    <td>21.3</td>
    <td>16.4</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>192</td>
    <td>25.0</td>
    <td>38.8</td>
    <td>11.9</td>
    <td>49.0</td>
    <td>59.7</td>
    <td>18.8</td>
    <td>35.0</td>
    <td>1024</td>
    <td>59.0</td>
    <td>3.0</td>
    <td>24.9</td>
    <td>22.5</td>
    <td><b>26.6</b></td>
    <td>30.0</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>192</td>
    <td>29.2</td>
    <td>37.9</td>
    <td>15.4</td>
    <td>57.5</td>
    <td>59.6</td>
    <td>25.7</td>
    <td>39.1</td>
    <td>1024</td>
    <td>98.4</td>
    <td>4.0</td>
    <td>21.8</td>
    <td>18.2</td>
    <td>18.7</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>192</td>
    <td>32.4</td>
    <td>41.6</td>
    <td>19.2</td>
    <td>58.6</td>
    <td>64.4</td>
    <td>27.8</td>
    <td>42.0</td>
    <td>1024</td>
    <td>7.8</td>
    <td>2.0</td>
    <td>25.3</td>
    <td>22.2</td>
    <td>25.6</td>
    <td>18.5</td>
  </tr>
  <tr>
    <td>TailorKV-1</td>
    <td>64(+128)</td>
    <td><b>38.0</b></td>
    <td><b>44.3</b></td>
    <td><b>27.3</b></td>
    <td><b>60.2</b></td>
    <td><b>66.3</b></td>
    <td>24.3</td>
    <td><b>44.7</b></td>
    <td>128(+896)</td>
    <td><b>98.7</b></td>
    <td>2.5</td>
    <td><b>26.6</b></td>
    <td><b>24.0</b></td>
    <td>21.3</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>TailorKV-2</td>
    <td>64(+128)</td>
    <td>35.6</td>
    <td>43.5</td>
    <td><b>27.3</b></td>
    <td>60.1</td>
    <td>66.0</td>
    <td>23.5</td>
    <td>44.0</td>
    <td>128(+896)</td>
    <td>98.5</td>
    <td><b>4.5</b></td>
    <td>25.3</td>
    <td><b>24.0</b></td>
    <td>24.9</td>
    <td><b>39.4</b></td>
  </tr>
  <tr>
    <td>Yi-6B</td>
    <td>200k</td>
    <td>32.4</td>
    <td>15.3</td>
    <td>1.3</td>
    <td>49.9</td>
    <td>69.8</td>
    <td>9.5</td>
    <td>29.7</td>
    <td>200k</td>
    <td>99.2</td>
    <td>0.0</td>
    <td>25.2</td>
    <td>6.8</td>
    <td>26.9</td>
    <td>37.1</td>
  </tr>
  <tr>
    <td>StreamLLM</td>
    <td>192</td>
    <td>20.0</td>
    <td>11.6</td>
    <td><b>1.6</b></td>
    <td>34.0</td>
    <td>44.6</td>
    <td>4.0</td>
    <td>20.4</td>
    <td>1024</td>
    <td>2.0</td>
    <td>0.0</td>
    <td>21.8</td>
    <td>4.8</td>
    <td>25.8</td>
    <td>13.5</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>192</td>
    <td>24.2</td>
    <td>13.0</td>
    <td><b>1.6</b></td>
    <td>38.5</td>
    <td>51.2</td>
    <td>3.7</td>
    <td>23.3</td>
    <td>1024</td>
    <td>55.7</td>
    <td>2.5</td>
    <td>23.2</td>
    <td>4.5</td>
    <td><b>26.9</b></td>
    <td>26.4</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>192</td>
    <td>26.5</td>
    <td>12.5</td>
    <td>0.3</td>
    <td>46.9</td>
    <td>51.9</td>
    <td><b>8.5</b></td>
    <td>26.2</td>
    <td>1024</td>
    <td><b>99.5</b></td>
    <td>3.0</td>
    <td>22.0</td>
    <td>5.1</td>
    <td>26.6</td>
    <td>35.8</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>192</td>
    <td>30.4</td>
    <td>14.5</td>
    <td>0.6</td>
    <td>48.0</td>
    <td>55.8</td>
    <td>4.0</td>
    <td>27.3</td>
    <td>1024</td>
    <td>6.9</td>
    <td>1.5</td>
    <td>24.6</td>
    <td>5.7</td>
    <td><b>26.9</b></td>
    <td>16.3</td>
  </tr>
  <tr>
    <td>TailorKV-1</td>
    <td>64(+128)</td>
    <td><b>32.5</b></td>
    <td><b>15.4</b></td>
    <td>1.4</td>
    <td><b>49.7</b></td>
    <td>55.9</td>
    <td>4.0</td>
    <td><b>28.3</b></td>
    <td>128(+896)</td>
    <td>98.7</td>
    <td>2.5</td>
    <td><b>25.3</b></td>
    <td>7.7</td>
    <td>26.4</td>
    <td>37.2</td>
  </tr>
  <tr>
    <td>TailorKV-2</td>
    <td>64(+128)</td>
    <td><b>32.5</b></td>
    <td>15.3</td>
    <td>1.5</td>
    <td>49.1</td>
    <td><b>56.4</b></td>
    <td>4.0</td>
    <td>28.2</td>
    <td>128(+896)</td>
    <td>98.5</td>
    <td><b>3.0</b></td>
    <td><b>25.3</b></td>
    <td><b>8.0</b></td>
    <td>26.7</td>
    <td><b>37.3</b></td>
  </tr>
</table>

Table 3: Task performance (%) on LongBench and InfiniteBench. 13 sub-tasks of LongBench are aggregated into 6 classes, and 9 sub-tasks of InfiniteBench are aggregate into 5 classes. The aggregation of sub-tasks is discussed in Table 10 and Table 11, while the detailed results for all sub-tasks can be found in Table 14 and Table 15.

![Three line plots showing accuracy vs sequence length for different methods on Llama-3.1-8B-Instruct, Yi-9B-200K, and Yi-6B-200K](page_1012_1012_484_312.png)

Figure 6: The average accuracy of different methods on RULER. The sparsity-friendly layer in TailorKV uses 128+(896) tokens, while other methods use 1024 tokens. See Table 16 for details.

5 Experiments

5.1 Experimental Setup

Baselines and Benchmarks. We evaluate three widely used models with their respective context lengths: Llama-3.1-8B-Instruct (Dubey et al., 2024), Yi-6B-200K (01-ai, 2024a), and Yi-9B-200K (01-ai, 2024b). To demonstrate the superior performance of our method, we compare TailorKV with competitive baselines, including StreamingLLM (Xiao et al., 2024b), SnapKV (Li et al., 2024), Quest (Tang et al., 2024), and PQCache (Zhang et al., 2024a). To evaluate the performance in long-context scenarios, we employ three well-designed benchmarks, including LongBench (Bai et al., 2024), InfiniteBench (Zhang et al., 2024c), and RULER (Hsieh et al., 2024). Refer to Appendix G for further details.

Implementation. We set \( \tau \) to 0.2 for all models. TailorKV-1 and TailorKV-2 represent KV cache stored with 1-bit and 2-bit precision in quantization-friendly layers, respectively. The group size is 64, with the zero point and scaler stored in 16-bit. For sparsity-friendly layers, the number of tokens involved in attention computation is \( n_{\text{local}} + (n_{\text{topk}}) \), where \( n_{\text{local}} \) refers to the GPU budget and \( n_{\text{topk}} \) represents the additional communication overhead. The number of critical channels is 8 for LongBench and 12 for both InfiniteBench and RULER. The symbol \( Q \) represents quantization-friendly layers. Llama-3.1-8B is configured with \( Q = \{0\} \), while
Figure 7: Peak memory usage on A100 (80GB).

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="4">Llama-2-7B</th>
    <th colspan="4">Llama-3.1-8B</th>
  </tr>
  <tr>
    <th>16k</th>
    <th>32k</th>
    <th>64k</th>
    <th>96k</th>
    <th>16k</th>
    <th>32k</th>
    <th>64k</th>
  </tr>
  <tr>
    <td colspan="8">NVIDIA RTX 3090 (24GB, PCIe 1.0 link)</td>
  </tr>
  <tr>
    <td>Full Cache</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>0.033</td>
    <td>0.042</td>
    <td>OOM</td>
  </tr>
  <tr>
    <td>OffloadCache</td>
    <td>0.893</td>
    <td>1.776</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>0.242</td>
    <td>0.460</td>
    <td>OOM</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>OOM</td>
    <td>0.126</td>
    <td>OOM</td>
    <td>OOM</td>
  </tr>
  <tr>
    <td>TailorKV</td>
    <td>0.067</td>
    <td>0.087</td>
    <td>0.135</td>
    <td>0.176</td>
    <td>0.062</td>
    <td>0.067</td>
    <td>0.074</td>
  </tr>
  <tr>
    <td colspan="8">NVIDIA A100 (80GB, PCIe 4.0 link)</td>
  </tr>
  <tr>
    <td>Full Cache</td>
    <td>0.045</td>
    <td>0.077</td>
    <td>0.140</td>
    <td>OOM</td>
    <td>0.024</td>
    <td>0.033</td>
    <td>0.050</td>
  </tr>
  <tr>
    <td>OffloadCache</td>
    <td>0.433</td>
    <td>0.838</td>
    <td>1.767</td>
    <td>3.253</td>
    <td>0.124</td>
    <td>0.227</td>
    <td>0.435</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>0.108</td>
    <td>0.111</td>
    <td>0.114</td>
    <td>0.115</td>
    <td>0.104</td>
    <td>0.105</td>
    <td>0.108</td>
  </tr>
  <tr>
    <td>TailorKV</td>
    <td>0.041</td>
    <td>0.062</td>
    <td>0.098</td>
    <td>0.132</td>
    <td>0.045</td>
    <td>0.047</td>
    <td>0.054</td>
  </tr>
</table>

Table 4: Decoding latency(s) on different devices. Additional results are provided in Table 13.

Llama-2-7B, Yi-6B, and Yi-9B are configured with \( Q = \{0, 1\} \). Additional details are in Appendix D.

Hardware. The experiments are conducted under two different settings: the first equipped with an NVIDIA RTX 3090 GPU (24GB) and Intel Xeon Gold 6240 CPU, interconnected via PCIe 1.0 ×16 (4GB/s); the second equipped with an NVIDIA A100 GPU (80GB) and Intel Xeon Platinum 8369B CPU, interconnected via PCIe 4.0 ×16 (32GB/s).

5.2 Accuracy on Long Context Tasks

LongBench. As shown in Table 3, SnapKV and StreamingLLM degrade in performance due to the loss of crucial information. Although Quest and PQCache improve performance, their individual strategies face limitations under restricted budgets. TailorKV outperforms the best method by 2.32%, 5.42%, and 3.66% on Llama-3.1-8B, Yi-9B, and Yi-6B, respectively, by preserving the 1-bit KV cache for quantization-friendly layers and selecting 192 tokens for sparsity-friendly layers. Appendix F provides a discussion on retrieval accuracy of our sparsity-friendly layers compared to other methods.

InfiniteBench. Table 3 presents evaluations on the challenging benchmark InfiniteBench. As the context length increases, the clustering overhead of PQCache on the CPU grows. We restrict K-Means to one iteration for real-time inference, which compromises accuracy and exposes PQCache’s limitations. Notably, our hybrid strategy outperforms individual strategies, with an average performance loss under 1.5% compared to the full cache, especially excelling in dialogue, novel, and math tasks.

RULER. Figure 6 summarizes the accuracy on RULER, with the sequence length ranging from 4K to 128K. TailorKV captures crucial information from redundant contexts, leading to superior performance on most tasks, such as Needle-in-a-haystack, Question Answering, and Variable Tracking (detailed results provided in Table 16).

5.3 Efficiency Results

We evaluate peak memory usage and decoding latency in comparison with the full cache, OffloadCache, and PQCache. Specifically, the full cache is implemented by FlashAttention-2 (Dao, 2024) and OffloadCache is a script3 from the official library that prefetches next layer’s KV cache from the CPU memory to the GPU.

Peak Memory Usage. As shown in Figure 7, our method achieves superior memory efficiency compared to alternative methods, enabling deployment on lower-end GPUs such as the RTX 3090. Specifically, compared to full cache, TailorKV reduces GPU memory usage by approximately **73.8%** for Llama-2-7B with a sequence length of 128k.

End-to-End Latency. As shown in Table 4, the increasing sequence length causes out-of-memory errors in the full cache, PQCache, and OffloadCache on the RTX 3090. For 64k context on the A100, TailorKV achieves significant latency reductions compared to OffloadCache and PQCache: **18.0×** and **1.2×** faster than the MHA model, and

![Peak memory usage on A100 (80GB)](page_120_120_600_300.png)

Figure 7: Peak memory usage on A100 (80GB).

![Latency breakdown (ms) under different methods. Q and S denote the quantization-friendly layer and sparsity-friendly layer, respectively.](page_800_120_600_300.png)

Figure 8: Latency breakdown (ms) under different methods. Q and S denote the quantization-friendly layer and sparsity-friendly layer, respectively.

3https://github.com/huggingface/transformers
8.1× and 2.0× faster than the GQA model. TailorKV’s latency is comparable to that of full attention, as a result of multi-threading used to execute asynchronous tasks, which enables the overlap of computation and CPU-GPU communication.

Latency Breakdown. As depicted in Figure 8, we evaluate the breakdown of latency for a single Transformer block with a sequence length of 16k on the A100 GPU. Compared to PQCache, TailorKV reduces retrieval latency by 27.8% for the GQA model and 40.5% for the MHA model, and data transfer latency by 83.5% and 82.2% for the same models. This reduction is primarily attributable to our use of DGL (Wang et al., 2019) to directly transfer rows from a CPU tensor to the GPU device, whereas PQCache first gathers rows on the CPU and then transfers them to the GPU.

5.4 Ablation Study

We conduct ablation studies on the LongBench benchmark using the Llama-3.1-8B-Instruct model.

Effect of Tailored Strategies. As depicted in Figure 9 (a), 1-bit quantization is applied to certain layers, while only 64(+128) tokens are computed for the remaining layers, with the quantization-friendly layer defined as \( Q = \{0\} \). The results indicate that quantizing only the 0th layer yields the best performance, while quantizing sparsity-friendly layers degrades performance, highlighting the need for tailored compression strategies.

Effect of Dynamic Channels. Prior study (Yang et al., 2024b) employed offline calibration to statically select high-magnitude channels. However, we find that outliers may appear at any position, not fixed to specific channels (Section 3). Figure 9 (b) compares the performance of dynamic and static channels. In general, our dynamic retrieval method demonstrates better performance.

Effect of the Number of Critical Channels. In Figure 9 (c), we maintain the 64(+128) configuration and adjust the number of critical channels. Reducing the number of critical channels decreases retrieval latency. However, performance significantly degrades when the number is set to 2 or 4. Overall, selecting 8 critical channels achieves a favorable balance between performance and latency.

6 Related Work

Existing KV cache compression methods include eviction, selection, and quantization, with detailed comparisons in Appendix A. Eviction methods reduce KV cache size by evicting most tokens during inference. StreamingLLM (Xiao et al., 2024b) identifies ‘Attention Sinks’ by retaining the initial and the most recent tokens. H2O (Zhang et al., 2023), SnapKV (Li et al., 2024), and Scissorhands (Liu et al., 2023) estimate token importance based on historical attention scores. However, evicting dominant tokens may degrade accuracy in tasks like ‘needle-in-the-haystack’ and multi-turn dialogues.

Selection methods are more commonly used in sparse attention scenarios. Quest (Tang et al., 2024) retains the KV cache and utilizes paged keys for retrieving tokens, but it fails to reduce memory usage and suffers from lower recall. Instead, KV cache offloading methods like PQCache (Zhang et al., 2024a) and InfiniGen (Lee et al., 2024) approximate attention scores for identifying and loading critical tokens from CPU to GPU, though they face challenges in balancing computation and communication due to large KV cache loads. Some methods (Chen et al., 2025; Liu et al., 2024c) use LSH and KNN to retrieve critical tokens, which are processed on the CPU and subsequently merged with GPU outputs; however, imbalanced computation times may result in GPU idle time.

Quantization is a common compression technique that converts high-precision floats into low-precision integers. Existing methods employ vari-
ous solutions to minimize quantization error. For example, KVQuant (Hooper et al., 2024) isolates outliers for mixed precision, GEAR (Kang et al., 2024) utilizes SVD to recover residuals, and KIVI (Liu et al., 2024d) quantizes keys per channel and values per token. FlexGen (Sheng et al., 2023) reduces I/O transfer latency by quantizing the KV cache to 4-bits. However, none of these methods reduce the KV cache to 1-bit. By contrast, we focus on exploring layer characteristics and selecting the most suitable compression strategy.

7 Conclusion

In this paper, we propose TailorKV, an effective framework for KV cache management in LLMs. We begin by observing that different layers exhibit distinct compression preferences and categorize them into quantization-friendly and sparsity-friendly, each employing a tailored strategy. Specifically, quantization-friendly layers aggressively quantize the KV cache to 1-bit. Sparsity-friendly layers, on the other hand, dynamically retrieve dominant tokens based on large magnitudes in the query and key channels, integrating CPU-GPU co-design. Experiments across long-context benchmarks show that TailorKV effectively minimizes the usage of the KV cache while maintaining model performance, with an acceptable latency cost. Our hybrid framework demonstrates the potential of deploying LLMs on resource-limited GPUs, extending the application of LLMs to more devices while maintaining efficiency.

Limitations

Although TailorKV has demonstrated superior memory optimization and latency reduction in long-context scenarios, it still exhibits some limitations, which are summarized as follows: (1) TailorKV primarily focuses on improving the efficiency of the decode phase by asynchronously transferring tokens from the CPU memory to the GPU. However, it is challenging to completely overlap the offloading latency during the prefill phase. Moreover, the efficiency of the prefill phase in long-context scenarios is also important. It is noteworthy that our work is compatible with and complementary to approaches for prefilling acceleration (Jiang et al., 2024). (2) We have designed tailored strategies for different layers to facilitate deployment, and we are confident that TailorKV can be adapted on a head-wise basis. These issues hold significant importance, and we intend to further explore them in our future research.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 62472419, 62472420). We would like to thank the anonymous reviewers for their insightful comments.

References

01-ai. 2024a. Yi-6b-200k. https://huggingface.co/01-ai/Yi-6B-200K. Accessed: 2024-07-01.

01-ai. 2024b. Yi-9b-200k. https://huggingface.co/01-ai/Yi-9B-200K. Accessed: 2024-07-01.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. ArXiv preprint, abs/2303.08774.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proc. of ACL, pages 3119–3137. Association for Computational Linguistics.

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. ArXiv preprint, abs/2406.02069.

Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. 2025. MagicPIG: LSH sampling for efficient LLM generation. In The Thirteenth International Conference on Learning Representations.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

Tri Dao. 2024. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. ArXiv preprint, abs/2407.21783.
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. 2024. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. ArXiv preprint, abs/2407.11550.

Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. Zipcache: Accurate and efficient KV cache quantization with salient token identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270–1303.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shan-tau Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling.

Huiqiang Jiang, Yucheng Li, Chenguoding Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qi. 2024. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. 2024. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. ArXiv preprint, abs/2403.05527.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Liannin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.

Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. Infinigen: Efficient generative inference of large language models with dynamic kv cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155–172.

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Aycr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haf-fari, and Bohan Zhuang. 2024b. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems, 37:139997–140031.

Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chen-grudong Zhang, Bailiu Ding, Kai Zhang, et al. 2024c. Retrievalattention: Accelerating long-context llm inference via vector retrieval. ArXiv preprint, abs/2409.10516.

Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhou Xu, Anastasios Kyri-lidis, and Anshumali Shrivastava. 2023. Scis-sorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Proc. of NeurIPS.

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhou Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024d. Kivi: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.

Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155–170, Santa Clara, CA. USENIX Association.

Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. 2024. Sparq attention: Bandwidth-efficient LLM inference. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models.

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single GPU. In Proc. of ICML, volume 202 of Proceedings of Machine Learning Research, pages 31094–31116. PMLR.

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: query-aware sparsity for efficient long-context llm inference. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro,
Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. 2025. $text{D}_{-2}text{O}$: Dynamic discriminative operations for efficient long-context inference of large language models. In The Thirteenth International Conference on Learning Representations.

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. 2019. Deep graph library: A graph-centric, highly-performant package for graph neural networks. ArXiv preprint, abs/1909.01315.

Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024a. InFLLM: Training-free long-context extrapolation for LLMs with an efficient context memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024b. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations.

June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024a. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. ArXiv preprint, abs/2402.18096.

Shuo Yang, Ying Sheng, Joseph E Gonzalez, Ion Stoica, and Lianmin Zheng. 2024b. Post-training sparse attention with double sparsity. ArXiv preprint, abs/2408.07092.

Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2024a. Pqcache: Product quantization-based kv-cache for long context llm inference. ArXiv preprint, abs/2407.12820.

Libo Zhang, Zhaoning Zhang, Baizhou Xu, Songzhu Mei, and Dongsheng Li. 2024b. Dovetail: A cpu/gpu heterogeneous speculative decoding for llm inference. arXiv preprint arXiv:2412.18934.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024c. xBench: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15262–15277, Bangkok, Thailand. Association for Computational Linguistics.

Xuan Zhang, Cuxiao Du, Chao Du, Tianyu Pang, Wei Gao, and Min Lin. 2024d. Simlayerkv: A simple framework for layer-level kv cache reduction. ArXiv preprint, abs/2410.13846.

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Proc. of NeurIPS.
A  Comparison with Other Approaches

Figure 10 compares TailorKV with other methods: (a) Full cache retains the entire KV cache. (b) The eviction methods permanently evict specific tokens from each layer, leading to irreversible information loss since evicted tokens may be important later. (c) The selection methods offload the entire KV cache to the CPU, enabling tokens recall but incurring significant communication overhead because of the large volume of tokens involved. (d) Our method employs layer-specific compression strategies, facilitating more aggressive compression.

![Comparison of TailorKV with other methods in managing KV cache budget across layers.](page_120_370_600_246.png)

Figure 10: Comparison of TailorKV with other methods in managing KV cache budget across layers.

B  Inter-Layer Similarity

Let \( \mathbf{h}^{(l)} \) denote the hidden state at the \( l \)-th layer. To quantify the similarity between the hidden states of two adjacent layers, we employ cosine similarity, which is formally defined as:

\[
\operatorname{sim}(\mathbf{h}^{(l-1)}, \mathbf{h}^{(l)}) = \frac{\mathbf{h}^{(l-1)} \cdot \mathbf{h}^{(l)}}{\|\mathbf{h}^{(l-1)}\| \|\mathbf{h}^{(l)}\|}.
\] (11)

We define the query weight at the \( l \)-th layer as \( W_q^{(l)} \), and the query vector at the \( l \)-th layer is computed as:
\[
\mathbf{q}^{(l)} = \mathbf{W}_q^{(l)} (\mathbf{h}^{(l)}).
\] (12)

As shown in Figure 11, \( \mathbf{h}^{(l)} \) and \( \mathbf{h}^{(l-1)} \) closely resemble each other, allowing us to approximate the query at the \( l \)-th layer based on the hidden state from the \( l - 1 \)-th layer:
\[
\hat{\mathbf{q}}^{(l)} = \mathbf{W}_q^{(l)} (\mathbf{h}^{(l-1)}).
\] (13)

Existing research (Liu et al., 2024b) has elucidated that the KV cache exhibits similarity across adjacent layers. However, as illustrated in Figure 11, the similarity between \( \hat{\mathbf{q}}^{(l)} \) and \( \mathbf{q}^{(l)} \) exceeds that between \( \mathbf{q}^{(l-1)} \) and \( \mathbf{q}^{(l)} \), suggesting that using hidden states from the preceding layer enhances prediction accuracy.

![Cosine similarity between adjacent layers.](page_820_563_600_246.png)

Figure 11: Cosine similarity between adjacent layers.

C  Offline Identification on Different Datasets

As shown in Figure 12, the curves represent different datasets. The distribution of \( \mathcal{P} \) is consistent across various datasets for the same model, indicating that the metric \( \mathcal{P} \) effectively captures the characteristics of different layers, enabling appropriate compression strategy.

D  Baselines Settings

In Table 5 and Table 6, we present the configuration for the long-context methods employed in our experiments.

E  Comparison with Hybrid Method

To validate the effectiveness of our quantization-sparsity hybrid framework, we compare it to SimLayerKV (Zhang et al., 2024d), a similar hybrid method. SimLayerKV assumes that some layers in LLMs exhibit "lazy" behavior, retaining only the initial and most recent tokens, while "non-lazy" layers require full precision to retain all tokens. Table 7 presents the experimental results on LongBench. The results show that at an average compression rate of 34.2\( \times \), the performance of our method is comparable to that of SimLayerKV at a 1.53\( \times \) compression rate. Our approach achieves
Figure 12: Dense preference score \( \mathcal{P} \) for layers across different offline datasets.

<table>
  <tr>
    <th>Methods</th>
    <th>Configurations</th>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>num local: 128, num initial: 64</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>window size: 64, max capacity prompt: 128, kernel size: 7, pooling: max pooling</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>page size: 16, token budget: 196</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>partitions in PQ: 2, bits for PQ codes: 6, K-Means iterations: adaptive, \( n_{local} \): 64, \( n_{topk} \): 128</td>
  </tr>
  <tr>
    <td>TailorKV-1</td>
    <td>\( \tau \): 0.2, bit size: 1, group size: 64, \( n_{local} \): 64, \( n_{topk} \): 128, num critical channels: 8</td>
  </tr>
  <tr>
    <td>TailorKV-2</td>
    <td>\( \tau \): 0.2, bit size: 2, group size: 64, \( n_{local} \): 64, \( n_{topk} \): 128, num critical channels: 8</td>
  </tr>
</table>

Table 5: Configurations of long-context methods on LongBench.

<table>
  <tr>
    <th>Methods</th>
    <th>Configurations</th>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>num local: 896, num initial: 128</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>window size: 128, max capacity prompt: 896, kernel size: 7, pooling: max pooling</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>page size: 16, token budget: 1024</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>partitions in PQ: 2, bits for PQ codes: 6, K-Means iterations: 1 (exceeding 64k), \( n_{local} \): 128, \( n_{topk} \): 896</td>
  </tr>
  <tr>
    <td>TailorKV-1</td>
    <td>\( \tau \): 0.2, bit size: 1, group size: 64, \( n_{local} \): 128, \( n_{topk} \): 896, num critical channels: 12</td>
  </tr>
  <tr>
    <td>TailorKV-2</td>
    <td>\( \tau \): 0.2, bit size: 2, group size: 64, \( n_{local} \): 128, \( n_{topk} \): 896, num critical channels: 12</td>
  </tr>
</table>

Table 6: Configurations of long-context methods on InfiniteBench and RULER.

F Effectiveness of Dynamic Retrieval

Table 8 presents a comparison of retrieval accuracy between our sparsity-friendly layers and alternative methods, using the Llama-3.1-8B-Instruct model on the LongBench benchmark. Specifically, we retain full precision for the KV cache in the 0th layer of StreamLLM, SnapKV, and Quest, thereby preserving the global information in the 0th layer. TailorKV-1 and TailorKV-2 represent the quantization of the 0th layer’s KV cache to 1-bit and 2-bit precision, respectively.

The experimental results demonstrate that our retrieval method outperforms other sparse methods when the global information is preserved in the 0th layer. Specifically, TailorKV applies quantization to the 0th layer, whereas other methods use full precision (16-bit), and the attention calculation utilizes the same tokens from layer 1 to layer 31. This notable performance advantage highlights that our retrieval method effectively identifies the most important tokens, thereby minimizing the loss of crucial information.

G More Information on Models and Benchmarks

G.1 Baselines

In all of our experiments, we use pre-trained model weights obtained from Huggingface. These models are based on two representative attention structures:
(1) MHA: including Llama-2-7B-32K-Instruct\(^4\).
(2) GQA: including Llama-3.1-8B-Instruct\(^5\), Yi-

\(^4\)https://huggingface.co/togethercomputer/Llama-2-7B-32K-Instruct
\(^5\)https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
<table>
  <tr>
    <th>Methods</th>
    <th>Ratio</th>
    <th>Qspr</th>
    <th>MulFi</th>
    <th>HQA</th>
    <th>WMQA</th>
    <th>GRpt</th>
    <th>MuIN</th>
    <th>TREC</th>
    <th>SMSM</th>
    <th>TriQA</th>
    <th>Repo</th>
    <th>LCC</th>
    <th>PsgC</th>
    <th>PsgR</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td><i>Llama-3.1-8B</i></td>
    <td>1×</td>
    <td>45.5</td>
    <td>53.8</td>
    <td>54.7</td>
    <td>47.1</td>
    <td>34.9</td>
    <td>27.5</td>
    <td>73.0</td>
    <td>43.8</td>
    <td>91.6</td>
    <td>56.7</td>
    <td>63.4</td>
    <td>7.5</td>
    <td>99.5</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td>SimLayerKV</td>
    <td>1.53×</td>
    <td><b>45.6</b></td>
    <td>52.3</td>
    <td>54.5</td>
    <td>44.5</td>
    <td><b>32.2</b></td>
    <td>26.9</td>
    <td><b>71.5</b></td>
    <td><b>43.8</b></td>
    <td>91.3</td>
    <td><b>54.9</b></td>
    <td><b>62.8</b></td>
    <td><b>7.9</b></td>
    <td>95.5</td>
    <td>52.6</td>
  </tr>
  <tr>
    <td>TailorKV-1</td>
    <td>34.2×</td>
    <td>43.4</td>
    <td>53.0</td>
    <td><b>55.3</b></td>
    <td><b>46.5</b></td>
    <td>31.3</td>
    <td><b>27.2</b></td>
    <td>70.0</td>
    <td>42.5</td>
    <td>91.6</td>
    <td>54.8</td>
    <td>61.8</td>
    <td><b>7.9</b></td>
    <td><b>99.0</b></td>
    <td>52.6</td>
  </tr>
  <tr>
    <td>TailorKV-2</td>
    <td>32.7×</td>
    <td>44.8</td>
    <td><b>53.9</b></td>
    <td>54.8</td>
    <td>46.2</td>
    <td>31.9</td>
    <td>26.8</td>
    <td>70.5</td>
    <td>43.2</td>
    <td><b>92.3</b></td>
    <td>54.2</td>
    <td>62.1</td>
    <td>7.7</td>
    <td><b>99.0</b></td>
    <td><b>52.9</b></td>
  </tr>
</table>

Table 7: Performance comparison between TailorKV and SimLayerKV. TailorKV computes only 64 (+128) tokens for sparsity-friendly layers. SimLayerKV retains the most recent 1024 tokens for the "lazy" layers, while the "non-lazy" layers preserve full precision. Additionally, the threshold for SimLayerKV on Llama-3.1-8B is 0.9, with more than half of the layers being "non-lazy."

<table>
  <tr>
    <th>Methods</th>
    <th>Tokens</th>
    <th>Qspr</th>
    <th>MulFi</th>
    <th>HQA</th>
    <th>WMQA</th>
    <th>GRpt</th>
    <th>MuIN</th>
    <th>TREC</th>
    <th>SMSM</th>
    <th>TriQA</th>
    <th>Repo</th>
    <th>LCC</th>
    <th>PsgC</th>
    <th>PsgR</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td><i>Llama-3.1-8B</i></td>
    <td>128k</td>
    <td>45.5</td>
    <td>53.8</td>
    <td>54.7</td>
    <td>47.1</td>
    <td>34.9</td>
    <td>27.5</td>
    <td>73.0</td>
    <td>43.8</td>
    <td>91.6</td>
    <td>56.7</td>
    <td>63.4</td>
    <td>7.5</td>
    <td>99.5</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td>StreamLLM<sup>†</sup></td>
    <td>192</td>
    <td>21.4</td>
    <td>31.3</td>
    <td>46.5</td>
    <td>38.9</td>
    <td>17.9</td>
    <td>18.0</td>
    <td>40.0</td>
    <td>34.4</td>
    <td>75.7</td>
    <td>45.7</td>
    <td>50.7</td>
    <td>8.0</td>
    <td>99.0</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td>StreamLLM<sup>‡</sup></td>
    <td>192</td>
    <td>21.6</td>
    <td>30.8</td>
    <td>45.5</td>
    <td>39.0</td>
    <td>18.4</td>
    <td>17.9</td>
    <td>40.5</td>
    <td>34.1</td>
    <td>75.6</td>
    <td>45.5</td>
    <td>52.8</td>
    <td>8.0</td>
    <td>99.0</td>
    <td>40.7</td>
  </tr>
  <tr>
    <td>SnapKV<sup>†</sup></td>
    <td>192</td>
    <td>25.7</td>
    <td>44.7</td>
    <td>52.6</td>
    <td>43.7</td>
    <td>20.0</td>
    <td>20.5</td>
    <td>41.0</td>
    <td>39.6</td>
    <td>89.0</td>
    <td>48.7</td>
    <td>57.0</td>
    <td>8.0</td>
    <td>97.0</td>
    <td>45.2</td>
  </tr>
  <tr>
    <td>SnapKV<sup>‡</sup></td>
    <td>192</td>
    <td>32.4</td>
    <td>47.0</td>
    <td>54.6</td>
    <td>44.0</td>
    <td>21.9</td>
    <td>22.8</td>
    <td>48.0</td>
    <td>40.0</td>
    <td>90.3</td>
    <td>51.9</td>
    <td>59.9</td>
    <td>8.0</td>
    <td>98.0</td>
    <td>47.6</td>
  </tr>
  <tr>
    <td>Quest<sup>†</sup></td>
    <td>192</td>
    <td>35.9</td>
    <td>44.2</td>
    <td>52.8</td>
    <td>41.0</td>
    <td>17.7</td>
    <td>23.8</td>
    <td>63.0</td>
    <td>36.0</td>
    <td>86.0</td>
    <td>43.6</td>
    <td>52.3</td>
    <td>8.4</td>
    <td>96.5</td>
    <td>46.2</td>
  </tr>
  <tr>
    <td>Quest<sup>‡</sup></td>
    <td>192</td>
    <td>39.1</td>
    <td>45.1</td>
    <td>52.4</td>
    <td>43.4</td>
    <td>21.1</td>
    <td>25.6</td>
    <td>65.5</td>
    <td>38.7</td>
    <td>88.1</td>
    <td>44.8</td>
    <td>52.0</td>
    <td>8.1</td>
    <td>97.0</td>
    <td>47.8</td>
  </tr>
  <tr>
    <td>TailorKV-1</td>
    <td>64(+128)</td>
    <td>43.4</td>
    <td>53.0</td>
    <td><b>55.3</b></td>
    <td><b>46.5</b></td>
    <td>31.3</td>
    <td><b>27.2</b></td>
    <td>70.0</td>
    <td>42.5</td>
    <td>91.6</td>
    <td><b>54.8</b></td>
    <td>61.8</td>
    <td>7.9</td>
    <td><b>99.0</b></td>
    <td>52.6</td>
  </tr>
  <tr>
    <td>TailorKV-2</td>
    <td>64(+128)</td>
    <td><b>44.8</b></td>
    <td><b>53.9</b></td>
    <td>54.8</td>
    <td>46.2</td>
    <td><b>31.9</b></td>
    <td>26.8</td>
    <td><b>70.5</b></td>
    <td><b>43.2</b></td>
    <td><b>92.3</b></td>
    <td>54.2</td>
    <td>62.1</td>
    <td>7.7</td>
    <td><b>99.0</b></td>
    <td><b>52.9</b></td>
  </tr>
</table>

Table 8: Effectiveness of dynamic retrieval. Methods marked with † indicate that the 0th layer of the model retains the full-precision (16-bit) KV cache, while methods marked with ‡ indicate that all layers use the same compression strategy. TailorKV-1 and TailorKV-2 store the KV cache as 1-bit and 2-bit, respectively, in the 0th layer.

6B-200K<sup>6</sup>, and Yi-9B-200K<sup>7</sup>. Detailed information about the four models can be found in Table 9.

To showcase the state-of-the-art performance of our method, we compare TailorKV with the following baselines: (1) StreamingLLM (Xiao et al., 2024b): An eviction strategy that retains only the initial and most recent tokens. (2) SnapKV (Li et al., 2024): An eviction strategy that chooses clustered important KV positions. (3) Quest (Tang et al., 2024): A selection strategy that determines page criticality through paged key. (4) PQ-Cache (Zhang et al., 2024a): A selection strategy that retrieves Top-K tokens through vector quantization.

G.2 Benchmarks

LongBench. A benchmark is conducted across six categories: summarization, code completion, synthetic tasks, few-shot learning, and single/multi-document question answering. Table 10 presents detailed information on the 13 datasets in Long-Bench.

InfiniteBench. A benchmark designed to assess the ability of language models to process, understand, and reason with extremely long contexts (200k+ tokens). We test the Llama3 and Yi models with context lengths of 128K and 200K, truncating inputs beyond these limits. Table 11 provides details of the 9 datasets in InfiniteBench.

RULER. A benchmark intended to assess the long-context modeling capabilities of language models, covering question answering, retrieval, aggregation, and multi-hop tracing. This benchmark consists of 13 representative tasks, with sequence lengths ranging from 4K to 128K. For each task, we employed 25 samples. Detailed information is provided in Table 12.

H Detailed Results

H.1 Accuracy on Long Context Tasks

Table 14 and Table 15 present experimental results for LongBench and InfiniteBench. Table 16 shows accuracy results for sequence lengths of 64k and 128k on RULER.

H.2 Efficiency Results

In Table 13, we present the end-to-end latency for Llama-3.1-8B-Instruct, Llama-2-7B-32K-Instruct, Yi-6B-200K, and Yi-9B-200K. The results indicate that our method achieves efficiency closest to that of the original model.
<table>
  <tr>
    <th>Name</th>
    <th>Claimed Length</th>
    <th>Query Heads</th>
    <th>KV Heads</th>
    <th>Num Layers</th>
    <th>Q</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>128k</td>
    <td>32</td>
    <td>8</td>
    <td>32</td>
    <td>{0}</td>
  </tr>
  <tr>
    <td>Llama-2-7B-32K-Instruct</td>
    <td>32k</td>
    <td>32</td>
    <td>32</td>
    <td>32</td>
    <td>{0, 1}</td>
  </tr>
  <tr>
    <td>Yi-6B-200K</td>
    <td>200k</td>
    <td>32</td>
    <td>4</td>
    <td>32</td>
    <td>{0, 1}</td>
  </tr>
  <tr>
    <td>Yi-9B-200K</td>
    <td>200k</td>
    <td>32</td>
    <td>4</td>
    <td>48</td>
    <td>{0, 1}</td>
  </tr>
</table>

Table 9: Details of models. \( \mathbb{Q} \) denotes the quantization-friendly layer.

<table>
  <tr>
    <th>Label</th>
    <th>Task</th>
    <th>Capability</th>
    <th>Metric</th>
    <th>Avg Len</th>
    <th>#data</th>
  </tr>
  <tr>
    <td>Qspr</td>
    <td>Qasper</td>
    <td>Single-Doc. QA (SD.QA)</td>
    <td>F1</td>
    <td>3,619</td>
    <td>200</td>
  </tr>
  <tr>
    <td>MulFi</td>
    <td>MultiFieldQA-en</td>
    <td>Single-Doc. QA (SD.QA)</td>
    <td>F1</td>
    <td>4,559</td>
    <td>150</td>
  </tr>
  <tr>
    <td>HQA</td>
    <td>HotpotQA</td>
    <td>Multi-Doc. QA (MD.QA)</td>
    <td>F1</td>
    <td>9,151</td>
    <td>200</td>
  </tr>
  <tr>
    <td>WMQA</td>
    <td>2WikiMultiHopQA</td>
    <td>Multi-Doc. QA (MD.QA)</td>
    <td>F1</td>
    <td>4,887</td>
    <td>200</td>
  </tr>
  <tr>
    <td>GRpt</td>
    <td>GovReport</td>
    <td>Summarization (Summ)</td>
    <td>Rouge-L</td>
    <td>8,734</td>
    <td>200</td>
  </tr>
  <tr>
    <td>MulN</td>
    <td>MultiNews</td>
    <td>Summarization (Summ)</td>
    <td>Rouge-L</td>
    <td>2,113</td>
    <td>200</td>
  </tr>
  <tr>
    <td>TREC</td>
    <td>TREC</td>
    <td>Few-shot Learning (FS.L)</td>
    <td>Accuracy (CLS)</td>
    <td>5,177</td>
    <td>200</td>
  </tr>
  <tr>
    <td>SMSM</td>
    <td>SAMSum</td>
    <td>Few-shot Learning (FS.L)</td>
    <td>Rouge-L</td>
    <td>6,258</td>
    <td>200</td>
  </tr>
  <tr>
    <td>TriQA</td>
    <td>TriviaQA</td>
    <td>Few-shot Learning (FS.L)</td>
    <td>F1</td>
    <td>8,209</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Lcc</td>
    <td>LCC</td>
    <td>Code Completion (Code)</td>
    <td>Edit Sim</td>
    <td>1,235</td>
    <td>500</td>
  </tr>
  <tr>
    <td>Repo</td>
    <td>RepoBench-P</td>
    <td>Code Completion (Code)</td>
    <td>Edit Sim</td>
    <td>4,206</td>
    <td>500</td>
  </tr>
  <tr>
    <td>PsgC</td>
    <td>PassageCount</td>
    <td>Synthetic (Synth)</td>
    <td>Accuracy (EM)</td>
    <td>11,141</td>
    <td>200</td>
  </tr>
  <tr>
    <td>PsgR</td>
    <td>PassageRetrieval-en</td>
    <td>Synthetic (Synth)</td>
    <td>Accuracy (EM)</td>
    <td>9,289</td>
    <td>200</td>
  </tr>
</table>

Table 10: Details of LongBench.

<table>
  <tr>
    <th>Label</th>
    <th>Task</th>
    <th>Context</th>
    <th>Capability</th>
    <th>Metric</th>
    <th>Avg len</th>
    <th>#Examples</th>
  </tr>
  <tr>
    <td>R.PK</td>
    <td>Retrieve.PassKey</td>
    <td>Fake Book</td>
    <td>Retrieve (Retr)</td>
    <td>Accuracy</td>
    <td>122.4k</td>
    <td>590</td>
  </tr>
  <tr>
    <td>R.Num</td>
    <td>Retrieve.Number</td>
    <td>Synthetic</td>
    <td>Retrieve (Retr)</td>
    <td>Accuracy</td>
    <td>122.4k</td>
    <td>590</td>
  </tr>
  <tr>
    <td>En.Dia</td>
    <td>En.Dia</td>
    <td>Script</td>
    <td>Dialogue (Dia)</td>
    <td>Accuracy</td>
    <td>103.6k</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Sum</td>
    <td>En.Sum</td>
    <td>Fake Book</td>
    <td>Novel</td>
    <td>Rouge-L-Sum</td>
    <td>171.5k</td>
    <td>103</td>
  </tr>
  <tr>
    <td>En.MC</td>
    <td>En.MC</td>
    <td>Fake Book</td>
    <td>Novel</td>
    <td>Accuracy</td>
    <td>184.4k</td>
    <td>229</td>
  </tr>
  <tr>
    <td>En.QA</td>
    <td>En.QA</td>
    <td>Fake Book</td>
    <td>Novel</td>
    <td>QA F1 Score</td>
    <td>192.6k</td>
    <td>351</td>
  </tr>
  <tr>
    <td>Zh.QA</td>
    <td>Zh.QA</td>
    <td>New Book</td>
    <td>Novel</td>
    <td>QA F1 Score</td>
    <td>2068.6k</td>
    <td>175</td>
  </tr>
  <tr>
    <td>Math.F</td>
    <td>Math.Find</td>
    <td>Synthetic</td>
    <td>Math</td>
    <td>Accuracy</td>
    <td>87.9k</td>
    <td>350</td>
  </tr>
  <tr>
    <td>Code.D</td>
    <td>Code.Debug</td>
    <td>Code Document</td>
    <td>Code</td>
    <td>Accuracy</td>
    <td>114.7k</td>
    <td>394</td>
  </tr>
</table>

Table 11: Details of InfiniteBench.

<table>
  <tr>
    <th>Label</th>
    <th>Task</th>
    <th>Category</th>
  </tr>
  <tr>
    <td>N-S1</td>
    <td>Single NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>N-S2</td>
    <td>Single NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>N-S3</td>
    <td>Single NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>N-MK1</td>
    <td>Multi-keys NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>N-MK2</td>
    <td>Multi-keys NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>N-MK3</td>
    <td>Multi-keys NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>N-MV</td>
    <td>Multi-values NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>N-MQ</td>
    <td>Multi-queries NIAH</td>
    <td>Retrieval</td>
  </tr>
  <tr>
    <td>VT</td>
    <td>Variable Tracking</td>
    <td>Multi-hop Tracing</td>
  </tr>
  <tr>
    <td>CWE</td>
    <td>Common Words</td>
    <td>Aggregation</td>
  </tr>
  <tr>
    <td>FWE</td>
    <td>Frequent Words Extraction</td>
    <td>Aggregation</td>
  </tr>
  <tr>
    <td>QA-1</td>
    <td>Question Answering</td>
    <td>Question Answering</td>
  </tr>
  <tr>
    <td>QA-2</td>
    <td>Question Answering</td>
    <td>Question Answering</td>
  </tr>
</table>

Table 12: Details of RULER.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">16k</th>
    <th colspan="5">32k</th>
    <th colspan="5">64k</th>
    <th colspan="5">96k</th>
    <th colspan="5">128k</th>
  </tr>
  <tr>
    <th colspan="5">Llama-3.1-8B-Instruct</th>
    <th colspan="5">Llama-2-7B-32K-Instruct</th>
    <th colspan="5">Yi-6B-200K</th>
    <th colspan="5">Yi-9B-200K</th>
  </tr>
  <tr>
    <td>Full Cache</td>
    <td>0.024</td><td>0.033</td><td>0.050</td><td>0.062</td><td>0.082</td>
    <td>0.045</td><td>0.077</td><td>0.140</td><td>OOM</td><td>OOM</td>
    <td>0.019</td><td>0.021</td><td>0.029</td><td>0.036</td><td>0.044</td>
    <td>0.029</td><td>0.032</td><td>0.043</td><td>0.055</td><td>0.070</td>
  </tr>
  <tr>
    <td>OffloadCach</td>
    <td>0.124</td><td>0.227</td><td>0.435</td><td>0.743</td><td>0.992</td>
    <td>0.433</td><td>0.838</td><td>1.767</td><td>3.253</td><td>4.468</td>
    <td>0.066</td><td>0.118</td><td>0.221</td><td>0.325</td><td>0.430</td>
    <td>0.102</td><td>0.205</td><td>0.417</td><td>0.626</td><td>0.843</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>0.104</td><td>0.105</td><td>0.108</td><td>0.108</td><td>0.110</td>
    <td>0.108</td><td>0.111</td><td>0.112</td><td>0.115</td><td>0.120</td>
    <td>0.085</td><td>0.087</td><td>0.090</td><td>0.092</td><td>0.094</td>
    <td>0.130</td><td>0.138</td><td>0.139</td><td>0.144</td><td>0.150</td>
  </tr>
  <tr>
    <td>TailorKV</td>
    <td>0.045</td><td>0.047</td><td>0.054</td><td>0.054</td><td>0.056</td>
    <td>0.041</td><td>0.062</td><td>0.098</td><td>0.132</td><td>0.170</td>
    <td>0.041</td><td>0.042</td><td>0.046</td><td>0.049</td><td>0.056</td>
    <td>0.066</td><td>0.067</td><td>0.072</td><td>0.076</td><td>0.079</td>
  </tr>
</table>

Table 13: Decoding latency(s) on A100 (80G).

I  Attention Visualization Across Models

As shown in Figure 13, the attention patterns of different models closely match the results predicted by our usage metric \( \mathcal{P} \). Specifically, the quantization-friendly layers of Llama-2-7B-32K-Instruct and Yi-6B-200K are identified as the 0th and 1st layers. In these layers, attention patterns are dense, while the other layers are sparse. Similarly, the quantization-friendly layer of Llama-3.1-8B-Instruct is the 0th layer, where attention pattern is dense, with sparse features in the remaining layers.

J  Observations on QKV

Figure 14 illustrates the distribution patterns of queries, keys, and values across different attention heads in Llama-3.1-8B-Instruct. Although outliers appear in both the keys and queries, the locations of the outlier channels are not consistently fixed.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Tokens</th>
    <th colspan="3">SD.QA</th>
    <th colspan="3">MD.QA</th>
    <th colspan="3">Summ</th>
    <th colspan="3">FS.L</th>
    <th colspan="3">Code</th>
    <th colspan="3">Synth</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Qspr</th><th>MulFi</th><th>HQA</th><th>WMQA</th><th>GRpt</th><th>MuLN</th><th>TREC</th><th>SMSM</th><th>TriQA</th><th>Repo</th><th>LCC</th><th>PsgC</th><th>PsgR</th>
  </tr>
  <tr>
    <td><i>Llama-3.1-8B</i></td><td>128k</td><td>45.5</td><td>53.8</td><td>54.7</td><td>47.1</td><td>34.9</td><td>27.5</td><td>73.0</td><td>43.8</td><td>91.6</td><td>56.7</td><td>63.4</td><td>7.5</td><td>99.5</td><td>53.8</td>
  </tr>
  <tr>
    <td>StreamLLM</td><td>192</td><td>21.4</td><td>31.3</td><td>46.5</td><td>38.9</td><td>17.9</td><td>18.0</td><td>40.0</td><td>34.4</td><td>75.7</td><td>45.7</td><td>50.7</td><td>8.0</td><td>99.0</td><td>40.6</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>192</td><td>25.7</td><td>44.7</td><td>52.6</td><td>43.7</td><td>20.0</td><td>20.5</td><td>41.0</td><td>39.6</td><td>89.0</td><td>48.7</td><td>57.0</td><td>8.0</td><td>97.0</td><td>45.2</td>
  </tr>
  <tr>
    <td>Quest</td><td>192</td><td>35.9</td><td>44.2</td><td>52.8</td><td>41.0</td><td>17.7</td><td>23.8</td><td>63.0</td><td>36.0</td><td>86.0</td><td>43.6</td><td>52.3</td><td>8.4</td><td>96.5</td><td>46.2</td>
  </tr>
  <tr>
    <td>PQCache</td><td>192</td><td>45.6</td><td>51.2</td><td>53.8</td><td>45.3</td><td>29.0</td><td>25.9</td><td>69.5</td><td>41.2</td><td>91.3</td><td>54.1</td><td>58.5</td><td>8.2</td><td>99.0</td><td>51.7</td>
  </tr>
  <tr>
    <td>TailorKV-1</td><td>64(+128)</td><td>43.4</td><td>53.0</td><td>55.3</td><td>46.5</td><td>31.3</td><td>27.2</td><td>70.0</td><td>42.5</td><td>91.6</td><td>54.8</td><td>61.8</td><td>7.9</td><td>99.0</td><td>52.6</td>
  </tr>
  <tr>
    <td>TailorKV-2</td><td>64(+128)</td><td>44.8</td><td>53.9</td><td>54.8</td><td>46.2</td><td>31.9</td><td>26.8</td><td>70.5</td><td>43.2</td><td>92.3</td><td>54.2</td><td>62.1</td><td>7.7</td><td>99.0</td><td>52.9</td>
  </tr>
  <tr>
    <td><i>Yi-9B</i></td><td>200K</td><td>38.4</td><td>34.9</td><td>52.7</td><td>36.7</td><td>31.0</td><td>26.7</td><td>77.0</td><td>14.9</td><td>90.0</td><td>67.4</td><td>71.9</td><td>2.5</td><td>67.5</td><td>47.0</td>
  </tr>
  <tr>
    <td>StreamLLM</td><td>192</td><td>22.3</td><td>20.4</td><td>36.7</td><td>30.6</td><td>11.8</td><td>10.3</td><td>45.5</td><td>9.2</td><td>77.7</td><td>49.6</td><td>54.1</td><td>3.5</td><td>26.0</td><td>30.6</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>192</td><td>26.7</td><td>23.3</td><td>44.2</td><td>33.4</td><td>11.5</td><td>12.3</td><td>44.5</td><td>13.4</td><td>89.1</td><td>56.6</td><td>62.8</td><td>1.6</td><td>36.0</td><td>35.0</td>
  </tr>
  <tr>
    <td>Quest</td><td>192</td><td>32.4</td><td>26.0</td><td>44.2</td><td>31.5</td><td>14.3</td><td>16.5</td><td>73.0</td><td>13.4</td><td>86.2</td><td>55.6</td><td>63.7</td><td>3.9</td><td>47.5</td><td>39.1</td>
  </tr>
  <tr>
    <td>PQCache</td><td>192</td><td>36.9</td><td>27.9</td><td>47.6</td><td>35.5</td><td>19.3</td><td>19.2</td><td>74.0</td><td>12.2</td><td>89.6</td><td>62.0</td><td>66.8</td><td>4.6</td><td>51.0</td><td>42.0</td>
  </tr>
  <tr>
    <td>TailorKV-1</td><td>64(+128)</td><td>37.7</td><td>38.2</td><td>52.8</td><td>35.8</td><td>29.8</td><td>24.9</td><td>76.0</td><td>15.2</td><td>89.5</td><td>64.3</td><td>68.4</td><td>3.5</td><td>45.0</td><td>44.7</td>
  </tr>
  <tr>
    <td>TailorKV-2</td><td>64(+128)</td><td>37.6</td><td>33.6</td><td>52.6</td><td>34.4</td><td>29.4</td><td>25.3</td><td>76.0</td><td>15.0</td><td>89.3</td><td>63.5</td><td>68.5</td><td>3.0</td><td>44.0</td><td>44.0</td>
  </tr>
  <tr>
    <td><i>Yi-6B</i></td><td>200K</td><td>25.4</td><td>39.5</td><td>14.8</td><td>15.8</td><td>2.8</td><td>0.01</td><td>72.5</td><td>7.7</td><td>69.7</td><td>58.5</td><td>61.2</td><td>3.5</td><td>15.5</td><td>29.7</td>
  </tr>
  <tr>
    <td>StreamLLM</td><td>192</td><td>11.0</td><td>29.0</td><td>11.1</td><td>12.1</td><td>3.1</td><td>0.1</td><td>41.5</td><td>5.1</td><td>55.5</td><td>43.1</td><td>46.2</td><td>3.0</td><td>5.0</td><td>20.4</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>192</td><td>14.9</td><td>33.5</td><td>13.1</td><td>13.0</td><td>3.3</td><td>0.01</td><td>45.0</td><td>6.6</td><td>63.8</td><td>48.8</td><td>53.7</td><td>3.0</td><td>4.5</td><td>23.3</td>
  </tr>
  <tr>
    <td>Quest</td><td>192</td><td>19.9</td><td>33.2</td><td>11.8</td><td>13.2</td><td>0.5</td><td>0.1</td><td>67.0</td><td>9.4</td><td>64.5</td><td>50.2</td><td>53.7</td><td>3.5</td><td>13.5</td><td>26.2</td>
  </tr>
  <tr>
    <td>PQCache</td><td>192</td><td>24.1</td><td>36.8</td><td>13.6</td><td>15.3</td><td>1.3</td><td>0.01</td><td>68.5</td><td>7.6</td><td>68.1</td><td>54.1</td><td>57.4</td><td>2.5</td><td>5.5</td><td>27.3</td>
  </tr>
  <tr>
    <td>TailorKV-1</td><td>64(+128)</td><td>24.5</td><td>40.5</td><td>15.6</td><td>15.3</td><td>2.7</td><td>0.1</td><td>72.0</td><td>8.5</td><td>68.3</td><td>55.2</td><td>56.5</td><td>2.5</td><td>5.5</td><td>28.3</td>
  </tr>
  <tr>
    <td>TailorKV-2</td><td>64(+128)</td><td>24.1</td><td>40.8</td><td>15.2</td><td>15.5</td><td>3.0</td><td>0.01</td><td>72.0</td><td>8.6</td><td>66.7</td><td>54.8</td><td>57.9</td><td>2.5</td><td>5.5</td><td>28.2</td>
  </tr>
</table>

Table 14: Results on LongBench (Bai et al., 2024) of different methods.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Tokens</th>
    <th colspan="12">RPK R.Num En.Dia Sum En.MC En.QA Zh.QA Math.F Code.D Avg.</th>
  </tr>
  <tr>
    <th>R.PK</th><th>R.Num</th><th>En.Dia</th><th>Sum</th><th>En.MC</th><th>En.QA</th><th>Zh.QA</th><th>Math.F</th><th>Code.D</th><th>Avg.</th>
  </tr>
  <tr>
    <td><i>Llama-3.1-8B</i></td><td>128K</td><td>100.0</td><td>99.3</td><td>19.0</td><td>26.8</td><td>65.9</td><td>14.8</td><td>13.3</td><td>34.0</td><td>22.8</td><td>44.0</td>
  </tr>
  <tr>
    <td>StreamLLM</td><td>1024</td><td>3.3</td><td>3.0</td><td>7.0</td><td>12.7</td><td>66.3</td><td>5.9</td><td>9.7</td><td>34.0</td><td>22.8</td><td>18.3</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>1024</td><td>100.0</td><td>93.2</td><td>9.5</td><td>22.4</td><td>65.5</td><td>10.4</td><td>11.3</td><td>34.0</td><td>22.8</td><td>41.0</td>
  </tr>
  <tr>
    <td>Quest</td><td>1024</td><td>100.0</td><td>28.9</td><td>14.0</td><td>12.2</td><td>69.8</td><td>9.2</td><td>11.4</td><td>34.0</td><td>25.1</td><td>33.8</td>
  </tr>
  <tr>
    <td>PQCache</td><td>1024</td><td>8.6</td><td>2.5</td><td>15.0</td><td>18.9</td><td>65.9</td><td>12.6</td><td>12.6</td><td>34.0</td><td>23.3</td><td>21.5</td>
  </tr>
  <tr>
    <td>TailorKV-1</td><td>128+(896)</td><td>99.8</td><td>73.2</td><td>18.0</td><td>22.8</td><td>66.4</td><td>13.6</td><td>13.0</td><td>34.0</td><td>22.8</td><td>40.4</td>
  </tr>
  <tr>
    <td>TailorKV-2</td><td>128+(896)</td><td>100.0</td><td>89.4</td><td>18.5</td><td>24.1</td><td>66.8</td><td>14.4</td><td>12.9</td><td>34.0</td><td>22.8</td><td>42.6</td>
  </tr>
  <tr>
    <td><i>Yi-9B</i></td><td>200K</td><td>100.0</td><td>100.0</td><td>2.5</td><td>8.2</td><td>65.0</td><td>10.8</td><td>16.7</td><td>23.4</td><td>26.3</td><td>39.2</td>
  </tr>
  <tr>
    <td>StreamLLM</td><td>1024</td><td>2.5</td><td>0.5</td><td>2.5</td><td>6.4</td><td>66.8</td><td>8.8</td><td>15.0</td><td>23.7</td><td>21.3</td><td>16.4</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>1024</td><td>99.8</td><td>18.3</td><td>3.0</td><td>8.6</td><td>67.6</td><td>8.4</td><td>14.9</td><td>22.5</td><td>26.6</td><td>30.0</td>
  </tr>
  <tr>
    <td>Quest</td><td>1024</td><td>100.0</td><td>96.9</td><td>4.0</td><td>3.4</td><td>58.5</td><td>12.5</td><td>12.7</td><td>18.2</td><td>18.7</td><td>36.1</td>
  </tr>
  <tr>
    <td>PQCache</td><td>1024</td><td>9.6</td><td>5.9</td><td>2.0</td><td>8.7</td><td>66.3</td><td>11.2</td><td>14.9</td><td>22.2</td><td>25.6</td><td>18.5</td>
  </tr>
  <tr>
    <td>TailorKV-1</td><td>128+(896)</td><td>100.0</td><td>98.3</td><td>2.5</td><td>7.3</td><td>64.2</td><td>15.1</td><td>19.7</td><td>24.0</td><td>21.3</td><td>39.2</td>
  </tr>
  <tr>
    <td>TailorKV-2</td><td>128+(896)</td><td>100.0</td><td>99.7</td><td>4.5</td><td>8.2</td><td>65.1</td><td>11.2</td><td>16.6</td><td>24.0</td><td>24.9</td><td>39.4</td>
  </tr>
  <tr>
    <td><i>Yi-6B</i></td><td>200K</td><td>100.0</td><td>98.4</td><td>1.0</td><td>3.4</td><td>53.3</td><td>18.2</td><td>26.0</td><td>6.8</td><td>26.9</td><td>37.1</td>
  </tr>
  <tr>
    <td>StreamLLM</td><td>1024</td><td>3.3</td><td>0.6</td><td>0.0</td><td>3.9</td><td>52.8</td><td>10.2</td><td>20.2</td><td>4.8</td><td>25.8</td><td>13.5</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>1024</td><td>100.0</td><td>99.1</td><td>3.0</td><td>3.6</td><td>52.4</td><td>13.4</td><td>18.8</td><td>5.1</td><td>26.6</td><td>35.8</td>
  </tr>
  <tr>
    <td>Quest</td><td>1024</td><td>100.0</td><td>3.7</td><td>1.5</td><td>4.4</td><td>51.5</td><td>16.4</td><td>26.2</td><td>5.7</td><td>26.9</td><td>16.3</td>
  </tr>
  <tr>
    <td>PQCache</td><td>1024</td><td>10.1</td><td>3.7</td><td>1.5</td><td>4.4</td><td>51.5</td><td>16.4</td><td>26.2</td><td>5.7</td><td>26.9</td><td>16.3</td>
  </tr>
  <tr>
    <td>TailorKV-1</td><td>128+(896)</td><td>100.0</td><td>97.5</td><td>2.5</td><td>4.4</td><td>53.3</td><td>17.4</td><td>26.0</td><td>7.7</td><td>26.4</td><td>37.2</td>
  </tr>
  <tr>
    <td>TailorKV-2</td><td>128+(896)</td><td>100.0</td><td>97.0</td><td>3.0</td><td>4.0</td><td>53.3</td><td>18.0</td><td>25.8</td><td>8.0</td><td>26.7</td><td>37.3</td>
  </tr>
</table>

Table 15: Results on InfiniteBench (Zhang et al., 2024c) of different methods.
<table>
  <tr>
    <th>Methods</th>
    <th>N-S1</th>
    <th>N-S2</th>
    <th>N-S3</th>
    <th>N-MK1</th>
    <th>N-MK2</th>
    <th>N-MK3</th>
    <th>N-MV</th>
    <th>N-MQ</th>
    <th>VT</th>
    <th>CWE</th>
    <th>FWE</th>
    <th>QA-1</th>
    <th>QA-2</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th colspan="15">Sequence Length = 64k</th>
  </tr>
  <tr>
    <td><i>Llama-3.1-8B</i></td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>96.0</td>
    <td>99.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>14.8</td>
    <td>92.0</td>
    <td>60.0</td>
    <td>52.0</td>
    <td>85.6</td>
  </tr>
  <tr>
    <td>StreamLLM</td>
    <td>8.0</td>
    <td>4.0</td>
    <td>0.0</td>
    <td>8.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>5.0</td>
    <td>3.0</td>
    <td>2.4</td>
    <td>0.8</td>
    <td>72.0</td>
    <td>28.0</td>
    <td>28.0</td>
    <td>12.2</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>96.0</td>
    <td>84.0</td>
    <td>0.0</td>
    <td>88.0</td>
    <td>32.0</td>
    <td>0.0</td>
    <td>40.0</td>
    <td>69.0</td>
    <td>74.4</td>
    <td>0.8</td>
    <td>41.3</td>
    <td>56.0</td>
    <td>48.0</td>
    <td>48.4</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>88.0</td>
    <td>100.0</td>
    <td>60.0</td>
    <td>92.0</td>
    <td>72.0</td>
    <td>0.0</td>
    <td>93.0</td>
    <td>90.0</td>
    <td>80.0</td>
    <td>8.4</td>
    <td>70.6</td>
    <td>52.0</td>
    <td>52.0</td>
    <td>66.0</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>36.0</td>
    <td>60.0</td>
    <td>12.0</td>
    <td>68.0</td>
    <td>48.0</td>
    <td>4.0</td>
    <td>16.0</td>
    <td>37.0</td>
    <td>52.8</td>
    <td>0.0</td>
    <td>73.3</td>
    <td>56.0</td>
    <td>48.0</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td><b>TailorKV-1</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>96.0</b></td>
    <td><b>100.0</b></td>
    <td><b>96.0</b></td>
    <td><b>28.0</b></td>
    <td><b>99.0</b></td>
    <td><b>100.0</b></td>
    <td><b>85.6</b></td>
    <td><b>18.4</b></td>
    <td><b>57.3</b></td>
    <td><b>56.0</b></td>
    <td><b>48.0</b></td>
    <td><b>75.7</b></td>
  </tr>
  <tr>
    <td><b>TailorKV-2</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>96.0</b></td>
    <td><b>68.0</b></td>
    <td><b>97.0</b></td>
    <td><b>98.0</b></td>
    <td><b>88.0</b></td>
    <td><b>19.6</b></td>
    <td><b>62.7</b></td>
    <td><b>60.0</b></td>
    <td><b>56.0</b></td>
    <td><b>80.4</b></td>
  </tr>
  <tr>
    <th colspan="15">Sequence Length = 128k</th>
  </tr>
  <tr>
    <td><i>Llama-3.1-8B</i></td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>96.0</td>
    <td>56.0</td>
    <td>24.0</td>
    <td>39.0</td>
    <td>76.0</td>
    <td>24.8</td>
    <td>0.8</td>
    <td>73.3</td>
    <td>32.0</td>
    <td>57.3</td>
  </tr>
  <tr>
    <td>StreamLLM</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>8.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>3.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.4</td>
    <td>62.6</td>
    <td>20.0</td>
    <td>16.0</td>
    <td>8.5</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>88.0</td>
    <td>4.0</td>
    <td>0.0</td>
    <td>16.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>5.0</td>
    <td>7.0</td>
    <td>15.2</td>
    <td>0.0</td>
    <td><b>65.3</b></td>
    <td>28.0</td>
    <td>20.0</td>
    <td>19.1</td>
  </tr>
  <tr>
    <td>Quest</td>
    <td>72.0</td>
    <td>84.0</td>
    <td>0.0</td>
    <td>52.0</td>
    <td>20.0</td>
    <td>0.0</td>
    <td>28.0</td>
    <td>30.0</td>
    <td>20.0</td>
    <td>1.6</td>
    <td>56.0</td>
    <td>24.0</td>
    <td>20.0</td>
    <td>31.3</td>
  </tr>
  <tr>
    <td>PQCache</td>
    <td>16.0</td>
    <td>20.0</td>
    <td>0.0</td>
    <td>28.0</td>
    <td>8.0</td>
    <td>0.0</td>
    <td>5.0</td>
    <td>3.0</td>
    <td>10.4</td>
    <td>0.0</td>
    <td>50.6</td>
    <td>24.0</td>
    <td>24.0</td>
    <td>14.5</td>
  </tr>
  <tr>
    <td><b>TailorKV-1</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>96.0</b></td>
    <td><b>12.0</b></td>
    <td><b>24.0</b></td>
    <td><b>41.0</b></td>
    <td><b>65.0</b></td>
    <td><b>28.8</b></td>
    <td><b>1.2</b></td>
    <td><b>58.7</b></td>
    <td><b>28.0</b></td>
    <td><b>24.0</b></td>
    <td><b>52.2</b></td>
  </tr>
  <tr>
    <td><b>TailorKV-2</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>100.0</b></td>
    <td><b>24.0</b></td>
    <td><b>28.0</b></td>
    <td><b>40.0</b></td>
    <td><b>67.0</b></td>
    <td><b>42.4</b></td>
    <td><b>0.8</b></td>
    <td><b>57.3</b></td>
    <td><b>32.0</b></td>
    <td><b>24.0</b></td>
    <td><b>55.1</b></td>
  </tr>
</table>

Table 16: Accuracy (%) of different methods and models on RULER (Hsieh et al., 2024) evaluated at length of 64k and 128k. The sparsity-friendly layer in TailorKV uses 128+(896) tokens, while other methods use 1024 tokens.
(a) Visualization of attention weights on Llama-2-7B-32K-Instruct.

(b) Visualization of attention weights on Llama-3.1-8B-Instruct.

(c) Visualization of attention weights on Yi-6B-200K.

Figure 13: Visualization of attention weights across the 2WikiMQA dataset.
Figure 14: Magnitude of query, key and value for Llama-3.1-8B-Instruct.

Reviewer paper summaries: The paper proposes TailorKV to compress the hybrid KV cache from quantization-friendly layers and sparsity-friendly layers. For identifying these two type of layers, the paper presents an offline metrics to identify layer-specific compression preference. The experimental results demonstrate that the proposed method is nearly lossless with minimal GPU memory consumption and comparable latency.

This paper mainly represents identification metrics to identify layer-specific compression preferences and introduces an optimized inference system that focuses on utilizing a combination of compression and offloading techniques in different layers. This paper also provides detailed explanation and experiments demonstrating performance downgrade but latency optimization compared to baseline optimization techniques. It also gives performance comparisons when different layers are applied with quantization (Tailored Strategies) and the effects of dynamic/static approaches and number of channels.

This paper proposes TailorKV, a hybrid KV cache compression method that applies quantization to the early layers of a language model and uses a sparse cache for the later layers. The authors empirically observe that attention patterns vary across layers and argue that not all parts of the model should be compressed in the same way. They find that quantization and sparsity are each more effective on different layers, motivating a layer-wise, tailored approach. TailorKV uses techniques to automatically identify the most suitable compression strategy for each layer, prefetch necessary data, and overlap CPU-GPU communication with computation to improve efficiency.
Summary of strengths: 1. The paper is well-written and easy to understand.
    
2. The paper proposes a hybrid KV cache compression framework combining quantization and offloading.
    
3. The paper proposes a metrics to identify two types of layers: quantization-friendly layers and sparsity-friendly layers.
    
4. The paper reveals observations on compression preferences of layers (quantization-friendly layers have higher sparse errors) and correlate with outliers.
    
5. Extensive experiments demonstrate that the proposed method is nearly lossless, with minimal GPU memory consumption and acceptable latency.

(1) This paper uniquely identifies different layers' preferences for compression techniques and uses detailed experiments to focus on the effects they bring.  
(2) This paper utilizes experiments to support the improvement gained by using dynamic (Critical) channels instead of static channels.  
(3) The system's overview and details are clearly illustrated and analyzed.

- The proposed approach is well-motivated using detailed empirical observations.
- The experiments are quite comprehensive.
Summary of weaknesses: 1. In Table 1, the computing of performance loss is not explained. Additionally, the term “KIVI“ is not clarified.
    
2. In line 205 to 209, the reason why quantization-friendly layers focus on global information while sparsity-friendly layers focus on crucial information should be discussed.
    
3. The paper does not explore the effectiveness of the proposed method on larger models. Larger (deeper) models introduce more attention layers, which may reduce the contribution of the proposed quantization strategy, making KV cache compression primarily dependent on sparsity.
    
4. Missing baselines (SnapKV, Quest) in comparison of peak memory and latency.

(1) As the core of the metrics, the threshold that determine whether a layer is Quantization-friendly is not been thoroughly discussed.

- While the paper presents a well-engineered system, its novelty is relatively limited. The proposed approach largely appears to be a combination of existing techniques (quantization and token selection) applied in a hybrid fashion rather than introducing a fundamentally new idea.
- Several design choices raise concerns:
1. Extremely aggressive 1-bit quantization is applied to the early layers. Given that errors in early layers can propagate and amplify through the network, this design is risky. Prior work [1] has shown that naive 1-bit quantization can significantly degrade model quality. It remains unclear whether this aggressive quantization strategy would generalize well to other models.
2. The assumption that hidden states from the previous layer can accurately identify critical tokens for the current layer may not hold universally.
3. Dynamic retrieval requires a lot of work to be done on the CPU. This could become a bottleneck, especially if the GPU is much faster than the CPU. If the CPU can't keep up, it might slow down or stall the whole inference process.
- Some technical details are underspecified. In particular: How is the 1-bit quantization actually performed? What is the mathematical formulation? How does this quantization differ from existing quantization methods?

References

[1] Zhang, Tianyi, et al. "Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization." Advances in Neural Information Processing Systems 37 (2024): 3304-3331.
Comments, suggestions and typos: N/A

(1) Experiments and discussions on choosing the Quantization-Friendly threshold.

Please see weaknesses.
Overall assessment scores: 3.5

3.5

2.5
Metareview score: 3.5
Metareview: This paper presents TailorKV, a hybrid KV cache optimization framework designed to improve memory and latency efficiency for long-context inference in LLMs. TailorKV introduces a layer-specific strategy that categorizes transformer layers into either quantization-friendly or sparsity-friendly, based on empirical measurements of attention sparsity. Quantization-friendly layers undergo aggressive 1-bit quantization, while sparsity-friendly layers apply selective KV cache offloading and token retrieval. TailorKV further overlaps CPU-GPU communication with computation through dynamic prefetching, maximizing throughput and minimizing overhead.
Summary of reasons to publish: 1. While prior works have studied quantization and sparsity separately, TailorKV contributes a novel per-layer classification mechanism that combines the two in a complementary and hardware-friendly fashion.

2. The paper presents comprehensive benchmarks on real-world long-context tasks with both small and large LLMs.

3. TailorKV includes a hardware-aware implementation that effectively overlaps CPU-GPU communication with computation.