Accurate KV Cache Quantization with Outlier Tokens Tracing
Authors: Yi Su, Yuechi Zhou, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang
Accepted at: acl25

Accurate KV Cache Quantization with Outlier Tokens Tracing

Yi Su1,2*, Yuechi Zhou1,2*, Quantong Qiu1,2, Juntao Li1,2†
Qingrong Xia3, Ping Li3, Xinyu Duan3, Zhefeng Wang3, Min Zhang1,2
1School of Computer Science and Technology, Soochow University
2Key Laboratory of Data Intelligence and Advanced Computing, Soochow University
3Huawei Cloud
yisunlp@outlook.com; {ljt,minzhang}@suda.edu.cn

Abstract

The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recomputation during inference, it also introduces additional memory overhead. KV Cache quantization presents a promising solution, striking a good balance between memory usage and accuracy. Previous research has shown that the Keys are distributed by channel, while the Values are distributed by token. Consequently, the common practice is to apply channel-wise quantization to the Keys and token-wise quantization to the Values. However, our further investigation reveals that a small subset of unusual tokens exhibit unique characteristics that deviate from this pattern, which can substantially impact quantization accuracy. To address this, we develop a simple yet effective method to identify these tokens accurately during the decoding process and exclude them from quantization as outlier tokens, significantly improving overall accuracy. Extensive experiments show that our method achieves significant accuracy improvements under 2-bit quantization and can deliver a 6.4 times reduction in memory usage and a 2.3 times increase in throughput1.

1 Introduction

Large Language Models (LLMs) have significantly impacted various industries due to their powerful capabilities (Achiam et al., 2023; Touvron et al., 2023a,b; Dubey et al., 2024; Jiang et al., 2023). However, their auto-regressive nature makes the generation process slow. Although using KV Cache can reduce decoding complexity from \( O(n^2) \) to \( O(n) \) by storing the Keys and the Values computed during inference, it introduces substantial memory overhead. This overhead scales with sequence length, batch size, and hidden dim, often creating a memory bottleneck and placing considerable pressure on resources during deployment. As a result, optimizing KV Cache management to enhance resource utilization and improve model throughput remains a critical challenge.

KV Cache affects throughput in two primary ways. First, its memory usage limits the scalability of batch sizes, reducing parallelism during decoding, and thus lowering throughput. Second, attention computation is delayed while waiting for the KV Cache to be transferred from memory to the computation unit. As the size of the KV Cache grows, the transmission time increases, decreasing throughput. Existing approaches mainly address this issue by optimizing hardware scheduling (Aminabadi et al., 2022; Dao et al., 2022; Sheng et al., 2023; Kwon et al., 2023) and reducing the size of the KV Cache (Liu et al., 2024b; Hooper et al., 2024; Kang et al., 2024; Zhang et al., 2023; Xiao et al., 2024). In this paper, we focus on the latter approach, KV cache compression.

One method of reducing the size of the KV Cache is to reduce the number of values that need to be stored, which is related to the shape of the KV Cache: [num_layers, batch_size, num_heads, sequence_length, head_dim]. There are various compression methods in each dimension, including layer-wise KV Cache sharing (Wu and Tu, 2024; Brandon et al., 2024; Zuhri et al., 2024; Mu et al., 2024), prefix sharing (Juravsky et al., 2024; Zhu et al., 2024), head-wise KV Cache sharing (Shazeer, 2019; Ainslie et al., 2023), token eviction (Xiao et al., 2024; Zhang et al., 2023; Ge et al., 2024), and low-rank projection (Wang et al., 2024; Yu et al., 2024; Chang et al., 2024).

Another strategy for reducing the size of KV Cache is quantization. However, unlike weight quantization, KV Cache quantization poses unique challenges due to the uneven distribution of the
Keys and Values (Kang et al., 2024). To enhance quantization accuracy, various methods have been proposed, including using low-rank matrices to approximate the error before and after quantization (Kang et al., 2024), smoothing Key distributions through specific mappings (Ashkboos et al., 2024; Chang et al., 2024), channel-wise Key and token-wise Value asymmetric quantization (Liu et al., 2024b; Hooper et al., 2024), non-uniform quantization (Hooper et al., 2024; Dettmers et al., 2022), mixed-precision quantization (Dong et al., 2024), and Block Floating Point (BFP) quantization (Trukhanov and Soloveychik, 2024). Among these methods, channel-wise Key and token-wise Value asymmetric quantization has garnered much attention for its high accuracy and tuning-free nature. This technique operates under the assumption that some channels of the Keys have huge magnitudes and that the distribution of the Keys within the same channel is relatively uniform.

However, our further exploration reveals that a few outlier tokens deviate from this assumption. Their Keys have very small magnitudes in the outlier channels with large magnitudes, which greatly affects the accuracy of quantization. Based on these observations, we propose KV Cache Quantization with Outlier Tokens Tracing (OTT), a simple yet effective method that identifies these tokens and excludes them from the quantization process, thereby improving quantization accuracy. With hardware-friendly implementation, OTT achieves significant accuracy improvements under 2-bit quantization, resulting in a 6.4x reduction in memory usage and a 2.3x increase in throughput.

Overall, our contributions are as follows:

• We investigate the outlier channels of the KV Cache and identify that some outlier tokens deviate from the previous assumptions.

• We propose KV Cache Quantization with Outlier Tokens Tracing (OTT), a simple yet effective method to identify and exclude these tokens during quantization, thus improving overall quantization accuracy.

• Our method achieves significant accuracy improvements under 2-bit quantization, yielding a 6.4x reduction in memory usage and a 2.3x increase in throughput.

2 Background

Implementation of KV Cache. Transformer-based (Vaswani, 2017) LLMs typically utilize KV cache to prevent the redundant calculation of the attention scores and accelerate auto-regressive decoding. The generation process of LLMs with KV cache is divided into the prefill phase and the decoding phase (Patel et al., 2024). Given a prompt \( X = \{x_0, x_1, \ldots, x_{n-1}\} \) and tensor \( X \in \mathbb{R}^{b \times n \times d} \) after embedding, where \( b \) is the batch size, \( n \) is the length of the prompt, and \( d \) represents the hidden size, we will briefly describe the calculation process of the attention block, and we omit the number of heads in the multi-head attention mechanism.

i) During the prefill phase, the Keys \( K_{<n} \) and Values \( V_{<n} \) are computed and cached by transforming \( X \) through the Key and Value weight matrices \( W_k, W_v \in \mathbb{R}^{d \times d} \) of each layer, which can be formulated as:

\[
K_{<n} = XW_k, \quad V_{<n} = XW_v.
\]

ii) During the decoding phase, only the Keys and Values of the new token \( x_n \) need to be calculated, which are then combined with the cached Keys and Values to compute the new attention scores and outputs. For the current input tensor \( X_n \in \mathbb{R}^{b \times 1 \times d} \), we update the KV cache as follows:

\[
K = K_{<n} \| K_n, \quad V = V_{<n} \| V_n,
\]

where \( K_n = X_n W_k \) and \( V_n = X_n W_v \). We calculate the new attention output ATT as follows:

\[
Q_n = X_n W_q, \quad \text{ATT} = \sigma \left( \frac{Q_n K^T}{\sqrt{d_k}} \right) V, \tag{1}
\]

where \( W_q \) is the query weight matrix, \( \sqrt{d_k} \) is the normalization factor, and \( \sigma \) is the softmax function.

Necessity of compression. While KV cache reduces the computational complexity from \( O(n^2) \) to \( O(n) \), it introduces substantial GPU memory overhead, particularly with long sequence lengths and large batch sizes. For example, in the case of LLaMA-3-8B (Dubey et al., 2024), where the number of layers \( m_{layers} \) is 32, the number of heads \( h \) is 8, the head dimension \( d \) is 512, the input length \( l \) is 8192, and the batch size \( b \) is 64, performing inference with fp16 precision (which uses 2 bytes per value) requires \( 4bhdlm_{layers} \) bytes to store the KV cache—equivalent to 256GB of memory. Thus, effectively compressing the KV cache is crucial to reducing GPU memory usage.
Uniform Quantization. In this paper, we focus on compressing the KV cache by reducing the bit-width needed to represent cached tensors. A straightforward approach is Uniform Quantization (Jacob et al., 2018), which maps continuous numerical data to a discrete domain. Specifically, to quantize a high-precision matrix (e.g., float32) \( X \) to a matrix \( X' \) with \( b \)-bit precision, we first determine the quantization step size \( q \). Each element \( X_{i,j} \in X \) can be quantized to \( Q(X_{i,j}) \) as follows:

\[
Q(X_{i,j}) = \lfloor (X_{i,j} - X_{min}) / q \rfloor,
\]
\[
q = (X_{max} - X_{min}) / (2^b - 1),
\]

where \( \lfloor \cdot \rfloor \) is the rounding function.

Group Quantization. However, Uniform Quantization does not fully exploit the distribution characteristics of the data, which can lead to significant quantization errors, especially when there are outliers. A more advanced technique is Group Quantization (Yao et al., 2022), which divides the matrix into multiple groups, expecting the data within each group to share similar distribution characteristics. Unlike Uniform Quantization, Group Quantization allows each group to have different quantization parameters, such as step size. This flexibility enables the method to better adapt to the local characteristics of the data, thereby reducing quantization errors while maintaining a low bit-width. The channel-wise Key quantization and token-wise Value quantization proposed by KIVI (Liu et al., 2024b) is a type of Group Quantization.

3 Method

In this section, we propose KV Cache Quantization with Outlier Tokens Tracing (\( OTT \)). We start with a preliminary exploration of the Keys and Values before introducing our method.

3.1 Exploration of the Keys and Values.

We conduct a series of preliminary experiments to gain a deeper understanding of the Keys and Values. For illustration, we take a sentence generated by LLaMA-2-7b-chat-hf 2 as an example. Table 5 in the Appendix presents the prompt and the generated context.

Distribution of the Keys and Values. Figure 1a and Figure 1b display the magnitude of the Keys and Values from layer 10, head 17. Notably, some channels exhibit exceptionally large Keys, and within these channels, the distribution of the Keys appears relatively uniform. In contrast, the Values have no distinct characteristics. These observations are consistent with KIVI (Liu et al., 2024b).

Distribution in outlier channels. We further investigate the distribution of these outlier channels. Figure 1d shows the Keys in an outlier channel from layer 10, head 17 (we plot the first 100 tokens). While the Keys generally exhibit a uniform distribution, a few tokens are notable exceptions. This pattern becomes clearer after sorting, as shown in Figure 1e, where some Keys have very small values while others are significantly larger. These exceptions can substantially increase \( X_{max} - X_{min} \) in Equation 2 during quantization, ultimately diminishing quantization accuracy. Statistical analysis of these outliers can be found in Appendix F.

Identifying Outlier Tokens. Intuitively, tokens with very small magnitude of the Keys in outlier channels are also likely to have smaller magnitude overall. To test this hypothesis, we plot the Keys from an outlier channel and the magnitude of the Keys across all channels (we plot the first 300 tokens). As shown in Figure 4 in the Appendix, the results confirm our assumption, suggesting that we can efficiently and accurately identify these outlier tokens with the magnitude of the Keys.

Removing Outlier Tokens. From our analysis, outlier tokens significantly impact the accuracy of quantization. By excluding these outlier tokens during quantization and retaining them with full precision, we can greatly reduce the loss of the attention output. To investigate this further, we retain different numbers of tokens based on different selection criteria and compare the L1 loss of attention outputs before and after quantization. The results (Figure 1c) reveal that retaining tokens with the largest keys yields the worst performance, while retaining those with the smallest Keys achieves the best results, aligning with our previous findings.

3.2 \( OTT \): KV Cache Quantization with Outlier Tokens Tracing

From the above observations, we find that some outlier tokens can seriously affect the accuracy of quantization. So, we attempt to dynamically identify these tokens during the quantization process, exclude them during quantization, and retain their full-precision representations. Our method consists of two components: quantization and decoding.

Quantization We define a fixed-size outlier pool with a capacity of \( outlier\_num \) to store the
Figure 1: Observations from preliminary experiments: (a) The Keys are distributed by channel and have some outlier channels. (b) The distribution of the Values does not exhibit any notable characteristics. (d) In certain outlier channels, a few tokens with low magnitude of Keys disrupt the originally uniform distribution within these channels. (e) Visualization of the sorted Keys in an outlier channel shows a rapid increase from a low value to very high values. (c) The L1 loss of attention output before and after quantization by retaining full-precision tokens based on different criteria. The best result is retaining full-precision tokens with the smallest magnitude of the Keys.

Keys and Values of the outlier tokens. Following KIVI (Liu et al., 2024b), we use channel-wise Key quantization and token-wise Value quantization. We quantize KV Cache every \( G \) (group size, a hyper-parameter in our method) steps. So, at each quantization step, there are \( G \) tokens to quant. Based on the rule from Section 3.1, we compute the magnitude of the Keys of each token as the criteria, and all the tokens (tokens to quant and tokens in the outlier pool) compete for a position in the outlier pool according to the criteria. Once selected as an outlier token, the Keys and Values of the token are replaced with the mean values of all tokens to eliminate their impact on quantization. When the outlier pool is full and replacements are needed, the tokens that are originally in the outlier pool but are defeated by the newly added tokens should return to their original positions. But for the sake of simplicity, we retain an additional pool to store these tokens, and when this pool is full, we stop identifying outlier tokens.

Decoding We maintain three types of KV Cache: the quantized KV Cache, the full-precision KV Cache, and the KV Cache stored in the outlier pool.

The full-precision KV Cache includes group tokens (when the group is not full, these tokens are not quantized and are kept in full precision) and recent tokens (a full-precision sliding window for the nearest tokens). First, the Query is multiplied by all three types of Keys, and we concatenate the results to produce the attention scores. Next, we multiply these scores by their corresponding Values from each type and sum them to generate the final attention output. To enhance decoding efficiency, we utilize a CUDA fused kernel to multiply full-precision and quantized matrices. We provide a simple mathematical formulation in Appendix E.

4 Experiments

4.1 Settings

Baselines and Models. KIVI (Liu et al., 2024b) is currently one of the strongest tuning-free baseline with high compression efficiency and accuracy. We use the same setting as KIVI, so we compare OTT with KIVI and vanilla FP16 in our main experiments. Due to differences in settings (e.g., compression frequency, compression factor, and the number of full-precision tokens), we do not include
Figure 2: Overview of \( OTT \). Top: Decoding stage. Multiply the Query by each type of the Keys and concatenate the results to obtain the attention scores. Multiply the attention scores by each type of the Values and sum the results to get the attention output. Bottom: Quantization stage. Process the outlier tokens before quantization.

other KV Cache compression methods in the main experiment. However, additional comparisons with these methods are provided in Appendix B. We use greedy decoding in our experiments. We select two famous model families: LLaMA (Touvron et al., 2023b; Dubey et al., 2024) and Mistral (Jiang et al., 2023). Specifically, we select LLaMA-2-7B-chat-hf, LLaMA-2-13B-chat-hf, LLaMA-3-8B-Instruct, and Mistral-7B-Instruct-v0.2. We also add experiments on LLaMA-2-70B-chat-hf in Appendix B.

Tasks. We evaluate our methods and the baselines on two benchmarks according to the length of input texts. For normal context length evaluation, we use arithmetic reasoning task Gsm8k (Cobbe et al., 2021), mainstream language and symbolic reasoning task BBH (Suzgun et al., 2023), and code completion task HumanEval (Chen et al., 2021). For long context length evaluation, we choose four types of tasks in LongBench (Bai et al., 2024a) including Document QA, Summarization, Few-shot Learning and Code completion. We provide more results of different benchmarks and baselines in Appendix G.

Details. We implement both KIVI and \( OTT \) under 2-bit quantization. For KIVI, the group size (G) and residual length (R, size of the sliding window storing the nearest tokens) are set to 128. For \( OTT \), we use G = 128, R = 32, and set outlier\_num to 3. Notably, we set outlier\_num to 0 for the first and second layers because we find that shallow layers have no outlier tokens (Ablation in Section 4.4). Regarding the additional pool used to store tokens evicted from the outlier pool, we find that a very small size is sufficient to retain all the eliminated tokens. Therefore, we set its size to 32. There are some differences in the processing of group and residual tokens between \( OTT \) and KIVI. KIVI compresses the Keys every G steps, while compressing the Values at each step. We choose to compress both the Keys and Values every G steps for more consistent processing of KV Cache. GSM8K and BBH are tested under LM Eval (Gao et al., 2024). Humaneval follows the settings from InstructEval\(^{3}\). The experiments are conducted on NVIDIA A100 40G GPUs unless otherwise specified.

4.2 Results

4.2.1 Normal context length evaluation

Table 1 presents the results of the normal context length evaluation across different models and methods. For Gsm8k and BBH, we report accuracy in the setting of few-shot, few-shot CoT, and zero-shot CoT. For HumanEval, we report pass@1 and pass@10 in the zero-shot setting. The results illustrate that our method significantly outperforms KIVI across all settings. Notably, on BBH (3-CoT, LLaMA-3-8B-Instruct), \( OTT \) achieves a 12.93% improvement over KIVI. Compared to FP16, \( OTT \) incurs minor accuracy loss in most settings. The largest accuracy drop occurs on BBH (3-CoT, Mistral-7B-Instruct), likely due to the high complexity of the task and the long generation length required. Overall, \( OTT \) can achieve significant performance improvements over KIVI.

\footnotetext{3https://github.com/declare-lab/instruct-eval}
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="3">LLaMA-2-7B-chat-hf</th>
    <th colspan="3">LLaMA-2-13B-chat-hf</th>
    <th colspan="3">LLaMA-3-8B-Instruct</th>
    <th colspan="3">Mistral-7B-Instruct</th>
  </tr>
  <tr>
    <th>Fp16</th><th>KIVI</th><th>Ours</th>
    <th>Fp16</th><th>KIVI</th><th>Ours</th>
    <th>Fp16</th><th>KIVI</th><th>Ours</th>
    <th>Fp16</th><th>KIVI</th><th>Ours</th>
  </tr>
  <tr>
    <td>Gsm8k (8)</td>
    <td>21.99</td><td>16.30</td><td><b>21.38</b></td>
    <td>36.54</td><td>28.51</td><td><b>36.09</b></td>
    <td>74.91</td><td>63.15</td><td><b>72.55</b></td>
    <td>42.91</td><td>37.38</td><td><b>41.17</b></td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>21.30</td><td>17.51</td><td><b>18.20</b></td>
    <td>37.00</td><td>31.77</td><td><b>36.92</b></td>
    <td>76.72</td><td>66.79</td><td><b>75.06</b></td>
    <td>42.99</td><td>37.45</td><td><b>41.39</b></td>
  </tr>
  <tr>
    <td>+ 0-CoT</td>
    <td>24.11</td><td>21.61</td><td><b>22.59</b></td>
    <td>32.60</td><td>29.19</td><td><b>31.31</b></td>
    <td>40.64</td><td>37.54</td><td><b>42.68</b></td>
    <td>40.18</td><td>33.81</td><td><b>37.98</b></td>
  </tr>
  <tr>
    <td>BBH (3)</td>
    <td>33.34</td><td>32.48</td><td><b>33.36</b></td>
    <td>37.61</td><td>36.20</td><td><b>37.43</b></td>
    <td>45.77</td><td>44.19</td><td><b>45.60</b></td>
    <td>42.10</td><td>40.29</td><td><b>42.02</b></td>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>40.21</td><td>34.00</td><td><b>35.17</b></td>
    <td>47.38</td><td>41.02</td><td><b>44.37</b></td>
    <td>68.18</td><td>47.38</td><td><b>60.31</b></td>
    <td>51.33</td><td>36.42</td><td><b>41.93</b></td>
  </tr>
  <tr>
    <td>+ 0-CoT</td>
    <td>35.00</td><td>33.30</td><td><b>34.25</b></td>
    <td>35.86</td><td>33.57</td><td><b>34.80</b></td>
    <td>51.37</td><td>44.19</td><td><b>48.89</b></td>
    <td>41.74</td><td>37.83</td><td><b>40.19</b></td>
  </tr>
  <tr>
    <td>HE (p@1)</td>
    <td>12.19</td><td>9.75</td><td><b>11.58</b></td>
    <td>7.92</td><td>7.31</td><td><b>7.92</b></td>
    <td>40.24</td><td>28.05</td><td><b>40.85</b></td>
    <td>40.24</td><td>32.92</td><td><b>35.36</b></td>
  </tr>
  <tr>
    <td>HE (p@10)</td>
    <td>17.07</td><td>12.19</td><td><b>14.63</b></td>
    <td>13.41</td><td>11.58</td><td><b>15.24</b></td>
    <td>69.51</td><td>56.09</td><td><b>67.68</b></td>
    <td>54.87</td><td>50.00</td><td><b>54.26</b></td>
  </tr>
  <tr>
    <td>Average</td>
    <td>25.65</td><td>22.14</td><td><b>23.90</b></td>
    <td>31.04</td><td>28.14</td><td><b>30.51</b></td>
    <td>58.42</td><td>48.16</td><td><b>56.70</b></td>
    <td>44.55</td><td>38.26</td><td><b>41.79</b></td>
  </tr>
</table>

Table 1: Results on GSM8K, BBH, and HumanEval (HE). Bold: the best results. We report accuracy for Gsm8k, BBH and Pass@k for HumanEval. Pass@k (p@k) refers to running each test question k times and calculating the average pass rate of the generated code. OTT outperforms KIVI across all tasks, achieving the best results.

4.2.2 Long context length tasks evaluation
The main results of long context length evaluation are in table 2. Our method outperforms KIVI in most settings, with only a tiny performance gap compared to the Fp16 baseline. While KIVI maintains good accuracy on most tasks, it occasionally experiences significant performance drops (e.g., LLaMA-3-8B-Instruct, LCC: 56.58% → 44.42%). However, OTT does not encounter this situation, which suggests that our method achieves higher quantization accuracy than KIVI.

4.3 Efficiency comparison
Additionally, to validate the memory reduction and throughput improvements achieved by OTT, we conduct three experiments: a throughput test, a memory test, and a longest sentence test. The throughput test measures the number of tokens generated per second as the batch size varies while keeping the input and output lengths fixed. The memory test tracks memory usage as the batch size changes, also with fixed input and output lengths. The longest sentence test assesses the memory required for inference as the output length increases infinitely (until out-of-memory), with a fixed batch size of 1 and an input length of 1. We use the LLaMA-2-7B-chat-hf model for our experiments, and set the input length to 64 and the output length to 384 for both the throughput and memory tests. Figure 3 illustrates the results.

Figure 3a shows that when the batch size is small, OTT performs slightly slower than the FP16 baseline. However, as the batch size increases, OTT demonstrates a significant speed advantage. Our method is consistently faster than KIVI at any batch size because it does not require compressing the Values at each step. Although processing outlier tokens introduces some additional computational overhead, the outlier pool is very small, and the compression frequency is low. As a result, the overhead is negligible in the overall decoding process.

From Figure 3b, it is evident that quantization significantly reduces memory usage compared to the FP16 baseline. OTT requires slightly more memory than KIVI, this is because OTT tends to retain more full-precision tokens. However, as the sequence length increases, this impact diminishes. Figure 3c provides a clearer view of the compression ratio (represented by the slope of each line) for the KV Cache. When the sequence length becomes sufficiently large, the effects of full-precision tokens are negligible. Notably, the compression ratio of KIVI and OTT reaches approximately 6.4x. We provide more time analysis in Appendix C and G

4.4 Ablation studies
Group size and residual length. Group size and residual length are critical hyperparameters in OTT. Theoretically, a larger group size allows more values to be quantized at each step, which can reduce quantization accuracy due to the increased range of \( X_{max} - X_{min} \). On the other hand, a larger group size decreases memory usage by requiring fewer quantization coefficients to be retained. Conversely, increasing the residual length requires more memory since a larger full-precision KV Cache must
<table>
  <tr>
    <th>Model</th>
    <th>Qasper</th>
    <th>GovReport</th>
    <th>MultiNews</th>
    <th>TREC</th>
    <th>TriviaQA</th>
    <th>SamSum</th>
    <th>LCC</th>
    <th>RepoBench-P</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Fp16</td>
    <td>20.04</td>
    <td>25.08</td>
    <td>23.02</td>
    <td>59.67</td>
    <td>85.39</td>
    <td>39.28</td>
    <td>59.59</td>
    <td>48.04</td>
    <td>45.01</td>
  </tr>
  <tr>
    <td>LLaMA-2-7B-chat-hf</td>
    <td><b>KIVI</b> 20.43</td>
    <td>19.97</td>
    <td>19.82</td>
    <td>59.67</td>
    <td><b>85.16</b></td>
    <td>37.70</td>
    <td>58.73</td>
    <td>47.24</td>
    <td>43.59</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>19.95</td>
    <td><b>21.56</b></td>
    <td><b>20.81</b></td>
    <td><b>59.67</b></td>
    <td>85.00</td>
    <td><b>39.10</b></td>
    <td><b>59.44</b></td>
    <td><b>48.51</b></td>
    <td><b>44.26</b></td>
  </tr>
  <tr>
    <td>Fp16</td>
    <td>17.42</td>
    <td>25.65</td>
    <td>23.35</td>
    <td>64.00</td>
    <td>86.52</td>
    <td>40.49</td>
    <td>49.80</td>
    <td>47.13</td>
    <td>44.30</td>
  </tr>
  <tr>
    <td>LLaMA-2-13B-chat-hf</td>
    <td><b>KIVI</b> 20.10</td>
    <td>20.65</td>
    <td>21.10</td>
    <td>63.67</td>
    <td>86.39</td>
    <td>39.51</td>
    <td>49.10</td>
    <td>43.95</td>
    <td>43.06</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>18.81</td>
    <td><b>22.29</b></td>
    <td><b>21.69</b></td>
    <td><b>64.00</b></td>
    <td><b>86.81</b></td>
    <td><b>40.35</b></td>
    <td><b>51.14</b></td>
    <td><b>47.71</b></td>
    <td><b>44.10</b></td>
  </tr>
  <tr>
    <td>Fp16</td>
    <td>37.54</td>
    <td>31.04</td>
    <td>25.58</td>
    <td>69.67</td>
    <td>89.85</td>
    <td>40.50</td>
    <td>56.58</td>
    <td>51.01</td>
    <td>50.22</td>
  </tr>
  <tr>
    <td>LLaMA-3-8B-Instruct</td>
    <td>KIVI 34.88</td>
    <td>28.43</td>
    <td>24.78</td>
    <td>69.33</td>
    <td>89.57</td>
    <td>40.09</td>
    <td>44.42</td>
    <td>45.54</td>
    <td>47.13</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>36.75</b></td>
    <td><b>30.74</b></td>
    <td><b>24.94</b></td>
    <td><b>69.67</b></td>
    <td><b>89.74</b></td>
    <td><b>40.39</b></td>
    <td><b>52.37</b></td>
    <td><b>48.82</b></td>
    <td><b>49.18</b></td>
  </tr>
  <tr>
    <td>Fp16</td>
    <td>24.35</td>
    <td>33.05</td>
    <td>25.77</td>
    <td>67.00</td>
    <td>86.84</td>
    <td>40.95</td>
    <td>57.24</td>
    <td>49.84</td>
    <td>48.13</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.2</td>
    <td><b>KIVI</b> 24.20</td>
    <td>30.98</td>
    <td>25.10</td>
    <td>66.33</td>
    <td>85.40</td>
    <td>41.05</td>
    <td>55.70</td>
    <td>48.18</td>
    <td>47.12</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>23.78</td>
    <td><b>31.37</b></td>
    <td><b>25.35</b></td>
    <td><b>66.33</b></td>
    <td><b>86.18</b></td>
    <td><b>41.25</b></td>
    <td><b>55.89</b></td>
    <td><b>48.32</b></td>
    <td><b>47.31</b></td>
  </tr>
</table>

Table 2: Main results on LongBench. We report accuracy for TREC, Rouge-L for GovReport and SamSum, edit similarity (Levenshtein distance (Svyatkovskiy et al., 2020)) for LCC and RepoBenchP, and F1 score for the other tasks. **Bold**: the best results for each setting. *OTT* demonstrates superior performance on average.

![Three line plots showing throughput, memory usage by batch size, and memory usage by length for different methods](page_101_1012_1242_246.png)

Figure 3: Experiments on throughput and memory: (a) Comparison of throughput (tokens/s) for different methods across different batch sizes on NVIDIA A800 80G. (b) Peak memory usage (including model weights and other components) at different batch sizes on NVIDIA A800 80G. (c) Peak memory usage (including model weights and other components) at different sequence lengths when batch size = 1 on NVIDIA A100 40G. The results shows that *OTT* achieves a peak memory reduction of up to 6.4x and a throughput increase of 2.3x.

be retained, but it also improves accuracy. Thus, selecting an appropriate group size and residual length is critical to balancing memory usage and accuracy. We explore the impact of group size and residual length with group sizes of {32, 64, 128} and residual lengths of {0, 8, 16, 32, 64, 128}. Table 3 reports the results for LLaMA-3-8B-Instruct on Gsm8k 8-shot and 8-shot CoT under different configurations. When the group size is fixed, we observe a clear upward trend in accuracy as the residual length increases. However, when the residual length is fixed, the effect of group size shows no clear pattern, likely because the token distribution is relatively uniform, meaning that increasing group size has a limited impact. Since increasing the group size can improve the compression ratio (if not consider the group tokens), we tend to choose a larger group size. For our main experiments, we choose a group size of 128 and a residual length of 32 to balance performance and compression ratio.

The number of outlier tokens. We explore the effect of varying *outlier_num* from 0 to 6, keeping all other settings unchanged. Table 4 presents the results for LLaMA-3-8B-Instruct on Gsm8k (8-shot and 8-shot CoT). The results show that retaining even a single outlier token can significantly improve performance, but further increases in *outlier_num* yield diminishing returns, eventually plateauing performance. However, the increase in *outlier_num* may result in more memory overhead, leading to a decrease in compression ratio. Considering that a small *outlier_num* is already sufficient to significantly improve the accuracy, we set *outlier_num = 3* for our main experiments.

Outlier tokens in shallow layers. We observe that there are no outlier tokens in the shallow layers (see Figure 5 and Figure 6 in Appendix, the Keys in the shallow layers does not exhibit the characteristics discussed in Section 3.1), suggesting that *outlier_num* should be set to 0 in these layers.
<table>
  <tr>
    <th>G</th>
    <th>R</th>
    <th>Gsm8k(8)</th>
    <th>Gsm8k(8-CoT)</th>
    <th>G</th>
    <th>R</th>
    <th>Gsm8k(8)</th>
    <th>Gsm8k(8-CoT)</th>
    <th>G</th>
    <th>R</th>
    <th>Gsm8k(8)</th>
    <th>Gsm8k(8-CoT)</th>
  </tr>
  <tr>
    <td>32</td>
    <td>0</td>
    <td>70.05</td>
    <td>73.16</td>
    <td>64</td>
    <td>0</td>
    <td>68.92</td>
    <td>72.63</td>
    <td>128</td>
    <td>0</td>
    <td>70.96</td>
    <td>73.54</td>
  </tr>
  <tr>
    <td>32</td>
    <td>8</td>
    <td>71.95</td>
    <td>74.22</td>
    <td>64</td>
    <td>8</td>
    <td>70.05</td>
    <td>73.01</td>
    <td>128</td>
    <td>8</td>
    <td>72.51</td>
    <td>74.15</td>
  </tr>
  <tr>
    <td>32</td>
    <td>16</td>
    <td>72.78</td>
    <td>74.83</td>
    <td>64</td>
    <td>16</td>
    <td>70.89</td>
    <td>73.84</td>
    <td>128</td>
    <td>16</td>
    <td>72.93</td>
    <td>74.37</td>
  </tr>
  <tr>
    <td>32</td>
    <td>32</td>
    <td>72.78</td>
    <td>74.53</td>
    <td>64</td>
    <td>32</td>
    <td>72.40</td>
    <td>75.06</td>
    <td>128</td>
    <td>32</td>
    <td>72.55</td>
    <td>75.06</td>
  </tr>
  <tr>
    <td>32</td>
    <td>64</td>
    <td>74.00</td>
    <td>76.88</td>
    <td>64</td>
    <td>64</td>
    <td>73.69</td>
    <td>76.42</td>
    <td>128</td>
    <td>64</td>
    <td>73.24</td>
    <td>75.36</td>
  </tr>
  <tr>
    <td>32</td>
    <td>128</td>
    <td>73.77</td>
    <td>77.33</td>
    <td>64</td>
    <td>128</td>
    <td>74.68</td>
    <td>76.42</td>
    <td>128</td>
    <td>128</td>
    <td>73.24</td>
    <td>76.65</td>
  </tr>
</table>

Table 3: Results of different G and R. The settings in the main experiment are indicated with underlines.

<table>
  <tr>
    <th>outlier_num</th>
    <th>Gsm8k(8)</th>
    <th>Gsm8k(8-CoT)</th>
    <th>Layers</th>
    <th>Gsm8k(8)</th>
    <th>Gsm8k(8-CoT)</th>
  </tr>
  <tr>
    <td>0</td>
    <td>62.09</td>
    <td>68.31</td>
    <td>None</td>
    <td>72.48</td>
    <td>75.59</td>
  </tr>
  <tr>
    <td>1</td>
    <td>71.80</td>
    <td>75.74</td>
    <td>0</td>
    <td>72.78</td>
    <td>75.44</td>
  </tr>
  <tr>
    <td>2</td>
    <td>71.57</td>
    <td>75.06</td>
    <td>0.1</td>
    <td>72.55</td>
    <td>75.06</td>
  </tr>
  <tr>
    <td>3</td>
    <td>72.55</td>
    <td>75.06</td>
    <td>0 ~ 2</td>
    <td>71.49</td>
    <td>74.68</td>
  </tr>
  <tr>
    <td>4</td>
    <td>72.25</td>
    <td>75.97</td>
    <td>0 ~ 3</td>
    <td>71.80</td>
    <td>73.64</td>
  </tr>
  <tr>
    <td>5</td>
    <td>72.18</td>
    <td>75.74</td>
    <td>0 ~ 4</td>
    <td>70.96</td>
    <td>74.53</td>
  </tr>
  <tr>
    <td>6</td>
    <td>72.18</td>
    <td>75.89</td>
    <td>0 ~ 5</td>
    <td>69.60</td>
    <td>74.00</td>
  </tr>
</table>

Table 4: Ablation study of outlier_num. The settings in the main experiment are indicated with underlines. Left: results on Gsm8k with different outlier_num. Right: results on Gsm8k with outlier_num = 0 in shallow layers.

To explore this further, we set outlier_num to 0 in consecutive shallow layers and evaluate the performance on Gsm8k (8-shot and 8-shot CoT) using LLaMA-3-8B-Instruct. For example, “0 ~ 2” means that outlier_num is set to 0 for the first three layers of the model. Table 4 shows that the impact is minimal in the shallowest layers but becomes more significant as we move deeper into the model. Based on these results, we set outlier_num = 0 for the first two layers in all models for our main experiments.

5 Related work

Efficient Inference of LLMs. Large Language Models often have enormous parameters, leading to significant computational costs on inference. To address this, some researchers have employed parameter pruning techniques to eliminate redundant or less important parameters, thereby compressing LLMs (Ma et al., 2023; Xia et al., 2024; Frantar and Alistarh, 2023). Other studies have focused on quantizing model weights, reducing their size and the number of arithmetic operations required for inference. For example, GPTQ (Frantar et al., 2022) uses second-order information to quantize models to 3 or 4-bit precision while maintaining accuracy. AWQ (Lin et al., 2024) preserves critical weights based on the activation distribution, quantizing the remaining weights to lower bit precision. These methods can be combined with KV Cache compression to achieve a better performance.

KV Cache Compression. KV Cache compression can significantly reduce the size of KV Cache with minimal accuracy loss. Liu et al. (2024b) find that some outlier channels in the Keys have very large magnitudes, resulting in a significant loss. Hooper et al. (2024) find that quantizing the Key cache before applying rotary positional embeddings reduces the negative impact of quantization. Xiao et al. (2024) propose StreamingLLM, which retains the initial and final tokens of the input. Similarly, Sun et al. (2024) find a "massive activations" pattern in LLMs, where a few activations have much higher values than others. These values stay stable across inputs and act as critical bias terms in the model. Zhang et al. (2023) find that only a minority of tokens influence the output.

6 Conclusion

In this paper, we start from the assumptions of KIVI and further explore the distribution of the Keys in the outlier channels. We observe that a few outlier tokens deviate from the assumptions of KIVI. Quantizing these tokens has detrimental effects, as it increases the quantization errors of other tokens. Building on these observations, we propose KV Cache Quantization with Outlier Tokens Tracing (OTT), which leverages the magnitude of the Keys to dynamically trace these tokens during decoding, excluding them from the quantization process while retaining their full-precision representations. Extensive experiments show that our
method achieves significant improvements in accuracy, along with substantial reductions in memory usage and increases in throughput.

Limitations

Although OTT has achieved excellent results, there are still some limitations:

• Due to the presence of the group and nearest tokens, we cannot ensure that all tokens are quantized at every moment. When the sequence length is very short and the batch size is very large, the compression ratio of OTT is reduced. In extreme cases, when the sequence length is shorter than the group size, OTT does not perform any compression.

• OTT occasionally still incurs a little loss on specific datasets under 2-bit quantization. This may be related to the difficulty of the datasets and the required generation length. When the generation length is very long, OTT may face an unacceptable risk of loss due to error accumulation.

Acknowledgements

We want to thank all the anonymous reviewers for their valuable comments. This work was supported by the National Science Foundation of China (NSFC No.62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and the Priority Academic Program Development of Jiangsu Higher Education Institutions.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901.

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15. IEEE.

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quartor: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024a. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3119–3137. Association for Computational Linguistics.

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazhen Xu, Lei Hou, Yuxiao Dong, et al. 2024b. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204.

William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. 2024. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981.

Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, and Kai-Chiang Wu. 2024. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332.
Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. 2024. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR.

Elias Frantar, Saleh Ashkboos, Torsten Hoefer, and Dan Alistarh. 2022. GPTQ: accurate post-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach', Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive kv cache compression for llms. In The Twelfth International Conference on Learning Representations.

Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. 2024. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:2405.14256.

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. CoRR, abs/2401.18079.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2704–2713. Computer Vision Foundation / IEEE Computer Society.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.

Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christopher Ré, and Azalia Mirhoseini. 2024. Hydragen: High-throughput llm inference with shared prefixes. arXiv preprint arXiv:2402.05099.

Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. 2024. GEAR: an efficient KV cache compression recipe for near-lossless generative inference of LLM. CoRR, abs/2403.05527.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Liannin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Aycr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Clara, CA, USA, May 13-16, 2024. mlsys.org.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhao Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024b. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Xinyin Ma, Gongfan Fang, and Xincho Wang. 2023. Lim-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Qiaozhi He, Murun Yang, Tong Xiao, and Jingbo Zhu. 2024. Cross-layer attention sharing for large language models. arXiv preprint arXiv:2408.01890.

Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118–132. IEEE.

Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150.

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094–31116. PMLR.

Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13003–13051. Association for Computational Linguistics.

Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellilcode compose: Code generation using transformer. In Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pages 1433–1443.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Nikolai Trukhnov and Ilya Soloveychik. 2024. Accurate block quantization in llms with outliers. arXiv preprint arXiv:2403.20137.

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Y Wang, D Ma, and D Cai. 2024. With greater text comes greater necessity: Inference-time training helps long text generation. arXiv preprint arXiv:2401.11504.

Haoyi Wu and Kewei Tu. 2024. Layer-condensed kv cache for efficient inference of large language models. arXiv preprint arXiv:2405.10637.

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2024. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694.

Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. 2024. Effectively compress kv heads for llm. arXiv preprint arXiv:2406.07056.

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Lei Zhu, Xinjiang Wang, Wayne Zhang, and Rynson WH Lau. 2024. Relayattention for efficient large language model serving with long system prompts. arXiv preprint arXiv:2402.14808.

Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, and Alham Fikri Aji. 2024. Mlkv: Multi-layer key-value heads for memory efficient transformer decoding. arXiv preprint arXiv:2406.09297.
A Additional Figures and Tables

<table>
  <tr>
    <th>Input:</th>
    <td>
Which methods did Socrates employ to challenge the prevailing thoughts of his time?<br>
</td>
  </tr>
  <tr>
    <th>Output:</th>
    <td>
Which methods did Socrates employ to challenge the prevailing thoughts of his time?<br>
 Socrates employed several methods to challenge the prevailing thoughts of his time, including:
      <ol>
        <li>Questioning: Socrates used questioning as a primary method to challenge the beliefs of others. He would ask probing questions to expose the contradictions and inconsistencies in their arguments, and to lead them to a deeper understanding of their own beliefs.</li>
        <li>Dialogue: Socrates believed in the power of dialogue to promote critical thinking and understanding. He would engage in conversations with others, using his questions and their responses to challenge their beliefs and to encourage them to think more deeply about their ideas.</li>
        <li>Irony: Socrates often used irony to challenge the beliefs of others. By using irony, he could expose the flaws in their arguments and lead them to see things from a different perspective.</li>
        <li>Paradox: Socrates often used paradoxical statements to challenge the prevailing beliefs of his time. By presenting seemingly contradictory statements, he could encourage his interlocutors to think more critically about their beliefs and to question their assumptions.</li>
        <li>Maieutics: Socrates believed in the power of midwifery, or maieutics, to bring forth new ideas and insights. He would use his questions and dialogue to help others discover new truths and to challenge their existing beliefs.</li>
        <li>Analogy: Socrates often used analogies to help others understand complex ideas and to challenge their beliefs. By using comparisons and contrasts, he could help his interlocutors to see things from a different perspective and to develop a deeper understanding of the issues at hand.</li>
        <li>Refutation: Socrates was skilled at refuting the arguments of others by exposing their flaws and inconsistencies. He would use his questions and dialogue to challenge the beliefs of others and to lead them to a deeper understanding of the issues at hand.</li>
        <li>Affirmation: Socrates also used affirmation to challenge the prevailing thoughts of his time. By affirming the beliefs of others and building on them, he could encourage them to think more deeply about their ideas and to develop a more nuanced understanding of the issues at hand.</li>
        <li>Silence: Socrates believed that sometimes the best way to challenge the prevailing thoughts of his time was through silence. By remaining silent in the face of foolish or false arguments, he could encourage others to think more critically about their beliefs and to question their assumptions.</li>
      </ol>
      Overall, Socrates employed a wide range of methods to challenge the prevailing thoughts of his time, including questioning, dialogue, irony, paradox, maieutics, analogy, refutation, affirmation, and silence. Through these methods, he was able to encourage critical thinking and to promote a deeper understanding of the issues at hand.
    </td>
  </tr>
</table>

Table 5: Example generated by LLaMA-2-7B-chat-hf.
Figure 4: The Keys in an outlier channel (up) and the magnitude of the Keys overall (down).

Layer 0 key cache        Layer 0 value cache        Layer 10 key cache        Layer 10 value cache
Layer 20 key cache        Layer 20 value cache        Layer 31 key cache        Layer 31 value cache

Figure 5: Magnitude of the keys and Values for Llama-2-7B-chat-hf in head 17.

Figure 6: The Keys in an outlier channel (up) and the sorted Keys in an outlier channel (down).
B Additional Experiment Results

B.1 Experiments on LLaMA-2-70b-chat-hf

To validate the performance on larger models, we conduct additional experiments on LLaMA-2-70b-chat-hf. The experimental setup is completely consistent with the main experiment. The result in Table 6 shows that \( OTT \) can still achieve higher accuracy advantages on larger models based on KIVI.

<table>
  <tr>
    <th>70b-chat-hf</th>
    <th>Gsm8k(8)</th>
    <th>Gsm8k(8-cost)</th>
    <th>Gsm8k(9-cost)</th>
    <th>BBH(3)</th>
    <th>HE(p@1)</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>56.03</td>
    <td>55.04</td>
    <td>48.98</td>
    <td>47.09</td>
    <td>16.46</td>
    <td>44.72</td>
  </tr>
  <tr>
    <td>KIVI</td>
    <td>51.63</td>
    <td>50.49</td>
    <td>46.40</td>
    <td>46.08</td>
    <td>14.02</td>
    <td>41.72</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>52.92</td>
    <td>52.54</td>
    <td>49.05</td>
    <td>46.48</td>
    <td>15.85</td>
    <td>43.37</td>
  </tr>
</table>

Table 6: Experiments on LLaMA-2-70b-chat-hf

B.2 Comparison with token eviction methods

We add some comparisons with the token eviction methods. The previous token eviction methods are mostly evaluated on LongBench, so we also conduct experiments on LongBench using LLaMA-2-7b-chat-hf. The input length of LongBench is relatively long, while the output length is relatively short, which may be more conducive to the performance of the token eviction methods. The baselines include StreamingLLM (Xiao et al., 2024), H2O (Zhang et al., 2023), and SnapKV (Li et al., 2024). In order to maintain the simplicity and consistency of the settings for comparison, we only perform token eviction in the prefill stage, and retain all KV caches in the decode stage. In addition, we make some adjustments to H2O based on SnapKV’s strategy, selecting only the queries in the sliding window for attention score selection (which was later verified to be superior to H2O’s strategy). In order to maintain the overall compression ratio consistent with \( OTT \), we choose to evict 84% of the tokens in the prefill stage, which have the closest compression ratio to \( OTT \). For H2O, the number of recent tokens and heavy hitters is the same. For StreamingLLM, we do not adjust its position id during decoding phase. So, the process of token eviction is as follows:

• In the prefill stage, use queries in the sliding window to calculate the attention score with other tokens, and perform token eviction according to the strategies of StreamingLLM, H2O, and SnapKV respectively.

• During the decode phase, attention calculation is performed directly without token eviction.

The results are shown in Table 7 Among these methods, SnapKV achieves the best results. But even under more favorable settings, the result is still slightly lower than \( OTT \).

B.3 Comparison with ZipCache

We compare \( OTT \) with ZipCache (He et al., 2024), and in order to maintain consistent compression rates, we set 20% of the tokens to 4-bit quantization and 80% to 2-bit quantization. The other hyperparameters in ZipCache are the same. We conduct our experiments on GSM8K, BBH and HumanEval with LLaMA-2-7b-chat-hf. The results in Table 8 show that ZipCache is weaker than KIVI and \( OTT \).

C Additional Time Analysis

We provide a more detailed analysis of the time cost. The computational overhead comes from two aspects. In the compression stage, we calculate the magnitude of each token’s key, perform comparison, select the index, and quantify it. The cost of the outlier operation is relatively high compared to quantization, but the compression is only performed every \( G \) steps, so this time cost can be almost negligible compared to the whole decoding process. In the attention calculation stage, we need to calculate the qkv in the outlier pool and cover the attention score according to the outlier token index, which has a certain cost. We plot the detailed time consumption in the attention block in Figure 7, and the outlier operation accounts for about 18%. Considering the pre-processing, post-processing and FFN calculation in the entire forward step, the time proportion of outlier operations is very small.

![Pie chart showing time proportion in the attention block: 55%, 18%, 13%, 14%](page_1012_1342_410_246.png)

Figure 7: The time proportion in the attention block.
<table>
  <tr>
    <th>LLaMA-2-7b-chat</th>
    <th>Qasper</th>
    <th>GovReport</th>
    <th>MultiNews</th>
    <th>TREC</th>
    <th>TriviaQA</th>
    <th>SamSum</th>
    <th>LCC</th>
    <th>Repobench-P</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>20.04</td>
    <td>25.08</td>
    <td>23.02</td>
    <td>59.67</td>
    <td>85.39</td>
    <td>39.28</td>
    <td>59.59</td>
    <td>48.04</td>
    <td>45.01</td>
  </tr>
  <tr>
    <td>KIVI</td>
    <td><b>20.43</b></td>
    <td>19.97</td>
    <td>19.82</td>
    <td>59.67</td>
    <td><b>85.16</b></td>
    <td>37.7</td>
    <td>58.73</td>
    <td>47.24</td>
    <td>43.59</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>18.96</td>
    <td>18.73</td>
    <td>19.64</td>
    <td>59</td>
    <td>84.84</td>
    <td>38.22</td>
    <td><b>60.5</b></td>
    <td><b>50.08</b></td>
    <td>43.75</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>17.51</td>
    <td>18.85</td>
    <td>19.88</td>
    <td>50</td>
    <td>84.22</td>
    <td>38.09</td>
    <td>58.23</td>
    <td>49.66</td>
    <td>42.05</td>
  </tr>
  <tr>
    <td>Streaming</td>
    <td>15.31</td>
    <td>19.39</td>
    <td>18.99</td>
    <td>51</td>
    <td>83.11</td>
    <td>36.8</td>
    <td>57.57</td>
    <td>47.33</td>
    <td>41.19</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>19.95</td>
    <td><b>21.56</b></td>
    <td><b>20.81</b></td>
    <td><b>59.67</b></td>
    <td>85</td>
    <td><b>39.1</b></td>
    <td>59.44</td>
    <td>48.51</td>
    <td><b>44.26</b></td>
  </tr>
</table>

Table 7: Experiments on three additional eviction-based methods on LLaMA-2-7b-chat-hf.

<table>
  <tr>
    <th>LLaMA-2-7b-chat</th>
    <th>Gsm8k(8)</th>
    <th>8-cot</th>
    <th>0-cot</th>
    <th>BBH(3)</th>
    <th>3-cot</th>
    <th>0-cot</th>
    <th>HE(p@1)</th>
    <th>p@10</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>21.99</td>
    <td>21.3</td>
    <td>24.11</td>
    <td>33.34</td>
    <td>40.21</td>
    <td>35.00</td>
    <td>12.19</td>
    <td>17.07</td>
    <td>45.01</td>
  </tr>
  <tr>
    <td>KIVI</td>
    <td>16.3</td>
    <td>17.51</td>
    <td>21.61</td>
    <td>32.48</td>
    <td>34</td>
    <td>33.30</td>
    <td>9.75</td>
    <td>12.19</td>
    <td>43.59</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>19.86</b></td>
    <td><b>19.33</b></td>
    <td><b>22.52</b></td>
    <td><b>33.33</b></td>
    <td><b>34.43</b></td>
    <td><b>33.74</b></td>
    <td><b>11.58</b></td>
    <td><b>14.63</b></td>
    <td><b>43.75</b></td>
  </tr>
  <tr>
    <td>ZipCache</td>
    <td>15.92</td>
    <td>17.74</td>
    <td>20.02</td>
    <td>32.85</td>
    <td>33.9</td>
    <td>32.35</td>
    <td>9.45</td>
    <td>15.24</td>
    <td>42.05</td>
  </tr>
</table>

Table 8: Experiments on ZipCache.

D Proof of Low-Magnitude Keys Disrupting Attention Weights

We formalize the claim that low-magnitude keys in outlier channels disrupt attention weights through two steps:

D.1 Quantization Error

• Definitions:
  – Let \( K_c \in \mathbb{R}^n \) be an outlier channel containing \( n \) Key values where \( \exists \) a subset \( S \subset \{1, \ldots, n\} \) with \( |S| = m \ll n \) such that:
    \[
    \begin{cases}
      K_{c,i} \in [\mu - \sigma, \mu + \sigma], \forall i \notin S \\
      (\text{uniform distribution}) \\
      K_{c,j} \in [\epsilon, \delta], \forall j \in S \\
      \text{where } 0 < \epsilon \ll \mu - \sigma
    \end{cases}
    \]
  • Quantization Parameters:
    – Full range: \( X_{\max} = \max(K_c),\ X_{\min} = \min(K_c) \)
    – Quantization step: \( q = \frac{X_{\max} - X_{\min}}{2^b - 1} \)
  • Key Observation: The presence of low-magnitude outliers forces:
    \[
    X_{\min} \leq \epsilon \ll \mu - \sigma \quad \text{and} \quad X_{\max} \geq \mu + \sigma
    \]
    \[
    \implies q = \frac{(\mu + \sigma) - \epsilon}{2^b - 1} \gg \frac{2\sigma}{2^b - 1}
    \]
  • Result: Low-magnitude outliers inflate quantization step size, leading to larger approximation errors for all tokens in the channel.

D.2 Error Propagation to Attention Weights

• Attention Score Calculation: For query vector \( Q \in \mathbb{R}^d \) and quantized Key matrix \( K' \):
  \[
  A_i = \frac{Q K'_i}{\sqrt{d}} \quad \text{and} \quad A_i^{\text{quant}} = A_i + \frac{Q (K'_i - K_i)}{\sqrt{d}}
  \]
  \[
  \Delta A_i = \frac{Q (K'_i - K_i)}{\sqrt{d}}
  \]

• Error Analysis:
  – For outlier channel \( c \):
    \[
    \mathbb{E}[|K'_{c,i} - K_{c,i}|] \propto q
    \]
    \[
    \Delta A_i \sim \sum_{c=1}^d Q_c (K'_{c,i} - K_{c,i})
    \]
  • Key Observation: In outlier channels where \( |Q_c| \) is typically large (by definition of being “outlier channels”), the quantization error gets amplified by:
    \[
    \Delta A_i \propto Q_c (K'_{c,i} - K_{c,i}) \approx Q_c \cdot q
    \]
  • Result: The error in quantization steps propagates to attention weights.

E Mathematical Formulation of OTT

Outlier Token Identification For a token \( t_i \) with Key vector \( K_i \), its outlier score \( S_i \) is computed as the magnitude of its Keys, typically measured via the \( L_1 \)-norm:
\[
S_i = \|K_i\|
\]
Tokens with smaller \( S_i \) are identified as outliers since their Keys deviate significantly from the uniform distribution in outlier channels.
Competition Mechanism  At each quantization step (every \( G \) tokens), tokens in the current group \( \mathcal{T} = \{t_1, t_2, \ldots, t_G\} \) compete with the existing outlier pool \( \mathcal{O} \) (capacity \( N \)) for inclusion. The process involves:

1. **Score Calculation:** Compute \( S_i \) for all tokens in \( \mathcal{T} \) and \( \mathcal{O} \).

2. **Token Ranking:** Combine \( \mathcal{T} \) and \( \mathcal{O} \), then sort all tokens by \( S_i \) in ascending order:

   \[
   \text{Sorted List} = \operatorname{argsort}(S_i), \quad \forall t_i \in \mathcal{T} \cup \mathcal{O}
   \]

3. **Outlier Pool Update:** Select the top-\( N \) tokens with the smallest \( S_i \) to form the new outlier pool:

   \[
   \mathcal{O}_{\text{new}} = \{t_j \mid j \in \text{top-}N \text{ indices of Sorted List}\}
   \]

4. **Replacement Handling:** Tokens evicted from \( \mathcal{O} \) (if \( |\mathcal{T} \cup \mathcal{O}| > N \)) are stored in an auxiliary pool or discarded. The retained outlier tokens are excluded from quantization and stored in full precision.

Mathematical Formulation  Let \( \mathcal{O}^{(t)} \) denote the outlier pool at step \( t \), and \( \mathcal{T}^{(t)} \) the current token group. The update rule is:

\[
\mathcal{O}^{(t+1)} = \arg\min_{\text{top-}N \text{ tokens by } S_i} \left( \mathcal{O}^{(t)} \cup \mathcal{T}^{(t)} \right)
\]

Outlier tokens are excluded from quantization, while non-outliers are quantized using channel-wise (Keys) and token-wise (Values) methods as in KIVI.

F Statistical Analysis of Preliminary Results

We conduct additional analysis experiments on LongBench using LLaMA-2-7b-chat-hf. We record the keys of Layer 10, Head 16 of the first 1024 tokens during the generation process and analyze the distribution of the outlier channel (i.e., the channel with the largest magnitude) among these keys. Specifically, for each example, we identify the outlier channel of the key in Layer 10, Head 16, then divide the entire range of channel values into ten equal parts and record which range each token’s value fall into. Finally, we average the results across the entire dataset. The results are shown in Table 9, which confirms our previous hypothesis.

G Additional Benchmarks and Baselines

To validate the effectiveness of OTT, we also add other baselines and benchmarks. Among them, the benchmarks include Needle-in-a-Haystack and Ruler, while the baselines include ZipCache and Gear.

**RULER** (Hsieh et al., 2024): This benchmark evaluates models’ ability to handle complex reasoning tasks. It involves tasks that require understanding and linking various pieces of information, making it essential for assessing skills in multi-step reasoning and logical analysis.

**Needle-in-a-Haystack** (Liu et al., 2024a): This benchmark focuses on testing if models can find important details in long texts. It checks how well models can spot useful information in a lot of text, which is key for tasks like finding facts or answering questions by pulling out parts of the text.

**GEAR** (Kang et al., 2024) compensates for compression-induced errors by combining low-rank and sparse matrices, achieving near-lossless results in 2-bit quantization integrated with KIVI (Liu et al., 2024b).

**ZipCache** (He et al., 2024) achieves accurate KV cache compression by introducing a channel-separable tokenwise quantization scheme, an improved salient token identification metric based on normalized attention scores, and an efficient approximation method for fast attention implementations.

We adjust the hyper-parameters of various methods to thoroughly observe their performance. For KIVI and OTT, we vary the group size \( G \) and the residual length \( R \). For ZipCache, we vary the k unimportant ratio \( k \) and v unimportant ratio \( v \). The unimportant tokens are stored in 2-bit and the important ones are stored in 4-bit. For Gear, we set the low rank \( r \) to 2, group size \( G \) to 128, streaming gap to 100, outlier ratio to 0.01. Note that Gear will use much more memory and is much slower than KIVI and OTT because it add additional low-rank and outlier operations based on KIVI.

G.1 Results on Needle-in-a-Haystack

Figure 9 and 10 shows the results of different methods and models on Needle-in-a-Haystack. The results on LLaMA-3-8B-Instruct shows that all methods can perform well under this setting. The results on LLaMA-2-7B-chat-hf show that Gear performs the best across all methods, while it sacrifices memory and throughput. OTT performs better
than KIVI (increasing the accuracy from 93.1% to 99.2%). ZipCache performs the worst.

G.2 Results on Ruler
We validate the effectiveness of various methods on Ruler. The results are shown in Table 10. Similar to previous findings, Gear still achieves the best accuracy among all methods, thanks to its higher computational cost and memory usage. OTT achieves better accuracy than KIVI when \( G = 32 \) and \( G = 128 \), demonstrating the effectiveness of our method. ZipCache also achieve the worst results, with significant losses on both models.

G.3 Full Results on LongBench
We test the performance of each method on LongBench under more settings and complete all the results for LongBench. The results are shown in Table 11. OTT achieves almost no loss on LongBench, performing nearly as well as Gear, and clearly outperforming ZipCache and KIVI.

G.4 Results on Helmet
We test the performance of each method on Helmet (Yen et al., 2024) benchmark using LLaMA-3-8B-Instruct. We set the max length to 8192. The results are shown in Table 12. The results show that our method performs better than the baselines under fair comparison.

G.5 Results on LongBench-v2
We test the performance of each method on LongBench-v2 (Bai et al., 2024b) using LLaMA-3-8B-Instruct. We set the max length to 8192. The results are shown in Table 13. The results show that our method performs better than the baselines under fair comparison.

G.6 Results on Longer Models
We test the performance of each method on Ruler with Llama-3-8B-ProLong-512k-Instruct and longer context lengths (64k, 128k), the results are shown in Table 14. We can conclude from the tables that our method can perform well on extreme-long scenarios.

G.7 Throughput and Memory Analysis
To fully demonstrate the memory compression and throughput of different methods, we conduct additional experiments on memory usage and throughput. We use LLaMA-2-7B-chat-hf with an input length of 64, output length of 384, and batch size of 128 to carry out our experiments on an NVIDIA A100 40GB GPU. We record the throughput and memory peak for each method. The results are shown in Figure 8. We omit Gear because its codebase only supports fake compression, making it impossible to measure its actual memory usage and throughput. Although Gear supports some true compression, it does not handle outliers in its true compression, which is inconsistent with the settings used in our experiments above. From the figure, we can conclude that OTT has slightly higher throughput than KIVI, likely because its handling of residual tokens is simpler than that of KIVI. Additionally, both OTT and KIVI show significantly higher throughput than ZipCache. In terms of memory usage, OTT consumes slightly more memory than KIVI, primarily because it needs to store more tokens. This difference may become more pronounced as the batch size increases. On the other hand, ZipCache uses the least GPU memory, indicating that it has a higher compression ratio.
<table>
  <tr>
    <th>Dataset</th>
    <th>0%-10%</th>
    <th>10%-20%</th>
    <th>20%-30%</th>
    <th>30%-40%</th>
    <th>40%-50%</th>
    <th>50%-60%</th>
    <th>60%-70%</th>
    <th>70%-80%</th>
    <th>80%-90%</th>
    <th>90%-100%</th>
  </tr>
  <tr><td>qasper</td><td>0.20</td><td>0.35</td><td>0.18</td><td>0.02</td><td>0.62</td><td>8.53</td><td>29.88</td><td>37.29</td><td>19.44</td><td>3.49</td></tr>
  <tr><td>triviaqa</td><td>0.20</td><td>0.36</td><td>0.16</td><td>0.01</td><td>0.31</td><td>6.42</td><td>27.36</td><td>38.78</td><td>22.23</td><td>4.17</td></tr>
  <tr><td>trec</td><td>0.21</td><td>0.30</td><td>0.07</td><td>0.01</td><td>0.13</td><td>1.90</td><td>20.80</td><td>46.38</td><td>24.42</td><td>5.79</td></tr>
  <tr><td>samsam</td><td>0.20</td><td>0.41</td><td>0.18</td><td>0.01</td><td>0.18</td><td>4.33</td><td>25.38</td><td>38.52</td><td>25.36</td><td>5.24</td></tr>
  <tr><td>lcc</td><td>0.20</td><td>0.31</td><td>0.18</td><td>0.05</td><td>0.63</td><td>7.03</td><td>25.32</td><td>36.57</td><td>24.85</td><td>4.85</td></tr>
  <tr><td>repobench-p</td><td>0.20</td><td>0.34</td><td>0.14</td><td>0.04</td><td>0.58</td><td>7.00</td><td>27.65</td><td>38.84</td><td>21.22</td><td>3.99</td></tr>
  <tr><td>mult_nerws</td><td>0.20</td><td>0.37</td><td>0.18</td><td>0.01</td><td>0.45</td><td>7.44</td><td>25.99</td><td>36.61</td><td>20.78</td><td>4.70</td></tr>
  <tr><td>multiNliQA_en</td><td>0.20</td><td>0.37</td><td>0.21</td><td>0.02</td><td>0.71</td><td>8.17</td><td>28.25</td><td>37.12</td><td>20.93</td><td>4.01</td></tr>
  <tr><td>hotpotqa</td><td>0.20</td><td>0.38</td><td>0.16</td><td>0.01</td><td>0.39</td><td>7.62</td><td>30.12</td><td>40.05</td><td>18.62</td><td>2.45</td></tr>
  <tr><td>2wikimqa</td><td>0.20</td><td>0.38</td><td>0.17</td><td>0.02</td><td>0.33</td><td>6.68</td><td>28.88</td><td>40.49</td><td>20.06</td><td>2.81</td></tr>
  <tr><td>gov_report</td><td>0.20</td><td>0.35</td><td>0.29</td><td>0.02</td><td>0.48</td><td>6.71</td><td>27.46</td><td>39.02</td><td>21.63</td><td>3.84</td></tr>
  <tr><td>passage_count</td><td>0.20</td><td>0.36</td><td>0.16</td><td>0.01</td><td>0.44</td><td>6.62</td><td>28.51</td><td>38.87</td><td>20.91</td><td>3.92</td></tr>
  <tr><td>passage_retrieval_en</td><td>0.20</td><td>0.38</td><td>0.17</td><td>0.01</td><td>0.60</td><td>7.17</td><td>28.39</td><td>38.01</td><td>21.11</td><td>3.97</td></tr>
</table>

Table 9: Statistical analysis of outlier distribution.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="6">Single NIAH</th>
    <th colspan="4">Multi-key NIAH</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>S-NIAH-1</th>
    <th>S-NIAH-2</th>
    <th>S-NIAH-3</th>
    <th>MK-NIAH-1</th>
    <th>MK-NIAH-2</th>
    <th>MK-NIAH-3</th>
    <th>MQ-NIAH</th>
    <th>MV-NIAH</th>
    <th>CWE</th>
    <th>FWE</th>
    <th>VT</th>
  </tr>
  <tr>
    <td colspan="12">LLaMA-2-7B-chat-hf</td>
  </tr>
  <tr>
    <td>FP16</td><td>100.00</td><td>92.80</td><td>90.00</td><td>84.00</td><td>67.40</td><td>52.80</td><td>76.85</td><td>80.45</td><td>83.72</td><td>80.67</td><td>92.12</td><td>81.89</td>
  </tr>
  <tr>
    <td>GEAR</td><td>47.40</td><td>42.20</td><td>42.40</td><td>43.60</td><td>36.00</td><td>19.60</td><td>41.05</td><td>43.10</td><td>53.86</td><td>75.13</td><td>49.60</td><td>44.90</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.7,v=0.8)</td><td>39.60</td><td>27.80</td><td>10.40</td><td>23.60</td><td>5.00</td><td>0.00</td><td>24.50</td><td>18.95</td><td>52.02</td><td>68.73</td><td>44.48</td><td>28.64</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.6,v=0.6)</td><td>44.70</td><td>31.40</td><td>17.40</td><td>31.40</td><td>6.40</td><td>0.00</td><td>30.95</td><td>28.15</td><td>52.98</td><td>70.44</td><td>44.04</td><td>32.32</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.5,v=0.5)</td><td>43.00</td><td>35.80</td><td>16.60</td><td>36.20</td><td>9.20</td><td>0.20</td><td>34.49</td><td>31.70</td><td>58.93</td><td>67.33</td><td>44.77</td><td>34.77</td>
  </tr>
  <tr>
    <td>KIVI(G=128,R=128)</td><td>47.00</td><td>39.40</td><td>26.80</td><td>41.40</td><td>13.20</td><td>0.20</td><td>37.05</td><td>38.40</td><td>52.76</td><td>71.27</td><td>45.84</td><td>37.57</td>
  </tr>
  <tr>
    <td>KIVI(G=32,R=128)</td><td>46.20</td><td>42.20</td><td>36.60</td><td>41.40</td><td>25.20</td><td>3.80</td><td>41.55</td><td>42.20</td><td>58.94</td><td>73.27</td><td>47.40</td><td>41.71</td>
  </tr>
  <tr>
    <td>Ours(G=128,R=32)</td><td>46.80</td><td>39.60</td><td>28.60</td><td>41.40</td><td>18.40</td><td>0.20</td><td>38.75</td><td>39.75</td><td>52.04</td><td>72.27</td><td>46.60</td><td>38.58</td>
  </tr>
  <tr>
    <td>Ours(G=32,R=128)</td><td>46.40</td><td>42.00</td><td>38.40</td><td>43.00</td><td>27.20</td><td>5.40</td><td>41.35</td><td>41.95</td><td>55.16</td><td>74.13</td><td>48.76</td><td>42.16</td>
  </tr>
  <tr>
    <td colspan="12">LLaMA-3-8B-Instruct</td>
  </tr>
  <tr>
    <td>FP16</td><td>100.00</td><td>98.20</td><td>97.00</td><td>99.20</td><td>91.60</td><td>95.80</td><td>99.75</td><td>97.45</td><td>97.82</td><td>82.27</td><td>98.28</td><td>96.12</td>
  </tr>
  <tr>
    <td>GEAR</td><td>100.00</td><td>98.20</td><td>97.00</td><td>99.20</td><td>91.80</td><td>87.80</td><td>99.70</td><td>96.80</td><td>97.82</td><td>81.53</td><td>98.24</td><td>95.28</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.7,v=0.8)</td><td>99.80</td><td>97.00</td><td>77.80</td><td>91.60</td><td>69.20</td><td>12.40</td><td>94.45</td><td>94.40</td><td>96.82</td><td>81.73</td><td>95.64</td><td>82.80</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.6,v=0.6)</td><td>99.60</td><td>97.80</td><td>82.00</td><td>93.80</td><td>73.00</td><td>19.80</td><td>97.45</td><td>96.05</td><td>98.62</td><td>84.27</td><td>96.16</td><td>85.00</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.5,v=0.5)</td><td>99.80</td><td>97.40</td><td>84.30</td><td>96.80</td><td>78.20</td><td>31.20</td><td>98.65</td><td>97.55</td><td>96.98</td><td>82.53</td><td>96.60</td><td>87.19</td>
  </tr>
  <tr>
    <td>KIVI(G=128,R=128)</td><td>96.00</td><td>97.80</td><td>88.60</td><td>96.00</td><td>75.20</td><td>11.00</td><td>95.30</td><td>94.90</td><td>96.80</td><td>86.27</td><td>88.64</td><td>82.79</td>
  </tr>
  <tr>
    <td>KIVI(G=32,R=128)</td><td>100.00</td><td>97.40</td><td>95.40</td><td>97.80</td><td>87.60</td><td>62.40</td><td>98.80</td><td>98.60</td><td>95.26</td><td>82.20</td><td>96.20</td><td>91.97</td>
  </tr>
  <tr>
    <td>Ours(G=128,R=32)</td><td>99.80</td><td>97.20</td><td>92.20</td><td>96.20</td><td>79.40</td><td>27.60</td><td>96.80</td><td>95.45</td><td>95.38</td><td>80.47</td><td>94.76</td><td>86.84</td>
  </tr>
  <tr>
    <td>Ours(G=32,R=128)</td><td>100.00</td><td>97.60</td><td>96.20</td><td>97.80</td><td>86.00</td><td>69.60</td><td>98.90</td><td>97.85</td><td>97.18</td><td>82.20</td><td>95.80</td><td>92.64</td>
  </tr>
</table>

Table 10: Performance comparison of different methods on RULER for LLaMA-2-7B-chat-hf and LLaMA-3-8B-Instruct. Bold text represents the best performance.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Single-Document QA</th>
    <th colspan="4">Multi-Document QA</th>
    <th colspan="3">Summarization</th>
    <th colspan="4">Few-shot Learning</th>
    <th colspan="5">Synthetic</th>
    <th colspan="3">Code</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>NFE-en</th>
    <th>Quasar</th>
    <th>HopQNA</th>
    <th>2WuMQA</th>
    <th>GovReport</th>
    <th>MultiNews</th>
    <th>TREC</th>
    <th>TriviQA</th>
    <th>SAMSum</th>
    <th>PCount</th>
    <th>PRc</th>
    <th>Lcs</th>
    <th>RB-P</th>
  </tr>
  <tr>
    <td colspan="22">LLaMA-2-7B-chat-hf</td>
  </tr>
  <tr>
    <td>FP16</td><td>20.04</td><td>85.39</td><td>59.67</td><td>39.28</td><td>59.49</td><td>48.04</td><td>23.02</td><td>34.34</td><td>35.19</td><td>25.14</td><td>28.08</td><td>6.33</td><td>15.33</td><td>37.17</td>
  </tr>
  <tr>
    <td>GEAR</td><td>19.36</td><td>85.58</td><td>59.67</td><td>38.34</td><td>58.01</td><td>46.44</td><td>20.32</td><td>34.94</td><td>34.24</td><td>32.06</td><td>21.41</td><td>6.33</td><td>14.67</td><td>36.26</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.7,v=0.8)</td><td>19.20</td><td>84.03</td><td>59.33</td><td>38.14</td><td>53.26</td><td>45.81</td><td>18.48</td><td>28.05</td><td>33.23</td><td>30.12</td><td>17.31</td><td>6.48</td><td>12.67</td><td>34.32</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.6,v=0.6)</td><td>19.28</td><td>84.31</td><td>59.00</td><td>39.41</td><td>56.54</td><td>46.28</td><td>19.47</td><td>29.98</td><td>33.83</td><td>31.13</td><td>18.58</td><td>7.44</td><td>14.67</td><td>35.38</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.5,v=0.5)</td><td>19.54</td><td>85.02</td><td>59.33</td><td>39.50</td><td>55.75</td><td>45.14</td><td>20.40</td><td>29.80</td><td>33.94</td><td>31.09</td><td>19.80</td><td>6.33</td><td>14.67</td><td>35.43</td>
  </tr>
  <tr>
    <td>KIVI(G=128,R=128)</td><td>20.43</td><td>85.16</td><td>59.67</td><td>37.70</td><td>58.73</td><td>47.24</td><td>19.82</td><td>31.03</td><td>34.65</td><td>30.38</td><td>19.97</td><td>6.33</td><td>11.67</td><td>35.60</td>
  </tr>
  <tr>
    <td>KIVI(G=32,R=128)</td><td>19.81</td><td>84.92</td><td>59.67</td><td>38.77</td><td>58.73</td><td>47.24</td><td>20.03</td><td>34.77</td><td>31.39</td><td>31.88</td><td>18.07</td><td>7.98</td><td>14.00</td><td>36.29</td>
  </tr>
  <tr>
    <td>Ours(G=128,R=32)</td><td>19.95</td><td>85.00</td><td>59.57</td><td>39.10</td><td>59.44</td><td>48.51</td><td>20.81</td><td>34.12</td><td>34.98</td><td>31.87</td><td>21.56</td><td>7.33</td><td>14.33</td><td>36.33</td>
  </tr>
  <tr>
    <td>Ours(G=32,R=128)</td><td>21.34</td><td>84.94</td><td>59.67</td><td>39.04</td><td>59.48</td><td>47.64</td><td>22.09</td><td>32.38</td><td>34.64</td><td>32.43</td><td>24.73</td><td>7.33</td><td>14.33</td><td>36.90</td>
  </tr>
  <tr>
    <td colspan="22">LLaMA-3-8B-Instruct</td>
  </tr>
  <tr>
    <td>FP16</td><td>37.54</td><td>89.85</td><td>69.67</td><td>40.50</td><td>56.68</td><td>51.01</td><td>25.58</td><td>40.56</td><td>49.81</td><td>34.93</td><td>31.04</td><td>12.94</td><td>83.67</td><td>47.98</td>
  </tr>
  <tr>
    <td>GEAR</td><td>37.55</td><td>89.85</td><td>69.67</td><td>40.02</td><td>56.42</td><td>50.47</td><td>25.52</td><td>40.41</td><td>48.80</td><td>34.93</td><td>30.53</td><td>12.61</td><td>83.33</td><td>47.79</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.7,v=0.8)</td><td>36.91</td><td>89.98</td><td>69.33</td><td>40.71</td><td>42.30</td><td>44.84</td><td>23.98</td><td>41.88</td><td>49.01</td><td>33.87</td><td>28.13</td><td>14.35</td><td>82.00</td><td>49.55</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.6,v=0.6)</td><td>36.61</td><td>90.14</td><td>69.33</td><td>40.58</td><td>45.82</td><td>44.96</td><td>24.70</td><td>39.32</td><td>50.05</td><td>33.46</td><td>29.31</td><td>13.60</td><td>84.33</td><td>46.32</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.5,v=0.5)</td><td>35.86</td><td>89.86</td><td>69.00</td><td>39.97</td><td>46.41</td><td>44.64</td><td>25.36</td><td>40.36</td><td>49.75</td><td>33.69</td><td>30.12</td><td>12.53</td><td>83.33</td><td>46.22</td>
  </tr>
  <tr>
    <td>KIVI(G=128,R=128)</td><td>34.88</td><td>89.57</td><td>69.33</td><td>40.09</td><td>44.24</td><td>45.78</td><td>24.78</td><td>39.19</td><td>49.65</td><td>34.19</td><td>28.11</td><td>11.50</td><td>82.00</td><td>45.66</td>
  </tr>
  <tr>
    <td>KIVI(G=32,R=128)</td><td>37.27</td><td>89.88</td><td>70.00</td><td>40.46</td><td>47.29</td><td>45.20</td><td>24.34</td><td>41.29</td><td>49.87</td><td>35.05</td><td>30.38</td><td>12.67</td><td>83.67</td><td>48.60</td>
  </tr>
  <tr>
    <td>Ours(G=128,R=32)</td><td>39.74</td><td>89.74</td><td>69.67</td><td>40.57</td><td>48.82</td><td>49.44</td><td>41.57</td><td>50.37</td><td>53.32</td><td>30.74</td><td>14.83</td><td>11.53</td><td>81.27</td><td>47.34</td>
  </tr>
  <tr>
    <td>Ours(G=32,R=128)</td><td>36.71</td><td>90.56</td><td>70.00</td><td>40.67</td><td>52.65</td><td>47.76</td><td>25.34</td><td>50.36</td><td>53.13</td><td>31.16</td><td>12.53</td><td>83.33</td><td>47.12</td>
  </tr>
</table>

Table 11: Performance comparison of OTT with GEAR, ZipCache, KIVI and FP16 on LongBench for LLaMA-3-8B-Instruct and LLaMA-2-7B-chat-hf. OTT generally achieves improvements over previous KV cache compression methods across various LLMs. Bold text represents the best performance.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">ruler_recall</th>
    <th colspan="2">substring_exact_match</th>
    <th colspan="2">NDCG@10</th>
    <th colspan="2">str_em</th>
    <th colspan="2">citation_rec</th>
    <th colspan="2">qampari_rec_top5</th>
    <th colspan="2">citation_prec</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>sub_mlk_2</th>
    <th>sub_mlk_3</th>
    <th>sub_mlk_m</th>
    <th>json_kv</th>
    <th>hotspot</th>
    <th>formal_psg</th>
    <th>alce_asgpl</th>
    <th>alce_pamprj</th>
    <th>alce_asgpl</th>
    <th>alce_pamprj</th>
    <th>alce_asgpl</th>
    <th>alce_pamprj</th>
    <th>alce_asgpl</th>
    <th>alce_pamprj</th>
  </tr>
  <tr>
    <td>full</td>
    <td>100.00</td>
    <td>100.00</td>
    <td>99.75</td>
    <td>98.00</td>
    <td>61.00</td>
    <td>55.27</td>
    <td>41.37</td>
    <td>8.47</td>
    <td>6.03</td>
    <td>17.80</td>
    <td>12.88</td>
    <td>9.46</td>
    <td>50.84</td>
  </tr>
  <tr>
    <td>Gear</td>
    <td>100.00</td>
    <td>92.00</td>
    <td>99.75</td>
    <td>92.00</td>
    <td>60.67</td>
    <td>54.01</td>
    <td>42.07</td>
    <td>7.20</td>
    <td>6.19</td>
    <td>18.00</td>
    <td>10.85</td>
    <td>8.22</td>
    <td>49.25</td>
  </tr>
  <tr>
    <td>KIVI(G=32,R=128)</td>
    <td>99.00</td>
    <td>85.00</td>
    <td>99.50</td>
    <td>77.00</td>
    <td>59.67</td>
    <td>41.52</td>
    <td>41.00</td>
    <td>5.83</td>
    <td>7.79</td>
    <td>14.80</td>
    <td>7.72</td>
    <td>7.81</td>
    <td>45.55</td>
  </tr>
  <tr>
    <td>ZipCache(k=0.5,v=0.5)</td>
    <td>98.00</td>
    <td>51.00</td>
    <td>99.25</td>
    <td>45.00</td>
    <td>59.00</td>
    <td>44.88</td>
    <td>35.47</td>
    <td>5.49</td>
    <td>7.28</td>
    <td>6.80</td>
    <td>7.44</td>
    <td>4.95</td>
    <td>38.71</td>
  </tr>
  <tr>
    <td>Ours(G=32,R=128)</td>
    <td>99.00</td>
    <td>87.00</td>
    <td>99.75</td>
    <td>80.00</td>
    <td>61.00</td>
    <td>42.92</td>
    <td>44.50</td>
    <td>9.52</td>
    <td>6.51</td>
    <td>17.40</td>
    <td>12.01</td>
    <td>8.33</td>
    <td>47.33</td>
  </tr>
</table>

Table 12: Performance comparison of OTT with GEAR, ZipCache, KIVI and FP16 on HELMET for LLaMA-3-8B-Instruct. OTT generally achieves improvements over previous KV cache compression methods across various LLMs. Bold text represents the best performance.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="6">Easy Hard Short Medium Long Overall</th>
  </tr>
  <tr>
    <th>FP16</th>
    <th>GEAR</th>
    <th>KIVI (G=32, R=128)</th>
    <th>ZipCache (k=0.5, v=0.5)</th>
    <th>Ours (G=32, R=128)</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>27.6</td>
    <td>27.0</td>
    <td>25.6</td>
    <td>25.6</td>
    <td>33.3</td>
    <td>27.2</td>
  </tr>
  <tr>
    <td>GEAR</td>
    <td>27.6</td>
    <td>27.0</td>
    <td>25.6</td>
    <td>25.6</td>
    <td>33.3</td>
    <td>27.2</td>
  </tr>
  <tr>
    <td>KIVI (G=32, R=128)</td>
    <td>26.6</td>
    <td>22.2</td>
    <td>23.9</td>
    <td>21.9</td>
    <td>27.8</td>
    <td>23.9</td>
  </tr>
  <tr>
    <td>ZipCache (k=0.5, v=0.5)</td>
    <td>26.6</td>
    <td>25.4</td>
    <td>26.1</td>
    <td>23.3</td>
    <td>30.6</td>
    <td>25.8</td>
  </tr>
  <tr>
    <td>Ours (G=32, R=128)</td>
    <td>25.5</td>
    <td>26.7</td>
    <td>26.7</td>
    <td>22.8</td>
    <td>32.4</td>
    <td>26.2</td>
  </tr>
</table>

Table 13: Performance comparison of OTT with GEAR, ZipCache, KIVI and FP16 on LongBench_v2 for LLaMA-3-8B-Instruct. OTT generally achieves improvements over previous KV cache compression methods across various LLMs. Bold text represents the best performance.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">Single NIAH</th>
    <th colspan="5">Multi-key NIAH</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>S-NIAH-1</th>
    <th>S-NIAH-2</th>
    <th>S-NIAH-3</th>
    <th>MK-NIAH-1</th>
    <th>MK-NIAH-2</th>
    <th>MK-NIAH-3</th>
    <th>MQ-NIAH</th>
    <th>MV-NIAH</th>
    <th>CWE</th>
    <th>FWE</th>
    <th>VT</th>
  </tr>
  <tr>
    <td colspan="13">max_length=64k</td>
  </tr>
  <tr>
    <td>FP16</td>
    <td>100</td>
    <td>99.4</td>
    <td>100</td>
    <td>99</td>
    <td>99.8</td>
    <td>99.4</td>
    <td>98.85</td>
    <td>95.8</td>
    <td>8.42</td>
    <td>76</td>
    <td>97.96</td>
    <td>88.60</td>
  </tr>
  <tr>
    <td>GEAR</td>
    <td>100</td>
    <td>99.4</td>
    <td>100</td>
    <td>99</td>
    <td>99.6</td>
    <td>90.2</td>
    <td>94.4</td>
    <td>96</td>
    <td>8.62</td>
    <td>76.53</td>
    <td>97.96</td>
    <td>87.43</td>
  </tr>
  <tr>
    <td>KIVI(G=32,R=128)</td>
    <td>99.8</td>
    <td>99.2</td>
    <td>97.8</td>
    <td>95.6</td>
    <td>97.2</td>
    <td>78.4</td>
    <td>94.5</td>
    <td>93</td>
    <td>8.26</td>
    <td>78.73</td>
    <td>92.24</td>
    <td>84.98</td>
  </tr>
  <tr>
    <td>Ours(G=32,R=128)</td>
    <td>99.8</td>
    <td>99.2</td>
    <td>99.2</td>
    <td>96</td>
    <td>97.6</td>
    <td>82</td>
    <td>97.5</td>
    <td>94.5</td>
    <td>9.28</td>
    <td>78.53</td>
    <td>93.92</td>
    <td>86.14</td>
  </tr>
  <tr>
    <td colspan="13">max_length=128k</td>
  </tr>
  <tr>
    <td>FP16</td>
    <td>100</td>
    <td>94</td>
    <td>100</td>
    <td>93</td>
    <td>100</td>
    <td>100</td>
    <td>98.75</td>
    <td>96.25</td>
    <td>0.3</td>
    <td>82.67</td>
    <td>96.6</td>
    <td>87.42</td>
  </tr>
  <tr>
    <td>GEAR</td>
    <td>100</td>
    <td>94</td>
    <td>98</td>
    <td>93</td>
    <td>100</td>
    <td>87</td>
    <td>98.5</td>
    <td>96.25</td>
    <td>0.3</td>
    <td>81.67</td>
    <td>96.6</td>
    <td>85.94</td>
  </tr>
  <tr>
    <td>KIVI(G=32,R=128)</td>
    <td>100</td>
    <td>92</td>
    <td>99</td>
    <td>91</td>
    <td>95</td>
    <td>69</td>
    <td>93</td>
    <td>87.5</td>
    <td>0.4</td>
    <td>80.33</td>
    <td>89.4</td>
    <td>81.51</td>
  </tr>
  <tr>
    <td>Ours(G=32,R=128)</td>
    <td>100</td>
    <td>94</td>
    <td>99</td>
    <td>92</td>
    <td>95</td>
    <td>73</td>
    <td>96.25</td>
    <td>90.25</td>
    <td>0.4</td>
    <td>82.67</td>
    <td>88.8</td>
    <td>82.85</td>
  </tr>
</table>

Table 14: Performance comparison of Llama-3-8B-ProLong-512k-Instruct with longer context lengths (64k, 128k).

![Throughput (left) and memory usage (right) of different methods under LLaMA-2-7B-chat-hf, input length=64, output length=384, batch size=128 in NVIDIA A100 40G.](page_1012_1342_627_246.png)

Figure 8: Throughput (left) and memory usage (right) of different methods under LLaMA-2-7B-chat-hf, input length=64, output length=384, batch size=128 in NVIDIA A100 40G.
Figure 9: Results of Needle-in-a-Haystack on LLaMA-2-7B-chat-hf with 4k context size. The vertical axis of the table represents the depth percentage, and the horizontal axis represents the token length.
Figure 10: Results of Needle-in-a-Haystack on LLaMA-3-8B-Instruct with 8k context size. The vertical axis of the table represents the depth percentage, and the horizontal axis represents the token length.

Reviewer paper summaries: This paper presents a novel improvement method for KV cache quantization, which is an important area of research given the computational demands of large language models. The authors make a valuable observation regarding small-magnitude outlier keys, which deviate from the typical assumptions in KV cache quantization. By retaining the precision of these outliers, the proposed method effectively mitigates quantization errors. The experiments further validate the effectiveness of the approach.

This paper proposes a new quantization method for KV cache in LLMs, called OTT (Outlier Tokens Tracing). The authors observe that previous KV cache quantization approaches fail to account for outlier tokens, which can greatly impact model quality. The key contribution of this paper is identifying that a small subset of tokens exhibits clear outlier patterns in the KV activations. To address this, OTT dynamically identifies these outlier tokens during decoding and excludes them from quantization. The method introduces a fixed-size outlier pool to store these outlier tokens and implements CUDA kernels for efficient inference under quantization. Experiments show the effectiveness of the method with multiple models on diverse tasks.
Summary of strengths: 1. The paper makes a novel and insightful observation about small-magnitude outlier keys, which has been less explored compared to large outliers. This observation is valuable and leads to a practical solution that makes sense and is inspiring for future research.

2. The proposed method is simple and avoids complex operations that could introduce additional latency. This simplicity is a significant advantage, especially in deployment scenarios where efficiency is crucial.

This paper has the following strengths:
1. The analysis for motivating and validating the approach is thorough. The authors presented a comprehensive investigation of key distribution patterns with clear visualizations. There are detailed ablation studies validating all key design choices.
2. The method has empirical significance. Consistent improvements are shown across multiple architectures and diverse tasks.
Summary of weaknesses: 1. While the observation about small-magnitude outliers is interesting, it is unclear how frequently such outliers occur across different tasks and datasets. Given that these outliers seem to be rare (from Figure 1e), there is a possibility that this observation may not be universally applicable. In tasks where such outliers are absent, the proposed method might not offer significant benefits. The paper should provide more discussion on the generalizability of this observation.

1. The technical contributions presented by this paper are limited. The quantization method proposed largely follow KIVI: it uses per-channel for keys and per-token for values, and a residual cache. The outlier pattern has been observed by previous works such as ZipCache [1].
2. Details of the proposed method are unclear in multiple places. 
- On line 284: "the tokens that are originally in the outlier pool but are defeated by the newly added tokens should return to their original positions," how can these tokens return to their original positions? The X_min, X_max constants within the same group are determined without these outlier tokens, so if they return to their original positions and become quantized, the model quality can degrade significantly.
- On line 286: "But for the sake of simplicity, we retain an additional pool to store these tokens, and when this pool is full, we stop identifying outlier tokens." This is quite confusing. I would suggest the authors use mathematical notations to describe what they are proposing here.
- On line 302: "To enhance decoding efficiency, we utilize a CUDA fused kernel to multiply full-precision and quantized matrices efficiently." Can the authors please share more details on the kernel?
- Does OTT quantize tokens before rotary positional embeddings are applied or after?
- How are outlier tokens identified? How do they compete with each other? Can the authors please describe the process using mathematical notations?

[1] ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification. https://arxiv.org/abs/2405.14256v1
Comments, suggestions and typos: 1. Given that most prior work has focused on large outliers, what additional experiments or analyses were conducted to ensure that the proposed method does not negatively impact performance when small outliers are not present?

1. Line 280: should be "an outlier"
2. Line 430: should be "time analysis"
Overall assessment scores: 3.5

2
Metareview score: 3
Metareview: The paper introduces OTT, a method to accurately quantize the KV cache for large language models by detecting and excluding a small subset of outlier tokens from quantization. This approach reduces quantization error, lowers memory usage, and improves inference throughput.
Summary of reasons to publish: The work presents a straightforward method for identifying outlier tokens—less considered in prior quantization methods—and demonstrates end-to-end efficiency improvements in latency and memory usage (Reviews pM8X and UBnA). The experimental analysis, including ablation studies and evaluations across multiple architectures and tasks, adds empirical support to the design choices (Review HLFQ). Additionally, despite building on existing techniques like those in KIVI and ZipCache, OTT shows tangible benefits, particularly in handling longer input sequences where memory overhead is critical (Review pM8X).