LSH Tells You What To Discard: An Adaptive Locality-Sensitive Strategy for KV Cache Compression

Authors: Tahseen Rabbani, Minghui Liu, Tony O'Halloran, Ananth Sankaralingam, Mary-Anne Hartley, Furong Huang
Primary area: foundation or frontier models, including LLMs
Keywords: kv cache, locality-sensitive hashing, compression

Abstract:
Transformer-based large language models (LLMs) use the key-value (KV) cache to significantly accelerate inference by storing the key and value embeddings of past tokens. However, this cache consumes significant GPU memory. In this work, we introduce LSH-E, an algorithm that uses locality-sensitive hashing (LSH) to compress the KV cache. LSH-E quickly locates tokens in the cache that are cosine dissimilar to the current query token. This is achieved by computing the Hamming distance between binarized Gaussian projections of the current token query and cached token keys, with a projection length much smaller than the embedding dimension. We maintain a lightweight binary structure in GPU memory to facilitate these calculations. Unlike existing compression strategies that compute attention to determine token retention, LSH-E makes these decisions pre-attention, thereby reducing computational costs. Additionally, LSH-E is dynamic -- at every decoding step, the key and value of the current token replace the embeddings of a token expected to produce the lowest attention score. We demonstrate that LSH-E can compress the KV cache by 30\%-70\% while maintaining high performance across reasoning, multiple-choice, long-context retrieval and summarization tasks.

Decision: reject
Reviewer scores: 5, 3, 5, 1, 3, 6

Reviews and discussion:
<thread>
creation date:
1730696058369

thread_type:
official_review

<thread_object>
creator: reviewer_a2yh

<content>
summary:
This paper introduces a KV cache compression method based on LSH and shows that LSH-E can achieve good downstream performance on various downstream tasks with a 30%- 70% compression ratio.

strengths:
This paper applies novel LSH methods to KV cache problems. The motivations and reasons why LSH can produce a good performance are well discussed. Besides this, a static compression rate of 30% - 70% is also helpful for many LLM serving systems, given the accuracy is preserved.

weaknesses:
1. There is no comparison with other static KV compression baselines, including H2O, streamingLLM, and SnapKV. If this problem is solved, I will raise my score.
2. Only the memory compression ratio is shown. I will ask for the wall clock speedups (latency or throughput).

questions:
Besides the problems mentioned in Weakness,
1 Does this method work well with quantization (KIVI, AWQ)?
2 How long does LSH-E increase first token latency?

These two questions can be left for future work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699749147

thread_type:
official_review

<thread_object>
creator: reviewer_rwsu

<content>
summary:
This paper proposes a method that uses LSH to perform kv cache eviction. The provided experiments show that the proposed method outperforms the baseline.

strengths:
Strong Points
----
S1. The problem of the paper is well-motivated. 

S2. The proposed algorithm is simple and clear with illustrative example.

S3. The proposed method outperforms the baseline L2.

weaknesses:
Weak Points
----
W1. Important related studies and baselines are missing:
Singhania, P., Singh, S., He, S., Feizi, S., & Bhatele, A. (2024). Loki: Low-Rank Keys for Efficient Sparse Attention. arXiv preprint arXiv:2406.02542.
Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., & Han, S. (2024). Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference. arXiv preprint arXiv:2406.10774.

W2. The key measures of the targeted task should be have more accurate inference with lower memory footprint and latency. I do not agree with the methodology of not comparing with other "non attention-free" methods.

W3. The presentation of experiments need to be improved: Lack of discussions and intuitions in the experiment analysis. For example, why does LSH-E outperform Full in Figure 4a; why does LSH-E become worse than L2 after 50% cache budget in Figure 4b? We have many subsubsections in the experiments, but most contents in those are barely text illustration of the figure and result while no discussion of why we would have those results.

W4. The execution time of the proposed system is missing.

W5. The discussion of the error introduced by the LSH is not included. I wonder what if we use cosine similarity to evict the cache instead of LSH, how will be the accuracy, latency, and memory usage?

W6. In the supplementary materials, we see more experiments with more baselines that are better than L2. I wonder the reason why the authors do not include them.


Presentation
----
P1. Line 180 "heavy hitters' -> ``heavy hitters''
P2. The axis captions of the figures are too small to be seen.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701323200

thread_type:
official_review

<thread_object>
creator: reviewer_nvgh

<content>
summary:
This paper introduces LSH-E, an algorithm for compressing the key-value (KV) cache in large language models (LLMs) using locality-sensitive hashing (LSH). Despite the availability of prior work—including KDEformer, Hyperattention, SubGen, and QJL—that similarly utilizes LSH for efficient attention and memory management, these related efforts are not cited here. LSH-E leverages Hamming distance calculations in a binary space following a Quantized Johnson-Lindenstrauss (JL) transform (SimHash) to identify and evict tokens with low relevance to the current query, resulting in memory savings. This pre-attention approach provides a lightweight, GPU-efficient solution for long-context tasks, although its effectiveness ultimately depends on the algorithm’s CUDA implementation efficiency.

strengths:
The use of theoretical approaches such as SimHash, a highly efficient hashing method, is a valuable aspect of this work, contributing to both the effectiveness and scalability of the proposed method.

weaknesses:
- The term "novel" should not be used for LSH in this context, as it is not a new approach and has appeared in prior work. Specifically, the methods used in KDEformer, Hyperattention, QJL, and SubGen demonstrate significant overlap, yet these works are not cited here, despite their relevance.

- The experimental setup lacks comprehensiveness; comparisons with alternative methods like H2O, SubGen, and other established baselines should be included to provide a more robust evaluation.

- The datasets used in the experiments are not sufficiently large for evaluating performance in long-context scenarios. Given that these methods target long-sequence processing, experiments should ideally use token sizes over 50,000. LongBench or other large-scale datasets would be more appropriate for a thorough evaluation.

- Additionally, runtime metrics should be reported to assess the efficiency of token generation and to substantiate the computational benefits claimed in the paper.

KDEformer : https://proceedings.mlr.press/v202/zandieh23a.html
HyperAttention : https://arxiv.org/abs/2310.05869
SubGen :  https://arxiv.org/abs/2402.06082
QJL : https://arxiv.org/abs/2406.03482

questions:
- Could you provide a plot showing the distortion error introduced by LSH compression across different levels of compression? Specifically, how does the approximation quality change as more tokens are evicted or as the quantization parameters are adjusted?

- Given that LSH-E’s efficiency largely depends on its CUDA implementation, can you elaborate on any specific optimizations made within the CUDA code?

- Could you clarify how LSH-E handles multi-head attention? Specifically, is each head processed separately with its own LSH compression, or is there a shared mechanism across heads?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716280569

thread_type:
official_review

<thread_object>
creator: reviewer_yqyi

<content>
summary:
This paper presents new methods to accelerate inference of auto-regressive transformers used in most modern-day decoder-based LLM architectures. Indeed, the main drawback of existing systems is the size of the "KV  Cache" or Key-Value Cache which is used during the attention mechanism. To speed up the attention calculation, most systems have a cache which remembers the keys and values of commonly used tokens, to avoid recomputing it for each token decoding.  However ,such a cache, for it to be performant at inference time, must scale quadratically with the sequence length, and linear in number of layers and attention heads. 

(Authors: please explain why for the uninformed reader -- this is stated in the intro, but without explanation)

In this paper, the authors present an LSH based method to evict far-away key tokens. Indeed, suppose we have an LSH which gets a binary encoding of any vector using random hyperplane projection method (SIMHASH). 
Then, we can first pre-process and compute the hamming distance between query token and all key tokens, and evict the farthest one, as this is the one least likely to affect the overall attention soft-max operation.

They implement this simple scheme and provide a range of quality vs cache size metrics comparing with one other KV-cache called L2-Dropout Cache, which drops the keys based on their magnitudes.

strengths:
Studies an important problem of much significance in todays LLM era. 

Presents a simple yet elegant approach

Does good evaluations on a range of use-cases

weaknesses:
Why is there no timing experiment, since that will be one key benefit of caching.

Why only restrict to attention-free cache policies and specifically only compare with the L2-dropout baseline?

Conceptually, what is the key difference with Reformer? I have not read that paper but you mention in passing that it is using LSH and simhash also. Is which cells to evaluate vs what to evict the only difference between Reformer and your work? If so, worth comparing with Reformer also in plots?

What is the rationale of the policy? Why can't a token just evicted become relevant again? I guess is there some language-based "locality of reference"?

Do ablation of the hardcoded bits, i.e., you mention you hard-cache the first few and last few tokens. What is the contribution of this to your overall success metrics?

The eviction policy is not clearly understandable in how it aggregates the hamming distances over time steps. Is it only based on the most recent time step, or some more complex rule?

questions:
Line 52: "However, this L2 dropout strategy only performs well on
long-context retrieval tasks. It is specialized to retain only those tokens with the highest attention" -- be more specific. Why is this?

Line 57: "wide variety of tasks?" -- how do you define this?

Line 145: Formally for our setup, distd(x, y) cos θx,y, here it is more a measure of cosine similarity than distance. Misleading, perhaps?

Line 419: did you mean "LSH dimension does significantly impact performance" --> does not?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731441980121

thread_type:
official_review

<thread_object>
creator: reviewer_r9hv

<content>
summary:
The idea is to reduce KV Cache by evicting and permanently dropping tokens at each position in the query. The heuristic used is to evict the lowest attention scored keys ( which is essentially similar to H2O / Scissorhands which preserve top attention scored keys). The difference is to use LSH to do a approximate score ranking to avoid SoftMax for exact computation.

strengths:
Uses LSH to approximate attention computation for eviction (if you compare to h2o / scissorhands)

weaknesses:
- Novelty: The novelty is limited.
- H2O / Scissorhands are known to not perform well on longbenchmark. Can we see some results on longbenchmark like passage retrieval datasets ?
- Missing baselines --only baseline used is L2 norm. 
- Limited evaluation. can we get more results on longbenchmark at different budgets with standard baselines.

questions:
see questions above,
</content>
</thread_object>

</thread>
<thread>
creation date:
1732396452473

thread_type:
official_review

<thread_object>
creator: reviewer_htgz

<content>
summary:
LLMs utilize KV cache to accelerate inference but take up significant GPU memory. LSH-E is an algorithm that uses LSH to compress the KV cache by evicting tokens that are cosine dissimilar. The token eviction happens pre-attention, thus making this method computationally affordable.

strengths:
1. The small size of the KV cache allows it to be stored in GPU memory, eliminating latency from moving data between CPU and GPU.
2. KV cache eviction happens before attention computation, cutting down on unnecessary and expensive attention computations.
3. The greedy eviction approach makes it computationally very affordable.

weaknesses:
1. It would be helpful to have an ablation study of LSH-E's performance with different numbers of first and recent tokens cached.
2. The benchmarks seem limited; there are only two datasets per task and the improvement over the baseline is not very significant in Needle-in-a-Haystack, Common Words, and MedQA Multiple Choice.
3. Evaluation does not include end-to-end speedup numbers, making it more difficult to see the ultimate impact of the contribution.
4. The greedy eviction algorithm assumes that the attention score between a particular key vector and the current query vector is representative of the attention score with subsequent query vectors. While there is ample empirical exploration on the correlation between attention and inverted LSH hamming distance, I could not find provable theoretical guarantees about the quality of the KV cache under this greedy eviction strategy or empirical observations about the consistency of attention scores across query vectors that suggest the soundness of this assumption. This is in contrast to other greedy approaches such as H2O that uses *accumulated* attention to be more robust to variations between individual query tokens.

questions:
1. Under "Configuration and Setup", it is mentioned that you "keep the most recent 10 tokens and the first 4 tokens of the prompt always in the KV cache." Is the L2 eviction baseline also configured this way?
2. How well does LSH-E perform without keeping the most recent 10 tokens and the first 4 tokens?
3. Is it possible to perform more evaluations on LongBench tasks?
4. Do you have empirical results that show that the attention score for the current token is a reasonable proxy for attention scores for subsequent token, or that a low attention score for a current query token implies that the key token will not be critical to subsequent query tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732760771278

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal Pt 1

comment:
# General Comments

Thank you to all reviewers for your thoughtful feedback and constructive suggestions. Your comments have been invaluable in helping us refine and strengthen our work. We are encouraged by the recognition of the computational efficiency and simplicity of our proposed LSH-E method, as well as its potential as a practical strategy for KV cache compression in resource-constrained scenarios. 

# Reviewer Highlights

We would like to summarize the highlights that reviwers appreciated in our paper:

* Reviewer HtGz believes that our attention-free approach is "**computationally** very **affordable**" and "cuts down on unnecessary and expensive attention computations".
* Reviewer NvGH, R9hV and a2yh commend our novel use of "LSH to approximate attention computation".  NvGH comments that it "contributes to both the **effectiveness** and **scalability** of the proposed method". a2yh comments that "the motivations and reasons why LSH can produce a good performance are **well discussed**."
* Reviewr yqyi remarks that our approach is "**simple** yet **elegant**" and that we did "**good evaluations** on a range of use-cases".
* Reviewer rWSu notes that our approach is "**simple** and **clear** with **illustrative examples**". 

# Summary of Changes

To address the feedbacks from the reviewer, we made the following improvements:

## 1. Additional Benchmarks and Baselines
Reviewers suggested that we should add tasks with even longer context length. In response, we expanded the experiments to include two new tasks from the LongBench benchmarks: MultiNews and GovReport. Both are long-context summarization tasks. 

Additionally, comparisons to well-cited KV cache compression strategies, such as H2O, Scissorhands, and FastGen, were added to contextualize LSH-E's performance against state-of-the-art baselines. We have updated existing experiments in the paper to include these new baselines. We also provide results of the two summarization tasks in Table 1 below. 

In these new experiments, LSH-E consistently demonstrats comparable or superior Rouge L scores across various cache budgets. In the MultiNews summarization task, LSH-E achieves higher Rouge L score at most cache budgets, outperforming all baselines, demonstrating LSH-E’s robustness and effectiveness in handling very large context lengths.

## 2. Throughput Analysis
Another addition to this rebuttal is the inclusion of throughput metrics. We provide decoding and prefill tokens per second results on the LongBench MultiNews task. LSH-E is 1.5-2x faster than H2O and Scissorhands, and 17x faster than FastGen at the prefill stage. Even without low-level optimizations (e.g., expressing hash tables in binary bits), LSH-E proved to be as fast as the L2 strategy in decoding and significantly faster than attention-based baselines. 

This speedup was achieved while maintaining competitive quality metrics, demonstrating the computational efficiency of LSH-E. The throughput results address reviewer concerns about runtime metrics and substantiate the claimed computational benefits.

### Table 1: Results of LongBench MultiNews Summarization with Throughput Metrics
|  |  | GovReport | MultiNews |  |  |
|---|---|---|---|---|---|
| Strategy | Cache Budget | Rouge L | Rouge L | Decode Toks Per Sec | Prefill Toks Per Sec |
| Full | 100% | 0.230 | 0.192 | 16.071 | 16573.492 |
| LSH-E | 30% | 0.202 | 0.180 | 22.880 | 20293.524 |
| L2 | 30% | 0.201 | 0.165 | 23.981 | 20628.160 |
| H2O | 30% | 0.219 | 0.175 | 21.555 | 13025.776 |
| Scissorhands | 30% | 0.214 | 0.175 | 21.448 | 13004.254 |
| LSH-E | 50% | 0.217 | 0.186 | 22.846 | 20459.961 |
| L2 | 50% | 0.214 | 0.174 | 16.013 | 15851.952 |
| H2O | 50% | 0.225 | 0.181 | 21.973 | 13969.985 |
| Scissorhands | 50% | 0.219 | 0.182 | 20.978 | 13549.967 |
| LSH-E | 70% | 0.223 | 0.187 | 22.914 | 21002.334 |
| L2 | 70% | 0.223 | 0.187 | 24.305 | 21303.763 |
| H2O | 70% | 0.229 | 0.184 | 21.793 | 14050.521 |
| Scissorhands | 70% | 0.226 | 0.183 | 21.705 | 13954.693 |
| LSH-E | 90% | 0.228 | 0.185 | 22.873 | 21229.230 |
| L2 | 90% | 0.230 | 0.186 | 24.010 | 21305.693 |
| H2O | 90% | 0.227 | 0.181 | 21.665 | 14007.697 |
| Scissorhands | 90% | 0.230 | 0.182 | 21.411 | 14025.440 |
| Fastgen | Attention recovery frac 70% | 0.192 | 0.129 | 12.752 | 1171.069 |
| Fastgen | Attention recovery frac 75% | 0.231 | 0.174 | 12.291 | 1157.987 |
| Fastgen | Attention recovery frac 80% | 0.232 | 0.184 | 11.850 | 1142.679 |
| Fastgen | Attention recovery frac 85% | 0.236 | 0.183 | 11.658 | 1164.689 |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732760923101

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal Pt 2

comment:
## 3. Ablation Studies on Attention Sink Tokens and Recent Tokens
To address concerns about hardcoding specific tokens for retention, we conducted ablation studies on the impact of retaining attention sink tokens (first 4 tokens) and recent tokens (last 10 tokens). The results revealed that disabling these features led to performance degradation. For example, at a 50% cache budget on GSM8K, LSH-E without sink and recent tokens scored a Rouge L of 0.173 compared to 0.393 with these features enabled.

This study not only validated the necessity of maintaining these tokens for optimal performance but also aligned LSH-E’s configuration with standard practices in competing methods like H2O and Scissorhands. We hope that the ablation results strengthened the empirical foundation of our method, demonstrating that these design choices are essential and justified.

### Table 2: Ablation of Attention Sink Tokens and Recent Tokens on GSM8K Free Response Question Answering
| Strategy | Cache Budget (%) | BertScore F1 | Rouge L | ChatGPT as a Judge Avg |
|---|---|---|---|---|
| LSH-E | 30% | 0.873 | 0.341 | 3.173 |
| LSH-E no sink & recent | 30% | 0.652 | 0.048 | 1.028 |
| L2 | 30% | 0.865 | 0.288 | 1.880 |
| L2 no sink & recent | 30% | 0.844 | 0.228 | 1.270 |
| LSH-E | 50% | 0.880 | 0.393 | 4.110 |
| LSH-E no sink & recent | 50% | 0.777 | 0.173 | 1.513 |
| L2 | 50% | 0.875 | 0.355 | 2.936 |
| L2 no sink & recent | 50% | 0.866 | 0.318 | 2.217 |
| LSH-E | 70% | 0.881 | 0.401 | 4.313 |
| LSH-E no sink & recent | 70% | 0.841 | 0.295 | 2.687 |
| L2 | 70% | 0.879 | 0.386 | 3.689 |
| L2 no sink & recent | 70% | 0.876 | 0.374 | 3.390 |
| LSH-E | 90% | 0.881 | 0.403 | 4.388 |
| LSH-E no sink & recent | 90% | 0.868 | 0.363 | 3.630 |
| L2 | 90% | 0.881 | 0.400 | 4.208 |
| L2 no sink & recent | 90% | 0.880 | 0.397 | 4.110 |

## 4. Attention Loss Analysis

We added an analysis of attention loss for LSH-E, L2 and Scissorhands, quantifying the discrepancy introduced by the eviction strategy compared to maintaining the full cache. We measured the atttention loss of each attention head and report the average. Attention loss is defined as the sum of the attention probabilities for evicted tokens. Or equivalently, 1 - the sum of the attention probabilities for the tokens in the compressed cache.
            
The attention loss was measured at 50% cache budget using prompts from the GSM8K question answering dataset. As per Table 5, all three methods have low attention loss at 50% cache budget, and LSH-E has lower attention loss compared to L2 and scissorhands, proving LSH-E's ability of keeping high attention tokens in the KV cache.  

By quantifying attention loss, we demonstrated that LSH-E introduces minimal deviation from full-cache attention, addressing concerns about the theoretical guarantees of its quality.

### Table 3: Attention Loss
| Strategy       | Attention Loss    |
|----------------|-------------------|
| LSH-E          | 0.03357896805     |
| L2             | 0.03403072357     |
| Scissorhands   | 0.04483547211     |


## 5. Clarified Novelty and Conceptual Differences
We clarified the conceptual distinctions between LSH-E and related works such as Reformer, H2O, and SubGen and updated the related works section of our paper. While LSH-E uses LSH for token eviction, Reformer and similar methods use LSH to accelerate attention computation. This distinction underscores LSH-E’s novelty as a probabilistically guaranteed attention-free token eviction strategy, separating it from approaches like L2 eviction. Additionally, it does not require scanning the entirety of the context like existing approaches, which risks VRAM blowup.

This clarification strengthens the claim of LSH-E's novelty and highlights its practical advantages, particularly in memory-constrained scenarios.

## 6. Improved Presentation and Addressed Minor Issues

We addressed several presentation issues, including improving axis captions in figures and fixing typographical errors. These changes enhanced the paper’s readability. Moreover, we expanded the discussion of results, providing intuitive explanations for observed trends, such as why LSH-E outperforms Full in certain cases and why its performance degrades at lower cache budgets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732761255073

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal Pt 3

comment:
## 7. Updated Ablation Studies on Attention Sink Tokens and Recent Tokens

We performed additional ablations including using only sink and recent tokens as a strategy, and L2 and LSH-E with only sink tokens, and with only recent tokens. The LSH dimension was set to 16 bits. The number of sink tokens is 4 and the number of recent tokens is 10 except for the Sink & Recent strategy, which keeps (cache_size - 4) most recent tokens. From the results we found that sink tokens have a bigger impact on the performance of LSH while recent tokens impact L2 more. Please see updated results in Table 4 below. 

#### Table 4: Ablation of Attention Sink Tokens and Recent Tokens on GSM8K Free Response Question Answering
| Cache Budget | Strategy | Bert F1 | Rouge L | GPT Rouge | GPT Coherence | GPT Faithfulness | GPT Helpfulness |
|---|---|---|---|---|---|---|---|
| 10% | LSH-E | 0.831 | 0.157 | 1.018 | 1.387 | 1.147 | 1.083 |
| 10% | LSH-E no sink no recent | 0.708 | 0.025 | 1.000 | 1.000 | 1.000 | 1.000 |
| 10% | LSH-E no sink | 0.713 | 0.027 | 1.000 | 1.000 | 1.000 | 1.000 |
| 10% | LSH-E no recent | 0.847 | 0.189 | 1.100 | 2.002 | 1.348 | 1.326 |
| 10% | L2 | 0.826 | 0.151 | 1.005 | 1.293 | 1.098 | 1.033 |
| 10% | L2 no sink no recent | 0.804 | 0.130 | 1.000 | 1.088 | 1.030 | 1.016 |
| 10% | L2 no sink | 0.836 | 0.178 | 1.026 | 1.600 | 1.138 | 1.096 |
| 10% | L2 no recent | 0.829 | 0.171 | 1.014 | 1.394 | 1.098 | 1.032 |
| 10% | Sink & Recent | 0.843 | 0.176 | 1.040 | 1.882 | 1.298 | 1.248 |
| 30% | LSH-E | 0.873 | 0.341 | 2.520 | 3.767 | 3.216 | 3.190 |
| 30% | LSH-E no sink no recent | 0.744 | 0.068 | 1.004 | 1.024 | 1.018 | 1.006 |
| 30% | LSH-E no sink | 0.744 | 0.066 | 1.006 | 1.018 | 1.028 | 1.002 |
| 30% | LSH-E no recent | 0.873 | 0.342 | 2.546 | 3.956 | 3.340 | 3.472 |
| 30% | L2 | 0.865 | 0.288 | 1.356 | 2.428 | 1.895 | 1.841 |
| 30% | L2 no sink no recent | 0.844 | 0.228 | 1.040 | 1.478 | 1.292 | 1.268 |
| 30% | L2 no sink | 0.865 | 0.290 | 1.474 | 2.750 | 2.010 | 2.102 |
| 30% | L2 no recent | 0.846 | 0.238 | 1.032 | 1.478 | 1.320 | 1.272 |
| 30% | Sink & Recent | 0.868 | 0.310 | 1.910 | 3.432 | 2.616 | 2.682 |
| 50% | LSH-E | 0.880 | 0.393 | 3.457 | 4.530 | 4.212 | 4.241 |
| 50% | LSH-E no sink no recent | 0.803 | 0.178 | 1.322 | 1.570 | 1.696 | 1.424 |
| 50% | LSH-E no sink | 0.802 | 0.179 | 1.362 | 1.554 | 1.684 | 1.440 |
| 50% | LSH-E no recent | 0.880 | 0.399 | 3.624 | 4.638 | 4.338 | 4.446 |
| 50% | L2 | 0.875 | 0.355 | 2.190 | 3.494 | 3.035 | 3.027 |
| 50% | L2 no sink no recent | 0.866 | 0.318 | 1.548 | 2.690 | 2.320 | 2.308 |
| 50% | L2 no sink | 0.876 | 0.359 | 2.492 | 3.710 | 3.170 | 3.276 |
| 50% | L2 no recent | 0.866 | 0.319 | 1.570 | 2.686 | 2.382 | 2.336 |
| 50% | Sink & Recent | 0.879 | 0.385 | 3.412 | 4.488 | 4.054 | 4.122 |
| 70% | LSH-E | 0.881 | 0.401 | 3.734 | 4.671 | 4.404 | 4.444 |
| 70% | LSH-E no sink no recent | 0.847 | 0.295 | 2.350 | 2.818 | 2.912 | 2.612 |
| 70% | LSH-E no sink | 0.847 | 0.295 | 2.332 | 2.794 | 2.888 | 2.600 |
| 70% | LSH-E no recent | 0.881 | 0.402 | 3.884 | 4.790 | 4.546 | 4.650 |
| 70% | L2 | 0.879 | 0.386 | 2.934 | 4.184 | 3.817 | 3.820 |
| 70% | L2 no sink no recent | 0.876 | 0.374 | 2.684 | 3.836 | 3.510 | 3.528 |
| 70% | L2 no sink | 0.879 | 0.390 | 3.266 | 4.370 | 4.018 | 4.104 |
| 70% | L2 no recent | 0.876 | 0.374 | 2.718 | 3.842 | 3.522 | 3.516 |
| 70% | Sink & Recent | 0.881 | 0.401 | 3.810 | 4.720 | 4.428 | 4.508 |
| 90% | LSH-E | 0.881 | 0.403 | 3.837 | 4.722 | 4.468 | 4.525 |
| 90% | LSH-E no sink no recent | 0.868 | 0.363 | 3.222 | 3.784 | 3.826 | 3.618 |
| 90% | LSH-E no sink | 0.869 | 0.363 | 3.248 | 3.822 | 3.854 | 3.628 |
| 90% | LSH-E no recent | 0.882 | 0.406 | 4.018 | 4.788 | 4.562 | 4.650 |
| 90% | L2 | 0.881 | 0.400 | 3.569 | 4.578 | 4.324 | 4.361 |
| 90% | L2 no sink no recent | 0.880 | 0.397 | 3.460 | 4.486 | 4.210 | 4.282 |
| 90% | L2 no sink | 0.881 | 0.402 | 3.752 | 4.658 | 4.388 | 4.470 |
| 90% | L2 no recent | 0.880 | 0.397 | 3.438 | 4.482 | 4.188 | 4.238 |
| 90% | Sink & Recent | 0.881 | 0.405 | 4.006 | 4.792 | 4.572 | 4.644 |
| 100% | Full | 0.882 | 0.403 | 3.845 | 4.716 | 4.499 | 4.545 |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732761276822

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal Pt 4

comment:
# Final Comment
We greatly appreciate reviewer feedback. Our rebuttal addresses all questions and concerns. We would appreciate it if the reviewers could update their scores accordingly. Please let us know if you have more comments or questions.

# References

[1] Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., ... & Chen, B. (2023). H2o: Heavy-hitter oracle for efficient generative inference of large language models.

[2] Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., ... & Shrivastava, A. (2024). Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. 

[3] Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023). Efficient streaming language models with attention sinks.

[4] Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., ... & Li, J. (2023). Longbench: A bilingual, multitask benchmark for long context understanding.

[5] Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., & Gao, J. (2023). Model tells you what to discard: Adaptive kv cache compression for llms.

[6] Devoto, A., Zhao, Y., Scardapane, S., & Minervini, P. (2024). A Simple and Effective $ L_2 $ Norm-Based Strategy for KV Cache Compression. arXiv preprint arXiv:2406.11430.

[7] Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L. and Dong, Y. (2023). Longbench: A bilingual, multitask benchmark for long context understanding.

[8] Charikar, M. S. (2002, May). Similarity estimation techniques from rounding algorithms. 

[9] Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020). Reformer: The efficient transformer.
</content>
</thread_object>

</thread>


Metareview: