SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction

Authors: Xuan Zhang, Cunxiao Du, Chao Du, Tianyu Pang, Wei Gao, Min Lin
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Long-Context, KV Cache

Abstract:
Recent advancements in large language models (LLMs) have extended their capabilities to handle long contexts. However, increasing the number of model layers and the length of input sequences significantly escalates the memory required to store key-value (KV) cache, posing challenges for efficient inference. To mitigate this issue, we present SimLayerKV, a simple yet effective method that reduces inter-layer KV cache redundancies by selectively dropping cache in identified lazy layers. Our approach is based on the observation that certain layers in long-context LLMs exhibit "lazy" behavior, contributing less to modeling long-range dependencies compared to non-lazy layers. By analyzing attention weight patterns, we find that the behavior of these lazy layers is consistent across tokens for a given input. This insight motivates our SimLayerKV, which identifies lazy layers and reduces their KV cache accordingly. SimLayerKV is training-free, generalizable, and can be implemented with only seven lines of code. We conduct extensive experiments on three representative LLMs, e.g., LLaMA2-7B, LLaMA3-8B, and Mistral-7B across 16 tasks from the LongBench benchmark. The results demonstrate that SimLayerKV achieves a KV cache compression ratio of 5$\\times$ with only a 1.2\% performance drop when combined with 4-bit quantization.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730549737695

thread_type:
official_review

<thread_object>
creator: reviewer_uyjg

<content>
summary:
The paper introduces SimLayerKV, a method aimed at addressing the increased memory requirements for storing key-value (KV) caches in large language models (LLMs) that handle long contexts. It identifies "non-lazy" layers in these LLMs that contribute significantly to modeling long-range dependencies. By implementing a novel KV cache eviction strategy that selectively drops caches in less critical layers based on attention weight patterns, SimLayerKV effectively reduces memory usage related to inter-layer KV cache redundancies.

strengths:
- The study identifies non-lazy layers within large language models and introduces an innovative KV cache eviction method that significantly reduces the memory usage of KV caches.
- The proposed approach is training-free, generalizable, and can be implemented in just seven lines of code, demonstrating its ease of application.
- Experiments conducted on three representative LLMs across 16 tasks from the LongBench benchmark illustrate that SimLayerKV, when combined with 4-bit quantization, achieves high KV cache compression ratios with only a minimal drop in performance.

weaknesses:
- The paper notes that KV caches in non-lazy layers must be fully retained, which does not fundamentally solve the substantial overhead caused by KV caches. This could still result in high memory usage. Methods like H2O are able to drastically reduce the memory footprint of KV caches.
- There is a lack of comparison with existing advanced KV cache eviction methods, such as H2O, SnapKV, and PyramidKV.
- The method proposed by the paper appears trivial, and it is unclear whether non-lazy layers will still be present in larger models or how this phenomenon may relate to different datasets.

questions:
- What is the underlying cause of non-lazy layers?
- Many papers have analyzed the relationships between cross-attention layers. How does this relate to the non-lazy layers identified in this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618331962

thread_type:
official_review

<thread_object>
creator: reviewer_fsab

<content>
summary:
This paper proposes a simple and effective method, SimLayerKV, which reduces inter-layer KV cache redundancy by selectively discarding the cache of “lazy” layers. The authors found that, in long-context LLMs, certain layers contribute minimally to modeling long-distance dependencies, displaying “lazy” behavior. By analyzing attention weight patterns, they observed that these lazy layers consistently exhibit this behavior throughout the generation process for a given input. SimLayerKV identifies these lazy layers and reduces their KV cache accordingly without altering the cache of non-lazy layers or merging caches across layers. Extensive experiments on three representative LLMs demonstrate that SimLayerKV, combined with 4-bit quantization, achieves a 5x KV cache compression rate with only a 1.2% performance drop.

strengths:
1. This paper addresses a good research topic: efficient LLM inference.
    
2. The paper is well-organized.
    
3. The proposed method is clearly presented.

weaknesses:
1. The designed identification algorithm does not meets the observation. It will also treat the layer that attend all of the tokens, including the intial, the recent tokens, and other tokens in the sequence as the lazy layer.
    
2. Why only use the only one token rather than few tokens, as in prefilling stage, in decoding to detect the lazy layers?
    
3. It seems that there are a lot of parameters need to be manually set in this algorithm, including the X_intial, X_recent, and W_last, making the designed algorithm less practical. Moreover, these parameters are also not covered in the ablation study and the authors did not explain how they configure them in the experiments.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730635921462

thread_type:
official_review

<thread_object>
creator: reviewer_yxrn

<content>
summary:
This paper aims to optimize the efficiency of the LLM inference by reducing the KV embeddings that are needed to be cached in the memory. Based on the observations of the attention weights of different layers, the authors propose an algorithm to identify the “lazy” layers, whose attentions weights of the initial tokens and recent tokens are predominately larger than those of other tokens, and then trim the KV cache of these lazy layers so as to reduce the memory cost. Some experiments on LLaMA2-7B, LLaMA-3-8B, and Mistral-7B show the effectiveness of the proposed method.

strengths:
1. The research question is interesting and promising for the highly-efficient inference of LLM. Since the KV cache grows linearly with the number of layers, it’s natural to optimize this problem from the layer’s perspective. Recently, increasingly more studies are focused on this topic.
2. From what I understand, the proposed method might be able to save the memory cost not only for decoding, but also prefilling, which is useful for those case where the prompt is way longer than the generated response.
3. The paper has reasonable design of experiments.

weaknesses:
1. The way to identify the lazy layers involves quite a few hyper-parameters, such as the length of $X_{last}$, $X_{initial}$, $X_{recent}$, and $\delta$. There is no systematic way to tune those hyper-parameters, which makes it too empirical.
2. Since there are two ways to identify the lazy layers, does it mean the lazy layers in prefilling and decoding are different? 
3. Can we identify the lazy layer on-the-fly during the inference? because in practice we usually do not have a small subset of data to identify the lazy layers first, then conduct the real inference.
4. A big missing part of this paper is the result of efficiency, like token throughput, latency or memory cost of the proposed method, since the goal of the algorithm is to optimize the efficiency. 
5. There are quite a few related works trying to identify the optimal KV cache strategies for different layers [1][2][3], it’s unclear what are the cons and pros of those methods compared with the proposed one.

[1] https://arxiv.org/pdf/2410.10819

[2] https://arxiv.org/pdf/2404.04793

[3] https://arxiv.org/pdf/2310.01801

questions:
please refer to the Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1731005290441

thread_type:
official_review

<thread_object>
creator: reviewer_ktmt

<content>
summary:
With the recent increase in context lengths of large language models (LLMs), the memory required to store key-value (KV) cache has grown significantly. This paper aims to reduce inter-layer KV cache redundancies by selectively dropping caches in identified "lazy" layers. The paper first observes that certain layers in long-context LLMs are "lazy," primarily focusing on semantically unimportant tokens (the initial few tokens and the most recent few tokens) when performing attention. Furthermore, lazy layers are less important than non-lazy layers in long-context generation. The paper also finds that the laziness behavior is consistent across tokens for a given input and is easily identifiable. Based on these observations, the paper proposes SimLayerKV, a simple strategy that identifies lazy layers at either the prefill or decode stage and trims the lazy layers to reduce inter-layer KV cache redundancy. To demonstrate SimLayerKV's effectiveness, the paper evaluates it on popular benchmarks like LongBench and Ruler, showing a maximum compression ratio of 5x with only a 1.2% drop in performance. Unlike existing works, this paper is distinct in leveraging inter-layer KV cache redundancies and requires no additional training.

strengths:
- The paper is novel in its exploration of better inter-layer KV cache trimming without additional training.
- The paper is well written

weaknesses:
- It is not clear why SimLayerKV is orthogonal to existing KV cache trimming or compressing methods
- The compression ratio of 1.6x on average without 4 bit Quantization is not significant
- There is performance degradation on more complex tasks
- The proposed way of identifying lazy layers is not flexible enough

questions:
- In Figure 2, the paper demonstrates the attention patterns during long-context generation in layers 0, 10, 20, and 30, thereby categorizing layers into two types: lazy and non-lazy. How do the insights gained regarding attention patterns in this paper compare to prior work, such as MInference1.0, which identifies three sparse patterns (A-shape, Vertical-Slash, and Block-Sparse)? Do these findings align?

- Could you clarify the lazy layer identification algorithm further? The paper suggests two methods for identifying lazy layers—during the prefill stage and the decode stage. Specifically, when is this identification executed during online inference? How frequently is it updated during benchmarking, and how often would it be updated in an online inference scenario? For multi-round conversations, where inputs from a single user may vary significantly between rounds, how does SimLayerKV address this in its design?

- In Section 2 on related work, the paper references prior research on KV cache trimming, compression, and selection. It includes comparisons with MiniCache, StreamingLLM, and SnapKV as baselines. Why does the paper not include comparisons with more intra-layer trimming methods? Can you provide examples illustrating why inter-layer methods are orthogonal to intra-layer methods? Additionally, have you conducted experiments to demonstrate that integrating these two approaches does not significantly degrade performance?

- In Table 1, the paper compares the performance of SimLayerKV and baseline methods. On the LongBench benchmarks, why does SimLayerKV+Q outperform SimLayerKV in many cases, especially in nearly half of the tests for the Mistral-7B-Instruct model?

- What configuration is used for the StreamingLLM baseline? In Table 6, StreamingLLM achieves a 6-8x higher compression ratio than SimLayerKV; could you provide additional comparison results where both methods achieve similar compression ratios?

- In Section 6.3 on Ruler experiments, SimLayerKV shows a performance drop (8.2% on average) on Multiple Queries NIAH and significant degradation in Common Words Extraction (from 75.1% to 48.6% for a 32k context length). The paper attributes this to the data-dependent nature of lazy layer identification, specifically a fixed selection of lazy layers across the entire benchmark task. Have you conducted experiments to verify this hypothesis? Additionally, have you considered dynamically updating the selected lazy layers during runtime? What are possible solutions to fix this issue?

- What is the overhead introduced by lazy layer identification in terms of latency, and how does it affect the system’s overall throughput? Have you conducted end-to-end serving experiments to demonstrate SimLayerKV's deployment potential?

- How might performance and compression ratios change if individual heads within a layer are considered? Have additional experiments been conducted on this aspect? Would controlling for smaller granularities potentially improve performance, or could it lead to worse outcomes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732352844731

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Revision

comment:
We thank all reviewers for their constructive feedback, and we have responded to each reviewer individually. We have also uploaded a **Paper Revision** including additional results and illustrations:

* $\\textrm{\\color{blue}Section 6.3 lines 483-485}$ (Page 9): Throughput improvement of our SimLayerKV. (Reviewer **KtMT**, Reviewer **YxRN**)
* $\\textrm{\\color{blue}Table 5}$ (Page 16): Pseudo code in torch style for our SimLayerKV-prefilling with flash attention. (Reviewer **KtMT**)
* $\\textrm{\\color{blue}Table 8}$ (Page 16):  Experiment results on combining SimLayerKV and intra-layer KV cache compression
method SnapKV. (Reviewer **KtMT**, Reviewer **UyjG**)
* $\\textrm{\\color{blue}Table 10}$ (Page 19):   Performance comparison of SimLayerKV and SqueezeAttention under similar compression ratio. (Reviewer **YxRN**)
* $\\textrm{\\color{blue}Table 11}$ (Page 19):   Performance on larger models (LLaMA3-70B-Instruct), and compression ratio across
different datasets in Longbench benchmark. (Reviewer **UyjG**)
* $\\textrm{\\color{blue}Table 12}$ (Page 19):   Effect of hyperparameters on lazy layer identification using LLama3-8B-Instruct. (Reviewer **KtMT**, Reviewer **YxRN**, Reviewer **fSad**)
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:
The paper introduces SimLayerKV, a method for reducing KV cache memory usage in LLM inference by identifying and compressing "lazy" layers that primarily attend to initial and recent tokens. The approach requires no model training and can achieve significant memory reduction while maintaining model performance.

Main Strengths:

- Novel approach focusing on inter-layer KV cache compression
- Training-free and simple to implement
- Compatible with other compression methods like SnapKV
- Demonstrates good empirical results across multiple models and benchmarks
- Significant throughput improvements (up to 2.17x)

Main Weaknesses:

- Several hyperparameters require tuning, though additional experiments show robustness
- Some performance degradation on complex tasks like Multiple Queries NIAH
- Lack of comprehensive runtime comparisons with state-of-the-art methods initially
- Theoretical analysis of lazy layer behavior could be strengthened