VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration

Authors: Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, Panpan Xu
Primary area: generative models
Keywords: KV Cache Compression, Vision-Language Models, Inference Acceleration, Sparsity, Modality

Abstract:
Vision-Language Models (VLMs) have demonstrated impressive performance across a versatile set of tasks. A key challenge in accelerating VLMs is storing and accessing the large Key-Value (KV) cache that encodes long visual contexts, such as images or videos. While existing KV cache compression methods are effective for Large Language Models (LLMs), directly migrating them to VLMs yields suboptimal accuracy and speedup. To bridge the gap, we propose VL-Cache, a novel KV cache compression recipe tailored for accelerating VLM inference. In this paper, we first investigate the unique sparsity pattern of VLM attention by distinguishing visual and text tokens in prefill and decoding phases. Based on these observations, we introduce a layer-adaptive sparsity-aware cache budget allocation method that effectively distributes the limited cache budget across different layers, further reducing KV cache size without compromising accuracy. Additionally, we develop a modality-aware token scoring policy to better evaluate the token importance. Empirical results on multiple benchmark datasets demonstrate that retaining only 10% of KV cache achieves accuracy comparable to that with full cache. In a speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding by up to 7.08x, while reducing the memory footprint of KV cache in GPU by 90%.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730093168265

thread_type:
official_review

<thread_object>
creator: reviewer_x8f2

<content>
summary:
The paper tackles the challenge of accelerating vision-language models (VLMs) by leveraging a sparse Key-Value (KV) cache. Unlike existing compression methods for large language models (LLMs), which are less effective for VLMs, the proposed VL-Cache takes advantage of unique sparsity patterns in visual and text tokens. It uses layer-adaptive budget allocation to allocate cache resources based on sparsity levels and modality-aware token scoring to determine token importance with post-vision text tokens. By retaining only the most crucial cache entries, VL-Cache effectively reduces memory usage and latency. Experimental results on three lmms-eval tasks demonstrate the method’s effectiveness.

strengths:
The paper is well-written and easy to follow. The sparsity analysis of prefilling and decoding effectively motivates the use of adaptive sparsity. The proposed method achieves significant speedup in decoding, particularly with long prefill tokens.

weaknesses:
Please see the questions.

questions:
1.	Some notations in the proposed method lack clarity. For instance, the meanings of $\Gamma$ and $Z$ are not sufficiently explained, making it difficult to understand their roles in the algorithm. Providing more detailed explanations would improve the clarity of the presentation.

2.	The authors state that the maximum batch size is constrained by peak memory usage during prefill rather than by the KV cache size, suggesting that compressing the KV cache does not increase the batch size. Does this imply that the proposed KV cache compression does not actually reduce overall memory usage? If so, why is that the case? This raises concerns about whether the title “KV Cache Compression” accurately reflects the method’s ability to save GPU memory, as the proposed approach appears to be more of a sparse computation technique applied during decoding.

3.	In Table 3, the reported speedup is based on generating a total of 100 output tokens. It would be helpful if the authors could provide the exact prefill and decoding times separately. Given that with a prompt length of 128k and only 100 output tokens, the prefill time is likely to dominate the overall processing time, having this detailed breakdown would clarify the impact of the proposed method on different stages of the inference process.

4.	In Section 4.2, the authors point out that accumulated attention has the limitation of assigning disproportionately high scores to earlier tokens due to summation over the full query dimension. To address this, they propose accumulated post-vision attention. However, there is another approach [1] that addresses the same issue using normalized attention. It would be helpful if the authors could include more discussion on this alternative method and provide additional experimental comparisons to better illustrate the benefits of their approach.

5.	The experimental evaluation is insufficient, as it lacks comparisons with two important baselines for VLM acceleration: FastV [1] and HiRED [2]. Additional comparisons with these methods are needed to strengthen the evaluation.

6.	Although the proposed method targets vision-language models, the current evaluation lacks coverage of a comprehensive range of tasks. To better demonstrate its effectiveness, it would be beneficial for the authors to include additional results on video benchmarks, such as Video-MME [4]. This would help showcase the method’s applicability to video tasks.

Reference:

[1] ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification. NeurIPS 2024.

[2] An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. ECCV 2024.

[3] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments. arXiv 2024.

[4] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730216609587

thread_type:
official_review

<thread_object>
creator: reviewer_kipr

<content>
summary:
The authors aim to accelerate VLMs by reusing KV caches for encoding visual contexts. The authors observe the limitations of existing KV cache compression methods and propose a specific version for VLMs. Based on the observation of sparsity patterns in prefilling and decoding stages, the main idea is to design a cache allocation method that balances the cache size and model accuracy. Evaluations based on vision-language benchmarks show the performance of model accuracy and speedups.

strengths:
1.	The authors identify the I/O bottleneck between GPU’s HBM and SRAM in the decoding stage of VLMs, motivating the innovative design of KV cache compression. This understanding helps to address the challenge of scaling VLMs and optimizing their performance.
2.	Through Figure 1, the authors illustrate the distinctive attention sparsity patterns between VLMs and LLMs, emphasizing the need for modality-aware compression techniques tailored for VLMs.
3.	I have learned a lot from the author's observations. The KV cache budget and compression ratios differ in layers and modalities. This provides good insights into the optimization of VLM cache compression.
4.	To compress the VLM cache, the authors propose two key methods: a sparsity-aware cache budget allocation strategy and a modality-aware token scoring policy, to optimize the cache budget allocation and token score measurement, respectively.
5.	The experimental results show that the proposed methods achieve significant inference speedups and GPU memory savings.

weaknesses:
1.	In Figure 1, the authors measure attention scores to verify the modality boundary, but the specific methodology for calculating these scores remains unclear. Please provide additional details on the measurement process.
2.	The term “post-vision attention” is frequently used but not explicitly defined, making the relevant sentences difficult to follow (“the attention patterns from the output language tokens is much closer aligned with the language tokens that follow the visual tokens in the prompt (the post-vision attention) rather than the visual tokens themselves”). A clear description of post-vision attention at its first occurrence would enhance the paper's readability.
3.	The threshold filter in Section 3.1 utilizes the hyper-parameter p to control sparsification. Conducting ablation studies on the impact of different p values would provide deeper insights into the robustness and sensitivity of the proposed method.
4.	To Algorithm 1, while it dynamically allocates the cache budget based on each prompt, its computational complexity raises my concern. Please analyze the computational overhead (in lines 11 to 14).
5.	Besides, Algorithm 1 does not explicitly reflect the modality-aware design as has been highlighted in this paper. Please clarify how the algorithm incorporates modality properties.
6.	In experiments, the I/O traffic amount caused by KV caching should be reported.

questions:
Please refer to the questions in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730360462538

thread_type:
official_review

<thread_object>
creator: reviewer_xvzl

<content>
summary:
This paper introduces a modality-aware KV cache compression method for Vision-Language Models (VLM). It introduces a dynamic cache budget allocation mechanism and employs post-vision language tokens to compute attention scores.

strengths:
The insight that the attention patterns of generated tokens closely align with language tokens is valuable.

weaknesses:
1.	Algorithm 1 only outlines two procedures without detailing how these procedures are integrated into the algorithm.
2.	Definition 3.1 and the following text is unclear. The terms Psi(S) and S_Psi are used without clear explanation. A precise definition of Cache Hit Rate is essential.
3.	Figure 4 lacks clarity. It does not effectively illustrate how the budget is allocated or how post-vision attention is utilized.
4.	In the related work, ZipCache proposes a policy using normalized attention scores. This should be considered and compared.
5.	The KV cache for generated tokens is not addressed.
6.	The performance of VL-Cache is not convincingly demonstrated in Table 2. VL-Cache does not significantly outperform other methods, while H2O has achieved the strongest performance on many benchmarks. It is recommended to test VL-Cache on more challenging benchmarks.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644021321

thread_type:
official_review

<thread_object>
creator: reviewer_xu5m

<content>
summary:
This paper addresses the challenge of efficiently storing and accessing large Key-Value (KV) caches in Vision-Language Models (VLMs), which are crucial for handling visual contexts like images or videos. Existing KV cache compression methods developed for Large Language Models (LLMs) don't perform optimally when applied to VLMs. To solve this, the authors propose VL-Cache, a compression method designed specifically for VLM inference. VL-Cache utilizes a layer-adaptive, sparsity-aware cache allocation and a modality-aware token scoring approach to reduce cache size without losing accuracy. Experimental results show that this method retains comparable accuracy with only 10% of the original cache, accelerates token generation by up to 2.33x, decoding by up to 7.08x, and reduces GPU memory usage by 90%.

strengths:
1. The experimental results are very good; the performance is significantly better compared to other KV-cache methods.
2. Sparsity-aware layer-wise KV cache allocation and modality-aware token scoring are meaningful and have also been proven effective in the experiments.

weaknesses:
1. The writing is somewhat complex and unclear; it takes a few careful readings to understand. Adding more illustrations to explain the method would improve it.
2. For MathVista, it seems that the KV-cache isn’t important? Even with only 1% of it, the performance is quite good, and sometimes the performance with 100% KV-cache isn’t even the best. Could you explain that?
3. In terms of writing, Section 3's PRELIMINARY EXPERIMENT is too long, while there’s relatively little written about the methodology. This writing style increases the burden on the reader. Readers likely want to see the method as soon as possible, so the focus should be on discussing your approach.

questions:
Please see the section on weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732436471692

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
We would like to thank all reviewers for their time and effort in reviewing our work. We are glad that our work receives the following recognition from our reviewers:

1. In terms of motivation and methodology: “the insight that the attention patterns of generated tokens closely align with language tokens is valuable” (XVzL); the observation that “KV cache budget and compression ratios differ in layers and modalities” provides “good insights into the optimization of VLM cache compression” (KiPr); “the sparsity analysis of prefilling and decoding effectively motivates the use of adaptive sparsity” (X8F2);“sparsity-aware layer-wise KV cache allocation and modality-aware token scoring are meaningful and have also been proven effective in the experiments” (XU5m). 
2. In terms of experimental results: “the experimental results are very good; the performance is significantly better compared to other KV-cache methods” (XU5m); “the experimental results show that the proposed methods achieve significant inference speedups and GPU memory savings” (KiPr); “the proposed method achieves significant speedup in decoding” (X8F2).

We show additional experimental results requested by multiple reviewers in this reply and have addressed other concerns in individual threads. We have also uploaded an updated version of our paper for further reference.

Both Reviewer XVzL and X8F2 have requested comparisons of our method with ZipCache’s scoring policy, which uses normalized attention scores to evict tokens. We show the results in the table below. We found that our method presents consistently better performance with 10% KV cache. In our updated paper, we also show that using post vision attention (our method) leads to higher cache hit rate as well. Interestingly, normalized attention scores proposed in ZipCache do not consistently improve upon accumulated attention scores used in H2O as shown in the ZipCache paper when tested on vision-language tasks, which highlights the importance of specialized scoring policies as we proposed in VL-Cache for VLMs.

Dataset | Model | Method | 1% | 5% | 10% | 20% | 40% | 60% | 80% | 100%
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
Coco-Caption (Metric: CIDEr) | LLaVA-Mistral-7B | VL-Cache | 2.64 | **82.53** | **100.36** | **102.06** | **99.93** | **101.07** | 100.08 | 100.68
Coco-Caption (Metric: CIDEr) | LLaVA-Mistral-7B | ZipCache | **4.4** | 53.00 | 66.41 | 92.36 | 99.51 | 100.71 | **102.86** | 100.68
Coco-Caption (Metric: CIDEr) | LLaVA-1.6-34B | VL-Cache | 0.00 | **120.11** | **137.35** | **139.42** | 138.58 | 139.19 | **138.01** | 135.07
Coco-Caption (Metric: CIDEr) | LLaVA-1.6-34B | ZipCache | 0.00 | 12.92 | 131.61 | 134.98 | **140.59** | **140.63** | 137.90 | 135.07
DocVQA (Metric: ANLS) | LLaVA-Mistral-7B | VL-Cache | **43** | **59** | **62** | **64** | **67** | 67 | **67** | 68
DocVQA (Metric: ANLS) | LLaVA-Mistral-7B | ZipCache | 41 | 40 | 60 | 45 | 65 | **68** | **67** | 68 
DocVQA (Metric: ANLS) | LLaVA-1.6-34B | VL-Cache | **41** | **82** | **84** | **85** | **85** | **85** | **85** | 85
DocVQA (Metric: ANLS)| LLaVA-1.6-34B | ZipCache | 0 | 55 | 74 | 81 | **85** | **85** | **85** | 85
MathVista (Metric: ACC) | LLaVA-Mistral-7B | VL-Cache | **38** | **36** | **39** | **40** | **40** | **42** | **42** | 41
MathVista (Metric: ACC) | LLaVA-Mistral-7B | ZipCache | 35 | 35 | **39** | 34 | 40 | 42 | 40 | 41
MathVista (Metric: ACC) | LLaVA-1.6-34B | VL-Cache | **41** | **42** | **42** | **44** | **45** | **45** | 42 | 43
MathVista (Metric: ACC) | LLaVA-1.6-34B | ZipCache | 28 | 35 | 41 | 41 | 42 | 43 | **43** | 43
</content>
</thread_object>

</thread>
<thread>
creation date:
1732733372790

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank-You Note

comment:
To all reviewers,

Thanks again for taking the time to review our paper and rebuttal. Your constructive feedbacks have helped us greatly in improving the quality of our submission, and we really appreciate that!


Best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces VL-Cache, a novel KV cache compression method tailored for accelerating vision-language model inference, which significantly reduces memory usage and latency while maintaining high accuracy.

After rebuttal and discussions, this paper receives all positive ratings. After carefully  reviewing the paper and all the reviewers' comments and discussions, The AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.