VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration

## Abstract

Abstract Vision-Language Models (VLMs) have demonstrated impressive performance across a versatile set of tasks. A key challenge in accelerating VLMs is storing and accessing the large Key-Value (KV) cache that encodes long visual contexts, such as images or videos. While existing KV cache compression methods are effective for Large Language Models (LLMs), directly migrating them to VLMs yields suboptimal accuracy and speedup. To bridge the gap, we propose VL-Cache, a novel KV cache compression recipe tailored for accelerating VLM inference. In this paper, we first investigate the unique sparsity pattern of VLM attention by distinguishing visual and text tokens in prefill and decoding phases. Based on these observations, we introduce a layer-adaptive sparsity-aware cache budget allocation method that effectively distributes the limited cache budget across different layers, further reducing KV cache size without compromising accuracy. Additionally, we develop a modality-aware token scoring policy to better evaluate the token importance. Empirical results on multiple benchmark datasets demonstrate that retaining only 10% of KV cache achieves accuracy comparable to that with full cache. In a speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding by up to 7.08x, while reducing the memory footprint of KV cache in GPU by 90%.

## 1 Introduction

Vision-Language Models (VLMs) have recently emerged as powerful tools for a broad range of multi-modal tasks (Liu et al., ; Chen et al., ; Bai et al., ). As these models improve at processing long visual context – such as high-resolution images, multiple images and multi-frame videos (Li et al., ) – the number of visual tokens increase rapidly. Consequently, deploying VLMs demands substantial GPU memory capacity, bandwidth, and computational resources, leading to high inference latency and cost.

Similarly to Large Language Models (LLMs) (Chang et al., ), VLMs decode tokens sequentially in an auto-regressive loop. The key and value pairs of the input prompt and of the generated output tokens are stored in GPU memory (where they are known as the KV cache) and reused at each decoding step to avoid recomputation. As the context length grows, KV cache not only occupies a larger amount of GPU memory, but also increases inference latency due to data movement between GPU’s high-bandwidth memory (HBM) and its on-chip memory (SRAM) in each decoding step (Dao, ; Hong et al., ). This is a significant challenge for scaling VLMs, because large KV cache is required to hold the input images and video frames. For example, with a batch size of four prompts, five images in each prompt, and each image using 2K visual tokens, serving the LLaVA-1.6-34B model requires 110 GB of HBM capacity just for the KV cache of visual tokens.

Figure: (a) LLMs Attention Score Matrix
Refer to caption: extracted/5963551/figures/llm_attn.png

KV cache sparsification is a promising method of reducing GPU memory requirements and inference latency for LLMs (Zhang et al., ; ; Yang et al., ; He et al., ; Li et al., ). Prior methods leveraged sparsity in the attention scores matrix of LLMs to evict insignificant tokens from KV cache, while preserving important contextual information needed for output token generation. Besides reducing the GPU memory footprint, this method also reduces the token generation latency by minimizing the amount of data movement and floating-point operations (FLOPs) in the attention layer.

We discover that VLMs and LLMs exhibit significantly different attention sparsity patterns, as illustrated in Figure 1. In VLMs, a clear modality boundary exists between visual tokens and the subsequent language tokens. Specifically, the attention patterns from the output language tokens is much closer aligned with the language tokens that follow the visual tokens in the prompt (the post-vision attention) rather than the visual tokens themselves. Previous KV compression methods are modality-unaware and incorrectly conflate these two sources of attention scores. Unsurprisingly, our experiment in Figure 5 shows that KV cache compression methods designed for LLMs have suboptimal results when applied to VLMs.

Next, we measure that attention sparsity ratios vary in 70% to 99% range across transformer layers. Our observations in Section 3.1 show that previous KV compression methods suboptimally distribute the cache budget between layers, which leads to either under- or over-compression of information in KV cache. Finally, we observe that sparsity ratios differ between visual and language tokens as well, so an optimal cache budget allocation between layers can not be done before sparsity is measured in a particular prompt.

To address these gaps, we propose VL-Cache, a novel KV cache compression method for accelerating Vision-Language Model inference. Our method fully utilizes cross-modality and layer-wise attention sparsity patterns in VLMs to dynamically prune KV cache with minimal loss of task-level accuracy. To the best of our knowledge, this is the first work that investigates attention sparsity in VLMs and the first work that specifically optimizes KV cache compression for VLMs. In particular, we propose (1) sparsity-aware KV cache budget allocation between the transformer layers at inference time, and (2) modality-aware scoring policy for token eviction.

Experimental results in Section 5.1 show that our method retains 98% of the original task-level accuracy while using only 10% of KV cache for the majority of vision-language tasks from several datasets. In a speed benchmark, our method reduces end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding in particular by up to 7.08x, while allocating 90% less GPU memory for KV cache. In inference scenarios where KV cache size is the limiting factor to higher concurrency, VL-Cache enables up to 10x higher concurrency after KV cache compression.

Overall, our contributions in this paper are summarized as follows:

- •
VLMs Attention Sparsity Profile. We uncovered unique attention sparsity patterns in the prefill and decoding phases of VLM inference on a variety of multi-modal tasks, which is drastically different from those of LLMs.
- •
Layer-Adaptive Sparsity-Aware Cache Budget Allocation. We proposed to allocate each layer’s KV cache budget with consideration of that layer’s attention sparsity at inference time. Even with 10% KV cache budget, we retained high accuracy on popular benchmark tasks.
- •
Modality-aware Token Scoring Policy. We observed that language-to-vision attention robustly includes information about the importance of visual tokens. We treat visual and language attention scores differently to better retain important tokens.

## 2 Background

In this section, we detail the inference procedure of the widely adopted VLM architecture with vision & language input and text output, where image tokens are projected as soft prompts (Li et al., ; Team et al., ; Islam & Moushi, ; Bai et al., ; Anthropic, ). We also introduce the formulation of KV cache compression and the approach of existing works (Zhang et al., ; ; Liu et al., ; Ge et al., ; Tang et al., ; Li et al., ; Lee et al., ; Yu et al., ).

### 2.1 VLM inference

#### Prefill phase.

The input to VLMs includes both images and language, where images are processed by the visual encoder to generate visual tokens. Subsequently, a projection layer, such as a simple multi-layer perceptron, maps these visual tokens to a unified embedding space. Meanwhile the language prompt is fed to a tokenizer and embedding layer to create the initial hidden state for language tokens. For notation simplicity, we denote a sequence of $m$ prompt tokens, including both visual and language ones, as $\{x_{1},...,x_{m}\}$. These tokens are processed by the language model in parallel to calculate the probability of the first decoded token $P_{\theta}(x_{m+1}|x_{1},...,x_{m})$. Simultaneously, the key vectors $\{k_{1}^{(l)},...,k_{m}^{(l)}\}$ and value vectors $\{v_{1}^{(l)},...,v_{m}^{(l)}\}$ at each transformer layer $l$ are cached in GPU memory to avoid recomputation in the next decoding step.

#### Decoding phase.

Once decoding starts, the language model in VLMs takes effect and generates one token per step in an auto-regressive loop. At step $i$, the model receives the token $x_{m+i}$ as input and calculates the probability $P_{\theta}(x_{m+i+1}|x_{1},...,x_{m+i})$. Each decoding step involves generating new key vectors $k_{m+i}^{(l)}$ and value vectors $v_{m+i}^{(l)}$, which are appended to the previously cached key-value pairs for each layer, resulting in $\{k_{1}^{(l)},...,k_{m+i}^{(l)}\}$ and $\{v_{1}^{(l)},...,v_{m+i}^{(l)}\}$. In case of long contexts, such as multiple or high resolution images, the key-value cache can grow significantly larger than the model parameters and other intermediate tensors, making memory capacity and bandwidth major performance bottlenecks.

### 2.2 KV Cache Compression

To address the bottleneck of storing and accessing the large KV cache during decoding, many researchers have focused on KV cache compression to maintain only a subset of the full KV cache for more efficient decoding while minimizing the accuracy loss. There are two main design dimensions to such algorithms: how many cache tokens should be kept at each layer, and which tokens to evict during compression.

#### Budget Allocation.

Since the transformer architecture consists of multiple identical layers, a straightforward strategy is to allocate an equal budget of KV cache slots to each layer (Xiao et al., ; Zhang et al., ; He et al., ). More recently, inspired by the observation that removing cache tokens at different layers results in varying degrees of performance loss, PyramidKV (Zhang et al., ) and PyramidInfer (Yang et al., ) proposed a decay schedule to assign more cache budget to shallower layers and observed improved accuracy results relative to the baseline with equal budget per layer.

#### Token Scoring Policy.

For a given cache token budget, a token scoring policy $\psi$ is required to rank the importance of KV cache tokens and decide which tokens to keep. Let $n$ be the count of cache tokens at layer $l$, and $S=\{0,1,...,n\}$ be the indices of these cache tokens. We define $\psi:S\rightarrow\mathbb{R}^{n}$, with the output being the scores for current cache tokens. Given these scores, indices with top-$k$ scores, $S_{\psi}\coloneqq\{i_{1},i_{2},...,i_{k}:\psi(S)_{i_{j}}\geq\psi(S)_{x\in[n]
\backslash\{i_{1},i_{2},...,i_{k}\}}\}$, where $k\in[1,n)$, are selected. Recent works have shown that attention scores serve as an effective source to design such policies. StreamingLLM (Xiao et al., ) found that high attention scores are frequently assigned to initial tokens and achieved length generalization by keeping only the initial and most recent cache tokens while removing intermediate ones. H2O (Zhang et al., ) uses accumulated attention scores to identify crucial tokens to retain.

We make the observation that, fundamentally, KV cache compression methods only work because inference with a transformer layer is a sparse process. Therefore, in this work, we leverage attention sparsity as the unified guidance to design both the cache budget allocation mechanism and the token eviction policy for KV cache compression in VLMs.

## 3 Preliminary Experiment

The attention mechanism for visual and language tokens is a key aspect of VLMs. Therefore, further optimization of the KV cache compression methods for VLMs requires careful investigation of the attention patterns with relevant input prompts. Motivated by this need, we conducted preliminary experiments to explore the VLM attention. We randomly sampled data from three multi-modal datasets — DocVQA (Mathew et al., ), MathVista (Lu et al., ), and Coco-Caption (Chen et al., ). These datasets cover of a wide range of visual tasks, including OCR, visual diagram reasoning, and world knowledge. We selected one of the state-of-the-art VLMs — LLaVA-Mistral-7B (Liu et al., ) — and recorded the attention score matrix until the generation process completes. Initially, we investigate attention sparsity across layers, and then propose two metrics to quantitatively assess how the model attends to visual and language tokens in each layer and evaluate different token scoring policies. We conclude our main findings in the end of this section which lead to our algorithm design described in Section 4.

### 3.1 Measuring Attention Sparsity

In this section, we measure the sparsity of the attention score matrix in different transformer layers during the prefill and decoding phases. First, we apply a filter with a relative threshold $p$ to the attention score matrix $A$:

$$ $\operatorname{\mathbf{ThresholdFilter}}(A,p)_{ij}=\begin{cases}A_{ij}&\;\;\; \text{if}\;A_{ij}\geq p\cdot\max_{j}(A_{ij})\\ 0&\;\;\;\text{otherwise}\end{cases}$ (1) $$

where threshold $p\in(0,1)$ controls the strength of induced sparsification, following Zhang et al. (). We also heuristically set $p=1\%$, such that the filtered-out scores have an insignificant impact on the output of the transformer layer. As will be discussed in Section 4.1, we also need to measure sparsity at inference time, so we prefer a metric with faster runtime. $\operatorname{\mathbf{ThresholdFilter}}$ is asymptotically faster than alternative methods of truncating a distribution (such as top-p and top-k), since it does not require the attention scores to be sorted. After filtration, we calculate sparsity $\operatorname{\mathbf{\gamma}}^{(l)}\in[0,1]$ of layer $l$ as count of zero entries, normalized by the size of the lower triangular portion of the attention scores matrix:

$$ $\operatorname{\mathbf{\gamma}}^{(l)}\coloneq\frac{\sum_{i\geq j}\mathbbm{1}[{ \operatorname{\mathbf{ThresholdFilter}}(A^{(l)},p)_{ij}=0}]}{|\{A_{ij}^{(l)}:i \geq j\}|}$ (2) $$

Figure: (a) Prefill Sparsity
Refer to caption: extracted/5963551/figures/prefill_sparsity.png

We calculated the average sparsity over a sample from the evaluation datasets, and then plotted the range of each layer’s sparsity with different attention heads in Figure 3. We observe that in the prefill phase (Figure 2(a)), the first two layers exhibit significantly lower sparsity and higher density compared to other layers. Additionally, some layers in the middle also demonstrate higher density than their neighboring layers. During the decoding phase (Figure 2(b)), a similar trend is observed as in the prefill phase, except for the second layer, which becomes much more sparse. Moreover, the Pearson Correlation Coefficient (James, ) is used to quantitatively compare the shape of layer-wise sparsity curves, resulting in an average shape similarity of 0.695, indicating that the curves are highly similar in both trend and magnitude. Based on these observations, we expect that the aggregate attention sparsity during the prefill phase could effectively predict the required KV cache size for robust decoding in each layer of the transformer model.

Previous methods, such as H2O (Zhang et al., ) and Keyformer (Adnan et al., ), allocate the same cache size across all layers, leading to insufficient allocation for high-information layers and wasteful allocation for low-information layers. In a more recent work, PyramidKV (Zhang et al., ) and PyramidInfer (Yang et al., ) monotonically decrease the KV cache size with depth of each layer, which we find to be suboptimal as well. Our observations reveal a more nuanced sparsity pattern, where different layers demand non-monotonically varying cache sizes to retain context information during decoding. We propose that sparsity based on $\operatorname{\mathbf{ThresholdFilter}}$ can serve as a reliable indicator for efficient cache allocation, allowing us to leverage insights from the prefill phase at inference time to optimize KV cache allocation for the decoding phase.

### 3.2 Measuring Cache Hit Rate

In Figure 1(b) of Section 1, we previously noted a distinct boundary in VLM attention between vision and language tokens in the prompt. Based on these observations, we hypothesize that retaining tokens based on post-vision prefill attention instead of the full-prompt prefill attention would preserve important cache tokens with higher recall. To validate this hypothesis, we define $\operatorname{\mathbf{CacheHitRate}}$ to measure the fraction of important tokens that are preserved after eviction when different scoring policies are applied. Let $m$ be the number of prompt tokens, $Q_{:m},K_{:m}\in\mathbb{R}^{m\times d}$ be the corresponding query and key matrices, and $Q_{m+1}\in\mathbb{R}^{d}$ be the query vector of the first decoding token.

#### Definition 3.1

(CacheHitRate), given $A_{m+1}\coloneq softmax(\frac{Q_{m+1}{K_{:m}}^{T}}{\sqrt{d}})\in\mathbb{R}^{m}$, we define:

- •
$\psi^{*}:S\rightarrow A_{m+1}$ as the optimal scoring function based on the true attention scores in decoding;
- •
$S_{\psi^{*}}$ as the top-$k$ tokens in the order given by $\psi^{*}$, which makes it the optimal policy for any $k$;
- •
$\operatorname{\mathbf{CacheHitRate}}$ as $\frac{|S_{\psi}\cap S_{\psi^{*}}|}{|S_{\psi^{*}}|}$, the percentage of $S_{\psi^{*}}$ that is also preserved by $S_{\psi}$, which denotes indices of cache tokens kept by any policy $\psi$.

We note that during inference, $A_{m+1}$ is not available because we aim to compress KV cache before the decoding pass to lift the memory bottleneck and boost decoding throughput. Thus, multiple $\psi$s have been proposed to approximate $\psi^{*}$ in recent works.

Accumulated Attention (prior work) uses the cumulative attention score along the query dimension, $\psi(S)\coloneq\sum_{i}softmax(\frac{Q_{:m}{K_{:m}}^{T}}{\sqrt{d}})_{i}$ to rank the importance of cache tokens.

Accumulated Sliding Window Attention (prior work) also computes accumulated attention scores along the query dimension but only over a recent window. This policy is defined as $\psi(S)\coloneq\sum_{i}softmax(\frac{Q_{m-w:m}{K_{m-w:m}}^{T}}{\sqrt{d}})_{i}$, where $w$ is a fixed window size.

Accumulated Post-vision Attention (ours) is based on the observation of the similarity between attention from post-vision prompt tokens and from decoding tokens as shown in Figure 1(b). We adopt the term post-vision to distinguish language tokens that follow visual tokens from instructions that may precede the images in the prompt. Formally, we use the portion of the attention scores $Q_{m-\tau:m}$, where $\tau$ is the count of language tokens that follow the vision tokens in the prompt. Our policy is defined by $\psi(S)\coloneq\sum_{i}softmax(\frac{Q_{m-\tau:m}{K_{m-\tau:m}}^{T}}{\sqrt{d}}
)_{i}$. During prefill, one can re-interpret our policy as a dynamic prompt-dependent sliding window with window size set to the length of the post-vision language prompt instead of a static prompt-independent value. From Figure 3, we find our scoring policy leads to consistently higher cache hit rate across layers than other policies.

## 4 VL-Cache Method

Motivated by observations from preliminary experiments, we introduce our method VL-Cache, which strategically combines sparsity-aware cache budget allocation and modality-aware token scoring policy to improve VLM’s performance under limited KV cache budget, in terms of both accuracy and efficiency.

Specifically, we use Post-vision Attention to compute both inter-layer sparsity and intra-layer token importance. The former guides how many cache tokens should be allocated at each layer, while the latter dictates which $k$ tokens within a layer should be kept due to their importance. A high-level description of our method is visualized in Figure 4. For brevity, we will use $A^{\prime}$ to denote the Post-vision Attention matrix in this section.

Figure: Figure 4: VL-Cache Overview. In the prefill stage, the cache budget for each layer is dynamically allocated according to the layer-wise sparsity. Then Post-vision Attention is employed to select both critical visual and language tokens.
Refer to caption: extracted/5963551/figures/arch.png

We argue that using Post-vision Attention has two important advantages. Firstly, computing layer-wise sparsity using Post-vision Attention results in much lower memory and latency $O(\tau m)$ compared to using the full attention matrix $O(m^{2})$ as in Zhang et al. (), since the visual tokens dominate the prompt length, $\tau\ll m$ for current vision-language tasks. Second, with a fixed cache budget, using Post-vision Attention leads to better preservation of important tokens, as measured by high $\operatorname{\mathbf{CacheHitRate}}$ in Figure 3. In this section, we will detail how we perform sparsity-aware KV cache budget allocation and cache token eviction guided by Post-vision Attention before reviewing the strong performance of VL-Cache in Section 5.

### 4.1 Sparsity-aware KV Cache Budget Allocation

Before determining the exact tokens to evict, we need to allocate the KV cache budget, which is the percentage of the KV cache to retain at each layer. Based on our observations from Section 3.1, we implement a sparsity-aware layer-wise KV cache allocation approach with two steps during the prefill phase. First, we apply $\operatorname{\mathbf{ThresholdFilter}}$ pruning (with $p=1\%$) to the Post-vision Attention scores and calculate the layer-wise sparsity. Second, given a target KV cache budget for the whole model, we distribute this budget across layers based on each layer’s sparsity ratio. This method optimizes the use of limited memory to store an appropriate amount of context information in each layer. Note that the allocation occurs only once and right after prefill phase, so the latency overhead can be amortized across multiple decoding steps.

Figure: Algorithm 1 Sparsity-Aware Cache Budget Allocation

In Algorithm 1, we present our KV cache budget allocation algorithm, where $\alpha$ is a hyper-parameter that encodes the desired KV cache budget for the whole model. When the accuracy is not satisfactory, a higher $\alpha$ can be used to keep more KV cache. Finally, we would like to point out the key difference between our algorithm and PyramidKV (Zhang et al., ): our allocation budget is customized for each prompt based on its sparsity pattern, instead of using a fixed layer-wise budget for all prompts, thus granting additional flexibility.

### 4.2 Modality-Aware Token Scoring Policy

After we decide the cache budget for layer $l$, we choose a subset of $k^{(l)}$ cache tokens from the full cache. Prior works (that were targeting KV cache compression for LLMs) have explored several scoring policies, such as Accumulated Attention in H2O (Zhang et al., ) and Accumulated Sliding Window Attention in Li et al. (); Zhang et al. (); Yang et al. (), as we discussed earlier in Section 3.2. In this section, we will explain in-depth why Accumulated Post-vision Attention provides a better scoring policy than prior policies for VLMs.

While Accumulated Attention serves as a simple yet effective baseline, one critical issue is that summation over the full query dimension unavoidably assigns high scores to earlier tokens. Observing this length bias, Accumulated Sliding Window Attention only sums attention scores over a recent window. We confirm that this indeed leads to higher $\operatorname{\mathbf{CacheHitRate}}$, as shown in Figure 3. However, as we can see in Figure 1(a) and 1(b), VLMs have a significantly different attention pattern than LLMs. Along the query dimension, we observe a clear modality boundary: visual tokens attend to other visual tokens rather uniformly while language tokens only pay attention to a few visual tokens with high concentration. If we were to apply Accumulated Attention as the scoring policy, the signal for critical cache tokens will be buried as we sum scores from all the preceding visual tokens. Using a recent window helps, but a fixed window size can easily be too large or too small as the length of the post-vision prompt varies from case to case. Therefore, we introduce Accumulated Post-vision Attention as an optimized scoring policy for VLMs by implementing a dynamic, prompt-specific window size. The advantage of our scoring policy is evident by its high $\operatorname{\mathbf{CacheHitRate}}$ in Figure 3, and will be further demonstrated through accuracy comparisons in Section 5.

## 5 Experiments

In our experiments, we evaluate VL-Cache across representative VLMs that can handle image, video and language inputs. We use the state-of-the-art open-source LLaVA family, including LLaVA-Mistral-7B (Liu et al., ) and LLaVA-1.6-34B (Liu et al., ).
As shown in Table 1, they all share the same visual model (openai/clip-vit-large-patch14-336 (Radford et al., )) but are fine-tuned from different language backbones (e.g., Mistral (Jiang et al., ), Nous-Hermes-2-Yi-34B (Research, )). Also, the former model uses Grouped Query Attention (GQA) (Ainslie et al., ), while the latter model uses Multi-Head Attention (MHA) (Vaswani et al., ).

Implementation Details. We use an AWS EC2 P4 instance equipped with 8 A100 40GB GPUs for evaluation.
First, we sample three tasks from lmms-eval (Bo Li* & Liu, ), including Coco-Caption (Chen et al., ), DocVQA (Mathew et al., ), and MathVista (Lu et al., ). These tasks are representative, and cover OCR, reasoning, and world knowledge domains. Second, we compare accuracy of our approach against full-cache baselines and previous KV cache sparsification methods including StreamingLLM (Xiao et al., ), H2O (Zhang et al., ) and PyramidKV (Zhang et al., ). We apply KV cache sparsification in line with these baselines by retaining the most recent tokens and selecting the Top-K tokens according to their corresponding scoring policies. All baselines are configured with their default settings, except that the KV cache budget is scaled proportionally to the prompt length, and the recent token window size is fixed at 10% of this budget to enable a fair comparison. Finally, we benchmark latency with varying context lengths and batch sizes.

**Table 1: LLaVA Family Architecture**
| Model | Visual Model | Language Model | Attention Type |
| --- | --- | --- | --- |
| llava-v1.6-mistral-7b | openai/clip-vit-large-patch14-336 | Mistral-7B-Instruct-v0.2 | GQA |
| llava-v1.6-34b | openai/clip-vit-large-patch14-336 | Nous-Hermes-2-Yi-34B | MHA |

### 5.1 Accuracy Evaluation

The accuracy evaluation results are shown in Figure 5 and Table 2. We report the average accuracy score with KV cache budget varying from 1% to 100% of prompt length. Overall, VL-Cache outperforms other compression methods across a range of KV cache budgets and different language model backbones.

Figure: (a) LLaVA-V1.6-Mistral-7B
Refer to caption: extracted/5963551/figures/Mistral-7B-CoCoCaption.png

For the Coco-Caption dataset (Figure 5 (a), (c) and Table 2), all baselines maintain high accuracy when the KV cache budget exceeds 40% for VLMs. However, as the KV cache budget is further reduced, accuracy significantly declines. It is worth noting that even with only 5% to 10% KV cache budget, VL-Cache can achieve accuracy that is comparable to the full KV cache. H2O and StreamingLLM allocate the same cache budget across all layers, causing denser layers to miss important tokens in the context, while sparser layers contain redundant tokens. PyramidKV statically allocates KV cache in a monotonically decreasing manner, which is ineffective for all input queries.

For the DocVQA dataset (Figure 5 (b), (d) and Table 2), PyramidKV demonstrates strong performance; however, our VL-Cache consistently outperforms all other baselines. H2O and StreamingLLM lack robustness, with performance significantly declining as the KV cache is further compressed.

**Table 2: Performance of VLMs with different compression methods and datasets**
| Dataset | Model | Method | 1% | 5% | 10% | 20% | 40% | 60% | 80% | 100% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Coco-Caption<br>(Metric: CIDEr) | LLaVA-Mistral-7B | VL-Cache | 2.64 | 82.53 | 100.36 | 102.06 | 99.93 | 101.07 | 100.08 | 100.68 |
| H2O | 7.87 | 64.45 | 90.36 | 104.04 | 102.64 | 101.21 | 102.86 | 100.68 |  |  |
| PyramidKV | 4.01 | 23.21 | 66.41 | 80.76 | 97.76 | 100.75 | 101.38 | 100.68 |  |  |
| StreamingLLM | 0.05 | 11.82 | 33.98 | 91.87 | 101.47 | 101.07 | 101.6 | 100.68 |  |  |
| LLaVA-1.6-34B | VL-Cache | 0 | 120.11 | 137.35 | 139.42 | 138.58 | 139.19 | 138.01 | 135.07 |  |
| H2O | 0 | 20.87 | 58.14 | 126.8 | 136.27 | 140.89 | 140.3 | 135.07 |  |  |
| PyramidKV | 0 | 93.47 | 116.91 | 129.59 | 138.53 | 138.17 | 140.15 | 135.07 |  |  |
| StreamingLLM | 0 | 5.69 | 8.23 | 21.57 | 131.51 | 134.26 | 139.65 | 135.07 |  |  |
| DocVQA<br>(Metric: ANLS) | LLaVA-Mistral-7B | VL-Cache | 43 | 59 | 62 | 64 | 67 | 67 | 67 | 68 |
| H2O | 37 | 40 | 56 | 59 | 66 | 66 | 67 | 68 |  |  |
| PyramidKV | 42 | 58 | 60 | 61 | 66 | 64 | 68 | 68 |  |  |
| StreamingLLM | 20 | 32 | 47 | 45 | 52 | 60 | 60 | 68 |  |  |
| LLaVA-1.6-34B | VL-Cache | 41 | 82 | 84 | 85 | 85 | 85 | 85 | 85 |  |
| H2O | 28 | 41 | 75 | 82 | 82 | 84 | 85 | 85 |  |  |
| PyramidKV | 43 | 79 | 83 | 85 | 85 | 86 | 85 | 85 |  |  |
| StreamingLLM | 0 | 16 | 34 | 48 | 76 | 80 | 82 | 85 |  |  |
| MathVista<br>(Metric: ACC) | LLaVA-Mistral-7B | VL-Cache | 38 | 36 | 39 | 40 | 40 | 42 | 42 | 41 |
| H2O | 32 | 35 | 36 | 38 | 38 | 40 | 39 | 41 |  |  |
| PyramidKV | 37 | 36 | 38 | 40 | 40 | 41 | 40 | 41 |  |  |
| StreamingLLM | 30 | 35 | 33 | 34 | 40 | 39 | 38 | 41 |  |  |
| LLaVA-1.6-34B | VL-Cache | 41 | 42 | 42 | 44 | 45 | 45 | 42 | 43 |  |
| H2O | 37 | 40 | 40 | 39 | 42 | 44 | 44 | 43 |  |  |
| PyramidKV | 39 | 41 | 42 | 44 | 46 | 41 | 44 | 43 |  |  |
| StreamingLLM | 29 | 35 | 34 | 28 | 38 | 42 | 42 | 43 |  |  |

### 5.2 Speed Benchmark

In order to show the speed advantage of VL-Cache, we measure the GPU kernel latencies of prefill and decoding forward passes with synthetic prompts, following the method in Kwon et al. (). We vary the size of the prompt from 1K tokens to 128K tokens to scale our method to a very long context. Batch sizes vary from 1 to 64 and are static, meaning that all requests get prefilled and decoded concurrently. We assume that the prompt template format remains similar to our accuracy benchmarks, so we use the last 50 tokens of the prompt to determine which tokens to evict from the KV cache. For both prefill and decoding in the baseline, we used default settings from the HuggingFace implementation (^1^11https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one), including CUDA-based FlashAttention-v2. To optimize performance in our VL-Cache, we applied our Triton-based solution for self-attention forward pass, layer-wise sparsity evaluation, and modality-aware token scoring, as detailed in the appendix A.3. The speedup is calculated as $\frac{\textbf{Baseline latency}}{\textbf{VLCache latency}}$.

In Table 3, we observe that with 50 query tokens for calculating attention statistics, the overhead of our method is just 1-6% of the prefill latency. See Appendix A.4 for detailed measurements of the overhead and a discussion on how to reduce the overhead of statistics calculation for a large number of query tokens. During decoding, we run 99 forward passes for a total of 100 output tokens. We observe up to 7x decoding speedups that are attributed to the reduced size of the KV cache, which we compressed to 10% for this benchmark.

Overall, we see that the end-to-end speedup is bounded by the prefill latency, which is mostly unchanged by VL-Cache. For example, with prompt length of 128K and batch size of 1, the decoding speedup of 7.08x is diluted by prefill taking 53% of end-to-end latency, which results in 1.66x end-to-end speedup. End-to-end speedup will monotonically approach the decoding speedup as the count of output tokens increases. This highlights that KV cache sparsity will give the best speed advantage in tasks with long outputs, such as image captions, video descriptions, chain-of-thought multi-modal reasoning, etc.

| Batch Size | Prompt Length | Prefill Speedup | Decoding Speedup | End-to-End Speedup |
| --- | --- | --- | --- | --- |
| 1 | 2000 | 0.96 | 1.19 | 1.16 |
| 1 | 8000 | 0.97 | 1.70 | 1.49 |
| 1 | 32000 | 0.99 | 3.32 | 1.85 |
| 1 | 128000 | 0.99 | 7.08 | 1.66 |
| 4 | 2000 | 0.98 | 1.68 | 1.50 |
| 4 | 8000 | 0.98 | 3.16 | 1.95 |
| 4 | 32000 | 0.99 | 6.07 | 2.06 |
| 16 | 2000 | 0.98 | 3.03 | 1.99 |
| 16 | 8000 | 0.99 | 5.61 | 2.27 |
| 64 | 2000 | 0.98 | 5.23 | 2.33 |

Table: Table 3: Performance metrics by batch size and prompt length for 100 output tokens.

Figure: Figure 6: Server-level throughput v.s. request-level latency curve (prompt length = 2K). Labeled points indicate batch size.

In our implementation of both the baseline and VL-Cache, maximum batch size is limited by peak memory usage during prefill instead of KV cache size, so compression of KV cache does not lead to higher batch size. In future work, continuous batching and chunked prefill could be used to eliminate the prefill memory bottleneck, which would expose our method’s advantage in raising maximum batch size when the size of KV cache is the bottleneck to higher batch size.

Finally, we summarize the trade-off between request-level latency and server-level throughput in Figure 6. We note that VL-Cache offers both higher peak throughput and lower latency for any desired server-level throughput.

## 6 Conclusion

In this paper, we propose VL-Cache, a novel KV cache compression optimized for VLMs. We discovered the unique sparsity patterns of visual and language tokens throughout the prefill and decoding phases. With these observations, we introduce a modality-aware token scoring policy and sparsity-aware cache budget allocation to reduce KV cache size without accuracy loss. Empirical results on multiple benchmark datasets demonstrate that when maintaining only 10% of the KV cache, our method achieves accuracy comparable to the full KV cache and outperforms all existing methods. In a speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by up to 2.33x relative to the full KV cache.

## Appendix A Appendix

### A.1 Vision-Language Prompt Template Construction

Constructing prompt templates in image-based conversations is a common practice for VLMs (Li et al., ; Team et al., ; Islam & Moushi, ; Bai et al., ; Anthropic, ), as it instructs language models to generate more accurate responses. For example, as illustrated in Figure 7, the input image is processed through a visual encoder and a projection layer to generate an image embedding, represented by the <image>. For language input, beyond user input, a prompt template is employed. With the appropriate prompt template design, regardless of the original image order from user input (whether before or after language inputs), there will always be a language-based instruction or question in the post-vision position, providing a strong signal for our VL-Cache to evict insignificant visual tokens.

Figure: Figure 7: Vision-Language Prompt Template Construction. Regardless of the image order (before or after language inputs), with an appropriate prompt template design, there will always be a language-based instruction or question in the post-vision position.
Refer to caption: extracted/5963551/figures/template.png

### A.2 Extended Related Works

The KV cache, while essential for transformer-based LLM and VLM family, demands significant computational resources and memory, limiting inference speed. To address these challenges, researchers have explored various KV cache compression techniques, such as KV cache sparsification, quantization, or a combination of both.

KV cache sparsification. Heavy-Hitters (H2O) (Zhang et al., ) employs cumulative attention scores to greedily evict unimportant tokens. However, this method tends to accumulate more attention on the initial tokens, introducing bias and negatively impacting the identification of key tokens during decoding. ZipCache (He et al., ) further normalizes the cumulative attention scores, leading to more precise prediction. Keyformer (Adnan et al., ) proposes a novel score function to predict the importance of each token and only window tokens and dynamic key tokens are kept in the KV cache. PyramidKV (Yang et al., ) allocates progressively smaller KV cache sizes as layers get deeper. This approach can achieve an absolute accuracy improvement of nearly 20.5 on specific tasks.

KV cache quantization. This method reduces the size of the KV cache by utilizing lower-bit representations of the key and value pairs. In particular, FP8 and INT8 quantization of KV caches are commonly used for LLM inference. Further compression of KV cache into lower-bits is also being explored, e.g. KIVI (Liu et al., ) compresses KV cache into 2-bit representations.

Combination of quantization and sparsification. Quantization and sparsification (Yang et al., ) technically could have orthogonal implementations that can be enabled together to obtain further. However, since both are lossy methods that can impact the accuracy of downstream tasks, careful accuracy evaluation of the quality of the output need to be conducted.

The methods mentioned above mostly address LLM only and there is limited exploration for KV cache sparsification in VLMs. VL-Cache proposed in this paper analyzes and utilizes the unique sparsity pattern in VLMs, which results in better accuracy for VLM inference.

### A.3 Efficient Implementation

In order to discard tokens that received low attention during prefill, we need to calculate 2 statistics from the attention score matrix during the prefill stage: (1) the average attention score for each token in key dimension; and (2) the count of attention scores that are smaller than $p\%$ of the maximum within the query $Q$ dimension.

Regular attention kernels, such as FlashAttention (Dao, ) and PagedAttention (Kwon et al., ), do not materialize the attention scores in HBM, so we need a new memory-efficient kernel to calculate the attention statistics. Overall, we aim to schedule these operations to ensure that: (1) The attention mask is not written to HBM; (2) the $QK$ product is not written to HBM; (3) the attention scores are not written to HBM; (4) similarly-parallelized operations (e.g. div and add) are fused to avoid intermediate tensors in HBM.

In the initial attempt, we used TorchCompiler with the Triton backend, which addressed requirements (1), (3), and (4), but not (2). An efficient fusion of the matmul operation with softmax is described in the FlashAttention paper and can be implemented in Triton, but TorchCompiler was not able to apply this optimization. We will now describe the optimal algorithm that satisfies all performance requirements.

The key to efficient fusion of matmul and softmax is to partition the work along the $Q$ dimension and then hold a single tile of $Q$ tensor in SRAM, while continuously loading tiles of the other tensor and computing the reductions without writing the $QK$ product back to HBM. The 2 reductions required by softmax (pre-exponentiation max, and post-exponentiation sum) are both along the $K$ dimension, so partitioning along the $Q$ dimension is required for these operations. The online softmax algorithm can be used to update the partially computed sum.

For average attention score in the KV context, we need to reduce the attention scores over the $Q$ dimension, which is not aligned with the previous computation. Instead, we need a second kernel which partitions the work over the $K$ dimension, so that a compute block can reduce over the $Q$ dimension. However, since we do not want to materialize the attention scores, the second kernel needs to recompute softmax again. In order do that, we re-use the max and sum over Q dimension from the first kernel, and these 2 tensors (both $O(Q)$ space) are the only intermediate tensors that are stored in HBM.

For sparsity ratio, counting the scores below the threshold is not possible until the maximum along the $K$ dimension is known. We can use the intermediate maximum tensor that was produced by the first kernel to count the sparse scores in the second kernel. Since each thread block will output its own count, we need the 3rd kernel to sum up the partial counts.

In summary, the optimized computation consists of 3 kernels executed serially:

- 1.
Softmax + row-wise stats (reductions over K dimension). Reads: $Q$ and $K$ from HBM. Computes and writes to HBM:
(a)
pre-exp max over $K$ dimension (used in softmax for numerical stability)
(b)
post-exp sum over $K$ dimension (used in softmax as the normalization factor)
- 2.
Softmax + column-wise stats (reductions over Q dimension). Reads: Q, K, pre-exp max, and post-exp sum from HBM. Computes and writes to HBM:
(a)
the post-softmax sum over the Q dimension
(b)
count of attention scores below threshold over the Q dimension
- 3.
Sum reduction of the partial count of attention scores below threshold

We implemented these 2 kernels in Triton and observed that each had roughly the same latency as the FlashAttention kernel, implying that this partioning is optimal. However, even with this approach, we would want to use only a few dozen tokens for querying statistics during prefill.

### A.4 Speed Benchmark Results

In this section, we report detailed metrics from the speed benchmark results. We will also describe the speed benchmark methodology a little more. We used Torch Profiler to measure GPU kernel latencies, and then added up the latencies from different kernels to get a total latency of a particular operation. This breakdown of latency by GPU kernels enabled us to 1) avoid measuring CPU latency that does not relate to our method, 2) group the latency into different buckets for reporting. Empty cells mean that the
inference server crashed due to insufficient GPU memory during prefill.

The following table presents the cumulative latency of all GPU kernels that are related to computing attention statistics from 50 query tokens, eviction scores for all prompt tokens, and copying the KV cache tensor after eviction of 90% of the KV cache into contiguous memory. This prefill overhead is stated in milliseconds:

| batch size | 1000 | 2000 | 4000 | 8000 | 16000 | 32000 | 64000 | 128000 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 5.20 | 7.20 | 10.10 | 17.30 | 28.30 | 49.70 | 100.20 | 189.60 |
| 2 | 6.40 | 9.50 | 15.80 | 29.00 | 51.80 | 92.50 | 188.50 |  |
| 4 | 8.50 | 15.20 | 27.40 | 52.50 | 93.30 | 175.80 |  |  |
| 8 | 13.70 | 26.80 | 50.90 | 96.60 | 178.40 |  |  |  |
| 16 | 25.20 | 51.30 | 95.60 | 183.50 |  |  |  |  |
| 32 | 48.50 | 97.10 | 182.60 |  |  |  |  |  |
| 64 | 91.60 | 186.70 |  |  |  |  |  |  |

We define the prefill latency as the latency of generating 1 output token – starting with input token IDs and ending with output logits, not including tokenization and sampling. The following table presents the prefill speedup ($\frac{\textbf{prefill}+\textbf{overhead}}{\textbf{prefill}}$). We observe that the overhead is never more than 6% and becomes smaller as the prompt gets longer:

| batch size | 1000 | 2000 | 4000 | 8000 | 16000 | 32000 | 64000 | 128000 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 0.94 | 0.96 | 0.97 | 0.97 | 0.99 | 0.99 | 0.99 | 0.99 |
| 2 | 0.96 | 0.97 | 0.97 | 0.98 | 0.98 | 0.99 | 0.99 |  |
| 4 | 0.97 | 0.98 | 0.98 | 0.98 | 0.98 | 0.99 |  |  |
| 8 | 0.98 | 0.98 | 0.98 | 0.99 | 0.99 |  |  |  |
| 16 | 0.98 | 0.98 | 0.99 | 0.99 |  |  |  |  |
| 32 | 0.98 | 0.98 | 0.98 |  |  |  |  |  |
| 64 | 0.99 | 0.98 |  |  |  |  |  |  |

We calculated decoding latency as the difference between end-to-end latency and the prefill latency. Since we use static batching with a constant number of output tokens, it’s easy to decompose the end-to-end latency into its components without concerns for queue waiting, scheduler stalling, sequence termination, etc. The following table presents the decoding speedup ($\frac{\textbf{decoding latency with full KV cache}}{\textbf{decoding latency
with 10\% of KV cache}}$):

| batch size | 1000 | 2000 | 4000 | 8000 | 16000 | 32000 | 64000 | 128000 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 1.10 | 1.19 | 1.37 | 1.70 | 2.31 | 3.32 | 5.17 | 7.08 |
| 2 | 1.18 | 1.36 | 1.68 | 2.27 | 3.16 | 4.56 | 6.78 |  |
| 4 | 1.36 | 1.68 | 2.26 | 3.16 | 4.39 | 6.07 |  |  |
| 8 | 1.73 | 2.29 | 3.13 | 4.37 | 5.87 |  |  |  |
| 16 | 2.19 | 3.03 | 4.18 | 5.61 |  |  |  |  |
| 32 | 2.94 | 4.12 | 5.53 |  |  |  |  |  |
| 64 | 3.83 | 5.23 |  |  |  |  |  |  |

The end-to-end speedup is calculated as $\frac{\textbf{Baseline latency}}{\textbf{VLCache latency}}$. VLCache latency includes prefill latency, VLCache overhead, and 99 decoding passes to generate a total of 100 output tokens:

| batch size | 1000 | 2000 | 4000 | 8000 | 16000 | 32000 | 64000 | 128000 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 1.09 | 1.16 | 1.30 | 1.49 | 1.71 | 1.85 | 1.89 | 1.66 |
| 2 | 1.16 | 1.30 | 1.49 | 1.73 | 1.89 | 1.97 | 1.94 |  |
| 4 | 1.30 | 1.50 | 1.74 | 1.95 | 2.06 | 2.06 |  |  |
| 8 | 1.54 | 1.79 | 2.00 | 2.17 | 2.23 |  |  |  |
| 16 | 1.75 | 1.99 | 2.17 | 2.27 |  |  |  |  |
| 32 | 1.98 | 2.19 | 2.32 |  |  |  |  |  |
| 64 | 2.18 | 2.33 |  |  |  |  |  |  |

### A.5 Measuring attention to Visual and Language Tokens

As we have established in the Section 3.1, the attention sparsity during decoding can be predicted from attention scores in prefill phase. To further understand the importance of visual tokens and language tokens in the prompt, we examine the division of attention from decoding tokens to these two modalities of prompt tokens.

We propose $\operatorname{\mathbf{Contribution}}$ for quantitative analysis and visualization of the layer-wise modality-specific attention patterns. Let $T$ be the current sequence length, $t$ be the index of the first decoded token, and $A^{(l)}\in\mathbb{R}^{T\times T}$ be the attention matrix at layer $l$ for one specific head.

$$ $\operatorname{\mathbf{Contribution}}_{mod}^{(l)}\coloneq\frac{1}{T-t+1}\sum_{i \geq t}{\frac{\sum_{j\in J_{mod}}{\operatorname{\mathbf{ThresholdFilter}}(A^{( l)},p)_{ij}}}{\sum_{j\in J_{all}}{\operatorname{\mathbf{ThresholdFilter}}(A^{( l)},p)_{ij}}}}$ (3) $$

Here, we use $J_{mod}$ and $J_{all}$ to denote KV cache indices of one selected modality (language or vision) and all modalities respectively in the input prompt. As shown in Figure 9, VLMs allocate primary attention to visual tokens in the first layer, while in the second layer, the contributions of visual and language tokens are comparable. From the third layer onward, language tokens dominate, with a slight increase in visual token contribution in the middle layers. Additionally, we also propose $\operatorname{\mathbf{Coverage}}$ to analyze the ratio of the number of tokens from a specific modality, defined as follows:

$$ $\operatorname{\mathbf{Coverage}}_{mod}^{(l)}\coloneq\frac{1}{T-t+1}\sum_{i\geq t }\frac{\sum_{j\in J_{mod}}{\operatorname{\mathbf{TopK}}(A^{(l)},k)_{ij}}}{\sum _{j\in J_{all}}{\operatorname{\mathbf{TopK}}(A^{(l)},k)_{ij}}}$ (4) $$

Specifically,

$$ $\operatorname{\mathbf{TopK}}(A,k)_{ij}=\mathbbm{1}[{A_{ij}\in\{A_{i}^{(r-k+1)} ,A_{i}^{(r-k+2)},...,A_{i}^{(r)}\}]}$ (5) $$

Here, we define $k$ as $\lfloor\alpha\cdot T\rfloor$, $r$ as the number of columns of matrix $A$, and overload notations to use $A_{i}^{(n)}$ as the $n$-th order statistic of the row vector. Additionally, $\alpha\in(0,1)$ denotes the token budget threshold (e.g. $\alpha=10\%$ means only the 10% of sequence length of KV cache is retained). After KV cache compression with $\operatorname{\mathbf{TopK}}$ selection, we assess the ratio of visual attention to language attention among the remaining tokens. In Figure 9, with $\alpha=10\%$ applied, a similar trend in coverage is observed compared to Figure 9. Notably, in the middle layer, visual tokens constitute a larger proportion than their contribution, indicating that the contribution per token is smaller.

Furthermore, since contribution of language and visual tokens differ by layer, the optimal budget for each layer may depend on the ratio of visual to language tokens in the prompt. In contrast, previous pyramid-style cache allocation methods do not adapt to the input prompts at inference time.

Figure: Figure 8: Contribution
Refer to caption: extracted/5963551/figures/token_contri.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the section on weakness.
soundness: 3
strengths: 1. The experimental results are very good; the performance is significantly better compared to other KV-cache methods.
2. Sparsity-aware layer-wise KV cache allocation and modality-aware token scoring are meaningful and have also been proven effective in the experiments.
summary: This paper addresses the challenge of efficiently storing and accessing large Key-Value (KV) caches in Vision-Language Models (VLMs), which are crucial for handling visual contexts like images or videos. Existing KV cache compression methods developed for Large Language Models (LLMs) don't perform optimally when applied to VLMs. To solve this, the authors propose VL-Cache, a compression method designed specifically for VLM inference. VL-Cache utilizes a layer-adaptive, sparsity-aware cache allocation and a modality-aware token scoring approach to reduce cache size without losing accuracy. Experimental results show that this method retains comparable accuracy with only 10% of the original cache, accelerates token generation by up to 2.33x, decoding by up to 7.08x, and reduces GPU memory usage by 90%.
weaknesses: 1. The writing is somewhat complex and unclear; it takes a few careful readings to understand. Adding more illustrations to explain the method would improve it.
2. For MathVista, it seems that the KV-cache isn’t important? Even with only 1% of it, the performance is quite good, and sometimes the performance with 100% KV-cache isn’t even the best. Could you explain that?
3. In terms of writing, Section 3's PRELIMINARY EXPERIMENT is too long, while there’s relatively little written about the methodology. This writing style increases the burden on the reader. Readers likely want to see the method as soon as possible, so the focus should be on discussing your approach.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses above.
soundness: 3
strengths: The insight that the attention patterns of generated tokens closely align with language tokens is valuable.
summary: This paper introduces a modality-aware KV cache compression method for Vision-Language Models (VLM). It introduces a dynamic cache budget allocation mechanism and employs post-vision language tokens to compute attention scores.
weaknesses: 1.	Algorithm 1 only outlines two procedures without detailing how these procedures are integrated into the algorithm.
2.	Definition 3.1 and the following text is unclear. The terms Psi(S) and S_Psi are used without clear explanation. A precise definition of Cache Hit Rate is essential.
3.	Figure 4 lacks clarity. It does not effectively illustrate how the budget is allocated or how post-vision attention is utilized.
4.	In the related work, ZipCache proposes a policy using normalized attention scores. This should be considered and compared.
5.	The KV cache for generated tokens is not addressed.
6.	The performance of VL-Cache is not convincingly demonstrated in Table 2. VL-Cache does not significantly outperform other methods, while H2O has achieved the strongest performance on many benchmarks. It is recommended to test VL-Cache on more challenging benchmarks.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to the questions in the weakness part.
soundness: 2
strengths: 1.	The authors identify the I/O bottleneck between GPU’s HBM and SRAM in the decoding stage of VLMs, motivating the innovative design of KV cache compression. This understanding helps to address the challenge of scaling VLMs and optimizing their performance.
2.	Through Figure 1, the authors illustrate the distinctive attention sparsity patterns between VLMs and LLMs, emphasizing the need for modality-aware compression techniques tailored for VLMs.
3.	I have learned a lot from the author's observations. The KV cache budget and compression ratios differ in layers and modalities. This provides good insights into the optimization of VLM cache compression.
4.	To compress the VLM cache, the authors propose two key methods: a sparsity-aware cache budget allocation strategy and a modality-aware token scoring policy, to optimize the cache budget allocation and token score measurement, respectively.
5.	The experimental results show that the proposed methods achieve significant inference speedups and GPU memory savings.
summary: The authors aim to accelerate VLMs by reusing KV caches for encoding visual contexts. The authors observe the limitations of existing KV cache compression methods and propose a specific version for VLMs. Based on the observation of sparsity patterns in prefilling and decoding stages, the main idea is to design a cache allocation method that balances the cache size and model accuracy. Evaluations based on vision-language benchmarks show the performance of model accuracy and speedups.
weaknesses: 1.	In Figure 1, the authors measure attention scores to verify the modality boundary, but the specific methodology for calculating these scores remains unclear. Please provide additional details on the measurement process.
2.	The term “post-vision attention” is frequently used but not explicitly defined, making the relevant sentences difficult to follow (“the attention patterns from the output language tokens is much closer aligned with the language tokens that follow the visual tokens in the prompt (the post-vision attention) rather than the visual tokens themselves”). A clear description of post-vision attention at its first occurrence would enhance the paper's readability.
3.	The threshold filter in Section 3.1 utilizes the hyper-parameter p to control sparsification. Conducting ablation studies on the impact of different p values would provide deeper insights into the robustness and sensitivity of the proposed method.
4.	To Algorithm 1, while it dynamically allocates the cache budget based on each prompt, its computational complexity raises my concern. Please analyze the computational overhead (in lines 11 to 14).
5.	Besides, Algorithm 1 does not explicitly reflect the modality-aware design as has been highlighted in this paper. Please clarify how the algorithm incorporates modality properties.
6.	In experiments, the I/O traffic amount caused by KV caching should be reported.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	Some notations in the proposed method lack clarity. For instance, the meanings of $\Gamma$ and $Z$ are not sufficiently explained, making it difficult to understand their roles in the algorithm. Providing more detailed explanations would improve the clarity of the presentation.

2.	The authors state that the maximum batch size is constrained by peak memory usage during prefill rather than by the KV cache size, suggesting that compressing the KV cache does not increase the batch size. Does this imply that the proposed KV cache compression does not actually reduce overall memory usage? If so, why is that the case? This raises concerns about whether the title “KV Cache Compression” accurately reflects the method’s ability to save GPU memory, as the proposed approach appears to be more of a sparse computation technique applied during decoding.

3.	In Table 3, the reported speedup is based on generating a total of 100 output tokens. It would be helpful if the authors could provide the exact prefill and decoding times separately. Given that with a prompt length of 128k and only 100 output tokens, the prefill time is likely to dominate the overall processing time, having this detailed breakdown would clarify the impact of the proposed method on different stages of the inference process.

4.	In Section 4.2, the authors point out that accumulated attention has the limitation of assigning disproportionately high scores to earlier tokens due to summation over the full query dimension. To address this, they propose accumulated post-vision attention. However, there is another approach [1] that addresses the same issue using normalized attention. It would be helpful if the authors could include more discussion on this alternative method and provide additional experimental comparisons to better illustrate the benefits of their approach.

5.	The experimental evaluation is insufficient, as it lacks comparisons with two important baselines for VLM acceleration: FastV [1] and HiRED [2]. Additional comparisons with these methods are needed to strengthen the evaluation.

6.	Although the proposed method targets vision-language models, the current evaluation lacks coverage of a comprehensive range of tasks. To better demonstrate its effectiveness, it would be beneficial for the authors to include additional results on video benchmarks, such as Video-MME [4]. This would help showcase the method’s applicability to video tasks.

Reference:

[1] ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification. NeurIPS 2024.

[2] An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. ECCV 2024.

[3] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments. arXiv 2024.

[4] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv 2024.
soundness: 3
strengths: The paper is well-written and easy to follow. The sparsity analysis of prefilling and decoding effectively motivates the use of adaptive sparsity. The proposed method achieves significant speedup in decoding, particularly with long prefill tokens.
summary: The paper tackles the challenge of accelerating vision-language models (VLMs) by leveraging a sparse Key-Value (KV) cache. Unlike existing compression methods for large language models (LLMs), which are less effective for VLMs, the proposed VL-Cache takes advantage of unique sparsity patterns in visual and text tokens. It uses layer-adaptive budget allocation to allocate cache resources based on sparsity levels and modality-aware token scoring to determine token importance with post-vision text tokens. By retaining only the most crucial cache entries, VL-Cache effectively reduces memory usage and latency. Experimental results on three lmms-eval tasks demonstrate the method’s effectiveness.
weaknesses: Please see the questions.

Meta-review: The paper introduces VL-Cache, a novel KV cache compression method tailored for accelerating vision-language model inference, which significantly reduces memory usage and latency while maintaining high accuracy.

After rebuttal and discussions, this paper receives all positive ratings. After carefully  reviewing the paper and all the reviewers' comments and discussions, The AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.

Decision: Accept (Poster)