VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
Authors: Dezhan Tu
Danylo Vashchilenko
Yuzhe Lu
Panpan Xu
Venue: ICLR 2025 Poster (2025)
Keywords: KV Cache Compression
Vision-Language Models
Inference Acceleration
Sparsity
Modality
Primary area: generative models
TL;DR: We propose VL-Cache, a novel KV cache compression recipe tailored for vision-language model inference acceleration.

Abstract: Vision-Language Models (VLMs) have demonstrated impressive performance across a versatile set of tasks. A key challenge in accelerating VLMs is storing and accessing the large Key-Value (KV) cache that encodes long visual contexts, such as images or videos. While existing KV cache compression methods are effective for Large Language Models (LLMs), directly migrating them to VLMs yields suboptimal accuracy and speedup. To bridge the gap, we propose VL-Cache, a novel KV cache compression recipe tailored for accelerating VLM inference. In this paper, we first investigate the unique sparsity pattern of VLM attention by distinguishing visual and text tokens in prefill and decoding phases. Based on these observations, we introduce a layer-adaptive sparsity-aware cache budget allocation method that effectively distributes the limited cache budget across different layers, further reducing KV cache size without compromising accuracy. Additionally, we develop a modality-aware token scoring policy to better evaluate the token importance. Empirical results on multiple benchmark datasets demonstrate that retaining only 10% of KV cache achieves accuracy comparable to that with full cache. In a speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding by up to 7.08x, while reducing the memory footprint of KV cache in GPU by 90%.

Published as a conference paper at ICLR 2025
VL-CACHE: SPARSITY AND MODALITY-AWARE KV CACHE COMPRESSION FOR VISION-LANGUAGE MODEL INFERENCE ACCELERATION
Dezhan Tu
Danylo Vashchilenko
Department of Electrical and Computer Engineering University of California, Los Angeles Los Angeles, CA, USA dztuβg ucla.edu
AWS AI
Amazon
New York, NY, USA
vdanyloß amannylolamazo.....m .om com
Yuzhe Lu
Panpan Xu
AWS AI
AWS AI
Amazon
Amazon
Santa Clara, CA, USA mozhelu@anee zon com
Santa Clara, CA,, USA
xupanpan ama zon com
ABSTRACT
Vision-Language Models (VLMs) have demonstrated impressive performance across versatile set of tasks. A key challenge in accelerating VLMs is storing and accessing the large Key-Value (KV) cache that encodes long visual contexts, such as images or vidcos. While existing KV cache compression methods are effec- tive for Large Language Models (LLMs), directly migrating them to VLMS yields suboptimal accuracy and speedup. To bridge the gap, we propose VL-Cache, a novel KV cache compression recipe tailored for accelerating VLM inference. this paper, we first investigate the unique sparsity pattern of VLM attention by dis- tinguishing visual and text tokens in prefill and decoding phases. Based on these observations, we introduce layer-adaptive sparsity- cache budget alloca- tion method that effectively distributes the limited cache budget across different layers, further reducing KV cache size without compromising accuracy. Addi- tionally, we develop a modality-awar token scoring policy to better evaluate the token importance. Empirical results on multiple benchmark datasets demonstrate that retaining only 10% of KV cache achieves accuracy comparable to that with full cache. speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding by up to 7.08x, while reducing the memory footprint of KV cache in GPU by 90%.
INTRODUCTION
Vision-Language Models (VLMs) have recently emerged powerful tools for a broad range of multi-modal tasks (Liu et al. 2023b; Chen et al. 2023 Bai et al. 2023). As these models improve processing long visual context such as high-resolution images, multiple images and multi-frame videos (Li et al., 2024a) the number of visual tokens increase rapidly. Consequently, deploying VLMs demands substantial GPU memory capacity, bandwidth, and computational resources, lead- ing to high inference latency and cost.
Similarly to Large Language Models (LLMs) (Chang et al. 2024), VLMs decode tokens sequen- tially in an auto-regressive loop. The key and value pairs of the input prompt and of the generated output tokens are stored in GPU memory (where they are known as the KV cache) and reused at each decoding step avoid recomputation. As the context length grows, KV cache not only occu- pies a larger amount of GPU memory, but also increases inference latency due to data movement between GPU's high-banden bandwidth memory (HBM) and its on-chip memory (SRAM) in each decoding
Work done during internship Amazon AWS AT

---

Published as a conference paper at ICLR 2025
Hong et al. 2024). This is significant challenge for scaling VLMs, because large KV cache is required to hold the imput images and video frames. For example, with batch size of four prompts, five images in each prompt, and each image using 2K visual tokens, serving the LLaVA-1.6-34B model requires 110 GB of HBM capacity just for the KV cache of visual tokens.
step (Dao 2023
Language
Language
Vision
Language
0
0.5k
critical tokens
0.2k
query dimension
query dimension
1.0k
0.5k
1.5k
clear modality boundary
0.8k
2.0k
post-vision attention
1.0k
2.5k
Prefill
Prefill
3.0k
decoding 1 .22k
decading
0
0.2k 0.5k
0.8k
1.0k
1.2k
0.5k 1.0k 1.5k 2.0k 2.5k 3.0k key dimension
key dimension
(a) LLMs Attention Score Matrix
(b) VLMs Attention Score Matrix
Figure 1: Attention Score Matrix from LLaVA v1.6 Mistral 7B with (a) language-only context, and (b) language and vision context. A deeper red color indicates higher attention score. Both matrices indicate that critical tokens used in the decoding phase are primarily consistent with those in the prefill context. The key difference is that, in (b) VLMs attention, clear modality boundary emerges along the query dimension.
KV cache sparsification a promising method of reducing GPU memory requirements and inference latency for LLMs (Zhang et al.) 2024b a: Yang et al. 2024a He et al. 2024 Li et al. 2024b). Prior methods leveraged sparsity in the attention scores matrix of LLMs evict insignificant tokens from KV cache, while preserving important contextual information needed for output token generation. Besides reducing the GPU memory footprint, this method also reduces the token generation latency by minimizing data movements and floating-poin operations (FLOPs) in the attention layer.
We discover that VLMs and LLMs exhibit significantly different attention sparsity patterss, as il- lustrated in Figure I] In VLMs, clear modality boundary exists between visual tokens and the subsequent language tokens. Specifically, the attention patterns from the output language tokens is much closer aligned with the language tokens that follow the visual tokens in the prompt (the post-vision attention) rather than the visual tokens themselves. Formally, we define post-vision attention as the sub attention score matrix that's sliced along the query dimension to only include language prompt tokens after vision tokens, as illustrated in Figure Previous KV compression methods modality- modality-unaware unaware and incorrectly conflate these two sources of attention scores. Un- surprisingly, our experiment in Figure 9 shows that KV cache compression methods designed for LLMs have suboptimal results when applied VLMs.
Next, we measure that attention sparsity ratios vary in 70% to 99% range across transformer lay- ers. Our observations in Section 3.1 show that previous KV compression methods suboptimally distribute the cache budget between layers, which leads to either under- or over-compression of in- formation in KV cache. Finally, we observe that sparsity ratios differ between visual and language tokens as well, so an optimal cache budget allocation between layers can not be done before sparsity is measured in particular prompt.
To address these gaps, propose VL-Cache, novel KV cache compression method for acceler- ating Vision- Language Model inference. Our method fully utilizes cross-modality and layer-wise attention sparsity patterns in VLMs to dynamically prune KV cache with minimal loss of task-level accuracy. To the best of our knowledge, this is the first work that investigates attention sparsity in VLMs and the first work that specifically optimizes KV cache compression for VLMs. In partic- ular, we propose (1) sparsity-aware KV cache budget allocation between the transformer layers at inference time, and (2) modality- scoring policy for token eviction.
2

---

Published as a conference paper at ICLR 2025
Experimental results in Section 5.1 show that method retains 98% of the original task-level accuracy while using only 10% of KV cache for the majority of vision-language tasks from several datasets. In speed benchmark, our method reduces end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding in particular by up to 7.08x, while allocating 90% less GPU memory for KV cache. In inference scenarios where KV cache size is the limiting factor to higher concurrency, VL-Cache enables up to 10x higher concurrency after KV cache compression.
Overall, our contributions in this paper are summarized as follows:
VLMs Attention Sparsity Profile. We uncovered unique attention sparsity patterns in the prefill and decoding phases of VLM inference on a variety of multi-modal tasks, which is drastically different from those of LLMs.
Layer-Adaptive Sparsity-Aware Cache Budget Allocation. We proposed allocate each layer's KV cache budget with consideration of that layer's attention sparsity at inference time. Even with 10% KV cache budget, we retained high accuracy on popular benchmark tasks.
Modality Token Scoring Policy. We observed that language- vision attention robustly includes information about the importance of visual tokens. We treat visual and language atten- tion scores differently better retain important tokens.
2 BACKGROUND
In this section, we detail the inference procedure of the widely adopted VLM architecture with vision & language input and text output, where image tokens are projected soft prompts (Li et al. 2024a
Team al. 2023 Islam Moushi 2024 Bai et al. 2023 Anthropic 2023). We also introduce the
Cormulation of K cache compression and the approach of existing works (Zhang et al.] 2024b
Liu et al. 2024a Ge al. 2023 Tang et al. 2024 et al. 2024b Lee et al. 2024 Yu et al. 2024
2.1
VLM INFERENCE
Prefill phase. The input to VLMs includes both images and language, where images are processed by the visual encoder generate visual tokens. Subsequently, projection layer, such as a simple multi-layer perceptron, maps these visual tokens to a unified embedding space. Meanwhile the language prompt is fed to a tokenizer and embedding layer to create the initial hidden state for language tokens. For notation simplicity,
denote sequence of prompt tokens, including both visual and language ones, as {x1, Lm These tokens are processed by the language model in parallel to calculate the probability of the first decoded token Po ((f/m+11/11_-0 (2, Simultaneously,
the key vectors {K1²) ..... k!!l)) } and value vectors {u] ..... um } at each transformer layer are cached in GPU memory to avoid recomputation in the next decoding step.
Decoding phase. Once decoding starts, the language model in VLMs takes effect and generates one token per step in auto-regressive loop. At step i, the model receives the token Tm-ii input and calculates the probability Po Po(Imtit1l (i,,...T++++) Each decoding step involves generating (1)
new key vectors k09 and value vectors um-i which are appended to the previously cached key-
value pairs for each layer, resulting in {k10) ..... ( and {uf)) .... Um+i)- } In case of long contexts, such as multiple or high resolution images, the key-value cache can grow significantly larger than
(1)
the model parameters and other intermediate tensors, making memory capacity and bandwidth major performance bottlenecks.
2.2 KV CACHE COMPRESSION
To address the bottleneck of storing and accessing the large KV cache during decoding, many searchers have focused on KV cache compression to maintain only subset of the full KV cache for more efficient decoding while minimizing the accuracy loss. There are two main design dimensions to such algorithms: how many cache tokens should be kept at each layer, and which tokens to evict during compression.
Budget Allocation. Since the transformer architecture consists of multiple identical layers, a straightforward strategy is allocate equal budget of KV cache slots each layer (Xiao al.)
3

---

Published as a conference paper at ICLR 2025
2023b
Zhang et al. 2024b He al. 2024). More recently, inspired by the observation that moving cache tokens at different layers results in varying degrees of performance loss, PyramidKV Zhang et al., 2024a) and PyramidInfer (Yang et al. 2024a) proposed a decay schedule to assign more cache budget to shallower layers and observed improved accuracy results relative to the base- line with equal budget per layer.
Token Scoring Policy. For a given cache token budget, token scoring policy ai is required to rank the importance of KV cache tokens and decide which tokens to keep. Let be the count of cache tokens at layer I, and S = {0, 1,.... n be the indices of these cache tokens. We define ui : S - IR", with the output being the scores for current cache tokens. Given these scores, indices with top-k scores, S = {i1, i2..... is (S),, 2 =E))==] (ft
.g....... } where k [1, are selected. Recent works have shown that attention scores serve as effective source design such policies. StreamingL.AM LLM (Xiao et al. (223b)) found that high attention scores are frequently assigned to initial tokens and achieved length generalization by keeping only the initial and most recent cache tokens
while removing intermediate ones. H2O (Zhang et al. 2024b) uses accumulated attention scores to identify crucial tokens to retain.
We make the observation that, fundamentally, KV cache compression methods only work because inference with a transformer layer is sparse process. Therefore, this work, we leverage attention sparsity the unified guidance to design both the cache budget allocation mechanism and the token eviction policy for KV cache compression in VLMs.
PRELIMINARY EXPERIMENT
The attention mechanism for visual and language tokens is a key aspect of VLMs. Therefore, further optimization of the KV cache compression methods for VLMs requires careful investigation of the attention patterns with relevant input prompts. Motivated by this need, we conducted preliminary experiments explore the VLM attention. We randomly sampled data from three multi-modal datasets Doc (Mathew et al. 2021). Math (Lu et al. (2023). and Coco-Caption (Chen) jet al., 2015). These datasets cover wide range of visual tasks, including OCR, visual diagram reasoning, and world knowledge. We selected one of the state-of-the-ae af- VI.Ms, LLaVA-Mistral (Liu et al.] 2023b), and recorded the attention score matrix until the generation process completes. We will leverage insights from these analyses to motivate our algorithm design in Section4
3.1 MEASURING ATTENTION SPARSITY
In this section, we measure the sparsity of the attention score matrix in different transformer layers during the prefill and decoding phases. First, we apply a filter with relative threshold p to the attention score matrix A:
AAy
if Af 2 p. max, (Ai)) otherwise
ThresholdFiltee(AA.pp)) (A, 0
where threshold p (0, 1) controls the strength of induced sparsification, following Zhang et al. (2024b). We also heuristically set = 1%, such that the filtered-out scores have little impact on the output of the transformer layer. After filtration, we calculate sparsity 700)) E [o, 1] of layer as count of zero entries, normalized by the size of the lower triangular portion of the attention scores matrix:
y00 Eizn ThreesoldFilleer p)5 = 0]
144 : :12 5
We calculated the average sparsity over sample from the evaluation datasets, and then plotted the range of each layer's sparsity with different attention heads in Figure 2 We observe that in the prefill phase (Figure 2a). the first two layers exhibit significantly lower sparsity and higher density compared to other layers. Additionally, some layers in the middle also demonstrate higher density than their neighboring layers. During the decoding phase (Figure 2b), similar trend is observed. Based on these observations, we expect that the aggregate attention sparsity during prefill could effectively predict the required KV cache size for robust decoding.
4

---

Published as a conference paper at ICLR 2025
Prefill Sparsity
Decoding Sparsity
1.0
Average Cache Hit Rate
1.00
1.00
Cache Htt Rate
0.8
Sparsity
0.75
0.6
0..50
0.50
0.4
0.25
0.25
0.20
10
15 20
25
30
0
5
10
15
20
25
30
0
10
15
20
25
30
Layer Index
layer index
layer index
Figure 3: Average Cache Hit Rate. The accumulated post- Figure 2: Layer-wise attention sparsity in prefill and decoding vision attention (ours) demon- phases. Different layers exhibit varying degrees of sparsity; The strates a higher cache hit rate layer-wise sparsity trend the decoding phase is similar to that in compared to the other two to- the prefill phase.
(a) Prefill Sparsity
(b) Decoding Sparsity
ken scoring policies.
We note that previous methods, such as H2O (Zhang et al. 2024b) and Keyformer (Adnan et al.) 2024), allocate the same cache size across all layers, leading to insufficient allocation for high- information layers and wasteful allocation for low-information layers. More recent work including PyramidKV [Zhang et al. (0244) and PyramidInfer (Yang et al., 2024a) monotonically decrease the KV cache size with depth of each layer, which we find to be suboptimal well. Our observations reveal a more nuanced sparsity pattern, where different layers demand non- non-monotonically varying cache sizes to retain context information during decoding.
3.2 MEASURING CACHE HIT RATE
In Figure Tb of Section [ we previously noted a distinct boundary in VLM attention between vi- sion and language tokens in the prompt. Based on these observations, we further hypothesize that retaining tokens based on post-vision prefill attention instead the full full-prompt prefill attention would preserve important cache tokens with higher recall. To validate this hypothesis, CacheHitRate to measure the fraction of important tokens that are preserved after eviction when different scoring policies are applied. Let be the number of prompt tokens, Q.m. K.n R. m be the corresponding query and key matrices, and Qm+1 E Rd be the query vector of the first decoding token. Also recall from Section 2.2 that denotes the indices of prompt cache tokens, and v denotes a scoring policy that maps each token index to a real number as their importance.
define
Definition 3.1 (CacheHitRatte), given Am+1 == softmar
Qm+1 K.m Vd E Rm, we define:
v S - Am+1 as the optimal scoring function since Am+1 is true attention scores in decoding: Spr.k a as the top-k tokens selected per vi ranking, which we treat as the ground-truth for all k;
CacheHitRRte as Se.tnSpr the percentage of true top-k tokens S & that is also preserved [8 by Sp.k. which are top-k tokens kept by any policy sb.
We note that during inference, Ant1 is not available because we aim to compress KV cache before the decoding pass lift the memory bottleneck and boost decoding throughput. Thus, multiple ehs have been proposed to approximate zi* in recent works. In the following paragraphs, we use (S) to indicate the token scores assigned to S with arbitrary policy vb. For brevity, we assume the so ffna outputs below are applied necessary causal masks to serve as attention scores.
Accumulated Attention (prior work) uses the cumulative attention score along the query dimension, (S) = E, softmar( Q. "n ); to rank the importance of cache tokens.
Vā
Normalized Attention (prior work) instead computes the average attention scores along the query dimension based on the observation that accumulated scores disproportionately favors earlier tokens.
The policy can be written as (5)) == à Σ. fo foftrrr Q... K........ ): where vector n denotes the number of unmasked elements in each column and @ indicates element- multiplication.
vi
Sliding Window Attention (prior work) also computes accumulated attention scores along the query dimension but only over recent window. This policy is defined ((S)
5

---

Published as a conference paper at ICLR 2025
vd da where fixed window size. Since is fixed, whether the score is normalized does not change the ranking of cache tokens for evicting purposes.
Σ, soflmart Qm-mom K unmm
Post-vision Attention (ours) is based the observation of the similarity between attention from post-vision prompt tokens and from decoding tokens as shown in Figure Ib We adopt the term post-vision to distinguish language tokens that follow visual tokens from instructions that precede the images in the prompt. Formally, we use the portion of the attention scores Q where the count of language tokens that follow the vision tokens in the prompt. Our policy is defined
policy dynamic prompt-depen dempdddd sliding window with window size set to the length of the post-vision language prompt instead of static prompt-independee value. From Figure E we find our scoring policy based on post-vision attention leads consistently higher cache hit rate across layers than other policies.
by (b)) == Σ, softmaz' Qm v.nu Km 1om ))- During prefill, one interpret
vd
VL-CACHE METHOD
Motivated by observations from preliminary experiments, introduce method VL-Cache, which strategically combines sparsity-aware cache budget allocation and modality token scor- ing policy to improve VLM's performance under limited KV cache budget, in terms of both accuracy and efficiency.
Specifically, we use Post-vision Attention compute both inter-layer sparsity and intra-layer token importance. The former guides how many cache tokens should be allocated at each layer, while the latter dictates which k tokens within layer should be kept due their importance. A high- level description of our method is visualized in Figure For brevity, we will use A' to denote the Post-vision Attention matrix in this section.
Embed Prompts
Prefill
Decode Reduced KV Cache
A chat between curious humen and an denalled, and polioo arewers to the human's queetion.
Budgert
based on
Compute
Layer-wise
Budget
Value
VLM
Rank Tokens
Kay
Value
Key
Key
The shows vast, emppy beach surset survise. warm, den hues. Someone appeare paragi ding...
Layer
image and provide a brief descripttion.
Layer O
Layer
Post-Vision Attention
LLM KV
Figure 4: VL-Cache Overview. During prefill, post-vision attention matrices are utilized to compute sparsity-driven layer-wise budgets and rank the importance of cache tokens. Unimportant tokens are then evicted, allowing lower memory usage and accelerated decoding with reduced KV cache.
We argue that using Post-vision Attention has two important advantages. Firstly, computing layer- wise sparsity using Post-vision Attention results in much lower memory and latency O(rm) com- pared to using the full attention matrix 0(m²) in (Zhang al. 2024b) and (He al. 2024). since the visual tokens dominate the prompt length, < m for current vision-language tasks. Sec- ond, with a fixed eache budget, using Post-vision Attention leads better preservation of important tokens, as measured by high CacheHitRate in Figure 3 In this section, we will detail how we perform sparsity-aware KV cache budget allocation and cache token eviction guided by Post-vision Attention before reviewing the strong performance of VI.-Cache in Section 5
4.1 SPARSITY-AWARE KV CACHE BUDGET ALLOCATION
Before determining the exact tokens evict, need to allocate the KV cache budget, which is the percentage of the KV cache to retain at each layer. Based on our observations from Section 3.1 we implement a sparsity-aware layer-wise KV cache allocation approach with steps during the prefill phase, detailed in Algorithm below. First, we apply ThresholdFilter pruning (with p 1%) to the Post-vision Attention scores and calculate the layer-wise sparsity (line 4-8 and 13-
18). Second, given target KV cache budget for the whole model, we distribute this budget across

---

Published as a conference paper at ICLR 2025
layers based each layer's spansity ratio (line 9-12). This method optimizes the use of limited memory to store an appropriate amount of context information in each layer. Since the allocation occurs only once right after prefill, the latency overhead is amortized across multiple decoding steps.
Algorithm Sparsity- Aware Cache Budget Allocation
1: Input: query and key Q, K E xx number of layers L, number of heads Ht, length of post-vision prompt т, overall cache budget a. procedure
K, a. L, H) 3: D main method to compute layer-wise cache budgel.
2:
4: [ t 0
T stores layer and head-wise sparsity
5:
for = I. do
6:
for h ('(0) +
H do ComputePost VisionSparsity((..
7:
(h k b by head-averaged sparsity for each layer b Z is the sum of non-sparse ratios across layers as the normalizer b B stores cuche token budget for each layer
b> invoke helper method below
h
8:
- (1)
9:
+ Σ, 1 107
10:
8[L]] <
11:
for
1 = - do
12:
g(D < clip( 10 aL, 0.01,)
39
allocate budget by normalized non-sparse ratio K)
13:
procedure
14:
helper method to compute post-vision attention head sparsity.
15:
Q' x mm
Q' is post-vision query
16:
A' + softmar(
K"
D A' is post-vision attention
Va
17:
7' Ei+m-rz 72)
1 I/ThresholFFitteri p),=00
of is ratio of zeros in unmasked entries
RA m m-r23) 1231
18:
return 7
19:
Output: layer-wise budget 3
In Algorithm 1 we present our KV cache budget allocation algorithm, where is hyper-paramett, that encodes the desired KV cache budget for the whole model. When the accuracy is not satisfac- tory. a higher a can be used to keep more KV cache. Finally, we would like to point out the key difference between our algorithm and PyramidKV (Zhang et al.] [2024a): our allocation budget is customized for each prompt based on its sparsity pattern, instead of using a fixed layer-wiss wise budget for all prompts, thus granting additional flexibility.
4.2 MODALITY-AWARE TOKEN SCORING POLICY
After we decide the cache budget for layer I, we choose a subset of k(()) cache tokens from the full cache. Prior works (that were targeting KV cache compression for L.MMs) have explored several scoring policies, such as Accumulated Attention in H20 (Zhang et al. 2024b). Normalized Attention in ZipCache He al. al. (2024) and Sliding Window Attention Li et al. (2024b); /Zang et al. (0024a); Yang et al. (2024a), as we discussed earlier in Section 3.2 In this section, we will explain in-depth why Pist-vision Attention provides a better scoring policy than prior policies for VLMs.
While Accumulated Attention serves as a simple yet effective baseline, one critical issue is that sum- mation over the full query dimension unavoidably assigns high scores to earlier tokens. Observing this length bias, Sliding Window Attention only sums attention scores over a recent window. We con- firm that this indeed leads to higher CacheHit.Rate Rate, as shown in Figure 3 However, as we can in Figure Th VLMS have significantly different attention pattern than L.MSs. Along the query dimension, we observe a clear modality boundary: visual tokens attend to other visual tokens rather uniformly while language tokens only pay attention few visual tokens with high concen- tration. If we were to apply Accumulated or Normalized Attention as the scoring policy, the signal for critical cache tokens will be buried as we sum or average scores from all the preceding visual tokens. Using a recent window helps, but a fixed window size can easily be too large or too small as the length of the post-vision prompt varies from case to case. Therefore, we introduce Post-vision optimized scoring policy for VLMs by implementing dynamic, prompt-specific window size. The advantage of our scoring policy evident by its high Rate in Figure 3l and will be further demonstrated through accuracy comparisons in Section 5
Attention
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
In our experiments, we evaluate VL-Cache across representative VLMs that can handle image, videos, and language inputs. We use the state-of-the-arr open-source LLaVA family, including LLaVA-Mistral-7H (Liu et 2023b) and LLaVA-1.6-34B (Liu al.)
2023a). They all share the (Radford et al. 2021)) but are fine-tuned Hae--eemes------eeee
same visual model peenaallclp-ii--laag-paahhh----- from different language backbones (e.g.. Mistral (Jiang et al. (023).
(Research] )023)). Also, the former model uses Grouped Query Attention (GQA) (Ainslie et al. 2023). while the latter model uses Multi-Head Attention (MHA) (Vaswani et al. 2017).
Implementation Details. We use AWS EC2 P4 instance equipped with 8 A100 40GB GPUs for evaluation. First, we sample three tasks from Imms-cval (Bo Li & Liu 2024), including Coco-Caption (Chen et al., 2015), Doc (Mathew et al. 202ID. and Math Vista (Lu et al, 2023 These tasks are representative, and cover OCR, reasoning, and world knowledge domains. Second, compare accuracy of approach against full-cache baselines and previous KV cache sparsification methods including StreamingLLM (Xiao et al. (023a). H2O (Zhang et al. 2024b), ZipCache (He al. 2024) and PyramidKV (Zhang al. 2024a). In appendix AA.3] show more comprehensive experimental results with additional datasets (TextVQA (Singh et al. 2019), ChartQA (asrry al. 2022)) and methods that do not focus on KV cache compression but are still relevant (FastV (Chen et al. (0255) and HiRED (Arif et al.) 2024). We apply KV cache sparsification in line with these baselines by retaining the most recent tokens and selecting the Top-K tokens according to their corresponding scoring policies. All baselines are configured with their default settings, except that the KV cache budget is scaled proportionally to the prompt length, and the recent token window size is fixed at 10% of this budget to enable a fair comparison. Finally, benchmark latency with varying context lengths and batch sizes.
5.1 ACCURACY EVALUATION
The accuracy evaluation results are shown in Figure 9 and Table 3 We report the average accuracy score with KV cache budget varying from 1% 100% of prompt length. Overall, VL-Cache outper- forms other baselines across range of KV cache budgets and different language model backbones.
CoCo-Caprion
DoCVOA
70
40
80
CIDE:
60
30
40
30
20
100
25
0.
20
10
00
100
10
20 40
100
Token Budgets (%6
Token Budgets (%)
Token Budgets (%)
(a) LLLVVAVV..-Mistrrlll
(b) -Livtall.--MMstrra-7777
(c) VI
Coloo Caption
DecvOA
Mathvista
125
80
45
100
60
40
CIDE
40
50
20
25
25
0
10
20 40
60
80
100
5
10
20
40
60
80
100
10
20
40
60 80 160
Token Budgets (%)
Token Budgets (%)
Taken Budgets (%)
(d) LLaVA-VV.6-34B
(e) LLaVA-V11.66
(0) LLLVA--V1.6-34B
Figure 5: Evaluation results on different datasets with varied cache budgets. VL-Cache achieves comparable accuracy against full-cache and outperforms multiple baselines with limited KV cache budget. Interestingly, we found that VLMs occasionally perform slightly better with a partial KV cache. We attribute it to the regularization effect of KV cache compression.
8

---

Published as a conference paper at ICLR 2025
For the Coco-C Seco-Caption dataset (Figure 9 (a), (c) and Table B). all baselines maintain high accuracy when the KV cache budget exceeds 40% for VLMs. However, as the KV cache budget further reduced, accuracy significantly declines. It is worth noting that even with only 5% to 10% KV cache budget, V1.-Cache can achieve accuracy that is comparable to the full KV cache. H2O and StreamingI.LM allocate the same cache budget across all layers, causing denser layers to miss im- portant tokens in the context, while sparser layers contain redundant tokens. PyramidKV statically allocates KV cache in a monotonically decreasing manner, which is ineffective for all input queries. For the Doc dataset (Figure mance; however, VL-Cache consistently outperforms all other baselines. H2O and StreamingLLMM lack robustness, with performance significantly declining as the KV cache is further compressed. For the MathVista dataset (Figure 9 (c). (f) and Table our method again outperforms other meth- ods in most cases. Meanwhile, multiple methods achieve near full-cache performance with 1% KV cache. It's likely due to the fact that 54% of the dataset are multiple-choice questions, which are less
9 (b), (d) and Table (3.. PyramidKV demonstrates strong perfor-
challenging than free-form questions.
Dataset
Model
Method
1%
5%
10%
20%
40 %
60%
80%
100%
VL-Cache
2.64
82.53
100.36
102.00
99.93
101.07
100.08
100 68
101.21
LLaAA-Mistral-TB
PyramidKV
4.01
23.21
66.41
80.76
97.76
100.75
101.38
100 68
SteamingLLM
0.05
11.82
33.98
91.87
101.47
101.07
101.6
100.68
Cocu Caption
ZipCache
4.4
53.00
66.41
92.36
99.51
100.71
102.86
100.68
(Metrie: CIDEr)
VL-Cache
120.11
137.35
139.42
138.58
139.19
138.01
135.07
H2O
20.87
58.14
136.27
140.30
135.07
LL
PyramidKV
93.47
116.91
129.59
138.53
138.17
140.15
135.07
StreamingL.LM
21.57
131.51
134.26
135.07
ZipCache
12.92
131 61
134.98
140.59
140.63
137.90
135.07
VI-Cache
43
59
64
67
67
67
68
H2O
66
67
68
LLVasMMst
PyramidKV
42
58
60
61
66
64
68
68
StreamingLLM
60
DocVQA (Metre:: ANLS)
ZipCache
41
40
60
45
65
68
67
68
VL-Cache
85
85
85
H2O
28
41
82
82
84
85
85
LLaVA-1.6 34B
PyramidKV
79
86
85
85
StreamingLLM
76
80
82
85
ZipCache
55
74
81
85
85
85
85
VL-Cache
42
42
41
H2O
32
35
36
38
38
40
39
41
LLaVA-Mistral-7B
PyramidKV
41
40
41
StreamingLLM
30
35
33
34
40
39
38
41
MathVista (Metrie: ACC)
ZipCache
39
40
42
40
41
VL-Cache
45
42
H2O
37
40
40
39
42
44
44
43
LLaVA-L6634B
PyramidKV
44
StreamingL.LM
29
35
34
28
38
42
42
43
ZipCache
Table 1: With 10% KV cache, VL-Cache achieves close to 100% of full cache performance on most datasets and models and significantly outperformed other methods. Since all KV cache compression methods will converge full cache performance as the more cache budget is granted, strong perfor- mances at low cache budgets attest to VL-Cache's effectiveness in retaining important KV tokens.
5.2 SPEED BENCHIARK
In order to show the speed advantage of VL-Cache, measure the GPU kernel latencies of prefill and decoding forward passes with synthetic prompts. following the method in Kwon et al. (2033) We vary the size of the prompt from IK tokens 128K tokens to scale our method context. Batch sizes vary from to 64 and are static, meaning that all requests get prefilled and decoded concurrently. We assume that the prompt template format remains similar our accuracy benchmarks, so we use the last 50 tokens of the prompt to determine which tokens to evict from the KV cache. For both prefill and decoding the baseline, we used default settings from the Hug- gingFace implementation including CUDA-based FlashAttention-v22 To optimize performance in our VL-Cache, we applied our Triton-based solution for self-attention forward pass, layer-wise
very long
hhtp:///minnnnnneeooooooc//oooon infer gpu one

---

sparsity evaluation, and modality-awart token scoring, as detailed in appendix A.5 The speedup is Published as a conference paper at ICLR 2025
calculated VL.Cache latency"
Baseline latency
In Table 2 we observe that with 50 query tokens for calculating attention statistics, the overhead of our method is just 1-4% of the prefill latency. See Appendix for detailed measurements of the overhead and discussion how to reduce the overhead of statistics calculation for a large number of query tokens. During decoding, we run 99 forward passes for total of 100 output tokens. We observe up 7x decoding speedups that are attributed the reduced size of the KV cache, which we compressed to 10% for this benchmark.
Overall, we see that the end-to-end speedup is bounded by the prefill latency, which is mostly un- changed by VL-Cache. For example, with prompt length of 128K and batch size of 1, the decoding speedup of 7.08x is diluted by prefill taking 53% of end-to-end latency, which results in 1.66x end-to-end speedup. End-to-end speedup will monotonically approach the decoding speedup as the count of output tokens increases. This highlights that KV cache sparsity will give the best speed advantage in tasks with long outputs, such as image captions, video descriptions, chain-of-thought multi-modal reasoning, etc.
800
Batch Size
Prompt Length
Prefill Speedup
Decoding Speedup
End-to-End Speedup
64
IC Cache Strategy
Pull KV Cache
VLCoche 110% budget
1uFal Jatunty Io. per bolem
2k
0.96
1.19
1.16
600
8k
0.97
1.70
1.49
500
32k
0.99
3.32
1.85
400
128k
0.99
7.08
1.66
300
4
0.98
1.68
1.50
16
200
4
8k
0.98
3.16
1.95
4
32k
0.99
6.07
2.06
100
16
2k
0.98
3.03
1.99
100
125
16
8k
0.99
5.61
2.27
wwwer/tro.ghhou..eer.oo
64
2k
0.98
5.23
2.33
Figure 6: Server-level throughput v.s. request-level latency curve (prompt length 2K). Labeled points indicate batch size.
Table 2: Performance metrics by batch size and prompt length for 100 output tokens.
In our implementation of both the baseline and VL-Cache, maximum batch size is limited by peak memory usage during prefill instead of KV cache size, so compression of KV cache does not lead higher batch size. In future work, continuous batching and chunked prefill could be used to eliminate the prefill memory bottleneck, which would expose our method's advantage in raising maximum batch size when the size of KV cache is the bottleneck higher batch size. Finally, we summarize the trade-off between request-level latency and server-level throughput in Figure 6 We note that VL-CCche offers both higher peak throughput and lower latency for any desired server-level throughput.
CONCLUSION & FUTURE WORK
In this paper, we propose VL--achee, novel KV cache compression optimized for VLMs. We p discovered the unique sparsity patterns of visual and language tokens throughout the prefill and decoding phases. With these observations, we introduce modality-aware token scoring policy and sparsity-aware cache budget allocation to reduce KV cache size without accuracy loss. Empirical results on multiple benchmark datasets demonstrate that when maintaining only 10% of the KV cache, our method achieves accuracy comparable to the full KV cache and outperforms all existing methods. a speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by to 2.33x relative to the full KV cache. As for future work, we identify two opportunities. Firstly, our method only focused on compressing prefill KV cache due to the high prompt to output token ratio (median 320x) of current vision- linguuee tasks. Periodical compression of decoded tokens can be developed for long output tasks while striking a balance between latency overhead and memory savings. Secondly, it would be interesting extend our method for video models since video inputs will impose even more memory pressure due to KV cache.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, 1lya Soloveychik, and Pu- rushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:1 6:114-127, 2024.
Joshua Ainslie, James Lee- Tee-Th Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi- query transformer models from multi-head check- points. arXiv preprint ar3iv 13245, 2023.
Anthropic. Introducing the next generation of claude, 2023, URL httpttss///...attrroa.. com/ news c aude- seees//lladee222 3- 20-ffmml family/
Kazi Hasan Ibn Arif, Jin Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: crention-guider token dropping for efficient inference of high-resolution environments. arXiv preprint arXiv:2408. 10945,
vision-language models in 2024.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. 2023.
Kaichen Zhang* Fanyi Pu* Xinrun Du Yuhao Dong Haotian Liu Yuanhan Zhang Ge Zhang Chunyuan Li Bo Li*, Peiyuan Zhang* and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024. EvolvingLMMs Lab/ Lmms-eval
URL htttp::/github... com/
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang. et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-45,2)) 2024.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang- An image is worth 1/2 tokens after layer 2: Plug-and-plas inference acceleration for large vision- language models. In European Conference Computer Vision, pp. 19-35. Springer. 2025.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision lan- guage models: Smaller, faster, stronger. arXiv preprint 21X1v:2310.09199 2023.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitiick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 100400550.000: 2015.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) 2023.
Suyu Ge. Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01800, 2023.
Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:240S.14256, 14256, 2024.
Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Yuhan Dong, Yu Wang, et al. Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics. Proceedings of Machine Learning and Systems, 6: 1-18-161, 2024.
Raisa Islam and Owana Marzia Moushi. Gpt-4o: The cutting-edge advancement in multimodal 11m.
Authorea Preprints, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 1-626, 2023.
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. (InfiniGen]:: Efficient generative inference of large language models with dynamic [KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 155-172, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, August 2024a. URL https: //11
ava-vl gi thu..hu....... /o/blo///o og w0w/////0000--000 05- ava ava-oneviln sion/
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404. 14469, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023b.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Vietor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for 11m kv cache compression at test time, Advances Neural Information Processing Systems, 36, 2024a.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu, Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Ahmed Masry, Xuan Long Do Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar, Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. 8763. PMLR, 2021.
Nous Research. Nous hermes 2 on yi 34b, 2023. Woou:Reseearcch/oos---eeere -aieees----i- -2Y1--3BB
URL
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint arXiv:2406. 10774, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud. Yonghui Wu, Jean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv: / 2023.
12

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 220..33997777 17453, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. The Twelfth International Conference on Learning Rep- resentations, 9 2023b.
Dongjie Yang. XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyra- mid kv cache compression for high-throughput Ilm inference. arXiv preprint arXiv:2405. 12532, 2024a.
June Yong Yang. Byeongwook Kim, Jeongin Bac, Beomseok Kwon, Gunho Park, Eunho Yang. Se Jung Kwon, and Dongsoo Lee, No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402. 18096, 2024b.
Hao Yu, Zelan Yang. Shen Li, Yong Li, and Jianxin Wu. Effectively compress heads for Ilm.
arXiv preprint arXiv:2406.07056 2024.
Yichi Zhang. Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02009, 2024a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances Neural Information Processing Systems, 36, 2024b.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 VISION-LANGUAGE PROMPT TEMPLATE CONSTRUCTION
Constructing prompt templates in image-based conversations is common practice for VLMs LLi ct al., 2024aa Team et al. 2023; Islam & Moushi, 2024 Bai et al. 2023 Anthropic 2023) as it
instructs language models generate more accurate responses. For example, as illustrated in Figure
7 the input image is processed through visual encoder and projection layer to generate an image embedding, represented by the <image>, For language imput, beyond user input, a prompt template employed. With the appropriate prompt template design, regardless of the original image order from user input (whether before or after language inputs), there will always be language- instruction or question in the post-vision position, providing strong signal for our VL-Cache to evict insignificant visual tokens.
carelully observe image provice gleen mage.
imagee emmpyy
The slouds
Assetant
warn golden hara, suggusting It's wtter
Gehin Convenation
Coratrct Yours tongagg Tomplet
Encoddrr
Projection
simage
chot betweer aurlous hurran and anti cot intellgences aulitart The asiitant giezs hoplid, detslled and solte aneweet the humon's
quesi on. USE sinagerna Please caretully
obsenve image and prodid
descrition giron Image. Assaeoi
Plense canetully cissenee the mmae and provice
à brief descristion the given image.
Figure 7: Vision-Language Prompt Template Example.
A.2 ABLATION STUDIES ON THRESHOLD p.
In VL-Cache, we utilized ThresholdFilter in eq to sparsify the attention matrix. It uses relative threshold p that we heuristically set to 0.01 in our experiments. To understand the robustness of our method to this hyperparameter, 7B and Coco-Caption. We vary p from 0.0001 to 0.1 and show the corresponding performance the following plot. We observe that overall, VL-Cache is robust to the choice of small performance variations, 2) higher token budgets seem to be even more robust to p. and 3) there isn't an't single threshold that works consistently better across all cache budgets, but p 0.01 does seem to be a decent choice.
conducted
ablation study using LLaVA-V 1--A-V1.6--M 1.--Misrrll-
has relatively
Ablation Study for Thresholds
102
100
CIDEr
98
96
Taken
10%
04
Thter Budgee:: 40%
0.1 Take- Buckee: 20%
Budkyst 40%
Budget: 10%
0.000
0.001
0.02
Threshold p
Figure 8: performance on Coco-Caption under various cache budgets and threshold values with VL-Cache.
14

---

Published as a conference paper at ICLR 2025
A.3
FULL EXPERIMENTAL RESULTS
Dataset
Model
Method
1%
5%
10%
20%
40%
60 %
80%
100%
VL-Cache
2.64
82.53
100.36
102.06
99.93
101.07
100.08
100.68
H20
7.87
64.45
90.36
104.04
102.64
101.21
102.86
100.68
PyramidKY
23.21
97.76
100.75
101.38
LLVA--Mittal-1
0.05
11.82
33.98
91.87
101.47
101.07
101.6
100.68
ZipCache
FastV
9.98
41.17
88.82
97.47
105.40
110.60
106,90
100.68
Coco-Caption (Merric: CIDFr)
HIRED
13.87
70.90
115.37
109.91
115.09
115.09
105.93
100.68
VI-Cache
120.11
137.35
139.42
138.58
139.19
138.01
335.0
H2O
20.87
58.14
126.8
136.27
140.89
140.30
135.07
PyramidKV
93.47
116.91
129. 59
138.53
138..1
140.15
135.07
LLaVA-1.6-34B
SuvamingLLM
5.69
8.23
21.57
131.51
134.26
139.65
135.07
ZipCache
12.92
131.61
134.98
140.59
140.63
135.07
FastV
3.10
15.86
32.55
80.04
112.99
16..5
120.70
135.07
VL-Cache
43
59
62
64
67
67
67
6$
PyramidKV
42
58
60
61
66
64
68
68
LLaVA-Mistra-7B
Stvaming
ZipCache
40
45
65
68
67
FastV
63
64
DocVQA (Metric: ANLS)
HiRED
38
63
63
VL-Cache
85
85
H2O
82
82
84
PyramidKV
85
85
86
85
LLaVA-L.6-34BB
LL
16
76
80
ZipCache
55
81
85
85
85
85
FastVV
28
HiRED
25
44
52
71
76
81
84
RS
VL-Cache
H2O
32
35
36
38
38
40
39
41
PyramidKV
Li
30
35
33
34
40
39
38
41
ZipCache
40
FastVV
29
38
39
40
MathVista Acc)
HiRED
40
40
VI. Cache
42
45
45
42
H20
40
42
44
PyraiiKY
46
41
LLAVA-L.6344B
SireamingLLM
29
35
34
28
38
42
42
43
ZipCache
Fast
27
27
32
37
37
35
37
43
HiRED
VI- Cache
20
33
41
38
40
40
41
41
H20
PyramidKV
27
29
39
39
38
40
41
41
LLaVA-Mistal-7B
StramiingLLM
39
37
ZipCache
33
40
40
FastV
41
42
CharqQA
HiRED
47
44
(Metric: Acc)
VL-Cache
51
55
54
55
55
H20
42
54
53
PyramidKV
52
56
54
54
55
LLAVA-L.6-4BB
SueamingLLM
43
ZipCache
43
55
52
54
54
54
54
HiRED
11
20
27
44
49
56
54
54
VL-Cache
65.4
H20
43.0
56.5
63.4
63.4
62.4
65.4
65.4
65.4
PyramidVV
35.0
50.1
62.8
65.4
65.4
65.4
65.4
LLaVA-Mistrll-B8
StraamingLLM
19.5
31.4
43,2
53.6
$9.8
59.9
61.8
654
ZipCache
35.6
51.2
59.4
63.4
65.4
65.0
65.4
65.4
FastV
11.1
38.5
48.2
61.1
63.4
61.7
62.7
TedVQA Acc)
HiRED
23.6
54.6
55.4
64.8
64.0
62.4
63.7
65.4
VL-Cache
12.9
70.7
71.4
73.4
72.4
73.6
72.7
72.7
H20
18.3
47.0
63.2
69.5
71.5
74.4
72.7
72.7
PyramidKV
73.4
74.3
1.
SineamingLLMM
0.0
9.6
19.5
34.7
70.4
70.4
71.4
72.7
ZipCache
FastVV
3.6
6.3
11.9
18.7
42.6
60.2
64.4
72.7
HiRED
62.1
69.9
71.2
71.7
72.7
Table 3: Performance of VLMs with different compression methods and datasets. With 10% KV cache, VL-Cache approaches 100% of full cache performance on most datasets and models and significantly outperformed other methods.
15

---

Published as a conference paper at ICLR 2025
CoCo-Caption
CoCo-Cate CoCo-Caption
120
100
125
80
100
CIDER
CIDEr
60
40
50
Pyramistt
Pyromiky
20
SteamingLLM
25
SteamngLUM
2yCashe
2pCache
Podt
Festy
0
20 40
10
80 100
Token Budgets (%)
Token Budgets (%)
(a) LLaVA-VI..-Mistral-7B
(b) LLaVA-V1.6-34B
DocVQA
DocVQA
70
60
50
anis
40
30
Pyraiittt
20
Pyroniaty
SmeamingLM
SteamingLM
ZyCashe
20
Porty
Festy
HRED
10
20 40
60
80 100
Token Budgets (%)
Token Budgets (%)
(c) LLLVA-VI.6-Misttrll
(d) LL.VA-V1.6634BB
MathVista
MathVista
40
40
35
ACc
Acc
35
30
Cachc
30
SteamingUM
25
Fall
7pCache
25
Ralty
HRED
HRED
20 Token Budgets (%)
60 80 100
5
10
20
40
60
80 100
Token Budgets (%)
L.LLVA-VV..-Mistalll
(f) LOAVA-VV.66344
Figure 9: Evaluation results on different datasets with varied cache budgets. VL-Cache achieves comparable accuracy against full-cache and outperforms multiple baselines with limited KV cache budget
A.4 EXTENDED RELATED WORKS
The KV cache, while essential for transformer-basee LLM and VLM family, demands significant computational resources and memory, limiting inference speed. To address these challenges, re- searchers have explored various KV cache compression techniques, such as KV cache sparsification, quantization, or combination of both.
KV cache sparsification. Heavy-Hitter Hitters (H20) (Zhang et al., 2024b) employs cumulative attention scores to greedily evict unimportant tokens. However, this method tends to accumulate more atten- tion on the initial tokens, introducing bias and negatively impacting the identification key tokens
16

---

Published as a conference paper at ICLR 2025
ChartQA
ChartQA
50
relaxed overall
40
relaxed overall
40
30
30
Cacach
20
VuCache
R2D Pynmisty SreamingLU
P.D
20
Pyrmiaty
10
SteamngguM
2,ccche
10
1
10
20
40
60
80
100
5
10
20
40
60
80 100
Token Budgets (%)
Token Budgets (%)
(g) LLaVA--V..6-Mistral-77
(h) LLaVA- V..VA-
TextVQA
TextVQA
60
60
exact match
50
exact match
40
40
30
Cacht
VL Cachee 120
120
Pynnisty SmominglLM
20
SteamingLLH
2pCashe
1
10
20
40
60
80 100
5
10
20
40
60 80 100
Token Budgets (%)
Token Budgets (%)
(i) L0
(j) LLVVA-V1.6-344
Figure 9: Additional evaluation results on different datasets with varied cache budgets, demonstrat- ing VL-Cache's 's consistent performance on wide range of settings.
during decoding. ZipCache (He et al. 2024) further normalizes the cumulative attention scores, leading to more precise prediction. Keyformer (Adnan al. (0244) proposes novel score function to predict the importance of each token and only window tokens and dynamic key tokens are kept in the KV cache. PyramidKV (Yang et al., 2024a) allocates progressively smaller KV cache sizes as layers get deeper. This approach can achieve an absolute accuracy improvement of nearly 20.5 on specific tasks.
KV cache quantization. This method reduces the size of the KV cache by utilizing lower-bit representations of the key and value pairs. In particular, FP8 and INT8 quantization of KV caches commonly used for LLM inference. Further compression of KV cache into lower-bits is also being explored, e.g. KIVI (Liu et al. 2024b) compresses KV cache into 2-bit representations.
Combination of quantization and sparsification. Quantization and sparsification (Yang et al.) 2024b) technically could have orthogonal implementations that can enabled together to obtain further. However, since both lossy methods that can impact the accuracy of downstream tasks, careful accuracy evaluation of the quality of the output need to be conducted.
Visual Token Reduction. In the domain of VLMs, most works focus pruning embedded image features either before they enter the LLMs or within the hidden states of the LLMSs. HiRED AAni et al. 2024) leverages sparse attention patterns in vision transformers (e.g., CLIP-VIT) to discard visual tokens, effectively extracting key objects from images and reducing computational resources required during subsequent LLMs generation. However, disregards textual prompts, potentially dropping essential tokens unexpectedly. FastV (Chen et al. 2025). on the other hand, prunes hidden states specific layer within LLMs, significantly accelerating computation later layers. How-
17

---

Published as a conference paper at ICLR 2025
ever, it applies uniform token budget across all subsequent layers, resulting in suboptimal accuracy when the token budget is highly constrained.
The KV cache compression methods mentioned above primarily target LLMs, while in VLMs, most approaches focus on reducing visual tokens in the input image or the hidden states. And there has been limited exploration of KV cache sparsification specifically for VLMs. VL-Cache proposed in this paper analyzes and utilizes the unique sparsity pattern in VLMs, which results in better accuracy for VLM inference.
A.5 EFFICIENT IMPLEMENTATION
In order to discard tokens that received low attention during prefill, we need to calculate statistics from the attention score matrix during the prefill stage: (1) the average attention score for each token in key dimension; and (2) the count of attention scores that are smaller than p% of the maximum within the query Q dimension. The ThresholdFilter cally faster than alternative methods of truncating a distribution (such as top-p and top-k), since it does not require the attention scores to be sorted.
adopted in our algorithm is asymptoti-
Regular attention kernels, such as FlashAttention 2023), do not materialize the attention scores in HBM, we need new memory-efficient kernel to calculate the attention statistics. Overall, we aim to schedule these operations to ensure that: (1) The attention mask is written HBM: (2) the QK product is written HBM; (3) the attention scores are not written to HBM: (4) mmmilarly-parllelizee operations (e.g. div and add) are fused to avoid intermediate tensors in HBM.
Dao 2023 and PagedAttention (Kwon et al,
In the initial attempt, we used TorchCompiler with the Triton backend, which addressed require- ments (1), (3), and (4), but not (2). An efficient fusion of the matmul operation with softmax is described in the FlashAttention paper and can be implemented in Triton, but TorchCompiler was not able to apply this optimization. We will now describe the optimal algorithm that satisfies all performance requirements.
The key to efficient fusion of matmul and softmax to partition the work along the Q dimension and then hold a single tile of Q tensor in SRAM, while continuously loading tiles of the other tensor and computing the reductions without writing the QK product back to HBM. The 2 reductions required by softmax ipre-exponentiation max, and post-exponentiation sum) are both along the K dimension, so partitioning along the Q dimension is required for these operations. The online softmax algorithm can be used update the partially computed sum.
For average attention score in the KV context, we need to reduce the attention scores over the Q dimension, which is not aligned with the previous computation. Instead, we need second kernel which partitions the work over the K dimension, so that a compute block can reduce over the Q dimension. However, since we do not want materialize the attention scores, the second kernel needs to recompute softmax again. In order do that, we re-use the max and sum over Q dimension from the first kernel, and these tensors (both (()) space) are the only intermediate tensors that are stored in HBM.
For sparsity ratio, counting the scores below the threshold is not possible until the maximum along the K dimension is known. We can use the intermediate maximum tensor that was produced by the first kernel to count the sparse scores in the second kernel. Since each thread block will output its own count, we need the 3rd kernel to sum up the partial counts.
In summary, the optimized computation consists of 3 kernels executed serially:
1. Softmax + row-wise stats (reductions over K dimension). Reads: Q and K from HBM. Computes and writes HBM:
(a) pre-exp max over K dimension (used in softmax for numerical stability)
(b) post-exp sum over dimension (used in softmax as the normalization factor)
2. Softmax + column- wise stats (reductions over Q dimension). Reads: Q, K, pre-exp max, and post-exp sum from HBM. Computes and writes to HBM:
(a) the post-softmax sum over the Q dimension
(b) count of attention scores below threshold over the dimension
18

---

Published as a conference paper at ICLR 2025
3. Sum reduction of the partial count of attention scores below threshold
We implemented these 2 kernels Triton and observed that each had roughly the same latency as the FlashAttention kernel, implying that this partioning is optimal. However, with this approach, we would want to use only a few dozen tokens for querying statistics during prefill.
A.6 SPEED BENCHMARK RESULTS
In this section, we present detailed metrics from the speed benchmark results, including exact times for both the prefill and decoding stages. also provide an overview the benchmarking method- ology. Using Torch Profiler, measured GPU kernel latencies and aggregated them to compute the total latency for each operation.
Prefill Latency (ms)
Decoding Latency (ms)
Speedup
Batch Size
Prompt Length
Full Cache (Baseline)
VL-CCahe (10%, Ours)
Full Cache (Baseline)
VL-Cache
Prefill Specdup
Decoding Speedup
End-to-End Speedup
(10%, Ours)
2k
330.7
345.8
3821.8
3222.3
0.96
1.19
1.16
8k
1396.1
1434,9
5877.1
3459. 1
0.97
1.70
1.49
32k
7257.2
7354.9
14247.2
4294.0
0,99
3.32
1.85
128k
59740.9
60063.9
52547.0
7423.0
0.99
7.08
1.66
4
2k
1287.0
1315.4
6156.0
3661.0
0.98
1.68
1,50
4
8k
5486.5
5604.0
14229.1
4499.6
0.98
3.16
1.95
4
32k
28908.7
29206.7
47321.1
7795.3
0.99
6.07
2.06
16
2k
5038.3
5144.3
15078.8
4978.7
0.98
3.03
1.99
16
8k
21843.6
22168.4
47688.7
8494.8
0.99
5.61
2.27
2k
20113.2
20448.2
49666.6
9499.4
0.98
5.23
2.33
Table 4: Detailed performance metrics by batch size and prompt length for 100 output tokens
This breakdown latency by GPU kernels enabled us to 1) avoid measuring CPU latency that does not relate method, 2) group the latency into different buckets for reporting. Empty cells mean that the inference server crashed due to insufficient GPU memory during prefill.
The following table presents the cumulative latency of all GPU kernels that are related to computing attention statistics from 50 query tokens, eviction scores for all prompt tokens, and copying the KV cache tensor after eviction of 90% of the KV cache into contiguous memory. This prefill overhead stated in milliseconds:
batch size
1000
2000
4000
8000
16000
32000
64000
128000
5.20
7.20
10.10
17.30
28.30
49.70
100.20
189.60
2
6.40
9.50
15.80
29.00
51.80
92.50
188.50
4
8.50
15.20
27.40
52.50
93.30
175.80
13.70
26.80
50.90
96.60
178.40
16
25.20
51.30
95.60
183.50
48.50
97.10
182.60
64
91.60
186.70
We define the prefill latency as the latency of generating output token - starting with input token IDs and ending with output logits, not including tokenization and sampling. The following table presents the prefill speedup purefilttverreeen We observe that the overhead is never more than 6% and becomes smaller as the prompt gets longer:
batch size
1000
2000
4000
8000
16000
32000
64000
128000
0.94
0.96
0.97
0.97
0.99
0.99
0.99
0.99
2
0.96
0.97
0.97
0.98
0.98
0.99
0.99
4
0.97
0.98
0.98
0.98
0.98
0.99
0.98
0.98
0.98
0.99
0.99
16
0.98
0.98
0.99
0.99
0.98
0.98
0.98
64
0.99
0.98
19

---

Published as a conference paper at ICLR 2025
We calculated decoding latency as the difference between end-to-end latency and the prefill la- tency. Since we use static batching with a constant number of output tokens, it's easy to de- compose the end-to-end latency into its components without concerns for queue waiting, sched- uler stalling, sequence termination, etc. The following table presents the decoding speedup cache
decoding lateney
decoding latency with 10% KV cache )
batch size
1000
2000
4000
8000
16000
32000
64000
128000
1.10
1.19
1.37
1.70
2.31
3.32
5.17
7.08
2
1.18
1,36
1.68
2.27
3.16
4.56
6.78
1.36
1.68
2.26
3.16
4.39
6.07
8
1.73
2.29
3.13
4.37
5.87
2.19
3.03
4.18
5.61
32
2.94
4.12
5.53
64
3.83
5,23
The end-to-end speedup is calculated as VLCache latency"
Baseline latency
VI.Cache latency includes prefill latency,
VLCache overhead, and 99 decoding passes to generate a total of 100 output tokens:
batch size
1000
2000
4000
8000
16000
32000
64000
128000
1.09
1.16
1.30
1.49
1.71
1.85
1.89
1.66
2
1.16
1.30
1.49
1.73
1.89
1.97
1.94
1.30
1.50
1.74
1.95
2.06
2.06
1.54
1.79
2.00
2.17
2.23
16
1.75
1.99
2.17
2.27
32
1.98
2.19
2.32
64
2.18
2.33
A.7
MEASURING ATTENTION TO VISUAL AND LANGUAGE TOKENS
As we have established in the Section 3.1 the attention sparsity during decoding be predicted from attention scores prefill phase. To further understand the importance of visual tokens and language tokens in the prompt, examine the division of attention from decoding tokens these two modalities of prompt tokens.
We propose Contribution for quantitative analysis and visualization of the layer-wise modality- specific attention patterns. Let 7' be the current sequence length, be the index of the first decoded token, and A() E RTT be the attention matrix at layer for one specific head.
Eje ThresholdFilter((4 (A)) Ejela ThresholdFilter (A(),))
Contribution mod
(3)
Here, we use Jmod and Jali to denote KV cache indices of one selected modality (language or vision) and all modalities respectively in the input prompt. As shown in Figure I0, VLMs allocate primary attention to visual tokens in the first layer, while the second layer, the contributions of visual and language tokens comparable. From the third layer onward, language tokens dominate, with slight increase in visual token contribution in the middle layers. Additionally, we also propose Coverage analyze the ratio of the number of tokens from specific modality, defined as follows:
E Eje TApK(A
Coverage mod
(4)
T
EjeJau TopK (AO) 440,k)s
12t
Specifically,
TopK(A, 1[Ai E (A(r-++1) (lr 1
(5)
20

---

Published as a conference paper at ICLR 2025
Here, we define l as |a T, as the number of columns of matrix A, and overload notations to A!") as the n-th order statistic of the row vector. Additionally, a E (0, 1) denotes the token budget threshold (e.g.
10% means only the 10% of sequence length of KV cache is retained). After KV cache compression with TopK selection, we assess the ratio of visual attention to language attention among the remaining tokens. In Figure [1] with
10% applied, similar trend in coverage is observed compared to Figure [O Notably, in the middle layer, visual tokens constitute a larger proportion than their contribution, indicating that the contribution per token is smaller.
Furthermore, since contribution of language and visual tokens differ by layer, the optimal budget for each layer may depend on the ratio of visual language tokens in the prompt. contrast, previous pyramid-style cache allocation methods do not adapt to the input prompts at inference time.
Average Contribution in Decoding
Average Coverage in Decoding
1.0
1.0
Contributio
0.8
0.8
0.6
Coverag
0.6
Visual Taken Contribution Taken Contribution
Visual Taken Coverage Textual Tiken Coverage
0.4
0.4
0.2
0.2
0.0
0.0
0
5
10
15
20
25
30
10
15
20
25
30
Layer Index
Layer Index
Figure 10: Contribution
Figure 11: Coverage
21


Reviews:
[{"review_id": "Yeqyd4CEjq", "reviewer": "Reviewer_XU5m", "summary": "This paper addresses the challenge of efficiently storing and accessing large Key-Value (KV) caches in Vision-Language Models (VLMs), which are crucial for handling visual contexts like images or videos. Existing KV cache compression methods developed for Large Language Models (LLMs) don't perform optimally when applied to VLMs. To solve this, the authors propose VL-Cache, a compression method designed specifically for VLM inference. VL-Cache utilizes a layer-adaptive, sparsity-aware cache allocation and a modality-aware token scoring approach to reduce cache size without losing accuracy. Experimental results show that this method retains comparable accuracy with only 10% of the original cache, accelerates token generation by up to 2.33x, decoding by up to 7.08x, and reduces GPU memory usage by 90%.", "questions": "Please see the section on weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The experimental results are very good; the performance is significantly better compared to other KV-cache methods.\n2. Sparsity-aware layer-wise KV cache allocation and modality-aware token scoring are meaningful and have also been proven effective in the experiments.", "weaknesses": "1. The writing is somewhat complex and unclear; it takes a few careful readings to understand. Adding more illustrations to explain the method would improve it.\n2. For MathVista, it seems that the KV-cache isn’t important? Even with only 1% of it, the performance is quite good, and sometimes the performance with 100% KV-cache isn’t even the best. Could you explain that?\n3. In terms of writing, Section 3's PRELIMINARY EXPERIMENT is too long, while there’s relatively little written about the methodology. This writing style increases the burden on the reader. Readers likely want to see the method as soon as possible, so the focus should be on discussing your approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tCR22tkGL4", "reviewer": "Reviewer_XVzL", "summary": "This paper introduces a modality-aware KV cache compression method for Vision-Language Models (VLM). It introduces a dynamic cache budget allocation mechanism and employs post-vision language tokens to compute attention scores.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The insight that the attention patterns of generated tokens closely align with language tokens is valuable.", "weaknesses": "1.\tAlgorithm 1 only outlines two procedures without detailing how these procedures are integrated into the algorithm.\n2.\tDefinition 3.1 and the following text is unclear. The terms Psi(S) and S_Psi are used without clear explanation. A precise definition of Cache Hit Rate is essential.\n3.\tFigure 4 lacks clarity. It does not effectively illustrate how the budget is allocated or how post-vision attention is utilized.\n4.\tIn the related work, ZipCache proposes a policy using normalized attention scores. This should be considered and compared.\n5.\tThe KV cache for generated tokens is not addressed.\n6.\tThe performance of VL-Cache is not convincingly demonstrated in Table 2. VL-Cache does not significantly outperform other methods, while H2O has achieved the strongest performance on many benchmarks. It is recommended to test VL-Cache on more challenging benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AgphlmAAhL", "reviewer": "Reviewer_KiPr", "summary": "The authors aim to accelerate VLMs by reusing KV caches for encoding visual contexts. The authors observe the limitations of existing KV cache compression methods and propose a specific version for VLMs. Based on the observation of sparsity patterns in prefilling and decoding stages, the main idea is to design a cache allocation method that balances the cache size and model accuracy. Evaluations based on vision-language benchmarks show the performance of model accuracy and speedups.", "questions": "Please refer to the questions in the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe authors identify the I/O bottleneck between GPU’s HBM and SRAM in the decoding stage of VLMs, motivating the innovative design of KV cache compression. This understanding helps to address the challenge of scaling VLMs and optimizing their performance.\n2.\tThrough Figure 1, the authors illustrate the distinctive attention sparsity patterns between VLMs and LLMs, emphasizing the need for modality-aware compression techniques tailored for VLMs.\n3.\tI have learned a lot from the author's observations. The KV cache budget and compression ratios differ in layers and modalities. This provides good insights into the optimization of VLM cache compression.\n4.\tTo compress the VLM cache, the authors propose two key methods: a sparsity-aware cache budget allocation strategy and a modality-aware token scoring policy, to optimize the cache budget allocation and token score measurement, respectively.\n5.\tThe experimental results show that the proposed methods achieve significant inference speedups and GPU memory savings.", "weaknesses": "1.\tIn Figure 1, the authors measure attention scores to verify the modality boundary, but the specific methodology for calculating these scores remains unclear. Please provide additional details on the measurement process.\n2.\tThe term “post-vision attention” is frequently used but not explicitly defined, making the relevant sentences difficult to follow (“the attention patterns from the output language tokens is much closer aligned with the language tokens that follow the visual tokens in the prompt (the post-vision attention) rather than the visual tokens themselves”). A clear description of post-vision attention at its first occurrence would enhance the paper's readability.\n3.\tThe threshold filter in Section 3.1 utilizes the hyper-parameter p to control sparsification. Conducting ablation studies on the impact of different p values would provide deeper insights into the robustness and sensitivity of the proposed method.\n4.\tTo Algorithm 1, while it dynamically allocates the cache budget based on each prompt, its computational complexity raises my concern. Please analyze the computational overhead (in lines 11 to 14).\n5.\tBesides, Algorithm 1 does not explicitly reflect the modality-aware design as has been highlighted in this paper. Please clarify how the algorithm incorporates modality properties.\n6.\tIn experiments, the I/O traffic amount caused by KV caching should be reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DixzKq25nF", "reviewer": "Reviewer_X8F2", "summary": "The paper tackles the challenge of accelerating vision-language models (VLMs) by leveraging a sparse Key-Value (KV) cache. Unlike existing compression methods for large language models (LLMs), which are less effective for VLMs, the proposed VL-Cache takes advantage of unique sparsity patterns in visual and text tokens. It uses layer-adaptive budget allocation to allocate cache resources based on sparsity levels and modality-aware token scoring to determine token importance with post-vision text tokens. By retaining only the most crucial cache entries, VL-Cache effectively reduces memory usage and latency. Experimental results on three lmms-eval tasks demonstrate the method’s effectiveness.", "questions": "1.\tSome notations in the proposed method lack clarity. For instance, the meanings of $\\Gamma$ and $Z$ are not sufficiently explained, making it difficult to understand their roles in the algorithm. Providing more detailed explanations would improve the clarity of the presentation.\n\n2.\tThe authors state that the maximum batch size is constrained by peak memory usage during prefill rather than by the KV cache size, suggesting that compressing the KV cache does not increase the batch size. Does this imply that the proposed KV cache compression does not actually reduce overall memory usage? If so, why is that the case? This raises concerns about whether the title “KV Cache Compression” accurately reflects the method’s ability to save GPU memory, as the proposed approach appears to be more of a sparse computation technique applied during decoding.\n\n3.\tIn Table 3, the reported speedup is based on generating a total of 100 output tokens. It would be helpful if the authors could provide the exact prefill and decoding times separately. Given that with a prompt length of 128k and only 100 output tokens, the prefill time is likely to dominate the overall processing time, having this detailed breakdown would clarify the impact of the proposed method on different stages of the inference process.\n\n4.\tIn Section 4.2, the authors point out that accumulated attention has the limitation of assigning disproportionately high scores to earlier tokens due to summation over the full query dimension. To address this, they propose accumulated post-vision attention. However, there is another approach [1] that addresses the same issue using normalized attention. It would be helpful if the authors could include more discussion on this alternative method and provide additional experimental comparisons to better illustrate the benefits of their approach.\n\n5.\tThe experimental evaluation is insufficient, as it lacks comparisons with two important baselines for VLM acceleration: FastV [1] and HiRED [2]. Additional comparisons with these methods are needed to strengthen the evaluation.\n\n6.\tAlthough the proposed method targets vision-language models, the current evaluation lacks coverage of a comprehensive range of tasks. To better demonstrate its effectiveness, it would be beneficial for the authors to include additional results on video benchmarks, such as Video-MME [4]. This would help showcase the method’s applicability to video tasks.\n\nReference:\n\n[1] ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification. NeurIPS 2024.\n\n[2] An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. ECCV 2024.\n\n[3] HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models in Resource-Constrained Environments. arXiv 2024.\n\n[4] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. The sparsity analysis of prefilling and decoding effectively motivates the use of adaptive sparsity. The proposed method achieves significant speedup in decoding, particularly with long prefill tokens.", "weaknesses": "Please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)