PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
Authors: Ao Wang
Hui Chen
Jianchao Tan
Kefeng Zhang
Xunliang Cai
Zijia Lin
Jungong Han
Guiguang Ding
Venue: NeurIPS 2025 poster ()
Keywords: large vision-language models
KV cache compression
Primary area: deep_learning
TL;DR: A kv cache compression method for large vision-language models

Abstract: Recently, large vision-language models (LVLMs) have rapidly gained popularity for their strong generation and reasoning capabilities given diverse multimodal inputs. However, these models incur significant computational and memory overhead during inference, which greatly hinders the efficient deployment in practical scenarios. The extensive key-value (KV) cache, necessitated by the lengthy input and output sequences, notably contributes to the high inference cost. Based on this, recent works have investigated ways to reduce the KV cache size for higher efficiency. Although effective, they generally overlook the distinct importance distributions of KV vectors across layers and maintain the same cache size for each layer during the next token prediction. This results in the significant contextual information loss for certain layers, leading to notable performance decline. To address this, we present PrefixKV. It reframes the challenge of determining KV cache sizes for all layers into the task of searching for the optimal global prefix configuration. With an adaptive layer-wise KV retention recipe based on binary search, the maximum contextual information can thus be preserved in each layer, facilitating the generation. Extensive experiments demonstrate that our method achieves the state-of-the-art performance compared with others. It exhibits superior inference efficiency and generation quality trade-offs, showing promising potential for practical applications. Code is available at
https://github.com/THU-MIG/PrefixKV.

PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Folloving Models Need for Efficient Generation
Ao Wang Hui Chen²*
Kefeng Zhang Xunliang Cai Guiguang Ding BBNist, Tsinghua University *Department of Automation, Tsinghua University
Jianchao Tan
Zijia Lin'
Jungong Han
School of Software, Tsinghua University Meituan Inc.
Abstract
Recently, large vision-language models (LVLMMs) have rapidly gained popularity for their strong generation and reasoning capabilities given diverse multimodal inputs. However, these models incur significant computational and memory over- head during inference, which greatly hinders the efficient deployment in practical scenarios. The extensive key-value (KV) cache, necessitated by the lengthy input and output sequences, notably contributes to the high inference cost. Based on this, recent works have investigated ways to reduce the KV cache size for higher efficiency. Although effective, they generally overlook the distinct importance distributions of KV vectors across layers and maintain the same cache size for each layer during the next token prediction. This results in the significant contex- tual information loss for certain layers, leading notable performance decline. To address this, we present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence. It reframes the challenge of determining KV cache sizes for all layers into the task of searching for the optimal global prefix configuration. With an adaptive layer-wise KV re- tention recipe based on binary search, the maximum contextual information thus be preserved in each layer, facilitating the generation. Extensive experiments demonstrate that method achieves the state-of-the- ofe-the-art performance compared with others. exhibits superior inference efficiency and generation quality trade- offs, showing promising potential for practical applications. Code is available ptttps://gitan com THU-MIG/UUAAHHMM MIC//MIffix/ii
Introduction
Recent years have witnessed the significant advancements of large vision-language models 13 [0]. Based on the powerful Large Language Models (LLMs) [16] 49 27 3 61). these models integrate visual inputs, showing strong generation and reasoning 52
(LVLMs) 37 66 25 35
abilities for various multimodal tasks. They demonstrate inspiring application potential in various
fields like autonomous driving 12 56 and intelligent medical analyses
29
However, despite their remarkable capabilities, the efficient deployment of LVLMs encounters notable challenges in real-world scenarios. This stems from the typical Transformer architecture employed in the LVLMs, which necessitates the global interaction of tokens. During autoregressive decoding, the key and value vectors of previous tokens are thus required be stored as the KV cache and subsequently retrieved by the output token The KV cache grows linearly with the number of
'Equal contributions. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

processed tokens, which leads to notable memory overhead and heavy burden on GPU communication under lengthy sequences. This causes suboptimal efficiency, resulting in the inference bottleneck.
Given this, recent works have investigated pruning unimportant KV vectors or merging adjacent
vectors to reduce the KV cache size while preserving the model performance [40
68 60
67]. For
example, H₂O [68] discards less important ones based on the attention scores. Elastic Cache [40]
identifies the important KV vectors as the anchor points and merges the surrounding less important
cache with these anchors. While effective, existing works 44] 68 60] typically apply a uniform strategy that retains the same number of KV vectors for each layer to generate next token efficiently, often overlooking the layer-wise heterogeneous characteristics. Our analyses in Sec. 3.2 reveal the notably distinct importance distributions of KV vectors across layers, highlighting the need for tailored retention recipe for each layer adaptively. Intuitively, the importance distribution is concentrated in certain layers while relatively dispersed in other layers. As suboptimal solution, retaining the same cache size for each layer results in obvious information loss in dispersed layers, and redundant cache in concentrated layers, as shown in Fig. (a).
To address this, we present PrefixKV, a new KV cache compression method for efficient and ac- curate model generation. We define the prefix
By B.
Globol profi codiscoorig ggon
05 odd 0.06
(828
of KV cache as the top ones in the priority se- quences that are sorted KV vectors according to the normalized importance. The prefix KV vec- tors of each layer represents the retained cache, making the challenge of determining the optimal layer-wise wise cache size for the next token genera- tion equivalent to identifying the optimal global prefix configuration. To derive such configura- tion for a given compression ratio budget, we leverage the prefix cumulative priority as the measurement for the amount of reserved contex- tual information in each layer. Binary search is then utilized to obtain the desirable information retention ratio, enabling the layer-wise KV reten- tion aligns with the overall budget and maintains the ideal cumulative priority. This ensures that each layer can preserve maximal contextual in- formation after compression, keeping the high generation quality of models, as shown in Fig. (b). Extensive experiments show that our method achieves the state-of-th aof-the-artt art performance compared with existing works. It can greatly boost inference efficiency while well maintaining the model's strong capabilities. exhibiting promising potential for welldwrldd applications. Notably, with compression budget of 20%. it provides 1.8x x inference speedup for -LAVA-1.57BB attaining competitive performance compared with original model.
60) Predicss
UA
0.05
33 R, (Raym poot raci))
(125)
(325 [02
(31 Olus
C25 R clayee prefis miiu)
Potained
EV
Figure 1:
between previous methods and ours. Previous methods often keep the same prefix length for priority sequences of KV, i.e re- training the same cache size for each layer. This causes notable information loss for certain layers. In this example, the first layer loses 30% of infor- mation. In contrast, we derive the optimal global prefix configuration to preserve as much informa- tion as possible in each layer. In this example, both layers can retain 90% of information, thereby en- hancing performance.
Comparison
2
Related Work
2.1
Vision instruction-followin model.
The progress of large vision-language models (LVLMss) has significantly expanded the capabilities of large language models (LLMs) [1522] 53 16, (2 50 51 49 by incorporating visual information,
leading powerful generation and reasoning ability for multimodal tasks [37 35
13 71
14 9
45 30 5)
These models typical use linear projection [37] or perceivers (26) to integrate visual
representations into the input of LLMs directly. Then, they are further finetuned on high-quality instructional datasets, which include the image-text pairs and the language instructional commands. These models can thus follow multimodal instructions effectively and accurately. Inspired by the notable advancements, subsequent works have sought to enhance their abilities of region-level
grounding 77] (6 43 10], video comprehension 65 31, industrial anomaly recognition 22 62
and biomedical image understanding 29 54] etc.
Preffxx refers top elements sorted by importance instead of position.

---

2.2 KV cache compression.
While powerful, existing LVLMs typically encounter high inference costs due to the large parameter count and complex computation, hindering the practical applications. Given this, numerous efforts
distillation 21 471, and pruning 44 48 24), etc. These methods typically focus on the compression on the model level, reducing inference overhead brought by the parameters. Meanwhile, the KV cache on the data level also incurs significant memory usage and GPU communication overhead during inference. To address this, recent works have investigated various ways to compress the KV cache (60) 40
have focused on improving the inference efficiency, including quantization [3 59
23 20
For example, Streaming [60] leverages the distance to the output token as the importance indicator and preserves starting KV vectors and those adjacent to the output token. O2O [68] utilizes the attention scores importance metric for KV vectors and retains the important ones during generation. KVMerger [57] identifies sets of suitable KV states for merging and uses gaussian kernel weighted aggregation. Elastic Cache [40] divides the KV cache into several buckets according the positions of important KV cache and merges the vectors in the same bucket into one. While effective, they often overlook the distinct importance distribution across layers and retain the same KV cache size in each layer, causing notable information loss. Moreover, despite works like (67 20 28 explore layer-wise KV cache size allocation, they often depend manually designed patterns, which still achieve inferior model's generation capability.
46
55 57
3 Methodology
In Sec. 3.1 we first introduce the inference process of LVLMs and the general KV cache compression framework. Then, in Sec. 3.2 we formalize the process of KV compression as retaining the prefix of KV cache and uncover the fact of diverse importance distributions of KV vectors in layers. In Sec. 3.3 we further present PrefixKV, to deliver the ideal layer-wise cache size for the next token prediction by binary searching for the optimal global prefix configuration.
3.1 Preliminary
LVLMs exhibit exceptional capabilities for multimodal instructions. Given input which often consists of the system prompt and user instruction, they first embed the text into the token embeddings through the pre-defined vocabulary and transform the image as flattened patches through the visual encoder. Then, the model enters the prefilling stage, where all tokens interact with each other through the self-attention mechanism. Meanwhile, the key and value vectors of each token are stored as the KV cache, which keeps the contextual information for generation. Subsequently, the model transitions to the decoding stage and outputs the response in an autoregressive way. In each step, the latest predicted token serves as the input and it interacts with the cached KV vectors for preceding information by the self-attention module. It is worth noting that the KV cache size proportional to the length of processed tokens. This can consume notable memory resources and result in bottleneck in inference speed. It calls for KV cache compression methods to reduce the KV cache size effectively.
The general KV cache compression framework in existing works consist of two stages 40 68 60
Firstly, after prefilling, the importance of each KV vector is derived based on the attention scores or the distance to the generated output. Then, the most important ones are retained in the cache while the less important KV vectors are removed to reduce the memory footprint. Mcanwhile, the reserved KV cache sizes in layers meet the requirement of the compression ratio budget. Secondly, during decoding stage, with the inclusion of KV vectors in cache for newly generated tokens, the importance metric of each KV vector is updated. Less critical ones are removed to ensure that the cache size consistently aligns with the overall budget. Our method also follows this general framework, as shown in Fig. 3 Specifically, after prefilling, we retain the most important KV vectors by the ideal global prefix configuration. In the decoding, we follow [40] to prune the vectors at a fixed distance to the latest generated token and maintain the global prefix configuration to meet target budget. Additionally, our method is also complementary to prefilling acceleration ways like FastV [
For more preliminary, please refer to previous works
69
63
70
3

---

3.2 Layer-wise KV Cache Importance Distribution
Mathematical notations for KV compression. We first introduce the necessary notations to formalize the compression procedure. Specifically, we follow 44) 68 employ the attention scores as the importance metric, which indicates the amount of contextual information of each KV vector. We suppose that the model consists of I. transformer layers. For the 1-th, layer, its input tokens {t) &i,... "f.,, } interact with each other in the multi-head self-attention module, where N is the token number. For the i-th head, we denote the query, key, and value vectors of the token by as in ki and of respectively. Then, the causal attention score matrix Aj = {a (a. NNN can be derived
by aj i.m. = Σjem explai "M " where aj a.m""" indicates the attention score of token t7" with respect to token Ey in the i-th head. Then, the total attention score that each token E receives in the i-th head q. For each ki"" and v¹ we follow 44 to define their importance metric as the averaged total attention score of Ef' across all heads, i... I = Average, (E,, of " where Average, denotes the average operation for heads. It is noted that the importance metric of each KV vector varies each layer but is the same across heads. Then, for compression ratio budget r. the top R² proportion of KV vectors with the highest importance are retained in each head at the (-th layer. Besides, the adoption of (Ri, R2.... R satisfy the requirement of Σt_ R,N = rLN.
expla) ")
can be derived by Σm 4i
Distinet importance distributions across lay- ers. Existing KV cache compression meth- ods [68] 60 40] often retain the same number of KV vectors for each layer. This, however, overlooks the diverse contextual information distribution across layers and causes notable valuable information loss. To uncover this fact, we leverage the lorenz curve [19] 18] to charac- terize the importance distribution across differ- ent layers. Specifically, for the n-th KV vector of 1-th layer, we first obtain its importance ra- tio relative to all tokens by normalization, i.e.,
0.8-
Equality
Comusalive Prortity
0.6-
Layor 1 (0.24)
Layer 17 (1:4)
Layer 3 (0.39)
Lays 19 (0
0.4
Previous
(045)
Lays (0.45
Ri-2
Layar (045)
Layer 0.44
0.2
Layer 9 (0.22)
11 10
Lays 25 (.45
Layer (1.45
Layer (0.39)
Layss (0.44
Layer 15 (0.43)
Lays 31 (0.42)
0.0
0.2
0.4
0.6
0.8
1.0
Prefix Size Ratio
IT = ΣN
17
We then sort the importance
Figure 2: The lorenz curve of priority sequence for KV vectors in different layers. We observe that different layers exhibit diverse importance distri- butions in the KV cache. Previous methods (the dashed black line) that keep the same prefix cause the notable information loss in layers with dis- persed distributions. In contrast, our method (the dashed red line) maximally retains the amount of contextual information of each layer by adaptively maintaining the maximal prefix cumulative priority. The numbers in parentheses in the legend repre- sent the gini coefficient of priority sequence in each layer. A higher gini index indicates more con- centrated importance distribution. It quantitatively demonstrates the varying importance distributions of KV vectors across layers.
set II., ...,; } in the descending order to derive the priority sequence of KV vectors. In the priority sequence, the vectors ranked higher have larger importance, making the KV com- pression equivalent to retaining the prefix of KV cache. Suppose that the sorted indices are {s/. .... s }. for each prefix size ratio o of this sequence, we can obtain its cumulative pri-
ority by P = Ejson I Pt indicates the amount of contextual information kept in the 1-th layer after retaining top o proportion of the most important KV vectors. By deriving the prefix size ratio sequence { 1 I N ..., 1) and its corresponding cumulative priority sequence
{P, P ..... P!), ). we can then obtain the lorenz curve of importance distribution. As shown in Fig. 2 we observe that the cumulative priority growth trends vary significantly across layers. Previous works generally retain the same layer-wise cache size, i.e., adopting the same prefix size ratio of
KV cache with R = r, vj E [1, L). In this scenario, as shown in the dashed black line, different
layers exhibit markedly distinct cumulative priorities, i.e., PR This suggests an uneven retention of contextual information across layers, where layers showing rapid growth trends retain a substantial amount, whereas those with slower growth trends retain relatively little. This disparity leads to obvious information loss in layers with slow growth and adversely impacts generation quality. We also quantitatively show the diverse importance distribution by gini coefficient
[15 19 It is defined as the area that lies between the equality line (the dotted pink line in Fig. 2 and the lorenz curve. The
smaller it is, the more uniform the importance distribution, and vice versa. As shown in the legend in

---

Prdillia
Decoding
Atsention soom
Priority Sequenoss
Cumulative poudty sequesions
Globdl prefts
KV cache
cudfquestoon
Tornfomer Layar!
Transforaa Laper.
10.25 0225- 02
025 02 baa
Iepettacntt It
Inrssomer Lapert
Prefts KV
Trarsforue Layert
0. -. no:
0.0
R2
Aupha
T Trandomer 2
Trarsfomer Laper2
0. 04
Trandomer Lager
Tranformr Layer!
102 0.3 00,02 0.00
Binery sourch Phote Sec riio p
103
028
Ingetwint
Nemmlize & Sort Z?
glubal prefix contiuuatioon (R, R; R.....u RR.) (b)
R,
6:)
Figure 3: (a) The inference process of L.LMs,, where the orange and green rectangles denote the KV cache generated during prefilling and utilized during decoding, respectively. After prefilling, the KV cache is layer-wisely compressed according to the proportions specified by PrefixKV, i.e.,
[R] R R During decoding, as the sequence lengthens and cache increases, the KV cache consistently maintains the derived compression proportions by pruning KV at fixed distance [40]. (b) The overview of PrefixKV. It employs binary search for cumulative priority sequences of KV to derive the optimal global prefix configuration, which delivers ideal cache size ratio for each layer.
Fig. different layers exhibit varying gini coefficients, further demonstrating the heterogeneity of KV cache in layers. This calls for adaptively determining the prefix of KV cache for each layer.
3.3 PrefixKV
Based on above observations, we introduce PV With layer-wise cache size scheme formalized as global prefix configuration, it employs binary search to derive ideal solution, as shown in Fig. 3 (b).
Global prefix configuration. All layers' cache size ratios {RI,... ..., Rt i.e., the prefix size ratios of KV cache, constitute the global prefix configuration space of the model. The goal is to identify the optimal global prefix configuration maintain the high quality of model generation. Given the compression ratio budget r, we discover information retention ratio p to derive such configuration. Specifically, with the priority sequences, p represents the cumulative priority threshold. For the I-th layer, its proportion of retained KV vectors is thus the minimum prefix size ratio o such that the cumulative priority Pi is larger than or equal to p, ie., R₁ = min((o] Pi 2 p}). Meanwhile, the value of satisfies the requirement of the compression ratio budget, ie..
Σ R. = Σ min ((o 2 p)) rL. Σn:
(1)
The corresponding prefix configurations (R1, R ensure that maximal prefix cumulative priority is kept layer-wisely. Finding p is needed for global prefix configuration.
Binary search for optimal configuration. Due to the numerous possible values of p, obtaining p is quite challenging. We propose employing binary search to efficiently derive p for adaptive layer-wist KV reten- tion. Formally, we start with the interval of [P1. P2] where P1 = 0 and p2 = 1. We try p = tr = 0.5 and calculate its corresponding compression budget dif- ference 5 = E, R² LL. If 5 is equal to zero, we have found the value for p. If it is smaller than zero, we set Pi = p; otherwise, we set P2 = p. Then, we update =PLt2 PLtzz and repeat the process until mecting the constraint. Algo. I preseents the process. In this way, shown in the dashed red line in Fig. 2 different layers can consistently keep the maximal amount of the contextual information. It ensures that the valuable KV vectors are cached and accessible during generation across all layers. The quality of the model's outputs can be maintained even after significant compression of the KV cache. In practice, p that satisfies 8 = 0 may not exist. We can thus set a small threshold for 8 to terminate the search and scale the resulting global prefix configuration to meet the target budget.
Algorithm 1: Binary Search for ratio p
Initialize P1 < 0. + 1;
while pl < P2 do
p eutp2
Σ R E,
Pf p\);
d= b -LL
i = 0 then return p;
else if 8 < 0 then Pi + p;
else P2 + p
p =
2
return p:
Moreover, we observe that the cumulative priority sequences of layers are similar and robust across different samples. Therefore, given compression ratio budget, we can leverage random samples to
5

---

Table 1: Comparison with SOTA methods on LLaVA-Deseription with the PPL/ ROUGE metrics under various compression ratio budgets. Note that lower PPL is better, while a higher ROUGE is better. The results without the KV compression are 3.20/ / 0.62 for LLaVV--.5--BB and 2.73 / 0.63 for LLL.VA-1.5-13B respectively. It shows that our method consistently achieves superior performance.
Model|
Method
10%
20%
30%
40%
50%
60%
70%
80%
90%
Local
66.0/0.22
105/
70.0/0.18
33.8/0.11 47.5/0.17
1..7/0.30
0.40
0..5
4.03
H₂O
54.5/0.28
48.3/0.31
32.0/.33
18.3/0.32
02.9/0.34
0,40
4.28/0.51
3.72/0.57 4.16/0.53
7B
Pyramid
14.3/0.31
12,4/0.31
7.16/0.31
5.75
3.50/0.51
3.47 /0.5
3.41
/0.59
3.20/0.74 3.20/0.73
Elastic
18.0/0.29
14.0/0.29
11.8/0.29
7.38
6,31/
5.97
3.66
3,55
3.58/ 0.57
Ours
4.41
3.69/0.5
3.48 0.55
/0.57
3.41
/.59
3.25/
/.63
3.20/
3.20/0.76
Local 60.0/0.15
139 /0.12
56.3/0.11
16.1
13.2/0,31
7.06
3.72
/0.8
3.72/
3.25 /0.55
H₂O 12.4/0.39
10.4/0.9
8
0.46
3.78/0.49
3.58 0.49
3.16
/0.55
3.28/
/.57
3.06/01 0.59
13B
Elastic
5.75 14.9/0.30
/0.35
4.41
00.40
3,55
0..00
3.36 / 0.52
3.28
01,53
2.97 /0.58
/0.60
3.02 0..59
Ours
2.72/0.45 3.17 / 0.53
2.97 / 0.59
2.92/ 0.60
2.89/ 0.60
2.84/ 0.61 2.77 0.69
2.73/ 0.74
2.73 0.79
Table 2: Comparison with SOTA methods on MM- Met with the PPL / ROUGE metrics under various compression ratio budgets. The results without the KV compression are 5.28 / 0.58 for LLa LL and 4.72 / 0.58 for 1..55.31111 respectively.
Model
Method
10%
20%
30%
40%
50%
60%
70%
80%
90%
Local
109 /0.11
90.0 0.00
99.0/0.13
99.0/0.16
66.0/0.16
/0,7
12.4/0.34
7.88 /0.41
6.28 / 0,46
H₂O
1.8/0.25
120/0.26
35.3 72.5/0.29
18.6/0.30
0..3
7.09 /0.44
6.22/0.46
5.72 0.49
7B
Pyramid
20.8
10.4/0.28
7.50/0.30
5.75 /0.34
5.43
5.50/0.46
5.41
5.28
5.28/0.73
Elastic
40.5/
21.0/0.25
14.9/0.29
11.3/0.29
9.06
/ 0.32
7.63
5.97
/0.46
5.53 5.56
0.54
Ours
0..39
/0.41
0.46
5.53
0.46
5.50/ 0.48
5.44/ 0,50
5,38 0.59
5,28 0.74
5.28/ / 0.77
Local
135/
120/0.14
77.0/0,44
0.28
40,5
18.0/0334
9.06
/0.42
6.63
/0.39
5.41 /0.43
H₂O
3.36/0.36
3..6/0.38
20.8/0.40
10.6/0.43
7.75 /,39
6,28/0.44
0.44
5.63/0.46
/046
5,25
/.47
/..56
13B
Elastic
34.3/0.28
11.6/0.34
8.00
6,31
/0.44
5.81 0.44
/
4.97
/0.52
4.81
/0.5
/0.56
Ours
6.28 0.48
5,16
4.88 0.52
4.78 / 0.52
4.72 0.55
4.72/ 0.57
4.72 0.64
4.69/ 0.75
0.79
derive the corresponding value and the optimal global prefix configuration [R1, R2. R2} for each layer offline, as analyzed in Fig. 4 Tab. 0 and Tab. [ The configuration can thus be adopted for the model during inference, which avoids the online binary search and shows good generalizability.
4 Experiments
4.1 Experimental Settings
We follow 440 to employ LLaVA-1.5-7B (35] and LLaVA-L.5-13B [35), and leverage the LLa Description [40] and MM-Vet [64] cerruction-follwwiing datasets for evaluation. LLaVA-Desrription a curated subset of 1000 detailed description instructions from the LLa VA-1.5 training set [40] MM- Vet encompasses diverse set of tasks designed to comprehensively evaluate the model performance in both understanding and generation. Besides, we also re-conduct the instruction tuning for LLaVA- 1.5 models, to exclude the LLa LLaVA-Deseription for preventing data leakage during evaluation. We follow [40 to employ the perplexity (PPL) and the ROUGE score [2] metrics. Specifically, PPL quantifies the exponential value of the cross-entropy loss between the predicted next token and ground truth. lower PPL indicates the better generation quality. The ROUGE score calculates the longest common subsequence between the generated output and the reference outputs, with FI score used for evaluation. A higher ROUGE score indicates the better consistency with reference responses.
Following [40] we utilize the model without cache compression to generate the reference output for the ROUGE score evaluation. Besides, in practice, multiple generation texts of model can be different when the temperature is not zero. Thus, for the compression budget of 100%, i.e., without compression, we measure the ROUGE score of two generation outputs and it is thus smaller than 1. For other budgets, we use the temperature of 0 for the reproducibility and the ROUGE score may thus exceed the uncompressed one. We simply leverage 10 random samples from the training set of model for the global prefix configuration estimation offline. During decoding, we follow 44) to fix the distance to the latest token for pruning and maintain the layer-wise cache size ratios to satisfy the target budget. We compare with the state-of-the ofth-offthe-art Elastic Cache [40], Heavy- -Hiteer Oracle (68).

---

Table Inference time for our method under compression ratio budgets of 20% 40% 60% / 80%.
OOM indicates out of memory.
Batch
Model
Token
Latency (8)
Throughput (token/s)
Size
Size
Length
PrefixKV
Full Cache
PrefixKV
Full Cache
13B
1024+512
20.0/24.3 / / 29.7
30.5
204.6/168.1 / 148.7 137.6
134.1
16
13B
6244256
15.94////119//1
17.8
349.5 / 256.3 236.5
230.2
16
7B
1024+512
16.8 22.5 26.6/
26.6/
30.7
486.7 / 363.3 / 307.9 276.9
266.6 OOM
48
7B
624+256
13.1 O0// /OM/ OOM
0OM
934,4/00M/ OOM OOM
StreamingLLM [60], and PyramidKV [67]. which are termed as Elastic, HzO, Local, and Pyramid
for brevity, respectively.
4.2 Main Results
As shown in Tab. and Tab. 2 our method consistently achieves the state-of-the-art performance compared with others across various compression budgets and different model scales. For example, as shown in Tab. I under compression budget of 50% LLaVA-Description significantly outperforms HzO and Elastic cache by 9.49 and 2.90 in PPL for LLaVA-1.5-7B. also surpasses the Local cache by notable margin of 0.39 in ROUGE score. We also observe that our advantages over others can be further amplified as the compression budget decreases. Under a compression budget of 20%, our PrefixKV can still maintain a satisfactory PPL of 3.69, which is reduced by 8.71, 44.6, and 10.3 compared with Pyramid, H2O, and Elastic, respectively. For large model of LLLVA--15--333 example, as shown Tab. 2 our PrefixKV outperforms Elastic and H₂O cache by 1.53 and 5.82 PPL, respectively, under a compression budget of 40% MM-Vet. Furthermore, our method shows the minimal performance decline compared with the original model without KV compression in various scenarios. For example, shown in Tab. for LLa nearly identical PPL and ROUGE scores compared with the uncompressed model at any compression budget exceeding 30%. Under compression budget of 20%, our PrefixKV only leads the decline of 0.44 PPL. These results well demonstrate the superiority of ours.
our PrefixKV
method also demonstrates the notable superiority over others. For
method can maintain the
4.3 Model Analyses
We present comprehensive analyses for our method. Following [40], experiments are conducted on MM- Vet based on LLaVA-1.5-7B with PPL for evaluation, by default.
Inference efficiency. We evaluate the inference efficiency of the model with our KV compression method to verify its benefit for acceleration. We follow [40 to conduct the evaluation in two practical scenarios. Specifically, firstly, we construct the input with 1024 prompt tokens and generate 512 tokens. Secondly. we employ a shorter input with 624 prompt tokens and generate 256 tokens, which means the minimal prompt length with only the image tokens and system prompts. We measure the inference time on the NVIDIA A A10 GPÜ and use the batch size which maximizes the available memory to simulate the realistic deployment scenarios for better efficiency and throughput. As shown in Tab. 3 our method shows the notable inference speedups under various compression budgets compared with original model with full KV cache. For example, under the compression budget of 20% with the batch size of 16 for LLaVA-1.5-7BB our method demonstrates 1.8x inference speedup in terms of throughput. Besides, it also reduces the memory usage, avoiding OOM and enabling efficient inference with large batch size of 48. We also note that in this scenario, our method maintains the superior performance with only 0.69 PPL decline compared with the original model. These results well demonstrate the benefit of our method in practical deployment for efficient LVLMs.
Global prefix configuration matters. We verify the effectiveness of our method in identifying the ideal global prefix configuration. We first introduce the baseline, which keeps the same retained KV cache size for all layers. As shown Tab. 4 our PrefixKV consistently brings performance benefit under various compression budgets. For example, under the compression budget of 30%, it surpasses the baseline by significant margin of 14.7 PPI.. Besides, as the compression budget gradually decreases from 90% to 10%, its superiority becomes increasingly evident, showing increasing
7

---

Table 4: Prefix config.
(Uncompressed: 5.28).
Table 5: Sample numbers. (Uncompressed:: 5.28).
Method
|10% 20% 30% 40% 50% 60% 70% 80% 90%
Number] 20% 30% 40% 50% 60% 70% 80% 90%
Baseline 141.8 26.6 20.4 15.4 11.8 9.06 6.47 5.75 5.72
1
7.63 6.03 5.72 5,53 5.53
5.44
5.38
5.28 5.28
Pyramid 20.8 10.4 7.50 5.75 5.63 5,50 5.41 5.28 5,28
5
7.50 6.03 5.72 5.53 5,50
5,41 5.38 5.28 5.28
PrefixKV
7.38 7.33 5.97 5.72 5.53 5.50
5.44 5,38 5.28 5.28
10
7.38 5.97 5.72 5.53 5.50
5,44
5.38
5.28 5.28
20
7.38 5.97 5.72 5.53 5.50 5,41 5.38 5.28 5.28
performance improvements. We also compare ours with PyramidKV 66 which manually allocates larger cache size in shallow layers and smaller size in deep layers. Since it compresses the cache only after the prefilling, we integrate it in the baseline for fair comparisons. As shown in Tab. 4 our strategy achieves superior performance over PyramidKV in various scenarios, especially under the low compression budget. These results show that compared with the same cache size across layers and the manually designed scheme by PyramidK V. our method can identify better global prefix configuration, which retains the overall contextual information more effectively.
Effect of offfine estimation. Given a compres-
0.9
sion budget, we estimate the optimal KV cache size for each layer by random samples offlin,, which avoids the overhead of online searching. To verify its effectiiveness, we first examine the
0.45
Retained KV cache slae ratio
0.8
0.7-
-0.40
cocficimnt
variation of retained KV cache size ratios at each layer across different samples. As shown
0.5
-0.35
in Fig. 4 different samples exhibit the similar cache size ratios for each layer. This indicates the potential of using the samples to estimate the optimal cache size ratios offline. We further present the comparison results between offline estimation and online binary searching for each sample in Tab. 6 It can be observed that of- fline estimation can obtain the comparable per- formance with the online searching. Therefore, our method can be integrated into the models efficiently. without the extra inference overhead. We also inspect the impact of adopting differ- ent numbers of random samples. As shown in Tab. 5 the performance is stable and robust across different sample sizes. Besides, employing a single sample can achieve good performance, which shows the effectiveness of offline estimation.
0.4
0.30
0.3
0.2
0.25
10
20
Layer
Figure 4: The retained KV cache size ratios for each layer of 100 random samples under the com- pression ratio of 50% and their gini coefficients of the priority sequences for KV vectors in each layer. shows that different samples exhibit similar and robust characteristics, showing the reasonableness of offline estimation.
Robustness of offfine estimation. To inspect the robustness of offline estimation, measure the standard deviation (std) the retained KV cache size ratios obtained via online binary search across all samples, well as the mean absolute difference (mad) between these values and the configuration of offline estimation. We present the results in Tab. 7 quantitatively shows that the variance of retained KV cache size ratios across samples is small, indicating the feasibility of offline estimation. The small mean absolute difference supports the efficacy of offline estimation. We also conduct experiments using random samples from different domains and with different instruction complexities to further verify the robustness. Specifically, for different domains, we leverage random samples from open-knowledge visual question answering (VQA), optical character recognition (OCR), and complex reasoning (reasoning), respectively. In Tab. g we present the PPI. results. We observe that the performance remains stable across different domains. Besides, for instruction complexities, we obtain random samples with different instruction lengths: shorter than 100, between 100 and 500, and longer than 500. We present the results in Tab. 9 It shows that our method also exhibits robustness to the instruction complexity, showing favorable generalizability. For the binary search for offline estimation, we also present analyses for its sensitivity and convergence behavior, Specifically, we conduct experiments under the compression ratio of 50% using varying approximation threshold 8 to inspect the robustness of binary search. We present the average search steps and PPL results in Tab. TO, It shows that the search consistently converges within 7-12 steps, with stable performance across different settings. We also fix the number of search steps of binary search for evaluation. As shown in Tab. II the performance remains stable across difference scenarios, showing the reliability.
8

---

Table 6: Offline estimation. (rarriionnnnnooopeeeeeeee 5.28).
Table 7: Deviation analyses.
Method 10% 20% 30% 40% 50% 60% 70% 80% 90%
10% 20%
30%
40%
50%
60%
70%
80%
90%
Offline 7.38 5.97 5.72 5.53
5.50
5.44 5.38
5.28 5.28
0.010
0.021
0.023
0.024
0.023
0.020
0.013
0.011 0.006
mad 0.010 0.021
0.024
0.024
0.023
0.020
0.014
0.011 0.006
Online 7.38 5.97 5.66 5.53 5.50 5.41 5.38 5.28 5.28
Table 8: Robustness to domain.
Table 9: Robustness to instruction complexity.
Domain
10% 20% 30% 40% 50% 60% 70% 80% 90%
Length
10% 20% 30% 40% 50% 60% 70% 80% 90%
VQA OCR
7.45 5.97 5.72 5.53 5.50 5.41 5.38 5.28 5.28
(0, 100]
7.38 6.01 5.66 5.53 5.53 5.44 5,38 5.28 5.28
7.38 6.01 5.72 5,53
5.53 5,41 5,38 5.28 5.28
(100, 500] 7.42 5.97 5.72 5,53 5,50 5,44 5,38 5,28 5,28
reasoning 7.40 5.97 5.72 5.53 5.50 5.44 5.38 5.28 5.28
(500, inf) 7.38 5.97 5.72 5.54 5,50 5.41 5.38 5.28 5.28
Table 10: Robustness to 8
Table 11: Robust. to step
Table 12: Merging. (Uncompressed: 5.28).
1000125
0.025
0.05 0.075 0.1
steps] 7
11
Method 10% 20% 30% 40% 50% 60% 70% 80% 90%
avg. step
PPL
5,50
11.5
5.50
10.8
5.50
9.3
5.51
855 7.8
5,53
PPT
5555 5.505.55555
PrefixKV 7.38 5.97 5.72 5.53
5.50
5.44
5.38 5.28 5.28
Position
7.63 6.06 5.75 5,53 5.44
5.31
5.28 5.28 5.28 5.28 5.28
5.31
Feature 7.38 5.97 5.72 5.53 5.44 5.31
Relation between KV cache size and importance distribution. We derive better KV cache size for each layer based on the importance distribution. To provide deeper insights for their relation, we visualize the KV cache size ratios and the gini coefficients of priority sequence for different samples in Fig. 4 We observe that layers with higher gini coefficients, ie., with more concentrated importance distributions typically have fewer KV cache sizes, and vice versa. This qualitatively validates the reliability of our method, as layers with concentrated importance distributions can retain most information with fewer KV vectors, while layers with more dispersed importance distributions note that the retained KV cache size exhibits W-shaped
require larger KV cache sizes. Besides, trend across layers. This inspires that utilizing more cost-effective attention mechanism in shallow to mid layers, as well as in mid to deep layers, may lead to more efficient model architecture. Besides, we also observe that the distributions of retained KV cache size ratios and Gini coefficients obtained from different samples exhibit low variance, indicating favorable consistency in importance patterns. The empirical stability further verifies the reliability of offline estimation using few samples, as they can approximate the overall layer-wise importance distribution.
Eviction or merging. Our PrefixKV shows favorable performance by cache eviction, i.e., retaining the prefix vectors and pruning the unimportant ones. To mitigate the information loss of eviction, previous research explores merging the less important KV vector with its nearest important one in position (40). Therefore, we inspect the performance variation under combining our PrefixKV with cache merging. In addition to merging based on positional distance, we also examine the way based on the feature similarity, which are explored in the token merging field 6 58). Specifically, we denote the I-th layer's retained KV vector set as Of and pruned set as Dj. respectively. The cache merging operation is the same for key and value vectors in different heads, we thus proceed with the key vector for illustration and omit the head superscript. For each pruned key vector ky" where m E Dt, we obtain its matching metric ent to each retained one k." where n E 27. For each k,, we obtain the argmax,, (c"""), The vectors are
set of pruned vectors that match with it by T = {m E 32 then merged to update k," by k." = Average (kj" Iu E T U {m))). .. We experiment with matching metrics based on the positional distance and feature cosine similarity by ctTh -Im n| and
"TA" [|]]]] which are denoted as "Position" and "Feature", respectively, As shown in Tab. 12
=
A," A2"
integrating "Position" can lead to inferior performance over PrefixKV in certain scenarios. We note that our method enables the maximal preservation of important KV vectors, and merging based on the positional distance, however, could introduce the interference to important cache due to the feature discrepancies among the vectors (58]. Besides, we observe that with less feature dispersion, combining "Feature" can bring the marginal improvements. This indicates that our method can well retain the significant contextual information across layers and eliminate the need for cache merging to reduce information loss, demonstrating its efficacy.
Analyses for the feature disturbance. To further verify the effectiveness of our method in pre- serving valuable contextual information across layers, inspect the feature perturbation caused

---

Eady
2.4
Lotal
231
II30
InfaKY
Method
10%
20%
30%
40%
50%
60%
70%
80%
90%
1.3
Local
314
185
93.0
54.5
70.0
43.8
24,6
17.5
11.4
H₂O
72.5
58.0
43.3
29.3
19.1
15,4
12.9
11.1
9.50
Elastic
404
66.0
33.8
20.4
12.8
10.4
8.63
8.63
8.63
Ours
16.6
9.94
8,50
8.13
7.88
7.09
6.41
6.28
6.28
Layer Figure 5: Mean absolute error vis.
Table 13: Results on Qwen- VL. (Uncompressed: 6.28).
by KV cache compression for output tokens each layer. Specifically, for each output token, we calculate the mean absolute error between its feature with and without compression. We employ the compression budget of 50% and visual the average error across output tokens for 100 random samples across layers. As shown in Fig. [ our method consistently exhibits lower mean absolute error between features compared with others. This shows that our method can introduce less interference to the features of output tokens, demonstrating enhanced retention of contextual information and resulting in improved generation quality.
Generalizability on other LVLMs. Following 40 we conduct experiments on Qwen- VL [2] to verify the general effectiveness of our method. In Tab. [3 our method consistently exhibits superiority over others across various compression budgets, highlighting generalizability.
Comparison results on LLMs. Our method can also be adopted for the KV cache compression in LLMs. Following [67]. we conduct experiments on LongBench
Table 14: Results on LLMs.
Table 15: With quant.
Dataset [Base. Locul H₂O Pyramid Ours
Method|12.54 25% 50% 100%
LongB. Ncedle, 100 30.4 49.3
41.6 32.0
35.0
37.1
37.5
KIVI
41.2 41.6 41.6
and (38 17 based
97.3
97.6
Ours
40.2 41.1 41.5 41.6
Needle-in-a-HayStee We set the KV cache size budget 128 and present the results in Tab. [4 We on LLAMA-3-8B [6..
can see that our PrefixKV also outperforms others in such scenarios. It well reserves the valuable information for model's generation in limited KV cache budget but under long context. The results further exhibit the general efficacy of our method. Besides, cache quantization methods for LLMs. We present the comparison results with advanced KIVI [39 on LongBench for LLaMA-3-8B in Tab. 15 We find that in long context scenarios, for different budgets, KV cache favors different compression dimensions, i.e., precision in quantization and length in pruning. For extreme 12.5% compression budget, the length dimension shows more redundancy and our method performs better. For larger budgets, removing precision redundancy preserves more information and KIVI obtains slightly better performance. Moreover, our pruning method is complementary to quantization method and can be combined to eliminate redundancy in both dimensions. For example, combining KIVI under 4 bit with ours can achieve competitive performance of 40.0 with 3.12%% compression ratio budget, leading to stronger efficacy.
also compare ours with the KV
5
Conclusion
In this paper, we present PrefixKV to effectively compress KV cache for efficient generation of large vision-language models (LVLMs). It derives the optimal KV cache size for each layer by searching for the ideal global prefix configuration with the priority sequences of KVV Maximal preservation of contextual information is thus ensured layer-wisely, contributing to high-quality model generation. Extensive experiments show that method achieves the state-of-the-ar performance compared with others. It provides notable inference speedups while maintaining the generation capability, showing the superiority for practical applications.
Acknowledgments
This work was supported by National Natural Science Foundation China (Nos. 62525103, 624B2082, 62271281. 62441235) and Beijing Natural Science Foundation (L244726). It was also supported by Tsinghua University Meituan Joint Institute for Digital Life. We are also grateful to Jiaxin Li for his thoughtful contribution.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv22333.08774 2023.
[2]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
[3]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16600,999,666, 2023.
[4] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, al. Longhench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. /4508, 2023.
[5] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
[6] Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer. and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:22 10.09461, 2022.
[7] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. arXiv preprint arXiv:2312.00784. 2023.
[8] Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen. Towards generic anomaly detection and understanding: Large-scale visual-linguisste model (gpt-4v) takes the lead. arXiv preprint arXiv:2311.02788 2023.
[9] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023.
[10] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXwv:230.115795. 2023.
[11] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. arXiv preprint 2673644403.0 2024.
[12] Can Cui, Yunsheng Ma, Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang. Kuci-Da Liao, et al. A survey multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958-979, 2024.
[13) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vasion-language models with instruction tuning, 2023.
[14] Wenhao Dong, Haodong Zhu. Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Guodong Guo, and Baochang Zhang. Fusion-mamba for cross-modality object detection. IEEE Transactions on Multimedia, 2025.
[15] Robert Dorfman. A formula for the gini coefficient. The review of economics and statistics, pages 146-149, 1979.
[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint ar 277722240,.2178,, 2024.

---

[17] Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402. 10171, 2024.
[18] Joseph L Gastwirth. A general definition of the lorenz curve. Econometrica: Journal of the Econometric Society, pages 1037-1039, 1971.
[19] Joseph L Gastwirth. The estimation of the lorenz curve and gini index. The review of economics and statisticy, pages 306-316, 1972.
[20] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01800 2023.
[21] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
[22] Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anoma- lygpt: Detecting industrial anomalies using large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1932-1940, 2024.
[23] Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression. arXiv preprint arXiv:2410.066 240.088844 2024.
[24] Wenxuan Huang. Zijie Zhai, Yunhang Shen, Shaoshen Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876, 2024.
[25] IDEFICS.
Introducing idefics: An open reproduction of state-of-the-art visual language model. idefics 2023.
[26] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learming, pages 4651-4664. PMLR, 2021.
[27] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
[28] Wonbeom Lee, Jungi Lee, Junghwan Sco, and Jaewoong Sim. [InfiniGen]: Efficient generative inference of large language models with dynamic (KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155-172, 2024.
(29) Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian I Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-visiee assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
[30] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024.
[31] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llav: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1013. 2023.
[32] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[33] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
12

---

[34] Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang. Chenyu You, Chenyang Wang, Lei Lu, Zhangdaihong Liu, Yefeng Zheng, Xu Sun, et al. medical multimodal large language model for future pandemics. NPJ Digital Medicine, 6(1):226, 2023.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 26296-26306, 2024.
[36] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[38] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
[39] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.
[40] Zuyan Liu, Benlin Liu, Jiahui Wang, Yuhao Dong, Guangyi Chen, Yongming Rao, Ranjay Krishna, and Jiwen Lu. Efficient inference of vision instruction-following models with elastic cache. arXiv preprint arXiv:2407.18121, 2024.
[41] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 00221702-222222 2023.
[42] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[43] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[44] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 3600-624, 2023.
[45] Junbo Qiao, Wei Li, Haizhen Xie, Hanting Chen, Jie Hu, Shaohui Lin, and Jungong Han. Lipt: Latency-aware image processing transformer. IEEE Transactions on Image Processing, 2025.
[46] Siyu Ren and Kenny Q Zhu. On the efficacy of eviction policy for key-value constrained generative language model inference. arXiv preprint arXiv2240..06262, 2024.
[47] Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, et al. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprint arXiv:2408.1588) 2024.
[48] Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning and distillation in practice: The minitron approach. arXiv preprint 11779 2024.
[49] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Bapistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
[50] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
13

---

[51] Gemma Team, Morgane Riviere. Shreya Pathak. Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408. 001 18, 2024.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manee-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977, 2023.
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 00928, 2023.
(54) Minh-Hao Van, Prateek Verma, and Xintao Wu. On large visual language models for medical imaging analysis: An empirical study. In 2024 IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), pages 172-176. IEEE, 2024.
[55] Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang. and Li Yuan. L: Look-once optimization in kv cache for efficient multimodal long-context inference. arXiv preprint arXiv:2406. 18139, 2024.
[56] Wenhai Wang, Jiangwei Xie, Chuan Yang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09225,, 2023.
[57] Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for Ilms on long-context tasks. arXiv preprint arXiv:2407.08454 2024.
[58] Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2092-2101, 2023.
[59] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[60] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprini 2r09:: 0020099.74453 /7453, 2023.
[61] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint ar09v 10305, 2023.
[62] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint 1724422309.77421, 91):1, 2023.
[63] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.123. 13549, 2023.
[64] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint 2xxv:2308.002900 2023.
[65] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-ttueed audio-visual language model for video understanding. arXiv preprint Nttiv::23360028885. 2023.
[66] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest arXiv preprint arXiv:2307.03601 2023,
14

---

[67] Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, arXiv preprint urXiv:2406.02069 2024.
[68] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
[69] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[70] Zixuan Zhou, Xuefei Ning. Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, al. survey on efficient inference for large language models, arXiv preprint arXiv:2404. 14294, 2024.
[71] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 200-:22444.002, 2023.
15

---

Table 16: Multi-round and long text VQA scenar- ios.
Table 17: High-resolution scenarios.
Dataset
Baseline
Local
H₂O
Elastic.
Ours
Dataset
Baseline
Local
H2O
Elastic.
Ours
MM-
68068
7817
4514
6.4
Multi-round
3.2
28.4
7.3
5.8
3,5
Long text
2.6
14.3
6.1
3.3
2.7
A
Implementation Details
We conduct experiments using Pytorch 42 on NVIDIA 3090 GPUs. Following [40], we leverage
two different metrics, Le., PPL and ROUGE score 32 to comprehensively assess the performance of model with KV cache compression.
Specifically, PPL measures the similarity between the probability distribution predicted by the model and that of the ground truth. Given the prefix prompt æ, suppose that the sequence of ground-trult subsequent words is t t2...ty distribution of vocabulary and ground-truth word by
then obtain the cross-entropy loss for each predicted probability
1 lpg(p(t, x. tit)...t-1)),.
(2)
where & denotes the cross-entropy loss for the i-th predicted word and p(t: æ, f1t2..t;-11 indicates the predicted probability of word t, under the input sequence of æ and ti t1t2...i-1- Then, PPL can be derived by the exponential of the average loss across words, i.e.,
Eile N
PPL exp(
(3)
The lower score of PPL represents the closer predicted probability distribution to the ground truth, suggesting stronger generative capabilities of the model.
Moreover, the ROUGE score quantifies the similarity between the model's generated content and the reference sentences. We utilize the widely adopted ROUGE-L F1 score, which identifies the longest common subsequence (LCS) between them. To calculate it, we first obtain the ROUGE-L precision score, which represents the proportion of the LCS found in the model's generation. Besides, the ROUGE-L recall score is assessed by evaluating the proportion the LCS present the reference content. The Fl score then be derived by
2 (precision recall)
F1
precision + recall
The higher score of ROUGE F1 represents that the model's output is more similar the reference content, indicating the better retention of the model's abilities. Besides, when the KV cache com- pression ratio budget is 100%, set the temperature to 0.2 measure the ROUGE score of two generation results, following 44
B More Model Analyses
We present experiments and analyses for our method in different settings. Specifically, following (24), we eval- uate our method in multi-round and long text VQA sce- narios. Since the datasets in 24 are not released, we
09.
Behited KW codhe nibe att
0.5
construct them following the implementation details. We
experiment with LLaVA- 1A-1.511.. 7B (35) under 50% compres-
0.5
sion budget and report the PPL (lower is better) in Tab, [6] where "Baseline" indicates the uncompressed result. It can be observed that our method also significantly out- performs others on such benchmarks, further showing its superiority. We also employ LLaVA-NeXT-7B [36] for assessing our method in scenarios with high-resolution images. We conduct experiments on MM- Vet 64 under
0.3
10
35
21
25
30
Layer
Figure 6: The retained KV cache size ratios of 10 random samples under 50% compression ratio for Qwen- VL.
the compression ratio budget of 50%. Tab. 17 presents the
16

---

Table 18: Our method can effectively preserve both the model's s overall and fine-grained perceptual abilities. The model adequately captures the overall background, like snowy landscape and clear blue sky, as well as the fine-grained details, like least ten people skiing. In contrast, with other methods, the model may struggle to provide comprehensive perception.
11
User
What's happening in the scene?
Local
The snowyards,
H₂O
The image of group of people are skiers are skiing down a snowy mountain slope, the image of a group of people are skiing down snowy mountain scene of people are skiing down a snowy mountain...
Elastic
The image shows group of people are skiers are enjoying snowy mountain skiing in the snowy mountain scene with a group of people skiers are skiing down a snowy mountain.
Ours
The image captures group of people skiing on snowy moun- tain slope. There are at least ten people visible in the scene, scattered across the slope, enjoying the winter sport. Some of the skiers are closer to the foreground, while others are further back, creating a sense of depth in the image. The snowy land- scape and the clear blue sky make for beautiful backdrop for the winter sports enthusiasts.
PPL results (lower is better). It shows that method also obtains notable improvement compared with others. These results well verify the efficacy and practicality our method in various scenarios. Besides, also inspect the retained cache size distribution of Qwen-VI for different samples in Fig. 6 It presents the similar W-shaped trend across layers as Fig. [4] We hypothesize this similar behavior stemming from the hierarchical processing of LVLMs: (1) Shallow layers captures fine- grained syntactic and visual features, with dispersed foundational token-level information across KV cache vectors and favor larger cache size. (2) Deep layers influence output reasoning and generation. The contextual information helps preserve the fidelity of high-level representations essential for final prediction, relying on larger retention cache. (3) Middle layers serve as intermediate abstraction stage, with feature representations aggregated and compressed. The importance distribution of KV cache vectors thus more concentrated, requiring fewer cache vectors.
C More Chat Generation Results
We present comparative analyses of PrefixKV with different KV cache compression methods [60
Tab. 21 and Tab. 22 We set the KV cache budget to 20% for inspection based VL. 35 can be observed that in various challenging scenarios. our method can consistently provide comprehensive and helpful responses by retaining the overall contextual information across layers effectively. For example, with our method, the model can reasonably infer that the man carrying surfboard possibly prepares to go surfing in the water, in Tab. 19 can also maintain the model's 's capability recognizing the six different cakes in Tab. [20 and perceiving the somewhat messy overall atmosphere in Tab. 22 Besides, it can well help the model observe various elements during generation, including the clear blue sky in Tab. [8 and palm trees in Tab. 19 etc. In contrast, with other methods, the model struggles to generate coherent outputs, with missing elements, repetitive texts, and abrupt sentences. The results highlight the superiority of method over others, showing its effectiveness and robustness for practical applications.
68 [40] in Tab. 18 Tab. 19 Tab. 20
17

---

Table 19: Our method can retain the valuable contextual information for generation effectively. For example, with our method, the model can make reasonable inference regarding the context of possibly preparing to go surfing for the man with the surfboard.
User
Can you elaborate the elements of the picture provided?
Local
The man, the beach, the beach, the beach, the beach, the beach, the beach,...
HgO
The image features man is walking along the beach scene shows man walking along the beach is man walking along the beach...
Elastic
The image shows man is a man standing on a man standing on beach, the beach, the beach, the image features man surfboard.
Ours
The image features a man walking along a beach, carrying a surfboard under his arm. He appears to be heading towards the water, possibly preparing to go surfing. The beach is lined with palm trees, adding to the tropical atmosphere. There are several other people scattered around the area, some closer to the water and others further back the beach.
Table 20: Our method can help maintain the focus key elements in the image, like wooden tray and cakes. The model thus capable of accurately describing the characteristics of salient elements, such six cakes and different shapes. In contrast, with other methods, the model may overlook important elements in the responses.
User
What this photo about?
Local
The table, the table, the table, the table, the table, the table...
H₂O
The image features a wooden cake stand is a wooden cake display wooden cake display of cake is wooden cake is wooden cake...
Elastic
The image of wooden table is wooden table is wooden table is a wooden table...
Ours
The image features wooden tray with a variety of small cakes and pastries on There are total of six cakes on the tray, each with different shapes and sizes. The tray is placed on wooden surface, and the arrangement of the cakes creates appealing presentation.
18

---

Table 21: With our method, the model can identify the main features of the image with the low KV
cache budget. It can accurately understand the scene and the relationships among salient objects.
User
Can you describe the main features of this image for me?
Local
The Airplane plane plane plane plane plane plane plane?
H₂O
The scene of the image shows large airport scene of a group of people are group of people are gathered around the airport scene of a group of people...
Elastic
The scene of the image of large airport scene of group of people are gathered around a crowd of people are gathered around group of people a group of people are standing in front of large airplane, which is a large airplane. The airplane. The airplane parked the tarmac. The airplane is parked on the runway. The airplane is parked on the tarmac. The airplane is parked on the tarmac...
Ours
The image features large United States Air Force airplane parked on runway, with group of people gathered around it. There are at least 13 people visible in the scene, with some standing closer to the airplane and others further away. The individuals appear to be observing waiting for boarding. The airplane is positioned on the runway, with the main body of the aircraft visible in the foreground and the tail end extending towards the background.
aircraft or possibly
Table 22: With our method, the model can provide accurate and detailed descriptions. Compared with others, our method can help the model discover various things and perceive the somewhat messy atmosphere after KV cache compression.
User
What the key elements in this picture?
Local
The books, books on bookshelves of books on bookshelf books on bookshelf books on bookshelf books...
H₂O
The image of bedroom scene of bedroom with bedroom with a bedroom scene of a bedroom with a bedroom...
Elastic
The image of bed is a bed is bed is bed is a bed is bed is bed bed a bed...
Ours
The image features bed with a large pile of books on it. The books are of various sizes and are spread across the bed, with some lying horizontally and others vertically. The bed is situated in appears to be somewhat messy.
with window, and the overall atmosphere
19

---

D Limitation and Societal Impact
Limitation. Due the limited academic resources, we do not evaluate our method on models with larger scales. In addition, the combination of our method with other acceleration ways like distillation and pruning is also worth exploring. which leave as the future work.
Societal Impact. Our proposed method can benefit the widely adoption of large vision-language models in various scenarios, like autonomous driving and medical image analyses, etc. We also aware of the potential risks associated with improper use and committed to preventing this.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately present the contribution and scope of our work in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We present the detailed information for reproducing the results in the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: Our code and data is available at https://githhb. com/ THU- MIG/ PrefixKV We provide sufficient information in the paper and appendix to faithfully reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 /mips.cccpublic//uu.ide http://gggiieee/eeeeububbimi
ssinisolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the details in the paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We did not report the error bars because the results are stable and consistent.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computer resources the paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///eeurrr..cccooobb.e/ttte
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss impacts the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used the publicly available dataset in our work. We cited the corresponding papers in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in work does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
27


Reviews:
[{"review_id": "fepHUBmoVJ", "reviewer": "Reviewer_hgee", "summary": "This paper proposes PrefixKV, a method for adaptive KV cache compression in large vision-language models (LVLMs), targeting efficient generation during inference. The core idea is to optimize the retained key-value (KV) cache size per layer, rather than applying a uniform retention policy.  PrefixKV introduces a binary search mechanism to derive a global prefix configuration that maximizes contextual information retention while respecting a global compression budget. The method is evaluated on several models (LLaVA-1.5-7B, 13B, Qwen-VL) and datasets (LLaVA-Description, MM-Vet), showing superior performance to state-of-the-art methods such as H2O and Elastic Cache under a range of compression budgets", "questions": "1. Could you expand on why 10 samples suffice for offline estimation across different datasets and tasks? Are there scenarios where this assumption might break?\n2. How well does a prefix configuration estimated on one dataset (e.g., LLaVA-Description) generalize to another (e.g., MM-Vet)? Have you tested cross-task portability?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths.\n\n-  The paper provides strong motivation for layer-wise KV retention, backed by empirical observations such as Lorenz curves and Gini coefficients. The insight into variable importance distributions across layers is well-founded and novel.\n- The method provides substantial inference speedups (e.g., 1.8× on LLaVA-1.5-7B) while retaining performance, making it highly relevant for real-world deployment.\n\nWeaknesses.\n- While the binary search approach is efficient, the paper does not deeply analyze the sensitivity or convergence behavior of the algorithm (e.g., robustness to approximation thresholds, search steps).\n- Some figures (e.g., Fig. 1 and Fig. 3) are packed and slightly difficult to interpret. More concise visuals or splitting into sub-figures could improve clarity.\n- The term “prefix” is overloaded in the Transformer literature and may confuse readers. A brief clarification early on (e.g., prefix in sorted importance vs. positional prefix) would help.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and insightful comments. Thank you for liking the strong motivation, well-founded and novel insight, substantial inference speedups, and relevant for real-world deployment. We provide the response to each comment in the point-to-point manner below. Please feel free to contact us if you have further concerns.\n\nQ1: While the binary search approach is efficient, the paper does not deeply analyze the sensitivity or convergence behavior of the algorithm (e.g., robustness to approximation thresholds, search steps).\n\nA1: Thanks. We first conduct experiments under the compression ratio of 50% using varying approximation threshold $\\delta$ to evaluate the sensitivity and convergence behavior of binary search. We present the average search steps and PPL results on MM-Vet with LLaVA-1.5-7B in the below table. It shows that the search consistently converges within 7–12 steps, with stable performance across different settings, showing the robustness. \n\n| $\\delta$ | Average search step | Result |\n| -------- | ------------------- | ------ |\n| 0.0125   | 11.5                | 5.50   |\n| 0.025    | 10.8                | 5.50   |\n| 0.05     | 9.3                 | 5.50   |\n| 0.075    | 8.5                 | 5.51   |\n| 0.1      | 7.8                 | 5.53   |\n\nFurthermore, we also fix the number of search steps of binary search for evaluation, to verify the robustness of our approach to different search steps. As shown in the below table, the performance remains stable across difference scenarios, showing the reliability. We will include more results and analyses in the revision.\n\n| Search step | Result |\n| ----------- | ------ |\n| 7           | 5.53   |\n| 8           | 5.51   |\n| 9           | 5.51   |\n| 10          | 5.50   |\n| 11          | 5.50   |\n| 12          | 5.50   |\n\nQ2: Some figures (e.g., Fig. 1 and Fig. 3) are packed and slightly difficult to interpret. More concise visuals or splitting into sub-figures could improve clarity.\n\nA2: Thanks. We will split the packed figures, e.g., Fig.1 and Fig.3, into clearer subfigures with more concise visuals to improve readability. For example, for Fig.1, we will organize (a) and (b) into one subfigure, and present the global prefix configuration space into another subfigure. For Fig.3, for better interpretation, we will move the prefilling and decoding processes into one subfigure, and the overview of our method into another subfigure with clearer layout.\n\nQ3: The term “prefix” is overloaded in the Transformer literature and may confuse readers. A brief clarification early on (e.g., prefix in sorted importance vs. positional prefix) would help.\n\nA3: Thanks. To mitigate confusion, in the revision, we will clarify early and clearly that our usage of the term \"prefix\" refers to the top KV cache vectors based on importance scores, not their original sequence positions. For example, we will add \", where \"Prefix\" means the top-ranked KV based on importance rather than position in the original sequence\" immediately after \"we present PrefixKV\" in Line 12.\n\nQ4: Could you expand on why 10 samples suffice for offline estimation across different datasets and tasks? Are there scenarios where this assumption might break?\n\nA4: Thanks. We use random samples in training set to estimate the cumulative priority sequences of layers and determine the optimal global prefix configuration offline. As shown in Fig. 4 in the paper, the distributions of retained KV cache size ratios and Gini coefficients obtained from different samples exhibit low variance, indicating favorable consistency in importance patterns. The empirical stability justifies the use of as few as 10 samples for offline estimation, as they can approximate the overall layer-wise importance distribution. We also measure the standard deviation (std) of the retained KV cache size ratios obtained via online binary search across all samples, as well as the mean absolute difference (mad) between these values and the configuration of offline estimation. We present the results in the below table. It quantitatively shows that the variance of retained KV cache size ratios across samples is small, indicating the feasibility of offline estimation. The small mean absolute difference also supports the effectiveness of offline estimation. We will incorporate these results in the revision and also release the code for the analyses.\n\n| Ratio | 10%   | 20%   | 30%   | 40%   | 50%   | 60%   | 70%   | 80%   | 90%   |\n| ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| std   | 0.010 | 0.021 | 0.023 | 0.024 | 0.023 | 0.020 | 0.013 | 0.011 | 0.006 |\n| mad   | 0.010 | 0.021 | 0.024 | 0.024 | 0.023 | 0.020 | 0.014 | 0.011 | 0.006 |\n\nBesides, we also acknowledge that in special settings (e.g., adversarial scenario), samples may exhibit higher variability in cumulative priority sequences. Consequently, increasing the number of samples may be required to ensure reliable offline estimation. In this way, the overall distribution can be approximated more accurately.\n\nQ5: How well does a prefix configuration estimated on one dataset (e.g., LLaVA-Description) generalize to another (e.g., MM-Vet)? Have you tested cross-task portability?\n\nA5: Thanks. We use random samples from the training set of model for offline estimation of global prefix configuration, which is then leveraged on the evaluation benchmarks under zero-shot settings. Since the training set is disjoint from the evaluation benchmarks, such assessment can effectively show the capability preservation of the model before and after compression. Thus, the superior results of our method in Table 1 and Table 2 in the paper also show its favorable generalizability and cross-task portability. Following your suggestion, we also conduct experiments to use samples of LLaVA-Description for offline estimation and evaluate on MM-Vet to inspect the generalizability. For baseline, we also use the samples of MM-Vet for offline estimation. We present the PPL results based on LLaVA-1.5-7B in the below table. It further shows that our method can maintain favorable cross-task performance, exhibiting portability and generalizability.\n\n| Offline estimation | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  |\n| ------------------ | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| LLaVA-Description  | 7.40 | 5.99 | 5.72 | 5.53 | 5.50 | 5.44 | 5.38 | 5.28 | 5.28 |\n| MM-Vet             | 7.38 | 5.97 | 5.66 | 5.53 | 5.50 | 5.41 | 5.38 | 5.28 | 5.28 |"}, {"review_id": "EotudxT6Sd", "reviewer": "Reviewer_rNU4", "summary": "This paper tackles the KV cache overhead in large vision-language models (LVLMs). The core argument is that the importance of KV cache vectors is heterogeneous across layers, a claim supported by an analysis of attention distributions. To leverage this, the authors propose PrefixKV, a method that assigns a unique, adaptive compression ratio to each layer. These layer-specific ratios are determined by searching for a single information retention threshold that satisfies a global compression budget. Crucially, this configuration is calculated offline, imposing no overhead during inference.", "questions": "(1) A major practical advantage of your method is the offline estimation of the global prefix configuration from a small sample set. Could you provide more analysis on the robustness of this process? For instance, how much does the derived configuration vary when using different small sets of calibration samples, especially samples from different domains or with different instruction complexities? A convincing analysis, even on a small scale, demonstrating the stability of the derived configuration would significantly strengthen the paper's practical claims.\n(2) The use of different temperature settings for the baseline (temp > 0) and the proposed method (temp = 0) leads to counter-intuitive ROUGE scores where PrefixKV surpasses the full-cache model (Table 1). Could you please provide results from a revised experiment where both the full-cache baseline and PrefixKV use the exact same deterministic generation settings (i.e., temperature 0)? This would provide a more direct and interpretable measure of information preservation.\n(3) The comparison with PyramidKV [65] in Table 4 is crucial, as it isolates the benefit of your search-based adaptive strategy from a manually-designed one. Could you elaborate on your interpretation of these results? Specifically, what is the core mechanism that allows your method to so clearly outperform PyramidKV, especially at aggressive compression ratios (e.g., 10-30%)? A more detailed analysis would help to better situate the novelty of your contribution, moving it beyond \"layer-wise adaptation is good\" to \"our method for finding the adaptation strategy is superior.\"\n(4) The W-shaped curve for retained cache sizes in Figure 4 is an interesting finding. Is there any intuition as to why shallow and deep layers require more cache than middle layers? Is this pattern consistent across different model families, or is it specific to the LLaMA architecture? A brief discussion on this could offer valuable insights into the inner workings of LVLMs and enhance the paper's significance.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "\tStrengh:\n(1) The paper's approach is well-motivated and principled, grounded in a rigorous analysis of KV cache importance heterogeneity. The use of Lorenz curves provides a strong quantitative justification for the proposed layer-wise compression.\n(2) The PrefixKV method is straightforward and elegant. It cleverly reframes a complex multi-layer optimization problem into a simple binary search for a global threshold, and its ability to be pre-computed offline makes the solution highly practical for deployment.\n(3) The experiments show that the method surpasses previous methods significantly.\n\n\tWeakness:\n(1) While the paper presents a well-engineered and effective solution, the core idea of layer-wise KV cache allocation is not entirely novel. The authors acknowledge prior works [65, 19, 27] that explore this direction. The primary contribution thus shifts from the conceptual breakthrough of \"adaptive allocation\" to the specific algorithm for achieving it (binary searching for a global cumulative priority threshold), which, while elegant, can be seen as a strong incremental improvement rather than a fundamentally new paradigm.\n(2) The practicality of the method hinges on the claim that an optimal prefix configuration can be determined offline from a very small number of samples (as few as one, per Table 6). While experimental results are stable, this assumption might not hold under more diverse, out-of-distribution, or adversarial scenarios. The paper lacks a rigorous analysis of the conditions under which this offline estimation remains robust, which is a critical aspect for real-world reliability.\n(3) The experimental setup for ROUGE evaluation, where the uncompressed reference is generated with non-zero temperature while compressed models use temperature 0 (lines 237-239), complicates interpretation. This leads to reported ROUGE scores for the proposed method that are higher than the \"full cache\" baseline (e.g., 0.76 vs 0.62 in Table 1), which is counter-intuitive. A more direct comparison using the same generation parameters for both the baseline and the proposed method would provide a clearer measure of quality preservation.\n(4) The most direct competitor, PyramidKV [65]—which employs a non-uniform, \"manually designed\" layer-wise allocation—is only compared in a supplementary analysis (Table 4). For a stronger claim, PrefixKV should be benchmarked against PyramidKV and other potential adaptive baselines within the main result tables (Tables 1 and 2). The current presentation, which primarily contrasts with uniform allocation methods, may overstate the performance gains attributable to the proposed search strategy versus the general benefit of adaptivity.", "was_revised": true, "final_justification": "The authors responded thoughtfully and strengthened the paper. While minor concerns remain, the work presents a solid contribution to the field and meets the acceptance criteria at this stage.", "rebuttal": "We sincerely appreciate your valuable feedback and insightful comments.\n\nQ1: While the paper presents a well-engineered... \n\nA1: Thanks. While prior works have explored layer-wise KV cache allocation, our contribution lies in introducing a novel analytical perspective. Instead of relying on manual or heuristic ways, we propose the cumulative priority sequence to capture KV cache importance distribution across layers, using Lorenz curves and Gini coefficients to quantify inter-layer disparity. This reframes the KV cache compression as a global prefix configuration problem. Our binary search method naturally follows from this formulation, offering an efficient and principled solution. We believe this perspective provides fresh insights into KV cache compression for LVLMs.\n\nQ2: The practicality of the method hinges on the claim...\n\nA2: Thanks. We obtain random samples from the diverse training corpus of model that covers various scenarios, including open-knowledge visual question answering, optical character recognition, region-level understanding, detailed description, and complex reasoning, etc. Besides, the random samples for offline estimation are disjoint from the benchmarks, showing the robustness of our method under out-of-distribution scenarios. For adversarial scenarios, which lacks available standardized dataset for LVLMs, we acknowledge it as a valuable direction and will investigate in the future work. Additionally, we conduct experiments to evaluate our method in different practical settings, including multi-round and long text VQA scenarios in Table 1 in the supplementary, and high-resolution scenarios in Table 2 in the supplementary. The results show that our method enjoys favorable performance, verifying the practical reliability.\n\nQ3: The experimental setup for ROUGE evaluation...\n\n& Q6: The use of different temperature settings...\n\nA3 & A6: Thanks. We follow prior work [1] for ROUGE evaluation, where for the compression budget of 100%, the score is measured with non-zero temperature. The baseline results also align with the prior work [1], which leads to counter-intuitive comparison. We understand the concern and conduct experiments for the baseline with the same generation parameters, which results in 0.78 / 0.80 for LLaVA-1.5-7B / 13B in Table 1. We will incorporate these results in the revision.\n\nQ4: The most direct competitor, PyramidKV...\n\nA4: Thanks. We conduct experiments with PyramidKV for comparisons in Table 1 and Table 2 based on LLaVA-1.5-7B. The following tables present the benchmark results. We observe that our method consistently outperforms it across datasets and compression ratios. These results suggest that our method can identify better KV cache size allocation for compression. We think that this stems from our method's global prefix configuration optimization from cumulative priority sequences across layers, allowing layer-wise maximal contextual information preservation and thus better remaining performance even at high compression ratios. Please refer to A7 for more detailed explanation. We will include more comparison results in the revision to better highlight the improvements of our method.\n\nTab 1. PPL / ROUGE score on LLaVA-Description.\n\n| Method    | 10%         | 20%         | 30%         | 40%         | 50%         | 60%         | 70%         | 80%         | 90%         |\n| --------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| PyramidKV | 14.3 / 0.31 | 12.4 / 0.31 | 7.16 / 0.31 | 5.75 / 0.37 | 3.80 / 0.51 | 3.47 / 0.55 | 3.41 / 0.59 | 3.20 / 0.73 | 3.20 / 0.74 |\n| Ours      | 4.41 / 0.43 | 3.69 / 0.51 | 3.48 / 0.55 | 3.41 / 0.57 | 3.41 / 0.58 | 3.41 / 0.59 | 3.25 / 0.63 | 3.20 / 0.74 | 3.20 / 0.76 |\n\nTab 2. PPL / ROUGE score on MM-Vet.\n\n| Method    | 10%         | 20%         | 30%         | 40%         | 50%         | 60%         | 70%         | 80%         | 90%         |\n| --------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| PyramidKV | 20.8 / 0.26 | 10.4 / 0.28 | 7.50 / 0.30 | 5.75 / 0.34 | 5.63 / 0.46 | 5.50 / 0.46 | 5.41 / 0.53 | 5.28 / 0.73 | 5.28 / 0.75 |\n| Ours      | 7.38 / 0.39 | 5.97 / 0.41 | 5.72 / 0.46 | 5.53 / 0.46 | 5.50 / 0.48 | 5.44 / 0.50 | 5.38 / 0.59 | 5.28 / 0.74 | 5.28 / 0.77 |\n\nQ5: A major practical advantage of your method is the offline...\n\nA5: Thanks. We conduct experiments using random samples from different domains and with different instruction complexities to verify the robustness of our offline estimation. Specifically, for different domains, we leverage random samples from open-knowledge visual question answering (VQA), optical character recognition (OCR), and complex reasoning (reasoning), respectively. In the below table, we present the PPL results on MM-Vet based on LLaVA-1.5-7B. We observe that the performance remains stable across different domains, showing the robustness. \n\n| domain    | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  |\n| --------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| VQA       | 7.45 | 5.97 | 5.72 | 5.53 | 5.50 | 5.41 | 5.38 | 5.28 | 5.28 |\n| OCR       | 7.38 | 6.01 | 5.72 | 5.53 | 5.53 | 5.41 | 5.38 | 5.28 | 5.28 |\n| reasoning | 7.40 | 5.97 | 5.72 | 5.53 | 5.50 | 5.44 | 5.38 | 5.28 | 5.28 |\n\nBesides, for instruction complexities, we obtain random samples with different instruction lengths: shorter than 100, between 100 and 500, and longer than 500. We present the results in the below table. It shows that our method also exhibits robustness to the instruction complexity, showing favorable generalizability.\n\n| Instruction         | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  | 70%  | 80%  | 90%  |\n| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| length <= 100       | 7.38 | 6.01 | 5.66 | 5.53 | 5.53 | 5.44 | 5.38 | 5.28 | 5.28 |\n| 100 < length <= 500 | 7.42 | 5.97 | 5.72 | 5.53 | 5.50 | 5.44 | 5.38 | 5.28 | 5.28 |\n| length > 500        | 7.38 | 5.97 | 5.72 | 5.54 | 5.50 | 5.41 | 5.38 | 5.28 | 5.28 |\n\nAdditionally, we also follow A4 for Reviewer hgee to measure the mean absolute difference between retained KV cache size ratios obtained via online binary search for all samples and offline-estimated ones by samples with different domains and instruction complexities. The average results across ratios are (VQA: 0.016), (OCR: 0.017), (reasoning: 0.016), (length <= 100: 0.016), (100 < length <= 500: 0.017) , and (length > 500: 0.018). This further shows the favorable robustness of our offline estimation.\n\nQ7: The comparison with PyramidKV in Table 4 is crucial...\n\nA7: Thanks. We think that the core mechanism lies in that our method **globally** optimizes the allocation of KV cache across layers based on cumulative priority sequences. It ensures that each layer retains its most informative portion of KV cache, preserving maximal contextual information. In contrast, PyramidKV employs a manually-designed one (larger cache size in shallow layers and smaller size in deep layers) that may discard high-priority KV cache in some layers while preserving less important ones in others. This allocation becomes especially suboptimal at aggressive compression ratios (e.g., 10-30%), because important KV vectors are more likely to be dropped due to the limited budget. This results in noticeable contextual information loss and clearly inferior performance compared with ours.\n\nQ8: The W-shaped curve for retained cache sizes...\n\nA8: Thanks. We conduct experiments to inspect the retained cache size distribution of Qwen-VL under the compression ratio of 50%, which also presents the similar pattern (Due to that the image is not allowed in the response, we present the average results in the below table). \n\n| Layer     | 1      | 2      | 3      | 4      | 5      | 6      | 7      | 8      | 9      | 10     | 11     | 12     | 13     | 14     | 15     | 16     |\n| --------- | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| Ratio     | 0.91   | 0.76   | 0.73   | 0.41   | 0.34   | 0.34   | 0.35   | 0.40   | 0.51   | 0.53   | 0.57   | 0.56   | 0.57   | 0.58   | 0.54   | 0.51   |\n| **Layer** | **17** | **18** | **19** | **20** | **21** | **22** | **23** | **24** | **25** | **26** | **27** | **28** | **29** | **30** | **31** | **32** |\n| Ratio     | 0.54   | 0.52   | 0.50   | 0.43   | 0.47   | 0.43   | 0.35   | 0.33   | 0.37   | 0.31   | 0.40   | 0.37   | 0.43   | 0.50   | 0.63   | 0.80   |\n\nOur intuition for this draws from the hierarchical processing of LVLMs: (1) Shallow layers are responsible for capturing fine-grained syntactic and visual features from the input, where the foundational token-level information is dispersed across KV cache vectors and favor larger retained cache size. (2) Deep layers directly influence output reasoning and generation. The contextual information across KV cache vectors helps preserve the fidelity of high-level representations essential for final prediction, which relies on larger retention cache size. (3) Middle layers often serve as an intermediate abstraction stage, where feature representations are aggregated and compressed. The importance distribution of KV cache vectors is thus more concentrated, which requires fewer cache vectors to be retained. Prior works on pruning and redundancy in models [2,3] also show that middle layers exhibit more representational redundancy, making them more tolerant to compression. We will incorporate more discussion in the revision to offer more insights.\n\n[1] Efficient Inference of Vision Instruction-Following Models with Elastic Cache. ECCV 2024.\n\n[2] Streamlining Redundant Layers to Compress Large Language Models. ICLR 2025.\n\n[3] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. ACL Findings 2025."}, {"review_id": "NerJ2Azy1S", "reviewer": "Reviewer_K11i", "summary": "The paper proposes a method for efficient attention key-value caching in the context of LVMs. In particular, the authors propose a variable-rate caching mechanism based on the relative importance of cached values across layers. For a fixed ``budget” compression rate, the method aims to optimally estimate the best amount of key-value pairs to be cached per layer, with such optimal ratio found through binary search. The authors apply their method across a variety of models including LlaVa and MM-Vet, as well as in Qwen-VL and LLM models. A set of ablation studies show the efficiency of their proposed approach. The proposed method shows to produce state of the art results for high levels of compression, illustrating the benefit of the proposed approach in low-regime scenarios.", "questions": "My questions are embedded in the weaknesses section above. In particular:\n\n- Novelty/results tradeoff against PyramidKV\n\n- Offline estimation being on par with online estimation\n\n- Merging tokens better than discarding them.", "limitations": "No, there is not such section in the paper. The authors should add such section in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n\nThe paper proposes a simple yet efficient idea, namely that of varying the number of cached tokens per layer, according to their estimated relevance to future attention maps. \n\n\nI find it interesting the fact that offline results are on par with online results, illustrating that some pre-computed relevance is enough to have good and efficient compression. \n\n\nThe results show that for high compression levels their proposed approach is better than alternative methods like H2O or Elastic.\n\n\nThe paper includes a good amount of ablation studies including the tradeoff between compression ratio and results, the effect of online vs offline caching, or the effect of merging rather than discarding tokens. \n\n\nWeaknesses:\n\n\nThe idea, while overall efficient, is relatively simple and the novelty is a bit limited. I see that in Table 4, the proposed approach is almost on par with PyramidKV except for very big compression rates. The comparison novelty-results against PyramidKV is in my opinion a bit limited. \n\n\n\nI find it a bit surprising that the Offline estimation produces nearly exactly the same results as that of online caching. This is very surprising as one would expect that online caching should always be related to current sequences and therefore some variability would be expected in these results in favor of online caching. In a similar way, I find it really surprising that a single sample is enough for the ratio optimality search in an offline setting, with results that are pretty much the same as those of online caching. A single example might contain a rather different semantic information and provoke completely different activations in the attention maps at different layers. The results suggest that the percentage of attention values that are relevant is almost fixed for each layer and is independent of the input content. In other words, the same percentage of tokens will be relevant at each layer for a completely variety of samples, regardless the entropy of each input sequence. This requires further investigation or explanation because it is rather surprising. \n\n\n\n\nThe results in Table 7 also suggest that merging tokens is a better option than directly disregarding them. This requires further investigation against methods for token merging.", "was_revised": true, "final_justification": "The rebuttal has partially addressed my concerns and therefore, considering the other reviewers' consideration for the paper, I choose to keep my original rating. I would like to ask the authors to further investigate the reason why offline and online estimation produce similar results, and add such findings to the final manuscript.", "rebuttal": "We sincerely appreciate your valuable feedback and insightful comments. Thank you for liking the state of the art results, simple yet efficient idea, and a good amount of ablations. We provide the response to each comment in the point-to-point manner below. Please feel free to contact us if you have further concerns.\n\nQ1: The idea, while overall efficient, is relatively simple and the novelty is a bit limited. I see that in Table 4, the proposed approach is almost on par with PyramidKV except for very big compression rates. The comparison novelty-results against PyramidKV is in my opinion a bit limited.\n\nA1: Thanks. Unlike PyramidKV which employs a manually-designed allocation strategy, we quantitatively analyze the inter-layer disparity of KV cache importance distribution across layers through the new cumulative priority sequence perspective, and optimize the global prefix configuration for KV cache retention by binary search. It enables **adaptive** layer-wise cache size budget **with maximal contextual information**. In Table 4, we note that when compression is light, more KV cache redundancy remains in layers, so both methods can preserve sufficient information to maintain generation quality. However, under aggressive compression ratios, the important KV cache vectors are more easily to be discarded and the KV cache size allocation thus becomes more critical. In these scenarios, our method consistently outperforms PyramidKV by notable margins. This underscores the practical advantage and effectiveness of our method, especially in situations where high KV cache compression ratio is essential, such as on resource-constrained mobile and edge devices.\n\nQ2: I find it a bit surprising that the Offline estimation produces nearly exactly the same results as that of online caching. This is very surprising as one would expect that online caching should always be related to current sequences and therefore some variability would be expected in these results in favor of online caching. In a similar way, I find it really surprising that a single sample is enough for the ratio optimality search in an offline setting, with results that are pretty much the same as those of online caching. A single example might contain a rather different semantic information and provoke completely different activations in the attention maps at different layers. The results suggest that the percentage of attention values that are relevant is almost fixed for each layer and is independent of the input content. In other words, the same percentage of tokens will be relevant at each layer for a completely variety of samples, regardless the entropy of each input sequence. This requires further investigation or explanation because it is rather surprising.\n\nA2: Thanks. We understand the concern. We note that as shown in Fig. 4 in the paper and the analyses in A5 for Reviewer rNU4, the distributions of cumulative priority sequence are similar and exhibit beneficial low variance across different samples, domains and instruction complexities. Such property enables offline estimation from a few random samples to provide a favorable approximation of the overall optimal global prefix configuration, i.e., the layer-wise KV cache retention size. In A4 for Reviewer hgee, we also measure the standard deviation (std) of the retained KV cache size ratios obtained via online binary search across all samples, and the mean absolute difference (mad) between those and the configuration of offline estimation. The quantitative results of small standard deviation and small mean absolute difference further verify the effectiveness of offline estimation. The small standard deviation also verifies the feasibility of offline estimation by one sample. We will incorporate these results in the revision and also release the code for the analyses. Although offline estimation may not yield the optimal configuration for every individual input, we hypothesize that it can still identify a generally effective pattern that is less dependent on specific samples and more reflective of the underlying learning mechanisms of model. This hypothesis offers a potentially interesting perspective for understanding the internal learning dynamics of models, which we will further investigate in future work.\n\nQ3: The results in Table 7 also suggest that merging tokens is a better option than directly disregarding them. This requires further investigation against methods for token merging.\n\nA3: Thanks. We acknowledge the better option of merging tokens compared with directly dropping them. Meanwhile, we note that the improvements are relatively marginal. Following your suggestion, we will also explore more effective merging techniques in the future work. \n\nQ4: There is not limitation section in the paper. The authors should add such section in the paper.\n\nA4: Thanks. We present the limitation section in the supplementary."}, {"review_id": "lJGL2S7oJY", "reviewer": "Reviewer_ByA6", "summary": "The paper introduces PrefixKV, an innovative method for KV cache compression in LVLMs. It adjusts layer-wise KV cache retention ratio based on varying importance distribution of KV vectors across layers. By utilizing offline binary search to determine the optimal prefix configuration, PrefixKV aims to enhance inference efficiency while maintaining high-quality generation. The paper presents extensive experiments, showing superiority in terms of inference speed and performance.", "questions": "See the weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.It is interesting to leverage the varying layer-wise KV importance for developing a KV cache budget strategy with the analyses from the Lorenz curve and Gini coefficient perspectives. The offline estimation of the global prefix configuration ensures that the method does not incur additional inference costs, making it practical for deployment. \n2. The method achieves state-of-the-art performance across various compression ratios, maintaining high performance even at low compression ratios. The experiments are conducted across multiple models and benchmarks, which present comprehensive analyses and comparisons and verify the efficacy of the proposed method. \n\nWeaknesses:\n1. The related work section is somewhat short. A more detailed introduction and discussion of recent KV cache compression techniques would provide a clearer context.\n2. The \"prefix\" in PrefixKV might be somewhat misleading. Specifically, \"prefix\" usually denotes the initial segment of a sequence, but the proposed method preserves tokens based on significance, rather than position in the sequence.", "was_revised": true, "final_justification": "My concerns are well addressed in the rebuttal, and I maintain my original score to accept this paper.", "rebuttal": "We sincerely appreciate your valuable feedback and insightful comments. Thank you for liking the extensive experiments, practical for deployment, and state-of-the-art performance. We provide the response to each comment in the point-to-point manner below. Please feel free to contact us if you have further concerns.\n\nQ1: The related work section is somewhat short. A more detailed introduction and discussion of recent KV cache compression techniques would provide a clearer context.\n\nA1: Thanks. We will enrich the related work section to provide a more comprehensive overview and clearer context in the revision. For example, we will elaborate on the background and advantages of KV cache compression. We will also provide more detailed descriptions of the motivations, methods, and results of KV cache compression techniques, and further enrich the discussions and analyses.\n\nQ2: The \"prefix\" in PrefixKV might be somewhat misleading. Specifically, \"prefix\" usually denotes the initial segment of a sequence, but the proposed method preserves tokens based on significance, rather than position in the sequence.\n\nA2: Thanks. We use the term \"prefix\" to refer to the top KV cache vectors sorted by importance score. We understand the concern and will clarify this more early and clearly in the revision. We will also consider revising the name of our method, e.g., TopKV, to avoid misleading."}]

Author rebuttal:

Meta-review: This paper proposes PrefixKV, a framework for adaptive per-layer KV cache compression in large vision-language models, with a focus on efficient inference. 

The paper is well-motivated, providing a principled analysis of inter-layer heterogeneity.  PrefixKV introduces an efficient binary search strategy to allocate compression ratios per layer under a global budget. The method achieves substantial inference speedups and state-of-the-art performance across multiple models and datasets.  
Related approaches (e.g., PyramidKV) have been proposed previously, and the main advancement lies in the automated allocation approach rather than a new paradigm. 

Recommendation: 
Acceptance. The paper meets standards of soundness and originality for a conference paper, with reasons for acceptance outweighing reasons for rejection, although some missing clarity merit caution. Further discussion of limitations, clarifications, and more direct comparison with prior adaptive methods (e.g., PyramidKV) are encouraged for the camera-ready version.
Decision: Accept (poster)