Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models
Authors: Xuyang Liu
Xiyan Gui
Yuchao Zhang
Linfeng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Vision-language Models
Inference Efficiency
Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: MixKV enhances baseline KV cache compression for LVLMs by balancing importance and diversity, delivering gains under extreme compression without sacrificing inference efficiency.

Abstract: Recent large vision-language models (LVLMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet the resulting key-value (KV) cache expansion creates a critical memory bottleneck that fundamentally limits deployment scalability.
While existing KV cache compression methods focus on retaining high-importance KV pairs to minimize storage, they often overlook the modality-specific semantic redundancy patterns that emerge distinctively in multi-modal KV caches.
In this work, we first analyze how, beyond simple importance, the KV cache in LVLMs exhibits varying levels of redundancy across attention heads. 
We show that relying solely on importance can only cover a subset of the full KV cache information distribution, leading to potential loss of semantic coverage.
To address this, we propose MixKV, a novel method that mixes importance with diversity for optimized KV cache compression in LVLMs. MixKV adapts to head-wise semantic redundancy, selectively balancing diversity and importance when compressing KV pairs. Extensive experiments demonstrate that MixKV consistently enhances existing methods across multiple LVLMs. Under extreme compression (budget=64), MixKV improves baseline methods by an average of 5.1% across five multi-modal understanding benchmarks and achieves remarkable gains of 8.0% and 9.0% for SnapKV and AdaKV on GUI grounding tasks, all while maintaining comparable inference efficiency. Furthermore, MixKV extends seamlessly to LLMs with comparable performance gains.

Published as a conference paper at ICLR 2026
MIXING IMPORTANCE WITH DIVERSITY: JOINT OP- TIMIZATION FOR KV CACHE COMPRESSION IN LARGE VISION-LANGUAGE MODELS
Xuyang Liu Xiyan Gui Yuchao Zhang'
Linfeng Zhang |s
EPIC Lab, Shanghai Jiao Tong University Sichuan University
3 Huazhong University of Science and Technology Homepage: https: htttps:/xuuaag--aal -liua github 1o/MixKV/
ABSTRACT
Recent large vision-language models (LVLMs) demonstrate remarkable capabil- ities in processing extended multi-modal sequences, yet the resulting key-value (KV) cache expansion creates a critical memory bottleneck that fundamentally limits deployment scalability. While existing KV cache compression methods fo- cus on retaining high-importance KV pairs to minimize storage, they often over- look the modality-specific semantic redundancy patterns that emerge distinctively in multi-modal KV caches. In this work, we first analyze how, beyond simple importance, the KV cache in LVLMs exhibits varying levels of redundancy across attention heads. We show that relying solely on importance can only cover subset of the full KV cache information distribution, leading to potential loss of semantic coverage. To address this, we propose MixKV, a novel method that mixes im- portance with diversity for optimized KV cache compression in LVLMSs Mi xKV adapts to head-wise semantic redundancy. selectively balancing diversity and im- portance when compressing KV pairs. Extensive experiments demonstrate that Mi XKV consistently enhances existing methods across multiple LVLMs. Under extreme compression (budget=64), Mi xKV improves baseline methods by an av- erage of % across five multi-modal understanding benchmarks and achieves remarkable gains of 8.0% and 9.0% for SnapKV and AdaKV on GUI ground- ing tasks, all while maintaining comparable inference efficiency. Furthermore, Mi xKV extends seamlessly to L.MSs with comparable performance gains.
INTRODUCTION
Large vision-language models (LVLMs) (Li et al., 2024a; Chen et al., 2024d) have achieved re- markable performance in multimodal understanding by effectively integrating visual information and user instructions into the input space of large language models (LLLMs) (Grattafiori et al., 2024; Yang et al., 2025a). With the growing demand for understanding long-context visual inputs, includ- ing high-resolution images (Bai et al., 2025b; Zhu et al., 2025) and long videos (Shu et al., 2025; Qin al., 2025), LVLMs must process an increasing number of visual tokens. However, process- ing such long-context inputs generates numerous key-value (KV) pairs in the KV cache of LLMs, substantially increasing GPU memory consumption and degrading computational efficiency due to memory access latency and bandwidth constraints (Liu et al., 2025c; Shao et al., 2025b).
To address the KV storage overhead, two main approaches have emerged. Token compression meth- ods (Yang et al., 2025b; Chen et al., 2024a; Liu et al., 2025b) directly compress visual tokens to in- directly reduce KV cache storage, but often underperform in high-resolution fine- aee-ggriiedd tasks like text understanding (Singh et al., 2019) and document processing (Mathew et al., 2021). More effec- tive KV cache compression methods directly evict KV pairs in the LLM to minimize storage while preserving performance (Wan et al., 2024; Li et al., 2024b), thereby enhancing decoding efficiency and throughput. However, current KV cache compression methods for LVLMs predominantly rely on attention-baue based importance scores to decide which KV pairs to retain (Tao et al., 2025a; Wang et al., 2025b). While this strategy does reduce the KV cache size, it fails to consider the intrin- sic semantic characteristics of KV pairs in multi-modal settings. To bridge this gap, we conduct comprchensive analysis and identify two key characteristics of KV pairs in LVLMs:
"qqual contribution. Corresponding author: zhangl i wnw.ingnnnnsuu..om edu.

---

Published as a conference paper at ICLR 2026
Ceenc, Layer 6, Head
Qwand Lager 15, Head
Qwan2 Lager lead
Say Seriarly Outribtter Compariaan
Theer ondex
Take
Toban Irdde
Toee Index
Tooen Indlec
Orline Seeiatty
14
Varue Simvirrty Dslnbbuten Comparioon
Index
Toom redex
Come
Colnne Unalaityy
(a) Cosine Similarity between All Tokens
(b) Similarity Distribution
Figure 1: Visualization of KV cache redundancy across different models. (a) presents the simi- larity of keys across the same layer and head, with Qwen2-Vi VL (bottom) processing vision-language information and Qwen2 (top) handling pure text information. (b) quantifies the average similarity distribution of keys (top) and values (bottom) across all layers and heads for Qwen2 and Qwen2-VL. VI.
(I) Vision-Language Redundancy Differences: Visual information in LVLMs contains signifi- cantly more semantic redundancy than textual information in LLMs. Images often contain repeti- tive visual elements (e.g., similar textures, repeated patterns), leading to higher semantic similarity among KV pairs during vision-language processing. Figure provides compelling evidence: (a) shows that Qwen2-VL exhibits much denser sighsiimllarity regions compared to the more diverse patterns of Qwen2, while (b) reveals that keys in Qwen2 peak around 0.2-0.4 average similarity whereas Qwen2-VL. keys peak around 0.6-0.8, 2-3x increase. This demonstrates that KV pairs in LVLMs exhibit substantially higher semantic redundancy than in LLMs.
(II) Head-wise Redundancy Differences: Within LVLMs, different attention heads focus on dis- tinct multi-modal aspects (Wang et al., 2025b). Some heads capture global features with lower redundancy, while others focus on local details with higher semantic similarity. Figure 2 illustrates this phenomenon across multiple tasks: for Owwn2-VL-77 certain heads show extremely high av- erage similarity exceeding 0.9, while other heads maintain relatively low similarity below 0.3. This pattern is consistent across different vision-language tasks, indicating that KV pairs in LVLMs show varying degrees of semantic redundancy across attention heads in the LLM.
Furthermore, our analysis reveals that importance-basse compression methods fail to fully repli- cate the information distribution of the original KV cache, leading to potential information loss (Figure 3). Therefore, we argue that beyond importance, preserving diverse KV pairs at per-head granularity is essential for minimizing semantic redundancy while maintaining comprehensive infor- mation coverage. To this end, we propose Mi XKV, which adopts a principled "mixing importance with diversity" approach. Specifically,, MixKV extends existing importance-based KV compression methods by incorporating head-wise semantic diversity evaluation. By independently measuring mantic similarity within each attention head, Mi XKV adaptively balances importance and diversity per head achieve fine-grained joint optimization of KV cache compression LVLMs.
Mi xKV is plug-and-play framework that enhances existing KV compression methods with consis- tent performance gains, maintaining inference efficiency while hetter preserving the distributional properties of the original KV cache. In summary, the main contributions are as follows:
1. Semantic Redundancy Analysis. We conduct in-depth analyzes of KV caches in LVLMs, re- vealing substantial inherent semantic redundancy. Besides, we demonstrate that importance-bast methods fail to preserve full KV distribution coverage, exposing fundamental limitations.
2. Mixing Importance with Diversity. Based on our analysis, we propose MixXV XiVVV head-wise adaptive mechanism that quantifies semantic redundancy to create principled weighting between importance and diversity scores for the joint optimization of KV cache compression.
3. Comprehensive Experimental Validation. Extensive experiments across diverse multi-modal and text benchmarks demonstrate that Mi xXV yields consistent performance improvements for existing importance- based compression methods while maintaining inference efficiency.
2

---

Published as a conference paper at ICLR 2026
Devogg
Hertinbee
(a) Head Similarity of LLaVA-NeXT--Mittal-78
(b) Head Similarity of Qwen2-VL-78
Visualization of KV cache redundancy across different heads in LLMs. (a) and (b) present the average cosine similarity across heads of LLaV-NccTT-Mistaa- and Qwen2- within the LLMs, with brighter heads indicating greater similarity in semantic information.
Figure 2:
2 RELATED WORK
Large Vision-language Models. Current large vision-language models (LVLMs) integrate a vision encoder (i.e. ViT), a projector module, and a large language model (LLM) to enable multi-modal comprehension (Liu et al., 2023; 2024a). To meet the growing demand for high-resolution image understanding, recent LVLMs introduce higher-resolution inputs via dynamic cropping strategies, such as LLaVA-NeXT (Liu et al., 2024b) and Vn series (Chen et al., 2024c:b), or native resolution processing like Qwen2-VL. (Wang et al., 2024; Bai et al., 2025b) and GLM-4.5V (Hong et al., 2025). Additionally, video large language models (VideoLLMs) such LLaVA-Video (Zhang et al., 2024) and Video-XL-2 (Qin et al., 2025) process This trend dramatically increases visual token counts, leading to substantial computational costs and GPU memory burdens due to the key-value (KV) cache in the attention mechanism.
vidcos with thousands of frames.
Long-Context Optimization. Longer contexts generally improve the performance of LVLMs and enable more comprehensive multi-modal understanding (Wang et al., 2025c; Chen et al., 2025). Extensive work aims to make long-context processing more efficient and can be broadly categorized into three directions: (i) Efficient computational architectures, such as sparse attention (Li et al., 2025b; Xu et al., 2025), linear attention (Li al., 2025a), and state-space models (Gu & Dao, 2024), which reduce the quadratic complexity of attention with respect to sequence length and thus accelerate long-context processing: (ii) Model-centric compression, including network pruning (Ma et al., 2023), model quantization (Wang et al., 2025a), and knowledge distillation (Cai et al., 2025a), which reduce parameter count and thereby lower the computational and memory cost of long-context inference; and (iii) Data-centric compression, which reduces the effective context length or storage processed by the model, for example via token compression (Yang al., 2025b; Shao et al., 2025a), KV cache compression (Li al., 2024b; Wang et al., 2025b) or KV cache quantization (Liu et al., 2024d; Tao et al., 2025b), thus directly improving the efficiency of long-contex: computation. These three directions optimize long-context processing from complementary perspectives and are largely orthogonal to each other. Given that the context lengths required by modern applications have rapidly increased (Liu al., 2025c), in this work focus on the data-centric perspective and compress the stored KV cache to enable efficient long-context computation for LVLMs.
KV Cache Compression. The KV cache stores computed key-value (KV) pairs during the LLM's pre-filling phase to avoid redundant computations in decoding and enhance inference efficiency. However, long-context multi-moo multi-modal inputs impose sigmificant GPU memory burden on the KV cache. To alleviate this, several works propose KV cache compression techniques, categorized as: (i) vision token compression that directly compresses vision tokens (Chen et al., 2024a; Liu al., 2025a), and (ii) KV cache compression that compresses stored KV pairs during pre-filling (Zhang et al., 2023; Liu al., 2024e). Current KV compression methods are mainly designed for LLMs, such SnapKV (Li et al., 2024b), which clusters important KV positions using attention patterns from an observation window; KNorm (Devoto et al., 2024), which uses & key norms to score and retain KV pairs with lower norms; and AdaKV (Feng et al., 2025), which adaptively allocates eviction budgets across attention heads. Methods specifically designed for LVLMs include InfiniPot- V (Kim et al., 2025), which employs Value Norm for KV pair selection, and SparseMM (Wang al.,
3

---

Published as a conference paper at ICLR 2026
2025b), which allocates asymmetric budgets across attention heads based on their importance and retains high-attenn high-attention KV pairs from observation windows. Most existing compression methods follow the paradigm of retaining critical KV pairs and evicting less important ones.
Unlike prior works that focus primarily on importance-based selection, we identify a critical char- acteristic in LVLMs: heterogeneous head-wise redundancy, where KV caches exhibit varying de- grees of semantic redundancy across attention heads (Figure 2), causing importance-only methods retain KV pairs that fail cover the full information spectrum in high-redundancy heads. This motivates us to jointly consider importance and diversity for more effective KV cache compression.
3 METHODOLOGY
3.1 PRELIMINARY: LARGE VISION-LANGUAGE MODELS
LVLM Architecture. Contemporary large vision-language models (LVLMs) generally adopt VViT-Projector-LLMM"" architecture (Li et al., 2024a; Wang et al., 2024), which consists of three primary components. For input image I R x3 video V (vvv, The visual encoder (i.e. VIT) extracts visual features, yielding embeddings E E RN for images or E = )= E 12'xNXX
RT*xxxxx3, (i)
for videos; (ii) A projection layer, often a two-layer MLP, maps these to vision tokens F" E MxD' for images or F" = {f.")] and (iii) The LLM processes the combined visual and text tokens F' in two phases: During the pre- filling phase, it computes KV pairs for all input tokens (F" and F²) and stores them in the KV cache to avoid redundant computations; in the decoding phase, it auto-regressively generates responses, leveraging the KV cache for efficient retrieval of prior KV pairs in attention mechanisms:
Ji=1 E Xxx for videos, with M S N;
( FY
II p ( F", F",F
where Y= {( is the output sequence, and denotes the KV cache. Thus, LVLMs achieve multi-modal understanding based on visual inputs and user instructions.
KV Cache Compression. Multi-modal long-context sequences result in numerous KV pairs, which lead to significant memory overhead. KV cache compression addresses this challenge by intro- ducing a compression operator 4 which selectively reduces the number of stored KV cache (Wang et al., 2025b). Typically, & involves an evaluation function E that assigns scores = E(K), V%,,) to each KV pair i, with compression based on these scores by retaining top-b highest-scoring pairs given budget B, such that for each layer / and head h, KV pairs K, V RTXD (I' is se- quence length, D dimension per head) are compressed into compact representations K V = TKp Vh, {=)). KV cache compression targets KV tensors computed during pre-filling, reducing memory burden while supporting efficient attention in decoding.
3.2 ANALYSIS OF KV PAIRS CHARACTERISTICS
To optimize KV cache compression in LVLMs, we begin by analyzing key characteristics of the KV cache. An intuitive characteristic is importance, aimed at retaining KV pairs with greater signifi- cance while compressing those with lesser importance, thereby enabling efficient compression.
Importance Metries. Existing methods evaluate KV pair importance from two perspectives, intrin- sic and extrinsic, each employing distinct metrics to compute importance scores Sjmp:
Intrinsic Importance: Determined by inherent KV vector properties, including Key Norm (KNorm) (Devoto et al., 2024), which computes the & norm of each key vector with its nega- in (KNomm for compression scoring, and Value Norm (VNorm) (Kim tive assigned as sin iimp. - Simp.
et al., 2025), which calculates the & norm of each value vector, using it directly as mmp. in (VNorm) for scoring. Compression retains the KV pairs with higher sin smp. such that K; V = Sump. TKK, Vh, (T where simp. represents the intrinsic importance.
Extrinsic Importance: Quantified by average attention scores from an observation window at the prompt end (default length 32) (L.i et al., 2024b; Cai et al., 2025b), reflecting instruction
4

---

Published as a conference paper at ICLR 2026
L jewindow Attention ; is computed from attention weights between query Q3 and key K, balancing modalities effectivelyy This score serves as Simp,i for compression. Compression prioritizes pairs with higher simp.i such that K ho V = TopB(K),, Vh { -.=)). where mmp primarily reflects instruction relevance.
relevance. The score Simp.i |widdww
We argue that comprehensive assessment of KV pair importance requires integrating both in- trinsic and extrinsic perspectives. This ensures a balanced evaluation of inherent significance and instruction relevance. Specifically,, the integrated importance score is computed as:
Simp.i
s smp.i + sip..
where we employ (VNorm) as the default intrinsic component. To ensure compatibility between Sup,i VNorm and attention-ben based extrinsic importance, we normalize VNorm scores to [o, 1] and scale smp
4t pum -e" where iimp and Spom de- note respective mean values. Comprehensive ablation studies in Section 4.3 Table 4 validate the effectiveness this design choice. However, methods relying solely on importance suffer from a critical limitation: they preferentially retain semantically similar KV pairs, leading significant redundancy in the compressed cache and loss of global semantic coverage.
sin them to match attention score magnitudes: s swaled,; Snom,
Figure 3 further presents this limitation, which performs a t-SNE visualization of KV cache distributions. We ob- serve that methods relying only importance, such as SnapKV (Li et al., 2024b) (blue stars). fail to adequately cover the full KV cache information. In Figure 3, SnapKV primarily focuses on a small portion of the information, losing semantic coverage compared the full KV distribu- tion (light gray circles). This is because importance-basee methods prioritize task-relevant, highly similar informa- tion, often neglecting the broader diversity of KV pairs. As result, relying solely on importance introduces redun- dancy by retaining semantically similar KV pairs, which do not provide the full semantic richness of the KV cache. Therefore, effective KV cache compression in LVLMs quires incorporating diversity to retain non-redundant KV pairs. This enables KV cache compression methods to ap- proximate the full original semantic distríbution of the uncompressed KV cache more effectively.
FAERY Spapiyy
Figure 3: t-SNE visualization of KV cache distributions under different settings. "Full KV" represents the original KV distribution of LW without compression, serving as the reference distribution.
Diversity Metrics. Beyond importance, semantic diversity serves as another crucial characteristic for effective KV cache compression in LVLMs. We focus key diversity, as keys primarily govern the attention patterns and semantic focus within a head, making them the most direct levers for controlling information redundancy. To quantify diversity in computationally efficient manner, we adopt the negative cosine similarity between each key and the global average key, as it serves as an intuitive proxy for capturing the breadth of the semantic distribution. For each layer I and
head h, we first normalize each key vector: K, = ||K,,|| II' where K),, E R represents the th key vector and I' is the sequence length. We then compute single global key representation via averaging: K, = = +E.,K, K This ensures that the diversity scores, computed via cosine similarity, are obtained in linear time with respect to I'. Specifically, the diversity score for each KV pair is idiv - Ky Here, higher scores indicate greater distinctiveness from the global pattern, as the negative cosine similarity encourages the retention of diverse information.
K,,,
3.3 HEAD-WISE ADAPTIVE MIXING MECHANISM
Section 3.2 analyzes importance metrics Sump and diversity metrics Sav. both critical for compression. A natural question arises: How these complementary metrics be combined effectively?. While simply adding the importance and diversity scores (Simp + saw)) offers simplicity, Figure shows that varying semantic similarity levels across attention heads make uniform mixing sub-optimal.
5

---

Published as a conference paper at ICLR 2026
Our central insight is that heads exhibiting higher semantic redundancy should prioritize diver- sity preservation to prevent similar KV pair retention, while less redundant heads can emphasize importance-based selection. This motivates the head-wise adaptive weighting mechanism in Mi xkV.
Head-wise Redundancy Quantification. We develop a principled approach to quantify semantic redundancy within each attention head. For layer I and head h, we employ the off-diagonal average similarity of normalized key vectors as our redundancy measure.
Using the normalized key matrix K2 E TRTxD from diversity computation, we construct the simi- larity matrix R K (K,) E RTXT, By exploiting the relationship between dot products and norms, the total similarity sum becomes:
T <ER'
Σ R s
Kins
Σ
K
T
K
where Ry = normalization process, the off-diagonal average similarity is:
-
K h.s Given that normalized vectors yield unit diagonal elements due to the
T K -T T(T 1)
F/
This formulation ensures that as F/. + 1 (high redundancy), diversity weight increases prevent redundant retention, while F/2 (low redundancy) prioritizes importance-based selection. This mathematical property positions F/, as ideal adaptive weight for diversity scores.
Head-wise Adaptive Mixing. Based on the redundancy quantification, we develop the head-wise adaptive mixing function (-head (.). To ensure scale compatibility between importance and diversity
mmin((5 (15"")
scores, we first normalize diversity scores to [0, 1]: gdiv
max, (1 min, (xth) +c" then scale them to
adiv match the magnitude of importance scores: "scaled, adiv
Siup
3a
The comprehensive score is computed through our hisd adaptive mixing function Whead (.):
comp
W Wheadd
(imp + sdiv) = (1 fh)) Simp,i + F/. swaled,
(5)
Through this formulation, MixKV achieves adaptive adjustment: redundant heads (F), + 1) empha-
size diverse KV pairs, while less redundant heads (F), + 0) prioritize importance. This head-wise adaptation ensures the compressed KV cache preserves both critical information and semantic di- versity. KV compression is realized by selecting the top- B pairs with the highest comprehensive
scores: K, V = TopB Vh, (s:mmp) T 1). Notably, Figure 3 demonstrates that the adaptive mixing strategy
XKV enables SnapKV to leverage KV cache diversity, thereby capturing a broader range of information and encompassing wider distribution of the full KV cache (high- lighted red circles). See Figure? 7 for more visualizations and Appendix A.6 for the algorithm.
EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Model Details. We evaluate MixkV across a diverse of architectures to ensure generalizability: (Liu et al., 2024b), InternVL.3-B (Li et al., 2024a), and (iven2-VL-77 Instruct (Wang et al., 2024) for multi-modal understanding tasks; Owen2.55-VL-77B-Instt et al., 2025b) for GUI grounding tasks; and Mistral-7B- --t.---t--nnttttto..22222 (Jiang et al., 2023) and Llama- (Grattafiori et al., 2024) for text-only evaluation.
(Bai
Benchmark Details. We select range of multi-modal understanding benchmarks and com- prehensive text understanding benchmark for evaluation. For image understanding, we include DoA (Mathew et al., 2021), OCRBench (Liu et al., 2024c), TextVQA (Singh al., 2019), ChartQA (Masry et al., 2022), and TextCaps (Sidorov et al., 2020), along with ScreenSpot-v2 (Wu al., 2024) for GUI grounding. For text understanding, adopt LongBench (Bai et al., 2024).

---

Published as a conference paper at ICLR 2026
Table 1: Performance on multiple image understanding benchmarks. Since SparseMM (Wang et al., 2025b) does not provide head importance scores for InternVL3-8B (Zhu et al., 2025), we cannot reproduce their results on this model. "Full KV" means caching all KV pairs (upper bound).
DocVQA (%)
OCRBench (%)
TextVQA (%)
CharlQA (%)
TextCaps
Methods
256
128
64
256
128
64
256
128
64
256
128
64
256
128
64
LLawA-NeNT-Mistral--T
Full KV
63.6
52.9
65.7
52.9
0.707
Snapky
59.7
55.2
47.3
45.0
39.0
31.9
63.5
61.0
57.1
50.2
47.5
42.7
0.650
0.558
0.444
MMXXV
61.7
58.1
48.8
49.9
44.7
36.1
65.2
64.3
60.1
50.8
47.7
43.6
0.708
0.659
0.514
+2.0
+2.9
+1.5
+4.9
+5.7
+4.2
+1.7
+3,3
+3.0
+0.6
+0.2
+4..9
+0.058
+0.101
+0.070
PyramidKY
58.2
54.3
43.4
44.1
39.4
29.1
62.9
60.9
54.8
49.
47.1
40.8
0.621
0.553
0.407
MixXV
60.8
57.2
45.1
49.7
43.7
32.0
64.9
63.8
57.8
50.8
47.5
41.3
0.687
0.656
0,466
Arsutins
+2.6
+2.9
+1.7
+5.6
+4.33
+2.9
+2.0
+2.9
+3.0
+1.7
+0.4
+0.5
+0.066
+0.103
+0.059
AdaKV
45.1
0.566
0.440
Mi XXV
61.3
58.3
50.8
49.8
44.9
36.6
65.3
63.7
59.6
50.9
48.5
45.2
0.704
0.660
0.509
Abadiic
+2.1
+4.77
+4.5
+0.77
+0.055
+0.094
+0,669
SparseMM
61.6
60.8
57.6
51.9
50.7
46.2
65.1
64.7
62.8
51.9
51.2
48.9
0.680
0.634
0.524
+ MiXXV
61.9
61.0
59.2
50.8
50.4
49.5
65.2
65.0
64.4
51.8
51.5
50.6
0.682
0.652
0.575
Avodice
+0.3
+0.2
+1.6
-1.1
0.3
+3.3
+0.1
+0.3
+1.6
00.1
+0.3
+1.7
+0.002
+0.018
+0.051
InternVI. VL3-8B
Full KV
91.0
84.2
81.1
86.4
1.042
SnapKV
85.4
71.9
86.2
1.009
0.901
0.734
89.4
86.2
76.3
81.9
71.1
52.3
80.9
78.8
72.9
86.3
84.8
80.7
1.029
0.949
0.753
+2.1
+0.1
+0.99
+0,048
PyramidKY
87.2
82.7
69.7
70.9
58.4
41.8
78.3
75.3
67.2
85.7
84.0
78.0
0.896
0.809
0.632
+ -MIXVV
87.5
83.5
70.4
72.3
60.0
41.2
79.0
76.6
68.2
85.8
84.4
78.6
0.941
0,850
0.646
Amaliir
+0.3
+0.8
+0.7
+1.4
+1.6
-0.6
+0.7
+1.3
+1.0
+0.1
+0.4
+0.6
+0,045
+0.041
+0.014
AdaKY
89.2
86.0
77.2
80.8
70.2
53.1
80.4
78.0
71.8
86.2
84.4
80.4
1.013
0.921
0.759
*MIXVV
89.5
86.7
78.1
82.4
71.6
52.3
80.8
78.7
72.9
86.2
85.2
80.9
1.034
0,955
0,782
Assutin
+0.3
+0.7
+0.9
+1.6
+1.4
-0.8
+0.4
+0.7
+1.1
+0.0
+0.8
+0.5
+0.021
+0.034
+0.023
Qwen2-VLL -L--------nntt
Full KV
82.1
82.
1.469
SnapKV
88.0
80.1
66.5
77.3
71.9
62.4
80.3
77.5
69.9
81.3
79.6
75.5
1.360
1.142
0.794
+ MiXXK
90.5
82.6
67.9
79.3
75.4
66.0
81.9
80.6
72.5
81.6
81.2
77.6
1.470
1.342
0.878
Aqudice
+2.5
+2.5
+1.4
+2.0
+3.5
+3.6
+1.6
+3.1
+2.6
+0.33
+1.6
+2.1
+00.110
+0.2010
+0,084
PyramidKY
81.7
74.0
59.9
74.5
67.9
56.8
78.3
74.6
65.3
81.1
79.2
73.5
1.115
0.951
0.569
-ixXV
84.0
76.3
60.8
76.6
72.6
58.4
80.4
77.1
81.3
75.5
1.348
1.119
0.633
Abantins
+2.3
+2.3
+0.9
+2.1
+4.7
+1.6
+2.1
+2.5
+1.7
+0.2
+1.5
+2.0
+0.233
+0.168
+0,064
AdaKV
1.345
1.146
0.775
+ MixKV
90.3
82.1
67.8
79.3
74.7
65.5
81.8
79.6
71.2
81.5
80.9
77.4
1.448
1.275
0,878
Asasiii
+0.129
SparseMM
93.5
91.5
84.9
81.2
79.0
74.3
82.0
81.6
77.3
82.0
81.5
80.
1.482
1.430
1.038
-iXXV
93.8
92.7
86.4
82.0
81.0
77.1
82.0
82.0
80.9
81.6
81.8
81.4
1.480
1.459
1.303
Amallie
+0.3
+1.2
+1.5
+0.8
+2.0
+2.8
+0,0
+0.4
+3.6
-0.4
+0.3
+1.3
-0.002
+0.029
+0.265
Implementation Details. We integrate Mi XKV with various KV compression methods, includ-
ing SnapKV (Li et al., 2024b), PyramidKV (Cai et al., 2025b), AdaKV (Feng et al., 2025), and SparseMM (Wang et al., 2025b), across different KV cache budgets. Details are in Appendix A.3.
4.2
MAIN RESULTS
Performance on Multi-modal Understanding Benchmarks. Table summarizes MixkV inte- grated with baseline methods across models and benchmarks, and highlights three key advantages: (i) Universal improvements: MixKV consistently enhances baselines across architectures, tasks, and budgets, supporting the benefit of mixing importance with diversity. (ii) Broad applicabil- ity: it benefits wide range of paradigms, from simple baselines such as SnapKV to layer-wise or head-wise budget allocation methods (PyramidKV V and SparseMM), by modifying only the eval- uation function without changing the compression operator. (iii) Model compatibility: Mi XXV estimates head-wise redundancy on-the-fly per sample, avoiding offline statistics (as in SparscMM) and enabling a plug-and-pla, integration with existing LVLMs. Additional results in Table 7 and Table 6 further verify its robustness on larger LVLMs and MoE-basee based models, including Int. 38B (Zhu et al., 2025) and -ieeen3-VL330---333 TL-30B-AAB--nntrruce (Bai et al., 2025a).
Performance on GUI Grounding Benchmarks. Recent advancements in LVLMs demonstrate their capability to understand GUI scenarios (Cheng et al., 2024; Tang et al., 2025b;a), which rely on edge-side comprehension and necessitate KV compression to reduce storage demands. To this end,
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance on ScreenSpot-r2 GUI grounding benchmark with Qwen2.5-VL-7B-
Instruct. "Full KV" refers to caching all KV pairs of the LLM (upper bound).
Methods
Mobile Text
Mobile Icom/ Widget
Desktop
Desktop loon/Widget
Text
Web Icon/Widget
Average
128
64
128
128
64
128
128
64
128
128
64
Qwen2.5. In.t-tttrctt
Full KV
$7.7
91.2
88.5
SnapKV
65.5
28.6
78.7
53.1
86.1
57.2
74.3
57.1
76.9
26.6
74.4
49.8
75.3
46.9
MLXXV
86.6
35.5
85.3
60.2
87.1
71.1
75.0
68.7
85.0
$3.0
76.4
56.2
83.3
54.9
Abadix
+21.1
46.9
+1.0
+13.9
+00.
48.6
+8.1
46.4
42.0
46.4
48.0
PyrumidKV
11.0
82.0
33.0
75.0
69.2
20.5
71.4
+ -EXXVV
64.1
15.9
74.4
42.2
87.1
41.8
74.5
47.1
76.9
24.8
71.9
27.1
74.1
31.1
Studas
+18.6
+4.9
+12.3
+8.1
+5.1
+8.8
-0.7
-0.8
+7.7
+4.3
+0.5
+3.0
+8.5
+5.0
70.6
74.3
82.1
75.9
HLXEVV
94.1
49.0
88.6
66.8
89.7
75.3
75.0
68.6
85.0
61.5
76.9
63.1
86.0
62.7
+13.4
+13.8
+3.8
+7.6
-0.5
+4.7
+0.7
+5.0
+2.9
+12.0
+1.0
+6,9
+4.4
+9.0
Table 3: Performance on LongBench with -istral-7B-Insttuuct-
and Llama-3.1-8B-
Instruct. "Full KV" refers to caching all KV pairs of the LLM (upper bound).
Information Localization
Information Aggregation
Methods
Single-Doe QA
Multi-Doe QA
Summarization
Few shot
Syuthetic
Code
Mistral-7B I1B--rsttuu---01.
Pull KV
26.81
33.19
49.25
43.02
27.12
18.78
32.80
24.16
27.02
71.00
36.23
42.64
2.75
86.98
55.09
53,01
42.49
KV Carhe Budget = 1024
SnapKV
24.08
50.24
49.03
41.45
27.11
18:26
25.69
23.87
25.97
68.00
56. 25
42: 12.30
2.82
87.93
54.95
52.00
41.30
25.55
31.04
48.19
41.31
27.18
19.24
26.98
23,88
26.74
70.00
$6.46
43.77
2.90
85.99 99
55.02
31.25
41.60
+0.57
-0.550
-1.84
-0:14
+11.07
+0.95
+1.29
+11.11
+1..77
+2.00
+0.21
41.47
+0.00
1.194
-0.07
.0.72
+0.30
AdaKV
25.15
36,60
49.06
40.93
26.92
18.81
25.
23.96
25.84
69.00
56.
43.01
2.85
88.68
55.
52.46
41.54
48.85
41.96
18:27
26.77
43.44
2,82
55.65
51.87
Budget
Snap&y
39.70
17.38
23.25
24.20
11.54
3.24
86.29
53.71
51.19
23.56
48.95
48.36
25.86
17
24.63
23,36
25.32
66.00
56.23
42:25
3,00
53.87
51.40
0.13
+0.45
0.20
-0.66
+0.43
0.04
-1.33
+0.08
+1.12
0.00
+0.06
+0.71
0.22
+1.37
-0.16
+0.21
+0.37
AdaKV
24.35
27:33
48.76
40.07
26.38
17.97
23.75
23.51
24.31
67.50
56.
42.53
3.06
86.65
$3.90
$1.57
40.50
24.26
28.39
48.90
40.86
26.33
17:00
24.63
23.30
25.41
69.00
86.51
42.0 67
3.67
86-44
54.46
51.89
40.51
0.39
+1.06
+0.14
+0.79
0:05
0.90
+0.90
0.19
+1.10
+1.50
+0.13
+0.14
+0.01
00.21
+0.56
+0.12
+0.33
LJama- 3.1 Listtrct
Full KV
30:22
45.37
55.80
55.97
45.00
31.26
35.12
25.38
27.20
72,50
91.64
43.57
9.41
99.50
62,88
56,43
49.20
KV Cache Budget
1924
SnapKy
45.17
9.44
55.82
30.65
8.96
Auudes
+0.11
AdaK'V
28.16
54.68
56.14
45.19
30
78.
72.50
91.72
42.45
8.74
56.51
27.98
44.28
55.03
56.03
45.58
30.55
29.06
24.58
26.70
73.50
91.42
43.37
9.46
99.50
62.65
36.97
49.37
-0.18
+0.30
+0.35
-0.11
+0.39
+0.25
+0.71
.0.22
+0.59
+0.00
-0.30
40.39
+0.72
+0.00
-0.29
+0.46
+0.10
KV Cache Hudget - 511
SnapKV
27.42
38.95
53,57
55.20
$1.68
29.75
25.55
24.31
24.28
64,50
92.35
41.04
9.98
99.50
62,50
51.93
46.53
26.76
41.77
53.77
55.19
44.72
30.02
26.03
24.28
25.27
69.00
91.44
42.24
9.95
99.50
61.84
55.17
47.37
-0.55
+2.83
-0.20
-0.01
+0.04
+0.27
-0.48
+0.07
+0.99
+4.550
-0.91
+1.20
+0.00
40.00
-0 06.66
+0.24
+0.54
AdaKV 25.96
52.82
54.55
43.83
30.43
25.
92.05
42.10
9.45
99.50
62.58
55.59
46.42
53.18
55.47
43.88
91.01
42.79
99.51
46.75
Suder
+0.17
+0.05
-1.83
0.04
+0.23
we further evaluate the fundamental GUI grounding capability on ScreenSpot-v2 using Qwen2.5- VL-7B-Instruct (Bai al., 2025b). Table shows that integrating MixKV with baseline importance- based compression methods yields significant GUI grounding performance improvements. Notably, with a budget of 128, SnapKV (Li et al., 2024b) improves average precision from 75.3% to 83.3%, achieving performance boost of +7.9%. This further validates the effectiveness of Mi XKV and its potential for edge-side deployment of GUI agent models.
Performance on Long-Contest Text Benchmarks. Table 3 evaluates the integration of MixKV Jiang et al. (2023) and Llama3. 1-8B- Instruct (Grattafiori et al., 2024) using LongBench (Bai et al., 2024), demonstrating its applicability to long-context text tasks LLMs. Overall, Mi XXV yields consistent gains in average performance, with more pronounced improvements under tighter KV budgets. We also observe intriguing patterns: in Information Aggregation tasks like Summarization, Mi XiXVV substantially enhances baselines by preserving diverse KV pairs, enabling better global information coverage essential for synthesis. However, in Information Localization tasks, occasional declines occur, likely because these require focused retrieval of local salient details, and introducing diversity may dilute attention in LLMs,
with importance-bart based methods on -
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation on MixKV metrics. Simp extrinsic importance only (baseline). WWhead, offline head weights (OCRBench-derived, sample-shared r.)) Whead, online head weights (per-sample r
Settingo
Swapky
MakY
SparshMMM
SuuRkV
MakY
SpanaMMM
Stagy
MAKY
SporahiMM
Stugly
AdaY
SparoMMM
Saup
Adak'
Spansr
ULWWNNoNT.Meeed/Rt Bedgri-14
Piooe woggoooeeeuuunee Mesicc
19.009
ASTe PESROW Milky Aci0gge
91.5
96.3
644
43.
45.4
5U5
0,516
2306
+1.5
11.
18.
18.6
16.
18000
10
Qven2-VI /TBittutinet KV Carhe
AEpOT Oefioon Anguraanc
61.9
15.5
753
67.9
+2.4
+0.8
07.6
07.5
76.9
90.5
70.6
774
81.4
0935
0300
63.5
07.3
06.1
76.8
71.1
991.
72.5
72.3
81.3
1022
2573
1.901
07.8
72.8
77A
$3.00
$3
90.99
11.00
25:39
21:55
20:47
22.07
22.07
22.07
22.07
Merning
Merring
17:65
7666
17:65
17:65
PVE
Seepitv
WedrepKY
ParavoRRy
PMDRV
AdaKY
PM
SparwelV
Mielpernatt
Figure 4: Efficiency comparisons of total latency and peak memory. For context length of 32,000, "Full KV" refers to caching the entire sequence, whereas KV compression strategies employ budget of 64. The upper part total time, while the lower part is peak memory.
where head-wise semantic redundancy is inherently lower than in LVLMs (Figure 1). This highlights the task-dependent benefits of balancing importance and diversity.
4.3 ABLATION STUDIES AND ANALYSIS
Ablation of Different Metrics in MixKV. Table 4 systematically evaluates three components: (a)
importance scores combining extrinsie (s(mp) and two intrinsic measures (s (KNorm) (VNorm) ); (b)
Smm
diversity scores (saw);; and (c) head-wise adaptive mixing (W/eeal), We select LALLVAANNeT-Mssse 7B and Wiven2-VL-7B with different architectures ensure generalizability.
For importance metrics, we evaluate combinations of extrinsic importance with intrinsic measures:
(VNorm) Results show that jointly assessing extrinsic importance (KNorm) and simp + Sup
Simp + imp of keys and intrinsic importance of values provides consistent performance gains across scenar- underperforms as KNorm focuses on key magnitude, potentially misaligning with value-driven attention dynamics in LVLMs and leading to suboptimal KV pair iin (V proves more effective, with VNorm better capturing value retention. Conversely, Simp + imp
(KNorm)
ios. However, Simp + Simp
contributions to multi-modal attention, enhancing task relevance and compression efficiency.
Beyond importance, validate the effects of mixing importance with diversity. Results demon- strate that relying solely on diversity (sdi)) is inadequate, leading to performance degradation due to the disruption of original semantic information. Furthermore, jointly incorporating importance and diversity through Simp + Sav achieves notable performance improvements across diverse models and henchmarks. We observe that applying P/1, computed offline on OCRBench (Liu et al., 2024c), to other samples and benchmarks (WWhhead (Smm (S + sai)), further enhances performance compared to the direct combination Simp Sdiv, it accounts for varying redundancy levels across heads and adaptively adjusts importance and diversity weights. Advancing this, computing F/, per sam-

---

Published as a conference paper at ICLR 2026
ple for 1yhhed(Samp Sav) yields additional performance gains, since this approach allows MixkV to adaptively tune importance and diversity weights based on the characteristics of each processed sample, optimizing the mixing effect for superior performance. Moreover, experiments reveal that the inference costs of Whhead (Simp + -div) and W (Simp + &div) remain comparable, as the low computational cost of r calculation (Equation 4) does not affect the inference efficiency.
Efficiency Analysis of MixKV. Figure 4 compares the inference latency and peak memory con- sumption of the base model (Qwen2-VL-7B-Instnu with full KV cache), various standalone KV cache compression baselines, and XiV framework applied on top of these baselines. As expected, all compression methods reduce both latency and memory compared to the full KV cache baseline, validating their role in enhancing LVLM inference efficiency, Crucially, integrating MixKV with existing baseline compression methods yields substantial performance gains without degrading their original efficiency. This is because the overhead introduced by MixKV, primarily the lightweight mixing operation, scales linearly with the KV sequence length T and remains neg- ligible (e.g., less than 1% increase in latency) compared to the cost of the underlying compression method and the overall generation process. Collectively, the findings in Table 4 and Figure 4 affirm that Mi xKV achieves an exceptional balance between task performance and inference efficiency. More additional efficiency analysis is presented in Appendix A.5.
CONCLUSION
In this work, we analyze KV pair characteristics in LVLMs and identify two critical distinctions: LVLMs exhibit significantly higher semantic redundancy than LLMs, and attention heads demon- strate varying redundancy patterns. Based on these insights, we propose Mi XVV, which jointly optimizes importance and diversity for KV cache compression. MixKV quantifies semantic similar- ity within each attention head and adaptively balances importance and diversity weights, prioritizing diversity in high redundancy heads while emphasizing importance in low redundancy heads. Exten- sive experiments across multiple models and benchmarks confirm that Mi xKV consistently enhances existing compression methods while maintaining inference efficiency.
ACKNOWLEDGEMENTS
This research was supported by the Shanghai Science and Technology Program (Grant No. 25ZR1402278).
REFERENCES
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men. Lingchen Meng. Xuancheng Ren. Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qi- uyue Wang, Yuxuan Wang. Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Hu- men Zhong. Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report. arXiv preprint 1 2025a.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025b.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long con- text understanding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 3119-3137, 2024.
Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, Zhucun Xue, Yong Liu, and Xiang Bai. Llava-kd: A framework of distilling multimodal large
10

---

Published as a conference paper at ICLR 2026
language models. In Proceedings the pp. 229-2392244 209,22492222 2025a.
the IEEE/CVF International Conference Computer Vision,
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. In Second Conference on Language Modeling, 2025b.
URL https::///// wwtp//eeeeeeeeeee////p/ .et. forumm ad=
Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et Eagle 2.5: Boosting long-context post-training for frontier vision-language models. arXiv preprint arXiv:2504. 152254444..2,,,, 2025.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and and-play inference acceleration for large vision- language models. In Proceedings of the European Conference on Computer Vision, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong. Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang. Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to gpt- 4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185- 24198, 2024d.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9313 99322, 2024.
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective 1_2 norm-based strategy for kv cache compression. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 18476-18499, 2024.
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient Ilm inference. In Proceedings of the Advances in Neural Information Processing Systems, 2025.
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning. In Proceedings of the International Conference on Learning Representations, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint 207.222407..1783 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First conference on language modeling. 2024.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang. Jiale Cheng, Ji Qi. Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints, pp. arXiv-2507, 2025.
11

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: wwtp:i..orrg/b///31111 06825.
Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. Infinipot-v: Memory- constrained kv cache compression for streaming video understanding. arXiv preprint arXiv:2506. 15745, 2025.
Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.0811,, 2025a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0m 240.:2408..3326, 2024a.
Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, et al. Mminference: Accelerating pre-filling for long-context visual language models via modality- permutation sparse attention. In Proceedings of the International Conference on Machine Learning. 2025b.
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before gen- cration. In Proceedings of the Advances in Neural Information Processing Systems, volume 37, pp. 22947-22970, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceed- ings of the Advances in Neural Information Processing Systems, volume 36, pp. 34892-34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26286-26296, 2024a.
Haotian Liu, Chunyuan Li. Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. LLaVA-NeXT: Improved reasoning, oer, and world knowledge, 2024b. URL https: httboo/////////.ee 2204 1001130-100 1a0--llaa-neex
//llava-vl. github.
Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. Video compression comman- der: Plug-and-play inference acceleration for video large language models. arXiv preprint arXiv:2505. /4454, 2025a.
Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Lin- feng Zhang, Siteng Huang, and Honggang Chen. Global compression commander: Plug-and- play inference acceleration for high-resolution large vision-language models. arXiv preprint 1rXiv:2501.05179, 2025b.
Xuyang Liu, Zichen Wen, Shaobo Wang, Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, et al. Shifting ai efficiency from model-centric to data-centric compression. arXiv preprint rrXiv:2505.19445 2025c.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):22010, 2024c.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the International Conference on Machine Learning, 2024d.
Zuyan Liu, Benlin Liu, Jiahui Wang, Yuhao Dong, Guangyi Chen, Yongming Rao, Ranjay Krishna, and Jiwen Lu. Efficient inference of vision instruction- Proceedings of the European Conference on Computer Vision, pp. 54-69. Springer, 2024e.
models with elastic cache. In
12

---

Published as a conference paper at ICLR 2026
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 2200-2209, 2021.
Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu. Video-xl-2: Towards very long- video understanding through task-aware kv sparsification. arXiv preprint arXiv:2506. 19225, 2025.
Kele Shao, Keda Tao, Can Qin. Haoxuan You, Yang Sui, and Huan Wang. Holitom: Holistic token merging for fast video large language models. arXiv preprint arXhv:2505.21334 2025a.
Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang. Haoxuan You, Can Qin, Yang Sui, and Huan Wang. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios. arXiv preprint arXiv:2507.20196. 2025b.
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
26160-26169, 2025.
Olcksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanprect Singh. Textcaps: a dataset for image captioning with reading comprehension. In Proceedings of the European Conference on Computer Vision, pp. 742-758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards VQA models that read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2317-8326, 2019.
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. Gui-g?: Gaussian reward modeling for gui grounding. arXiv preprint arxiv:2507.15846 2025a.
Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, al. A survey on (m) llm-based gui agents. arXiv preprint a50iv:2504. 13865, 2025b.
Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025a.
Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1. x-bit kv cache quantization for video large language models. arXiv preprint arXiv:2503. 2025b.
Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4065- 4078, 2024.
Hongyu Wang, Shuming Ma, Lingxiao Ma, Lei Wang, Wenhui Wang. Li Dong, Shaohan Huang, Huaijie Wang. Jilong Xue, Ruiping Wang, al. Bitnet: 1-bit pre-training for large language models. Journal of Machine Learning Research, 26(125): 2025a.
Jiahui Wang. Zuyan Liu, Yongming Rao, and Jiwen Lu. Sparsemm: Head sparsity emerges from visual concept responses in milms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025b.
13

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayi- heng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing vision-language model's perception of the world at any resolution. arXiv preprini arXiv:2409. 12191, 2024.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, al. Interni13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025c.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for gener- alist gui agents. arXiv preprint arXiv:2410.232188 2024.
Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. In Proceedings of the International Conference on Machine Learning, 2025.
Yang, Baosong Yang. Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https:// /ttts: org/ abs/ 2202240722 10671.
An Yang. Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 098888 2025a.
Sengiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer better but not necessary in vision language models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, arXiv preprint 2 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient gener- ative inference of large language models. Proceedings of the Advances in Neural Information Processing Systems, volume 36, pp. 34661-34710 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ADDITIONAL EXPLANATION OF FIGURE 1-3
Figure I: To investigate the semantic redundancy patterns in KV caches across model architectures, we visualize the head-wise average cosine similarity of key vectors in Qwen2-7B (Yang et al., 2024) (an LLM) and Qwen2-VL- VL-7BB (Wang et al., 2024) (an LVLM), using representative samples from LongBench (Bai ct al., 2024) and OCRBench (Liu et al., 2024c). This comparison reveals that LVLMs exhibit substantially higher intra-head semantic redundancy than LLMs.
Figure 2: To quantify the variability of redundancy across attention heads within LVLMs, we ran- domly select 100 samples from each benchmark and compute the average cosine similarity of key vectors per head in Qwen2-VL-7B (Wang et al., 2024) and -aaaV--eee--Mssrral (Liu et al., 2024b). The results demonstrate significant head-wise heterogeneity in semantic redundancy, moti- vating our adaptive compression strategy.
To qualitatively compare the semantic coverage of different KV selection strategies, we randomly select one sample from TexiVQA (Singh et al., 2019) and apply PCA to project the key vectors from representative attention head (layer 23, head in the LLM) into a two-dimensional space. This head exhibits moderate semantic redundancy, making it an ideal case to illustrate how different compression strategies balance information retention. We visualize the distributions of keys retained under three settings: full KV cache, SnapKV (Li et al., 2024b), and our Mi XKV. In Figure 3, Mi xKV preserves a broader and more diverse set of key vectors compared to SnapKV.
Figure 3:
A.2 MORE DISCUSSIONS ON REDUNDANCY DIFFERENCES
To further validate the two types of "redundancy differences" introduced in Section 1, Figure 5 visu- alizes the head-wise KV cache redundancy of Qwen2 (Yang al., 2024) and Qwen2-Vi (Wang et al., 2024) when processing pure-text and vision-language inputs. The pure-text and vision- language results are obtained by averaging over 100 samples randomly drawn from LongBench (Bai et al., 2024) and TextVQA (Singh et al., 2019), respectively. For pure-text inputs, Qwen2 and Qwen2 VL exhibit highly similar redundancy patterns across heads, suggesting that the architec- tural difference alone does not account for the redundancy gap we observe.
From this visualization, we obtain two key findings: (I) Vision-Language Redundancy Differ- ences. When VI. processes vision-language inputs, the semantic redundancy of its KV cache is substantially higher than for pure-text inputs. which is consistent with our analysis in the main text. Notably, for some heads (e.g., Layer 29, Heads O and 1), the redundancy on vision-language data is more than twice that on text-only data, reflecting the inherently redundant nature of visual signals, whereas textual tokens tend to be more semantically diverse. (II) Head-wise Redundancy Differences. For both pure-text and vision-languae redundancy levels, and their overall patterns are highly similar: a head that is relatively more redun- dant on text remains relatively more redundant on vision-language inputs. We hypothesize that this is because different heads focus on different types of information: some heads primarily attend to local patterns and therefore exhibit higher semantic redundancy, while others capture more global information and consequently show much lower redundancy.
data, different heads exhibit markedly different
A.3 DETAILED EXPERIMENT SETTINGS
Model Details
We introduce more details of LVLMs used for evaluation in the main text:
LLAVA-NeXT (Liu et al., 2024b) improves upon LLaVA (Liu et al., 2023; 2024a) by sup- porting higher resolutions (4 x more pixels) and multiple aspect ratios using AnyRes technique. It employs a simple linear projector to connect vision features to the LLM, en- abling efficient multi-modal processing with unified interleaving of visual and text tokens.
InternVL3 (Zhu et al., 2025) is an advanced vision- langualanguuag model in the InternVL se- ries (Chen et al., 2024c:b), following the "ViT-MLP-LLI multi-modal pre-training for superior performance in multi-modal tasks, with dynamic res- olution handling and efficient alignment between vision and language components.
architecture. It features native
15

---

Published as a conference paper at ICLR 2026
XX
121
13
241
141
SM
141
681
524
821
65
33
120
04
2/1
300
54
53
514
115
Laye mide
1.6
8.1
3
614
122
034
625
1.4
043
612
240
010
Hoad Indox
Hcod ndoex
Head Indox
(a) Qwen2 for Pure Text Data
(bb Qwen2-VL for Pure Text Data
(c) Qwen2-VL for Vision-Language Data
Figure 5: Visualization of KV cache redundancy of Qwen2 and Qwen2-VL, on different data
types. The number each head is the average cosine similarity of the KV cache within that head.
Qwen2-VL (Wang et al., 2024) introduces Naive Dynamic Resolution to adaptively con- vert frames of any resolution into visual tokens. It utilizes multi-modal Rotary Position Embedding (M-RoPE) within a unified image-and-vides processing paradigm, enabling the handling of long videos for high-quality QA, dialogue, and content creation.
Mistral-7B (Jiang et al., 2023) is a dense transformer-basee LLM with 7B parameters. adopts Grouped -ueury-ueeruuuu Attention (GQA) and Sliding Window Attention (SWA), improving inference efficiency while supporting it delivers competitive performance across reasoning, coding, and dialogue tasks.
long-contee cong-context understanding. Despite its compact size,
Llama (Grattafiori et al., 2024) represents the latest generation of Llama models, offer- ing a family of parameter scales (c.g.. 8B, 70B). It leverages large-scale pretraining with optimized data curation and advanced instruction tuning. resulting in strong performance on benchmarks covering reasoning, knowledge-intensive tasks, and multi-turn dialogue.
Benchmark Details
We provide detailed introductions of benchmarks used in the main text:
DocVQA (Mathew et al., 2021) is a visual question answering benchmark on document images, focusing extracting and reasoning over information from scanned documents.
OCRBench (Liu et al., 2024c) evaluates OCR abilities, testing models on diverse text extraction tasks from images with varying fonts, layouts, and noise levels.
TextVQA (Singh et al., 2019) requires models to read and reason about text images to answer questions, emphasizing multi-modal integration of visual and textual information.
ChariQA (Masry al., 2022) assesses visual question answering on charts and graphs, requiring models to interpret data visualizations and answer related questions accurately.
TextCaps (Sidorov et al., 2020) is captioning benchmark for text-containing images, focusing on descriptions that accurately incorporate and describe textual content in scenes.
ScreenSpot-v2 (Wu al., 2024) is GUI grounding benchmark that evaluates model's ability to locate and identify specific UI elements (e.g., icons, text buttons) within screen- shots across diverse platforms including mobile, desktop, and web interfaces.
LongBench (Bai et al., 2024) evaluates long-context language understanding, with tasks testing models' handling of extended sequences across reasoning and comprehension.
16

---

Published as a conference paper at ICLR 2026
Table Performance of integrating HeadKY into MixkV on LongBench and LooGLE bench- marks using
Methods
Single Dec Q4
Qusper
HotpotQA
Multi-De OA
ZWRRMNQA
Musique
Doc.AA
Retriesal
Dependency QA
Timillne
Computation
Mistral
42.77
18.75
Cache Budget = 1024
Headky
25.58
31.28
50.54
4061
22.57
18.80
32.45
11.99
14887
0.49
9.56
9.21
Mi
26.26
31:20
50.07
40.99
27,88
19.93
32.72
12.06
14:73
0.50
9.33
9.16
Aruciez
+0.38
0.08
0.47
+0.35
+0.31
+1.13
+0.27
+0.13
0.14
+0.01
0.23
0.05
KV
Cache Budget
128
HeadKV
24.34
36.60
48.55
40
25.97
15.
30.25
10.48
12.72
0.53
10.04
8.44
MEXXY
24.39
27.70
49.85
42.48
27.21
15.40
31,17
10.62
13,08
0.73
10.31
Asuder
40.14
Baseline Details We provide a detailed introduction to the baseline importance-based KV cache compression methods used in the main text:
SnapKV (Li et al., 2024b) clusters important KV positions based on attention patterns observed from an initial window of tokens, enabling efficient KV cache compression by retaining only the most relevant clusters while maintaining high generation quality and reducing memory usage during inference.
PyramidKV (Cai et al., 2025b) dynamically adjusts KV cache sizes across different LLM layers in pyramidal manner, allocating more KV cache budget to lower layers for foun- dational information and less to higher layers for refined processing, based on information priority to optimize compression and performance.
AdaKV (Feng et al., 2025) adaptively allocates eviction budgets across attention heads LLMs by evaluating head-specific contributions, providing a plug-and-play solution for KV cache compression that significantly reduces memory footprint while preserving model performance generative inference tasks.
SparseMM (Wang al., 2025b) exploits sparsity patterns in visual attention heads of multi-modal models, assigning asymmetric KV cache budgets based on head importance for visual tokens, enabling modality-aware compression that effectively reduces storage requirements in vision-language models without sacrificing accuracy.
KNorm (Devoto et al., 2024) compresses KV cache using the f norm of key embeddings, keeping low-norm keys that correlate with high attention scores.
VNorm (Kim et al., 2025) ranks tokens by the Eg norm of their value embeddings to pre- serve semantically salient information.
A.4 ADDITIONAL EXPERIMENTS
Performance of MixKV with HeadKY for Long-Context Understanding. Table 5 further applies HeadKV (Fu et al., 2025) within our MixKV framework to validate its generality. Experimental re- sults show that integrating HeadKV into MixKV can improve pure-text long-context understanding. In particular, under the extreme compression setting (i.e., 128), we observe consistent and substantial gains across all tasks. This suggests that, when the KV cache budget is highly con- strained, it is crucial to preserve KV entries that are both important and diverse in order to maintain the long-context understanding ability of the baseline models.
Performance of MixKV on Larger LVLMs for Multi-Modal Understanding. Table 6 reports the results of integrating Mi xKV with baseline KV cache compression methods on the larger LVLM intemVL3-338 (Zhu et al., 2025) to evaluate its effectiveness for multi-moda, understanding, Exper- imental results show that MixKV consistently improves all baseline methods across benchmarks and KV cache budgets, demonstrating its robustness on larger LVLMs and further highlighting its practical value for real-world deployment.
Performance of Mi xKV MoE-based LVLMs for Multi-Modal Understanding, Table 7 further presents the results of integrating MixKV with SnapKV (Li et al., 2024b) on the MoE-based based LVLM Qwen3-VLL-30--AN Instrrutt evaluate its performance recent MoE-style LVLM architectures.
17

---

Published as a conference paper at ICLR 2026
Table 6: Performance of Applying MixkV
Doc (%)
OCRBench (%)
TextVQA (%)
ChartQA (%)
TextCaps
Methods
128
64
128
128
64
128
128
64
Intern InternVL3-333 VL3-38B
Full KV
93.5
85.9
83.8
88.6
0.953
SnapKV
87.5
85.2
77.8
64.3
82.0
78.5
87.5
85.2
0.932
0.822
MixkV
92.1
86.9
79.3
65.8
82.8
79.4
88.2
85.8
0.959
0.859
Abaseline
+4.6
+1.7
+0.8
+0.9
+0.7
+0.6
+0.027
+0.037
AdaKV
92.0
87.6
79.6
67.8
82.0
79.3
87.4
85.3
0.940
0.841
+ MixkV
92.3
88.5
81.1
69.2
82.9
80.2
88.2
86.0
0.961
0.859
A Ahsseline
+0.3
+0.9
+1.5
+1.4
+0.9
+0.9
+0.88
+0.77
+0.021
+0.018
Table 7: Performance of Applying MixKV to Sweea--LL-3UD-AAB---nttu
Doc (%)
OCRBench (%)
TextVQA (%)
CharlQA (%)
TextCaps
Methods
128
64
128
64
128
64
128
64
128
64
Qwen VL-300-A3BB ---sttuctttt
Full KV
94.5
84.0
83.5
85.1
0.287
SnapKV 91.9
83.8
71.0
55.2
75.3
75.3
83.8
79.8
0.314
0,272
+ MixKV
93.2
86.2
80.7
68.8
80.8
79.7
84.5
80.8
0,411
0.349
Ataseline
+1.3
+2.4
+9.7
+13.6
+5.5
+4.4
+0.7
+1.0
+0.097
+0.077
Experimental results show that MixKV can significantly boost SnapKV across various benchmarks; in particular, on OCRBench, brings a 13.6% improvement under the strict Budget=64 setting. These results demonstrate the strong effectiveness of MixXV MoE-based LVLMs and further highlight its potential for improving the efficiency of future LVLM inference.
A.5 MORE EFFICIENCY ANALYSIS OF MT XKV
Figure 6 further compares the total inference latency and peak GPU memory consumption across different KV cache compression settings. We observe that integrating MixKV with baseline methods (e.g., SnapKV, AdaKV) incurs negligible overhead, typically less than 1 increase in latency and no measurable rise in peak memory. while consistently achieving the same level of computational efficiency as the underlying baselines. This confirms that Mi MixK'S xKV preserves the original inference speed and memory footprint of the compression method it enhances, making it a truly plug-and-play framework.
A.6 ALGORITHM DETAILS OF MI XKV
Algorithm outlines the workflow of our MixXV framework, seamlessly integrated with SnapKV (Li et al., 2024b) to enhance its performance.
A.7 LIMITATIONS AND FUTURE WORK.
Our study is conducted on models up to the 8B parameter scale. Future work should validate the observed head-wise redundancy patterns and the effectiveness of MixKV generalize to significantly larger models (e.g., 70B+). This would be a crucial step for broader applicability.
A.8 THE USE OF LARGE LANGUAGE MODELS
In this study, we solely employ LLM-based language polishing refine sentence fluency and correct grammatical errors, without altering the technical content or experimental data of the paper.
18

---

Published as a conference paper at ICLR 2026
$3.80
53.48
53.43
53.83
TrAM Lateroy
27.99
20.31
27.53
27.59
Seaphiv
WedeoppiV
PyunidV
MaPymerikyy
AAWV
NaAdatl
Spmnali
WccGperaNV
22.07
22.87
Mamrrry (DB)
22.87
22.87
Momnnr
17:48
17:48
17.4
17.65
Fall
SeephV
Medroppy
KY
PynnsiiVV
MaPyrasedYy
Pull KV
MMNV
NaAday
MidkpasoW
$3.00
53.88
92.49
00.89
Toaal Literry
29.56
23.17
Tooll
20.05
22.07
22.87
Pesi Mamory (G8)
22.87
22.97
17.43
17.51
17.01
Pooi
F4ll KV
SeapRV
Wedropkv
Pat
PytesdiV
MaPyneveky
Full KV
AWVV
NaAdeV
Pull KV
SparwelM
ModigarsiN
Figure 6: More efficiency comparisons of total latency and peak memory. The top two rows of bars correspond to a budget of 128, while the bottom two rows correspond to a budget 256.
19

---

Published as a conference paper at ICLR 2026
Algorithm MixkV with SnapKV for KV Cache Compression
1: Input: Key-value pairs K) V query values Q, total memory budget B.
2: Output: Compressed KV pairs K, V%
3: Let B' = B |window| denote the adjusted budget for non-window KV pairs.
4: for each layer and head / do
5: Step 1: Compute Importance and Diversity Scores
6: Compute the intrinsic importance using VNorm, normalize, and scale:
V ll miny ( h.gll2
(12)
S
in scaled,
i= 1
max, V h.j (2) miny NJ (22)) +
gin
+ 6
norm
7: Compute the extrinsic importance average attention scores:
ex imp,
Σ
Attention Qj. K;),
i 1o
wwndow
JEwindow
8:
Combine importance scores:
Sump,i
imp
in
scaled
i to
9:
Normalize keys and compute diversity scores:
K
1
K! K hg
i
to
K
T
10: Step 2: Head-wise Adaptive Mixing
11:
Quantify redundancy and compute comprehensive scores:
T2
T
K
1=
K
T(T 1)
$/ min, (sddi)
8
div
scalod,
max (sdiv) min, ( + 8aiv +
Simp
i= to T
comp (1 F/)) Simp. P/.
div scaled,
i= 1 to
12: Select the top-B' B' KV pairs based comprehensive scores:
K V (OoB((K window], V (exclude window], (0:000 )
end for 14:
Return: Compressed KV cache C = [(K), ViJaa
20

---

Published as a conference paper at ICLR 2026
Full KY
Snapky
Snapky MixCV
Layer Head
(b) Layer Head 1
Full KV
Snupkt MioDY
(c) Layer 17 Head 0
(cl) Layer 18 Head 0
Full Snapky Mixit
Pull KV Snapky
(e) Layer Head
Layer 23 Head
Figure 7: More t-SNE visualization of KV cache distributions under different settings.
21


Reviews:
[{"review_id": "26F6gzGJA4", "reviewer": "Reviewer_nwar", "summary": "This paper proposes MixKV, a novel KV cache compression method that adaptively balances importance and diversity at the attention head level. MixKV quantifies semantic redundancy per head and mixes importance with diversity to select KV pairs for retention. Extensive experiments across multiple visual-text tasks and compression baselines show that MixKV consistently improves performance under tight memory budgets.", "questions": "None", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a thorough analysis of semantic redundancy in LVLMs, highlighting a previously underexplored limitation of importance-only KV compression. \n2. The adaptive mixing of importance and diversity per head is both intuitive and empirically validated.\n3. MixKV can be integrated with existing compression methods without modifying the underlying compression operator, making it practical for real-world deployment.", "weaknesses": "1. The paper does not include a direct comparison with closely related methods such as [HeadKV](https://arxiv.org/abs/2410.19258),  which would strengthen the empirical evaluation.\n2. The analysis of Cross-modality Redundancy Differences does not clearly illustrate the \"cross-modality\" aspect. From my understanding, it's more about *Two different modality*. \n3. While the paper discusses Head-wise Redundancy Differences, similar redundancy patterns may also exist in pure LLMs. The unique characteristics or observations specific to VLMs should be clarified and emphasized.\n4. In Section 4.2, it would be valuable to further discuss the observed patterns—specifically, why MixKV significantly improves baselines on Information Aggregation tasks but not on Information Localization tasks. Providing concrete examples would enhance the discussion.\n5. The definition of \"total latency\" in Figure 5 is unclear. Details such as prompt length, number of decoding steps, and how compression overhead is measured should be explicitly disclosed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UT2yiOBcj0", "reviewer": "Reviewer_eFa6", "summary": "This work introduces a novel KV cache compression method tailored for VLMs. The paper begins by presenting two core observations, supported by visualizations: (1) KV pairs in LVLMs exhibit substantially higher semantic redundancy than in text-only LLMs, and (2) KV pairs in VLMs show varying degrees of semantic redundancy across attention heads in the LLM. Motivated by these observations, the authors propose a new compression scheme named MixKV. The central idea is to perform head-wise KV cache compression by simultaneously considering both the semantic redundancy and the diversity of the tokens. The paper is evaluated across multiple distinct tasks and compared against several baselines, demonstrating consistent performance advantages under identical settings.\"", "questions": "1. What about more text tasks?\n2. There is still a noticeable performance degradation when compared to the full KV cache. Can the authors include a discussion comparing the proposed method with other KV cache optimization techniques, such as MLA or similar approaches?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.  The paper effectively identifies a key problem through insightful feature analysis and subsequently proposes a targeted solution. The motivation is well-grounded, the methodology is appropriate, and the results are solid and convincing.\n2.  Although the proposed KV cache compression scheme is designed for VLMs, its effectiveness is also validated on text-only tasks. This demonstrates the method's strong generalizability and versatility.\n3.  The experimental evaluation is thorough. The method demonstrates consistent performance gains across various tasks and when compared against multiple baseline methods.\n4.  The problem addressed by this paper is of high practical significance, and the proposed solution has strong potential for real-world application in resource-constrained environments.", "weaknesses": "1.  No significant flaws were identified.\n2.  However, the paper could be further strengthened by including an analysis of the persistent performance gap that remains when compared to the full KV cache.\n3.  Additionally, a broader discussion that horizontally situates the proposed method among other KV cache (or GPU memory) saving techniques would be valuable. While direct experimental comparisons are not strictly necessary, a qualitative discussion of the trade-offs relative to other approaches would enhance the paper's context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LMJsRneh0s", "reviewer": "Reviewer_vSGZ", "summary": "The paper introduces MixKV, a KV cache compression method for large vision-language models (LVLMs) that combines importance and diversity to optimize memory usage and maintain performance. By adapting to head-wise redundancy, MixKV improves compression efficiency and achieves significant performance gains on multiple benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The key advantage of MixKV is its comprehensive benchmarking across various tasks and models. It demonstrates consistent performance improvements across multiple multi-modal and text understanding benchmarks, including DocVQA, TextVQA, and LongBench, as well as GUI Grounding tasks.", "weaknesses": "1. I believe the paper spends unnecessary length discussing **modality-specific redundancy differences** and **head-wise redundancy differences**, as these concepts have already been well-established in previous works, such as **MMinference** and **VisionZip**. For instance, the **head-wise redundancy** can be directly observed in **MMinference's Fig. 1**, making it redundant to claim this in such detail. Additionally, I find it questionable to use **text on Qwen2** and **image on Qwen2-VL** to illustrate **modality-specific redundancy differences**. Normally, one would expect to examine both text and image together on a **Qwen2-VL** model to properly demonstrate this kind of redundancy, as that's where multi-modal data is processed simultaneously.\n\n2. MixKV essentially merges two well-established paradigms: **importance-based** selection (FastV,ZipVL) and **diversity-based** retention (VisionZip). This type of fusion has already been explored in various contexts, such as in **attention mechanisms** and **token selection** techniques. The novelty here lies mainly in **applying this fusion to KV cache compression**, rather than introducing a fundamentally new idea. \n\n3. The paper mainly tests on **images** or **multi-view data** and doesn't address the challenges of **long-context sequences**, like those found in **long video** tasks. Testing on a **20k context** in long video benchmarks would be essential to evaluate how well **MixKV** handles large, complex data with **long-range dependencies** and more diverse redundancy patterns, which might stress-test its memory compression capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ryd8SVi0dc", "reviewer": "Reviewer_nMmG", "summary": "This paper introduces MixKV, a simple, plug-and-play KV-cache compression method for (multi-modal) LLMs that mixes standard importance scores with a diversity metric at the per-head level. It estimates head-level redundancy via the off-diagonal average similarity of normalized keys and uses this statistic to adapt the diversity–importance mix, complementing attention- and value-norm–based importance signals. The approach is lightweight—requiring no architectural changes—and, across multiple LVLMs/LLMs under tight KV budgets, consistently outperforms importance-only baselines. Experiments on both multimodal understanding and language modeling tasks further demonstrate its effectiveness.", "questions": "1. The study is thorough at the 7B dense scale, but scalability to larger backbones and MoE architectures remains unclear. Results on a larger dense LVLM (e.g., 34B–72B) and on an MoE model such as Qwen/Qwen3-VL-30B-A3B-Instruct would help substantiate generalization. \n\n2. In Section 4.3 and Table 4, the “online head weights” setup is under-specified. Define what “online” means (e.g., per-token, no lookahead) and detail the procedure for computing and applying these weights—i.e., which signals are used and at what step/layer they are measured. \n\n3. The authors only report results on image understanding benchmarks. How does the proposed method perform on video understanding tasks, such as Video-MME [1]? Including evaluations on video benchmarks would help demonstrate the method’s generality across temporal visual–language settings. \n\n4. Table 3 presents results on long-context understanding tasks, but the reported LLMs are not specifically optimized for extended context lengths. To make the comparison more meaningful, it would be valuable to include models that have been explicitly tuned for long-context inference, such as Qwen/Qwen3-4B-Instruct-2507. This would better reflect the effectiveness of the proposed method under realistic long-context settings. \n\nReference: \n\n[1] Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to follow. It convincingly diagnoses the limitations of importance-only KV compression and motivates a diversity component, positioning the work clearly against prior baselines. The evaluation is comprehensive—spanning multiple models and benchmarks with budget sweeps and ablations (e.g., choice of intrinsic signal, effect of mixing)—and it substantiates the central claims.", "weaknesses": "The paper notes that LVLMs exhibit much higher key similarity. However, the proposed approach appears general and seemingly applicable to LLMs as well. Which components, if any, are specifically designed for LVLMs, and what prevents a straightforward application to text-only LLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)