Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models

## Abstract

Abstract Recent large vision-language models (LVLMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet the resulting key-value (KV) cache expansion creates a critical memory bottleneck that fundamentally limits deployment scalability.
While existing KV cache compression methods focus on retaining high-importance KV pairs to minimize storage, they often overlook the modality-specific semantic redundancy patterns that emerge distinctively in multi-modal KV caches.
In this work, we first analyze how, beyond simple importance, the KV cache in LVLMs exhibits varying levels of redundancy across attention heads.
We show that relying solely on importance can only cover a subset of the full KV cache information distribution, leading to potential loss of semantic coverage.
To address this, we propose MixKV , a novel method that mixes importance with diversity for optimized KV cache compression in LVLMs. MixKV adapts to head-wise semantic redundancy, selectively balancing diversity and importance when compressing KV pairs. Extensive experiments demonstrate that MixKV consistently enhances existing methods across multiple LVLMs. Under extreme compression (budget=64), MixKV improves baseline methods by an average of 5.1% across five multi-modal understanding benchmarks and achieves remarkable gains of 8.0% and 9.0% for SnapKV and AdaKV on GUI grounding tasks, all while maintaining comparable inference efficiency. Furthermore, MixKV extends seamlessly to LLMs with comparable performance gains. Our code is available at https://github.com/xuyang-liu16/MixKV .

## 1 Introduction

Large vision-language models (LVLMs) (Li et al., 2024a; Chen et al., 2024d) have achieved remarkable performance in multimodal understanding by effectively integrating visual information and user instructions into the input space of large language models (LLMs) (Grattafiori et al., 2024; Yang et al., 2025a). With the growing demand for understanding long-context visual inputs, including high-resolution images (Bai et al., 2025; Zhu et al., 2025) and long videos (Shu et al., 2025; Qin et al., 2025), LVLMs must process an increasing number of visual tokens. However, processing such long-context inputs generates numerous key-value (KV) pairs in the KV cache of LLMs, substantially increasing GPU memory consumption and degrading computational efficiency due to memory access latency and bandwidth constraints (Liu et al., 2025; Wan et al., 2024).

To address the KV storage overhead, two main approaches have emerged. Token compression methods (Yang et al., 2025b; Chen et al., 2024a) directly compress visual tokens to indirectly reduce KV cache storage, but often underperform in high-resolution fine-grained tasks like text understanding (Singh et al., 2019) and document processing (Mathew et al., 2021). More effective KV cache compression methods directly evict KV pairs in the LLM to minimize storage while preserving performance (Wan et al., 2024; Li et al., 2024b), thereby enhancing decoding efficiency and throughput. However, current KV cache compression methods for LVLMs predominantly rely on attention-based importance scores to decide which KV pairs to retain (Tao et al., 2025; Wang et al., 2025). While this strategy does reduce the KV cache size, it fails to consider the intrinsic semantic characteristics of KV pairs in multi-modal settings. To bridge this gap, we conduct a comprehensive analysis and identify two key characteristics of KV pairs in LVLMs:

(I) Cross-modality Redundancy Differences: Visual information in LVLMs contains significantly more semantic redundancy than textual information in LLMs. Images often contain repetitive visual elements (e.g., similar textures, repeated patterns), leading to higher semantic similarity among KV pairs during vision-language processing. Figure [1](https://arxiv.org/html/2510.20707v1#S1.F1) provides compelling evidence: (a) shows that Qwen2-VL exhibits much denser high-similarity regions compared to the more diverse patterns of Qwen2, while (b) reveals that keys in Qwen2 peak around 0.2-0.4 average similarity whereas Qwen2-VL keys peak around 0.6-0.8, a 2-3$\times$ increase. This demonstrates that KV pairs in LVLMs exhibit substantially higher semantic redundancy than in LLMs.

(II) Head-wise Redundancy Differences: Within LVLMs, different attention heads focus on distinct multi-modal aspects (Wang et al., 2025). Some heads capture global features with lower redundancy, while others focus on local details with higher semantic similarity. Figure [2](https://arxiv.org/html/2510.20707v1#S1.F2) illustrates this phenomenon across multiple tasks: for Qwen2-VL-7B, certain heads show extremely high average similarity exceeding 0.9, while other heads maintain relatively low similarity below 0.3. This pattern is consistent across different vision-language tasks, indicating that KV pairs in LVLMs show varying degrees of semantic redundancy across attention heads in the LLM.

Figure: Figure 1: Visualization of KV cache redundancy across different models. (a) presents the similarity of keys across the same layer and head, with Qwen2-VL (bottom) processing vision-language information and Qwen2 (top) handling pure text information. (b) quantifies the average similarity distribution of keys (top) and values (bottom) across all layers and heads for Qwen2 and Qwen2-VL.
Refer to caption: x1.png

Furthermore, our analysis reveals that importance-based compression methods fail to fully replicate the information distribution of the original KV cache, leading to potential information loss (Figure [3](https://arxiv.org/html/2510.20707v1#S3.F3)). Therefore, we argue that beyond importance, preserving diverse KV pairs at per-head granularity is essential for minimizing semantic redundancy while maintaining comprehensive information coverage. To this end, we propose MixKV, which adopts a principled “mixing importance with diversity” approach. Specifically, MixKV extends existing importance-based KV compression methods by incorporating head-wise semantic diversity evaluation. By independently measuring semantic similarity within each attention head, MixKV adaptively balances importance and diversity per head to achieve fine-grained joint optimization of KV cache compression in LVLMs.

MixKV is a plug-and-play framework that enhances existing KV compression methods with consistent performance gains, maintaining inference efficiency while better preserving the distributional properties of the original KV cache. In summary, the main contributions are as follows:

- 1.
Semantic Redundancy Analysis. We conduct in-depth analyses of KV caches in LVLMs, revealing substantial inherent semantic redundancy. Besides, we demonstrate that importance-based methods fail to preserve full KV distribution coverage, exposing fundamental limitations.
- 2.
Mixing Importance with Diversity. Based on our analysis, we propose MixKV, a head-wise adaptive mechanism that quantifies semantic redundancy to create principled weighting between importance and diversity scores for joint optimization of KV cache compression.
- 3.
Comprehensive Experimental Validation. Extensive experiments across diverse multi-modal and text benchmarks demonstrate that MixKV yields consistent performance improvements for existing importance-based compression methods while maintaining inference efficiency.

Figure: Figure 2: Visualization of KV cache redundancy across different heads in LLMs. (a) and (b) present the average cosine similarity across heads of LLaVA-NeXT-Mistral-7B and Qwen2-VL-7B within the LLMs, with brighter heads indicating greater similarity in semantic information.
Refer to caption: x2.png

## 2 Related Work

Large Vision-language Models. Current large vision-language models (LVLMs) integrate a vision encoder (i.e., ViT), a projector module, and a large language model (LLM) to enable multi-modal comprehension (Liu et al., 2023; 2024a). To meet the growing demand for high-resolution image understanding, recent LVLMs introduce higher-resolution inputs via dynamic cropping strategies, such as LLaVA-NeXT (Liu et al., 2024b) and InternVL series (Chen et al., 2024c; b), or native resolution processing like Qwen2-VL (Wang et al., 2024; Bai et al., 2025) and GLM-4.5V (Hong et al., 2025). Additionally, video large language models (VideoLLMs) such as LLaVA-Video (Zhang et al., 2024) and Video-XL-2 (Qin et al., 2025) process multi-frame videos with thousands of frames. This trend dramatically increases visual token counts, leading to substantial computational costs and GPU memory burdens due to the key-value (KV) cache in the attention mechanism.

KV Cache Compression. The KV cache stores computed key-value (KV) pairs during the LLM’s pre-filling phase to avoid redundant computations in decoding and enhance inference efficiency. However, long-context multi-modal inputs impose a significant GPU memory burden on the KV cache. To alleviate this, several works propose KV cache compression techniques, categorized as: (i) vision token compression that directly compresses vision tokens (Yang et al., 2025b; Chen et al., 2024a), and (ii) KV cache compression that compresses stored KV pairs during pre-filling (Zhang et al., 2023; Liu et al., 2024d). Current KV compression methods are mainly designed for LLMs, such as SnapKV (Li et al., 2024b), which clusters important KV positions using attention patterns from an observation window; KNorm (Devoto et al., 2024), which uses $\ell_{2}$ key norms to score and retain KV pairs with lower norms; and AdaKV (Feng et al., 2025), which adaptively allocates eviction budgets across attention heads. Methods specifically designed for LVLMs include InfiniPot-V (Kim et al., 2025), which employs Value Norm for KV pair selection, and SparseMM (Wang et al., 2025), which allocates asymmetric budgets across attention heads based on their importance and retains high-attention KV pairs from observation windows. Most existing compression methods follow the paradigm of retaining critical KV pairs and evicting less important ones.

Unlike prior works that focus primarily on importance-based selection, we identify a critical characteristic in LVLMs: heterogeneous head-wise redundancy, where KV caches exhibit varying degrees of semantic redundancy across attention heads (Figure [2](https://arxiv.org/html/2510.20707v1#S1.F2)), causing importance-only methods to retain KV pairs that fail to cover the full information spectrum in high-redundancy heads. This motivates us to jointly consider importance and diversity for more effective KV cache compression.

## 3 Methodology

### 3.1 Preliminary: Large Vision-language Models

LVLM Architecture. Contemporary large vision-language models (LVLMs) generally adopt a “ViT-Projector-LLM” architecture (Li et al., 2024a; Wang et al., 2024), which consists of three primary components. For an input image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ or video $\mathbf{V}=\{\mathbf{v}_{i}\}_{i=1}^{T}\in\mathbb{R}^{T\times H\times W\times 3}$: (i) The visual encoder (i.e., ViT) extracts visual features, yielding embeddings $\mathbf{E}\in\mathbb{R}^{N\times D}$ for images or $\mathbf{E}=\{\mathbf{e}_{i}\}_{i=1}^{T}\in\mathbb{R}^{T\times N\times D}$ for videos; (ii) A projection layer, often a two-layer MLP, maps these to vision tokens $\mathbf{F}^{v}\in\mathbb{R}^{M\times D^{\prime}}$ for images or $\mathbf{F}^{v}=\{\mathbf{f}_{i}^{v}\}_{i=1}^{T}\in\mathbb{R}^{T\times M\times D^{\prime}}$ for videos, with $M\leq N$; and (iii) The LLM processes the combined visual and text tokens $\mathbf{F}^{t}$ in two phases: During the pre-filling phase, it computes KV pairs for all input tokens ($\mathbf{F}^{v}$ and $\mathbf{F}^{t}$) and stores them in the KV cache to avoid redundant computations; in the decoding phase, it auto-regressively generates responses, leveraging the KV cache for efficient retrieval of prior KV pairs in attention mechanisms:

$$ $p\left(\mathbf{Y}\mid\mathbf{F}^{v},\mathbf{F}^{t}\right)=\prod_{j=1}^{L}p\left(\mathbf{y}_{j}\mid\mathbf{F}^{v},\mathbf{F}^{t},\mathbf{Y}_{1:j-1};\mathcal{C}\right),$ (1) $$

where $\mathbf{Y}=\{\mathbf{y}_{j}\}_{j=1}^{L}$ is the output sequence, and $\mathcal{C}$ denotes the KV cache. Thus, LVLMs achieve multi-modal understanding based on visual inputs and user instructions.

KV Cache Compression. Multi-modal long-context sequences result in numerous KV pairs, which lead to significant memory overhead. KV cache compression addresses this challenge by introducing a compression operator $\bm{\Phi}$, which selectively reduces the number of stored KV cache (Wang et al., 2025). Typically, $\bm{\Phi}$ involves an evaluation function $\mathcal{E}$ that assigns scores $s_{i}=\mathcal{E}(\mathbf{K}^{l}_{h,i},\mathbf{V}^{l}_{h,i})$ to each KV pair $i$, with compression based on these scores by retaining top-$b$ highest-scoring pairs given a budget $B$, such that for each layer $l$ and head $h$, KV pairs $\mathbf{K}^{l}_{h},\mathbf{V}^{l}_{h}\in\mathbb{R}^{T\times D}$ ($T$ is sequence length, $D$ dimension per head) are compressed into compact representations $\hat{\mathbf{K}}^{l}_{h},\hat{\mathbf{V}}^{l}_{h}=\text{TopB}(\mathbf{K}^{l}_{h},\mathbf{V}^{l}_{h},\{s_{i}\}_{i=1}^{T})$. KV cache compression targets KV tensors computed during pre-filling, reducing memory burden while supporting efficient attention in decoding.

### 3.2 Analysis of KV Pairs Characteristics

To optimize KV cache compression in LVLMs, we begin by analyzing key characteristics of the KV cache. An intuitive characteristic is importance, aimed at retaining KV pairs with greater significance while compressing those with lesser importance, thereby enabling efficient compression.

Importance Metrics. Existing methods evaluate KV pair importance from two perspectives, intrinsic and extrinsic, each employing distinct metrics to compute importance scores $s_{\text{imp}}$:

- •
Intrinsic Importance: Determined by inherent KV vector properties, including Key Norm (KNorm) (Devoto et al., 2024), which computes the $\ell_{2}$ norm of each key vector with its negative assigned as $s^{\text{in}}_{\text{imp},i}=-s^{\text{in (KNorm)}}_{\text{imp},i}$ for compression scoring, and Value Norm (VNorm) (Kim et al., 2025), which calculates the $\ell_{2}$ norm of each value vector, using it directly as $s^{\text{in}}_{\text{imp},i}=s^{\text{in (VNorm)}}_{\text{imp},i}$ for scoring. Compression retains the KV pairs with higher $s^{\text{in}}_{\text{imp},i}$, such that $\hat{\mathbf{K}}^{l}_{h},\hat{\mathbf{V}}^{l}_{h}=\text{TopB}(\mathbf{K}^{l}_{h},\mathbf{V}^{l}_{h},\{s^{\text{in}}_{\text{imp},i}\}_{i=1}^{T})$, where $s^{\text{in}}_{\text{imp},i}$ represents the intrinsic importance.
- •
Extrinsic Importance: Quantified by average attention scores from an observation window at the prompt end (default length 32) (Li et al., 2024b; Cai et al., 2025), reflecting instruction relevance. The score $s^{\text{ex}}_{\text{imp},i}=\frac{1}{|\text{window}|}\sum_{j\in\text{window}}\text{Attention}(\mathbf{Q}_{j},\mathbf{K}_{i})$ is computed from attention weights between query $\mathbf{Q}_{j}$ and key $\mathbf{K}_{i}$, balancing modalities effectively. This score serves as $s^{\text{ex}}_{\text{imp},i}$ for compression. Compression prioritizes pairs with higher $s^{\text{ex}}_{\text{imp},i}$, such that $\hat{\mathbf{K}}^{l}_{h},\hat{\mathbf{V}}^{l}_{h}=\text{TopB}(\mathbf{K}^{l}_{h},\mathbf{V}^{l}_{h},\{s^{\text{ex}}_{\text{imp},i}\}_{i=1}^{T})$, where $s^{\text{ex}}_{\text{imp},i}$ primarily reflects instruction relevance.

We argue that a comprehensive assessment of KV pair importance requires integrating both intrinsic and extrinsic perspectives. This ensures a balanced evaluation of inherent significance and instruction relevance. Specifically, the integrated importance score is computed as:

$$ $s_{\text{imp},i}=s^{\text{ex}}_{\text{imp},i}+s^{\text{in}}_{\text{imp},i}$ (2) $$

where we employ $s^{\text{in (VNorm)}}_{\text{imp},i}$ as the default intrinsic component. To ensure compatibility between VNorm and attention-based extrinsic importance, we normalize VNorm scores to $[0,1]$ and scale them to match attention score magnitudes: $s^{\text{in}}_{\text{scaled},i}=s^{\text{in}}_{\text{norm},i}\cdot\frac{\bar{s}^{\text{ex}}_{\text{imp}}}{\bar{s}^{\text{in}}_{\text{norm}}+\epsilon}$, where $\bar{s}^{\text{ex}}_{\text{imp}}$ and $\bar{s}^{\text{in}}_{\text{norm}}$ denote respective mean values. Comprehensive ablation studies in Section [4.3](https://arxiv.org/html/2510.20707v1#S4.SS3) Table [4](https://arxiv.org/html/2510.20707v1#S4.T4) validate the effectiveness of this design choice.
However, methods relying solely on importance suffer from a critical limitation: they preferentially retain semantically similar KV pairs, leading to significant redundancy in the compressed cache and loss of global semantic coverage.

Figure: Figure 3: t-SNE visualization of KV cache distributions under different settings. “Full KV” represents the original KV distribution of Qwen2-VL without compression, serving as the reference distribution.
Refer to caption: x3.png

Figure [3](https://arxiv.org/html/2510.20707v1#S3.F3) further presents this limitation, which performs a t-SNE visualization of KV cache distributions. We observe that methods relying only on importance, such as SnapKV (Li et al., 2024b) (blue stars), fail to adequately cover the full KV cache information. In Figure [3](https://arxiv.org/html/2510.20707v1#S3.F3), SnapKV primarily focuses on a small portion of the information, losing semantic coverage compared to the full KV distribution (light gray circles). This is because importance-based methods prioritize task-relevant, highly similar information, often neglecting the broader diversity of KV pairs. As a result, relying solely on importance introduces redundancy by retaining semantically similar KV pairs, which do not provide the full semantic richness of the KV cache.
Therefore, effective KV cache compression in LVLMs requires incorporating diversity to retain non-redundant KV pairs.
This enables KV cache compression methods to approximate the full original semantic distribution of the uncompressed KV cache more effectively.

Diversity Metrics. Beyond importance, semantic diversity serves as another crucial characteristic for effective KV cache compression in LVLMs. We focus on key diversity, as keys primarily govern the attention patterns and semantic focus within a head, making them the most direct levers for controlling information redundancy. To quantify diversity in a computationally efficient manner, we adopt the negative cosine similarity between each key and the global average key, as it serves as an intuitive proxy for capturing the breadth of the semantic distribution. For each layer $l$ and head $h$, we first normalize each key vector: $\hat{\mathbf{K}}^{l}_{h,i}=\frac{\mathbf{K}^{l}_{h,i}}{\|\mathbf{K}^{l}_{h,i}\|}$, where $\mathbf{K}^{l}_{h,i}\in\mathbb{R}^{D}$ represents the $i$-th key vector and $T$ is the sequence length. We then compute a single global key representation via averaging: $\hat{\bar{\mathbf{K}}}^{l}_{h}=\frac{1}{T}\sum_{i=1}^{T}\hat{\mathbf{K}}^{l}_{h,i}$. This ensures that the diversity scores, computed via cosine similarity, are obtained in linear time with respect to $T$. Specifically, the diversity score for each KV pair is $s^{\text{div}}_{i}=-\hat{\mathbf{K}}^{l}_{h,i}\cdot\hat{\bar{\mathbf{K}}}^{l}_{h}$. Here, higher scores indicate greater distinctiveness from the global pattern, as the negative cosine similarity encourages the retention of diverse information.

### 3.3 Head-wise Adaptive Mixing Mechanism

Section [3.2](https://arxiv.org/html/2510.20707v1#S3.SS2) analyzes importance metrics $s_{\text{imp}}$ and diversity metrics $s_{\text{div}}$, both critical for compression. A natural question arises: How can these complementary metrics be combined effectively?. While simply adding the importance and diversity scores ($s_{\text{imp}}+s_{\text{div}}$) offers simplicity, Figure [2](https://arxiv.org/html/2510.20707v1#S1.F2) shows that varying semantic similarity levels across attention heads make uniform mixing sub-optimal.

Our central insight is that heads exhibiting higher semantic redundancy should prioritize diversity preservation to prevent similar KV pair retention, while less redundant heads can emphasize importance-based selection. This motivates the head-wise adaptive weighting mechanism in MixKV.

Head-wise Redundancy Quantification. We develop a principled approach to quantify semantic redundancy within each attention head. For layer $l$ and head $h$, we employ the off-diagonal average similarity of normalized key vectors as our redundancy measure.

Using the normalized key matrix $\hat{\mathbf{K}}^{l}_{h}\in\mathbb{R}^{T\times D}$ from diversity computation, we construct the similarity matrix $\mathbf{R}^{l}_{h}=\hat{\mathbf{K}}^{l}_{h}(\hat{\mathbf{K}}^{l}_{h})^{T}\in\mathbb{R}^{T\times T}$. By exploiting the relationship between dot products and norms, the total similarity sum becomes:

$$ $\sum_{i,j=1}^{T}\mathbf{R}^{l}_{h,i,j}=\left(\sum_{i=1}^{T}\hat{\mathbf{K}}^{l}_{h,i}\right)\cdot\left(\sum_{j=1}^{T}\hat{\mathbf{K}}^{l}_{h,j}\right)=T^{2}\|\hat{\bar{\mathbf{K}}}^{l}_{h}\|_{2}^{2}$ (3) $$

where $\hat{\bar{\mathbf{K}}}^{l}_{h}=\frac{1}{T}\sum_{i=1}^{T}\hat{\mathbf{K}}^{l}_{h,i}$. Given that normalized vectors yield unit diagonal elements due to the normalization process, the off-diagonal average similarity is:

$$ $\bar{r}^{l}_{h}=\frac{T^{2}|\hat{\bar{\mathbf{K}}}^{l}_{h}|_{2}^{2}-T}{T(T-1)}.$ (4) $$

This formulation ensures that as $\bar{r}_{h}^{l}\to 1$ (high redundancy), diversity weight increases to prevent redundant retention, while $\bar{r}_{h}^{l}\to 0$ (low redundancy) prioritizes importance-based selection. This mathematical property positions $\bar{r}^{l}_{h}$ as an ideal adaptive weight for diversity scores.

Head-wise Adaptive Mixing. Based on the redundancy quantification, we develop the head-wise adaptive mixing function $W^{\text{head}}(\cdot)$. To ensure scale compatibility between importance and diversity scores, we first normalize diversity scores to $[0,1]$: $\tilde{s}^{\text{div}}_{i}=\frac{s^{\text{div}}_{i}-\min_{j}(s^{\text{div}}_{j})}{\max_{j}(s^{\text{div}}_{j})-\min_{j}(s^{\text{div}}_{j})+\epsilon}$, then scale them to match the magnitude of importance scores: $s^{\text{div}}_{\text{scaled},i}=\tilde{s}^{\text{div}}_{i}\cdot\frac{\bar{s}_{\text{imp}}}{\bar{\tilde{s}}_{\text{div}}+\epsilon}$.

Through this formulation, MixKV achieves adaptive adjustment: redundant heads ($\bar{r}^{l}_{h}\to 1$) emphasize diverse KV pairs, while less redundant heads ($\bar{r}^{l}_{h}\to 0$) prioritize importance. This head-wise adaptation ensures the compressed KV cache preserves both critical information and semantic diversity. KV compression is realized by selecting the top-$B$ pairs with the highest comprehensive scores: $\hat{\mathbf{K}}^{l}_{h},\hat{\mathbf{V}}^{l}_{h}=\text{TopB}(\mathbf{K}^{l}_{h},\mathbf{V}^{l}_{h},\{s^{\text{comp}}_{i}\}_{i=1}^{T})$. Notably, Figure [3](https://arxiv.org/html/2510.20707v1#S3.F3) demonstrates that the adaptive mixing strategy of MixKV enables SnapKV to leverage KV cache diversity, thereby capturing a broader range of information and encompassing a wider distribution of the full KV cache (highlighted in red circles). See Figure [6](https://arxiv.org/html/2510.20707v1#A1.F6) for more visualizations and Appendix [A.4](https://arxiv.org/html/2510.20707v1#A1.SS4) for the algorithm.

## 4 Experiments

### 4.1 Experimental Setting

Model Details. We evaluate MixKV across a diverse set of architectures to ensure generalizability: LLaVA-NeXT-Mistral-7B (Liu et al., 2024b), InternVL3-8B (Li et al., 2024a), and Qwen2-VL-7B-Instruct (Wang et al., 2024) for multi-modal understanding tasks; Qwen2.5-VL-7B-Instruct (Bai et al., 2025) for GUI grounding tasks; and Mistral-7B-Instruct-v0.2 (Jiang et al., 2023) and Llama-3.1-8B-Instruct (Grattafiori et al., 2024) for text-only evaluation.

Benchmark Details. We select a range of multi-modal understanding benchmarks and a comprehensive text understanding benchmark for evaluation. For image understanding, we include DocVQA (Mathew et al., 2021), OCRBench (Liu et al., 2024c), TextVQA (Singh et al., 2019), ChartQA (Masry et al., 2022), and TextCaps (Sidorov et al., 2020), along with ScreenSpot-v2 (Wu et al., 2024) for GUI grounding. For text understanding, we adopt LongBench (Bai et al., 2024).

**Table 1: Performance on multiple image understanding benchmarks with reduced KV cache budgets. Since SparseMM (Wang et al., 2025) does not provide head scores for InternVL3-8B (Zhu et al., 2025), we cannot reproduce their results on this model.**
| Methods | DocVQA (%) | OCRBench (%) | TextVQA (%) | ChartQA (%) | TextCaps |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 256 | 128 | 64 | 256 | 128 | 64 | 256 | 128 | 64 | 256 | 128 | 64 | 256 | 128 | 64 |  |
| LLaVA-NeXT-Mistral-7B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SnapKV | 59.7 | 55.2 | 47.3 | 45.0 | 39.0 | 31.9 | 63.5 | 61.0 | 57.1 | 50.2 | 47.5 | 42.7 | 0.650 | 0.558 | 0.444 |
| + MixKV | 61.7 | 58.1 | 48.8 | 49.9 | 44.7 | 36.1 | 65.2 | 64.3 | 60.1 | 50.8 | 47.7 | 43.6 | 0.708 | 0.659 | 0.514 |
| $\Delta_{\text{baseline}}$ | +2.0 | +2.9 | +1.5 | +4.9 | +5.7 | +4.2 | +1.7 | +3.3 | +3.0 | +0.6 | +0.2 | +0.9 | +0.058 | +0.101 | +0.070 |
| PyramidKV | 58.2 | 54.3 | 43.4 | 44.1 | 39.4 | 29.1 | 62.9 | 60.9 | 54.8 | 49.1 | 47.1 | 40.8 | 0.621 | 0.553 | 0.407 |
| + MixKV | 60.8 | 57.2 | 45.1 | 49.7 | 43.7 | 32.0 | 64.9 | 63.8 | 57.8 | 50.8 | 47.5 | 41.3 | 0.687 | 0.656 | 0.466 |
| $\Delta_{\text{baseline}}$ | +2.6 | +2.9 | +1.7 | +5.6 | +4.3 | +2.9 | +2.0 | +2.9 | +3.0 | +1.7 | +0.4 | +0.5 | +0.066 | +0.103 | +0.059 |
| AdaKV | 59.6 | 55.9 | 48.7 | 45.1 | 40.4 | 32.8 | 62.9 | 60.5 | 56.9 | 50.4 | 47.8 | 44.6 | 0.646 | 0.566 | 0.440 |
| + MixKV | 61.3 | 58.3 | 50.8 | 49.8 | 44.9 | 36.6 | 65.3 | 63.7 | 59.6 | 50.9 | 48.5 | 45.2 | 0.704 | 0.660 | 0.509 |
| $\Delta_{\text{baseline}}$ | +1.7 | +2.4 | +2.1 | +4.7 | +4.5 | +3.8 | +2.4 | +3.2 | +2.7 | +0.5 | +0.7 | +0.6 | +0.058 | +0.094 | +0.069 |
| SparseMM | 61.6 | 60.8 | 57.6 | 51.9 | 50.7 | 46.2 | 65.1 | 64.7 | 62.8 | 51.9 | 51.2 | 48.9 | 0.680 | 0.634 | 0.524 |
| + MixKV | 61.9 | 61.0 | 59.2 | 50.8 | 50.4 | 49.5 | 65.2 | 65.0 | 64.4 | 51.8 | 51.5 | 50.6 | 0.682 | 0.652 | 0.575 |
| $\Delta_{\text{baseline}}$ | +0.3 | +0.2 | +1.6 | -1.1 | -0.3 | +3.3 | +0.1 | +0.3 | +1.6 | -0.1 | +0.3 | +1.7 | +0.002 | +0.018 | +0.051 |
| InternVL3-8B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SnapKV | 89.2 | 85.4 | 75.7 | 80.6 | 69.0 | 53.1 | 80.4 | 78.2 | 71.9 | 86.2 | 84.6 | 79.8 | 1.009 | 0.901 | 0.734 |
| + MixKV | 89.4 | 86.2 | 76.3 | 81.9 | 71.1 | 52.3 | 80.9 | 78.8 | 72.9 | 86.3 | 84.8 | 80.7 | 1.029 | 0.949 | 0.753 |
| $\Delta_{\text{baseline}}$ | +0.2 | +0.8 | +0.6 | +1.3 | +2.1 | -0.8 | +0.5 | +0.6 | +1.0 | +0.1 | +0.2 | +0.9 | +0.020 | +0.048 | +0.019 |
| PyramidKV | 87.2 | 82.7 | 69.7 | 70.9 | 58.4 | 41.8 | 78.3 | 75.3 | 67.2 | 85.7 | 84.0 | 78.0 | 0.896 | 0.809 | 0.632 |
| + MixKV | 87.5 | 83.5 | 70.4 | 72.3 | 60.0 | 41.2 | 79.0 | 76.6 | 68.2 | 85.8 | 84.4 | 78.6 | 0.941 | 0.850 | 0.646 |
| $\Delta_{\text{baseline}}$ | +0.3 | +0.8 | +0.7 | +1.4 | +1.6 | -0.6 | +0.7 | +1.3 | +1.0 | +0.1 | +0.4 | +0.6 | +0.045 | +0.041 | +0.014 |
| AdaKV | 89.2 | 86.0 | 77.2 | 80.8 | 70.2 | 53.1 | 80.4 | 78.0 | 71.8 | 86.2 | 84.4 | 80.4 | 1.013 | 0.921 | 0.759 |
| + MixKV | 89.5 | 86.7 | 78.1 | 82.4 | 71.6 | 52.3 | 80.8 | 78.7 | 72.9 | 86.2 | 85.2 | 80.9 | 1.034 | 0.955 | 0.782 |
| $\Delta_{\text{baseline}}$ | +0.3 | +0.7 | +0.9 | +1.6 | +1.4 | -0.8 | +0.4 | +0.7 | +1.1 | +0.0 | +0.8 | +0.5 | +0.021 | +0.034 | +0.023 |
| Qwen2-VL-7B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SnapKV | 88.0 | 80.1 | 66.5 | 77.3 | 71.9 | 62.4 | 80.3 | 77.5 | 69.9 | 81.3 | 79.6 | 75.5 | 1.360 | 1.142 | 0.794 |
| + MixKV | 90.5 | 82.6 | 67.9 | 79.3 | 75.4 | 66.0 | 81.9 | 80.6 | 72.5 | 81.6 | 81.2 | 77.6 | 1.470 | 1.342 | 0.878 |
| $\Delta_{\text{baseline}}$ | +2.5 | +2.5 | +1.4 | +2.0 | +3.5 | +3.6 | +1.6 | +3.1 | +2.6 | +0.3 | +1.6 | +2.1 | +0.110 | +0.200 | +0.084 |
| PyramidKV | 81.7 | 74.0 | 59.9 | 74.5 | 67.9 | 56.8 | 78.3 | 74.6 | 65.3 | 81.1 | 79.2 | 73.5 | 1.115 | 0.951 | 0.569 |
| + MixKV | 84.0 | 76.3 | 60.8 | 76.6 | 72.6 | 58.4 | 80.4 | 77.1 | 67.0 | 81.3 | 80.7 | 75.5 | 1.348 | 1.119 | 0.633 |
| $\Delta_{\text{baseline}}$ | +2.3 | +2.3 | +0.9 | +2.1 | +4.7 | +1.6 | +2.1 | +2.5 | +1.7 | +0.2 | +1.5 | +2.0 | +0.233 | +0.168 | +0.064 |
| AdaKV | 87.4 | 81.2 | 67.1 | 77.8 | 71.0 | 62.1 | 79.9 | 77.0 | 70.3 | 80.8 | 79.6 | 75.9 | 1.345 | 1.146 | 0.775 |
| + MixKV | 90.3 | 82.1 | 67.8 | 79.3 | 74.7 | 65.5 | 81.8 | 79.6 | 71.2 | 81.5 | 80.9 | 77.4 | 1.448 | 1.275 | 0.878 |
| $\Delta_{\text{baseline}}$ | +2.9 | +0.9 | +0.7 | +1.5 | +3.7 | +3.4 | +1.9 | +2.6 | +0.9 | +0.7 | +1.3 | +1.5 | +0.103 | +0.129 | +0.103 |
| SparseMM | 93.5 | 91.5 | 84.9 | 81.2 | 79.0 | 74.3 | 82.0 | 81.6 | 77.3 | 82.0 | 81.5 | 80.1 | 1.482 | 1.430 | 1.038 |
| + MixKV | 93.8 | 92.7 | 86.4 | 82.0 | 81.0 | 77.1 | 82.0 | 82.0 | 80.9 | 81.6 | 81.8 | 81.4 | 1.480 | 1.459 | 1.303 |
| $\Delta_{\text{baseline}}$ | +0.3 | +1.2 | +1.5 | +0.8 | +2.0 | +2.8 | +0.0 | +0.4 | +3.6 | -0.4 | +0.3 | +1.3 | -0.002 | +0.029 | +0.265 |

**Table 2: Performance on ScreenSpot-v2 GUI grounding benchmark with Qwen2.5-VL-7B-Instruct. “Full KV” refers to caching all KV pairs of the LLM (upper bound).**
| Methods | Mobile Text | Mobile Icon/Widget | Desktop Text | Desktop Icon/Widget | Web Text | Web Icon/Widget | Average |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 128 | 64 | 128 | 64 | 128 | 64 | 128 | 64 | 128 | 64 | 128 | 64 | 128 | 64 |  |
| Qwen2.5-VL-7B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Full KV | 97.2 | 87.7 | 91.2 | 77.1 | 88.5 | 82.3 | 88.5 |  |  |  |  |  |  |  |
| SnapKV | 65.5 | 28.6 | 78.7 | 53.1 | 86.1 | 57.2 | 74.3 | 57.1 | 76.9 | 46.6 | 74.4 | 49.8 | 75.3 | 46.9 |
| + MixKV | 86.6 | 35.5 | 85.3 | 60.2 | 87.1 | 71.1 | 75.0 | 65.7 | 85.0 | 53.0 | 76.4 | 56.2 | 83.3 | 54.9 |
| $\Delta_{\text{baseline}}$ | +21.1 | +6.9 | +6.6 | +7.1 | +1.0 | +13.9 | +0.7 | +8.6 | +8.1 | +6.4 | +2.0 | +6.4 | +7.9 | +8.0 |
| PyramidKV | 45.5 | 11.0 | 62.1 | 34.1 | 82.0 | 33.0 | 75.0 | 47.9 | 69.2 | 20.5 | 71.4 | 24.1 | 65.6 | 26.1 |
| + MixKV | 64.1 | 15.9 | 74.4 | 42.2 | 87.1 | 41.8 | 74.3 | 47.1 | 76.9 | 24.8 | 71.9 | 27.1 | 74.1 | 31.1 |
| $\Delta_{\text{baseline}}$ | +18.6 | +4.9 | +12.3 | +8.1 | +5.1 | +8.8 | -0.7 | -0.8 | +7.7 | +4.3 | +0.5 | +3.0 | +8.5 | +5.0 |
| AdaKV | 80.7 | 35.2 | 84.8 | 59.2 | 90.2 | 70.6 | 74.3 | 63.6 | 82.1 | 49.6 | 75.9 | 56.2 | 81.6 | 53.7 |
| + MixKV | 94.1 | 49.0 | 88.6 | 66.8 | 89.7 | 75.3 | 75.0 | 68.6 | 85.0 | 61.5 | 76.9 | 63.1 | 86.0 | 62.7 |
| $\Delta_{\text{baseline}}$ | +13.4 | +13.8 | +3.8 | +7.6 | -0.5 | +4.7 | +0.7 | +5.0 | +2.9 | +12.0 | +1.0 | +6.9 | +4.4 | +9.0 |

Implementation Details. We integrate MixKV with various KV compression methods, including SnapKV (Li et al., 2024b), PyramidKV (Cai et al., 2025), AdaKV (Feng et al., 2025), and SparseMM (Wang et al., 2025), across different KV cache budgets. Details are in Appendix [A.2](https://arxiv.org/html/2510.20707v1#A1.SS2).

### 4.2 Main Results

Performance on Multi-modal Understanding Benchmarks. Table [1](https://arxiv.org/html/2510.20707v1#S4.T1) presents the integration of MixKV with baseline methods on various models and benchmarks, highlighting three key advantages: (i) Universal Improvements: MixKV enhances baselines across models, benchmarks, and budgets, confirming the necessity of mixing importance with diversity. (ii) Scalability to Compression Methods: Our approach benefits various paradigms from simple baselines like SnapKV to layer-wise or head-wise budget allocation methods PyramidKV and SparseMM, modifying only the evaluation function without adjusting the compression operator. (iii) Model Compatibility: MixKV efficiently analyzes head-wise redundancy per sample, enabling direct compatibility with existing LVLMs without offline statistics like SparseMM, enabling a plug-and-play implementation.

Performance on GUI Grounding Benchmarks. Recent advancements in LVLMs demonstrate their capability to understand GUI scenarios (Cheng et al., 2024; Tang et al., 2025b; a), which rely on edge-side comprehension and necessitate KV compression to reduce storage demands. To this end, we further evaluate the fundamental GUI grounding capability on ScreenSpot-v2 using Qwen2.5-VL-7B-Instruct (Bai et al., 2025). Table [2](https://arxiv.org/html/2510.20707v1#S4.T2) shows that integrating MixKV with baseline importance-based compression methods yields significant GUI grounding performance improvements. Notably, with a budget of 128, SnapKV (Li et al., 2024b) improves average precision from 75.3% to 83.3%, achieving a performance boost of +7.9%. This further validates the effectiveness of MixKV and its potential for edge-side deployment of GUI agent models.

Performance on Long-Context Text Benchmarks. Table [3](https://arxiv.org/html/2510.20707v1#S4.T3) evaluates the integration of MixKV with importance-based methods on Mistral-7B-Instruct-v0.2 Jiang et al. (2023) and Llama3.1-8B-Instruct (Grattafiori et al., 2024) using LongBench (Bai et al., 2024), demonstrating its applicability to long-context text tasks in LLMs. Overall, MixKV yields consistent gains in average performance, with more pronounced improvements under tighter KV budgets. We also observe intriguing patterns: in Information Aggregation tasks like Summarization, MixKV substantially enhances baselines by preserving diverse KV pairs, enabling better global information coverage essential for synthesis. However, in Information Localization tasks, occasional declines occur, likely because these require focused retrieval of local salient details, and introducing diversity may dilute attention in LLMs, where head-wise semantic redundancy is inherently lower than in LVLMs (Figure [1](https://arxiv.org/html/2510.20707v1#S1.F1)). This highlights the task-dependent benefits of balancing importance and diversity.

**Table 3: Performance on LongBench with Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct. “Full KV” refers to caching all KV pairs of the LLM (upper bound).**
| Methods | Information Localization | Information Aggregation | Avg. |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Single-Doc QA | Multi-Doc QA | Summarization | Few-shot | Synthetic | Code |  |  |  |  |  |  |  |  |  |  |  |  |
| NrtvQA | Qasper | MF-en | HotpotQA | 2WikiMQA | Musique | GovReport | QMSum | MultiNews | TREC | TriviaQA | SAMSum | PCount | PRe | Lcc | RB-P |  |  |
| Mistral-7B-Instruct-v0.2 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Full KV | 26.81 | 33.19 | 49.26 | 43.02 | 27.12 | 18.78 | 32.80 | 24.16 | 27.02 | 71.00 | 86.23 | 42.64 | 2.75 | 86.98 | 55.09 | 53.01 | 42.49 |
| KV Cache Budget = 1024 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SnapKV | 24.98 | 30.24 | 49.03 | 41.45 | 27.11 | 18.26 | 25.69 | 23.87 | 25.97 | 68.00 | 86.25 | 42.30 | 2.82 | 87.93 | 54.95 | 52.00 | 41.30 |
| + MixKV | 25.55 | 31.04 | 48.19 | 41.31 | 27.18 | 19.24 | 26.98 | 23.88 | 26.74 | 70.00 | 86.46 | 43.77 | 2.90 | 85.99 | 55.02 | 51.28 | 41.60 |
| $\Delta_{\text{baseline}}$ | +0.57 | +0.80 | -0.84 | -0.14 | +0.07 | +0.98 | +1.29 | +0.01 | +0.77 | +2.00 | +0.21 | +1.47 | +0.08 | -1.94 | +0.07 | -0.72 | +0.30 |
| AdaKV | 25.15 | 30.60 | 49.06 | 40.93 | 26.92 | 18.81 | 25.88 | 23.96 | 25.84 | 69.00 | 86.24 | 43.01 | 2.85 | 88.68 | 55.19 | 52.46 | 41.54 |
| + MixKV | 25.31 | 30.56 | 48.83 | 41.96 | 26.95 | 18.27 | 26.77 | 23.85 | 26.37 | 70.50 | 86.63 | 43.44 | 2.62 | 86.52 | 55.65 | 51.87 | 41.63 |
| $\Delta_{\text{baseline}}$ | +0.16 | -0.04 | -0.23 | +1.03 | +0.03 | -0.54 | +0.89 | -0.11 | +0.53 | +1.50 | +0.39 | +0.43 | -0.23 | -2.16 | +0.46 | -0.59 | +0.09 |
| KV Cache Budget = 512 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SnapKV | 23.69 | 27.71 | 49.16 | 39.70 | 25.44 | 17.38 | 23.31 | 23.28 | 24.20 | 66.00 | 86.17 | 41.54 | 3.24 | 86.29 | 53.71 | 51.19 | 40.13 |
| + MixKV | 23.56 | 28.19 | 48.96 | 40.36 | 25.86 | 17.34 | 24.63 | 23.36 | 25.32 | 66.00 | 86.23 | 42.25 | 3.02 | 87.66 | 53.87 | 51.40 | 40.50 |
| $\Delta_{\text{baseline}}$ | -0.13 | +0.48 | -0.20 | +0.66 | +0.42 | -0.04 | +1.32 | +0.08 | +1.12 | 0.00 | +0.06 | +0.71 | -0.22 | +1.37 | +0.16 | +0.21 | +0.37 |
| AdaKV | 24.35 | 27.33 | 48.76 | 40.07 | 26.38 | 17.97 | 23.73 | 23.51 | 24.31 | 67.50 | 86.38 | 42.53 | 3.06 | 86.65 | 53.90 | 51.57 | 40.50 |
| + MixKV | 24.26 | 28.39 | 48.90 | 40.86 | 26.33 | 17.07 | 24.63 | 23.32 | 25.41 | 69.00 | 86.51 | 42.67 | 3.07 | 86.44 | 54.46 | 51.69 | 40.81 |
| $\Delta_{\text{baseline}}$ | -0.09 | +1.06 | +0.14 | +0.79 | -0.05 | -0.90 | +0.90 | -0.19 | +1.10 | +1.50 | +0.13 | +0.14 | +0.01 | -0.21 | +0.56 | +0.12 | +0.31 |
| Llama-3.1-8B-Instruct |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Full KV | 30.22 | 45.37 | 55.80 | 55.97 | 45.00 | 31.26 | 35.12 | 25.38 | 27.20 | 72.50 | 91.64 | 43.57 | 9.41 | 99.50 | 62.88 | 56.43 | 49.20 |
| KV Cache Budget = 1024 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SnapKV | 27.10 | 43.91 | 55.07 | 55.60 | 45.17 | 30.47 | 27.84 | 24.44 | 25.75 | 69.00 | 91.89 | 42.69 | 9.44 | 99.50 | 62.49 | 56.30 | 48.86 |
| + MixKV | 27.50 | 44.19 | 55.42 | 55.82 | 45.40 | 30.65 | 28.83 | 24.75 | 26.26 | 70.00 | 91.62 | 42.88 | 8.96 | 99.50 | 62.69 | 56.41 | 49.30 |
| $\Delta_{\text{baseline}}$ | +0.40 | +0.28 | +0.35 | +0.22 | +0.23 | +0.18 | +0.99 | +0.31 | +0.51 | +1.00 | -0.27 | +0.19 | -0.48 | +0.00 | +0.20 | +0.11 | +0.44 |
| AdaKV | 28.16 | 43.98 | 54.68 | 56.14 | 45.19 | 30.30 | 28.35 | 24.80 | 26.11 | 72.50 | 91.72 | 42.48 | 8.74 | 99.50 | 62.94 | 56.51 | 49.27 |
| + MixKV | 27.98 | 44.28 | 55.03 | 56.03 | 45.58 | 30.55 | 29.06 | 24.58 | 26.70 | 72.50 | 91.42 | 43.37 | 9.46 | 99.50 | 62.65 | 56.97 | 49.37 |
| $\Delta_{\text{baseline}}$ | -0.18 | +0.30 | +0.35 | -0.11 | +0.39 | +0.25 | +0.71 | -0.22 | +0.59 | +0.00 | -0.30 | +0.89 | +0.72 | +0.00 | -0.29 | +0.46 | +0.10 |
| KV Cache Budget = 512 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SnapKV | 27.42 | 38.95 | 53.57 | 55.20 | 44.68 | 29.75 | 25.55 | 24.21 | 24.28 | 64.50 | 92.35 | 41.04 | 9.98 | 99.50 | 62.50 | 54.93 | 46.53 |
| + MixKV | 26.76 | 41.77 | 53.77 | 55.19 | 44.72 | 30.02 | 26.03 | 24.28 | 25.27 | 69.00 | 91.44 | 42.24 | 9.98 | 99.50 | 61.84 | 55.17 | 47.37 |
| $\Delta_{\text{baseline}}$ | -0.66 | +2.82 | +0.20 | -0.01 | +0.04 | +0.27 | +0.48 | +0.07 | +0.99 | +4.50 | -0.91 | +1.20 | +0.00 | +0.00 | -0.66 | +0.24 | +0.84 |
| AdaKV | 25.96 | 40.26 | 52.82 | 54.55 | 43.83 | 30.43 | 25.76 | 24.06 | 24.69 | 69.00 | 92.05 | 42.10 | 9.45 | 99.50 | 62.58 | 55.59 | 46.42 |
| + MixKV | 26.13 | 42.08 | 53.18 | 55.47 | 43.88 | 28.80 | 26.68 | 24.03 | 25.35 | 70.00 | 91.01 | 42.79 | 9.41 | 99.50 | 62.92 | 55.82 | 46.75 |
| $\Delta_{\text{baseline}}$ | +0.17 | +1.82 | +0.36 | +0.92 | +0.05 | -1.63 | +0.92 | -0.03 | +0.66 | +1.00 | -1.04 | +0.69 | -0.04 | +0.00 | +0.34 | +0.23 | +0.33 |

### 4.3 Ablation Studies and Analysis

Ablation of Different Metrics in MixKV. Table [4](https://arxiv.org/html/2510.20707v1#S4.T4) systematically evaluates three components: (a) importance scores combining extrinsic ($s^{\text{ex}}_{\text{imp}}$) and two intrinsic measures ($s^{\text{in (KNorm)}}_{\text{imp}}$, $s^{\text{in (VNorm)}}_{\text{imp}}$); (b) diversity scores ($s_{\text{div}}$); and (c) head-wise adaptive mixing ($W^{\text{head}}$). We select LLaVA-NeXT-Mistral-7B and Qwen2-VL-7B with different architectures to ensure generalizability.

For importance metrics, we evaluate combinations of extrinsic importance with intrinsic measures: $s^{\text{ex}}_{\text{imp}}+s^{\text{in (KNorm)}}_{\text{imp}}$ and $s^{\text{ex}}_{\text{imp}}+s^{\text{in (VNorm)}}_{\text{imp}}$. Results show that jointly assessing extrinsic importance of keys and intrinsic importance of values provides consistent performance gains across scenarios. However, $s^{\text{ex}}_{\text{imp}}+s^{\text{in (KNorm)}}_{\text{imp}}$ underperforms as KNorm focuses on key magnitude, potentially misaligning with value-driven attention dynamics in LVLMs and leading to suboptimal KV pair retention. Conversely, $s^{\text{ex}}_{\text{imp}}+s^{\text{in (VNorm)}}_{\text{imp}}$ proves more effective, with VNorm better capturing value contributions to multi-modal attention, enhancing task relevance and compression efficiency.

Beyond importance, we validate the effects of mixing importance with diversity. Results demonstrate that relying solely on diversity ($s_{\text{div}}$) is inadequate, leading to performance degradation due to the disruption of original semantic information. Furthermore, jointly incorporating importance and diversity through $s_{\text{imp}}+s_{\text{div}}$ achieves notable performance improvements across diverse models and benchmarks. We observe that applying $\bar{r}^{l}_{h}$, computed offline on OCRBench (Liu et al., 2024c), to other samples and benchmarks (${}^{\dagger}W^{\text{head}}(s_{\text{imp}}+s_{\text{div}})$), further enhances performance compared to the direct combination $s_{\text{imp}}+s_{\text{div}}$, as it accounts for varying redundancy levels across heads and adaptively adjusts importance and diversity weights. Advancing this, computing $\bar{r}^{l}_{h}$ per sample for $W^{\text{head}}(s_{\text{imp}}+s_{\text{div}})$ yields additional performance gains, since this approach allows MixKV to adaptively tune importance and diversity weights based on the characteristics of each processed sample, optimizing the mixing effect for superior performance. Moreover, experiments reveal that the inference costs of $W^{\text{head}}(s_{\text{imp}}+s_{\text{div}})$ and ${}^{\dagger}W^{\text{head}}(s_{\text{imp}}+s_{\text{div}})$ remain comparable, as the low computational cost of $\bar{r}^{l}_{h}$ calculation (Equation [4](https://arxiv.org/html/2510.20707v1#S3.E4)) does not affect the inference efficiency.

**Table 4: Ablation on MixKV metrics. $s^{\text{ex}}_{\text{imp}}$: extrinsic importance only (baseline). ${}^{\dagger}W^{\text{head}}$: offline head weights (OCRBench-derived, sample-shared $\bar{r}^{l}_{h}$). $W^{\text{head}}$: online head weights (per-sample $\bar{r}^{l}_{h}$).**
| Settings | DocVQA (%) | OCRBench (%) | TextVQA (%) | ChartQA (%) | TextCaps |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SnapKV | AdaKV | SparseMM | SnapKV | AdaKV | SparseMM | SnapKV | AdaKV | SparseMM | SnapKV | AdaKV | SparseMM | SnapKV | AdaKV | SparseMM |  |
| LLaVA-NeXT-Mistral-7B, KV Cache Budget = 64 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Effects of Different Importance Metrics |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| $s^{\text{ex}}_{\text{imp}}$ (baseline) | 47.3 | 48.7 | 57.6 | 31.9 | 32.8 | 46.2 | 57.1 | 56.9 | 62.8 | 42.7 | 44.6 | 48.9 | 0.444 | 0.440 | 0.524 |
| $s^{\text{ex}}_{\text{imp}}+s^{\text{in (KNorm)}}_{\text{imp}}$ | 46.7 | 48.2 | 55.4 | 30.7 | 32.0 | 41.6 | 56.4 | 56.6 | 60.9 | 42.6 | 43.8 | 47.4 | 0.445 | 0.444 | 0.482 |
| $s^{\text{ex}}_{\text{imp}}+s^{\text{in (VNorm)}}_{\text{imp}}$ | 48.2 | 49.7 | 59.1 | 34.4 | 35.0 | 49.0 | 58.0 | 57.9 | 64.0 | 43.7 | 45.3 | 50.3 | 0.470 | 0.469 | 0.544 |
| $\Delta_{\text{baseline}}$ | +0.9 | +1.0 | +1.5 | +2.5 | +2.2 | +2.8 | +0.9 | +1.0 | +1.2 | +1.0 | +0.7 | +1.4 | +0.026 | +0.029 | +0.020 |
| Effects of Different Mixing Strategies |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| $s_{\text{div}}$ | 34.2 | 35.8 | 51.8 | 28.4 | 28.8 | 43.9 | 54.5 | 55.2 | 63.4 | 32.3 | 34.2 | 48.1 | 0.487 | 0.504 | 0.534 |
| $s_{\text{imp}}+s_{\text{div}}$ | 48.8 | 50.6 | 59.1 | 36.1 | 36.1 | 49.5 | 59.8 | 59.1 | 64.4 | 43.5 | 45.4 | 50.9 | 0.516 | 0.504 | 0.573 |
| $\Delta_{\text{baseline}}$ | +1.5 | +2.1 | +1.6 | +4.2 | +3.8 | +3.3 | +3.0 | +2.7 | +1.6 | +0.9 | +0.6 | +1.7 | +0.070 | +0.069 | +0.051 |
| Qwen2-VL-7B-Instruct, KV Cache Budget = 64 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Effects of Different Importance Metrics |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| $s^{\text{ex}}_{\text{imp}}$ (baseline) | 66.5 | 67.1 | 84.9 | 62.4 | 62.1 | 74.3 | 69.9 | 70.3 | 77.3 | 75.5 | 75.9 | 80.1 | 0.794 | 0.775 | 1.038 |
| $s^{\text{ex}}_{\text{imp}}+s^{\text{in (KNorm)}}_{\text{imp}}$ | 66.1 | 66.5 | 81.2 | 56.2 | 56.0 | 68.8 | 67.9 | 67.2 | 72.6 | 71.4 | 72.8 | 76.3 | 0.766 | 0.769 | 0.927 |
| $s^{\text{ex}}_{\text{imp}}+s^{\text{in (VNorm)}}_{\text{imp}}$ | 67.2 | 67.4 | 84.7 | 64.9 | 64.4 | 75.5 | 71.5 | 70.3 | 79.7 | 77.3 | 77.4 | 80.8 | 0.862 | 0.854 | 1.259 |
| $\Delta_{\text{baseline}}$ | +0.7 | +0.3 | -0.2 | +2.5 | +2.3 | +1.2 | +1.6 | +0.0 | +2.4 | +1.8 | +1.5 | +0.7 | +0.068 | +0.079 | +0.221 |
| Effects of Different Mixing Strategies |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| $s_{\text{div}}$ | 44.0 | 44.3 | 60.4 | 50.7 | 50.3 | 68.4 | 59.8 | 59.7 | 78.4 | 64.7 | 65.2 | 78.5 | 0.739 | 0.711 | 1.113 |
| $s_{\text{imp}}+s_{\text{div}}$ | 67.6 | 67.6 | 86.3 | 65.0 | 63.6 | 76.9 | 72.2 | 70.6 | 80.8 | 76.6 | 77.0 | 81.4 | 0.905 | 0.869 | 1.291 |
| ${}^{\dagger}W^{\text{head}}(s_{\text{imp}}+s_{\text{div}})$ | 67.7 | 67.7 | 86.2 | 66.1 | 65.2 | 76.8 | 72.4 | 71.1 | 80.9 | 77.5 | 77.3 | 81.1 | 0.922 | 0.873 | 1.301 |
| $W^{\text{head}}(s_{\text{imp}}+s_{\text{div}})$ | 67.9 | 67.8 | 86.4 | 66.0 | 65.5 | 77.1 | 72.5 | 71.2 | 80.9 | 77.6 | 77.4 | 81.5 | 0.916 | 0.879 | 1.303 |
| $\Delta_{\text{baseline}}$ | +1.4 | +0.7 | +1.5 | +3.6 | +3.4 | +2.8 | +2.6 | +0.9 | +3.6 | +2.1 | +1.5 | +1.4 | +0.122 | +0.104 | +0.265 |

Efficiency Analysis of MixKV. Figure [4](https://arxiv.org/html/2510.20707v1#S4.F4) compares the inference latency and peak memory consumption of the base model (Qwen2-VL-7B-Instruct with full KV cache), various standalone KV cache compression baselines, and our MixKV framework applied on top of these baselines. As expected, all compression methods reduce both latency and memory compared to the full KV cache baseline, validating their role in enhancing LVLM inference efficiency. Crucially, integrating MixKV with existing baseline compression methods yields substantial performance gains without degrading their original efficiency. This is because the overhead introduced by MixKV, primarily the lightweight mixing operation, scales linearly with the KV sequence length $T$ and remains negligible (e.g., less than 1% increase in latency) compared to the cost of the underlying compression method and the overall generation process. Collectively, the findings in Table [4](https://arxiv.org/html/2510.20707v1#S4.T4) and Figure [4](https://arxiv.org/html/2510.20707v1#S4.F4) affirm that MixKV achieves an exceptional balance between task performance and inference efficiency. More additional efficiency analysis is presented in Appendix [A.3](https://arxiv.org/html/2510.20707v1#A1.SS3).

Figure: Figure 4: Efficiency comparisons of total latency and peak memory. For a context length of 32,000, “Full KV” refers to caching the entire sequence, whereas KV compression strategies employ a budget of 64. The upper part is total time, while the lower part is peak memory.
Refer to caption: x4.png

## 5 Conclusion

In this work, we analyze KV pair characteristics in LVLMs and identify two critical distinctions: LVLMs exhibit significantly higher semantic redundancy than LLMs, and attention heads demonstrate varying redundancy patterns. Based on these insights, we propose MixKV, which jointly optimizes importance and diversity for KV cache compression. MixKV quantifies semantic similarity within each attention head and adaptively balances importance and diversity weights, prioritizing diversity in high redundancy heads while emphasizing importance in low redundancy heads. Extensive experiments across multiple models and benchmarks confirm that MixKV consistently enhances existing compression methods while maintaining inference efficiency.

## References

- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Bai et al. (2024)
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al.
Longbench: A bilingual, multitask benchmark for long context understanding.
In *Proceedings of the Annual Meeting of the Association for Computational Linguistics*, pp. 3119–3137, 2024.
- Cai et al. (2025)
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao.
PyramidKV: Dynamic KV cache compression based on pyramidal information funneling.
In *Second Conference on Language Modeling*, 2025.
URL [https://openreview.net/forum?id=ayi7qezU87](https://openreview.net/forum?id=ayi7qezU87).
- Chen et al. (2024a)
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang.
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.
In *Proceedings of the European Conference on Computer Vision*, 2024a.
- Chen et al. (2024b)
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.
*arXiv preprint arXiv:2412.05271*, 2024b.
- Chen et al. (2024c)
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang.
How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.
*arXiv preprint arXiv:2404.16821*, 2024c.
- Chen et al. (2024d)
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai.
InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 24185–24198, 2024d.
- Cheng et al. (2024)
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu.
Seeclick: Harnessing gui grounding for advanced visual gui agents.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 9313–9332, 2024.
- Devoto et al. (2024)
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini.
A simple and effective l_2 norm-based strategy for kv cache compression.
In *Proceedings of the Conference on Empirical Methods in Natural Language Processing*, pp. 18476–18499, 2024.
- Feng et al. (2025)
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou.
Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference.
In *Proceedings of the Advances in Neural Information Processing Systems*, 2025.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Hong et al. (2025)
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al.
Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.
*arXiv e-prints*, pp. arXiv–2507, 2025.
- Jiang et al. (2023)
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.
Mistral 7b, 2023.
URL [https://arxiv.org/abs/2310.06825](https://arxiv.org/abs/2310.06825).
- Kim et al. (2025)
Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang.
Infinipot-v: Memory-constrained kv cache compression for streaming video understanding.
*arXiv preprint arXiv:2506.15745*, 2025.
- Li et al. (2024a)
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li.
Llava-onevision: Easy visual task transfer.
*arXiv preprint arXiv:2408.03326*, 2024a.
- Li et al. (2024b)
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen.
Snapkv: Llm knows what you are looking for before generation.
In *Proceedings of the Advances in Neural Information Processing Systems*, volume 37, pp. 22947–22970, 2024b.
- Liu et al. (2023)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
In *Proceedings of the Advances in Neural Information Processing Systems*, volume 36, pp. 34892–34916, 2023.
- Liu et al. (2024a)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 26286–26296, 2024a.
- Liu et al. (2024b)
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
LLaVA-NeXT: Improved reasoning, ocr, and world knowledge, 2024b.
URL [https://llava-vl.github.io/blog/2024-01-30-llava-next/](https://llava-vl.github.io/blog/2024-01-30-llava-next/).
- Liu et al. (2025)
Xuyang Liu, Zichen Wen, Shaobo Wang, Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, et al.
Shifting ai efficiency from model-centric to data-centric compression.
*arXiv preprint arXiv:2505.19147*, 2025.
- Liu et al. (2024c)
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai.
Ocrbench: on the hidden mystery of ocr in large multimodal models.
*Science China Information Sciences*, 67(12):220102, 2024c.
- Liu et al. (2024d)
Zuyan Liu, Benlin Liu, Jiahui Wang, Yuhao Dong, Guangyi Chen, Yongming Rao, Ranjay Krishna, and Jiwen Lu.
Efficient inference of vision instruction-following models with elastic cache.
In *Proceedings of the European Conference on Computer Vision*, pp. 54–69. Springer, 2024d.
- Masry et al. (2022)
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.
Chartqa: A benchmark for question answering about charts with visual and logical reasoning.
*arXiv preprint arXiv:2203.10244*, 2022.
- Mathew et al. (2021)
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.
Docvqa: A dataset for vqa on document images.
In *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pp. 2200–2209, 2021.
- Qin et al. (2025)
Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu.
Video-xl-2: Towards very long-video understanding through task-aware kv sparsification.
*arXiv preprint arXiv:2506.19225*, 2025.
- Shu et al. (2025)
Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao.
Video-xl: Extra-long vision language model for hour-scale video understanding.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 26160–26169, 2025.
- Sidorov et al. (2020)
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.
Textcaps: a dataset for image captioning with reading comprehension.
In *Proceedings of the European Conference on Computer Vision*, pp. 742–758. Springer, 2020.
- Singh et al. (2019)
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.
Towards VQA models that can read.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 8317–8326, 2019.
- Tang et al. (2025a)
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al.
Gui-g^2: Gaussian reward modeling for gui grounding.
*arXiv preprint arXiv:2507.15846*, 2025a.
- Tang et al. (2025b)
Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, et al.
A survey on (m) llm-based gui agents.
*arXiv preprint arXiv:2504.13865*, 2025b.
- Tao et al. (2025)
Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang.
Dycoke: Dynamic compression of tokens for fast video large language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2025.
- Wan et al. (2024)
Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan.
Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference.
In *Findings of the Association for Computational Linguistics: EMNLP 2024*, pp. 4065–4078, 2024.
- Wang et al. (2025)
Jiahui Wang, Zuyan Liu, Yongming Rao, and Jiwen Lu.
Sparsemm: Head sparsity emerges from visual concept responses in mllms.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2025.
- Wang et al. (2024)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin.
Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Wu et al. (2024)
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al.
Os-atlas: A foundation action model for generalist gui agents.
*arXiv preprint arXiv:2410.23218*, 2024.
- Yang et al. (2024)
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan.
Qwen2 technical report, 2024.
URL [https://arxiv.org/abs/2407.10671](https://arxiv.org/abs/2407.10671).
- Yang et al. (2025a)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025a.
- Yang et al. (2025b)
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia.
Visionzip: Longer is better but not necessary in vision language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2025b.
- Zhang et al. (2024)
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li.
Video instruction tuning with synthetic data.
*arXiv preprint arXiv:2410.02713*, 2024.
- Zhang et al. (2023)
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al.
H2o: Heavy-hitter oracle for efficient generative inference of large language models.
In *Proceedings of the Advances in Neural Information Processing Systems*, volume 36, pp. 34661–34710, 2023.
- Zhu et al. (2025)
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
*arXiv preprint arXiv:2504.10479*, 2025.

## Appendix A Appendix

### A.1 Additional Explanation of Figure 1 - 3

Figure [1](https://arxiv.org/html/2510.20707v1#S1.F1): To investigate the semantic redundancy patterns in KV caches across model architectures, we visualize the head-wise average cosine similarity of key vectors in Qwen2-7B (Yang et al., 2024) (an LLM) and Qwen2-VL-7B (Wang et al., 2024) (an LVLM), using representative samples from LongBench (Bai et al., 2024) and OCRBench (Liu et al., 2024c). This comparison reveals that LVLMs exhibit substantially higher intra-head semantic redundancy than LLMs.

Figure [2](https://arxiv.org/html/2510.20707v1#S1.F2): To quantify the variability of redundancy across attention heads within LVLMs, we randomly select 100 samples from each benchmark and compute the average cosine similarity of key vectors per head in Qwen2-VL-7B (Wang et al., 2024) and LLaVA-NeXT-Mistral-7B (Liu et al., 2024b). The results demonstrate significant head-wise heterogeneity in semantic redundancy, motivating our adaptive compression strategy.

Figure [3](https://arxiv.org/html/2510.20707v1#S3.F3): To qualitatively compare the semantic coverage of different KV selection strategies, we randomly select one sample from TextVQA (Singh et al., 2019) and apply PCA to project the key vectors from a representative attention head (layer 23, head 3 in the LLM) into a two-dimensional space. This head exhibits moderate semantic redundancy, making it an ideal case to illustrate how different compression strategies balance information retention. We visualize the distributions of keys retained under three settings: full KV cache, SnapKV (Li et al., 2024b), and our MixKV. In Figure [3](https://arxiv.org/html/2510.20707v1#S3.F3), MixKV preserves a broader and more diverse set of key vectors compared to SnapKV.

### A.2 Detailed Experiment Settings

#### Model Details

We introduce more details of LVLMs used for evaluation in the main text:

- •
LLaVA-NeXT (Liu et al., 2024b) improves upon LLaVA (Liu et al., 2023; 2024a) by supporting higher resolutions (4 $\times$ more pixels) and multiple aspect ratios using an AnyRes technique. It employs a simple linear projector to connect vision features to the LLM, enabling efficient multi-modal processing with unified interleaving of visual and text tokens.
- •
InternVL3 (Zhu et al., 2025) is an advanced vision-language model in the InternVL series (Chen et al., 2024c; b), following the “ViT-MLP-LLM” architecture. It features native multi-modal pre-training for superior performance in multi-modal tasks, with dynamic resolution handling and efficient alignment between vision and language components.
- •
Qwen2-VL (Wang et al., 2024) introduces Naive Dynamic Resolution to adaptively convert frames of any resolution into visual tokens. It utilizes multi-modal Rotary Position Embedding (M-RoPE) within a unified image-and-video processing paradigm, enabling the handling of long videos for high-quality QA, dialogue, and content creation.
- •
Mistral-7B (Jiang et al., 2023) is a dense transformer-based LLM with 7B parameters. It adopts Grouped-Query Attention (GQA) and Sliding Window Attention (SWA), improving inference efficiency while supporting long-context understanding. Despite its compact size, it delivers competitive performance across reasoning, coding, and dialogue tasks.
- •
Llama 3 (Grattafiori et al., 2024) represents the latest generation of Llama models, offering a family of parameter scales (e.g., 8B, 70B). It leverages large-scale pretraining with optimized data curation and advanced instruction tuning, resulting in strong performance on benchmarks covering reasoning, knowledge-intensive tasks, and multi-turn dialogue.

#### Benchmark Details

We provide detailed introductions of benchmarks used in the main text:

- •
DocVQA (Mathew et al., 2021) is a visual question answering benchmark on document images, focusing on extracting and reasoning over information from scanned documents.
- •
OCRBench (Liu et al., 2024c) evaluates OCR abilities, testing models on diverse text extraction tasks from images with varying fonts, layouts, and noise levels.
- •
TextVQA (Singh et al., 2019) requires models to read and reason about text in images to answer questions, emphasizing multi-modal integration of visual and textual information.
- •
ChartQA (Masry et al., 2022) assesses visual question answering on charts and graphs, requiring models to interpret data visualizations and answer related questions accurately.
- •
TextCaps (Sidorov et al., 2020) is a captioning benchmark for text-containing images, focusing on descriptions that accurately incorporate and describe textual content in scenes.
- •
ScreenSpot-v2 (Wu et al., 2024) is a GUI grounding benchmark that evaluates a model’s ability to locate and identify specific UI elements (e.g., icons, text buttons) within screenshots across diverse platforms including mobile, desktop, and web interfaces.
- •
LongBench (Bai et al., 2024) evaluates long-context language understanding, with tasks testing models’ handling of extended sequences across reasoning and comprehension.

#### Baseline Details

We provide a detailed introduction to the baseline importance-based KV cache compression methods used in the main text:

- •
SnapKV (Li et al., 2024b) clusters important KV positions based on attention patterns observed from an initial window of tokens, enabling efficient KV cache compression by retaining only the most relevant clusters while maintaining high generation quality and reducing memory usage during inference.
- •
PyramidKV (Cai et al., 2025) dynamically adjusts KV cache sizes across different LLM layers in a pyramidal manner, allocating more KV cache budget to lower layers for foundational information and less to higher layers for refined processing, based on information priority to optimize compression and performance.
- •
AdaKV (Feng et al., 2025) adaptively allocates eviction budgets across attention heads of LLMs by evaluating head-specific contributions, providing a plug-and-play solution for KV cache compression that significantly reduces memory footprint while preserving model performance in generative inference tasks.
- •
SparseMM (Wang et al., 2025) exploits sparsity patterns in visual attention heads of multi-modal models, assigning asymmetric KV cache budgets based on head importance for visual tokens, enabling modality-aware compression that effectively reduces storage requirements in vision-language models without sacrificing accuracy.
- •
KNorm (Devoto et al., 2024) compresses KV cache using the $\ell_{2}$ norm of key embeddings, keeping low-norm keys that correlate with high attention scores.
- •
VNorm (Kim et al., 2025) ranks tokens by the $\ell_{2}$ norm of their value embeddings to preserve semantically salient information.

### A.3 More Efficiency Analysis of MixKV

Figure [5](https://arxiv.org/html/2510.20707v1#A1.F5) further compares the total inference latency and peak GPU memory consumption across different KV cache compression settings. We observe that integrating MixKV with baseline methods (e.g., SnapKV, AdaKV) incurs negligible overhead, typically less than 1% increase in latency and no measurable rise in peak memory, while consistently achieving the same level of computational efficiency as the underlying baselines. This confirms that MixKV preserves the original inference speed and memory footprint of the compression method it enhances, making it a truly plug-and-play efficiency-preserving framework.

### A.4 Algorithm Details of MixKV

Algorithm [1](https://arxiv.org/html/2510.20707v1#alg1) outlines the workflow of our MixKV framework, seamlessly integrated with SnapKV (Li et al., 2024b) to enhance its performance.

Figure: Figure 5: More efficiency comparisons of total latency and peak memory. The top two rows of bars correspond to a budget of 128, while the bottom two rows correspond to a budget of 256.
Refer to caption: x5.png

Figure: Algorithm 1 MixKV with SnapKV for KV Cache Compression

Figure: Figure 6: More t-SNE visualization of KV cache distributions under different settings.
Refer to caption: x6.png

### A.5 Limitations and Future Work.

Our study is conducted on models up to the 8B parameter scale. Future work should validate if the observed head-wise redundancy patterns and the effectiveness of MixKV generalize to significantly larger models (e.g., 70B+). This would be a crucial step for broader applicability.

### A.6 The Use of Large Language Models

In this study, we solely employ LLM-based language polishing to refine sentence fluency and correct grammatical errors, without altering the technical content or experimental data of the paper.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: None
soundness: 2
strengths: 1. The paper provides a thorough analysis of semantic redundancy in LVLMs, highlighting a previously underexplored limitation of importance-only KV compression. 
2. The adaptive mixing of importance and diversity per head is both intuitive and empirically validated.
3. MixKV can be integrated with existing compression methods without modifying the underlying compression operator, making it practical for real-world deployment.
summary: This paper proposes MixKV, a novel KV cache compression method that adaptively balances importance and diversity at the attention head level. MixKV quantifies semantic redundancy per head and mixes importance with diversity to select KV pairs for retention. Extensive experiments across multiple visual-text tasks and compression baselines show that MixKV consistently improves performance under tight memory budgets.
weaknesses: 1. The paper does not include a direct comparison with closely related methods such as [HeadKV](https://arxiv.org/abs/2410.19258),  which would strengthen the empirical evaluation.
2. The analysis of Cross-modality Redundancy Differences does not clearly illustrate the "cross-modality" aspect. From my understanding, it's more about *Two different modality*. 
3. While the paper discusses Head-wise Redundancy Differences, similar redundancy patterns may also exist in pure LLMs. The unique characteristics or observations specific to VLMs should be clarified and emphasized.
4. In Section 4.2, it would be valuable to further discuss the observed patterns—specifically, why MixKV significantly improves baselines on Information Aggregation tasks but not on Information Localization tasks. Providing concrete examples would enhance the discussion.
5. The definition of "total latency" in Figure 5 is unclear. Details such as prompt length, number of decoding steps, and how compression overhead is measured should be explicitly disclosed.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. What about more text tasks?
2. There is still a noticeable performance degradation when compared to the full KV cache. Can the authors include a discussion comparing the proposed method with other KV cache optimization techniques, such as MLA or similar approaches?
soundness: 4
strengths: 1.  The paper effectively identifies a key problem through insightful feature analysis and subsequently proposes a targeted solution. The motivation is well-grounded, the methodology is appropriate, and the results are solid and convincing.
2.  Although the proposed KV cache compression scheme is designed for VLMs, its effectiveness is also validated on text-only tasks. This demonstrates the method's strong generalizability and versatility.
3.  The experimental evaluation is thorough. The method demonstrates consistent performance gains across various tasks and when compared against multiple baseline methods.
4.  The problem addressed by this paper is of high practical significance, and the proposed solution has strong potential for real-world application in resource-constrained environments.
summary: This work introduces a novel KV cache compression method tailored for VLMs. The paper begins by presenting two core observations, supported by visualizations: (1) KV pairs in LVLMs exhibit substantially higher semantic redundancy than in text-only LLMs, and (2) KV pairs in VLMs show varying degrees of semantic redundancy across attention heads in the LLM. Motivated by these observations, the authors propose a new compression scheme named MixKV. The central idea is to perform head-wise KV cache compression by simultaneously considering both the semantic redundancy and the diversity of the tokens. The paper is evaluated across multiple distinct tasks and compared against several baselines, demonstrating consistent performance advantages under identical settings."
weaknesses: 1.  No significant flaws were identified.
2.  However, the paper could be further strengthened by including an analysis of the persistent performance gap that remains when compared to the full KV cache.
3.  Additionally, a broader discussion that horizontally situates the proposed method among other KV cache (or GPU memory) saving techniques would be valuable. While direct experimental comparisons are not strictly necessary, a qualitative discussion of the trade-offs relative to other approaches would enhance the paper's context.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: The key advantage of MixKV is its comprehensive benchmarking across various tasks and models. It demonstrates consistent performance improvements across multiple multi-modal and text understanding benchmarks, including DocVQA, TextVQA, and LongBench, as well as GUI Grounding tasks.
summary: The paper introduces MixKV, a KV cache compression method for large vision-language models (LVLMs) that combines importance and diversity to optimize memory usage and maintain performance. By adapting to head-wise redundancy, MixKV improves compression efficiency and achieves significant performance gains on multiple benchmarks.
weaknesses: 1. I believe the paper spends unnecessary length discussing **modality-specific redundancy differences** and **head-wise redundancy differences**, as these concepts have already been well-established in previous works, such as **MMinference** and **VisionZip**. For instance, the **head-wise redundancy** can be directly observed in **MMinference's Fig. 1**, making it redundant to claim this in such detail. Additionally, I find it questionable to use **text on Qwen2** and **image on Qwen2-VL** to illustrate **modality-specific redundancy differences**. Normally, one would expect to examine both text and image together on a **Qwen2-VL** model to properly demonstrate this kind of redundancy, as that's where multi-modal data is processed simultaneously.

2. MixKV essentially merges two well-established paradigms: **importance-based** selection (FastV,ZipVL) and **diversity-based** retention (VisionZip). This type of fusion has already been explored in various contexts, such as in **attention mechanisms** and **token selection** techniques. The novelty here lies mainly in **applying this fusion to KV cache compression**, rather than introducing a fundamentally new idea. 

3. The paper mainly tests on **images** or **multi-view data** and doesn't address the challenges of **long-context sequences**, like those found in **long video** tasks. Testing on a **20k context** in long video benchmarks would be essential to evaluate how well **MixKV** handles large, complex data with **long-range dependencies** and more diverse redundancy patterns, which might stress-test its memory compression capabilities.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The study is thorough at the 7B dense scale, but scalability to larger backbones and MoE architectures remains unclear. Results on a larger dense LVLM (e.g., 34B–72B) and on an MoE model such as Qwen/Qwen3-VL-30B-A3B-Instruct would help substantiate generalization. 

2. In Section 4.3 and Table 4, the “online head weights” setup is under-specified. Define what “online” means (e.g., per-token, no lookahead) and detail the procedure for computing and applying these weights—i.e., which signals are used and at what step/layer they are measured. 

3. The authors only report results on image understanding benchmarks. How does the proposed method perform on video understanding tasks, such as Video-MME [1]? Including evaluations on video benchmarks would help demonstrate the method’s generality across temporal visual–language settings. 

4. Table 3 presents results on long-context understanding tasks, but the reported LLMs are not specifically optimized for extended context lengths. To make the comparison more meaningful, it would be valuable to include models that have been explicitly tuned for long-context inference, such as Qwen/Qwen3-4B-Instruct-2507. This would better reflect the effectiveness of the proposed method under realistic long-context settings. 

Reference: 

[1] Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv 2024.
soundness: 3
strengths: The paper is well written and easy to follow. It convincingly diagnoses the limitations of importance-only KV compression and motivates a diversity component, positioning the work clearly against prior baselines. The evaluation is comprehensive—spanning multiple models and benchmarks with budget sweeps and ablations (e.g., choice of intrinsic signal, effect of mixing)—and it substantiates the central claims.
summary: This paper introduces MixKV, a simple, plug-and-play KV-cache compression method for (multi-modal) LLMs that mixes standard importance scores with a diversity metric at the per-head level. It estimates head-level redundancy via the off-diagonal average similarity of normalized keys and uses this statistic to adapt the diversity–importance mix, complementing attention- and value-norm–based importance signals. The approach is lightweight—requiring no architectural changes—and, across multiple LVLMs/LLMs under tight KV budgets, consistently outperforms importance-only baselines. Experiments on both multimodal understanding and language modeling tasks further demonstrate its effectiveness.
weaknesses: The paper notes that LVLMs exhibit much higher key similarity. However, the proposed approach appears general and seemingly applicable to LLMs as well. Which components, if any, are specifically designed for LVLMs, and what prevents a straightforward application to text-only LLMs?


Decision: Accept (Poster)