EffiVLM-Bench: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Visual-Languge Models
Authors: Zekun Wang, MingHua Ma, Zexin Wang, Rongchuan Mu, hongtao liu, liping shan, Ming Liu, Bing Qin
Accepted at: acl25

EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models

Zekun Wang1*, Minghua Ma1*, Zexin Wang1*, Rongchuan Mu1*
Liping Shan2, Ming Liu1,2, Bing Qin1,2
1Harbin Institute of Technology, Harbin, China
2Pengcheng Laboratory, Shenzhen, China.
3Du Xiaoman Science Technology Co., Ltd, Beijing, China
{zkwang,mhma,zxwang,rcmu,mliu,qinb}@ir.hit.edu.cn
Project Page: https://effivlm-bench.github.io/

Abstract
Large Vision-Language Models (LVLMs) have achieved remarkable success, yet their significant computational demands hinder practical deployment. While efforts to improve LVLM efficiency are growing, existing methods lack comprehensive evaluation across diverse backbones, benchmarks, and metrics. In this work, we systematically evaluate mainstream acceleration techniques for LVLMs, categorized into token and parameter compression. We introduce EffiVLM-BENCH, a unified framework for assessing not only absolute performance but also generalization and loyalty, while exploring Pareto-optimal trade-offs. Our extensive experiments and in-depth analyses offer insights into optimal strategies for accelerating LVLMs. We open-source code and recipes for EffiVLM-BENCH to foster future research.

1 Introduction
Large vision-language models (LVLMs) (OpenAI, 2024; Team et al., 2024; Lu et al., 2024a; Wang et al., 2024a; Team, 2025) have rapidly advanced, transforming multimodal AI and showing great potential for real-world applications (Anthropic, 2024; Xu et al., 2024a; OpenAI, 2025; Li et al., 2025). However, their remarkable capabilities are often overshadowed by massive computational and memory costs, severely hindering practical deployment. To this end, some studies propose more efficient architectures (Chu et al., 2023; Zhou et al., 2024; Yao et al., 2024; Liu et al., 2024d) or incorporate distillation (Shu et al., 2024; Cai et al., 2024a; Li et al., 2024d) to improve efficiency. But these typically demand full retraining, incurring substantial overhead (An et al., 2024).

As a result, there has been a growing focus on training-free acceleration methods for LVLMs, which are more economical and can be broadly classified into two representative categories: token compression, eliminating redundant tokens in inputs (Chen et al., 2024b; Zhang et al., 2024b) or KV cache (Wan et al., 2024; Tu et al., 2024), and parameter compression, which reduces parameter size by pruning (Sung et al., 2023) or quantization (Lin et al., 2024; Yu et al., 2025). However, the scope of performance evaluation for these methods remains limited in several key aspects: (1) Outdated Model Architectures: Evaluations are often stuck on outdated models like LLaVA (Liu et al., 2023) or LLaVA-v1.5 (Liu et al., 2024a), without considering state-of-the-art LVLMs with the dynamic resolution processing mechanism (Li et al., 2024b; Wang et al., 2024a; Chen et al., 2024d). (2) Limited Benchmarks: Assessments typically use general VQA tasks, neglecting more challenging benchmarks that require advanced capabilities such as OCR or long-context generation. (3) Narrow Evaluation Metrics: The focus is solely on absolute performance, overlooking other critical metrics such as generational quality and loyalty of compression methods. This narrow focus leaves a significant gap in understanding how these techniques generalize across broader scenarios, which severely limits their practical use. Furthermore, there is a lack of systematic exploration into the crucial trade-offs between performance and efficiency (actual inference time), which is essential for real-world LVLM deployment.

To address these limitations, we propose EffiVLM-BENCH, a unified evaluation framework for systematically assessing training-free acceleration methods of LVLMs. EffiVLM-BENCH spans a wide range of representative model architectures and tasks, employing comprehensive metrics for performance, generalization, loyalty, and efficiency. With EffiVLM-BENCH, we conduct a thorough comparison of mainstream token and parameter compression methods. We explore Pareto-optimal performance-efficiency trade-offs and offer a nuanced understanding of their strengths and limita-
tions. We hope our EffiVLM-BENCH can provide the research community with insightful perspectives on LVLM acceleration, paving the way for more effective and sustainable deployment.

2 Related Work

Recent advancements in LVLMs (OpenAI, 2024; Team et al., 2024; Li et al., 2024b; Wang et al., 2024a; Deitke et al., 2024; Chen et al., 2024d; Team, 2025) have led to significant improvements in various multimodal tasks. Despite their remarkable capabilities, the computational overhead remains a critical bottleneck of LVLMs for real-world deployment. Some LVLM studies introduce more efficient model architectures (Chu et al., 2023; Zhou et al., 2024; Yao et al., 2024; Chen et al., 2024a; Luo et al., 2024) or knowledge distillation (Wang et al., 2021, 2023a; Li et al., 2024d) to enhance efficiency. However, these methods typically require training from scratch or retraining, which incurs significant computational overhead. Thus, increasing attention has been paid to training-free acceleration methods for LVLMs, the focus of this paper, which can be broadly categorized into token compression and parameter compression.

Token Compression The lengthy input tokens of LVLM impose a substantial computational burden, mainly due to the quadratic scaling of computational costs with the number of input tokens in the attention. Some studies focus on directly token pruning, which prune the redundant visual tokens during the forward process (Li et al., 2024c; Wang et al., 2023b; Cha et al., 2024; Huang et al., 2024). As for training-free methods, LLaVA-PruMerge (Shang et al., 2024) first prunes uninformative tokens in visual encoder and then merges the remaining ones using KNN techniques. Yang et al. (2024) also dynamically prunes the tokens in the visual encoder and then merges the pruned tokens to the next layer, while Chen et al. (2024b) and Zhang et al. (2024b) leverage textual information to guide visual token pruning in the initial layers of LLMs. Another line of work mainly focuses on KV cache compression. The inference time during LVLM generation is memory-bound, primarily due to the substantial memory consumption of KV caches, particularly when processing high-resolution images. KV cache compression leverages attention sparsity to select fewer key-value pairs, thereby reducing memory overhead and enhancing inference speed. Recent efforts in KV cache compression have evolved from techniques for LLMs (Xiao et al., 2023; Zhang et al., 2023b; Li et al., 2024e; Ge et al., 2024; Cai et al., 2024b; Xu et al., 2024b) to methods tailored for LVLMs. VL-Cache (Tu et al., 2024) introduces a modality-aware strategy that dynamically allocates cache budgets across layers and incorporates a token scoring mechanism tailored to the unique roles of visual and textual tokens. Similarly, LOOK-M (Wan et al., 2024) adopts a look-once optimization that minimizes redundant cache entries through eviction and KV pair merging. Both methods show that targeted compression can significantly reduce memory usage and accelerate decoding in multi-modal long-context scenarios while maintaining performance.

Parameter Compression Besides reducing the lengthy inputs, compressing the large model parameter size also benefits the efficiency of LVLM. Weight pruning removes redundant parameters to reduce parameter size and computations while preserving accuracy (Sun et al., 2023; Frantar and Alistarh, 2023; Wang et al., 2023a; An et al., 2024; Wang et al., 2024b). Quantization(Frantar et al., 2022; Lin et al., 2024) converts full-precision weights and activations into lower-precision formats (e.g., int8 or int4). Most existing work focuses on compressing the parameters of the LLM backbone in LVLMs, while recent efforts have also begun exploring the compression of the visual encoder (Yu et al., 2025). In this work, we concentrate on reducing the parameters of LLMs, which is a key bottleneck of LVLM efficiency.

3 EffiVLM-BENCH

In this section, we introduce the details of EffiVLM-BENCH, including metrics (Section 3.1) and tasks with model architectures (Section 3.2).

3.1 Metrics

Let \( c \) denote the compression method, \( m \) the target model, and \( b \) the target benchmark. \( \mathbb{E}(\cdot) \) denotes the mean operation.

Performance For each method \( c \) on model \( m \), we define the overall performance \( \mathrm{OP}^{m,c} \) as:

\[
\mathrm{OP}^{m,c} = \sqrt{\frac{1}{B} \sum_{b=1}^{B} \mathbb{E} \left( \frac{EM_b^{m,c}}{EM_b^m} \right)^2 },
\]

where \( EM_b^{m,c} \) is the evaluation performance metric (e.g., accuracy) and \( EM_b^m \) represents the corre-
sponding metric for the original model. \( B \) denotes the number of benchmarks.

**Generalization** To evaluate the generalization of compression method \( c \), we define the metric \( \mathrm{OG}^c \) as the coefficient of variation of performance across benchmarks and models:

\[
\mathrm{OG}^c = \frac{\sigma_b \left( \mathbb{E}_m \left[ \frac{EM_{b,m}^{m,c}}{EM_{b,m}^m} \right] \right)}{\mathbb{E}_{b,m} \left[ \frac{EM_{b,m}^{m,c}}{EM_{b,m}^m} \right]}
\]

where \( \sigma_b(\cdot) \) represents standard deviation across benchmarks. A lower \( \mathrm{OG}^c \) signifies more consistent relative performance (better generalization), while a higher value suggests greater sensitivity to variations in models or benchmarks.

**Loyalty** Loyalty (\( \mathrm{OL}^c \)) measures how well a compression method \( c \) preserves an original model’s predictions (\( P_b^m \)) with its compressed version (\( P_b^{m,c} \)). Ideally, compression should maintain the original model’s behavior, avoiding new biases or unexpected performance alterations (Xu et al., 2021). Specifically, we define the loyalty metric as:

\[
\mathrm{OL}^c = \mathbb{E}_{b,m} \left[ \mathbb{I}(P_b^{m,c}, P_b^m) \right]
\]

where \( \mathbb{I}(P_1, P_2) \) is an agreement function between predictions. Higher \( \mathrm{OL}^c \) indicates better loyalty.

**Efficiency** Actual inference time is our efficiency measure for a compression method \( c \). Unlike FLOPs or parameter counts, it directly reflects real-world latency, which varies significantly with model architecture (e.g., depth vs. width) even for similar theoretical complexities. Specifically, for model \( m \) using compression method \( c \) on benchmark \( b \), we measure the speedup of inference time per fixed number of samples:

\[
\mathrm{OE}^c = \mathbb{E}_{b,m} \left[ \frac{T_b^m}{T_b^{m,c}} \right],
\]

where \( T^{*,c} \) and \( T^* \) represent the latency of compressed and original models, respectively. More details of EffiVLM-BENCH are shown in Appendix A.4.

3.2 Tasks and Models

EffiVLM-BENCH evaluates across 17 widely-used benchmarks, including DocVQA(Mathew et al., 2020), ChartQA(Masry et al., 2022), TextVQA(Singh et al., 2019), OCRBench(Liu et al., 2024c), AI2D(Kembhavi et al., 2016), GQA(Hudson and Manning, 2019), MMMU(Yue et al., 2024), MME(Fu et al., 2024), Real-worldQA(x.ai), MMStar(Chen et al., 2024c), MathVista(Lu et al., 2024b), LLaVA-Wilder(Li et al., 2024a), MMBench(Liu et al., 2024b), MMVet(Yu et al., 2023), ImageDC(Li et al., 2024a), MPDocVQA(Tito et al., 2022) and MovieChat(Song et al., 2023). These cover diverse tasks—from document understanding and chart interpretation to real-world QA—across single-image, multi-image, and video scenarios (details in Appendix A.1). EffiVLM-BENCH includes three frontier LVLMs: LLaVA-OneVision(OV)-7B (Li et al., 2024b), Qwen2-VL-7B (Wang et al., 2024a), and InternVL2.5-38B (Chen et al., 2024d), spanning diverse model sizes for comprehensive evaluation. Furthermore, the modular design of EffiVLM-BENCH allows straightforward extension to support new tasks and emerging LVLMs.

4 Results

4.1 Token Compression

**Setup** We evaluate token compression effectiveness by examining two mainstream approaches: (1) token pruning, including FastV (Chen et al., 2024b), VisionZip (Yang et al., 2024), and PruMerge+ (Shang et al., 2024), which eliminates redundant visual tokens. (2) KV cache compression, including streamingLLM (Xiao et al., 2023), H2O (Zhang et al., 2023b), SnapKV (Li et al., 2024e), PyramidKV (Cai et al., 2024b), LOOK-M (Wan et al., 2024), and VL-Cache (Tu et al., 2024). Notably, the last two methods are specifically designed for LVLMs. More implementation details are in Appendix A.3 and A.4. We focus on retention budgets up to 40%, as methods generally maintain original performance at higher budgets. Main results of token pruning and KV cache compression are shown in Table 1 and Figure 1. More results of KV cache compression are provided in Appendix A.5 and Table 15.

**Observation 1** Token compression performance is task-dependent and shows significant sensitivity to benchmark and model. Most methods are stable at higher budgets but degrade sharply at 1%, especially on tasks requiring fine-grained visual detail (e.g., OCRBench) or long outputs (e.g., LLaVA-Wilder, ImageDC). For token pruning on a 1% budget, pruning tokens within the visual encoder (e.g., VisionZip and PruMerge+) consistently
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Budgets</th>
    <th colspan="2">Methods</th>
    <th colspan="13">Benchmarks</th>
    <th>OP</th>
  </tr>
  <tr>
    <th>FastV</th>
    <th>VisionZip</th>
    <th>DnC-VQA</th>
    <th>CharQA</th>
    <th>TextQNA</th>
    <th>AID</th>
    <th>MMAU</th>
    <th>RecoMMendQA</th>
    <th>MMStar</th>
    <th>MathVista</th>
    <th>LLaVA-Wilder</th>
    <th>MMVet</th>
    <th>ImageQC</th>
    <th>LLaVA-OneVision-7B</th>
  </tr>
  <tr>
    <td rowspan="4">LLaVA-OneVision-7B</td>
    <td>1%</td>
    <td>84.10</td>
    <td>91.18</td>
    <td>66.35</td>
    <td>55.04</td>
    <td>40.89</td>
    <td>170.42</td>
    <td>36.86</td>
    <td>28.97</td>
    <td>36.40</td>
    <td>49.10</td>
    <td>27.69</td>
    <td>12.30</td>
    <td>20.05</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>10%</td>
    <td>84.14</td>
    <td>91.48</td>
    <td>66.11</td>
    <td>54.18</td>
    <td>41.11</td>
    <td>170.42</td>
    <td>36.86</td>
    <td>28.97</td>
    <td>36.40</td>
    <td>49.10</td>
    <td>27.69</td>
    <td>12.30</td>
    <td>20.05</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>40%</td>
    <td>74.88</td>
    <td>86.22</td>
    <td>51.63</td>
    <td>43.44</td>
    <td>16.92</td>
    <td>58.16</td>
    <td>42.81</td>
    <td>39.20</td>
    <td>63.60</td>
    <td>73.09</td>
    <td>34.50</td>
    <td>76.60</td>
    <td>76.75</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>100%</td>
    <td>48.16</td>
    <td>52.77</td>
    <td>19.01</td>
    <td>12.53</td>
    <td>4.53</td>
    <td>166.94</td>
    <td>53.86</td>
    <td>44.63</td>
    <td>40.30</td>
    <td>66.90</td>
    <td>70.12</td>
    <td>31.70</td>
    <td>77.75</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td rowspan="4">Owen2-VL-7B</td>
    <td>1%</td>
    <td>56.98</td>
    <td>72.36</td>
    <td>35.21</td>
    <td>27.97</td>
    <td>8.57</td>
    <td>11.91</td>
    <td>58.97</td>
    <td>45.45</td>
    <td>45.06</td>
    <td>71.80</td>
    <td>78.86</td>
    <td>36.60</td>
    <td>87.50</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>10%</td>
    <td>60.20</td>
    <td>72.48</td>
    <td>38.08</td>
    <td>26.63</td>
    <td>46.33</td>
    <td>195.72</td>
    <td>57.75</td>
    <td>53.91</td>
    <td>50.80</td>
    <td>81.22</td>
    <td>47.70</td>
    <td>36.35</td>
    <td>86.35</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>40%</td>
    <td>67.04</td>
    <td>68.21</td>
    <td>50.08</td>
    <td>38.84</td>
    <td>61.23</td>
    <td>46.11</td>
    <td>195.68</td>
    <td>63.01</td>
    <td>51.17</td>
    <td>52.60</td>
    <td>71.60</td>
    <td>80.43</td>
    <td>48.30</td>
    <td>87.55</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>100%</td>
    <td>49.10</td>
    <td>67.79</td>
    <td>38.2</td>
    <td>61.78</td>
    <td>45.55</td>
    <td>192.40</td>
    <td>64.70</td>
    <td>53.38</td>
    <td>48.00</td>
    <td>68.10</td>
    <td>80.88</td>
    <td>46.50</td>
    <td>85.55</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td rowspan="4">InternVL2.5-38B</td>
    <td>1%</td>
    <td>87.00</td>
    <td>74.79</td>
    <td>59.5</td>
    <td>89.96</td>
    <td>61.92</td>
    <td>45.44</td>
    <td>197.41</td>
    <td>65.88</td>
    <td>58.75</td>
    <td>58.20</td>
    <td>71.40</td>
    <td>83.12</td>
    <td>55.80</td>
    <td>87.25</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>10%</td>
    <td>12.24</td>
    <td>22.45</td>
    <td>6.52</td>
    <td>39.79</td>
    <td>44.67</td>
    <td>133.01</td>
    <td>41.18</td>
    <td>26.83</td>
    <td>33.70</td>
    <td>51.40</td>
    <td>21.91</td>
    <td>18.10</td>
    <td>24.60</td>
    <td>0.51</td>
  </tr>
  <tr>
    <td>40%</td>
    <td>58.00</td>
    <td>101.87</td>
    <td>51.62</td>
    <td>60.48</td>
    <td>51.45</td>
    <td>177.22</td>
    <td>56.86</td>
    <td>35.70</td>
    <td>39.50</td>
    <td>63.00</td>
    <td>56.67</td>
    <td>34.60</td>
    <td>63.10</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>100%</td>
    <td>81.56</td>
    <td>81.81</td>
    <td>81.02</td>
    <td>62.30</td>
    <td>50.77</td>
    <td>232.78</td>
    <td>66.53</td>
    <td>57.11</td>
    <td>58.30</td>
    <td>72.80</td>
    <td>78.45</td>
    <td>68.35</td>
    <td>1.00</td>
    <td></td>
  </tr>
</table>

Table 1: Main results of various visual token prune methods on different models and benchmarks. Bold denotes the best result under the same setting.

outperforms those pruning in the LLM backbone (e.g., FastV). On LLaVA-OV-7B, FastV’s relative performance plummets to 48%, whereas VisionZip retains 75% (Further analysis in Section 5.3.). For KV cache compression, retaining sufficient tokens is crucial for preserving fine-grained information, but sensitivity varies across architectures and budgets. Notably, LLaVA-OV-7B processes thousands of tokens, so even at a 1% budget, it retains more tokens (40.17) than Owen2-VL-7B (7.62) and InternVL2.5-38B (15.14), leading to a smaller performance drop. These findings underscore the need to tailor compression methods to model and task specifics, and to rigorously assess their cross-benchmark generalization.

Observation 2 KV cache compression outperforms token pruning in generalization and loyalty. Table 2 shows H2O and PyramidKV leading in overall performance across metrics at high (40%) and low (10%) budgets, respectively. KV cache compression methods generally exhibit superior generalization and loyalty compared to token pruning. Thus, they should be prioritized when these aspects are critical. Meanwhile, we observe that LOOK-M performs well at higher budgets, but it exhibits a significant drop at lower budgets, which aligns with results in Figure 1, highlighting its high sensitivity under extremely aggressive constraints(detailed analysis is presented in Section 5.4).

<table>
  <tr>
    <th>Methods</th>
    <th colspan="2">OG<sup>c</sup> ↓</th>
    <th colspan="2">OL<sup>c</sup> ↑</th>
  </tr>
  <tr>
    <th></th>
    <th>1%</th>
    <th>40%</th>
    <th>1%</th>
    <th>40%</th>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>49.76</td>
    <td>12.05</td>
    <td>33.26</td>
    <td>85.98</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>30.93</td>
    <td><b>2.05</b></td>
    <td>53.02</td>
    <td><b>94.57</b></td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>31.98</td>
    <td>2.34</td>
    <td>55.85</td>
    <td>93.31</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>29.76</b></td>
    <td>5.31</td>
    <td><b>59.52</b></td>
    <td>91.19</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>51.72</td>
    <td>3.46</td>
    <td>29.67</td>
    <td>93.43</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>43.25</td>
    <td>3.80</td>
    <td>55.00</td>
    <td>92.04</td>
  </tr>
  <tr>
    <td>FastV</td>
    <td>58.46</td>
    <td>11.87</td>
    <td>40.89</td>
    <td>80.57</td>
  </tr>
  <tr>
    <td>VisionZip</td>
    <td>31.32</td>
    <td>5.86</td>
    <td>59.24</td>
    <td>80.52</td>
  </tr>
  <tr>
    <td>PruMerge+</td>
    <td>34.62</td>
    <td>13.47</td>
    <td>56.85</td>
    <td>83.07</td>
  </tr>
</table>

Table 2: Overall performance of generalization and loyalty for different token compression methods, with generalization calculated across all benchmarks and loyalty on a subset(MathVista, LLaVA-Wilder, MMVet). Bold indicates the best under the same setting.

Observation 3 Selecting token pruning or KV cache compression based on task statistics to achieve better performance-efficiency trade-off. Actual inference time can be broken down into two components: time-to-first-token (TTFT), which reflects the prefill overhead, and decoding time, which measures the latency of subsequent token generation. Figure 2 illustrates the trade-offs between efficiency and performance for token compression methods. KV cache methods yield lim-
Figure 1: Performance comparison of KV cache compression methods across multiple benchmarks and models.

ited TTFT speedup as they recompute attention weights for token selection during prefill\footnote{When using mechanisms like Flash-Attention2 (Dao, 2024) that don’t provide full attention matrices for selection, recalculation becomes necessary.}. This computation overhead is budget-independent, keeping their TTFT speedup nearly constant (see vertical lines in left part of Figure 2). In contrast, token pruning removes visual tokens during prefill, drastically reducing TTFT (e.g., 3.2× speedup at 1% budget), making it ideal for short-response tasks like VQA. For instance, at comparable overall performance, H2O achieves only 0.65× TTFT speedup, while VisionZip delivers 2.29× TTFT speedup. For decoding latency (right), KV cache and token pruning methods show similar speedups under the same budget. However, KV cache methods generally outperform token pruning at low budgets for tasks requiring long outputs (e.g., LLaVA-Wilder, ImageDC). Conversely, for tasks involving high-resolution images, the fixed cost of computing attention matrices in KV cache compression leads to significant memory overhead, favoring token pruning in these scenarios.

Observation 4 Consistent performance trends of token compression across single-image, multi-images, and video tasks. We further evaluate token compression methods on multi-image and video tasks. Table 3 shows the results based on Qwen2-VL-7B. Under a 1% budget, SnapKV achieves the best performance in MP-DocVQA, while LOOK-M suffers a significant drop, consistent with the results in DocVQA. In MovieChat, VL-Cache outperforms other methods. Notably, at higher budgets, some compression methods, such as SnapKV and VL-Cache, can maintain the original model’s performance while achieving substantial speedups. These findings indicate that token compression remains consistently effective across single-image, multi-image, and video scenarios. Future work will explore more challenging tasks for further validation.

4.2 Parameter Compression

Setup Pruning and quantization are two mainstream approaches for compressing LVLMs. We evaluate three pruning methods: EcoFLAP (Sung et al., 2023), Wanda (Sun et al., 2023), and SparseGPT (Frantar and Alistarh, 2023); and two
Figure 2: Trade-off Analysis on Token Compression Methods between Efficiency and performance. Left: Trade-off between TTFT (time-to-first-token) and Performance across different methods. Right: Trade-off between Speedup and performance across different methods. All metrics are expressed as ratios relative to the original model.

<table>
  <tr>
    <th>Benchmarks</th>
    <th>Methods</th>
    <th>1%</th>
    <th>10%</th>
    <th>40%</th>
    <th>100%</th>
  </tr>
  <tr>
    <td rowspan="5">MP-DocVQA (Split:val) (Metric:ANLS)</td>
    <td>StreamingLLM</td>
    <td>36.74</td>
    <td>51.38</td>
    <td>69.04</td>
    <td>85.91</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>38.73</td>
    <td>63.02</td>
    <td>80.67</td>
    <td>85.91</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>69.39</b></td>
    <td>80.73</td>
    <td>84.11</td>
    <td>85.91</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>63.44</td>
    <td>75.93</td>
    <td>79.33</td>
    <td>85.91</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>18.38</td>
    <td>57.09</td>
    <td>80.69</td>
    <td>85.91</td>
  </tr>
  <tr>
    <td rowspan="5">MovieChat (Split:test) (Metric:Acc)</td>
    <td>StreamingLLM</td>
    <td>13.11</td>
    <td>34.63</td>
    <td>37.03</td>
    <td>37.73</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>14.23</td>
    <td><b>37.15</b></td>
    <td>37.19</td>
    <td>37.73</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>28.29</td>
    <td>36.56</td>
    <td>37.53</td>
    <td>37.73</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>28.26</td>
    <td><b>37.15</b></td>
    <td>36.62</td>
    <td>37.73</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>11.95</td>
    <td>36.49</td>
    <td>37.57</td>
    <td>37.73</td>
  </tr>
  <tr>
    <td rowspan="2">VL-Cache</td>
    <td>28.51</td>
    <td>35.99</td>
    <td><b>37.65</b></td>
    <td>37.73</td>
  </tr>
</table>

Table 3: Results of token compression methods on Qwen2-VL-7B evaluated with multi-image and video benchmarks.

quantization methods: AWQ (Lin et al., 2024) and GPTQ (Frantar et al., 2022). EcoFLAP is specifically designed for LVLMs, while the others are widely-used for general LLM compression. Details are shown in Appendix A.2 and A.3.

Observation 5 Parameter compression generally preserve performance more effectively than token compression. Even at higher compression ratios (50% or 2:4 sparsity), the overall performance remains relatively stable. We observe that quantization methods, such as AWQ, tend to preserve higher performance compared to pruning. However, since neither method shortens input token sequences, token compression remains essential for tasks involving very long inputs or high-resolution images. Importantly, the two types of compression are orthogonal that can be effectively combined. Crucially, these two compression types are orthogonal and can be effectively combined and more results are shown in Appendix A.7.

5 Discussion

5.1 Revisiting Layer Adaptive Mechanism

While Cai et al. (2024b) shows that layer-adaptive sparsity benefits KV cache compression in LLM, EFFIVLM-BENCH results reveal that PyramidKV underperforms SnapKV at low sparsity budgets despite using the same token selection metric, due to its layer-adaptive strategy. This suggests that layer-adaptive sparsity is not always advantageous in LVLM compression, especially under low sparsity. To further probe this, we analyze VL-Cache, a layer-adaptive KV cache compression method proposed for LVLMs. As shown in Figure 3 and Appendix A.10: At 5% budget, the 0-th layer has nearly 7× of the average budget. This aggressive front-loading reduces budgets for subsequent layers, effectively starving them (less than the average). In addition, visualizations of the tokens selected in the 0-th layer (shown in Figure 8a and Appendix A.11) reveal that most of the tokens chosen are irrelevant. These findings suggest that reducing the budget of early layers while increasing the average allocation for later layers may lead to better performance. Based on the VL-Cache layer adaptive strategy, we propose a hybrid allocation strategy: a portion (40% or 80%) of the total bud-
<table>
  <tr>
    <th>Settings</th>
    <th>Methods</th>
    <th colspan="18">OP</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>DoxVQA</td>
    <td>CharVQA</td>
    <td>TextVQA</td>
    <td>OCRbench</td>
    <td>AIDD</td>
    <td>GQA</td>
    <td>MNLI</td>
    <td>MME</td>
    <td>RedwoodQA</td>
    <td>MMStar</td>
    <td>MultiVista</td>
    <td>LLaVA-Wilder</td>
    <td>LLaVA-Bench</td>
    <td>MMNet</td>
    <td>ImageDC</td>
    <td></td>
  </tr>
  <tr>
    <td>100%</td>
    <td>Original</td>
    <td>87</td>
    <td>80.00</td>
    <td>74.79</td>
    <td>595</td>
    <td>89.96</td>
    <td>61.92</td>
    <td>45.44</td>
    <td>1974.1</td>
    <td>65.88</td>
    <td>58.75</td>
    <td>58.20</td>
    <td>71.40</td>
    <td>83.12</td>
    <td>55.00</td>
    <td>87.25</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td rowspan="3">20%</td>
    <td>EcoFLAP</td>
    <td><b>87</b></td>
    <td><b>79.32</b></td>
    <td>74.70</td>
    <td>573</td>
    <td><b>89.86</b></td>
    <td>61.26</td>
    <td>46.56</td>
    <td>1951.0</td>
    <td>64.84</td>
    <td><b>60.27</b></td>
    <td>55.80</td>
    <td>70.20</td>
    <td>82.17</td>
    <td>57.70</td>
    <td>85.75</td>
    <td><b>0.995</b></td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>86</td>
    <td>78.84</td>
    <td><b>75.70</b></td>
    <td>595</td>
    <td>81.12</td>
    <td>61.71</td>
    <td><b>46.67</b></td>
    <td>1939.8</td>
    <td><b>66.14</b></td>
    <td>59.11</td>
    <td>54.10</td>
    <td>68.60</td>
    <td>82.17</td>
    <td><b>60.10</b></td>
    <td>86.90</td>
    <td>0.992</td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>87</td>
    <td>77.48</td>
    <td>73.92</td>
    <td>598</td>
    <td>80.67</td>
    <td><b>61.95</b></td>
    <td>45.78</td>
    <td>1957.2</td>
    <td><b>66.14</b></td>
    <td>58.28</td>
    <td><b>56.30</b></td>
    <td><b>70.30</b></td>
    <td><b>83.18</b></td>
    <td>55.40</td>
    <td>86.10</td>
    <td>0.986</td>
  </tr>
  <tr>
    <td rowspan="3">50%</td>
    <td>EcoFLAP</td>
    <td><b>82</b></td>
    <td>73.24</td>
    <td>71.06</td>
    <td>502</td>
    <td>75.78</td>
    <td>59.81</td>
    <td>41.56</td>
    <td>1732.8</td>
    <td>63.40</td>
    <td>54.23</td>
    <td><b>50.20</b></td>
    <td>53.80</td>
    <td>78.69</td>
    <td>39.80</td>
    <td>57.55</td>
    <td>0.877</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>81</td>
    <td><b>76.96</b></td>
    <td><b>71.30</b></td>
    <td>582</td>
    <td>72.12</td>
    <td>60.10</td>
    <td>41.67</td>
    <td>1726.8</td>
    <td><b>64.18</b></td>
    <td><b>54.41</b></td>
    <td>48.50</td>
    <td><b>65.20</b></td>
    <td>42.26</td>
    <td>47.20</td>
    <td><b>85.50</b></td>
    <td>0.900</td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>81</td>
    <td>72.08</td>
    <td>70.20</td>
    <td>534</td>
    <td><b>77.66</b></td>
    <td><b>61.16</b></td>
    <td><b>42.89</b></td>
    <td><b>1741.1</b></td>
    <td>66.01</td>
    <td>53.05</td>
    <td>49.30</td>
    <td>64.90</td>
    <td><b>79.82</b></td>
    <td><b>47.30</b></td>
    <td>85.00</td>
    <td><b>0.921</b></td>
  </tr>
  <tr>
    <td rowspan="3">2:4</td>
    <td>EcoFLAP</td>
    <td>67</td>
    <td>62.02</td>
    <td>60.70</td>
    <td>483</td>
    <td>65.19</td>
    <td>52.68</td>
    <td>32.89</td>
    <td><b>1309.1</b></td>
    <td>49.54</td>
    <td>45.75</td>
    <td>36.00</td>
    <td>49.30</td>
    <td>71.35</td>
    <td>30.20</td>
    <td>81.05</td>
    <td>0.760</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>69</td>
    <td><b>63.52</b></td>
    <td>59.71</td>
    <td><b>499</b></td>
    <td><b>67.39</b></td>
    <td>53.09</td>
    <td>34.22</td>
    <td>1282.7</td>
    <td>57.52</td>
    <td>43.09</td>
    <td><b>38.80</b></td>
    <td>51.60</td>
    <td><b>72.81</b></td>
    <td>30.60</td>
    <td><b>81.65</b></td>
    <td><b>0.779</b></td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>74</td>
    <td>62.68</td>
    <td><b>65.60</b></td>
    <td>426</td>
    <td>61.70</td>
    <td><b>56.97</b></td>
    <td><b>34.44</b></td>
    <td>1296.2</td>
    <td>62.35</td>
    <td><b>45.99</b></td>
    <td>35.60</td>
    <td><b>52.90</b></td>
    <td>35.59</td>
    <td>35.70</td>
    <td>81.30</td>
    <td>0.772</td>
  </tr>
  <tr>
    <td rowspan="2">W4A16</td>
    <td>AWQ</td>
    <td>84</td>
    <td>75.52</td>
    <td><b>72.78</b></td>
    <td>575</td>
    <td>85.23</td>
    <td>61.84</td>
    <td>45.44</td>
    <td>1978.3</td>
    <td>67.58</td>
    <td>53.70</td>
    <td>52.60</td>
    <td><b>70.50</b></td>
    <td>78.92</td>
    <td><b>54.80</b></td>
    <td>87.05</td>
    <td>0.972</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>86</td>
    <td><b>75.83</b></td>
    <td>72.36</td>
    <td>571</td>
    <td><b>85.40</b></td>
    <td><b>61.90</b></td>
    <td>44.82</td>
    <td>1970.2</td>
    <td>66.24</td>
    <td><b>56.25</b></td>
    <td><b>55.80</b></td>
    <td>70.30</td>
    <td>78.69</td>
    <td>54.30</td>
    <td>86.55</td>
    <td><b>0.975</b></td>
  </tr>
</table>

Table 4: Main results of various parameter compression methods on LLaVA-OneVision-7B evaluated on different tasks, grouped by Setting. † indicates that only the LLM backbone is quantized.

<table>
  <tr>
    <th>Models</th>
    <th>Benchmarks</th>
    <th>A-Only</th>
    <th>U-40%</th>
    <th>U-80%</th>
  </tr>
  <tr>
    <td rowspan="4">LLaVA-OneVision-7B</td>
    <td>DoxVQA</td>
    <td>82</td>
    <td>83</td>
    <td>84</td>
  </tr>
  <tr>
    <td>OCRbench</td>
    <td>372</td>
    <td>398</td>
    <td><b>410</b></td>
  </tr>
  <tr>
    <td>ChartQA</td>
    <td>74.12</td>
    <td>75.84</td>
    <td>76.24</td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>70.73</td>
    <td>71.83</td>
    <td>72.36</td>
  </tr>
  <tr>
    <td rowspan="4">Qwen2-VL-7B</td>
    <td>DoxVQA</td>
    <td>83</td>
    <td>85</td>
    <td>86</td>
  </tr>
  <tr>
    <td>OCRbench</td>
    <td>386</td>
    <td>410</td>
    <td><b>424</b></td>
  </tr>
  <tr>
    <td>ChartQA</td>
    <td>67.00</td>
    <td>67.84</td>
    <td>68.24</td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>68.17</td>
    <td>69.93</td>
    <td><b>70.98</b></td>
  </tr>
</table>

Table 5: VL-Cache Budget Allocation (5% Total Budget) with Hybrid Strategies: A-Only (adaptive-Only Allocation) vs. U-40% (40% uniform + 60% adaptive) vs. U-80% (80% uniform + 20% adaptive).

![Layer-wise Budget Allocation of VL-Cache on LLaVA-OneVision-7B (5% Total Budget Constraint, OCRBench Example)](page_1012_1342_377_246.png)

Figure 3: Layer-wise Budget Allocation of VL-Cache on LLaVA-OneVision-7B (5% Total Budget Constraint, OCRBench Example).

get was evenly distributed across all layers, and the remaining (60% or 20%) is adaptively allocated according to the original allocation strategy. As shown in Table 5, evenly allocating 80% of the total budget produced the best results.

5.2 Revisiting Head Adaptive Mechanism

Previous KV cache compression methods diverge on whether heads within a layer should select identical cached tokens. We explore head-adaptive token selection (i.e., allowing heads in the same layer to select different tokens) at a 1% budget. Table 6 shows that head-adaptivity significantly improves performance. This improvement likely stems from different KV or query heads capturing distinct information patterns, enabling head-adaptive selection to preserve critical information under tight budget constraints better. Detailed head attention distributions are in Appendix A.9.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Benchmarks</th>
    <th colspan="2">H2O</th>
    <th colspan="2">SnapKV</th>
    <th colspan="2">VL-Cache</th>
  </tr>
  <tr>
    <th>w/</th>
    <th>w/o</th>
    <th>w/</th>
    <th>w/o</th>
    <th>w/</th>
    <th>w/o</th>
  </tr>
  <tr>
    <td rowspan="4">LLaVA-OneVision-7B</td>
    <td>ChartQA</td>
    <td>67.32</td>
    <td>65.12</td>
    <td>74.64</td>
    <td>73.84</td>
    <td>68.64</td>
    <td>66.64</td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>64.96</td>
    <td>62.33</td>
    <td>65.62</td>
    <td>65.60</td>
    <td>61.76</td>
    <td><b>61.85</b></td>
  </tr>
  <tr>
    <td>MathVista</td>
    <td>42.40</td>
    <td>41.60</td>
    <td>54.40</td>
    <td>51.20</td>
    <td>52.20</td>
    <td>49.50</td>
  </tr>
  <tr>
    <td>LLaVA-Wilder</td>
    <td>62.50</td>
    <td>60.60</td>
    <td>63.70</td>
    <td>61.40</td>
    <td>62.80</td>
    <td>62.40</td>
  </tr>
  <tr>
    <td rowspan="4">Qwen2-VL-7B</td>
    <td>ChartQA</td>
    <td>54.36</td>
    <td>45.92</td>
    <td>55.96</td>
    <td>56.16</td>
    <td>57.76</td>
    <td>53.32</td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>51.16</td>
    <td>42.85</td>
    <td>57.38</td>
    <td>52.96</td>
    <td>56.09</td>
    <td>47.03</td>
  </tr>
  <tr>
    <td>MathVista</td>
    <td>49.00</td>
    <td>48.50</td>
    <td>49.90</td>
    <td>49.20</td>
    <td>52.90</td>
    <td>52.30</td>
  </tr>
  <tr>
    <td>LLaVA-Wilder</td>
    <td>57.20</td>
    <td>51.60</td>
    <td>57.70</td>
    <td>56.60</td>
    <td>54.30</td>
    <td>59.80</td>
  </tr>
</table>

Table 6: Effect of head-adaptive strategy for KV cache compression methods.

5.3 Attention Sink Tokens in LVLMs

Xiao et al. (2023) identified attention sinks in Transformers: a few initial tokens attract disproportionate attention regardless of semantic relevance, and their removal significantly degrades performance. We explore sink tokens in both textual and visual modalities within LVLMs. For textual sinks, comparing a random baseline against StreamingLLM (which preserves these sink tokens) shows StreamingLLM significantly improves performance at a 1% budget (Table 15), underscoring their importance. Similarly, visual information tends to concentrate on a set of image tokens after passing through the visual encoder Darzet et al. (2024). We find that FastV (using text-guided
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">Budgets</th>
  </tr>
  <tr>
    <th>10%</th>
    <th>20%</th>
    <th>40%</th>
  </tr>
  <tr>
    <td>FastV<sub>origin</sub></td>
    <td>31.04</td>
    <td>54.20</td>
    <td>67.40</td>
  </tr>
  <tr>
    <td>FastV<sub>A1</sub></td>
    <td>30.36</td>
    <td>52.40</td>
    <td>65.40</td>
  </tr>
  <tr>
    <td>FastV<sub>A2</sub></td>
    <td><b>45.56</b></td>
    <td><b>58.60</b></td>
    <td><b>70.04</b></td>
  </tr>
</table>

Table 7: Comparison of results for FastV<sub>origin</sub>, FastV<sub>A1</sub>, and FastV<sub>A2</sub> on ChartQA with Qwen2-VL-7B.

metrics for visual token pruning) underperforms VisionZip (relying solely on the visual encoder’s attention map). We hypothesize this is because visual sink tokens critically impact performance, yet text-guided metrics often fail to capture them. Two ablations on ChartQA using Qwen2-VL-7B (Table 7) further support this hypothesis: **A1**: Prohibiting FastV from selecting from the top 10% most critical visual tokens slightly degraded performance. **A2**: Conversely, forcing FastV to prioritize these top 10% tokens significantly improved performance. These results indicate that text-based selection can overlook critical visual sink tokens, and that retaining these sinks is vital, particularly under low budgets. The recent attention-sink-free gated attention model of Qiu et al. (2025) also warrants future exploration. More visualizations are available in Appendix A.12.

5.4 How to Merge Evicted Tokens

The merge operation shows promise for recovering evicted information (Bolya et al.). Current methods differ: LOOK-M and PruMerge+ merge evicted tokens into retained ones, whereas VisionZip concatenates them. This poses the question: What is an effective merge strategy for LVLMs? Our experiments reveal LOOK-M’s performance drops at low budgets. We hypothesize that at a 1% budget, its text-prior mechanism discards visual tokens, which are then merged into remaining text tokens. This cross-modal fusion, we argue, disrupts critical textual features (e.g., sink tokens), degrading performance.. Thus, we modify LOOK-M to merge evicted tokens only within the same modality. Table 8 shows our modification consistently improves performance over the original LOOK-M, underscoring that modality-specific merging is crucial for LVLM token compression. Specific examples are in Appendix A.13.

<table>
  <tr>
    <th rowspan="2">Budgets</th>
    <th rowspan="2">Methods</th>
    <th colspan="6">Benchmarks</th>
  </tr>
  <tr>
    <th>DavOQA</th>
    <th>ChartQA</th>
    <th>TextVQA</th>
    <th>OORibench</th>
    <th>MulVQA</th>
    <th>LLaVA-Walker</th>
  </tr>
  <tr>
    <td>1%</td>
    <td>LOOK-M<sub>origin</sub></td>
    <td>38</td>
    <td>34.44</td>
    <td>44.74</td>
    <td>81</td>
    <td>44.30</td>
    <td>43.40</td>
  </tr>
  <tr>
    <td></td>
    <td>LOOK-M<sub>change</sub></td>
    <td>44</td>
    <td>56.16</td>
    <td>48.12</td>
    <td>117</td>
    <td>51.80</td>
    <td>59.30</td>
  </tr>
  <tr>
    <td>5%</td>
    <td>LOOK-M<sub>origin</sub></td>
    <td>74</td>
    <td>73.72</td>
    <td>70.29</td>
    <td>375</td>
    <td>53.10</td>
    <td>65.50</td>
  </tr>
  <tr>
    <td></td>
    <td>LOOK-M<sub>change</sub></td>
    <td>74</td>
    <td>75.04</td>
    <td>70.89</td>
    <td>406</td>
    <td>53.50</td>
    <td>69.00</td>
  </tr>
</table>

Table 8: The performance comparison between LOOK-M with origin merge(LOOK-M<sub>origin</sub>) and modality-specific merge(LOOK-M<sub>change</sub>) on LLaVA-OV-7B.

ing token and parameter compression methods for LVLMs across diverse tasks, models, and metrics. Our empirical results reveal inherent trade-offs—performance, generalization, loyalty, and efficiency—tied to different compression strategies. Analysis of factors such as attention sinks and layer- or head-adaptive sparsity provides practical insights for optimizing these techniques. We hope that EffiVLM-BENCH offers a robust foundation for advancing future research in LVLM compression.

Limitations

Our work has several limitations: (1) Although EffiVLM-BENCH systematically evaluates both token and parameter compression approaches, it focuses primarily on a subset of representative LVLM models and tasks, leaving the performance on other architectures and more specialized domains unexplored. (2) We only consider training-free methods in this study, and the incorporation of training-based compression algorithms could provide deeper insights into the balance between performance gains and resource overhead. (3) Although our analysis delves into several mechanisms under extremely low budgets, there may be additional factors in token and parameter compression methods. In future work, we plan to extend EffiVLM-BENCH to include more tasks, models, and methods, offering a more comprehensive assessment of LVLM compression.

Acknowledgments

The work is supported by the National Key Research and Development Project (2022YFF0903301), the National Science Foundation of China (U22B2059,62276083). Ming Liu is the corresponding author. Besides, we express gratitude to DuXiaoman Technology for supporting the work.
References

Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 10865–10873. AAAI Press.

Anthropic. 2024. Developing a computer use model.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations.

Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, and Xiang Bai. 2024a. Llava-kd: A framework of distilling multimodal large language models. CoRR, abs/2410.16236.

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and 1 others. 2024b. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069.

Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2024. Honeybee: Locality-enhanced projector for multimodal LLM. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13817–13827. IEEE.

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024a. Allava: Harnessing gpt4v-synthesized data for A lite vision-language model. CoRR, abs/2402.11684.

Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024b. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024c. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330.

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/1504.00325.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jingou Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024d. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.

Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and 1 others. 2023. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886.

Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR).

Timothée Darzet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, and 1 others. 2023. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36:2252–2274.

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, and 32 others. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. CoRR, abs/2409.17146.

Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR.

Elias Frantar, Saleh Ashkboos, Torsten Hoeffer, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024. Mme: A comprehensive evaluation benchmark for multimodal large language models. Preprint, arXiv:2306.13394.

Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive KV cache compression for llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Lianyu Hu, Fanhua Shang, Liang Wan, and Wei Feng. 2024. illava: An image is worth fewer than 1/3 input tokens in large multimodal models. CoRR, abs/2412.06263.

Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaoshen Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shao-hui Lin. 2024. Dynamic-llava: Efficient multi-modal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00876.

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. Preprint, arXiv:1603.07396.

Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. 2024a. Llava-next: Stronger llms supercharge multimodal capabilities in the wild.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024b. Llava-oneview: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. 2024c. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392.

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. 2024d. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814.

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024e. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469.

Yunxin Li, Zhenyu Liu, Zitao Li, Xuanuy Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinpeng Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, and 3 others. 2025. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.04921.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Weiming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awg: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26286–26296.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024b. Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281.

Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024c. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12).

Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, and 8 others. 2024d. NVILA: efficient frontier visual language models. CoRR, abs/2412.04468.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhushou Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024a. Deepseek-v1: Towards real-world vision-language understanding. CoRR, abs/2403.05525.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024b. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR).

Yaxin Luo, Gen Luo, Jiayi Ji, Yiyi Zhou, Xiaoshuai Sun, Zhiqiang Shen, and Rongrong Ji. 2024. γ-mod: Exploring mixture-of-depth adaptation for multimodal large language models. CoRR, abs/2410.13859.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.

Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. 2020. Docvqa: A dataset for vqa on document images. corr abs/2007.00398 (2020). arXiv preprint arXiv:2007.00398.

OpenAI. 2024. Hello gpt-4o.
OpenAI. 2025. Introducing operator.

Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, and 1 others. 2025. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. arXiv preprint arXiv:2505.06708.

Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388.

Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, Haoyuan Li, Bolin Li, Zhelun Yu, Si Liu, Hongsheng Li, and Hao Jiang. 2024. Llava-mod: Making llava tiny via moe knowledge distillation. CoRR, abs/2408.15881.

Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326.

Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, and 1 others. 2023. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695.

Yi-Lin Sung, Jaehong Yoon, and Mohit Bansal. 2023. Ecoflap: Efficient coarse-to-fine layer-wise pruning for vision-language models. arXiv preprint arXiv:2310.02998.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, abs/2403.05530.

Qwen Team. 2025. Qwen2.5-vl.

Rubén Tito, Dimosthenis Karatzas, and Ernest Valveny. 2022. Hierarchical multimodal transformers for multi-page docvqa. arXiv preprint arXiv:2212.05935.

Dezhun Tu, Danylo Vashchilenko, Yuzhe Lu, and Pan-pan Xu. 2024. VI-cache: Sparsity and modality-aware kv cache compression for vision-language model inference acceleration. arXiv preprint arXiv:2410.23317.

Zhongwei Wan, Zhang Wu, Che Liu, Jinfa Huang, Zhi-hong Zhu, Peng Jin, Longyue Wang, and Li Yuan. 2024. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4065–4078.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi-hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xinsong Zhang. 2023a. Efficientvlm: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13899–13913. Association for Computational Linguistics.

Yuxin Wang, Minghua Ma, Zekun Wang, Jingchang Chen, Huiming Fan, Liping Shan, Qing Yang, Dongliang Xu, Ming Liu, and Bing Qin. 2024b. CFSP: an efficient structured pruning framework for llms with coarse-to-fine activation information. CoRR, abs/2409.15199.

Zekun Wang, Jingchang Chen, Wangchunshu Zhou, Tian Liu, and Bing Qin. 2023b. Smarttrim: Adaptive tokens and parameters pruning for efficient vision-language models. CoRR, abs/2305.15033.

Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, and Furu Wei. 2021. Distilled dual-encoder model for vision-language understanding. CoRR, abs/2112.08723.

x.ai. Grok-1.5 vision preview.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv.

Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, and Furu Wei. 2021. Beyond preserved accuracy: Evaluating loyalty and robustness of BERT compression. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 10653–10659. Association for Computational Linguistics.

Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024a. Aguvis: Unified pure vision agents for autonomous GUI interaction. CoRR, abs/2412.04454.

Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2024b. Think: Thinner key cache by query-driven pruning. CoRR, abs/2407.21018.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengya Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2024. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mlm on your phone. arXiv preprint arXiv:2408.01800.

JiangYong Yu, Sifan Zhou, Dawei Yang, Shuo Wang, Shouyu Li, Xing Hu, Chen Xu, Zukang Xu, Changyong Shu, and Zhihang Yuan. 2025. Mquant: Unleashing the inference potential of multimodal large language models via full static quantization. arXiv preprint arXiv:2502.00425.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Li-Juan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. Preprint. arXiv:2308.02490.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqiu Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR.

Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Zhiwei Liu. 2024a. Lmms-eval: Reality check on the evaluation of large multimodal models. Preprint. arXiv:2407.12772.

Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and 1 others. 2024b. Sparsevlm: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04417.

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. 2023a. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, volume 36, pages 34661–34710. Curran Associates, Inc.

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023b. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661–34710.

Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. 2024. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289.

A Appendix

A.1 Evaluation Benchmarks

We conduct evaluations in realistic scenarios, using diverse benchmarks that span a wide range of vision-language tasks. These are grouped into three categories:

(1) Chart, Diagram, and Document Understanding This category focuses on textual and diagrammatic comprehension. Benchmarks include DocVQA (Mathew et al., 2020), ChartQA (Masry et al., 2022), TextVQA (Singh et al., 2019), OCR-Bench (Liu et al., 2024c), and AI2D (Kembhavi et al., 2016). These test the ability to accurately detect and recognize text in various forms (printed, handwritten, or embedded in charts/diagrams) and to understand document layouts or diagrammatic structures.

(2) Perception and Multi-discipline Reasoning This category targets visual perception combined with logical or knowledge-intensive reasoning. Benchmarks include GQA (Hudson and Manning, 2019), MMMU (Yue et al., 2024), MME (Fu et al., 2024), MMbench (Liu et al., 2024b), MMVet (Yu et al., 2023), MMStar (Chen et al., 2024c), and MathVista (Lu et al., 2024b).

(3) Real-world Compositional & Interactive QA This category assesses interactive and compositional reasoning in realistic conversational settings. Benchmarks like LLava-Wilder (Li et al., 2024a), RealWorldQA (x.ai), and ImageDC (Li et al., 2024a) test a model’s ability to handle open-ended questions, multi-turn dialogues, and complex visual-linguistic grounding, as well as its adaptation to diverse topics and coherence over extended interactions.

Across these benchmarks, we evaluate various compression methods on multimodal understanding and reasoning. Input/output lengths differ significantly by task (Table 9). To assess KV cache methods on longer sequences, we select MPDocVQA (Tito et al., 2022) for multi-images and MovieChat (Song et al., 2023) for video.

A.2 Compression Method

Token Pruning Token pruning compression reduces computational overhead by eliminating redundant tokens during inference. We implement
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="13">Benchmarks</th>
  </tr>
  <tr>
    <th>DocVQA</th>
    <th>ChinRQA</th>
    <th>TextQQA</th>
    <th>OCR-Bench</th>
    <th>AID</th>
    <th>QQA</th>
    <th>NMMU</th>
    <th>NME</th>
    <th>RelevantQQA</th>
    <th>MMSur</th>
    <th>MultiViSta</th>
    <th>LLaVA-Wilder</th>
    <th>JMBSench</th>
    <th>MMVqa</th>
    <th>ImageBC</th>
  </tr>
  <tr>
    <td>Input<br>LLaVA-OneVision-7B<br>Qwen2-VL-7B<br>InternVL2.5-38B</td>
    <td>7413<br>2027<br>1835</td>
    <td>3882<br>406<br>1367</td>
    <td>5147<br>1010<br>1744</td>
    <td>1233<br>75<br>1212</td>
    <td>3497<br>598<br>1389</td>
    <td>2693<br>383<br>1800</td>
    <td>3613<br>730<br>1588</td>
    <td>5768<br>1141<br>1506</td>
    <td>7778<br>1776<br>1834</td>
    <td>2226<br>288<br>1478</td>
    <td>2995<br>465<br>1346</td>
    <td>4559<br>904<br>1337</td>
    <td>1913<br>251<br>1387</td>
    <td>5006<br>1018<br>1400</td>
    <td>2704<br>363<br>1492</td>
  </tr>
  <tr>
    <td>Output<br>LLaVA-OneVision-7B<br>Qwen2-VL-7B<br>InternVL2.5-38B</td>
    <td>2.72<br>1.82<br>1.72</td>
    <td>2.92<br>2.96<br>3.27</td>
    <td>2.69<br>2.96<br>2.86</td>
    <td>26.04<br>3.71<br>11.27</td>
    <td>1.00<br>1.00<br>1.00</td>
    <td>1.27<br>1.23<br>1.07</td>
    <td>1.05<br>1.42<br>1.07</td>
    <td>1.03<br>1.01<br>1.01</td>
    <td>1.02<br>1.01<br>1.01</td>
    <td>2.85<br>2.65<br>78.80</td>
    <td>232.12<br>55.54<br>257.73</td>
    <td>1.20<br>1.72<br>1.00</td>
    <td>59.07<br>24.70<br>145.62</td>
    <td>414.61<br>314.22<br>158.66</td>
  </tr>
</table>

Table 9: Input and Output token length of different models on various benchmarks. We select 100 samples from each benchmark for calculation.

methods including FastV (Chen et al., 2024b), VisionZip (Yang et al., 2024), and PruMerge+ (Shang et al., 2024), which selectively remove redundant tokens based on relevance scores or token-level importance metrics.

KV Cache Compression We explore several KV cache compression methods that reduce memory usage during inference by selectively retaining key-value states, including StreamingLLM (Xiao et al., 2023), H2O (Zhang et al., 2023a), SnapKV (Li et al., 2024e), PyramidKV (Cai et al., 2024b), LOOK-M (Wan et al., 2024), and VL-Cache (Tu et al., 2024).

Model Pruning Model pruning aims to reduce model size and inference cost while preserving performance. We evaluate EcoFLAP (Sung et al., 2023), Wanda (Sun et al., 2023), and SparseGPT (Frantar and Alistarh, 2023), applying these methods only to the LLM backbone of the LVLM.

Model Quantization Model quantization compresses models by reducing the precision of their weights and activations, thereby lowering memory requirements and accelerating computation. We consider techniques like AWQ (Lin et al., 2024) and GPTQ (Frantar et al., 2022) (applied only to the LVLM’s LLM backbone).

A.3 Details of Methods

Token Pruning Let \( T_v = \{ t_{v,j} \}_{j=1}^{l_v} \) be the set of \( l_v \) visual tokens, each \( t_{v,j} \in \mathbb{R}^d \). Given a retention budget \( b \in (0, 1] \), pruning aims to select \( l_v' = \lfloor b \cdot l_v \rfloor \) tokens. This typically involves a scoring function \( s : \mathbb{R}^d \to \mathbb{R} \) to assess token importance.

FastV uses a lightweight scoring strategy. For each visual token \( t_{v,j} \), its importance \( s(t_{v,j}) \) is its accumulated attention score among textual tokens. A binary mask \( m \in \{0, 1\}^{l_v} \) is defined by

\[
m_j = \begin{cases}
1, & \text{if } s(t_{v,j}) \geq \tau, \\
0, & \text{otherwise},
\end{cases}
\]

where the threshold \( \tau \) is set to ensure the binary mask \( m \) satisfies the following constraint:

\[
\sum_{j=1}^{l_v} m_j = \lfloor b \cdot l_v \rfloor.
\]

The pruned visual token set is given by

\[
T_v' = \{ t_{v,j} \mid m_j = 0 \}.
\]

VisionZip first selects an initial set of tokens to keep using accumulated attention (similar to FastV), and identifies the discarded set \( T_v'' \). To compensate for information loss from \( T_v'' \), VisionZip clusters \( T_d \) into \( k \) groups. The original paper sets \( k = \lfloor (b/6.4) \cdot l_v \rfloor \). For each cluster \( C_i \subseteq T_v'' \), its centroid \( c_i \) is computed:

\[
c_i = \frac{1}{|C_i|} \sum_{t \in C_i} t.
\]

The recycled token set is then defined as

\[
R_v = \{ c_i \mid i = 1, 2, \ldots, k \}.
\]

Finally, the tokens in \( R_v \) are concatenated with the retained tokens.

PruMerge+ also adaptively selects important tokens and merges less important ones to retain critical information. For each visual token \( t_{v,j} \), its importance score is computed via the [CLS] token’s attention weights in the vision encoder:

\[
s(t_{v,j}) = A_j^{[CLS]},
\]
where \( A^{[\text{CLS}]} \in \mathbb{R}^{l_v} \) denotes the attention weight from the [CLS] token to visual tokens. For architectures without [CLS] tokens, we compute row-wise averages of the attention matrix:

\[
s(t_{v,j}) = \frac{1}{l_v} \sum_{i=1}^{l_v} A_{ij}.
\]

The binary mask \( m \in \{0, 1\}^{l_v} \) is determined through interquartile range (IQR) based outlier detection:

\[
m_j = \begin{cases}
1, & \text{if } s(t_{v,j}) \geq \tau, \\
0, & \text{otherwise},
\end{cases}
\]

where threshold \( \tau \) is adaptively calculated using the IQR method. To ensure budget compliance, we enforce:

\[
\sum_{j=1}^{l_v} m_j = \lfloor b \cdot l_v \rfloor.
\]

If IQR selection yields fewer than \( l'_v = \lfloor b \cdot l_v \rfloor \) tokens (the target budget \( b \)), PruMerge+ first supplements this set by uniformly sampling from the remaining highest-scoring unselected candidates. All \( l'_v \) tokens in the final retained set then undergo feature merging to consolidate information. Unlike VisionZip, PruMerge+ merges information from pruned tokens into retained ones.

KV Cache Compression In the LVLM prefill stage, let \( l_v \) and \( l_t \) be the number of visual and text tokens, respectively, with total length \( l = l_v + l_t \). Let \( w \) be the recent window size and \( b \) the compression budget fraction. The attention map is \( A \in \mathbb{R}^{l \times l} \), where \( A_{ij} \) is the attention weight between query \( i \) and key \( j \). To compress the KV caches, we introduce:

\[
\mathcal{F} : \mathbb{R}^{l \times l} \to \{0, 1\}^l,
\]

which produces a binary mask \( \mathbf{m} \in \{0, 1\}^l \) indicating which tokens to retain (\( \mathbf{m}_j = 1 \)) or evict (\( \mathbf{m}_j = 0 \)). We consider four variants of \( \mathcal{F} \) that rank token importance:

• *Accumulated Attention* \( \mathcal{F}_{\text{acc}} \). We sum attention weights along the query tokens dimension:

\[
s_j = \sum_{i=1}^l A_{ij}.
\]

• *Normalized Attention* \( \mathcal{F}_{\text{norm}} \). We normalize attention weights for each query \( i \), then sum and average:

\[
\bar{s}_j = \text{Norm}(\sum_{i=1}^l A_{ij})
\]

• *Sliding Window Attention* \( \mathcal{F}_{\text{sw}} \). We compute accumulated attention scores along the query tokens but only over a recent window:

\[
\bar{s}_j = \sum_{i=l-w+1}^l A_{ij}.
\]

• *Post-Vision Attention* \( \mathcal{F}_{\text{pv}} \). Recognizing that in many LVLMs, the textual tokens following the visual tokens are more critical, this variant computes attention scores using only the queries from the text region. Formally, letting the text region be indexed by \( i = l_v + 1, \ldots, l \), we define:

\[
\bar{s}_j = \sum_{i=l_v+1}^l A_{ij}.
\]

These variants of \( \mathcal{F} \) offer different strategies for ranking tokens, allowing us to evaluate how attention-based selection influences KV cache compression in LVLMs. In experiments, H2O and LOOK-M use *Accumulated Attention*, SnapKV and PyramidKV use *Sliding Window Attention* and VL-Cache uses *Post-Vision Attention*.

Model Pruning aims to identify a binary mask \( S \in \{0, 1\}^{m \times n} \) so that the pruned weight matrix \( \widetilde{W} = W \odot S \) preserves performance while satisfying a desired sparsity level, i.e., \( \|S\|_0 = p \cdot (m \times n) \), with \( p \) being the fraction of weights to retain. An importance score \( s_{ij} \) is computed for each weight \( W_{ij} \). The binary mask is then determined by selecting the top-\( p \) fraction of weights with the highest scores.

*Wanda* measures the importance of each weight by the product of its magnitude and the norm of its corresponding input activation:

\[
s_{ij} = |W_{ij}| \cdot \|X_j\|_2,
\]

where \( X_j \) is the input activation vector associated with the \( j \)-th column of \( W \). This prioritizes weights coupled with strong activations, preserving contributions critical to model performance even if the weights themselves are small.
EcoFLAP prunes LVLMs layer-wise in a coarse-to-fine manner. The coarse phase computes layer i’s importance \( S(W_i) \) using an expected zeroth-order gradient approximation: \( \| \nabla_{W_i} \mathcal{L}(W_i, \mathcal{D}) \|_2 = \mathbb{E}_{d \sim \mathcal{D}}[\mathbb{E}_{z \sim N(0,1)} \| \frac{\mathcal{L}(W_i + \epsilon z, d) - \mathcal{L}(W_i - \epsilon z, d)}{2\epsilon} \|] \). The fine-grained step then uses same score as Wanda locally within each layer to prune weights according to these layer-specific sparsity ratios.

SparseGPT minimizes a layer’s output reconstruction error using a second-order approximation. Its importance metric is derived from an approximate diagonal of the Hessian:

\[
s_{ij}^{\text{SparseGPT}} = [\frac{\|W\|^2}{\operatorname{diag}(XX^T + \lambda I)^{-1}}]_{ij}
\]

Model Quantization Quantization aims to approximate a full-precision weight matrix \( W \in \mathbb{R}^{m \times n} \) with a low-bit representation while minimizing performance degradation. In our experiments, we consider two notable approaches of Post-Training Quantization: AWQ and GPTQ.

AWQ refines the PTQ process by incorporating input activation statistics to better preserve the layer output. Given the weight matrix \( W \) and its corresponding input activations \( X \), AWQ selects quantization parameters by solving:

\[
\min_{s, z} \| WX - \widehat{W} X \|_F^2
\]

with \( \widehat{W}_{ij} = s \cdot (\operatorname{round}\left(\frac{W_{ij}}{s}\right) - z) \).

GPTQ enhances PTQ by leveraging second-order information to minimize the output error induced by quantization. Starting from the uniform quantization of \( W \), GPTQ aims to adjust the quantization parameters so as to minimize the reconstruction error using a Taylor expansion involving an approximate Hessian \( H \) (often estimated via the diagonal of \( X^T X \)). This Hessian-guided correction allows GPTQ to achieve high quantization fidelity in one shot, even under aggressive bit-width reductions.

A.4 Implementation Details

Token Pruning To ensure fair comparison of token pruning methods, we standardize average token retention rates across layers. We evaluate rates of 1%, 5%, 10%, 20%, and 40%. Higher rates are omitted, as 40% retention typically preserves performance comparable to the original model. Additionally, because the metrics of these methods can be architecture-dependent, we adapt each method consistently across all evaluated LVLMs to maintain fair comparisons.

KV Cache Compression For fair comparison of KV cache compression methods, we standardize budget allocation. We test budgets of 1%, 5%, 10%, 20%, and 40% of the original cache size. Higher budgets are omitted as 40% typically yields performance comparable to the uncompressed model. During prefill, 10% of the currently allocated budget forms a recent window to retain the most recent tokens; the remainder is managed by each method’s specific mechanism. In the decoding, to maintain uniformity and fairness across all methods, we do not apply any additional KV cache compression.

Model Pruning In our experiments, we focus on pruning the LLM component. We utilize a consistent set of 128 samples from the COCO-Caption(Chen et al., 2015) as our validation set. We apply unstructured pruning at 20% and 50% sparsity levels, as well as semi-structured pruning under 2:4 setting. We do not apply structured pruning because these result in significant performance drops without recovery training (Wang et al., 2024b).

Model Quantization We focus exclusively on quantizing the LLM component, aligning with the existing LVLM quantization work. We utilize the same validation dataset as in our pruning experiments. For the LLM’s weights, we apply a W4A16g128 quantization scheme, where weights are quantized to 4 bits and activations are kept in FP16, and a group size of 128 is used during quantization. This scheme offers a balanced trade-off between model performance and efficiency.

Evaluation We use Imms-eval (Zhang et al., 2024a) framework to perform evaluation. Given the diversity of benchmarks and LVLMs, we select specific LVLMs and benchmarks tailored to each method type, which are detailed in Appendix A.1. The batch size is set to 1. We use 8 × NVIDIA A800 GPUs. For the efficiency experiments, we run 128 samples with an 8k input token length and 100 token output length on a single NVIDIA A800 GPU.

A.5 Detailed Results of KV Cache Compression

Table 15 presents the detailed results of various KV cache compression methods across different mod-
Figure 4: Visualizations of attention weight heatmaps from layers 0, 14, and 27 of Qwen2-VL-7B. Different heads within the same layer exhibit notable pattern variations.

els and benchmarks under varying cache budgets.

A.6 More Results of Token Compression
We additionally evaluate two recent token compression approaches: iLLaVA (Hu et al., 2024) and SparseVLM (Zhang et al., 2024b), which incorporate progressive sparsification strategy that gradually sparsifies tokens across multiple layers, rather than enforcing a target sparsity ratio at a single layer. We follow the original implementation details and the results against existing methods in our benchmark are shown in Table 10.

We find that: (i) iLLaVA demonstrates competitive performance at higher token budgets (40%) thanks to its iterative pruning strategy within the visual encoder. This strategy progressively prunes tokens across layers while preserving critical ones in earlier stages. However, at lower token budgets (1%), performance declines significantly similar as the phenomenon of FastV, likely due to the visual sink token: relying on text tokens to guide pruning within LLM may inadvertently remove essential visual sink tokens, leading to performance degradation (details are in Section 5.3). (ii) Regarding SparseVLM, compared to other token pruning methods that also exclusively prune visual tokens in the LLM component (e.g., FastV), it introduces an iterative pruning strategy and a more fine-grained, text-driven selection criterion to guide the pruning process. Consequently, SparseVLM achieves consistently strong performance across various compression budgets and tasks.

A.7 More Results of Parameter Compression

Table 11 shows the results of parameter compression methods on Qwen2-VL-7B across various benchmarks. To investigate the efficacy of combining token-level and parameter-level compression, we select AWQ and SnapKV as representative methods for token and parameter compression, respectively. The results on Qwen2-VL-7B are shown in Table 12. Combining AWQ with SnapKV boosts inference speed by 1.65× over AWQ alone, while maintaining comparable overall performance, particularly on multimodal reasoning tasks (MMMU, MathVista). This highlights the efficiency gains achievable with hybrid compression. However, for tasks demanding fine-grained visual perception (e.g., TextVQA), integrating token compression like SnapKV results in a marginal performance drop. This underscores that acceleration strategies must be tailored to specific target scenario characteristics.
<table>
  <tr>
    <th>Budgets</th>
    <th>Methods</th>
    <th>TextVQA</th>
    <th>AI2D</th>
    <th>MMMU</th>
    <th>MME</th>
    <th>MathVista</th>
  </tr>
  <tr>
    <td rowspan="3">1%</td>
    <td>VisionZip</td>
    <td><b>44.48</b></td>
    <td><b>72.11</b></td>
    <td>42.56</td>
    <td><b>1704.2</b></td>
    <td>39.30</td>
  </tr>
  <tr>
    <td>iLLaVA</td>
    <td>22.86</td>
    <td>68.78</td>
    <td><b>43.11</b></td>
    <td>1101.0</td>
    <td>37.50</td>
  </tr>
  <tr>
    <td>SparseVLM</td>
    <td>42.29</td>
    <td>71.07</td>
    <td>43.00</td>
    <td>1662.5</td>
    <td><b>39.40</b></td>
  </tr>
  <tr>
    <td rowspan="3">10%</td>
    <td>VisionZip</td>
    <td>57.26</td>
    <td>77.97</td>
    <td>44.11</td>
    <td><b>1915.0</b></td>
    <td><b>45.00</b></td>
  </tr>
  <tr>
    <td>iLLaVA</td>
    <td>64.88</td>
    <td>73.47</td>
    <td>44.66</td>
    <td>1773.9</td>
    <td>39.30</td>
  </tr>
  <tr>
    <td>SparseVLM</td>
    <td><b>65.72</b></td>
    <td><b>79.27</b></td>
    <td><b>45.22</b></td>
    <td>1841.8</td>
    <td>44.60</td>
  </tr>
  <tr>
    <td rowspan="3">40%</td>
    <td>VisionZip</td>
    <td>68.21</td>
    <td>83.84</td>
    <td>46.11</td>
    <td>1956.8</td>
    <td>52.60</td>
  </tr>
  <tr>
    <td>iLLaVA</td>
    <td><b>73.26</b></td>
    <td>87.62</td>
    <td><b>47.44</b></td>
    <td><b>1995.3</b></td>
    <td><b>56.60</b></td>
  </tr>
  <tr>
    <td>SparseVLM</td>
    <td>72.80</td>
    <td><b>88.66</b></td>
    <td>45.78</td>
    <td>1993.8</td>
    <td>54.10</td>
  </tr>
  <tr>
    <td>100%</td>
    <td>Original</td>
    <td>74.79</td>
    <td>89.96</td>
    <td>45.44</td>
    <td>1974.1</td>
    <td>58.20</td>
  </tr>
</table>

Table 10: Results of iLLaVA and SparseVLM on LLaVA-OneVision-7B model. Bold denotes the best result under the same setting.

<table>
  <tr>
    <th rowspan="2">Settings</th>
    <th rowspan="2">Methods</th>
    <th colspan="6">Qwen2-VL-7B</th>
    <th rowspan="2">OP</th>
  </tr>
  <tr>
    <th>CharQA</th>
    <th>TextVQA</th>
    <th>OCRBatch</th>
    <th>AI2D</th>
    <th>QQA</th>
    <th>MMMU</th>
    <th>MME</th>
    <th>RealworldQA</th>
    <th>MMSur</th>
    <th>MathVista</th>
    <th>LLaVA-Wilder</th>
    <th>ImageDC</th>
  </tr>
  <tr>
    <td rowspan="3">100%</td>
    <td>Original</td>
    <td>81.56</td>
    <td>81.82</td>
    <td>813</td>
    <td>91.02</td>
    <td>62.3</td>
    <td>50.77</td>
    <td>2327.78</td>
    <td>66.53</td>
    <td>57.11</td>
    <td>58.3</td>
    <td>72.7</td>
    <td>86.35</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>EcoFLAP</td>
    <td>81.36</td>
    <td>81.82</td>
    <td>805</td>
    <td>91.13</td>
    <td>62.30</td>
    <td>50.00</td>
    <td>2288.0</td>
    <td>66.67</td>
    <td>56.74</td>
    <td>57.10</td>
    <td>52.70</td>
    <td>87.33</td>
    <td>0.964</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>75.52</td>
    <td>80.64</td>
    <td>754</td>
    <td>89.67</td>
    <td>60.86</td>
    <td>46.11</td>
    <td>2090.8</td>
    <td>65.88</td>
    <td>53.02</td>
    <td>52.40</td>
    <td>50.20</td>
    <td>84.90</td>
    <td>0.918</td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>71.98</td>
    <td>77.45</td>
    <td>722</td>
    <td>82.32</td>
    <td>57.84</td>
    <td>39.56</td>
    <td>1871.9</td>
    <td>59.61</td>
    <td>45.99</td>
    <td>40.70</td>
    <td>44.50</td>
    <td>81.00</td>
    <td>0.842</td>
  </tr>
  <tr>
    <td rowspan="3">20%</td>
    <td>EcoFLAP</td>
    <td>81.08</td>
    <td>81.79</td>
    <td>800</td>
    <td>91.16</td>
    <td>62.32</td>
    <td>50.44</td>
    <td>2312.6</td>
    <td>66.54</td>
    <td>56.69</td>
    <td>58.90</td>
    <td>53.50</td>
    <td>87.13</td>
    <td>0.970</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>77.72</td>
    <td>80.61</td>
    <td>784</td>
    <td>88.63</td>
    <td>61.20</td>
    <td>45.00</td>
    <td>2129.1</td>
    <td>64.97</td>
    <td>52.74</td>
    <td>53.50</td>
    <td>51.20</td>
    <td>85.55</td>
    <td>0.924</td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>71.96</td>
    <td>78.08</td>
    <td>727</td>
    <td>82.29</td>
    <td>58.66</td>
    <td>38.78</td>
    <td>1967.3</td>
    <td>57.91</td>
    <td>45.63</td>
    <td>39.30</td>
    <td>44.50</td>
    <td>80.30</td>
    <td>0.845</td>
  </tr>
  <tr>
    <td rowspan="3">50%</td>
    <td>EcoFLAP</td>
    <td>81.16</td>
    <td>82.08</td>
    <td>809</td>
    <td>91.03</td>
    <td>62.32</td>
    <td>50.78</td>
    <td>2317.2</td>
    <td>67.45</td>
    <td>56.85</td>
    <td>59.30</td>
    <td>54.80</td>
    <td>88.37</td>
    <td>0.977</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>73.96</td>
    <td>80.09</td>
    <td>790</td>
    <td>88.44</td>
    <td>61.54</td>
    <td>44.89</td>
    <td>2106.9</td>
    <td>65.10</td>
    <td>53.42</td>
    <td>53.10</td>
    <td>52.40</td>
    <td>86.00</td>
    <td>0.924</td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>61.04</td>
    <td>76.60</td>
    <td>721</td>
    <td>81.19</td>
    <td>58.24</td>
    <td>36.89</td>
    <td>1675.00</td>
    <td>59.22</td>
    <td>42.31</td>
    <td>40.40</td>
    <td>48.70</td>
    <td>85.25</td>
    <td>0.832</td>
  </tr>
  <tr>
    <td rowspan="3">2:4</td>
    <td>EcoFLAP</td>
    <td>81.16</td>
    <td>82.08</td>
    <td>809</td>
    <td>91.03</td>
    <td>62.32</td>
    <td>50.78</td>
    <td>2317.2</td>
    <td>67.45</td>
    <td>56.85</td>
    <td>59.30</td>
    <td>54.80</td>
    <td>88.37</td>
    <td>0.977</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>73.96</td>
    <td>80.09</td>
    <td>790</td>
    <td>88.44</td>
    <td>61.54</td>
    <td>44.89</td>
    <td>2106.9</td>
    <td>65.10</td>
    <td>53.42</td>
    <td>53.10</td>
    <td>52.40</td>
    <td>86.00</td>
    <td>0.924</td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>61.04</td>
    <td>76.60</td>
    <td>721</td>
    <td>81.19</td>
    <td>58.24</td>
    <td>36.89</td>
    <td>1675.00</td>
    <td>59.22</td>
    <td>42.31</td>
    <td>40.40</td>
    <td>48.70</td>
    <td>85.25</td>
    <td>0.832</td>
  </tr>
</table>

Table 11: Results of various parameter compression methods on Qwen2-VL-7B.

<table>
  <tr>
    <th>Methods</th>
    <th>Speedup</th>
    <th>AI2D</th>
    <th>MMMU</th>
    <th>MME</th>
    <th>TextVQA</th>
    <th>MathVista</th>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>1.0x</td>
    <td>90.80</td>
    <td>57.20</td>
    <td>2320.6</td>
    <td>50.26</td>
    <td>66.53</td>
  </tr>
  <tr>
    <td>AWQ+SnapKV</td>
    <td>1.65x</td>
    <td>90.74</td>
    <td>57.40</td>
    <td>2235.9</td>
    <td>48.78</td>
    <td>66.80</td>
  </tr>
</table>

Table 12: Results on Qwen2-VL-7B on combining token and parameter compression.

A.8 Architectural Impact on Compression Effectiveness

We analyze how architectural differences in LVLM backbones (LLaVA-OneVision, Qwen2-VL, InternVL-2.5) affect compression effectiveness. Two primary aspects are considered:

• High-Resolution Image Processing. LLaVA-OneVision and InternVL-2.5 use anyres-like strategies, splitting high-resolution images into many patches, averaging 4180 visual tokens per input on our benchmark. In contrast, Qwen2-VL employs NaViT, a visual encoder with native dynamic-resolution support that merges visual tokens, reducing their average count to 762 (see Dehghani et al. (2023) for NaViT details). This difference in initial visual token count directly impacts token compression. For example, LLaVA-OneVision-7B consistently outperforms similarly sized Qwen2-VL-7B in various token compression budgets (Table 9). This suggests that a larger pool of initial visual tokens offers greater flexibility for compression, as its richer representation can tolerate more aggressive reduction without substantial performance loss.

• Parameter Size. To isolate the impact of model scale, we evaluate Qwen2-VL-2B against Qwen2-VL-7B using various parameter compression methods. The results are shown in Table 13.
<table>
  <tr>
    <th>Settings</th>
    <th>Method</th>
    <th>Size</th>
    <th>A12D</th>
    <th>MathVista</th>
    <th>MME</th>
    <th>MMU</th>
    <th>TexVQA</th>
    <th>OP</th>
  </tr>
  <tr>
    <td rowspan="2">100%</td>
    <td>Original</td>
    <td>2B</td>
    <td>83.83</td>
    <td>46.20</td>
    <td>1872.0</td>
    <td>40.85</td>
    <td>79.70</td>
    <td>1.0000</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td>
    <td>91.02</td>
    <td>58.30</td>
    <td>2327.7</td>
    <td>50.77</td>
    <td>81.82</td>
    <td>1.0000</td>
  </tr>
  <tr>
    <td rowspan="3">20%</td>
    <td>ECOFLAP</td>
    <td>2B</td>
    <td>80.99</td>
    <td>44.00</td>
    <td>1883.4</td>
    <td>39.44</td>
    <td>79.19</td>
    <td>0.9769</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td>
    <td>91.13</td>
    <td>57.10</td>
    <td>2288.0</td>
    <td>50.00</td>
    <td>81.82</td>
    <td>0.9900</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>2B</td>
    <td>81.99</td>
    <td>45.30</td>
    <td>1884.5</td>
    <td>39.89</td>
    <td>79.31</td>
    <td>0.9874</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>7B</td>
    <td>91.16</td>
    <td>58.90</td>
    <td>2312.6</td>
    <td>50.44</td>
    <td>81.79</td>
    <td>1.0000</td>
  </tr>
  <tr>
    <td rowspan="3">SparseGPT</td>
    <td>2B</td>
    <td>82.35</td>
    <td>46.30</td>
    <td>1852.9</td>
    <td>40.00</td>
    <td>79.44</td>
    <td>0.9901</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td>
    <td>91.03</td>
    <td>59.30</td>
    <td>2317.2</td>
    <td>50.78</td>
    <td>82.08</td>
    <td>1.0030</td>
  </tr>
  <tr>
    <td>ECOFLAP</td>
    <td>2B</td>
    <td>71.99</td>
    <td>33.50</td>
    <td>1585.2</td>
    <td>34.44</td>
    <td>76.39</td>
    <td>0.8497</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>7B</td>
    <td>89.67</td>
    <td>52.40</td>
    <td>2090.8</td>
    <td>46.11</td>
    <td>80.64</td>
    <td>0.9356</td>
  </tr>
  <tr>
    <td rowspan="3">50%</td>
    <td>Wanda</td>
    <td>2B</td>
    <td>72.83</td>
    <td>34.90</td>
    <td>1721.8</td>
    <td>36.44</td>
    <td>76.26</td>
    <td>0.8812</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td>
    <td>88.63</td>
    <td>53.50</td>
    <td>2129.1</td>
    <td>45.00</td>
    <td>80.61</td>
    <td>0.9350</td>
  </tr>
  <tr>
    <td>SparseGPT</td>
    <td>2B</td>
    <td>72.15</td>
    <td>35.70</td>
    <td>1535.1</td>
    <td>33.56</td>
    <td>75.75</td>
    <td>0.8472</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>7B</td>
    <td>88.44</td>
    <td>53.10</td>
    <td>2106.9</td>
    <td>44.89</td>
    <td>80.09</td>
    <td>0.9309</td>
  </tr>
  <tr>
    <td rowspan="3">2:4</td>
    <td>ECOFLAP</td>
    <td>2B</td>
    <td>44.79</td>
    <td>28.20</td>
    <td>1008.7</td>
    <td>27.11</td>
    <td>66.12</td>
    <td>0.6445</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td>
    <td>82.32</td>
    <td>40.70</td>
    <td>1871.9</td>
    <td>39.56</td>
    <td>77.45</td>
    <td>0.8313</td>
  </tr>
  <tr>
    <td>Wanda</td>
    <td>2B</td>
    <td>47.83</td>
    <td>27.70</td>
    <td>1207.2</td>
    <td>26.89</td>
    <td>67.01</td>
    <td>0.6695</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>7B</td>
    <td>82.29</td>
    <td>39.30</td>
    <td>1967.3</td>
    <td>38.78</td>
    <td>78.08</td>
    <td>0.8336</td>
  </tr>
  <tr>
    <td rowspan="3">SparseGPT</td>
    <td>2B</td>
    <td>49.92</td>
    <td>26.20</td>
    <td>1027.8</td>
    <td>29.22</td>
    <td>66.91</td>
    <td>0.6624</td>
  </tr>
  <tr>
    <td></td>
    <td>7B</td>
    <td>81.19</td>
    <td>40.40</td>
    <td>1675.0</td>
    <td>36.89</td>
    <td>76.60</td>
    <td>0.8000</td>
  </tr>
  <tr>
    <td>W4A16</td>
    <td>AWQ</td>
    <td>2B</td>
    <td>82.90</td>
    <td>46.50</td>
    <td>1835.4</td>
    <td>39.82</td>
    <td>79.34</td>
    <td>0.9893</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>7B</td>
    <td>90.80</td>
    <td>58.30</td>
    <td>2232.5</td>
    <td>48.78</td>
    <td>81.31</td>
    <td>0.9826</td>
  </tr>
</table>

Table 13: Results of various parameter compression methods on different model sizes of Qwen2-VL.

A.9 Visualization Head Attention Distribution
To illustrate how head-adaptive mechanisms can capture diverse attention patterns, we analyzed attention distributions across heads in several randomly selected layers. Qwen2-VL-7B was chosen for this visualization because its relatively fewer image tokens enhance attention map readability. Indeed, Figure 4 shows that different heads within the same layer exhibit distinct attention patterns.

A.10 Visualizations on VL-Cache Budget Allocation
We evaluated VL-Cache on eight benchmarks across three LVLMs, all at a 5% total budget. The models consistently exhibited similar, highly skewed layer-wise budget allocations (Figures 5 and 6). True to its design, VL-Cache’s layer-adaptive mechanism heavily favored early “dense” layers, particularly the first two. Allocation minima were observed at layer 4 for LLaVA-OV-7B and Qwen2-VL-7B, and at layer 5 for InternVL2.5-38B, with subsequent layers typically receiving far below-average budgets. This consistent front-loading suggests performance could be improved by reallocating budget from these over-resourced early layers to the under-resourced later ones. Effective layer-adaptive strategies therefore demand a more nuanced resource balance, rather than just aggressive front-loading.

A.11 Visualization of VL-Cache Token Selection
We visualize visual tokens selected by VL-Cache at layers 0, 1, and 2 in LLaVA-OV-7B (5% budget, across 6 benchmarks). Figure 8 reveals that for the first two layers (0 and 1), selected token distributions are strikingly similar: they predominantly cluster towards the end of the visual token sequence, potentially limiting their utility in deriving final answers. Layer 2, in contrast, displays a more uniform distribution of selected tokens across the entire sequence. Although VL-Cache’s strategy allocates a higher token density to these early layers, these observed patterns suggest such aggres-
Figure 5: Budgets distribution of LLaVA-OV-7B and Qwen2-VL-7B across 8 benchmarks under 5% budget. Each subplot shows a random sample from the respective benchmark.

sive, front-loaded budgeting may not be optimal.

A.12 Visualization of Visual Sink Tokens
To further illustrate the visual token sink phenomenon, we visualize attention heatmaps for 15 randomly selected ChartQA samples using LLaVA-OV-7B and Qwen2-VL-7B. Figure 10 clearly shows a visual sink emerging after the visual encoder’s forward pass: image information consistently converges to a limited subset of visual tokens. Furthermore, Figures 7 and 9 compare visual token selection by VisionZip (image-guided) against FastV (text-guided). VisionZip tends to retain these critical sink tokens, whereas FastV often fails to capture many of them, explaining observed performance differences.

A.13 Case of Different Merge Strategies
Table 14 contrasts LOOK-M’s outputs under two token merge strategies: cross-modal versus modality-specific. Modality-specific merging yields an accurate image caption; in contrast, cross-modal merging results in an image-irrelevant output.
Figure 6: Budgets distribution of InternVL2.5-38B across 8 benchmarks under 5% budget. Each subplot shows a random sample from the respective benchmark.

Figure 7: Visualization of token selection strategies on ChartQA with Qwen2-VL-7B under 10% budget. Left to right: attention heatmap, tokens retained by FastV and VisionZip. VisionZip selects more critical tokens.
Q: "Which color Bar shows the greatest Meat consumer?"
A: "Finland"

Q: "Which college's name is specified in the logo?"
A: "MEHARRY MEDICAL COLLEGE"

(a) Visualization of Layer 0 visual token selection: representative cases from ChartQA (left) and DocVQA (right)

Q: "What is the sum of the two numbers visible in the picture?"
A: "71"

Q: "Which spot is the red car parking on?"
A: "31"

(b) Visualization of Layer 1 visual token selection: representative cases from MathVista (left) and MMVet (right)

Q: "Where is this?"
A: "garage"

Q: "What is the number for southern homes?"
A: "648-HOME"

(c) Visualization of Layer 2 visual token selection: representative cases from OCRBench (left) and TextVQA (right)

Figure 8: Visualization of VL-Cache selected visual tokens in layers 0–2 on LLaVA-OneVision-7B. Highlighted regions with bounding boxes are critical to the answer.
(a) Visualization results of randomly selected cases from DocVQA

(b) Visualization results of randomly selected cases from MMMU

Figure 9: Visualization of token selection strategies on DocVQA and MMMU with Qwen2-VL-7B under 10% budget. Left to right: attention heatmap, tokens retained by FastV and VisionZip. VisionZip selects more critical tokens than FastV.
ChartQA

(a) Heatmap visualization results of ChartQA on LLaVA-OneVision-7B

ChartQA

(b) Heatmap visualization results of ChartQA on Qwen2-VL-7B

Figure 10: Heatmap visualizations of 15 randomly selected examples from the ChartQA benchmark on LLaVA-OneVision-7B and Qwen2-VL-7B.
Visual input example for image caption

![A classic Honda motorcycle parked outdoors on a gravel surface](page_374_382_1002_393.png)

User    Please give a brief description of this picture.

<table>
  <tr>
    <th>Qwen2-VL-7B</th>
    <td>The image shows a classic Honda motorcycle parked on a gravel surface. The motorcycle has a black and chrome finish, with a quilted seat and a small windshield.</td>
  </tr>
  <tr>
    <th>LOOK-M<sub>origin</sub></th>
    <td>The image shows a close-up of a person’s hand holding a small, round object. The object appears to be a small, metallic ball or a similar spherical item.</td>
  </tr>
  <tr>
    <th>LOOK-M<sub>change</sub></th>
    <td>The image shows a motorcycle parked outdoors. The motorcycle has a classic design with a prominent front wheel and a seat that appears to be made of leather.</td>
  </tr>
</table>

Table 14: Comparative analysis of response quality in Qwen2-VL-7B original model vs. LOOK-M with origin merge and modality-specific merge strategies under extreme compression ratios
<table>
  <tr>
    <th>Benchmarks</th>
    <th>Models</th>
    <th>Methods</th>
    <th>1%</th>
    <th>5%</th>
    <th>10%</th>
    <th>20%</th>
    <th>40%</th>
    <th>100%</th>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td>Random</td>
    <td>31</td>
    <td>55</td>
    <td>69</td>
    <td>79</td>
    <td>85</td>
    <td>87</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>42</td>
    <td>62</td>
    <td>71</td>
    <td>74</td>
    <td>79</td>
    <td>87</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>58</td>
    <td>78</td>
    <td>82</td>
    <td><b>86</b></td>
    <td><b>87</b></td>
    <td>87</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>76</b></td>
    <td><b>82</b></td>
    <td>83</td>
    <td>85</td>
    <td>86</td>
    <td>87</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>70</td>
    <td>76</td>
    <td>75</td>
    <td>78</td>
    <td>81</td>
    <td>87</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>38</td>
    <td>74</td>
    <td>81</td>
    <td>85</td>
    <td><b>87</b></td>
    <td>87</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>69</td>
    <td><b>82</b></td>
    <td><b>85</b></td>
    <td><b>86</b></td>
    <td><b>87</b></td>
    <td>87</td>
  </tr>
  <tr>
    <td rowspan="6">DocVQA (Split:test) (Metric:ANLS)</td>
    <td>Random</td>
    <td>13</td>
    <td>60</td>
    <td>78</td>
    <td>89</td>
    <td><b>94</b></td>
    <td>95</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>42</td>
    <td>56</td>
    <td>65</td>
    <td>74</td>
    <td>84</td>
    <td>95</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>47</td>
    <td>67</td>
    <td>79</td>
    <td>89</td>
    <td>93</td>
    <td>95</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>78</b></td>
    <td><b>90</b></td>
    <td><b>92</b></td>
    <td><b>92</b></td>
    <td><b>94</b></td>
    <td>95</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>66</td>
    <td>83</td>
    <td>82</td>
    <td>84</td>
    <td>87</td>
    <td>95</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>27</td>
    <td>58</td>
    <td>77</td>
    <td>89</td>
    <td><b>94</b></td>
    <td>95</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>52</td>
    <td>83</td>
    <td>91</td>
    <td><b>93</b></td>
    <td><b>94</b></td>
    <td>95</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td>StreamingLLM</td>
    <td>0</td>
    <td>46</td>
    <td>53</td>
    <td>63</td>
    <td>80</td>
    <td>94</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>58</td>
    <td>76</td>
    <td>86</td>
    <td>91</td>
    <td>93</td>
    <td>94</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>34</td>
    <td><b>91</b></td>
    <td><b>93</b></td>
    <td><b>93</b></td>
    <td><b>94</b></td>
    <td>94</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>53</td>
    <td>90</td>
    <td>92</td>
    <td>93</td>
    <td>93</td>
    <td>94</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>12</td>
    <td>54</td>
    <td>78</td>
    <td>90</td>
    <td>93</td>
    <td>94</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td><b>64</b></td>
    <td>87</td>
    <td>91</td>
    <td><b>93</b></td>
    <td>93</td>
    <td>94</td>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td>Random</td>
    <td>45.04</td>
    <td>59.76</td>
    <td>64.56</td>
    <td>69.52</td>
    <td>72.72</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>58.20</td>
    <td>65.44</td>
    <td>69.84</td>
    <td>73.88</td>
    <td>75.36</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>67.32</td>
    <td>75.48</td>
    <td>76.44</td>
    <td><b>77.48</b></td>
    <td><b>77.84</b></td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>74.64</b></td>
    <td><b>76.48</b></td>
    <td>76.68</td>
    <td>77.20</td>
    <td>77.28</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>72.68</td>
    <td>76.20</td>
    <td>76.08</td>
    <td>76.24</td>
    <td>76.80</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>34.44</td>
    <td>73.72</td>
    <td><b>76.72</b></td>
    <td>77.40</td>
    <td>77.72</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>66.64</td>
    <td>74.12</td>
    <td>75.96</td>
    <td>77.08</td>
    <td>77.56</td>
    <td>80.00</td>
  </tr>
  <tr>
    <td rowspan="6">ChartQA (Split:overall) (Metric:Acc)</td>
    <td>Random</td>
    <td>13.00</td>
    <td>45.96</td>
    <td>66.08</td>
    <td>72.32</td>
    <td>79.68</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>50.28</td>
    <td>61.52</td>
    <td>65.04</td>
    <td>69.60</td>
    <td>76.32</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>54.36</td>
    <td>70.12</td>
    <td>74.56</td>
    <td>78.60</td>
    <td>80.88</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>55.96</td>
    <td><b>77.32</b></td>
    <td><b>79.28</b></td>
    <td><b>80.20</b></td>
    <td><b>81.32</b></td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>67.24</b></td>
    <td>72.08</td>
    <td>76.08</td>
    <td>78.96</td>
    <td>80.40</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>23.00</td>
    <td>48.52</td>
    <td>68.24</td>
    <td>76.96</td>
    <td>79.48</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>53.32</td>
    <td>67.00</td>
    <td>74.28</td>
    <td>79.16</td>
    <td>80.68</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td>StreamingLLM</td>
    <td>0.68</td>
    <td>67.08</td>
    <td>71.68</td>
    <td>73.72</td>
    <td>80.00</td>
    <td>88.04</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>34.64</td>
    <td>80.32</td>
    <td>84.76</td>
    <td>86.88</td>
    <td>87.84</td>
    <td>88.04</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>60.60</td>
    <td>84.80</td>
    <td>86.64</td>
    <td>87.28</td>
    <td>87.92</td>
    <td>88.04</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>76.40</b></td>
    <td><b>85.36</b></td>
    <td><b>86.68</b></td>
    <td>87.24</td>
    <td>87.52</td>
    <td>88.04</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>8.80</td>
    <td>54.48</td>
    <td>75.48</td>
    <td>84.04</td>
    <td>87.16</td>
    <td>88.04</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>58.60</td>
    <td>82.48</td>
    <td>85.88</td>
    <td><b>87.64</b></td>
    <td><b>88.16</b></td>
    <td>88.04</td>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td>Random</td>
    <td>39.63</td>
    <td>57.15</td>
    <td>64.29</td>
    <td>68.96</td>
    <td>71.81</td>
    <td>74.79</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>47.01</td>
    <td>57.29</td>
    <td>64.49</td>
    <td>70.25</td>
    <td>71.41</td>
    <td>74.79</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>64.96</td>
    <td>72.16</td>
    <td><b>73.71</b></td>
    <td>74.14</td>
    <td>74.54</td>
    <td>74.79</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>65.62</b></td>
    <td><b>72.54</b></td>
    <td>70.86</td>
    <td>71.62</td>
    <td>73.77</td>
    <td>74.79</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>58.56</td>
    <td>68.39</td>
    <td>66.37</td>
    <td>66.60</td>
    <td>69.70</td>
    <td>74.79</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>44.74</td>
    <td>70.29</td>
    <td>73.07</td>
    <td><b>74.39</b></td>
    <td><b>74.76</b></td>
    <td>74.79</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>61.85</td>
    <td>70.73</td>
    <td>72.92</td>
    <td>73.46</td>
    <td>74.16</td>
    <td>74.79</td>
  </tr>
  <tr>
    <td rowspan="6">TextVQA (Split:val) (Metric:Acc)</td>
    <td>Random</td>
    <td>5.21</td>
    <td>48.04</td>
    <td>61.58</td>
    <td>72.78</td>
    <td>79.78</td>
    <td>81.82</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>31.60</td>
    <td>49.70</td>
    <td>58.89</td>
    <td>67.77</td>
    <td>75.18</td>
    <td>81.82</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>51.16</td>
    <td>64.50</td>
    <td>71.40</td>
    <td>77.85</td>
    <td>81.13</td>
    <td>81.82</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>57.38</td>
    <td><b>73.45</b></td>
    <td><b>78.37</b></td>
    <td><b>80.83</b></td>
    <td>81.19</td>
    <td>81.82</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>67.50</b></td>
    <td>71.01</td>
    <td>73.90</td>
    <td>77.89</td>
    <td>78.87</td>
    <td>81.82</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>31.89</td>
    <td>52.07</td>
    <td>70.24</td>
    <td>78.62</td>
    <td>81.24</td>
    <td>81.82</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>47.03</td>
    <td>68.17</td>
    <td>77.74</td>
    <td>80.81</td>
    <td><b>81.79</b></td>
    <td>81.82</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td>StreamingLLM</td>
    <td>0.83</td>
    <td>51.49</td>
    <td>53.24</td>
    <td>58.97</td>
    <td>70.28</td>
    <td>82.87</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>63.29</td>
    <td>76.08</td>
    <td>79.93</td>
    <td>81.96</td>
    <td>82.52</td>
    <td>82.87</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>36.56</td>
    <td><b>80.58</b></td>
    <td><b>82.15</b></td>
    <td><b>82.69</b></td>
    <td><b>82.77</b></td>
    <td>82.87</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>56.73</td>
    <td>79.70</td>
    <td>80.64</td>
    <td>81.91</td>
    <td>82.16</td>
    <td>82.87</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>12.56</td>
    <td>61.42</td>
    <td>76.33</td>
    <td>81.13</td>
    <td>82.34</td>
    <td>82.87</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td><b>64.40</b></td>
    <td>76.54</td>
    <td>80.35</td>
    <td>82.29</td>
    <td>82.71</td>
    <td>82.87</td>
  </tr>
</table>
<table>
  <tr>
    <th>Benchmarks</th>
    <th>Models</th>
    <th>Methods</th>
    <th>1%</th>
    <th>5%</th>
    <th>10%</th>
    <th>20%</th>
    <th>40%</th>
    <th>100%</th>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td>Random</td>
    <td>67</td>
    <td>335</td>
    <td>455</td>
    <td>530</td>
    <td>564</td>
    <td>595</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>94</td>
    <td>222</td>
    <td>320</td>
    <td>386</td>
    <td>502</td>
    <td>595</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>229</td>
    <td>407</td>
    <td><b>495</b></td>
    <td><b>554</b></td>
    <td>588</td>
    <td>595</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>281</b></td>
    <td><b>415</b></td>
    <td>445</td>
    <td>475</td>
    <td>553</td>
    <td>595</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>275</td>
    <td>337</td>
    <td>354</td>
    <td>358</td>
    <td>426</td>
    <td>595</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>81</td>
    <td>375</td>
    <td>463</td>
    <td>544</td>
    <td><b>589</b></td>
    <td>595</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>231</td>
    <td>372</td>
    <td>460</td>
    <td>505</td>
    <td>551</td>
    <td>595</td>
  </tr>
  <tr>
    <td rowspan="6">OCR(Bench (Split:test) (Metric:Acc)) Qwen2-VL-7B</td>
    <td>Random</td>
    <td>109</td>
    <td>192</td>
    <td>336</td>
    <td>491</td>
    <td>658</td>
    <td>813</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>116</td>
    <td>199</td>
    <td>248</td>
    <td>349</td>
    <td>465</td>
    <td>813</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>153</td>
    <td>334</td>
    <td>441</td>
    <td>570</td>
    <td>713</td>
    <td>813</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>199</td>
    <td><b>428</b></td>
    <td><b>542</b></td>
    <td><b>654</b></td>
    <td><b>725</b></td>
    <td>813</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>241</b></td>
    <td>417</td>
    <td>535</td>
    <td>642</td>
    <td>662</td>
    <td>813</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>112</td>
    <td>202</td>
    <td>299</td>
    <td>441</td>
    <td>587</td>
    <td>813</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>216</td>
    <td>386</td>
    <td>505</td>
    <td>592</td>
    <td>660</td>
    <td>813</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td>StreamingLLM</td>
    <td>51</td>
    <td>121</td>
    <td>162</td>
    <td>214</td>
    <td>399</td>
    <td>802</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>143</td>
    <td>487</td>
    <td>656</td>
    <td>752</td>
    <td>782</td>
    <td>802</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>359</td>
    <td><b>652</b></td>
    <td><b>720</b></td>
    <td><b>777</b></td>
    <td><b>791</b></td>
    <td>802</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>478</b></td>
    <td>637</td>
    <td>664</td>
    <td>734</td>
    <td>765</td>
    <td>802</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>18</td>
    <td>183</td>
    <td>491</td>
    <td>695</td>
    <td>771</td>
    <td>802</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>193</td>
    <td>487</td>
    <td>630</td>
    <td>733</td>
    <td>778</td>
    <td>802</td>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td>Random</td>
    <td>41.10</td>
    <td>44.20</td>
    <td>47.80</td>
    <td>53.50</td>
    <td>55.90</td>
    <td>58.20</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>40.40</td>
    <td>43.10</td>
    <td>45.20</td>
    <td>48.50</td>
    <td>54.40</td>
    <td>58.20</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>42.40</td>
    <td>48.90</td>
    <td>53.70</td>
    <td>55.40</td>
    <td><b>56.60</b></td>
    <td>58.20</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>54.40</b></td>
    <td><b>55.30</b></td>
    <td><b>56.30</b></td>
    <td><b>56.70</b></td>
    <td><b>56.60</b></td>
    <td>58.20</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>53.00</td>
    <td>54.10</td>
    <td>54.50</td>
    <td>56.50</td>
    <td>56.30</td>
    <td>58.20</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>44.30</td>
    <td>53.10</td>
    <td>53.60</td>
    <td>54.00</td>
    <td>55.10</td>
    <td>58.20</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>49.50</td>
    <td>53.40</td>
    <td>55.80</td>
    <td>55.90</td>
    <td>56.50</td>
    <td>58.20</td>
  </tr>
  <tr>
    <td rowspan="6">MathVista (Split:testmini) (format: COT) Qwen2-VL-7B</td>
    <td>Random</td>
    <td>45.00</td>
    <td>49.60</td>
    <td>54.00</td>
    <td>56.30</td>
    <td>57.20</td>
    <td>58.30</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>47.40</td>
    <td>54.80</td>
    <td>55.30</td>
    <td>56.20</td>
    <td>56.40</td>
    <td>58.30</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>49.00</td>
    <td>55.60</td>
    <td>55.80</td>
    <td>57.00</td>
    <td>57.80</td>
    <td>58.30</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>49.90</td>
    <td>56.50</td>
    <td>57.20</td>
    <td>57.40</td>
    <td><b>58.00</b></td>
    <td>58.30</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>50.30</b></td>
    <td><b>57.30</b></td>
    <td><b>57.60</b></td>
    <td><b>57.50</b></td>
    <td><b>58.00</b></td>
    <td>58.30</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>35.80</td>
    <td>54.30</td>
    <td>56.20</td>
    <td>55.50</td>
    <td>56.70</td>
    <td>58.30</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>52.30</td>
    <td>55.00</td>
    <td>54.80</td>
    <td>56.40</td>
    <td>56.70</td>
    <td>58.30</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td>StreamingLLM</td>
    <td>40.80</td>
    <td>48.00</td>
    <td>50.10</td>
    <td>54.50</td>
    <td>59.10</td>
    <td>70.20</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>50.40</td>
    <td>60.90</td>
    <td>64.60</td>
    <td><b>68.60</b></td>
    <td>70.50</td>
    <td>70.20</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>53.70</td>
    <td>64.30</td>
    <td><b>68.40</b></td>
    <td>68.40</td>
    <td>70.00</td>
    <td>70.20</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>56.10</b></td>
    <td><b>66.10</b></td>
    <td>66.20</td>
    <td>68.00</td>
    <td>69.60</td>
    <td>70.20</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>38.30</td>
    <td>56.50</td>
    <td>61.60</td>
    <td>66.70</td>
    <td>69.40</td>
    <td>70.20</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>53.90</td>
    <td>60.70</td>
    <td>65.00</td>
    <td>67.60</td>
    <td><b>70.60</b></td>
    <td>70.20</td>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td>Random</td>
    <td>19.60</td>
    <td>34.10</td>
    <td>40.80</td>
    <td>46.50</td>
    <td>47.70</td>
    <td>55.00</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>24.60</td>
    <td>39.80</td>
    <td>46.30</td>
    <td>48.10</td>
    <td>50.00</td>
    <td>55.00</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td><b>33.60</b></td>
    <td>46.10</td>
    <td><b>51.60</b></td>
    <td><b>53.20</b></td>
    <td><b>54.10</b></td>
    <td>55.00</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>32.90</td>
    <td><b>47.90</b></td>
    <td>50.20</td>
    <td>52.20</td>
    <td>53.30</td>
    <td>55.00</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>33.60</b></td>
    <td>43.60</td>
    <td>47.80</td>
    <td>47.20</td>
    <td>51.60</td>
    <td>55.00</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>20.60</td>
    <td>45.20</td>
    <td>49.10</td>
    <td>52.20</td>
    <td>53.60</td>
    <td>55.00</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>30.00</td>
    <td>42.50</td>
    <td>49.30</td>
    <td>50.20</td>
    <td>51.90</td>
    <td>55.00</td>
  </tr>
  <tr>
    <td rowspan="6">MMVet (Split:test) (Metric: GPT) Qwen2-VL-7B</td>
    <td>Random</td>
    <td>4.10</td>
    <td>10.90</td>
    <td>22.30</td>
    <td>33.90</td>
    <td>52.20</td>
    <td>65.40</td>
  </tr>
  <tr>
    <td>StreamingLLM</td>
    <td>1.10</td>
    <td>16.30</td>
    <td>28.00</td>
    <td>38.60</td>
    <td>44.60</td>
    <td>65.40</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>8.30</td>
    <td>27.30</td>
    <td>39.20</td>
    <td>53.70</td>
    <td>56.00</td>
    <td>65.40</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td><b>12.10</b></td>
    <td><b>33.50</b></td>
    <td><b>43.50</b></td>
    <td><b>55.10</b></td>
    <td><b>56.50</b></td>
    <td>65.40</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td>11.20</td>
    <td>31.20</td>
    <td>39.30</td>
    <td>46.70</td>
    <td>51.40</td>
    <td>65.40</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>1.40</td>
    <td>17.80</td>
    <td>30.40</td>
    <td>46.30</td>
    <td>55.30</td>
    <td>65.40</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>7.00</td>
    <td>25.80</td>
    <td>38.10</td>
    <td>47.40</td>
    <td>53.30</td>
    <td>65.40</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td>StreamingLLM</td>
    <td>0.00</td>
    <td>21.20</td>
    <td>32.20</td>
    <td>37.70</td>
    <td>53.70</td>
    <td>69.40</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>15.90</td>
    <td>47.50</td>
    <td>58.80</td>
    <td><b>68.00</b></td>
    <td>70.70</td>
    <td>69.40</td>
  </tr>
  <tr>
    <td>SnapKV</td>
    <td>23.00</td>
    <td><b>51.40</b></td>
    <td><b>63.60</b></td>
    <td>67.10</td>
    <td>69.20</td>
    <td>69.40</td>
  </tr>
  <tr>
    <td>PyramidKV</td>
    <td><b>33.30</b></td>
    <td>51.10</td>
    <td>57.70</td>
    <td>64.00</td>
    <td>67.10</td>
    <td>69.40</td>
  </tr>
  <tr>
    <td>LOOK-M</td>
    <td>2.80</td>
    <td>29.20</td>
    <td>50.50</td>
    <td>57.80</td>
    <td><b>71.10</b></td>
    <td>69.40</td>
  </tr>
  <tr>
    <td>VL-Cache</td>
    <td>16.50</td>
    <td>35.50</td>
    <td>48.40</td>
    <td>57.50</td>
    <td>67.00</td>
    <td>69.40</td>
  </tr>
</table>
<table>
  <tr>
    <th rowspan="2">Benchmarks</th>
    <th rowspan="2">Models</th>
    <th rowspan="2">Methods</th>
    <th colspan="10">%</th>
  </tr>
  <tr>
    <th>1%</th>
    <th>5%</th>
    <th>10%</th>
    <th>20%</th>
    <th>40%</th>
    <th>100%</th>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td rowspan="6">Qwen2-VL-7B</td>
    <td>Random</td><td>34.10</td><td>51.60</td><td>58.20</td><td>63.60</td><td>67.70</td><td>71.40</td>
  </tr>
  <tr>
    <td>StreamingLLM</td><td>48.00</td><td>65.70</td><td>69.20</td><td><b>71.20</b></td><td>69.40</td><td>71.40</td>
  </tr>
  <tr>
    <td>H2O</td><td>62.50</td><td><b>69.20</b></td><td><b>70.00</b></td><td>71.00</td><td>70.60</td><td>71.40</td>
  </tr>
  <tr>
    <td>SnapKV</td><td><b>63.70</b></td><td>66.20</td><td>69.40</td><td>71.00</td><td><b>70.90</b></td><td>71.40</td>
  </tr>
  <tr>
    <td>PyramidKV</td><td>62.30</td><td>67.20</td><td>66.80</td><td>67.50</td><td>69.00</td><td>71.40</td>
  </tr>
  <tr>
    <td>LOOK-M</td><td>43.40</td><td>65.50</td><td>65.70</td><td>66.80</td><td>68.40</td><td>71.40</td>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-Wilder (Split: test) (Metric: GPT)</td>
    <td rowspan="6">InternVL2.5-38B</td>
    <td>Random</td><td>12.50</td><td>21.00</td><td>31.50</td><td>42.30</td><td>59.90</td><td>72.70</td>
  </tr>
  <tr>
    <td>StreamingLLM</td><td>19.50</td><td>39.60</td><td>53.40</td><td>61.00</td><td>69.20</td><td>72.70</td>
  </tr>
  <tr>
    <td>H2O</td><td>37.20</td><td>57.30</td><td>62.00</td><td>65.60</td><td><b>72.50</b></td><td>72.70</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>27.70</td><td>58.10</td><td><b>64.50</b></td><td>66.60</td><td>69.50</td><td>72.70</td>
  </tr>
  <tr>
    <td>PyramidKV</td><td>36.30</td><td><b>58.50</b></td><td>64.10</td><td>65.00</td><td>68.70</td><td>72.70</td>
  </tr>
  <tr>
    <td>LOOK-M</td><td>19.10</td><td>52.30</td><td>60.50</td><td>67.80</td><td>69.80</td><td>72.70</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td rowspan="6">ImageDC (Split:DC100_EN) (Metric: GPT)</td>
    <td>Random</td><td>27.35</td><td>54.35</td><td>73.80</td><td>83.35</td><td>86.65</td><td>87.25</td>
  </tr>
  <tr>
    <td>StreamingLLM</td><td>38.40</td><td>77.95</td><td>80.50</td><td>85.25</td><td>86.50</td><td>87.25</td>
  </tr>
  <tr>
    <td>H2O</td><td><b>77.05</b></td><td><b>85.30</b></td><td><b>86.00</b></td><td><b>86.75</b></td><td>86.53</td><td>87.25</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>28.00</td><td>77.00</td><td>80.15</td><td>83.25</td><td>86.25</td><td>87.25</td>
  </tr>
  <tr>
    <td>PyramidKV</td><td>27.75</td><td>69.55</td><td>75.45</td><td>80.40</td><td>84.00</td><td>87.25</td>
  </tr>
  <tr>
    <td>LOOK-M</td><td>24.55</td><td>83.95</td><td>85.99</td><td>86.50</td><td><b>86.70</b></td><td>87.25</td>
  </tr>
  <tr>
    <td rowspan="6">LLaVA-OneVision-7B</td>
    <td rowspan="6">Qwen2-VL-7B</td>
    <td>Random</td><td>15.25</td><td>38.30</td><td>56.60</td><td>69.50</td><td>82.10</td><td>86.35</td>
  </tr>
  <tr>
    <td>StreamingLLM</td><td>16.70</td><td>31.35</td><td>76.95</td><td>82.15</td><td>85.10</td><td>86.35</td>
  </tr>
  <tr>
    <td>H2O</td><td><b>29.55</b></td><td><b>82.25</b></td><td><b>83.85</b></td><td><b>85.80</b></td><td><b>85.80</b></td><td>86.35</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>17.30</td><td>29.85</td><td>64.05</td><td>77.95</td><td>83.75</td><td>86.35</td>
  </tr>
  <tr>
    <td>PyramidKV</td><td>16.90</td><td>31.05</td><td>56.80</td><td>69.90</td><td>78.55</td><td>86.35</td>
  </tr>
  <tr>
    <td>LOOK-M</td><td>24.05</td><td>23.65</td><td>29.15</td><td>82.10</td><td>85.55</td><td>86.35</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td rowspan="6">Qwen2-VL-7B</td>
    <td>Random</td><td>15.05</td><td>17.80</td><td>41.80</td><td>70.35</td><td>82.30</td><td>86.35</td>
  </tr>
  <tr>
    <td>StreamingLLM</td><td>13.25</td><td>48.10</td><td>51.40</td><td>73.30</td><td>83.50</td><td>86.45</td>
  </tr>
  <tr>
    <td>H2O</td><td><b>59.60</b></td><td><b>83.55</b></td><td><b>85.05</b></td><td><b>85.85</b></td><td><b>86.65</b></td><td>86.45</td>
  </tr>
  <tr>
    <td>SnapKV</td><td>37.60</td><td>75.05</td><td>80.60</td><td>85.50</td><td>85.25</td><td>86.45</td>
  </tr>
  <tr>
    <td>PyramidKV</td><td>46.75</td><td>77.65</td><td>81.95</td><td>84.45</td><td>86.10</td><td>86.45</td>
  </tr>
  <tr>
    <td>LOOK-M</td><td>19.35</td><td>62.00</td><td>70.50</td><td>76.50</td><td>86.40</td><td>86.45</td>
  </tr>
  <tr>
    <td rowspan="6">InternVL2.5-38B</td>
    <td rowspan="6">ImageDC (Split:DC100_EN) (Metric: GPT)</td>
    <td>Random</td><td>24.90</td><td>56.55</td><td>71.40</td><td>82.95</td><td>85.50</td><td>86.45</td>
  </tr>
</table>

Table 15: Main results of various KV cache compression methods on different models and tasks. **Bold** denotes the best result under the same setting.

Reviewer paper summaries: This paper presents a comprehensive benchmark named EFFIVLM-BENCH to systematically assess training free acceleration methods for LVLMs. These methods are classified into two categories including token compression and parameter compression. The proposed benchmark covers a range of representative model architectures and multimodal tasks, incorporating comprehensive metrics that measure performance, generalization, loyalty, and efficiency.

The paper introduces EFFIVLM-BENCH, a well-structured and comprehensive benchmark that evaluates training-free acceleration methods for LVLMs across multiple dimensions, including performance, generalization, loyalty, and efficiency. Unlike existing studies that focus on limited metrics, this framework provides a multi-faceted evaluation, ensuring a more holistic understanding of model compression techniques.
Summary of strengths: (1) This paper addresses the challenges in evaluating efficient Large Vision-Language Models (LVLMs) and proposes a unified evaluation framework, conducting a detailed assessment of existing methods.

(2) The paper provides comprehensive analysis and experimental evaluations based on LLaVA-OneVision-7B, Qwen2-VL-7B, and InternVL2.5-38B, offering valuable insights into their performance.

(3) The evaluation code will be open-sourced, which could be beneficial for research on training-free paradigms.

1.	The paper introduces EFFIVLM-BENCH, which systematically evaluates training-free acceleration methods using multiple metrics, providing a well-rounded assessment.

2.	The paper explores Pareto-optimal trade-offs, helping practitioners select the best compression strategy for different deployment scenarios.

3.	The benchmark includes both token and parameter compression techniques, ensuring a thorough comparison of various acceleration strategies.

4.	The paper is well-structured and easy to understand, making it accessible to both researchers and practitioners in the field.
Summary of weaknesses: As a benchmark paper, the comparison needs to be more comprehensive. Some recent important models and methods are still missing.

1.	The paper treats Token Compression and Parameter Compression as separate evaluation categories. It remains unclear which method should be prioritized for different scenarios or whether a combination of both can achieve better efficiency-performance trade-offs. 

2.	The paper evaluates multiple LVLMs but does not analyze how different model architectures impact compression effectiveness.
Comments, suggestions and typos: (1) The paper would benefit from including model compression results across a wider range of model sizes, such as smaller models (e.g., 3B) and larger models (e.g., 72B). This would strengthen the contribution of the benchmark.

(2) Some recent methods are missing from the comparisons, such as SparseVLM [1], which should be included for a more comprehensive evaluation.

[1] Zhang Y, Fan C K, Ma J, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference[J]. arXiv preprint arXiv:2410.04417, 2024.

Please see weakness above
Overall assessment scores: 3.5

3.5
Metareview score: 3.5
Metareview: This paper presents EFFIVLM-BENCH for training-free acceleration evaluation in large visual-language models. Extensive experiments have been conducted to verify the effectiveness of the EFFIVLM-BENCH. 

In addition, the authors did a good job of addressing reviewers’ comments. I hope the authors reflect on the responses in the next version.
Summary of reasons to publish: 1. The presented EFFIVLM-BENCH is valuable for the community.
2. The experiments are extensive including many acceleration methods.