Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective
Authors: Lei Lei
Jie Gu
Xiaokang Ma
Chu Tang
Jingmin Chen
Tong Xu
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal large language model; token compression
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Existing Multimodal Large Language Models (MLLMs) process a large number of visual tokens, leading to significant computational costs and inefficiency. Instruction-related visual token compression demonstrates strong task relevance, which aligns well with MLLMs’ ultimate goal of instruction following. Previous works generally assume that visual tokens achieve better vision–language alignment in the shallow layers of LLMs, which have led to task-related token compression being primarily applied in intermediate LLM layers. In contrast, our study reveals that with proper selection, task-related token compression is feasible at the input stage of LLM with negligible performance loss. This new paradigm significantly reduces task-irrelevant visual tokens and its model-agnostic design enables application without modifying the LLM architecture. Specifically, we suggest that explainability methods for transformer-based architechtures can evaluate the global importance of each visual token with respect to the given instruction, which can effectively guide the task-related token compression for MLLMs. Furthermore, we propose to learn a mapping from the attention map of the first LLM layer to the explanation results, thereby avoiding the need for a full inference pass. Interestingly, this mapping can be learned using a simple and lightweight convolutional network, whose training is efficient and independent of MLLMs. Extensive experiments on 13 image and video benchmarks across three leading MLLMs (Qwen2-VL, LLaVA-OneVision, and VILA1.5) demonstrate the remarkable effectiveness and strong generalization of our approach. Additionally, our new compression paradigm achieves faster inference with reductions in both prefilling time and KV-cache memory.

Published as a conference paper at ICLR 2026
TASK-RELATED TOKEN COMPRESSION IN MULTI- MODAL LARGE LANGUAGE MODELS FROM AN EX- PLAINABILITY PERSPECTIVE
Lei Leit.3+ Jie Gu
Chu Tang² Jingmin Chen University of Science and Technology of China Rigghtly Robotics
Xiaokang Ma²
Tong Xu
3 Shanghai Innovation Institute
f11y,66,
edu.en; cn; {igu, xma, chu, tang, jingmin
ABSTRACT
Existing Multimodal Large Language Models (MLLMs) process a large num- ber of visual tokens, leading significant computational costs and inefficiency. Instructiin-rellted visual token compression demonstrates strong task relevance, which aligns well with MLLMs' ultimate goal of instruction following. Previous works generally assume that visual tokens achieve better vision-language align- ment in the shallow layers of LLMs, which have led task-related token compres- sion being primarily applied in intermediate LLM layers. In contrast, our study reveals that with proper selection, task-related token compression is feasible the input stage of LLM with negligible performance loss. This new paradigm sig- nificantly reduces task-irrelevant visual tokens and its model-agnostic design en- ables application without modifying the LLM architecture. Specifically, sug- gest that explainability methods for transformer-basee architechtures can evaluate the global importance of each visual token with respect to the given instruction, which can effectively guide the task-related token compression for MLLMs. Fur- thermore, we propose to learn mapping from the attention map of the first LLM layer to the explanation results, thereby avoiding the need for a full inference pass. Interestingly, this mapping can be learned using simple and lightweight convolu- tional network, whose training is efficient and independent of MLLMs. Extensive experiments on 13 image and video benchmarks across three leading MLLMs VL, LLaVA-One Vision, and VILA1.5) demonstrate the remarkable ef- fectiveness and strong generalization of our approach. Additionally, our new com- pression paradigm achieves faster inference with reductions in both prefilling time and K V-cache memory.
INTRODUCTION
With large language models (LLMs) providing a strong foundation Brown et (2020); OpenAl
2023); ; Touvron et al. 2023); Bi et al. (024), research on multimodal large language models (MLLMs)) gained significant momentum Liu et al. (2023); Chen et al. (202)): Zhu et al. (2024);
Bai et al. (2023). Considerable progress has been achieved in various image- and video-related tasks Chen et al. (2024d); Anil et al. (2023). A common paradigm among existing MLLMs is to jointly feed visual tokens (generated by vision encoder) and textual tokens into the LLM for cross- modal alignment and integration Liu et al. (2023); Zhu et al. (0024): Li et al. (202bb). This paradigm introduces substantial memory and computational overhead due to the high volume of visual tokens, which grows rapidly with higher resolutions or frame rates Wang et al. (2024b) ):: Zhang et al. (022a)). Consequently, there is pressing need for effective token compression techniques.
Previous exploration of visual token compression methods can be roughly divided into two cat- egories. The first aims to obtain more compact and fewer visual representations (especially for videos) in a aaskiinttruciion-anooo
manner (independent of LLM) Bolya et al. (2023): Yang et al.
WWrk done during interships in Rightly Robotics.
Corresponning author.

---

Published as a conference paper at ICLR 2026
2024); Shen et al. (2025b); Wang et al. (2025) : Shen al. (2025a). We argue that visual represen- tations are an integral part of MLLMs and serve as the foundation for achieving strong performance and generalization. Many state-of-the-an thtetattSSOOAAA MILMs already incorporate built-in, task-agnostic compression mechanisms (e.g., spatial and temporal pooling) instead of relying on separate com- pression techniques applied afterward Wang al. (2024b a): [i et al. (2024c). The second category focuses on selecting visual tokens that are most relevant to the given instruction. FastV Chen et al. (2024b) is a pioneering work that highlights the importance of retaining all shallow-layer visual tokens in LLMs for lossless compression. While this assumption has been adopted by many subse- quent studies Zhang et al.] (2024b); [Zhao et al. (2024); Xing et al. (2024): Taa al. (2025); Huang et al. (2024); Wen et al. (2025). we believe it remains open to question: Are all visual tokens in the shullow layers of LLM truly indispensable for tusk-related compression?
This paper seeks to answer the question of whether an effective task-related token compression approach prior to the LLMM exists but remains undiscovered, or whether it is inherently infeasible. To the best of our knowledge, our work is among the earliest efforts to investigate this issue. We first explore the use of explainability methods to assess visual token importance with respect to the architecture generally iteratively update
instruction. Explainability methods for transformer-base relevance map across layers using gradient-weighted multi-head attentions attentions/Chefer [Chefer et al.] (2021b)a) a),
which effectively captures the global relevance scores of visual tokens. Relevance scores indicating the contributions of input tokens to output can be used to rank and prune less important visual tokens for compression. Comprehensive experiments conducted on both image and video data across three representative MLLMs demonstrate the effectiveness of such compressor. The results indicate that, with proper selection, pruning tokens that are not relevant to the task at the LLM input stage is indeed feasible. Moreover, unlike previous works motivated by observations derived from specific network architectures (e.g., LLaVA) Chen al. (2024b); Tan et al. (2025), which limits their generality and transferability, our explainability-basse approach is broadly applicable. Rather than relying on the behaviors of specific models, it leverages the inherent characteristics of the applied model.
After validating that the explanation results are effective compression indicators, lightweight model capable of generating an alternative to the relevance map is further needed to enable effi- cient and practical deployment. Interestingly, this goal can be achieved by training a simple fully convolutional network that predicts relevance based on the first-layer attention map of the LLM. The training process is highly efficient (e.g., training a 5-layer network using only 10K image data) and does not involve any changes to the MLLM itself. Using the predicted relevance, token compression can be performed prior the prefill phase with negligible extra computational cost. As result, both computational and memory overhead during inference are significantly reduced, with no modifica- tions required to either the prefill or decode phases. Last but not least, our approach generalizes well across various architectures, benefiting from the broadly applicable nature of explainability methods and the -agnMsti design of the auxiliary training.
To thoroughly assess the capability of our approach, we apply it to three prominent models with dif- ferent architectures and visual representations: VILA1.5, LLaVA- -ne Vision, and Qwen2-VL. VL. We include 13 widely used image and video benchmarks that span a wide range of visual complexities and tasks, ensuring comprehensive evaluation. Notably, our method achieves significant compres- sion by pruning 75% of video tokens while retaining more than 97% of the original performance across all benchmarks for both VILAI.5 and ELaVA-OneVision Vision. It also performs well on image tasks, where up to 50% of image tokens can be removed with only minimal performance drop: maintaining over 96% of baseline performance for Qwen2-VL. VL. and LLaVA VA-One Vision.
In summary, the contributions of the work are threefold: (i) reveal that explainability methods can well evaluate the importance of visual tokens, enabling effective token compression. (ii) propose highly efficient token compressor by learning from explanation results. It allows token compres- sion to be performed before the LLM,, significantly reducing inference costs at both the prefill and decode phases. (iii) Validate the effectiveness and generalization of our method through extensive experiments on a wide range of image and video benchmarks across different Icading MLLMs.
2 RELATED WORK
Multimodal Large Language Models. Benefiting from advancements in large language models
(LLMMs) OpenAl 2023); Touvron et al. (2023) Bi et al. (2024), multimodal large language models
2

---

Published as a conference paper at ICLR 2026
Trace Back Relevance Maps
VM
F(x)
Ingut viduo/image V
Top-N%
comprased
vidoolmagge P
User Instraction
Bokenizer
Q
VM
X
R
Explainahiity -haued Comparaner
System prompt
toketiour
Token Compr ession prior to LM
takeniner
Figure Overview our method. The top portion illustrates the details of our explainability- based compression approach: an explainability method can reveal the important visual tokens (first row, Section 3.2p; a lightweight model can then be trained to approximate this explainability and serve compression indicator (second row, Section 3.3 The bottom portion shows general inference framework for MLLMs, where the resulting compressor applied at the LLM input stage.
(MLLMs) have gained considerable attention due to the powerful ability in multi modal understand- ing and reasoning Liu et al. (2023); Chen et al. (2023): Bai et al. ; Chen et al. (0024)) (Anil al. (202)). Recent advances Li et al. (2024a): Wang al. (2024b); /Zhang al. (2024a) tend to handle images with higher resolution and videos with more frames, which significantly increases the number of visual tokens and thus the computational burden. This reveals the necessity for ken compression strategies that can balance efficiency and effectiveness. Our work proposes new token compression paradigm, which removes insk-irreeevant visual tokens the LLM input stage, significantly reducing computational costs without sacrificing performance.
Visual Token Compression. Existing visual token compression methods for MLLMs can be broadly categorized into: maskiinstruction-agnooo
compression [Bolya et al. (2023); Yang et al. 2024 Shen et al. (2025b); Alvar et al.) (2025) and task/instruction-relaeed compression Chen et al. 2024b): Xing et al. (2024), Huang et al. (2024); Wen et al. (2025). The first category of meth- ods typically introduces additional modules to merge redundant visual tokens based on similarity, addressing the limitations of existing models. However, many recent works have developed tech- niques to obtain more compact visual representations when building MLLMs Wang et al. (2024b); compression
Chen al. (2024a). Notably, task/instruction-eelaan
further reduce the number of visual tokens on models already incorporate built-in compression mechanisms, offering greater potential for efficiency gains. FastV Chen et al. (2024b) represents typical method of the second category, which rely on shallow-layer attention maps the L.LM for compression. In this work, we explores the feasibility of an effective task-related token compression prior to the LLM, which functions independently of the architecture and can be applied broadly across different MLLMs.
METHOD
3.1 BACKGROUND AND MOTIVATION
Current Multimodal Large Language Models (MLLMs) typically follow a framework in which vi- sion encoder is incorporated to encode visual signals into sequence of tokens Liu et al. (2023); Bai et al. 2023); Chen et al. (2023). Specifically, multiple frames or patches are sampled from video or image, and their corresponding visual tokens encoded. These visual tokens then flattened and concatenated with textual prompt tokens before being fed into a Large Language Model (LLM) generate response. Formally, let V be the video image, and let VM and LM represent the
3

---

Published as a conference paper at ICLR 2026
User Iustruction According video, what do the theee carred lines criendin Pave bettore up zymbolize? A. fewr B. Stean, C. Popor, D.A
Vidvo frumee
Respons: C. lipar
Llave- Onettrion
Quon2 PL
VILA15
Figure 2: Visualization of R. obtained via the explainability method (left) and the correspond- ing token pruning results (right). Based on Rg the top 50% of visual tokens are retained and the rest masked in white. Given the instruction querying the "three curved lines". Rt highlights the regions corresponding to the "three curved lines", guiding the selective retention of the associated visual tokens. All three MLLMs generate the correct answer using only the retained tokens. More visualization cases are presented in Appendix
vision encoder and the language model, respectively. The visual token embeddings E,, can be rep- resented as E.. (V) E R N. xC where N₂ is the number of visual tokens and C' is the feature dimension. Let E, R.N and En E N. denote the token embeddings of the system prompt and user instruction, respectively. By feeding E, together with E, and E., into the LLM, a textual response is generated, i.e. Y (M,E,, Ey. Eu). An additional compression module Comp can be introduced to prune visual tokens. while keeping the MLLM architectre-iinluuinng both the VM and the LM unnhhaggee during this pruning process.
Eg can be considered as purppoe representations of visual signals that are task/instruction- agnostic. Recent advances have already incorporated built-in compression mechanisms (e.g.. spatial and temporal pooling) to reduce task-agnostic redundancy. enabling the construction of more com- pact E, when building MLLMs Wang et al. (2024b a); Li et al. (2024c). Rather than applying another round of task-agnostic compression to E, (as methods Bolya al. (2023); Shen et al. (025a)). our objective to further address task-related redundancy by evaluating the importance of each token in E. with respect to given instruction and selectively pruning those that contribute less. Recent advances have developed techniques to reduce the number of visual tokens to obtain a more compact E, when building MILMMs Wang et al. (2024b a): Li et al. (202c)). Therefore, instead of further compressing E.,. in isolation (namely task-agnostic compresssion methods like [Bolya et al.) 2023 Shen et al. (2025a) ), our objective is assess the importance of each token in E, with respect to a given instruction, and subsequently prune those that are less essential. Moreover, we investigate how to perform token compression prior to LLM computation, i.e., compressing E, to E. = Comp F,, Fu) E R.N.CC and then computing Y LM( (F,, E.,, F... where N, is much smaller than N,- In contrast previous methods Chen et al. (2024b); HHuang et al.) (2024), method does not require any modifications to the prefill and decode phases during inference, and computational and memory overhead can be significantly reduced in both phases.
The details of our approach are presented below. In Section 3.2 we introduce explainability methods to assess the importance of visual tokens and guide token compression. A learning mechanism is then proposed to predict the explanation results in Section 3.3 which ultimately enables effective token compression the LLM input stage.
3.2 TOKEN COMPRESSION WITH EXPLAINABILITY
To reduce task-related redundancy the token level, need estimate the contribution of each visual token to the model response. Explainability methods for LLMs facilitate this goal by gen- erating a relevance map through the integration of attention weights and corresponding gradients, effectively revealing where the model genuinely focuses. The resulting relevance map highlights the
A cross-modal projector is commonly employed in such architectures. For notational simplicity, we denote both the vision encoder and the projector by VM.
4

---

Published as a conference paper at ICLR 2026
contributions, enabling to rank and prune these visual tokens accordingly. The pipeline for this section is shown in the first row of Figure
Relevance Maps by Explainability Method. We adopt generie explainability method similar to Yao et al. (2024); Chefer et al. (2021) to compute the relevance of the response-to-vision The relevance values reveal the distribution of importance across visual tokens utilized by the LLM. Without loss of generality, assume that the LLM in an MLLM has L layers, and denote the generated trace back the semantic sequence of textual tokens as Y {yo, .1....., 1/--1) Specifically, relevance flow from generated tokens to raw visual inputs. For each at the 1-th generation step, the relevance map Re is first initialized as an identity matrix and then iteratively updated across layers. Denote A, and V as the multi- attention map and the corresponding gradients in the 1-th layer, obtained during the forward and backward passes, respectively. R₁ updated as:
R = R2 + (A(( @ VAA+) RR:
(1)
where O represents Hadamard product, and E, is the mean across the heads dimension. The update is performed from the 0-th layer to the last layer. In the end, the relevance of y to visual signals can be extracted by indexing the corresponding positions in the last row of R,, that is, R N, + Nv.. Finally, we aggregate visual relevance across all time steps t by averaging, obtaining the overall visual relevance scores R. E RIXN, with respect the current response. A detailed analysis of Eq. is provided in Appendix [A] This well-grounded importance assessment Rg can then be used rank and select visual tokens.
Visual Token Compression Using Relevance Scores. The importance of visual tokens related to the instruction can be ranked according R, We can prune the less important visual tokens down to target count of Ng. resulting compressed token embeddings E, as LLM input.
Observation. We visualize Ry and the corresponding token pruning results for LLaVA-One Vision, Qwen2-VL, and VILA1.5 Figure 2 While visualizations from different MILMS show varying appearances due to differences in how each model processes visual input, they exhibit clear com- mon patterns - R, for each model consistently highlights the regions corresponding to the "three curved lines" in the video, demonstrating the robustness of our method. Moreover, experimental results show that retaining 50% of the original visual tokens based on R, preserves over 98% of the performance on image benchmarks and 99% video benchmarks (see Section 4.2 for details). We draw the following conclusion: the explanation results faithfully capture the visual information essential for the MLLM to answer the question, and retaining only the corresponding visual tokens does not compromise model performance.
3.3 EXPLAINABILITY-BAN COMPRESSOR LEARNING
The relevance map offers valuable insights into achieving token compression at the LLM input level. However, its practical application is limited by the fact that R, is derived post-hoc- only after the model has already generated the output. To address this limitation, we propose to approximate Rg using a standalone module trained independently of the MLLM. By learning to capture attention patterns and generate relevance estimates R.,. this module enables token compression prior LLM without modifying or retraining the MILMM Importantly, this module prioritizes efficiency: it is lightweight, requires small amout of training data, and can be trained quickly, making it practically applicable. The pipeline for this section is shown in the second row of Figure
Model Architecture. As shown in Eq. D the relevance map is essentially obtained by aggregat- ing attention maps, suggesting that learning a mapping from attention maps to relevance maps is promising. Interestingly yet reasonably, we find in practice that simple convolutional network applied to the first-layer attention suffices, which guarantees the compressor's efficiency in terms of model size, training time, and computation (implementation details and efficiency analysis can be found in Appendix C and Appendix D) Formally, let A be the first-layer attention map. Similar Chen et al. (2024b): Zhao et al. (2024), we focus specifically on the attention scores that visual to- kens receive from textual instruction tokens. Accordingly, we extract the submap A E R.N. xN. by indexing the corresponding positions. We then average the N,, scores for each visual token to obtain a compact representation, resulting in AQ E RIIN. N. This averaged attention vector AQ is
We omit the head dimension for notational simplicity.
5

---

Published as a conference paper at ICLR 2026
subsequently fed into ID convolutional model fo to predict visual relevance:
Rg fo (A
Note that a softmax operation is applied at the end of fo. making R. a probability distribution. In addition, separate instance of fe is used for each MLLM, because trained approximate the explainability patterns specific to that particular MLLM.
Training Objectives. We process R. into the training label It by masking the bottom 50% val- ues Gu et al. (2021) and normalizing the remainder into a probability distribution. Instead of soft- max waxx yields near-uniform values due to the closeness of raw scores-we we normalize by di- viding each score by the total, thereby preserving relative differences. Finally, given Rt, and Ry, the Kullback-Leibe (KL) divergence is used to measure the difference, defining the loss function:
LKL (R,
Oberservation. The learned fo can be seamlessly integrated into the MLLM inference pipeline to generate R. which can guide the token compression. As shown in Figure visualization of R2 and R. is given in the first and second rows, along with their corresponding pruning results, respectively. One can see that R, closely resembles Rg. Important visual regions related to the question are highlighted in both maps. This observation provides evidence that the lightweight model fo can indeed be efficiently and effectively trained to approximate Rv. allowing lossless token compression at the LLM input stage. Quantitative experimental results further support this conclusion (see Section 4.3 for details).
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models. Experiments are conducted on three MLLMs with different architectures for extensive validation, i.e., LLa VA OneVinion-7B Li et al. (024a), Qwen2-VL-7B [Wang et al. (2024b) and VILAI.5-8B Liu et al. (024c). These models exemplify recent advances in handling high-resolution and long visual inputs. LLaVA-One
and Qwen2-VL VL support arbitrary resolution/length, with Qwen2-VL VL further introducing dynamic resolution and token aggregation for compact vi- sual representations. VILAI.5 applies spatial token compression when processing images or video frames. They thus provide strong basis for evaluating our task-related compression, which reduces instruction-related redundancy beyond their built-in instruction-agnosti compression.
Evaluation Tasks. We thoroughly evaluate our method on 13 widely used image and video bench- marks. For image tasks, MME Fu et al. (2023) (all-round capability). MMStar Chen et al. (2024c)
(data contamination), MMVet Yu et al. (2024) (subjective evaluation). SEED-Bench Li et al. (002a)) (all-round capability), POPE Li et al. (202cc) (hallucination evaluation). TextVQA [Singh et al. 2019) (OCR reasoning). and MMBench Liu et al. (2024a) (all-round capability) are included, cov- ering various aspects of MILMM performance. For video evaluation, select Video-MME(woo sub.) Fu et al. (022)), MVBench Li et al. (2024b), MMBench- Video Fang et al. (2024), NExT- QA Xiao al. 2021). ) and ActivityNetQA Yu et al. (2019), providing comprehensive coverage of video understanding abilities across different tasks and video durations. For comparison, we take existing SOTA task-related token compression methods such as Fast Chen et al. (2024b), Pyramid- Drop et al. (2024), Dart Wen et al. (2025) as the primary baselines, which perform compression in the LLM intermediate layers on visual tokens fused with textual information in the shallow layers.
Implementation Details. Training So- fo is implemented as five-layer convolutional network, with each layer employing a 1D depthwise separable convolution Chollet (2017). The training data of fo is collected from general-domain high-quality datasets: subset of LLaVA-Video Zhang (et al. (2024c) for videos and a subset of Infinity-MM. MM Gu et al. (2024) for images. To ensure high diversity, we adopt sampling strategy that covers wide range of task types and video duration. Implementation details including the sampling strategy of training data, the generation of R,, and the training procedure of fo are provided in Appendix
Inference. We conducted all experiments on A100 GPUs (80GB) and used VLMEvalKit Duan
al. 2024 for benchmarking. Details of the inference settings can be found in Appendix

---

Published as a conference paper at ICLR 2026
Table 1: The relevance R. effectively guides token compression under different retention ra- tios. Avg. means the average of performance preservation ratios across all image/video benmarks.
Method
Retention Ratio
Image Benchmark
Avg(%)
Video Benchmark
Avg.%)
MMR
MMStar
MMVet
Video-MME:
MYBench
MMBBenhh-V
LLaVA-One OiNN-Oo
II LVVA-OneVisio
50%
1974.2
59.7
47.2
98.1
54.3
41.1
0.40
99.5
w/GAF-Based Compressor
25%
1977.3
59.3
47.0
97.8
53.8
40.9
0.40
99.1
Owen2-VL
51.0
Qwen2-VI. VI.
50%
2297.1
60.3
53.2
99.5
51.0
50.7
1.19
99.1
wiGaF-Basee Compressor
25%
2299.1 1
58.7
51.7
97.7
50.3
49.7
1.17
97.5
1700.3
VII A1.5
50%
1740.5
37.2
38.0
98.4
47.9
34.2
1.26
99.8
wiGeAF-aased Compressor
25%
1722.1 1
35.7
35.6
94.7
47.1
35.1
1.28
100.7
Table 2: Compare explainability-bass. compressor image benchmarks. Values marked with in Retention Ratio denote the average retention ratio across LLM layers due to multi-stage com- pression in PDrop.
Method
Retention Ratio
FLOPs
MME
MMStar
MMVet
SRED
POPE
TextVQa MMBeach
Avg(S) %
LLaVA-OV
100%
100%
1997.7
60.5
48.7
76.7
87.4
69.3
80.6
100
LLVA--VV w/ FistV
50%
0.51%
679.2
42.7
78.8
60.1
10.8
54.4
47.7
56.0
LLAVA-OV w/ Pdrop
51%
0.51x
1974.7
55.4
41.7
74.8
87.0
64.9
79.9
95.1
LLAVA-OV Dart
50%
0.51
1977.5
55.2
12.3
74.3
85.8
65.4
79.2
95.0
LLaVA-OV
0.48x
1980.8
75.3
MaYA-OV FastV
25%
0.27x
527.7
20.6
10.6
42.8
47.3
PDrop
20.4
VLLVA-OV
0.27x
1905.0
26.2
0.24x
Qwen2-VL
1.00x
75.8
QwenZ-VL FastV
$6.0
Qwen2-VL w/ PDrop
51%*
0.51x
2283.1
55.4
46.3
23.0
86.3
80.9
79.7
95.2
Qwen2-VL VL w/ Durt
50%
0,51x
2290.0
55.5
19.4
72.4
86.6
82.7
80.1
96.1
Owen2-VL. m/ Ours
50%
0.49%
2288.3
55.9
51.9
73,2
86.4
80,9
97.4
QwenZ-VL LL w/ FustV
25%
0.27
1415.3
37.6
31.4
51.4
77.6
63.2
39.1
66.0
QwenZ-VL VL w/ PDrop
25%
0,25x
2216.3
51.1
42.3
67.4
83.2
75.4
77.0
89.7
Qwen2.VV. VL. w/ Damt
25%
0.27x
2184.5
51.3
45.6
68.2
84.3
76.7
27.5
91.1
Owen2-VL VL w/ Ours
25%
0.24x
2280.9
$1.8
47.3
67.9
84.8
80.0
77.7
92.9
Following prior works CChn et al. (2024b); Ye et al. (2025). for evaluating inference efficiency. For fair comparison, we configure baselines for comparable FLOPS (e.g., pruning the 2nd layer for FastV). Our method achieves significant reduction in inference cost with only negligible additional computation. We provide a comprehensive analysis of FLOPs of method, please refer Appendix D
report FLOPs the primary metric
4.2 EFFECTIVENESS OF COMPRESSION WITH EXPLAINABILLIT
We conduct experiments to verify whether the explanation results can guide token compression, i.e., compressing E, to E, according to R, and then feeding É, into LM to generate response. To assess effectiveness and generalization, we apply the method to three state-of-the-ar MLLMs and test them three image and three video benchmarks.
Table reports the results under retention ratios of 50% and 25%. The strong performance across multiple models and benchmarks demonstrates the effectiveness and broad applicability of such token compressor. For Qwen2-VL, reducing visual tokens by 50% still preserves over 99% of the original performance both image and video tasks. -LaVA-OneVision retains 99.19 19.1% its video performance even with only 25% of tokens. VILA reduces visual tokens to 98 per image frame 50% retention, while maintaining 98% of the original image performance and nearly unchanged video performance. These observations indicate that token compression based on relevance R. effectively preserves the visual tokens essential for MILMS answer the question.
an explainability-bon

---

Published as a conference paper at ICLR 2026
Table Compare explainability-basee compressor on video benchmarks. As videos generally exhibit greater visual redundancy, we also evaluate a lower retention ratio 10% to further assess compression robustness, with detailed results reported in the Appendix
Method
Retention Ratio
FLOPs
Video-MME
MVBench
MMBench- Video
Next-QA
Activity-QA
Avg.
multi-cholce open-ended
LLaVA-OV
100%
1.00x
53.6
41.2
0.41
79.2
49.0
56.9
100
LLaVA-OV FastV
50%
0.48x
42.3
25.0
0.35
66.5
36.0
48.5
78.0
LLaVA-OV w/ PDrop
0.47x
52.7
40.2
0.36
78.4
48.2
56.0
96.6
LLaVA-OV w/ Dart
50%
0.48x
53.2
40.0
0.36
78.0
49.0
56.2
96.9
LLaVA-OV w/ Ours
50%
0.46 x
53.4
40.5
0,43
78.6
49.7
56.5
100.4
FastV
39.6
23.6
1.LVV-OVV w/ PDmp
25%*
0.24
50.8
38.2
0.35
76.3
48.2
53.5
93.6
11aVA-OV w/ Dart
25%
0.25
51,5
38.7
0.33
76.6
47.0
55.1
93.3
LLaVA-OV
51.3
39,0
0,42
Qwen2-V1.
100%
1.00x
50.4
51.0
1.23
76.8
45.5
53.6
100
Qwen2-VT. VI. w/ FastV
50%
0.48x
32.4
36.1
0.52
43.9
28.3
38.2
61.4
Qwen2-VI VI. Prrop
50%*
0.47x
48.9
49.6
1.14
75.2
45.4
50.8
96.6
Qwen2-VT,
0.48
Qwen2-VL VL. w/ Ours
50%
0.46
50.0
49.8
1.18
75.6
45.9
52.4
98.3
Qwen2-Ve VL VastV
25%
0.25x
31.2
36.1
0,48
42.0
26.8
35.1
58.5
Qwen2-VL PDrop
25%
47.3
46.2
Qwen2-VL. VL. w/ Dart
25%
0.25
47.4
47.1
1.10
74.1
44.5
49.2
93.7
Qwen2-VI. VL w/ Ours
25%
0.22
48.1
46.7
1.11
74.2
44.3
$0.5
94,2
VILA
100%
1.00x
34.0
VILA FastV
50%
0.49x
42.2
20.7
0.98
62.9
36.8
47.1
80.1
VILA w/ PDrup
50%*
0.49x
47.3
35.0
1.22
69.4
45.8
55.1
99.2
VILA Dart
50%
0.49%
46.1
34.7
1.25
69.2
46.5
55.2
99.2
VILA Ours
50%
0.47
35.2
1.25
VILA w/ FastV
25%
0.26x
41.4
20.5
0.97
61.5
36.4
46.8
79.0
VILA PDrop
25%
0.26x
45.2
33.6
1.24
68.2
45.1
54.8
97.4
VILA
34.6
VILA w/ Ours
25%
0.23
45.5
35.6
1.22
69.4
46.4
54.8
99.0
4.3
EFFECTIVENESS OF EXPLAINARILITY
COMPRESSOR LEARNING
The performance of the Rudgudded token compressor is evaluated in this section. πč, is generated by the learned fe. and the token pruning is performed accordingly before the LLM computation. Five image and six video benchmarks included for evaluation.
Performance Comparison. Table presents the results of VA- Vision and Qwen2-Vy 2L under different token compression retention ratios on image benchmarks. We exclude VILA here because uses a fixed and relatively small number of image tokens, making compression less meaningful. As shown in the table, retention rate of 50%, our compressor demonstrates overall superiority over the baselines at comparable FLOOS, achieving average improvements of 2.6% and 1 1.2% across all benchmarks for ALLVA-OOn Vision and Qwen2-VI, respectively. When the retention rate is further reduced to 25%, the performance gains increase to 4.7% and 1. 1.6%, highlighting the enhanced robustness of method under higher compression rates, We conducted additional experiments on more image benchmarks; please refer to the Appendix F
In Table we evaluate the compression performance of LLaVA-OOeVVision Qwen2- VL, and VILA on video benchmarks. We make several observations. First, our compressor consistently outper- forms baselines at comparable FLOPs, regardless of the model and retention ratio. Both LLaVA- Vision and VILA are able to maintain 100% performance when 50% of the visual tokens are pruned. Second, VILA exhibits the smallest performance drop, while Qwen2- VI. shows the largest, likely due to its attention patterns being harder to capture. Importantly, sion can still further reduce token redundancy on both Qwen2-VL and VILA, demonstrating that complements the task-agnostic compression already presented in these models. Finally, comparing the results in Tables [ 2] and 3, the performance degradation from the R. Rg-guided compressor to the R, -guided compressor more pronounced in image tasks. This is likely also due to the greater redundancy in videos, which reduces the learning difficulty.
task-related compres-
8

---

Published as a conference paper at ICLR 2026
Part Cadly
12.4%%
435% HAHS
HOMSI 11.%% HEADTS
AJ5G KAMI
#ioo
FarViD
MLS
Nextjsout Mi
Vaunsip
PLOPEED
JMP 10.0%
-1.%
11.4%
-..% 183%
AS-i204 4.55 1441%
MVBBccc
Radar Vhineeeee Qwen2-VV 25% stesal
(b) Performance preservation")) tnetheds based LaVA--ueeViioo
Comparison results on larger images and longer videos. Performance preservation ratio measures the performance retained relative to Vanilla. Gray text denotes the gap from Vanilla and green text highlights improvements over the FastV Tasttopt.Conig Con.Config
Figure 3:
Table 4: Efficiency analysis based on Qwen2- VL on MMStar. We evaluate the inference costs in terms of total inference time, prefilling time, FLOPs, and KV cache memory. KV cache memory is computed with consideration of the Grouped Query Attention (GQA) used practical inference.
Method
Retention Ratio
FLOP
Total Inference
Prefilling Time
KV Cache
Total Speedup
Prefilling Speedup
MMStar
Time
Qwen2-Vr
100%
1,00x
15min24s
6min36s
71.2MB
1.00x
1.00x
61.1
Qwen2-VL w/ F
25%
0,27x
12min196 19s
4min14s
19.7MB
1.25x
1,56x
39.6
Qwen2-VL w/ PDrop
25%*
0,25x
12min15s
4min10s
18.
1.26x
1.58x
53.1
Qwen2-VL w/ Durt
25%
0,30x
12min20s
4min16s
21.6MB
1.25x
1.55x
54.3
Qwen2-VL VL w/ Ours
25%
0.24x
12min16s
4min08s
17.8MB
1.26x
1.60
55.8
Applying to Larger Images and Longer Videos. We evaluate the generalization of directly apply- ing the trained fe on larger images and longer videos; experimental results are shown in Figure 3 with configurations detailed in Appendix C Sub-figure(a) show the compression performance image and 6 video benchmarks based on LLLVA-One Vision and Qwen2-VL. Our method consis- tently outperforms Fast'V and achieves higher average performance than its optimal configuration, demonstrating strong generalization to larger images and longer videos than it seen during training. Detailed comparison results on these 10 benchmarks are provided in the Appendix F
Sub-figure(b) show the comparisons on two challenging benchmarks, i.e., MMStar and Bench. Several strong methods are introduced for comparison: Prune Huang al. (202)). FastVID et al. (2025a). VisionZip Yang et al. (2024). and PyramidDrop Xing et al. (2024). Our approach achieves SOTA performance with the lowest FLOPs. Remarkably, even directly applying the trained fo to longer videos with more frames, it still performs favorably compared to meth- ods specifically designed for videos (i.e., FastVID and PruneVID). Beyond superior performance, our lightweight compressor significantly improves MLLM inference efficiency with negligible ad- ditional cost. We follow Dart Wen et al. (2025) and report efficiency terms of total inference time, prefilling time, FLOPs, and KV cache memory, shown in Table 4 Our compressor achieves both the best performance and highest efficiency. The prefill-staee acceleration (1.60x) and reduced KV cache footprint (71.2MB - -17.8MB) enable efficient processing in prefill and decode stages, keeping total inference time comparable to other methods (see Appendix F] for more results).
Analysis and Discussion. We find it interesting that lightweight fo can achieve strong perfor- mance, thus we offer further discussion here: (i) The convolutional mapper Jo learns a structurally matched inpul-oulput mapping. As shown in Eq. D the relevance map Rg is essentially an iterative composition of attention maps A with their task-specific gradients across layers. Therefore, using A to predict R natural and intuitively well-motivated. Visualization results in Appendix show that fo exhibits task- behavior, adaptively focusing on the relevant visual regions according to the instruction. (ii) Shallow-layer attention maps contain vital information. strated in several works Chen et al. (2024b); Xing et al. (2024) that visual tokens make a greater contribution to output generation in the shallow layers compared to the deeper layers. Our experi- mental results are consistent with the finding, First, using the first-layer attention map the input of fo already provides a strong guidance for token compression. Moreover, the attention maps from shallow layers perform well overall; for detailed ablation results, please refer to the Appendix E (iii) We aim not to learn an identical R,, but rather its relatively large values. The loss function design in Section 3.3 masked the bottom 50% of label values, simplifying the learning task to the
has been demon-
9

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study on explainability methods for relevance map generation. We evaluate two strategies for aggregating multi-head attention maps gradient-weighte s summation and simple averaging to generate relevance maps for token compression on video and image benchmarks.
Model
Method
Rentation Ratio
Image Benchmark
Video Benchmark
Avg. %
MME
MMStar
MMVet
Video-MME
MVBench
MMBench-V
Vanilla
100%
1997.7
60.5
48.7
53.6
41.2
0.41
100
LLaVA-
Mean-weighted
1974.5
58.5
45,9
53.6
40.8
0.39
97.3
OneVision
50%
Grad-weighted
1974.2
47.2
98.8
Vanilla
100%
2295.1
60.4
54.0
50.4
51.0
1.23
100
Qwen2-VI.
Mean weighted
2300.6
58.2
49.2
49.9
49.9
1,15
96.3
Grad-weighted
2297.1
60,3
53.2
51.0
50.7
1.19
99.3
100%
1700.3
38.7
39.3
VILA1.5
Mean-weighted
1720.8
38.0
34.2
48.0
34.1
1.20
96.9
50%
Grad-weighted
1740.5
37.2
38.0
47.9
34,2
1.26
99.1
distribution of the top 50%. Our goal not to learn a precise one-to-one input-output mapping. Instead, the learning target is only to achieve identifying which regions are relevant to the instruc- tion, thereby enabling token pruning. One can observe the visualization results Appendix [ for evidence. The pruning results (based on R.) and the convolutional network's 's learned (based R.) are not identical, yet retains the same task-related regions and produces correct answers.
4.4 ABLATION STUDY
As shown in Eq. I the relevance map is updated by aggregating attention maps across layers, where the multiple heads in each layer are combined either via simple averaging or gradient-weighter averaging (used in approach). Table 3 shows that employing gradient-weige weighted aggregation to generate R,, for token compression performs consistently better than simple averaging across image and video benchmarks. A reasonable explanation is that differing contributions of attention heads make simple averaging prone to distorting relevance maps Voita et al. (2019). We also include ablation studies of different configurations for training fo in Appendix E specifically investigating (i) the effect of varying the depth of the convolutional network and (ii) the influence of selecting different attention layer as input for the compressor.
5 CONCLUSION
In this work, we demonstrate the feasibility of task-related visual token compression at the LLM in- put stage. We first demonstrate experimentally that the relevance scores derived from explainability methods well evaluate the task-related importance of visual tokens, which can be used for effective token compression. To enable efficient and practical deployment, network to learn a mapping from the L.M first-layer attention maps to the explainability-deen dderive relevance scores. Using the predicted relevance scores from lightweight model, token compression can be performed prior to the LLM. Extensive experiments demonstrate the effectiveness and gener- alizability of our raskerelated token compression method. Since the relevance scores are obtained via backward computations, their generation is eesource-intensio compressor training to high-resolution images or long video sequences. Future work will explore stronger compressors and the use of relevance scores guide token compression during training.
employ simple convolutional
This poses a challenge in scaling the
Reproducibility Statement. We provide the necessary information to facilitate reproducibility. Fx- and
perimental settings and implementation details are described in the Section 4.1 and Appendix [ All the datasets used our experiments are publicly available, and the preprocessing steps of training data are documented in Appendix
ACKNOWLEDGEMENTS
This work supported in part by the grants from National Science and Technology Major Project (No. 20232D0121144, and National Natural Science Foundation of China 13..2122 U22B2059)
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. In CVPR, 2025.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Sann-Bapistt Alayrac, et al. Gemini: A family of highly capable multimodal models. arXiv, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, et al. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv, 2023.
Oren Barkan, Edan Hauon, Avi Caciularu, Ori Katz, et al. Grad-sam: Explaining transformers via gradient self-attention maps. In CIKM, 2021.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, et al. Deepseek LLM: scaling open-source language models with longtermism. arXiv, 2024.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, et al. Token merging: Your vit but faster. In ICLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et Language models are few-shot learners. arXiv, 2020.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In CVPR, pp. 782-791, 2021a.
Hila Chefer, Shir Gur. and Lior Wolf. Generic attention modell explainability for interpreting bi- modal and encoder-decoder transformers. In ICCV, 202
Jieneng Chen, Luoxin Ye, Ju He, Zhaoyang Wang, and Daniel Khashabi. Efficient large multi-modal models via visual context compression. In NeurIPS, 2024a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, et al. An image is worth 1/2 tokens after layer 2: Blug-and-play inference acceleration for large vision- lssoonlanguage models. ECCV, 2024b.
Lin Chen, Jinsong Li. Xiaoyi Dong, Pan Zhang, al. Are on the right way for evaluating large vision-language models? NeurlPS, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, et al. Internvl: Scaling up vision foundation models and aligning generic visual-linguistic tasks. arXiv, 2023.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv, 2024d.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017.
Tri Dao. Daniel Y. Fu, Stefano Ermon, Atri Rudra, et al. Flashattention: Fast and memory--fficient exact attention with io-awareness. In NeurlPS, 2022.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In MM, 2024.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, et al. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. In NeurlPS, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv, 2023.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms video analysis. arXiv, 2024.
Jie Gu, Feng Wang, Qinghui Sun, Zhiquan Ye, et al. Exploiting behavioral consistence for universal user representation. In AAAI, 2021.
11

---

Published as a conference paper at ICLR 2026
Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, et al. Infinity-mmt Scaling multimodal perfor- mance with large-scale and high-quality instruction data. arXiv, 2024.
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, and Others. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018.
Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. arXiv, 2024.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, al. Llava-onevision: Easy visual task transfer. arXiv, 2024a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, et al. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024b.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, and Wayne Xin Zhao. Evaluating object ballucination in large vision- vision-language models. In EMNLP, 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, and Others. Mmbench: Is your multi-modal model an all-around player? ECCV, 2024a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang. and Others. Ocrbench: on the hidden mystery of OCR in large multimodal models. Sci. China Inf. Sci., 2024b.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, et al. NVILA: efficient frontier visual language models. arXiv, 2024c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, and Others. Learn to explain: Multimodal rea- soning via thought chains for science question answering. In NeurIPS, 2022.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar, Docvqa: dataset for VQA on docu- ment images. In WACV, 2021.
OpenAl. GPT-4 technical report. arXiv, 2023.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, al. Efficiently scaling trans- former inference. In Conf. Mach. Learn. Syst., 2023.
Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, et al. Fastvid: Dynamic density pruning for fast video large language models. arXiv, 2025a.
Leqi Shen, Tianxiang Hao, Tao He, Sicheng Zhao, al. Tempme: Video temporal token merging for efficient text-video retrieval. In ICLR, 2025b.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, and Others. Towards VQA models that can read. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2026
Xudong Tan, Peng Ye. Chongjun Tu, Jianjian Cao, et al. Tokencarve: information-pressrreee visual token compression in multimodal large language models. arXiv, 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, et al. Llama: Open and efficient foundation language models. arXiv, 2023.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, et al. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL, 2019.
Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, al. FOLDER: accelerating multi-
modal large language models with enhanced performance. arXiv, 2025.
Han Wang, Yuxiang Nie, Yongjie Ye, Guanyu Deng, et al. Dynamic-vim: Simple dynamic visual token compression for videollm. arXiv, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, model's 's perception of the world at any resolution. arXiv, 2024b.
Qwen2-vl: Enhancing vision-language
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, and Qintong Zhang. Stop looking for important tokens in multimodal language models: Duplication matters more. EMNLP, 2025.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, 2021.
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv, 2024.
Senqiao Yang. Yukang Chen, Zhuotao Tian, Chengyao Wang, et al. Visionzip: Longer is better but not necessary in vision language models. arXiv, 2024.
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, et al. Deco: Decoupling token compression from semantic abstraction in multimodal large language models. arXiv,, 2024.
Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training- free visual token pruning for multi-modal large language models. In AAAI, 2025.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, et al. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICMI., 2024.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, et al. Activitynet-qa: dataset for understanding complex web videos via question answering, In AAAI, 2019.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, al. Intemin--cooposerr2.5: A versatile large vision language model supporting long-contextual input and output. arXiv, 2024a.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, et al. Sparsevlm: Visual token sparsifi- cation for efficient vision-languaae model inference. arXiv, 2024b.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, et al. Video instruction tuning with synthetic data. arXiv, 2024c.
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, et A stitch in time saves nine: Small VLM is a precise guidance for accelerating large vlms. arXiv, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, et al. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A DETAILS OF RELEVANCE PROPAGATION EQUATION
The relevance propagation in Eq. [ follows the Generie Attention Explainability (GAE) frame- work Chefer et al. (2021b), which is powerful method to interpret predictions for Transformer- based architectures. We trace the contribution of each visual token the model's output by leverag- ing the self-attention modules within the GAE framework to assess token importance. Specifically, GAE first generates a relevance map A for each layer / by integrating raw attention map and its gradients with respect to the current output y/:
A' E, ((A' O VA')*),
(4)
where A' can be obtained through a forward pass and the related gradient V :- 2ye during backward pass. GAE computes the Hadamard product of the attention map and its gradient for the intuitive reasons that: 1) it captures how much attention each token receives from other tokens (information from the attention map). 2) identifies which tokens require more attention to effectively influence the output (information from the gradient). ()+ represents the operation of setting negative values to By zeroing out negative gradient components, GAE enforces a causal direction of influence and prevents the suppression of informative positive signals by accumulated negative components Chefer et al. (202 Barkan et al. (202I). Gradient can also be viewed as weight in the head aggregation to indicate corresponding importance Voita et al. (219)) thereby E,, performs sradient-weighte expectation over the head dimension.
a. can be cached
Since the attention module is followed by residual connection, GAE accumulate the relevance map by adding each layer's contribution A' to the aggregated relevance map Rt:
R=
The overall relevance map is initialized as the identity matrix with the intuition that each input token's 's relevance score is identical in the beginning. Then the relevance propagation updates the R from the 0-th layer to the last layer in the Transformer.
The traditional GAE method lacks mechanism for handling the token-by-token autoregressive outputs required by MILMs. Therefore, we adopt the stepwise relevance computation from R- GAE Yao al.] (2024). a GAE-derived method adapted to MILMs. For generated sequence of textual tokens
yoo V1..... ........-... -r-1). we obtain the overall relevance by averaging the GAE relevance maps computed using Eq. 5 at each decoding step
B MORE VISUALIZATION RESULTS
B. VISUALIZATION RESULTS ACROSS DIFFERENT MLLMS
We present visualization results for LLaVA-One Vision, Qwwn2-VL, and VILAI.5 on both video and image inputs in Figures 4 Given an input image video V, we first show the visual relevance scores R with respect the current response obtained using an explainability method. Based on R,, we visualize the results of token pruning at 50% and 25% retention ratios (labeled as Top-50% compressed V and Top-25% compressed V in the figures). Then, we visualize the pruning results produced by our trained compressor (fe) under the same compression ratios (labeled as Top-50% compressed & and Top-25% compressed y in the figures).
B.2 CASE STUDY: EXPLAINABILITY REVEALS INSTRUCTION- RELATED VISUAL TOKENS
To demonstrate the effectivvness of explainability methods in identifying visual tokens that are highly relevant to instructions, present 2 case studies covering both video and image inputs.
Given the same input V. the explainability method generates visual relevance scores Rg that se- lectively emphasize different visual tokens according to varying user instructions. As shown in Figure 9 when the user instruction specifically targets clothing- information, the visual to- kens corresponding the person's clothing in the video obtain higher relevance scores compared to
14

---

Published as a conference paper at ICLR 2026
User Instruction: According the video, what curved extending from bottom up symbolize? A. Heat flow. B. Stream., C. Vapor. Air. Response: C. Vapor
Input
R
Top-50% compressed
Top-25% compressed
Top-50% compressed /
T5p-25% compressed V
Figure 4: Video Input Visualizations for LLaVA-OneVision.
User Instruction: What the highest fueling cost? Response: The highest fueling cost, as indicated by the bar chart, is for the Ford F150, which is $/30.96.
Top-50% compressed
Top-25% compressed V
Top-50% compressed
Top-25% compressed V
Input V
R
Figure Image Input Visualizations for LLaVA-Onee Vision.
instructions requesting a general summary. Similarly, in Figure 10, visual tokens relevant to the user instruction exhibit higher relevance scores. When the user instruction specifies excluding Ford F150. the visual attention shifts primarily to the other two columns. In contrast, when the instruction highlights the highest fueling cost, the Ford F150 column attracts nearly all the attention.
From visualization standpoint, further corroborate that the explanation results faithfully reflect the critical visual information required by the MLLM to answer the question.
C IMPLEMANTATION DETAILS.
Generating R., To derive R,, our implementation employs eager attention, allowing access to full-layer attention maps required by the explainability method Chefer et al. (2021b). Compared to FlashAttention Dao et al. (2022) and inference based on KV cache Pope et al. (023), eager attention requires memory. To avoid out-of-memory
and ensure efficient data generation, limit the number of visual tokens to approximately 1500 per sample. Specifically, for video inputs, LLaVA-Oma Vision, VILA and Qwen2-VI. all set to sample 8 frames, resulting in 1569, 1568
15

---

Published as a conference paper at ICLR 2026
User Instruction: According to the video, what do three curved lines extending from bottom up symbolize? A. Heat flow. B. Stream., C. Vapor, Air. Response: C. Vapor
Input
R
Top-50% compressed
Top-25% compressed
Top-50% compressed /
Top-25% compressed /
Figure 6: Video Input Visualizations for Qwen2-VL.
User Instruction: What the highest fueling cost?
Response: The highest fueling cost, as indicated by the har chart, for the Ford F150, which is $130.96.
Top-50% compressed V
Top-25% compressed V
Top-50% compressed V
Top-25% compressed V
Input V
R
Figure 7: Image Input Visualizations for Qwen2-VL.
and 1296 visual tokens per video, respectively. For image inputs, LLLVA--OeVisiee and Qwen2- VL use similar image resolutions, resulting 1849 and 1500 visual tokens per image, respectively. VILA always processes an image as 196 tokens, eliminating the need for additional configuration. The generated R2 be used directly to guide token pruning or train fo-
Training fo- fo is implemented as a five-layer fully convolutional network with channel dimensions of 32, 64, 128, 256, and 512. Each layer employs a ID depthwise separable convolution Chollet (017)) i.e., depthwise convolution with a kernel size of followed by a pointwise convolution. An additional pointwise convolution layer applied at the end for channel aggregation. The network trained by using Adam Kingma & Ba (2015) with default settings and a batch size of 128. Training data is collected from open-source datasets: subset of LLa VA- Video Zhhang et al. (2024c) for videos and subset of Infinity-MM Gu et al. (2024) for images, each containing approximately 10K samples. Note that fe is specific to MLLM, so each MLLM generates its own A and R, based on the input image- video-text pair for training. The training is performed for roughly 100 epochs, taking about half an hour for image data and less than four hours for video data on a single A100 GPU.
16

---

Published as a conference paper at ICLR 2026
User Instruction: According to the video, what do three curved lines extending from bottom up symbolize? A. Heat flow. B. Stream., C. Vapor, Air. Response: C. Vapor
Input
R,
Top-50% compressed
Top 25% compressed
Top-50% compressed V
25% compressed 1
Figure 8: Video Input Visualizations for VILA.
t
Input V
User Instruction: Which color of clothing is worn by the first person selling bananas in the video? A. Blue. B. Purple., C. Black., D. Green. Response: A. Blue
R
Top-50% compressed
User Instruction: Which summarizes the content of the video? A. Supply and demand, B. Bananas supply,, C. Business competition, D. Banana selling. Response: Supply and demand.
R
Top-50% compressed P
SWAN GA18
Figure 9: Case Study 1.
Details of Data for Training fo. We train our explainability- based compressor based on subsets sampled from high-quality open-source datasets. First, the details of the sampling follows:
17

---

Published as a conference paper at ICLR 2026
User Instruction: What is the average total fueling cost excluding the Ford F150? Response: 76.55
User Instruction: What is the highest fueling cost?
Response: 130.96
Top-50%
Top-50%
Input V
R
compressed V
Input V
R
compressed 1
Figure 10: Case Study 2.
Image Dataset. For training the compressor used in image tasks, we sample subset of Infinity-MM that ensures high quality and diversity. The training set primarily consists of data used during Stage including 9k samples randomly sampled from the Data Generated by GPT-4 subset and 4k from Synthetic Data.
Video Dataset. For training compressor used in video tasks, we sample a subset of LLaVA- Video. Specifically, we include 7k samples from ELaVA-Videea 6k from NeXT-QA and 4k from ActivàtyNeIQA. Note that the training sets of NeXT-QA and ActivityNetQA have no overlap with the testing sets used in the evaluation. During sampling, since LLa VA-Video contains several parts categorized by task type (open-ended and multi-choice) and video duration (0-30s, 30-60s, 1-2min and 2-3min), we ensure balanced distribution by randomly selecting equal number of training examples from each part.
Moreover, assume that the visual attention distributions (R,) associated with correct answers exhibit higher quality than those that lead to incorrect answers. Therefore, when training fo for specific MILMM, the sampled data evaluated by this MILM,, and the samples with incorrect answers are filtered out. Only samples for which the MLLM produces correct answers are retained and used as training data. The number of the retained samples ranges from 8K 12K.
Inference. The learned fa can be seamlessly integrated into existing inference pipelines (no mod- ifications are required for the prefill and decode phases of LLM inference). More interestingly, fo is capable of processing longer A thanks to the fully convolution design. That is, our compres- sion method can handle larger images and longer videos, even though the visual token number is limited to approximately 1500 during training. Corresponding experiments have been conducted. In these experiments, Qwen2-VI. dynamically processes both images (with 'max_pixe max_pixels' set to half of its default value) and videos (with 'VIDEO_MAX PIXELS' and SPS_MAX_FRAMEE 384 x 28 and 32, respectively). These configurations are set to accommodate hardware resource constraints. LLa Vision also processes images dynamically with default settings, while sam- pling 32 frames per video as in Huaagg et al. (2024) for a fair comparison. For VILA, the input image size cannot be changed, and the number of input video frames is set 16.
set to
D EFFICIENCY ANALYSIS IN INFERENCE
To evaluate computational efficiency during inference, we report the FLOPs of the visual token part. Specifically, consider the FL.OPs of the multihead attention and the ferd-forward network (FFN) modules as:
FLOP Stayer 4nd + 2n Inm,
(6)
where n is the number of visual tokens, is the hidden state size, m is the intermediate size of the FFN, and I is the number of layers in the FFN. To compute the total FLOPs for the entire LLM, we simply multiply Eq. 6 by the number of Transformer layers Nr. i.e. FL.PPLIM = Nr (And + 2nd d Inm).
At the input stage of the LLM, our compressor introduces additional computation. First, we consider the FLOPs introduced by the first-layer attention map:
FL.OPsatn na? nd.
18

---

Published as a conference paper at ICLR 2026
Note that only the key projection computation for visual tokens and the attention computation from textual tokens to visual tokens are required, corresponding to the term nd and nd, respectively. Only the FLOPs incurred by the visual part are included.
Next, we account for the FLOPs introduced by the ID depthwise separable convolution:
FLOPS SCOTV
n(C + C Cous),
(8)
where C and Cuu denote the number of input and output channels of the I-th layer, respectively. We ensure that the output shape of each convolutional layer remains the same its input by applying appropriate padding with respect to the kernel size k. As a result, the number of visual tokens n remains constant across all layers. Then the total FLOPs is computed as the sum of the operations across all convolutional layers.
To intuitively understand the additional computational cost introduced by method, we adopt typical parameter configuration used in MILMss Specifically, we set the number of visual tokens 1568, the hidden dimension per FFN block (1 = 3). For the full L.M,, we consider 28-layer Transformer blocks (N₂ = 28). For fe, we follow the configuration described in Section 4.1 (Experimental Setup). Concretely, the convolutional network consists of 5 layers (L = 5) with kernel size k = 3, and channel dimensions increasing across layers: 32. 64, 128, 256, and 512. Based on these settings, FLOPsaeun amounts to approximately 0.02 trillion, FLOPscoav is approximately 0.0003 trillion, while FLOPSLLM reaches approximately 11.69 trillion. It can be observed that the computational overhead introduced by our compressor is negligible. The computational costs of these two parts account for only 0.17% % and 0.0026% of the total computational cost, respectively.
3584, the intermediate size m 18944, and assume 3 layers
The FLOPs reported in the Table 2 Table 3 Table Figure 3 are computed using standardized input setting. For image input, FLOPs are computed using 384 x 512 input image as the reference (the number of visual tokens is 1728 for LLa Vision and 1302 for Qwen2-VL), For video input, LLaVA-One Vision and VILAI.5 sample 32 and 16 frames, respectively, resulting in visual token counts of 6272 and 3136. We fix Qwen2-VLL VL's input to 32 frames at 720 x 1280 resolution (n=5824) for FI.OPs calculation.
E ADDITIONAL ABLATION RESULTS
We perform additional ablation studies to validate the design choices of our method, specifically examining (i) the depth of the convolutional network and (ii) the attention layer index used as input. For the convolutional depth, we conduct controlled experiments on Qwen2-VL using networks of 3, 5, 7, and 10 layers, while keeping all other factors fixed, including the training data, input rep- resentation A and training configurations (learning rate, batch size, and number of epochs). All variants follow consistent Haaneel-growth strategy in which the network starts with 32 channels and gradually increases capacity, with intermediate channel dimensions repeated to moderate growth and avoid overrparameeterization. The exact configurations are 3-layer: [32, 64, 128], 5-layer: [32, 64, 128, 256, 512], 7-layer: [32. 64, 128, 128, 256, 256, 512], and 10-layer: [32, 64, 128, 128, 256, 256, 512, 512, 512, 512]. For the study on attention-layer index, we train our lightweight model on Qwen2-VL using attention maps extracted from different transformer layers as input, specifically evaluating layers 1, 2, 4, and 6.
The results are reported in Table 6 and Table 7 respectively. Table 6 shows that the 5-layer model achieves the best performance under retention ratios of 50% and 25%. Increasing the depth to 7 or 10 layers leads to slight performance degradation esspeiaal under the more challenging 25% retention condition. This suggests that deeper networks begin to overfit or suffer from optimization difficulties given the simplicity of the task. In contrast, the 3-layer model lacks sufficient capacity to capture the relevance distribution. Therefore, the 5-layer architecture strikes the optimal balance between model capacity and task complexity, which justifies design choice.
Table shows that using multi-layer attention scores as input yields only marginal gains- tains++0.11 +0.1% average 50% retention (with 2 layers) and +0.6% at 25% retention (with 4 layers) yet incurs substantial computational overhead, with FLOPs increasing from 0.24x x to 0.43x at 25% retention, almost two times. Notably, the first-layer input achieves nearly the same performance as multi-layer
19

---

Published as a conference paper at ICLR 2026
Table Ablation study on the depth the convolutional network. In our work, we use a 5-layer convolutional network as the explainability- compressor before the LLM.
Methods
Conv Depth
Retention Ratio
FL.O%s
Image Benchmark
Video Benchmark
Avg-(%) (%)
MME
MMStar
MMVett
Video-MME
MVBench
MMBench-V
Qwea2-VL
100%
1.00x
2295.1
60.4
54.0
50.4
1.23
100
3
50%
0.49x
2265.0
55.5
50.4
49.8
49.3
1.15
95.5
1.18
7
50%
x.49
2279.4
56.1
51.9
49.8
49.4
1.17
96.5
Qwen2-VL.
10
50%
0.49x
2275.1
55.1
51.5
50.0
49.7
1,15
96.0
w/ Ours
25%
0.24x
2297.1
60.3
53.2
51.0
50.7
1.19
90.7
5
25%
0.24x
2299.1
58.7
51.7
50.3
49.7
1.17
91.7
7 255
0.24x
2299.1
58.7
51.7
50.3
49.7
1.17
91.1
10
25%
0.24x
2299.1
58.7
51.7
50.3
49.7
1.17
90.4
Table 7: Ablation study the attention layer index used input. In our work, the explainability-basee compressor takes the first-layer attention map A as its input.
Methods
Input Layer
Retention Ratio
FLOPs
Image Benchmark
Video Benchmark
Avg.(%)
MME
MMStar
MMNeet
Video-MME
MV MVBencc
MMBench-V
Qwea2-VL
2295.1
50.4
1.23
1
50%
x.49
2288.3
55.9
51.9
50.0
49.8
1.18
96.9
2
50%
0.53x
2279.8
55.8
52.0
50.1
50.2
1.18
97.0
0.60x
2303.0
1.15
Qwen2-VI. w/ Ours
6
50%
0.67
2278.1
56.7
49.5
49.4
49.6
1.19
96.1
1
25%
0.24x
2280.9
51.8
47.3
48.1
46.7
1.11
91.7
0.35
2272.0
47.8
4
25%
0.35
2270.2
52.1
48.0
48.0
48.1
1.11
92.3
6
255
0.43x
2252.3
52.1
47.2
47.4
47.6
1.11
91.6
variants while being significantly more efficient. Our work aims achieve an innovative paradigm shift that enables task-related token compression to be applied prior to the LLM,, which not only significantly reduces computation and memory overhead during both prefill and decode phases, but also allows deployment without modifying the LLM architecture. However, leveraging deeper-layer attention maps as input yields only marginal improvements while contradicting our design goal. Overall, the first-layer attention map suffices for purpose.
F ADDITIONAL EXPERIMENTAL RESULTS
For image tasks, further conducted evaluations across additional benchmarks, including GQA Hudson & Manning (2019) (real-world visual reasoning), ScienceQA (SQA)[ur et al. (2022) (scientific reasoning), VizWiz Gurari et al.] (2018) (real-world robustness), ChartQA Masry et al. (2022) (chart reasoning). Doc Mathew et al. (2021) (document reasoning), OCRBench Liu al. (2024b) (OCR reasoning), to comprehensively assess the effectiveness of our method. The results are presented in Table g our method consistently outperforms the baselines across diverse image benchmarks, including real-world visual reasoning and OCR-related tasks.
For video tasks, we also investigate more aggressive compression setting with 10% retention ratio in Table as a supplement to Table Our method attains the lowest FLOPs while preserving competitive accuracy, achieving average improvements of 3.7%, 3.2%, and 1.9% across all bench- marks for V...VAA-OeVViiiin Qwen2-VI. VI. and VIL.A, respectively, compared to the perf-perfo baseline. Notably, results, highlighting its robustness across different MLLMs.
under such extreme compression,
method consistently delivers strong
We provide full tables of results corresponding to the generalization experiments shown in the Fig- ure (3 (a) in the main text (Applying to Larger Images and Longer Videos), with detailed results for the image and video benchmarks listed in Table 10 and Table II respectively. In addition, pro- vide detailed comparison results shown the Figure (b) for two challenging benchmarks, MMStar and MVBench, in Table 12 and Table 13
20

---

Published as a conference paper at ICLR 2026
Table Compare explainability-base. compressor on image benchmarks. Values marked with in Retention Ratio denote the average retention ratio across LLM layers due to multi-stage com- pression in PDrop.
Retention Ratio
Method
FLOPS
GQA
SQA
VizWiz
ChariQA
DocVQA OCRBench
LLaVA OneVision
100%
1,00x
63.1
95.1
33.1
71.4
73.3
54.4
LLaVA VneVision FastV
0.51x
LLaVA-OneVisiee Vision w/ Pdrop
51%*
0.51x
61.0
92.0
29.7
62.0
58.0
42.0
LLaVA-One Vision w/ Dart
50%
0.51x
61.7
91.7
29.8
61.0
55.3
43.2
LLaWa-OmeVision
0.48x
LLaVA-One Viston w/ F
25%
0.27x
39.9
70.4
2.7
21.3
22.2
13.9
LLaVA OneVision w/ PDrop
25%
0.25x
57.2
87.9
25.3
43.4
34,9
30.2
LLaVA VneVision
0.27x
LLa LLaVA-OneVision w/ Ours
25%
0.24x
89.0
90.2
33.0
52.7
49,4
38.3
Qwen2 VI.
100%
1.00x
62.2
85.7
44.3
92.7
93.1
81.5
Qwen2-VL w/ FastV
0.51
32.5
51.3
59.2
56.3
Qwen2 VL w/ PDrop
51%*
0.51
60.8
83.5
41.9
77.0
78.5
78.8
Qwen2 VL w/ Dart
50%
0.51x
61.3
84.7
43.9
78.4
79.7
78.5
Qwen2-VL Ours
x.49
44.1
80.1
81.2
Qwen2 VL. w/ FastV
25%
0.27x
41.5
64.3
31.5
41.1
46.1
46.8
Qwen2-VL PDrop
25%*
0.25x
40.8
61.7
58.2
63.9
Qwen2-VL w/ Dart
0.27x
41.3
65.0
62.8
65.1
Qwen2-VL. w/ Ours
25%
0.24X
59.3
84.3
43.1
70.9
72.1
67.8
Table 9; Compare explainability compressor on video benchmarks.
Method
Retention Ratio
FLOPs
Video-MME
MVBench
MMBench- Video
Next-QA multi-choice open-ended
Activity-QA
Avg.
LLaVA-OV w/ FastV
10%
0.12x
37.7
22.4
0.27
60.3
30.6
39.9
66.5
LLaVA-OV w/ PDmp
10%*
0.10x
47.0
37.0
0.35
72.3
43.7
50.0
88.5
LLaVA-OV w/ Dart
10%
0.12x
47,3
36.9
0.36
72.7
43.5
50.3
89.1
10%
0.09x
37.4
Qwen2-VL VL
100%
1.00x
50.4
51.0
1.23
76.8
45.5
53.6
100
Qwen2-VI. FastV
10%
0.12x
29.1
37.5
0.44
39.4
23.3
32.0
54.9
Qwen2-VL, PDrop
10%*
0.10x
40.3
Qwen2-VL. VV w/ Dart
10%
0.12x
45.8
41.6
0.85
72.1
42.6
45.5
85.7
Qwen2-VL. VL w/ Ours
10%
0.09x
46.1
42.5
1.00
72.0
43.3
47.5
88.9
VILA
100%
1.0xx
VILA FastV
10%
0.12x
37.8
19.5
0.88
57.9
33.9
43.2
73.2
VILA w/ PDrop
10%*
0.11x
43.0
34.4
1.13
65.2
43.5
50.6
93.0
VILA Dart
10%
0.12x
42.9
34.8
1.14
93.4
Ours
10%
0.09x
43.6
35.0
Table 14 exhibits the additional efficiency analysis on VM Our lightweight compressor achieves substantial reductions KV-cache usage and accelerates the prefill stage, while achieving the high- est task scores and keeping overall inference time comparable to baselines. These results demon- strate that our approach maintains both strong task performance and computational efficiency.
G THE USE OF LARGE LANGUAGE MODELS(LLMS)
In preparing this manuscript, we used large language model (L.LM, specifically GPT-5-mini) solely general-purpose writing and editing assistant. The LLM was employed to improve clarity, gram- mar, and overall presentation of the text. All technical content, experiments results, and interpreta- tions were generated and verified by the authors. The LLM did contribute research ideation, experimental design, data analysis, or the writing of original technical content. The authors take full responsibility for all content presented in this paper.
21

---

Published as a conference paper at ICLR 2026
Table 10: Compare generalization performance of our compressor on image benmarks.
Rentention
Method
FLOPs
MME
MMStur
MMVet
SEED
Avg(%)
Ratio
LLaVA-OneVision
2002.0
76.7
MLNVA-OneVision w/ Fast LestVopucen
50%
0.51x
1990.3
57.3
48.4
75.7
95.9
LLaVA-One Vision w/ Ours
0.48x
1988.0
75.4
96.8
VAaVV-Oneeiin w/ Fast FastVopuCmula
25%
0.27x
1953.7
52.0
43.2
71.5
89.4
I.a Vi-oonniiioon w/ Ours
25%
0.24x
1985.8
52.6
45.8
73.1
91.9
Qwen2-VL
100%
1.00x
2316.6
61.1
51.7
76.4
100
Qwen2-VL w/ Comii
50%
0.51x
2295.8
51.7
52.4
74.8
98.2
Qwen2-VL.
2311.7
Qwen2-VL w/ Fast Fast
25%
0.27
2288.2
55.0
49.3
71.1
94.3
Qwen2-VL VL Ours
25%
0.24x
2283.1
55.8
71.0
95.3
Table 11: Compare generalization performance of our compressor on video benchmarks.
Method
Retention Ratio
FLOPs
Video-MME
MVBench
MMBench- Video
Next-QA
Activity-QA
Avg.(%
MC
OE
LLaVA-OV
100%
37.1
58.4
LLaVA-OV w/ Coumt
50%
0.48x
58.8
36.1
0.38
80.5 51.4
58.2
98.9
LLaVA-OV Ours
50%
0.46
58.8
37.2
0.38
80.2 $2.0
58.1
99.5
LLaVA-OV w/ Cast
25%
x
57.0
35.4
0.35
79.7
50.9
57.2
96.2
LLaVA-OV w/ Ours
25%
0.22x
56.5
36.9
0.36
79.2 51.0
58.0
97.3
Qwen2-VL
100%
1.00x
52.7
1.42
49.5
$7.6
Qwen2-Vi VL w/ PastVop.cuuta
50%
x
55.4
51.3
1.40
79.6
49.0
55.7
97.9
Qwen2-VL
50%
0.46x
51.4
48.7
$6.3
Qwea2-VE VL w/ FastVop..oot
25%
0.25x
$3.0
49.6
1.30
78.6
47.3
$2.0
93.6
Qwen2-VL. w/ Ours
25%
0.22
53.2
48.6
1.30
78.4
47.6
54.3
94.1
VILA
100%
1.00x
48.7
31.7
1.30
70.4
45.8
55.2
100)
VILA w/ FastVop
50%
0.49x
48.1
31.5
1.31
70.1
46.5
55.1
100.0
Ours
VII A w/ Fast
25%
0.26
46.3
31.8
1.26
69.6
45.6
54.6
98.3
VILA w/ Ours
25%
0.23
47.4
35.0
1.29
46.7
55.7
101.5
Table 12: Efficiency and performance comparison across different methods on MMStar. Values marked with indicate that the retention ratio refers to the average proportion of retained tokens across all LLM layers, due to multi-stage compression in PDrop. For FastV, the same retention ratio corresponds to different FLOPs when compression is applied at different layers (2nd and 4th).
Method
Ratio
FLOPS()
Preservation %
LLaVA-OneVisha
LLaVA-OneVision w/FastVV
25.0%
4.2
83.9
LLaVA-OneVision w/FastV
25.0%
3.5
66.3
LLVV-OOneVison w/PDrop
LLaVA-OneVision w/PDrop
2
3.2
80.2
LLa LLVAAOneeisson w/Ours
Qwea2-VL
100%
9.6
100
Qwen2.Ve w/TasVV
25.0%
3.1
90.0
Qwen2-VV VL
25.0%
2.4
22

---

Published as a conference paper at ICLR 2026
Table 13: Efficiency and performance comparison across different methods on MVBench. Val- ues marked with indicate that the retention ratio is reported from the original paper.
Retention Ratio
Performance Preservation
Method
FL.PP(()
LLaVA-One
100%
52.7
MNNA-OeeVssies w/FaxVID
25,0%
11.7
99.3
LLaVA LLaVA-OneVin Vision w/PruneVID
17.0%
11.9
99.1
LLaVA-One Vision w/FastV
16.1
LLaVA-OneVision w/VisionZáp
25.0%
11.7
94.4
LLaVA-OncVison w/Ours
25.0%
11.7
99.5
Qwen2-VI.
100%
48.4
Qwen2-Vi 2LL w/FastV
25.0%
14.9
94.1
Qwen2- 2L w/Ours
25.0%
10.9
92.2
VILAL.5
100%
27.0
VILAL5 w/FasVV
25.0%
8.2
100.3
VILALS w/Ours
25.0%
6.3
110.4
Table 14: Efficiency analysis based on Qwen2-VL on MMVet. We evaluate the inference costs in terms of total inference time, prefiling time, FLOPs, and KV cache memory. KV cache memory is computed with consideration of the Grouped Query Attention (GQA) used practical inference.
Method
Retention Ratio
FLOP
Total Inference
Time
Prefilling Time
KV Cache
Speedup
Prefilling Speedup
MMV
Qwen2.
7minn58s
71.2MB
Qwen2-VL FastV
25%
0.27x
6min50s
0min56s
19.7MB
1.17x
1.61x
Qwen2-VL VL PDrop
25%
0.25x
6min449s
0min55s
18. 1MB
1.17x
1.64x
47.0
Qwen2-VL VL Durt
25%
0.30x
6min51s
0min57s
21.6MB
1.16x
1.58x
QwenZ-VVe w/ Ours
25%
0.24x
6min50s
0min544
17.8MB
1.17x
1.67
50.8
23


Reviews:
[{"review_id": "b2qKgORrXt", "reviewer": "Reviewer_RQ2d", "summary": "This paper introduces a novel approach for task-aware visual token compression in multimodal large language models (MLLMs), aiming to eliminate instruction-irrelevant tokens at the LLM input stage to enhance inference efficiency—without modifying the underlying architecture. The key idea is to utilize explainability techniques to assign relevance scores to visual tokens with respect to a given instruction, guiding the compression process accordingly. Furthermore, the authors demonstrate that a lightweight convolutional network can be trained to map first-layer attention maps to explainability-derived importance scores, enabling token importance prediction without a full forward pass.", "questions": "Please refer to weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work tackles a critical bottleneck in MLLMs that high computational and memory overhead from large numbers of visual tokens by introducing an instruction-aware token compression paradigm that operates without architectural changes.\n2. The proposed compressors are lightweight, transferable, and require minimal retraining across diverse MLLM architectures, underscoring strong generalizability.\n3. Evaluation spans 11 benchmarks for both images and videos, with comprehensive ablations, efficiency analyses, and baseline comparisons.", "weaknesses": "1. The paper relies heavily on empirical findings and visual evidence, offering limited theoretical grounding. The mathematical formulation (e.g., the relevance propagation equation for 𝑅t in Section 3.2) lacks a deeper analysis of properties such as linearity, gradient behavior, and attribution faithfulness, especially under ambiguous or multi-factor instructions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lRIYmTZnSD", "reviewer": "Reviewer_M6Sq", "summary": "The paper proposes a task-related token compression paradigm for Multimodal Large Language Models (MLLMs) to address the high computational cost and inefficiency associated with a large number of visual tokens in the LLM input. The authors first use an explainability method (e.g., gradient-weighted multi-head attention) to assess the global importance of each visual token relative to a given instruction, generating a \"ground truth\" importance score. Building on this insight, they train a lightweight convolutional network to predict this importance score based solely on the LLM's first layer attention map. Experiments demonstrate the effectiveness of the proposed method.", "questions": "The paper mentions training using only 10K image samples. Please elaborate on the data sampling strategy (e.g., are they sampled from the test benchmarks? Are they general-domain? How is diversity ensured?) Is the small scale of the training data a potential limitation when generalizing to open-world or other benchmarks scenarios beyond the 11 evaluation benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand. Figure 1 and 2 are intuitive to understand the motivation and the big picture of the proposed method.\n\n2. The authors successfully demonstrate the feasibility and efficiency of performing task-related token compression at the LLM input stage with negligible performance loss, providing a new, more efficient compression strategy.\n\n3. Experiments validate the effectiveness across three distinct MLLM architectures (Qwen2-VL, LLaVA-One Vision, VILA1.5) and 11 diverse image and video benchmarks, showcasing its robustness and wide applicability.", "weaknesses": "1. The lightweight convolutional network is trained using only the first-layer attention map ($A^0$). A more in-depth analysis is needed to justify how such a small network, using only shallow information, can accurately and robustly predict the global, task-specific importance score, which typically requires information propagated through multiple LLM layers.\n\n2. This train-based methods may overfit to the training data. I wonder the performance on some OCR related benchmarks (TextVQA, Chartqa, DocVQA, OCRBench), since they are more challenging to validate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "87VVCu3j2i", "reviewer": "Reviewer_y6e3", "summary": "This paper aims to compress visual tokens for accelerating inference of multi-modal large language models (MLLMs).\n\nSpecifically, it transfers the explainability methods for transformer-based architectures to visual token pruning.\nThe explainability methods often keep a relevance map that could be used to measure the importance of visual tokens.\n\nTo validate the effectiveness of the method, experiments on various benchmarks (MME, MMStar, MMVet, Video-MME, MVbench, and MMBench-V) are conducted, demonstrating improvements over previous methods.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper writes clearly and is easy to follow.\n\n(2) It is interesting that the relevance map from explainability methods can accurately identify important visual tokens.", "weaknesses": "(1) To derive the relevance map, it requires the ground truth labels to calculate gradients. Thus, the paper proposes a lightweight model to distill knowledge from the derived relevance maps. However, it is challenging to make it generalizable to various data, as the lightweight model is small and the size of training data is also much smaller than that used for MLLMs.\n\nIt would be better to evaluate the model performance on various benchmarks, like GQA, SQA, VQAv2, VizWiz, and MMB.\n\n\n(2) For the lightweight model, it takes the first-layer attention scores as inputs.\n     Why not use the original visual tokens as inputs?     \n     Is it helpful to use multiple-layer attention scores?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2xddTs6caq", "reviewer": "Reviewer_66M8", "summary": "This paper proposes a visual token compression method for Multimodal Large Language Models (MLLMs) based on explainability approaches, enabling task-related token compression at the LLM input stage. The core innovation lies in using explainability methods to evaluate the importance of visual tokens and training a lightweight network to predict these importance scores. Evaluations on 3 MLLM models and 11 image/video benchmarks show that the method maintains high performance while reducing computational complexity, prefilling time, and KV-cache usage.", "questions": "1. Why is the network depth set to 5 layers? No ablation analysis is provided to justify this choice.\n2. Can relevance prediction be improved using more advanced architectures without sacrificing efficiency?\n3. The motivation explanation in Section 3.1 is insufficiently clear. Specifically, the causal link indicated by \"therefore\" in Lines 186–187 is not clear enough.\n4. Why is one layer of attention sufficient? The paper mentions \"first-layer attention suffices\" but provides no theoretical or empirical analysis to support this claim.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It achieves an innovative paradigm shift, as the study challenges the previous view that shallow-layer visual tokens are indispensable.\n2. The authors conducted comprehensive experiments to demonstrate the superiority of their method compared to other existing approaches.", "weaknesses": "Requires more discussion and ablation analysis; see Questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)