MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
Authors: Sixun Dong
Juhua Hu
Mian Zhang
Ming Yin
Yanjie Fu
Qi Qian
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient VLM Inference
Multimodal Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Vision-Language Models (VLMs) demonstrate impressive performance in understanding visual content with language instruction by converting visual inputs to vision tokens. However, redundancy in vision tokens results in the degenerated inference efficiency of VLMs. While many algorithms have been proposed to reduce the number of vision tokens, most of them apply only unimodal information (i.e., vision/text) for pruning and ignore the inherent multimodal property of vision-language tasks. Moreover, it lacks a generic criterion that can be applied to different modalities. To mitigate this limitation, in this work, we propose to leverage both vision and text tokens to select informative vision tokens by the coverage criterion. We first formulate the subset selection problem as a maximum coverage problem. Afterwards, a subset of vision tokens is optimized to cover the text tokens and the original set of vision tokens, simultaneously. The proposed method MMTok is extensively evaluated on benchmark datasets with different VLMs. The comparison illustrates that vision and text information are complementary, and combining multimodal information can surpass the unimodal baseline with a clear margin. Moreover, under the maximum coverage criterion on the POPE dataset, our method achieves a 1.87× speedup while maintaining 98.7\% of the original performance on LLaVA-NeXT-13B. Furthermore, with only four vision tokens, 87.7\% of the original performance is still preserved on LLaVA-1.5-7B. These results highlight the effectiveness of coverage in token selection.

Published as a conference paper at ICLR 2026
ММТок: MULTIMODAL COVERAGE MAXIMIZA- TION FOR EFFICIENT INFERENCE OF VLMS
Sixun Dong!t
Juhua Hu²
Mian Zhang
Ming Yin
Yanjie Fu'
Qi Qian
AArizona State University
University of Washington University of Texas at Dallas
ZZom Communications sixxndonng.ar ai, qianq. mai 1 10gmmmm 10ggmal.... com, juhuah@uxe edw., yanjie. fulasu. cdu
*Duke University
ABSTRACT
Vision-Language Models (VLMs) demonstrate impressive performance in un- derstanding visual content with language instruction by converting visual inputs vision tokens. However, redundancy in vision tokens results in the degraded inference efficiency of VLMs. While many algorithms have been proposed to reduce the number of vision tokens, most of them apply only unimodal informa- tion (i.e., vision/text) for pruning and ignore the inherent multimodal property of dision-language tasks. Moreover, it lacks a generic criterion that can be applied to different modalities. To mitigate this limitation, in this work, we propose to leverage both vision and text tokens select informative vision tokens by the coverage criterion. We first formulate the subset selection problem a maxi- mum coverage problem. Afterwards, subset of vision tokens is optimized cover the text tokens and the original set of vision tokens, simultaneously. The proposed method MMTok is extensively evaluated on benchmark datasets with different VLMs. The comparison illustrates that vision and text information complementary, and combining multimodal information can surpass the unimodal baseline with clear margin. Moreover, under the maximum coverage criterion the POPE dataset, our method achieves a 1.87x speedup while maintaining 98.7% 7% of the original performance tokens, 87.7% of the original performance is still preserved on LLa VA-1.5-BB. These results highlight the effectiveness of coverage token selection. The code is available at https: : //gitttt..b//// com/ Ironieser paasserrmmmtte /m
LLaVA-NeXT-1S. 133-BB.. Finally, with only four vision
MMTok
DivPrune
VisionZip
Sparse VLM
90%
80%
Token
LLVAA-I5-139
100
100
80
80
LLAVA- -o1-7-
160
192
128
60
0%
64
40
40
30%
20
20
64
32
16
8
64
32
16
8
(1) MMB
(2) POPE
160
90
80
LLaVA-Ner--1yy
60
70
20%
425
60
5%
QweB-2.5-VL-78
10%
V-L-BBB
40
50
16
(3)
MEME
(4) SEED-I
(a) Performance Preservation (%) Token Budgets
(b)
Cross-Architecture Performance
Figure I: MMTok demonstrates better performance across multiple benchmarks.
Work done during internship Zoom Communications.
E Corresponding author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
By converting the visual input to vision tokens, Vision-Language Models (VLMs) can leverage powerful Large Language Models (LLMs) to understand visual content as text (Liu et al., 2024b; Li et al., 2024b; Team et al., 2023). Unlike discrete text tokens, where the information is highly compressed, current vision encoders extract vision tokens directly from the original input patches, which are redundant according to previous studies (Bolya et al., 2022; He et al., 2022) and their count can far exceed that of text tokens. For example, given "Describe the image" with less than 10 text tokens, 2,880 vision tokens can be obtained from single image in LLaVA-NeXT (Liu et al., 2024a).
Since LLMS are built on self-attention layers ( (Vaswani et al., 2017) 7) that have a quadratic computational cost with respect the total number of tokens, the large volume of vision tokens can significantly challenge the inference efficiency of VLMs. To accelerate inference, many works (Shang et al., 2024; Yang et al., 2025a; Zhang et al., 2024) have been proposed to sample a subset of vision tokens for inference without compromising performance. While some work adopts an additional training process (Yang et al., 2025a) to enable vision token selection, in this work, we will focus on the training- free paradigm to reduce optimization efforts. Our experiments also confirm that the proposed training-free method can even outperform baselines with fine-tuning.
Despite different architectures of VLMs (Team, 2024; Bai et al., 2025; Guo et al., 2025), the leading performance is achieved by architectures employing a separate vision encoder to obtain vision tokens (Bai et al., 2025). In that architecture, both vision tokens and text tokens are available for token selection before applying LLMs. However, most of the existing work relies on unimodality for pruning while the multimodal information has not been explored sufficiently (Zhang al., 2024; Yang et al., 2025a; Alvar et al., 2025). For example, Sparse VLM (Zhang et al., 2024) mainly considers text tokens from language instruction to guide the pruning of vision tokens, while VisionZip (Yang et al., 2025a) heavily depends on the [CLS] ] vision token to select informative vision tokens. By investigating Saiiunlanguage tasks, we find that given the same image, the answers can be different due to user-specific text queries, while the same text instruction can be applied for different images, i.e. caption tasks. Therefore, a unimodal method is hard to capture sufficient information about target tasks, implying suboptimal performance for token selection.
In order to leverage both vision and text information to obtain informative vision tokens, in this work, we propose multimodal strategy for efficient inference. First, we formulate the token selection problem as a maximum coverage problem, which aims to cover the target tokens with a subset of source tokens. While the source tokens are vision-only, the target ones can come from either text or vision, respectively. Therefore, the framework can explicitly combine the information from different modalities. Then, we optimize the coverage problem by maximizing a submodular function defined on the similarity between target and source tokens. Although the original problem is NP-hard (Khuller al., 1999), simple greedy algorithm observe an approximate solution that is not worse than (1 - 1/e) of the optimal solution (Nemhauser et al., 1978). The main contributions of this work are summarized as follows.
We introduce the maximum coverage problem for vision token selection. The problem can be formulated as maximizing submodular function, which has an efficient algorithm to obtain near-optimal solution with a theoretical guarantee.
We apply the coverage criterion cover both the text tokens and the entire set of vision tokens with a subset of selected vision tokens. The tixtovision and rision-vision coverage explicitly help explore multimodal information for selection.
Experiments are conducted on benchmark datasets with diverse VLMs. The superior performance of the proposed method demonstrates the effectiveness of the proposed coverage criterion for the subset selection of vision tokens. For example, the proposed MM'Tok can achieve overall best performance under different settings as illustrated in Figure 1 (b) and shows the potential to compress to an extremely small number of vision tokens as in Figure (a).
RELATED WORK
VLMs, such as LLaVA (Liu et al., 2023), InstructBLIP (Dai et al., 2023), and Qwen (Bai et al., 2025), have become comnertone for multimodal understanding by integrating large-scale vision encoders
2

---

Published as a conference paper at ICLR 2026
Vision Large Language Models
Reduce 95% tokens
Multimodal Coverage Selection
Greedy Search
small
cat
standing
S
S/-1
5 (s.)
2880 tokens
Tokenizer
Vision Encoder
Whatis the small castandnnnn
Figure 2: Overview of MMTok framework. Our method optimizes two maximum coverage prob- lems simultaneously to leverage text-vision and vision-vision similarity for vision token selections.
(e.g., CLIP-ViT (Radford et al., 2021b)) with pre-trained language models. These models achieve strong performance by representing images as sequences of visual tokens, but their inference cost grows quadratically with token count, highlighting the need for more efficient processing.
Many vision token selection methods have been proposed recently, but most of them rely only on unimodal information for pruning (Yang al., 2025a; Shang al., 2024; Chen et al., 2024a; Zhang et al., 2024; Alvar et al., 2025). For example, VisionZip (Yang et al., 2025a) and FastV (Chen et al., 2024a) prune tokens using pre-trained attention signals, either ranking by [CLS] token attention (VisionZip) or discarding low-attention vision tokens in deeper layers (FastV). Besides ranking, DivPrune (Alvar al., 2025) to maximize the intra-set diversity. These methods rely vision information and may miss query- related semantics (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019). Sparse VLM (Zhang et al., 2024) instead uses text-to-vision attention for scoring, yet ignores the information from the whole image. To mitigate the gap between existing unimodal algorithms and target multimodal tasks, we propose coverage-bar based criterion to leverage both vision and text information sufficiently to select vision tokens effectively.
diversity-based criterion but only has vision tokens
3 THE PROPOSED METHOD
To leverage the power pre-trained models, many existing VLMs adopt a pre-trained vision encoder to extract vision tokens from images and then concatenate them with text tokens as input for the pre- trained LLMs. Although the simple architecture demonstrates promising performance, the inference efficiency be challenging. Concretely, given image, a pre-defined number of vision tokens will be obtained as (V1,..., Vn Even for a small 336 x 336 image, n is 576 with the VřT-L-336px from CLIP (Radford al., 2021a), which is much larger than that of the text tokens from the text query (Liu et al., 2023). The large n will significantly slow down the inference of LLMs, which relies on the self-attention operations, and the complexity quadratic the total number of tokens.
To accelerate the inference of VLMs, we propose to select an informative subset of vision tokens {ue Jees reduce the number of input tokens for LLM in VLM, where / (1....,7, nn. S C N, and (S] < n. Figure illustrates the framework of our method, and we describe it as follows.
3.1 VISION TOKEN SELECTION BY COVERAGE MAXIMIZATION
Unlike most of the existing work, we apply coverage as the main criterion for token selection. Given similarity matrix M E Rm.. defined between target tokens and source tokens, where denotes

---

Published as a conference paper at ICLR 2026
the number of target tokens and is the number of source tokens, subset S will be selected to maximize the similarity between the target and selected tokens as
m m
f(S; M)
max Mi,s; S arg max f(S;
(1)
S
a.k.a. covering the target tokens by an appropriate subset of source tokens. We first find that Eq. is popular submodular function (Leskovec et al., 2007). Proposition 1. (Leskovec et al., 2007) For all subsets A ACE B C N and s E EN \B, B.
[(A - ((A) 2 ( - f(B)
Maximizing submodular functions in general is NP-hard (Khuller et al., 1999), but a simple greedy algorithm can achieve good approximation.
Proposition 2. (Nemhauser et al., 1978) Let S denote the subset obtained by the greedy algorithm, then have
f(S) 2 (1 - 1/e)
max f(A)
A:AA=||S]
We elaborate how to apply the coverage function for token selections in the following subsections.
3.1.1 MAXIMUM TEXT- VISION COVERAGE
First, we consider covering the semantics from text tokens with source vision tokens, which aims to find the vision tokens related to the text input (e.g., query). Let ft...... tm denote the text tokens from the query. A similarity matrix between text and vision tokens can be obtained as
M t v
where M E Rm and Vi J. ||t.ll2 = l|v 22 = 1. To align the semantic similarity between text and vision, we adopt the vision tokens after the projection layer (i.e., those concatenated with text tokens input for LLMs). After obtaining the appropriate similarity matrix, subset of vision tokens can be selected to maximize the similarity between all text tokens and selected vision tokens for coverage as
S' = arg max f(S M*)
S
According to Proposition 2, a greedy algorithm as summarized in Alg. can approximate the optimal solution. It should be noted that the proposed Alg. 1 contains only simple operations (e.g., argmax, matrix multiplication, etc.) and thus efficient for implementation.
Algorithm A Greedy Algorithm to Cover Text Input with Vision Tokens
Algorithm 2 MMToK: Greedy Algorithm for Mul- timodal Coverage
1: Input: Similarity Matrix M
1: Input: Similarity Matrices M M vv k Initialize S = 0
2: Initialize S
3: for i = l....... ke do
3: for i 11=1,... 1,... ,k do
4: for s E N do
for s E A S do
5: Compute g(x) S(S Us; M()
5: Compute g(s) = M .
6: end for
end for
7: Obtain 84 arg max, g(s)
7: Obtain 8; = arg max, g(s)
8: S SSU
S S=SUs:
9:
end for
9: end for
10: return
10: return S
3.1.2 MAXIMUM VISION- VISION COVERAGE
Although text-vision coverage can explore vision information according text, may be insufficient due to vague text, e.g., "Please describe the image". Therefore, we propose to cover all vision information with limited number of vision tokens. Concretely, a vision- vision similarity matrix can be generated as
M v.
4

---

Published as a conference paper at ICLR 2026
where M R Unlike M that adopts vision tokens after the projection layer to align with text tokens, those before projection are more appropriate to capture similarity between vision tokens without mixing text information. We have v' to distinguish it from the one after projection (i.e., v).
Then, we can apply the greedy algorithm to select subset of vision tokens to cover the main informa- tion implied by the whole set of vision tokens. Obviously, vision-vision coverage is complementary to text-vision coverage, which also confirmed by our ablation study. The remaining challenge is to combine the two maximum coverage problems, which is described in the next subsection.
3.1.3 MAXIMUM MULTIMODAL COVERAGE
The maximum coverage problem be applied the original text and vision tokens simultaneously. However, " and ' have different shapes and similarity measurements. Therefore, their values must be aligned before fusion. To calibrate the similarity between different modalities, the score for each row, i.e., that for target tokens, is first normalized by a softmax operation as
exp(M اー exp
(M/)/T/T eXp Tw)
M
M.
where the softmax operation normalizes each row to a distribution over all vision tokens. Tr and T. aim to further normalize the distribution shape for text-vision and vision-vision, respectively.
After calibration, the final objective for multimodal coverage can be written as
M .M'') = f(S; ((") + af(S
where a used to weigh the importance of vision-vision vision coverage. Incorporating text-vision coverage with vision-vision coverage, the function in Eqn. 2 still submodular function as follows. Corollary 1. The sum of two submodular functions is submodular function.
Proof. It comes from the addition property of inequalities directly.
With Corollary we can apply a similar greedy algorithm to obtain the near-optimal solution for the multimodal scenario efficiently. The detailed algorithm is summarized in Alg. 2.
4 EXPERIMENTS
To evaluate the performance of the proposed method, MMTok, we conduct experiments on diverse benchmark datasets and VLMs with different architectures. For fair comparison, we use datasets adopted in VisionZip (Yang al., 2025a), which contains GQA (Hudson & Manning, 2019), MM- Bench (Liu et al., 2024c), MME (Fu et al., 2023), POPE (Li al., 2023b), ScienceQA(IMG) (Lu (Goyal et al., 2017). TextVQA (Singh et al., 2019). MMMU (Yue
et al., 2022), -esvt-Test-- et al., 2024), and SeedBench (Li et al., 2023a). Mcanwhile, five VLMs are applied for comparison, that is, I.LVVA-155-BB (Liu et al., 2023), LLLVA-L.5-13B (Liu et al., 2023), L.AVA-NeXT-7B (Liu et al., 2024a), LLaVA-NeXT--33B (Liu et al., 2024a), and recent model Qwen-2.5-Vi VL--BB (Bai al., 2025). Finally, we compare our method with state-of-the-art vision token pruning algorithms, including FastV (Chen et al., 2024a) (a vision-based method), SparseVLM (Zhang et al., 2024) language-based method), VisionZip (Yang al., 2025a) (a [CLS] -ispo-tancce-bbaeeee and DivPrune (Alvar et al., 2025) (a diversity- based method). We also include a fine-tuning based method, VisionZipé, in the comparison. We obtain the result of DivPrune through official code, and that for other baselines is directly from (Yang et al., 2025a). Evaluation is implemented under the Lmms umms-eval framework (Li et al., 2024a) with the details elaborated as follows.
method),
Implementation Details The proposed method relies on an appropriate similarity for coverage optimization. Since different layers may demonstrate different similarity measurements (Liu et al., 2023), we have vision tokens before the projection layer compute vision-vision similarity, while those after the projection layer are for eest-vision coverage. That because the latter layer aligns better with the text. We find that method is not sensitive to hyperparameters, ablation study. Therefore, we fix T 0.02, T, = 0.2, and a = 0.5 for all experiments if not otherwise specified.
shown in the
5

---

Published as a conference paper at ICLR 2026
Method
GQA
MMB
MME
POPE
SQA
VQA
VQA Teat
MMMU
SEED
Avg.
Total
Tokens
LLaVA-1.5-7B
61.90
64.70
1862.00
85,90
69.50
78.50
58.20
36,30
58.60
100%
Retain 192
Tokens
67%
FastV
52.70
61.20
1612.00
64.80
67.30
67.10
52.50
34,30
57.10
89.6%
Sparse VL.M
57.60
62.50
1721.00
83.60
69.10
75.60
56.10
33.80
55.80
95.5%
VisionZip
59.30
63.00
1782.60
85.30
68,90
76.80
57.30
36.60
56.40
97.9%
DivPrune
59.97
62.54
1762.23
87.00
68,66
76.87
56.97
35.44
58.71
98.0%
VisionZip
60.10
63,40
1834.00
84,90
68.20
77.40
57.80
36.20
57.10
98.4%
MMTok
60.07
63.40
1773.86
86.42
68.76
77.11
57.68
36.33
59.21
98.7 %
Retain 128 Tokens
4. 78%
V
49.60
56.10
1490.00
59.60
60.20
61.80
50.60
34.90
55.90
84,4%
V1.rseV..M
56.00
60.00
1696.00
80.50
67.10
73.80
54.90
33.80
53.40
92.9%
VisionZip
57.60
62.00
1761.70
83.20
68.90
75.60
56.80
37.90
54.90
96.8%
DivPrune
59.25
62.03
1718.22
86.72
68.66
75.96
56.06
35.56
56.98
96.9%
VisionZip
58.90
62.60
1823.00
83.70
68,30
76.60
57.00
37.30
55.80
97.7%
MMTok
59.29
62.29
1779.14
86,25
68.82
76.35
57,03
35.67
58.59
97.8%
Retain 64
Tokens
89%
FastV
46.10
48.00
1256.00
48.00
51.10
55.00
47.80
34.00
51.90
75.6%
Vp.
52.70
56.20
1505.00
75.10
62.20
68.20
51.80
32.70
51.10
86.9%
VisionZip
55.10
60.10
1690.00
77.00
69.00
72.40
55.50
36.20
52.20
93.2%
DivPrune
57.78
59.28
1674.40
85.56
68.07
74.11
54.69
35,56
55.13
94.8%
VisionZip
57.00
61.50
1756.00
80.90
68.80
74.20
56.00
35.60
53.40
95.0%
MMTok
58.29
61.17
1715.33
85.77
69.16
75.20
56.01
36.11
57.15
96.6%
Table I: Performance Comparison on LLa VA-1.555. More details Appendix Table 16.
LLaVA 1. 1.5.7B (2023)
LLaVA-1 1.5 1.5.13 (2023)
LLaVA-NeXTTS (2024k)
LLaVA LLaVA-NeXT-I1R -eXTT-13B (2024.)
Method
576 tokens
576 tokens
Upper(Up.) 2880 tokens
UppertUp.) 2880 tokeus
Compress Ratio
67%
L 78%
89%
67%
L 78%
1. 89%
1 78%
J 8.%
1.94%
78%
89%
J 9.1%
Remain Token
128
64
192
128
64
Up. 640
Up. 320
Up. 160
Up. 640
Up. 320
VisionZip
97.9%
96.8%
93.2%
97.9%
97.0%
93.7%
97.5%
94.5%
90.4%
97.7%
94.7%
91.4%
DivPrune
98.0%
96.9%
94.8%
98.2%
96.9%
95.3%
97.1%
95.1%
92.4%.
97.1%
94.5%
92.0%
VisionZip
98.4%
97.7%
95.0%
98.7%
97.4%
94.8%
98.9%
97.61%
95.05%
98.8%
97.8%
94.6%
MMTok
98.7%
97.8%
96.6%
98.7%
97.5%
96.4%
98.7%
97.3%
95.1%
98.2%
96.4%
95.1%
Table 2: Comparison on LLaVA-1.5 and LLaVA-NeXTI. Details are in Appendix Tables 16 to 19.
4.1 PERFORMANCE COMPARISON ON DIVERSE TASKS
LLa VA-1.5-BB First, we compare our method with baselines using LLaVA-1.5-7B, which is a popular benchmark for vision token selection. The model has a fixed number of vision tokens for arbitrary visual inputs. As shown in Table 1, given the original 576 tokens, MMTok achieves the best performance (preserving on average 18.7/97.8/99 6/96.6% original performance of L.La VA-1.5-..-- when retaining only 192/128/64 tokens (reducing by 67/78/89% of tokens compared to 576), respectively. Specifically, our method outperforms DivPrune by 1.8% when using budget 64 tokens. Although the gap decreases with more tokens as expected, MMTok still surpasses all baselines without fine- tuning by least 0.7% with 192 tokens. In addition, compared the fine-tuning method, the proposed method is 1.6% better than VisionZipé with 64 tokens, which shows the potential of the training-fret free strategy. Since VisionZip and DivPrum show much better performance than FastV and Sparse we will include only them for comparison in the following experiments.
LLAVA--L5-13B The average performance over all benchmark datasets and token budgets is reported in Table 2, while detailed results can be found in Appendix Table 17. Although the model is larger, the observation is similar to the above 7B counterpart, where our method consistently outperforms the baselines with a clear margin.
Ve- 7B and 13B In addition to models that have fixed number of vision tokens, we further evaluate our method LLa (Liu et al., 2024a), which dynamically samples up to five images and processes them individually, resulting in up to 2880 vision tokens. To align the comparison with real applications, keep the dynamic settings VisionZip (Yang et al., 2025a) and

---

Published as a conference paper at ICLR 2026
have token selection performed in a fixed ratio. For example, with a maximum budget of 160 tokens, we retain 32 tokens per image in up to five images (32 x 5 = 160). The retained number of tokens becomes 128 if only four images are sampled by the VLMs according to the ratio of 160/2, 880. The same setting is used for all baselines as a fair comparison. As shown in Table 2, our method retains more than 95% of the original performance using only 5.5% of the tokens with budget of 160 tokens, indicating substantial redundancy in vision tokens and the effectiveness of the proposed strategy. Detailed results can be found in Appendix Tables 18 and 19.
GQA
MMB
MME
POPE
VQA Teur
SQA
OCRBench Acc.
Avg.l
Method
Acc.
Acc. f
P+C
FI
Acc.
Acc.
Dynanic Resolation (MinPix
256
28
28, MaxPix
2048
28).
Upper Bound (100%)
Avg. Tokens T
358.5
276.9
867.6
359.6
976.5
323.0
652.8
Qwen-2.5-VL-7B
60.48
83,25
2327
86.16
77.72
87.46
83,80
100%
Fixed Resolution (MinPix
MaxPix
2048
28 x 28).
Upper Bound (100%)
Awen-2.5-VL-7B
58.59
83.59
2339
86.09
76.64
86.91
76.60
99.3%
Retain 20% T
71.7
55.4
173.5
71.9
195.3
64.6
130.6
80%
VisionZip
56.80
80.33
2174
83.38
70.43
84,23
59,50
94.2%
DivPrune
56.70
76.98
2163
80.59
65.86
80.91
48.10
91.5%
MMTok
58.09
79,30
2217
82.38
70.49
81.61
$9.60
94.6%
Retain 10% T
35.9
27.7
86.8
36.0
97.7
32.3
65.3
90%
VisionZip
52.47
75.60
2003
78.90
63.78
82.30
36.90
87.5%
DivPrunc
53.43
72.85
1957
74.99
59.59
79.57
37.30
84.7%
MMTok
55.09
74.74
2051
78.75
63.90
80.47
43.60
88.5%
Retain 5% T
17.9
13.8
43.4
18.0
48.8
16.2
32.6
95%
VistonZip
46.28
67.53
1677
66.38
54.49
79.57
19.70
75.4%
DivPrune
49.01
65.89
1739
68.45
52.02
77.05
24.90
76.3%
MMTok
50.66
65.89
1796
71.35
55.95
77.19
30.70
79.0%
Token
100% %
Qwen-2.5-VL-7B
31.84
20.10
935
0.00
38.93
71.10
1.80
33.8%
Table Comparison on Qwen-2.5-VL-77 Avg.t are computed over 5 datasets. *When no visual tokens are provided, Qwen-2.5-VL. outputs "No" for all questions, leading to 0% F1. More detailed results are Appendix Table 20.
Qwen-2.5-VI--7B Finally, we compare different algorithms on a more advanced VLM, that is, Qwen- 2.5.VL-7B. Unlike previous work, adopts dynamic resolution and a ooken-merging layer. Those strategies help reduce the total number of vision tokens while demonstrating better performance. For example, the average number of input tokens is only 359.6 in Qwen on POPE, significantly less than 2880 tokens in LALVA--eeT Therefore, it is more challenging to apply the token selection algorithm in this strong model. Following experiments for LLaVA-NeXT, we conduct the evaluation under dynamic resolution for all methods. Due to distinct image pre-processing strategies in Qwen, we include image datasets in this comparison. Since ScienceQA (SQA) is low-IC dataset that will be discussed in Section 4.2 and all baselines perform poorly on OCRBench, the average performance is computed across the remaining datasets. For MMTok, we reduce Te to 0.01 for all datasets while other parameters remained.
First, we compare the dynamic resolution to the fixed number of tokens in Qwen as shown in the first two rows of Table 3. Although the model can use fixed number of about 2,048 vision tokens for different tasks, the performance is worse than that of the dynamic strategy, which has much fewer tokens. It shows that vision tokens are quite redundant for VLM tasks, and the sophisticated strategies in Qwen already compress the number to hundreds, providing even better performance. Based on the challenging dynamic setting, we further investigate whether token selection is still valuable. From Table 3, we can find that our MMTok can preserve nearly 95% of the original performance while further reducing the number of vision tokens to 20%. This observation demonstrates that even for models with token compression, the remaining tokens can still be redundant. The proposed method MMTok can effectively explore the most informative tokens and further reduce the number of vision tokens from hundreds to tens. Furthermore, our method is better than VisionZip with different budgets, which confirms the efficacy of our proposed multimodal coverage strategy. Finally, we can observe that even without any vision tokens, Qwen's 's performance with all tokens. This reminds us to investigate the contribution of vision to vision-language tasks in the next subsection, which can help to better evaluate the performance of token selection methods.
SQA is still close to its version
7

---

Published as a conference paper at ICLR 2026
4.2 COMPARISON ON HIGH IC TASKS WITH LIMITED VISION TOKENS
Although multimodal tasks rely on images for answers, the contribution of vision varies. Table summarizes the performance with/without vision tokens on different datasets. It is interesting to observe that even without any vision tokens, LLaVA-1.5 still preserves 92% of the original performance on MMMU and 91% on ScienceQA. Those tasks may fail to help adequately assess the efficacy of vision token selection. To mitigate the issue, we introduce Image Contribution (IC) to quantify the relative performance gain from all vision tokens, TC = (PerfAil summarize IC values in Table 4. According to the table, we can identify 5 and high-IC datasets for LLaVA and LLaVA-NeXT, respectively. Then, we compare different algorithms on those datasets in Table 5. To evaluate the performance with extremely aggressive compression ratio, extend the experiments from 64 tokens to 2 tokens. The comparison shows that our method can substantially preserve the informative vision tokens for VL tasks. Moreover, illustrate the performance ratio compared to the original result in Figure 1. On POPE, our method maintains about 80% original performance with only vision tokens, showing the importance of appropriate vision tokens. More results can be found in Appendix Tables 21 and 22.
(erfo)/Pertls and
LLaVA-15-7B
NEXT-7B
Different Token Budgets
Dalaset
Model/Method
All Zero
IC
All Zero
IC
64
32
16
8
4
2
MMB
64.7/19.33
67.9/17.87
2.801
LLaVA-L.5-BB (MMB, POPE, MME, SEED, GOA)
POPE
85.9/44.64
0.924
86. 86.4/25.84
2.344
VisionZip
90.0%
83.5%
69.7%
48.9%
43.8%
43.0%
MME
1862870.89
0
1842/867
1.125
DivPrune
93.1%
89.6%
81.4%
68.4%
54.3%
50.1%
SEED-1
66.14/37.05
0.786
70.2/37.43
0.875
MMTok (Ours)
94.7%
91.0%
86.4%
79.8%
71.4%
62.1%
GQA
61.9/37.65
0.644
64,2/38.23
0.679
LLaVA NeXT-7B (MMB, POPE, MME, SEED, GQA, TeaV(A)
TexiVQA
58.2/41.66
61.3/37.77
VisionZip
93.2%
87.5%
76.8%
52.6%
39.4%
38.5%
SQA
69.5/63.51
0.094
70.2/64.60
0.087
DivPrune
94.2%
89.7%
85.7%
79.9%
71.0%
57.8%
MMMU
36.3/33.3
0.089
1/3
0,112
MMTok (Ours)
95.7%
92.8%
89.6%
85.2%
79.2%
71.0%
Table 4: Demonstration of Image Contri-
Table 5: Comparison on high-IC tasks with different token budgets.
bution (IC).
4.3 ABLATION STUDY
We conduct comprehensive ablation studies to demonstrate each component in MMTok. All experi- ments are performed on LLaVA-1.5-7B with 64 tokens unless otherwise specified.
Multimodal
GQA
MMB
MME P+C
POPE FI f
SQA Acc. f
VQA Acc.
MMMU Acc. 1
SEED Acc. 1
Avg.
Coverage
Acc.
Acc.
Total 576 Tokens (100%)
LLaVA-1.5-7B
61.9
64.7
1862
85.9
69.5
58.2
36.3
58.6
100.0%
Retain 64 Tokens
88.9%
T-V (A/"))
56.82
59.62
1632.47
83.56
68.72
51.97
35.33
56.36
93.8%
V-V (M
58.14
59,88
1662.34
83.43
67.67
53.93
35.33
56.90
94.7%
Softmax T-V (18%"))
56.66
58.85
1674.11
83.69
68.57
52.01
35.33
56.37
93.9%
Softmax V-V (M
57.97
60.31
1684.33
84.31
68.07
55.90
35.89
56,88
95.7%
MMTok (ffe' + M/ee')
58.29
61.17
1715.33
85.77
69.16
56.01
36.11
57.15
96.7%
Table 6: Ablation on multimodal coverage in MMTok. The best performance with token selection is highlighted in bold and the second-best is underlined.
Unimodal Coverage vs. Multimodal Coverage The proposed method contains both text-vision coverage (T-V) and vision-vision coverage (V-V). We evaluate each component separately in Table Compared with the original similarity matrix, the softmax variant can obtain similar even better performance, which shows that calibration on similarity matrices will not hurt the performance, Then, combining coverage optimization different modalities shows improvement of about 1% over unimodal coverage, which demonstrates the complementarity between T-V and V-V coverages for various tasks. More ablation experiments can be found in the Appendix.
Token Selection vs. Resizing Resizing the image resolution is an effective way to reduce the total number of tokens shown in (Yang et al., 2025b). We compare the resize strategy with token
8

---

Published as a conference paper at ICLR 2026
budgets in Table 7. First, can be more effective than resizing under the same token budget. This is because resizing would ignore the redundancy between vision tokens. More importantly, resizing, MMTok can work better than the counterpart with the same token budget on full images. For example, with about 10% original tokens, MMTok with resizing can achieve 2170 on MME while that on original image is only 2051. Exploring resizing with token selection sufficiently can be our future work.
find that selecting vision tokens from the original large image
observe that incorporating with
Model
Image Resize Ratio
Token Avg.
MME P+C
DLL-2.5-VL-7B
867.6
2327
MMToo MMTok
173.5
2217
86.8
2051
Resize image to fixed ratio of original height and width, respectively
Qwen2.5-VL-7B Qwen2.5-VL-70
I/2
459.1
2274
1/4
349.6
2238
@wen2.5-VL-7B
1/8
276.0
1793
Retain 55% Tokens on 1/4 Resized Image
MMTok
1/4
192.3
2254
Retain 40% Tokens on 1/4 Resized Image 139.8
MM'ok
1/4
2215
Retain 25% Tokens 1/4 Resized Image
MM'Tok
1/4
87.4
21
Table 7: Comparison with resize strategy MME with Qwen2.5-VI--7B. The original image is denoted as resize ratio of 1. Qwen has a default minimal number of vision tokens as 256 to obtain meaningful results.
Model
Upper Token
Total Inler T(s)
POPE Inter T(a)
GPU Util.
Memary (+25 42 (B)
POPE
SEED TestVQA Acc.
MME P4C:
MMB
GQA
Avg. (5)
Acc.
Acc.
Acc.
Single Performance. Upper
ALAVA-NeXTII
2880
15204
1705
86.7%
4.59
86.22
71 71.89
64.33
1900
69.16
65.38
100.0
VisionZip
Upper 160
7551
866
$2.4%
1.92
76.32
61.18
58.33
1738.24
64.78
57.77
89.6
DivPrune
Upper
MMTos
Upper 160
7768
913
58.0%
1.78
85.11
65.45
55.91
1811.35
65.89
61.94
93.7
Table 8: Comparison of Inference Efficiency. All results are reproduced under the same hardware and evaluation settings. The initial memory usage for loading the model is 25.42GB.
Inference Efficiency Besides effectiveness,, examine efficiency in real scenarios. To mimic real applications, we report the total running time on different datasets in Table First, we profile the computational cost POPE. Obviously, all token selection methods help reduce the utility percentage of the GPU by about 30%, which shows that pruning is helpful for inference. Then, with fixed memory cost 25, 25,42GB for model loading, these methods can also help reduce the usage of running- time memory by more than 58.2% compared to the baseline. This reduction in computation and memory helps significantly improve the inference time POPE, where both VisionZip and our method can reduce the running time by about 50%. DivPrune runs little bit slower due to different strategy for handling multiple crops its official code. Although our method introduces two subproblems, that is, T-V and V-V to optimize, the running time is almost the same as the fast unimodal method, i.e., VisionZip. which confirms the efficiency of MMTok. The running time accumulated over 6 tasks demonstrates similar observation, where the performance of MMTok is better than VisionZip by 4.1%. This further demonstrates the efficacy and efficiency of our proposal.
Visualization While MMTok will leverage text information for vision token selection, the vision- vision coverage in our framework can help preserve the vision information and reuse the selected vision tokens for multi-ture turn conversation as shown in Figure 3. We can observe that MMTok selects tokens with text from Q1 but oision-vision coverage enables the following questions to be answered correctly with the selected vision tokens.

---

Published as a conference paper at ICLR 2026
In addition, also demonstrate how the answer changes with the number of selected tokens in Figure 4. Given a simple question as in the first example, only vision tokens can obtain the right answer. However, for the challenging question in the last example, the answer changes even with 16 tokens. The phenomenon implies that the appropriate number of vision tokens also depends on the hardness of the questions. Hardness-aware vision token selection is interesting future direction.
handbag image?
Question:
output water?
32 (5.59)
(LANTS
MM Vay
MMTok:
MMTol: Yey
Question: Is there only one in the image?
MMTok:
Q2: Is there person in the Image?
MMTak:
patted MMTok: Yes
image?
MMToi:
No.
MMTol:: Yes
Q4: Is there a vase in the Inage?
Question: image shows python output of the code "x 15 snaller than 10'
MMTakk No
17/259 (8399) 10
@5: there dining
image?
MMMok: Ver
Q6: Is there cup in the image?
No
MMTOk: Yes-
MMTok:
MMTok:
Figure 3: Multi-turn conversation by ap- Figure Answer changes with different number plying MMTok only with text from QI. of tokens. Hard questions need more vision tokens.
CONCLUSION
In this work, we propose a multimodal coverage framework, MMTok, to guide vision token selection accelerate the inference of VLMs training-free manner. Extensive experiments on benchmark datasets and representative VLMs demonstrate that our method outperforms the unimodal baselines without compromising efficiency. While text input may carry limited semantic information target for vision tokens to cover, lightweight agent VLM can be leveraged to provide additional meaningful text tokens to guide the selection of vision tokens, which can be the future direction.
6 ETHICS STATEMENT
To the best of our knowledge, this work has no potential ethical issues to disclose.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility this work, all implementation details have been clearly described, and we have also released the code.
ACKNOWLEDGMENTS
The authors thank Dr. Yebowen Hu and Dr. Kaiqiang Song for their valuable discussions, as well as their respective assistance with experiments and computing resource scheduling, Dr. Juhua Hu supported by Advata and EWA Gift Funding. Dr. Yanjie Fu is supported by the National Science Foundation (NSF) through grant numbers: 2426340, 2416727, 2421865, 2421803. All opinions, findings, conclusions, and recommendations in this work are those of the authors and do not necessarily reflect the views of the funding agencies.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9392-9401, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint (tt.:2552.113222. 2025.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 2022.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: aegg-ndd-plyy inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-lange lsionnlanguage models? arXiv preprint 2xXv:2240,22033 2024b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-492677 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprchensive evaluation benchmark for multimodal large language models. arXiv:2306. 133994 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stee Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904-6913, 2017.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.0700 2025.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Drew Hudson and Christopher D Manning. GQA: new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint 1902.10.0 2019.
Samir Khuller, Anna Moss, and Joseph Naor. The budgeted maximum coverage problem. Inf. Process. Lett., 70(1):39-45 1999.
Jure Leskovec, Andreas Krause, Carlos Guestrin, Christos Faloutsos, Jeanne M. VanBriesen, and Natalie Glance. Cost-effective outbreak detection in networks. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Jose. California, USA, August 12- 15, 2007, pp. 420 429.. ACM, 2007.
Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models. https: : /tt:://iii /////ttthu www./vvoovinnnleeeeooooooe 2024a. Version 10. Zenodo.
Lab/ 1mms eval, March
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
11

---

Published as a conference paper at ICLR 2026
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong. Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403. 18814, 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jimpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv:2305.102 10355, 2023b.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, oet, and world knowledge, January 2024a. URL https:
//llava-vl. vitpraavi.gttttaa wit.tt.th...... iioiiii//ba /to/blog//24-- 01-00-00000000000 30- llava-next.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et Mmbench: your multi-modal model an all-around player? In European conference computer vision, pp. 216-233. Springer, 2024c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-25211 2022.
George Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming. 14(1):265-299 1978.
Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-modal proxy learning for zero-shot visual categorization with CLIP. In Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp.
8748-8763. PMLR, 2021b.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. pravereee Adaptive token reduction for efficient large multimodal models. arXiv preprint 1033:22403.15388 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Chameleon Team. Chameleon: Mixed-moke early-fusion foundation models. arXiv preprint
arXiv:2405.09813 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. 12X.i:2312.11881 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017.
12

---

Published as a conference paper at ICLR 2026
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. arXiv preprint 2019.
Senqiao Yang. Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-19802, 2025a.
Senqiao Yang, Junyi Li, Xin Lai, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Visionthink: Smart and efficient vision language model via reinforcement learning. arXiv preprint arXiv:2507.1344. 2025b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi musti-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma. Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevim: Visual token sparsifica- tion for efficient vision-langues model inference. arXiv preprint 14X4422410004417, 2024.
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
We did not use LLM at all during the idea and writing stage this work.
B EXPERIMENTS
B.1 ADAPTIVE TEMPERATURE T
To further improve the calibration between M and M an adaptive visual temperature can be applied for each example. Concretely, when fixing the maximal similarity between the target text The desired
tokens and the whole set of vision tokens can be obtained as f(V; M* ), letting S = The temperature Tu should lead to a similar magnitude for the vision-vision similarity. The optimization problem can be cast
min (N; " (W; M" T2
For the default f. is monotone T. which can be solved efficiently by bisection search. However, the diagonal elements in Mov' can mislead the optimization due to their fixed value of 1. To mitigate the issue, the k-th largest value is applied to search for the temperature
min F(N: - (W; fx(:; = Σ max M.
Moreover, f& is not guaranteed to be a monotone function to T and we can search the value in (Tr, rul as suggested in (Qian et al., 2023), where it shows that the temperature between vision-vision should be higher than that between text-vision due to the modality gap.
We perform the evaluation high IC tasks in Table 9, As discussed above, the second largest value is adopted for searching the temperature in the set of (0.05, 0.1, 0.15, 0.2]. While the variant with adaptive temperature, i.e., MMTokAdape shows slightly better performance with budget of 16 tokens, the results over different tasks are almost the same, demonstrating that our method is insensitive to hyperparameters.
GQA
MMB
POPE
MME P+C t
SEED
Avg.
Method
Acc.
Acc.
F1
Acc.
Upper Bound: LLaVA-1.5-7B (576 Tokens)
LLaVA-1.5 7B
61.9
64.7
85.9
1862
58.6
100%
Retain 16 Tokens | 99 27%
MMTok
53.31
54.30
79.79
1550.65
56.67 56.66
88.6%
MMTokAdee Aduge
53.31
54.30
79.83
1565.10
88.7%
Table Fixed vs. Adaptive Temperature. Evaluation on LLaVA-1.5 7B with adaptive temperature T E (0.05, 0.15, 0.2).
B.2 POOLING STRATEGY FOR TEXT
Given an LLM, each word can be tokenized into multiple tokens. recover the semantic information of words, we may aggregate tokens from the same word. In this experiment, we explore different pooling strategies when computing T-V similarity. Concretely, we consider the pooling process either before or after computing the similarity matrix, where Max-pooling selects the token with the maximum feature value or similarity, Mean-pooling averages similarity over tokens, and Hirst-pooling simply retains the first token of a word. As shown in Table 10, there is no pooling strategy that consistently yields the best performance across eight datasets. Therefore, our method does not apply word pooling for simplicity.
B.3 MMTOK FOR REASONING TASK
To demonstrate the efficacy of MMTok on reasoning task, we conduct the experiments on the MMStar dataset (Chen al., 2024b). In Table 11, we can observe that vision token selection also help
14

---

Published as a conference paper at ICLR 2026
Pooling Method
GQA Ace. f
MMB
MME
POPE
SQA
VQA
MMMU
SEED Acc.
Avg. f
Position
1
P+C1
FIT
Acc.
Ace.
1
MMTok LLaVA-1.5-70 with 64 Tokens (Baseline)
None
58.29
61.17
1715
85.77
69.16
56.01
36.11
57.15
100.0% %
Pre- Pee-Pooling (Before Similarity Colculation)
Mean
Pre
58.01
61.00
1703
85.75
69.11
55.73
36.00
57.13
99.7%
Max
Pre
58.26
61.17
1704
85.64
68.82
$5.82
35.89
56.94
99.7%
First
Pre
58.39
61.34
1709
85.67
68.77
55.76
36.11
56.90
99.8%
Post Pooling (After Similarity Calculation)
Mean
Post
58,20
61.00
1690
85.67
68.77
55.77
36.22
57.16
99.7%
Max
Post
58,36
61.00
1711
85.61
68.77
55.68
36.22
57.04
99.8%
First
Post
58.39
61.34
1709
85.67
68.77
55.76
36.11
56.90
99.8%
Table 10: Word token pooling strategies for token selection on L.LVA-1.5-7B, Pre-pooling aggre- gates subword tokens before similarity computation, while post-pooling applies pooling afterward. We evaluate three methods: Mean (average pooling), Max (maximum pooling), and First (first subword). The baseline applies no pooling. The best is in bold and the second-best is underlined.
reasoning tasks. It should be noted that the major issue on these challenging tasks is that the original performance without selection is already quite low. Therefore, is hard show the significant difference when the upper-bound is limited. Nevertheless, our method is still better than baselines with clear margin and by selecting 32 out of 576 tokens, MMTok able to recover the performance of the baseline.
MMStar Metries
Method
Coarse
Fine-Grained
Instance
Logical
Math
Sci&Tech
Average
Baseline
LLaVA-1.5-7B
63.63
25.63
38.89
28.92
26.60
18.48
33.69
64 Tokens
VisionZip DivPrune
55.27
22.92
39.32
27.95
24.76
24.62
32,47
56.35
19.50
36.72
27.73
26.70
18.94
30.99
Ours
59.08
22.66
39.51
29.66
28.39
20.66
33.33
32 Tokens
VisionZip DivPrune
48.58
19.04
39.73
29.69
22.91
21.95
30.32
54.82
21.07
37.03
27.82
24.18
19.32
30.71
Ours
59.56
25.71
40.49
29.94
27.92
17.37
33.50
16 Tokens
VisionZip DivPrune
43.76
21.34
32.58
25.97
23.18
19.96
27.80
49.99
21.45
38.37
28.45
21.54
18.58
29.73
Ours
56.32
21.58
39.48
30.22
23.98
15.16
31.12
8 Tokens
VisionZip
27.93
21.26
25.71
21.84
20.18
17.83
22,46
DivPrune
47.25
21.05
33.89
25.75
20.32
16.76
27.50
Ours
54.11
21.26
35.09
29.56
20.34
15.04
29.23
O Tokens
Baseline
31.85
19.10
23.77
23.61
14.28
16.11
21.45
Table 11: Comparison on MMStar with LLaVA-1.5-7B.
B.4 INFERENCE EFFICIENCY FOR QWEN2.5-VL-7B
Besides the evaluation in Table 8, we also evaluate the inference efficiency of 1wen2.5-VL-7B in Table 12 using the MME task. We find that vision token selection can also accelerate the inference of state-of-the-art VLMss.
15

---

Published as a conference paper at ICLR 2026
Model
Token
Inference Time(s)
GPU. util.
Memory (+15.87GB)
Avg.
A6000 GPU Performance on MME
Qwen2.5-VL-1
867.6
675
77.0%
3.05
VisionZip
86.8
508
66.3%
0.41
DivPrune
86.8
423
55.1% %
0.71
MMTok
86.8
419
60.0%
0.71
Table 12: Comparison Inference Efliciency on Qwen2.5-VL-7B, The initial memory usage for loading the model is 15.87GB.
B.5 EFFICIENCY EFFECT OF THE NUMBER OF INPUT OR SELECTED VISION TOKENS
In MMTok, the similarity matrix can be constructed efficiently using the pytorch built-in libraries. Then, for token selection, MM'ok proposes greedy algorithm only with max operations and can run in (len) to pick vision tokens, where n the total number of vision tokens. Therefore, our method can scale well with vision tokens and with the number of selected vision tokens. In Table 13, we show the running time MMTok with the varying number of input and selected vision tokens. We find that with the same number of imput tokens, the running time is almost linear in the number of selected tokens, which confirms our analysis. Moreover, even with 2880 input tokens, the running time of MMTok is less than 7ms, which is negligible for real applications. It should also be noted that even for 2880 input tokens, the computation only costs about 13.93 GFLOPs.
#Input
#Select
Time(ms)
#Input
#Select
Time(ms)
#Input
#Select
Time(ms)
2880
160
6.417
1728
96
3.862
576
1.267
2880
80
3.733
1728
48
2.247
576
16
0.774
Table 13: Running time (ms) of MMTok with different numbers of input and selected vision tokens on LLAVA-NeXT-7B, The reported result is averaged over 100 a A6000 GPU.
B.6 TOKEN SELECTION IN DECODER
Following the common practice, token selection has been conducted mainly after the vision encoder. In fact, token selection also happen in the decoder. We conduct preliminary experiment the decoder in Table 14. We try to further select the vision tokens from 160 to 80 for an intermediate layer (i.e., L-24) of the decoder in LLaVA-Next MME. We find that keep the similar performance and further improve the token efficiency.
Model
Upper Tokens
MME (P+C) 1
LLaVA-Next-13B
2880
1901
MMTok
160
1811
MMTok
160->80 (L.24)
1846
MMTok
80
1717
Table 14: Comparison with vision token selection during decoding. token selection in the 24th layer of decoder.
We have an additional vision
C IMPROVED MMTOK
Since LLaVA-1.5 does not fine-tune the vision tower and also does not mask padding patches, explicitly exclude padding patches from the candidate token set and fix overflow bug that wasted one token. As shown in Table 15, these changes substantially improve accuracy while using fewer tokens. For fair comparison, report the results without the fix in the main text.
16

---

Published as a conference paper at ICLR 2026
GQA
MMB
MME
POPE
SEED-I
Avg
Method
Ace.
P+CT
FI
Acc.
Vanilla Baseline (576 tokens)
L15VA-1.5-78
61.9
64.7
1862
85.9
66.14
100.0%
Tokens
MMTok
55.95
58.39
1625
82.95
59.81
91.0%
MMTok++
56.61
58.76
1636
83.44
59.85
91.6%
16 Tokens
MMTok
54.30
1551
79.79
56.67
86.4%
MMTok++
54.05
54.98
1581
80.79
57.13
87.5%
8 Tokens
MMTok
49.06
1355
78
52.74
79.8%
MMTok++
50.80
49.31
1395
79.75
53.59
81.4%
Tokens
MMTok
43.93
36.94
1290
74.84
48.10
71.4%
45.08
40.21
1294
76.36
49.34
73.6%
2 Tokens
MMTok MMTok++
40.58
69
1122
68.95
42.89
62.1%
42.18
31.36
1237
72.97
45.27
67.3%
0 Tokens
Baseline
37.65
19.33
971
44.64
37.03
50.2%
Table 15: Evaluate MMTok++ on LLaVA-1.5-7B with Extremely Less Token Budgets.
D SELECTED TOKENS VISUALIZATION
To provide an intuitive understanding of our token selection process, we visualize selected tokens and their nearest words in Figure 5 and compare them with the diversity-basee method, DivPrune. From the columns (b) and (c). we can observe that MMToo selects top patches according to the word to patch similarity, which aligns well with the question semantically. In contrast, shown in columns (d) and (c). DivPrune selected top patches without any close semantic relation to the question. This further demonstrates that MM'Tok help significantly reduce the number of tokens without losing the semantic relation to the questions, so as to provide better performance.
E COMPLETE EMPIRICAL RESULTS
This section shows per-dataset results for all models and token budgets, including LLaVA-1.5 (7B/13B) (Tables 16 and 17), LLaVA-NeXT (7B/13B) (Tables 18 and 19), and Qwen-2.5-VL (7B Table 20). We report both raw scores and percentage retention relative to the full-token setting. We also report results with an extremely low number of tokens LLaVA-1.5-77 and -LaVA-NeeT--1 (Tables 21 and 22).
17

---

Published as a conference paper at ICLR 2026
GQA
MMB Acc. f
MME P+C
POPE
SQA
VQA
VQA Text
MMMU Ace. 1
SEED Ace. f
Avg.
Method
Acc.
FI
Acc. 1
Acc. t
Acc. 7
Upper Bound, 576 Tokens (100%)
LLaVA-I.5
61.9
64.7
1862
85.9
69.5
78.5
58.2
36.3
58.6
100%
Vanilla
100%
100%
100%
100%
100%
100%
100%
100%
100%
Retain 192 Tökens
66.7%
FastV (2024a)
52.7
61.2
1612
64.8
67.3
67.1
52.5
34.3
57.1
89.6%
85.1%
94.6%
86.6%
75.4%
96.8%
85.5%
90.2%
94.5%
97.4%
SP.rseVL.M (2024)
57.6
62.5
1721
83.6
69.1
75.6
56.1
33.8
55.8
95.5%
93.1%
96.6%
92.4%
97.3%
99.4%
96.3%
96.4%
93.1%
95,2%
VisionZip
59.3
63.0
1782.6
85.3
68.9
76.8
57.3
36.6
56.4
97.9%
(2025a)
95.8%
97.4%
95.7%
99.3%
99.1%
97.8%
98.5%
100.8%
96.2%
DivPrune
59.97
62.54
1762.23
87.00
68.66
76.87
56.97
35.44
58.71
98.0%
(2025)
96.9%
96.7%
94.6%
101.3%
98.8%
97.9%
97.9%
97.6%
100.2%
VisionZip (2025a)
60.1
63.4
1834
84.9
68.2
77.4
57.8
36.2
57.1
98.4%
97.1%
98.0%
98.5%
98.8%
98.1%
98.6% 6%.6%
99.3%
99.7%
97,4%%
MMTok
60.07
63.40
1773.86
86.42
68.76
77.11
57.68
36.33
59.21
98.7%
(Ours)
97.0%
98.0%
95.3%
100.6%
98.9%
98.2%
99.1%
100.1%
101.0%
Retain 128
Tokens
77.8%
FastV
49.6
56.1
1490
59.6
60.2
61.8
50.6
34.9
55.9
84.4%
(2024a)
80.1%
86.7%
80.0%
69.4%
86.6%
78.7%
86.9%
96.1%
95.4%
SparseVLM
56.0
60.0
1696
80.5
67.1
73.8
54.9
33.8
53.4
92.9%
(2024)
90.5%
92.7%
91.1%
93.7%
96.5%
94.0%
94.3%
93.1%
91.1%
VisionZip (2025a)
57.6
62.0
1761.7
83.2
68.9
75.6
56.8
37.9
$4.9
96.8%
93.1%
95.8%
94.6%
96.9%
99.1%
96.3%
97.6%
104.4%
93.7%
DivPrune (2025)
59.25
62.03
1718.22
86.72
68.66
75.96
56.06
35.56
56.98
96.9%
95.7%
95.9%
92.3%
101.0%
98.8%
96.8%
96.3%
98.0%
97.3%
VisionZip (2025a)
58.9
62.6
1823
83.7
68.3
76.6
57.0
37.3
55.8
97.7%
95.2%
96.8%
97.9%
97.4%
98.3%
97.6%
97.9%
102.8%
95.2%
MMTok
59.29
62.29
1779.14
86.25
68.82
76.35
57.03
35.67
58.59
97.8%
(Ours)
95.8%
96.3%
95.5%
100.4%
99.0%
97.3%
98.0%
98.3%
100.0%
Retain 64 Tokens
88.9%
FastV (2024a)
46.1
48.0
1256
48.0
51.1
55.0
47.8
34,0
51.9
75.6%
74.5%
74.2%
67.5%
55.9%
73.5%
70.1%
82.1%
93.7%
88.6%
SparseVL.M
52.7
56.2
1505
75.1
62.2
68.2
51.8
32.7
51.1
86.9%
(2024)
85.1%
86.9%
80.8%
87.4%
89.4%
86.9%
89.0%
90.1%
87.2%
VisionZip (2025a)
55.1
60.1
1690
77.0
69.0
72.4
55.5
36.2
$2.2
93.2%
89.0%
92.9%
90.8%
89.6%
99.3%
92.2%
95.4%
99.7%
89.1%
DivPrune (2025)
57.78
59.28
1674.4
85.56
68.07
74.11
$4.69
35.56
$5.13
94.8%
93.3%
91.6%
89.9%
99.6%
97.9%
94.4%
94.0%
98.0%
94.1%
VisionZip a (2025a)
57.0
61.5
1756
80.9
68.8
74.2
56.0
35.6
53.4
95.0%
92.1%
95.1%
94.3%
94.2%
99 99.0%
94.5%
96.2%
98.1%
91.1%
MM/oö
58.29
61.17
1715.33
85,77
69.16
75.20
56.01
36.11
57.15
96.6%
(Ours)
94.2%
94.5%
92.1%
99.9%
99.5%
95.8%
96.3%
99.5%
97.5%
Table 16:
Performance Comparison on I. The vanilla number of visual tokens is 576. The first line of each method shows the raw benchmark accuracy, and the second line is the proportion relative to the upper limit. The last column is the average value.
18

---

Published as a conference paper at ICLR 2026
GQA f
MMB
MME P+C
POPE
SQA Acc.
VQA
VQA Tea
MMMU f
SEED-I
SEED
Method
Acc.
FI
f
Acc.
Ace.
Acc.
Upper Bound, 576 Tokens (100%)
LLaVA-1.5
63.2
67.7
1818
85.9
72.8
80.0
61.3
36.4
66.9
61.6
Vanilla 13B
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
Tokens
66.7%
VisionZip (2025a)
59.1
66.9
85.1
73.5
78.1
59.5
36.4
65.2
61.201
97.9%
93.5%
98.8%
96.5%
99.1%
101.0%
97.6%
97.1%
100%
97.5%
99.4%
DivPrune (2025)
59.42
66.58
1781.50
86.76
73.03
77.98
58.46
36.56
65.72
60.83
98.2%
94.0%
98.3%
98.0%
101 .05
100.3%
97.5%
95.4%
100.4%
98.2%
98.8%
VisionZip (2025a)
61.6
67.1
84.5
72.7
78.6
59.9
36.4
98.7%
97.5%
99.1%)
98.5%
98.4%
99.9%
98.3%
97.7%
100%
98.8%
MMTok
59.67
67.70
1784.16
86.15
73.62
78.30
59.64
36.78
65.49
61.17
(Ous)
94.4%
100.0%
98.1
100.35
101.1%
97.9%
97.3%
101.0%
97.9%
99.3%
Retain 128 Tokens
77.8%
VisionZip (2025a)
57.9
66.7
1743
85.21
74.0
76.8
58.7
36.1
63.8
59.74
97.0%
91.6%
98.5%
95.9%
99.2%
101.6%
96.0%
95.8%
99.25
95.4%
97.0%
DivPrune
58.89
66.07
1748.56
86.53
72.48
77.10
58.17
35.56
64.22
59.49
96.9%
(2025)
93.2%
97.6%
96.2%
100.7%
99.6%
96.4%
94.9%
97.7%
96.0%
96.6%
VisionZip (20225a)
60.1
67.6
1736
83.8
73.0
77.6
59.2
35.4
64.9
97.4%
95.1%
99.9%
95.5%
97.6%
100.3%
97.0%
96.6%
97.3%
97.0%
MMToo
58.98
67.18
1756.20
86.22
73.38
77.57
59.22
35.44
64.26
60.11
97.5%
(Ours)
93.3%
99.2%
96.6%
100.4%
100.8%
97.0%
66.6%
97.4%
96.19
97.6%
Retain Tokens
88,
VisionZip (2025a)
56.2
64.9
1676
76.0
74.4
73.7
57.4
36.4
60.4
57.131
93.7%
88.9%
95.9%
92.2%
88.5%
102.2%
92.1%
93.65
100%
90.3%
92.7%
DivPrune (2025)
57.66
64.60
1777.93
84,80
72.09
75.20
57.11
35.22
62.44
57.70
95.3%
91.2%
95.4%
97.8%
98.7%
99.0%
94.0%
93.2%
96.8%
93.3%
93.7%
VisionZip (2025a)
58.1
65.6
1671
81.6
72.3
75.2
58.5
35.3
61.4
94.8%
91.9%
96.9%
91.9%
95.0%
99.3%
94.0%
95.4%
97.0%
91.8%
MMToo (Ours)
58.42
65.72
1763.39
84,39
72.98
76.55
58.40
35.22
63.39
59.51
96.4%
92.4%
97.1%
97.0%
98.2%
100.2%
95.7%
95.3%
96.8%
94.8%
96.6%
Table 17: Performance Comparison on LLaVA-1155-13, The vanilla number of visual tokens is 576. The first line of each method shows the raw benchmark accuracy, and the second line is the proportion relative to the upper limit. SEED-I represents SEED-IMG, SEED represents SEED-ALL Following (Yang et al., 2025a), Avg. is based on SEED-I instead of SEED.
19

---

Published as a conference paper at ICLR 2026
GQA
MMB
MME
POPE
SQA
VQA V2
VQA Tee
MMMU
SEED-I
AvE
Method
Acc.
P+C1
FI
Acc. f
Acc. 1
Acc.
Acc.
Acc.
Avg. Images 12
4.90
4.12
4.53
4.00
3.85
4.98
4.98
4.07
4.72
Avg. Tokens (a 576)
2822.4 4
2373.12
2009 28
2822 40
2217.60
2808 48
2808.48
2344.32
2718.72
Upper Bound:
2880 (5 576) Tokens (100%)
LLaVA-NeXT Vanilla 7B
64.2
67.9
1842
86.4
70.2
80.1
61.3
35.1
70,2
100%
100%
100%:
100%
100%
100%
100%
100%
100%
100%
Upper: 5 x
527
638
521
SparseVLM (2024)
65.7
77.1
57.8
34.6
93.9%
96.8%
96.2%
96.4%
96.3%
94.3%
98.6%
VisionZip (2025a)
61.3
66.3
1787
86.3
68.1
79.1
60.2
34.7
66.7
97.5%
95.5%
97.6%
97.0%
99.9%
97.0%
98.8%
98.2%
98.9%
95.0%
DivPrune
61.58
65.38
1773.04
85,51
67.82
78.94
55.41
36.89
67.56
97.1%
(2025)
96.3%
96.3%
90.4%
105.1%
96.2%
VisionZip4 (2025a)
62.4
65.9
1778
87.6
67.9
79.9
60.8
37.2
67.8
98.9%
97.2%
97.15
96.5%
101.4%
96.7%
99.8%
99.2%
106.0%
96.6%
MMTak (Ours)
62.27
65.29
1829.28
86.74
68.47
79.31
58.97
37.22
67.74
97.0%
96.2%
99.3%
100.4%
97.5%
99.0%
96.2%
106.0%
96.5%
Upper: 64
320
264
319
261
302
88.9%
Spa (2024)
64.3
55.9
34.4
89.9%
94.7%
92.0%
95.9%
91.6%
91.2%
98.0%
VisionZip (2025a)
59.3
63.1
1702
82.1
67.3
76.2
58.9
35.3
63.4
94.5%
92.4%
92.9%
92.4%
95.0%
95.9%
95.1%
96.1%
100.6%
90.3%
DivPrune (2025)
59.63
63.66
1731.04
83.47
67.82
76.64
53.84
37.11
65.35
95.1%
92.9%
93.7%
94.0%
96.6%
96.6%
95.7%
87.8%
105.7%
93.1%
VisionZip4 (2025a)
64.4
1770
59.3
38.0
65.9
97.6%
95.0%
94.8%
96.1%
99.8%%
96.2%
97.9%
96.7%
108.3%
93.99
MMTok (Ours)
60.96
64.35
1 1799.33
85.76
67.33
77.68
56.93
38.00
66.29
97.3%
95.0%
94.8%
97.7%
99.3%
95.9%
97.0%
92.9%
108.3%
94.4%
Upper: 32
160
137
132
145
139
130
151
94.4%
SparseVLM (2024)
46.4
92.9%
83.7%
82.8%
75.7%
93.4%
VisionZip (2025a)
55.5
60.1
1630
74.8
68.3
71.4
56.2
36.1
58.3
90.4%
86.4%
88.5%
88.5%
86.6%
97.3%
89.1%
91.7%
102.8%
83.0%
DivPrune (2025)
57.79
62.29
1658.25
79.36
68.02
73.92
52.42
36.44
62.54
92.4%
90.0%:
91.7%
90.0%
91.9%
96.9%
92.3%
85.5%
103.8%
89.1
VisionZipa (2025a)
63.9
1699
57.3
37.7
62.9
95.0%
90.7%
94.1%
92.2%
96.5%
96.2%
94.4%
93.5%
107.4%
89.6%
MMTak
60.05
62.97
1715.55
83.87
67.97
75.62
$4.17
37.89
64.54
95.1%
(Ours)
93.5%
92.7%
93.1%
97.1%
96.8%
94.4%
88.4%
107.9%
91.9%
Table 18: Performance Comparison on LLa The vanilla number of visual tokens varies by dataset due to dynamic image processing (max 2880 for 5 images). '.' means performance not available in the original paper.
20

---

Published as a conference paper at ICLR 2026
GQA
MMB
MME P+C1
POPE
SQA
VQA
VQA
MMMU
SEED-I
Avg.
Method
Acc. 1
Acc.
FI
Acc. 1
Acc. 7
Acc.
Acc.
Acc.
Avg. Images a
4.90
4.12
4.53
4:90
4.98
4.07
4.72
Avg. Tokens 376)
2822.4 A
2373.
2009
2822. 40
2217.60
2868, 18
2568.
2344.32
2718.72
Upper Bound: 2880 (5 x 576) Tokens (100%)
LLaVA-NeXT Vanilla 13B
65.4
70.0
1901
86.2
73.5
81.8
64.3
36.2
71.9
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
Upper: 5 128 640
527
580
638
638
521
601
77.8%
VisionZip (2025a)
68.6
1871
62.2
97.7%
98.4%
96.7%
100.5%
95.7%
DivPrune (2025)
62.82
66.84
1832,76
86.17
71.84
79.87
57.54
37.78
69.38
97.1%
96.1%
95.5%
96.4%
99.9%
97.7%
97.6%
89.5%
104.4%
96.5%
VisionZips (2025a)
63.7
66.6
1829
86.3
73.2
81.2
64.4
38.1
69.2
98.8%
97.4%
95.1%
96.2%
100.1% 1%
99.69
99.3%
100.2%
105.2%
96.2%
MMTok (Ours)
67.44
1874.63
86.72
80.55
61.06
37.11
69.61
98.2%
97.4%
96.3%
98.6%
100.6%
98.5%
95.0%
102.5%
96.8%
Upper: x 64 = 320
314
264
290
314
246
319
319
261
302
88.9
VisionZip
60.7
67.2
1805
82.0
70.3
76.8
60.9
35.6
65.2
94.7%%
(2025a)
92.8%
96.0%
95.0%
95.1%
95.6%
93.9%
94.7%
98.3%
90.7%
DivPrune
61.03
65.46
1802.79
84.86
71.39
77.6
55.75
36.00
66.75
94.5%
(2025)
93.3%
93.5%
94.8%
98.4%
97.1%
94.9%
86.7%
99.4%
92.8%
VisionZip4 (2025a)
62.5
66.9
1861
85.7
72.7
80.0
63.2
36.9
67.9
97.8%
95.6%
95.6%
97.9%
99.4%
98.9%
97.8%
98.3%
101.9%
94.4%
MMTak
62.95
65.55
1840.10 10
85.88
72.38
78.79
58.88
36.33
67.81
(Ours)
96.3%
93.6%
96.8%
99.6%
98.5%
96.3%
91.6%
100.4%
94.3%
Upper:
159
151
94.4%
VisionZip
64.9
58.4
91.4%
(2025a)
88.4%
92.7%
91.5%
88.9%
94.3%
88.5%
90.8%
102.2%
85.0%
DivPrune (2025)
59.34
64.78
1699.83
82.16
70.55
74.72
54.65
35.89
63.80
92.0%
90.7%
92.5%
89.4%
95.3%
96.0%
91.3%
85.0%
99.1%
88.7%
VisionZipa (2025a)
59.7
65.3
1766
84.0
72.0
77.6
60.8
36.0
64.4
94.6%
91.3%
93.3%
92.9%
97.4%
98.0%
94.9%
94.6%
99.4%
89.6%
MMTok
61.94
09.89
1811.35
85.11
72.43
55.91
37.11
95.1%
94.7%
94,19
95.3%
98.7%
98.5%
93.9%
87.0%
102.5%
91.0%
Table 19: Performance Comparison on LLaVA-NeXT-13R. The vanilla upper number visual tokens is 2880. SEED-I represents SEED-IMG.
21

---

Published as a conference paper at ICLR 2026
GQA Acc. 1
MMB
MME P+C
POPE
VQA Tex
SQA
OCRBench
Avg.i
Method
Acc. 1
FI
Acc.
Acc. 1
Acc.
Dynamic Resolution (MinPix
256 x 28 X 28, MaxPix 2048 x 28 28),
Upper Bound (100%)
Avg. Tokens T
358.5
276.9
976.5
323.0
652.8
Qwen-2.5-VL.-77
60.48
83.25
2327
86.16
77.72
87.46 100%
83.80
100%
Dynamic Res.
100%
100%
100%
100%
100%
100%
Fixed Resolution (MinPix
MaxPix
2048 x 28 x 28),
Upper Bound
(100%)
Qwen-2.5-VI.-7B
58.59
83.59
2339
86.09
76.64
86.91
76.60
99.3%
Fixed Res.
96.9%
100.4%
100.5%
99.9%
98.6%
99.4%
91.4%
Retain 20% T
71.7
55.4
173.5
71.9
195.3
64.6
130.6
80 %
VisionZip
56.80
80.33
2174
83.38
70.43
84.23
59.50
94.2%
(2025a)
93.9%
96.5%
93.4%
96.8%
90.6%
96.3%
71.0%
DivPrune
56.70
76.98
2163
80.59
65.86
80.91
48.10
91.5%
(2025)
93.8%
92.5%
93.0%
93.5%
84.7%
92.5%
57.4%
MMTok
58.09
79.30
2217
82.38
70.49
81.61
59.60
94.6% %
(Ours)
96.0%
95.3%
95.3%
95.7%
90.7%.
93.3%
71.1%
Retain 10% T
35.9
27.7
86.8
36.0
97.7
32.3
65.3
90%
VisionZip (2025a)
52.47
75.60
2003
78.90
63.78
82.30
36.90
87.5%
86.8%
90.8%
1 %
91.6%
82.1
94.
44.0%
DivPrune (2025)
53.43
72.85
1957
74.99
59.59
79.57
37.30
84.7%
88.3%
87.5%
84.1%
87.0%
76.7%
91.0%
44.5%
MMTok (Ours)
55.09
74.74
2051
78.75
63.90
80.47
43.60
88.5 %
91.1%
89.8%
1
91.4%
82.2%
92.0%
52.1%
Retain 5% T
17.9
13.8
43.4
18.0
48.8
16.2
32.6
95%
VisionZip (2025a)
46.28
67.53
1677
66.38
54,49
79.57
19.70
75.4%
76.5%
81.1%
72. 1
77.1%
70.1%
91.0%
23.5%
DivPrune (2025)
49.01
65.89
1739
68.45
52.02
77.05
24.90
76.3%
81.0%
79.1%
74.7%
79.4%
66.9%
88. 1%
29.7%
MMTok
50.66
65.89
1796
71.35
55.95
77.19
30.70
%
(Ours)
83.8%
79.2%
77.2%
82.8%
72.0%
88.2%
36.6%
0 Token
100 %
Qwen-2.5-VL 7B Text-Only 54.3%
31.84
20.10
935
0.00*
38.93 50.8%
71.10
1.80
33.8%
24.0%
40.0%
0.0%
88.6%
2.1%
Avg. is the average perfor- mance over the 5 datasets: GQA, MMB, MME, POPE, and VQA Text The first line of each method shows the raw benchmark accuracy, and the second line is the proportion relative to the upper limit. " "Qwen-2.5-VL. outputs "No" " for all POPE questions when visual tokens provided, resulting in 0% Fl score.
Table 20: Performance Comparison on 2.wen22.5VV--7--Innte
22

---

Published as a conference paper at ICLR 2026
GQA
MMB
POPE
VQA Acc.
MMMU
SEED-F
Avg@8
Avg@s
299
2:80%
Method
Acc.
Acc.
P4C
FI
Acc.
Acc.
Acc.
78
Vanilia Baseline (576 tokons)
61,9
64.7
1862
85,9
69.5
58.2
36.3
66.14
100.0%
100.0%
8/8
8/8
90% Threshold
55.71
58.23
1675.80
77.31
62.55
52.38
32.67
59.53
90.0%
90.0%
80% Threshold
49.52
51.76
1489.60
68.72
$5.60
46.56
29.04
52.91
80.0%
80.056
64 Tokens
VisionZip DivPrune
55.1
60.1
1690
77.0
69.0
55.5
36.2
57.84
93.0%
90.0%
5/8
8/8
$7.78
59.28
1674.40
85.56
68.07
54.69
35.56
60.21
94.4%
93.1%
7/8
8/8
58.29
61.17
1715.33
85.77
69.16
56.01
36.11
61.29
96.1%
94.7%
8/8
8/8
Tokens
VisionZip
51.78
68.77
83.5%
DisPrune
89.6%
MMTok
55.95
58.59
1624.72
82.95
$3.70
35.33
59.81
93.0%
91.0%
8/8
16 Tokou
VisionZáp
46.72
45.70
1326.89
51.84
67.67
49.74
35.00
46.66
78.4%
69.7%
2/8
3/8
DivPrune
51.10
53.09
1518
69.56
69.41
50.01
35.44
52.72
86.3%
81.4%
2/8
778
MMTOk
$3.31
54.30
1580.65
79.79
68. 82
50.04
34.22
$6.67
88.9%
86.4%
3/8
8/8
Tokens
VisionZip
39.47
24.40
1069.94
23.66
64.30
44.62
33.67
38.46
63.3%
48.9%
2/8
2/8
DivPrune
46.09
43.13
1294
52,10
67. 67.92
45.21
34.00
46.68
76.4%
68.4%
2/8
2/8
49.06
49.06
1355.31
78.46
66.83
45.71
34.11
52.74
83.5%
79.8%
3/8
3/8
Tokens
VisionzZip
36.57
63.56
33.78
DisProne
MMTok
43.93
1290.31
43.52
34.00
77.5%
71.4%
3/8
Tokens
VisionZáp
35.94
16.84
890.28
26.48
63.31
39.55
33.78
34.62
58.4%
43.0%
2/8
2/8
DivPrune
38.58
21.48
991
37.60
64.60
42.16
33.44
38.43
63.5%
50.1%
2/8
2/8
MMToR
40.58
25.69
1122,42
68.95
64.90
42.42
32.67
42.89
70.9%
62.1
2/8
3/8
Tokonr
Baseline
37.65
19.33
970.89
44.64
63.51
41.66
33.33
37.03
63.2%
50.2%
2/8
2/8
Table 21: Extended Performance Comparison with Extremely Less Token Budgets on LLaVA- 1.5-7B. SEED-I indicts SEEDBench-Image Image. Avg@8 is across all datasets, while Avg@5 is High-IC datasets.
23

---

Published as a conference paper at ICLR 2026
GQA
MMB
VQA Text
MMMU
SEED-I
Avg@8
Avg@6
2.90%
2:80%
Method
Acc.
P-C1
FI
Acc.
Acc.
Acc.
Acc.
78
Vanilla Baseline (Upper 2850 tokens)
LLaVA-NeXT-78
64.2
67.9
1842
86.4
70.2
61.3
35.1
70.2
100.0%
100.0%
8/8
8/8
90% Threshold
$7.78
61.11
1657.80
77.76
63.18
55.17
31,59
63.18
90.0%
90.0%%
80% Threshold
51.36
54.32
1473.60
69.12
56.16
49.04
28.08
56.16
80.05:
80.0%:
Upper 32x5 Tokems (160 Tokens!
5.6%
VisionZip DivPrune
55.5
60.1
56.2
36.
90.6%
87.5%
8/8
57.79
MMTok
60.05
62.97
1716
83.87
67.97
$4.17
37,89
64.54
95.256
92.8%
7/88
8/8
16x.5 Tokens (80 Tokens)
2.8%
VisionZip
50.80
50.69
1431
61.82
66.93
51.65
34.44
51.77
81.8%
76.8%
278
3/8
DivPrune
55.73
39.97
1575
74.74
66.83
50.35
36.56
59.48
89.2%
85.7%
2/8
8/8
MMTak
58.23
62.54
1681
81.89
67.13
49.56
36.11
61.86
92.0%
89.6%
6/8
8/8
Upper 8x.5 Tokems (40 Tokens)
1.4%
VisionZip
41.87
28.35
999
21.22
64.25
42.85
31.44
41.93
62.1%
$2.6%
2/8
DivPrune
66.78
83.7%
MMTok
54.52
Upper 4x5
Tokenz)
0.7%
VisionZip
36.56
18.38
0.40
63.56
35.36
31.56
34.98
52.1% 1%
39.4%
2/8
DivPrune
49.57
48.54
1324
52.14
65.94
44.06
31,78
$1.25
76.3%
71.0%
2/8
2/8
MMTok
49.60
51.20
1457
82.41
66.88
42.33
33.67
55.34
83.39
79.25:
3/8
3/8
Upper 2x5 Tokens (10 Tokens)
0.33%
VisionZip
36.17
17.96
823
0.80
62.91
32,84
30.56
34.31
50.9%
38.5%
0/8
2/8
DivPrune
45.19
37.11
25.48
65.25
40.33
33.22
45.54
66.8%
57.8%
2/8
Baseline
38.23
Table 22: Extended Performance Comparison with Extremely Less Token Budgets on LLaVA- NeXT-7B. Avg@8 is across all datasets, while Avg@6 is across 6 High-IC datasets. The "x5" notation indicates maximum sampling to images. Average percentages are calculated relative the vanilla baseline for each metric.
24

---

Published as a conference paper at ICLR 2026
Question: ls there traffic light in the image?? Answer the question using single word phrase.
Wordi-Th
Neech-h--uuuhhss Nuthes
MKTok BivPrune
Question: Is there chair in the image? Answer the question using single word phrase.
Mrl Dwoin
Original
Nuches
MKTok DivPrune
Question: Is there clock in the image? Answer the question using single word or phrase.
Meric A1
Petches Simllactty
MMTikk
Devtrrune
(el MNNlok Diei/ruee
Question: Is there person in the image? Answer the question using a single word or phrase.
Original
Words Petches Simllactty
MbTikk
(dl Devtrune
(e] MNNikk Div//ruee
Question: Is there cheir in the image? Answer the question using single word or phrase.
Original Words-TIl Petches Similertty
MMTiok
(8) DevPrune
(e) MNTiok Div/rrue
Question: Is there an orange in the image? Answer the question using single word or phrase.
Original Worth-Til-Nitssee Similanty
MMTiak
(8) DevPrunn
(6) MNTiok DivPrune
Figure 5: Visualization of Selected Tokens. Compared with the diversity based method, DivPrune, our method selected token coverage necessary token associate to language context.
25


Reviews:
[{"review_id": "uYNJYG8yXX", "reviewer": "Reviewer_8R44", "summary": "This paper proposes MMTok, a training-free method for vision token selection in Vision-Language Models (VLMs) by formulating the problem as a multimodal maximum coverage problem.  The approach leverages both text and vision tokens to select a subset of vision tokens that maximally cover semantic information from both modalities.  The method is evaluated across multiple VLMs and benchmarks, showing strong performance in maintaining accuracy while reducing token counts and improving inference speed.", "questions": "See the weaknesses part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThe paper addresses an important and practical problem: improving inference efficiency of VLMs without fine-tuning. And the multimodal coverage formulation is well-motivated, combining both text-vision and vision-vision similarity in a principled manner.\n\n-\tExtensive experiments across multiple models (LLaVA-1.5, LLaVA-NeXT, Qwen) and datasets demonstrate the generality and effectiveness of the method.\n\n-\tThe method achieves impressive results, e.g., 1.87× speedup on LLaVA-NeXT-13B with 98.7% performance retention, and maintains strong performance even with very few tokens (e.g., 4 tokens on LLaVA-1.5-7B).", "weaknesses": "-\tMy main concern lies in the efficiency part. The authors only provide efficiency test on base version. The agent-based text enrichment is presented as optional and shows mixed results, but its computational overhead and when it is most beneficial are not thoroughly discussed.\n-\tWhat is the efficiency cost under the practical experimental settings? e.g. The experiments in Table 3 with Qwen2.5-VL.\n-\tHow does the time complexity of MMTok scale with the number of vision tokens, especially compared to the baselines?\n-\tThe visualization results are not fully convincing, which weaken the interpretability of the proposed approach.\n-\tThe performance drop is severe under some benchmarks, which yield questions about the availability among real-world scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "68rI1cNHSj", "reviewer": "Reviewer_FeoR", "summary": "This paper proposes MMTok, a training-free approach for vision token compression in VLMs. They formulate the multimodal token selection problem as a maximum coverage problem, define the coverage and select a subset of vision tokens to cover text tokens and original vision tokens via a greedy approach. Experimentally, they found MMTok achieve better performance compared to other visual token compression approaches on the basis of the LLaVA-Next and Qwen2.5-VL baseline.", "questions": "1. **Performance over reasoning VLMs.** Reasoning VLMs also face the problem of token redundancy. I am curious if MMTok work with reasoning VLMs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Reasonable idea.** Using cross-modal correlation for visual token compression is reasonable.\n\n2. **Solid experiments.** MMTok works with LLava-1.5, LLava-Next and Qwen2.5-VL baseline, especially with cutting-edge SOTA VLMs like Qwen2.5-VL. This demonstrates the robustness and effectiveness of this method.\n\n3. **Training-free.** MMTok is training-free and plug-and-play, easy to apply to multiple foundation VLMs.", "weaknesses": "1. **Compare with resize baseline.** Experiment results in [1] show that simply resizing the raw image yields good performance and low latency. I am looking forward to the author comparing your approach with the simple resize approach and adding a discussion in the paper.\n\n2. **Work on multi-turn conversation.** MMTok relies on text instruction to prune vision tokens, therefore hard to apply with multi-turn conversation. The author should discuss this situation and propose some solutions.\n\n3. **About the MMTok-Agent.** As shown in Table 2, looks like the performance gain of MMTok-Agent is limited over MMTok, and tha author didn't show this result over Qwen2.5-VL in Table 3.\n\n4. **Runtime.** The author should show the runtime and flops for MMTok.\n\n[1] VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gVwWm6rpQu", "reviewer": "Reviewer_TSCU", "summary": "This paper introduces MMTok, a multimodal token selection framework designed to enhance the inference efficiency of Vision-Language Models. The key idea is to leverage both vision and text modalities for vision token pruning, rather than relying on unimodal information as in prior work. The authors formulate the token selection as a maximum coverage problem, aiming to optimize a subset of vision tokens that best cover both text tokens and the original visual space. Additionally, a VLM agent is utilized to refine text representations, further improving pruning quality. Experimental results on multiple benchmark datasets and various VLM architectures demonstrate that MMTok effectively reduces redundancy and accelerates inference.", "questions": "1. Is there a **fundamental difference** between Coverage Maximization and maximizing Conditional Diversity for Token Pruning? This is important, as there are already works that perform token pruning based on differentiable diversity (e.g., [1]).\n\n[1] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs, NeurIPS 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Formulating vision token pruning as a maximum coverage problem is both novel and insightful. This formulation naturally captures visual–textual interactions by optimizing vision tokens to jointly cover textual semantics and the original visual space, providing a solid mathematical foundation for multimodal token selection.\n\n2. The authors conduct impressive experiments on five models and demonstrate improvements over previous methods. Moreover, the ablation study is detailed, particularly in the efficiency analysis section.\n\n3. The visualizations presented in the Appendix are impressive and highly detailed.", "weaknesses": "1. The proposed VLM agent offers limited contribution. As shown in Table 1, it yields at most a 0.2 improvement and even leads to performance degradation in some cases, while introducing additional time overhead.\n\n2. Compared with finetuned VisionZip, MMTok performs notably better under the 64-token setting. However, its advantage diminishes under the 128- and 192-token settings. I suggest evaluating MMTok on more challenging benchmarks, such as MMStar and MathVista, to better highlight its strengths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2B4o1epXtY", "reviewer": "Reviewer_EEd7", "summary": "This paper proposes a novel token pruning method for VLMs. The authors formulate token pruning as a maximum coverage submodular optimization problem: finding a subset of visual tokens that covers both the textual tokens (text-visual coverage) and the visual tokens themselves (visual-visual coverage). The authors also provide an optional agent (MMTok-Agent) that enriches text guidance by adding preliminary text output from a small VLM. Experimental results show that the proposed method achieves significant speedup while pruning most tokens.", "questions": "1. Unlike most approaches that select tokens at the decoder stage, what are the advantages of selecting tokens at the encoder stage? Why can't apply to both the encoder and decoder?\n\n2. Are there any datasets where performance drops significantly at higher pruning rates, while others remain relatively stable?\n\n3. To my limit knowledge, token pruning methods can ensure that overall performance does not drop significantly. However, they may introduce knowledge boundary drift, meaning that answers that were previously correct may now be incorrect, and vice versa. This possibility could have negative consequences in scenarios where critical questions must be answered correctly. If possible, could the authors conduct a simple analysis to see whether the proposed method leads to changes in individual case performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with easy-to-understand figures and tables.\n\n2. The authors provide excellent theoretical support, supplemented by experiments. The design is well-designed.\n\n3. The experiments are solid, experimented on nine datasets and various virtual machine models, demonstrating robust generalization capabilities.", "weaknesses": "1. The role of the agent is not very clear. From the experiments, it seems to bring only slightly improvements. Although the authors claim it does not affect inference efficiency, it still introduces additional computation, which somewhat contradicts the very movitation of token pruning.\n\n2. The authors provide solid theoretical justification and a novel selection methods; however, this method appears to be applicable only to the encoder side. Why not consider expand it to decoder? Please correct me if my understanding is mistake.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)