SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
Authors: Jinhong Deng
Wen Li
Joey Tianyi Zhou
Yang He
Venue: NeurIPS 2025 poster ()
Keywords: Visual Token Pruning
Efficient MLLM
Primary area: deep_learning
TL;DR: A novel visual token pruning method that jointly maximizes both the saliency and coverage of the selected visual tokens to better preserve semantic completeness.

Abstract: Multimodal Large Language Models (MLLMs) typically process a large number of visual tokens, leading to considerable computational overhead, even though many of these tokens are redundant. Existing visual token pruning methods primarily focus on selecting the most salient tokens based on attention scores, resulting in the semantic incompleteness of the selected tokens. In this paper, we propose a novel visual token pruning strategy, called **S**aliency-**C**overage **O**riented token **P**runing for **E**fficient MLLMs (SCOPE), to jointly model both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. Specifically, we introduce a set-coverage for a given set of selected tokens, computed based on the token relationships. We then define a token-coverage gain for each unselected token, quantifying how much additional coverage would be obtained by including it. By integrating the saliency score into the token-coverage gain, we propose our SCOPE score and iteratively select the token with the highest SCOPE score. We conduct extensive experiments on multiple vision-language understanding benchmarks using the LLaVA-1.5 and LLaVA-Next models. Experimental results demonstrate that our method consistently outperforms prior approaches.

20 21 have signifi- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
Jinhong Deng Wen Li Joey Tianyi Zhou Yang He
Univeessity of Electronic Science and Technology of China (UESTC)
"Shenzhen Institute for Advanced Study, UESTC CFAR, Agency for Science, Technology and Research (A*STAR), Singapore IIHPC Agency for Science, Technology and Research (A (A*STAR), Singapore [jhdengvision, liwenbnu) @gmail. com, (joey_zhou, he_yang) Ca Ba-star. edu.sg
Abstract
Multimodal Large Language Models (MLLMs) typically process large number of visual tokens, leading to considerable computational overhead, even though many of these tokens are redundant. Existing visual token pruning methods primarily focus on selecting the most salient tokens based on attention scores, resulting in the semantic incompleteness of the selected tokens. In this paper, we propose novel Oriented token Pruning
visual token pruning strategy, called Saliency-Covenn for Efficient MLLMs (SCOPE), to jointly model both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. Specifically, we introduce a set-coverage for given set of selected tokens, computed based on the token relationships. We then define coken-coverage gain for each unselected token, quantifying how much additional coverage would be obtained by including it. By integrating the saliency score into the token-coverage gain, we propose our SCOPE score and iteratively select the token with the highest SCOPE score. We conduct extensive experiments on multiple vision-language understanding benchmarks using the LLaVA-1.5 and LLaVA-Next models. Experimental results demonstrate that our method consistently outperforms prior approaches. Our code is available
Introduction
Recent advances in Multimodal Large Language Models (MLLMs) 24
25
51
47 8 by integrating powerful vision encoders (34] with autoregressive large language models 33 These systems typically tokenize visual inputs into sequences of patch-level embeddings (i.e., visual tokens), which are then fed into the language model via either projection modules 24 attention- based fusion mechanisms [19]. Despite its effectiveness, this paradigm incurs substantial computational overhead, particularly when processing high-resolution images or temporally dense video inputs. For instance, a ViT encoder [I] applied to a 448 x 448 image can generate over 1,000 visual tokens. This number increases rapidly in resoluesnn and video scenarios involving multiple frames. Since these tokens are jointly processed with textual tokens, the computational cost of self-attention grows quadratically with the
cantly advanced open-ended visual understanding tasks 27
number of visual tokens [30 25 limiting their deployment in practical applications such as edge
computing and robotics 17 33
"Corresponding Author.
This work was completed during Jinhong Deng's 's internship at CFAR, SSTAR
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Head Tokens
Tail
Salency
Saetsicc of d4
Scoams
MIKE
POPt
NIE
Altention Valu
0.10
Mac
0000
0.08
Repid Drop
0.06
917
942
Ours
3.04
These sail tokans are loss
0.02
distnguishable by their attention values.
912
Unselested Token
Token Rank (Sorted)
30
50.100 120
s0u0
66
(a) Semantic Completeness Analysis
(b) Skewed Attention Distribution
(c) Performance Comparison
Figure 1:
(a) Semantic Completeness Analysis. We visualize the selected tokens using saliency- based rule (Top) and our method (Bottom). The saliency score corresponds to the visual attention assigned to the CLS token. Our method selects tokens that maximize coverage while preserving the most dominant visual information.
(b) Skewed Attention Distribution. We show the averaged attention distribution of the top 128 tokens on the MME benchmark. The attention weights rapidly
flatten, making tail tokens less distinguishable based on their attention values. (c) Performance comparison with prior methods across various benchmarks. The model is LLa 7B, and the number of retained tokens is 64.
However, not all visual tokens contribute equally the final outputs of the language model [7]. Many background or repetitive patches carry redundant or less informative content 6 [.] This motivates the need for efficient visual token pruning or compression, aiming to retain only the most relevant tokens while discarding those that are redundant. To this end, recent works 77 41 49
have proposed various pruning strategies that select salient visual tokens based on attention scores, i.e. visual attention from text prompts or from the CLS token in vision transformers. For instance, VisionZIP 43 selects visual tokens that receive the highest attention from the CLS token.
While effective, saliency-based visual token pruning methods exhibit notable limitations in complex lssion-language tasks. First, they inevitably compromise semantic completeness by discarding key contextual information essential for comprehensive visual understanding. For example, in response to the question "Where is the cat?", attention may focus primarily on the object "cat" while neglecting its surrounding context. The saliency-based methods typically concentrate on small subset of visual tokens (see Fig. [ resulting in significant semantic loss. Moreover, saliency-bas based approaches often suffer from highly skewed attention distribution, where only a few tokens receive substantial attention while the rest exhibit nearly uniform (i.e., flat) attention values as shown Fig. [b)) This hampers the discriminability among tokens, making it difficult to differentiate potentially informative ones from truly redundant ones.
To address the above challenges, we propose novel visual token pruning strategy, named Saliency- Coverage Oriented token Pruning for Efficient MLLMs (SCOPE), which jointly models the saliency and coverage of selected visual tokens to preserve semantic completeness. Specifically, we first define set-coverage score for a selected token set based on token relationships and introduce a token-coverage gain for each unselected token, measuring the additional coverage achieved by including that token. We then propose SCOPE score to integrate the token saliency score into the token-coverage gain, and iteratively select the token with the highest SCOPE score. This enables our method to retain tokens that not only contribute the most salient information but also ensure broad semantic coverage (see Fig. [a).
To evaluate the effectiveness of our SCOPE, we conduct extensive experiments on variety of vision-language understanding benchmarks using popular MLLMs, including LLaVA-1.5 [24) and LLaVA-Next (25]. The results demonstrate that our method consistently outperforms prior approaches by significant margin (see Fig. [c)). For instance, SCOPE achieves 9x reduction in the number
of visual tokens while retaining 96.0% of the original performance on LLaVA- 7B 24
Our main contributions are summarized as follows:
We reveal the limitation of the saliency-based visual token pruning methods, which unfortunately ignore the semantic completeness of the selected visual tokens and suffer from a highly skewed attention distribution problem.

---

We propose a novel visual token pruning strategy, named Coverrge Oriented token Pruning for Efficient MLLMs (SCOPE), which jointly models saliency and coverage of the retained visual tokens to preserve semantic completeness.
We integrate SCOPE into representative MLLMs such as LLaVA-1.5 and LLaVA-Next without training, and demonstrate its effectiveness multiple vision-language benchmarks, achieving favorable trade-off between computational efficiency and task performance.
2 Related Work
Multimodal Large Language Models (MLLMs). Large Language Models (LLMs) 1 37 10
16 have achieved remarkable success in wide range of language understanding and generation tasks. Building on this foundation, Multimodal LLMs (MLLMs) 24 25 21
have shown impressive progress in visual understanding. A prevailing paradigm in MLLMs projects visual features into a sequence of visual tokens via a Ausion-to-languaee projector, and feeds them into the LLM alongside text tokens, exemplified by LLaVA 24 25]. Qwen-VL 4 and Mini-Gemini [2]].
50 19
However, real-world images are often high-resolution, resulting long visual token sequences that significantly slow down inference in MILMs 23 a 672 672 image into over 2,000 tokens. The situation worsens when handling multiple images or videos, further increasing the number of visual tokens. This highlights the need for effective strategies to reduce token length and accelerate vision-language inference.
30 20
9 For example, LLaVA-Nexx [25] converts
Visual Token Truning/Compressinn in MLLMs. A number of recent studies 44
7 have focused on reducing visual token redundancy in MLLMs without requiring additional model training. Most of these methods 77
40
49 rely on specific attention scores to rank token saliency, such as text-to-vision attention LLMs or CLS-token attention in vision transformers. They typically retain only the top-ranked tokens using top-k strategy, i.e., selecting tokens with the highest attention scores. For instance, FastV 7 leverages early-layer text-to-vision attention to retain salient tokens. SparseVLMM 49 uses important textual words as a rater to guide token selection. VisionZip [3 applies CLS-based attention in the vision transformer for token pruning. To further increase the information density of the selected tokens, several approaches attempt to merge semantically similar tokens 43 49 35 DivPrune 2 selects visual tokens by maximizing the diversity of selected tokens. In contrast, our method jointly considers both saliency and coverage, aiming to preserve semantic completeness while reducing token redundancy.
3 Method
In this section, we first introduce the preliminaries of visual token pruning and discuss the instantiation
of saliency-based pruning methods in 3.1. In S.2 provide coverage analysis and show that saliency-based methods often suffer from low coverage. Finally, we present our proposed Aaliency-Coverre Oriented token Pruning for Efficient MLLMs (SCOPE) in Sec 3.3
3.1 Preliminary
Visual Token Pruning. The core architecture of LLMs consists of stacked self-attention layers and networks (FFNs) (38], where the computational complexity grows quadratically with the input sequence length. In MILMs, input images are typically high-resolution, resulting in long sequences of visual tokens. For instance, L.LVVA (66] produces 576 visual tokens for single image, which is often significantly longer than the corresponding text input in many visual understanding tasks. Furthermore, visual tokens often exhibit substantial redundancy 7 due to repeated patterns and limited informational content in background regions.
Therefore, reducing the number of visual tokens is essential for enhancing the computational efficiency of MLLMs. In particular, = {v1..... .........,, UNJ denotes the full set of N visual tokens extracted from the image, where each token vi E IRd represents local region of the image. The goal of visual token pruning algorithm A is to select a small subset of visual tokens S = [vi,.... ux A(V), where K & N. The objective of visual token pruning is to ensure that the model's output based on S closely approximates the output based on the full set V. Formally, the pruning objective can be
3

---

formulated as:
min (M((S)T) M(V.T)). $
(1)
where M(- T denotes the output of the vision-language model given visual input (either V or S) and text input T, and a function to measure the output difference of LLM.
Saliency- Visual Token Pruning, The saliency-basees based visual token pruning methods aim to reduce token redundancy by retaining the most salient visual tokens while discarding the less informative ones. The core challenge lies in how to effectively measure the saliency of each visual token. Several prior works 41 43 estimate saliency by leveraging attention scores. Specifically, the attention matrix A is calculated as:
QK Vd
A Softmax
where is the embedding dimension, Q and K is the query and key matrices in the standard attention mechanism. These attention scores indicate the interaction strength between tokens, guid- ing the identification of highly salient tokens. In practice, in the vision encoder of CLIP [34]. the [CLS] token used to aggregate global information from the entire image. Therefore, the attention scores from the [CLS] token to the visual tokens serve as a reasonable proxy for to- ken saliency. Based on these saliency scores, token pruning methods typically adopt top-k selection strategy to retain only the most salient visual tokens. This approach effectively re- duces visual token redundancy and significantly accelerates MLLM inference across various tasks.
3.2 Coverage Analysis
Although salieney-based pruning methods effec- tively identify important tokens based on attention scores, they inevitably discard certain semantically critical tokens that are essential for comprehensive visual understanding. Semantic completeness, how- ever, is crucial for accurately responding to wide range of instruction prompts in MLLMs. Further- more, saliency-based approaches often suffer from highly skewed attention distributions, where a small subset of tokens receives disproportionately high at- tention, while the remaining tokens exhibit nearly uniform (i.e., flat) attention values. This skewness un- dermines token discriminability, making challeng- ing to distinguish between potentially informative tokens and truly redundant ones. To quantitatively as- sess the representational completeness of the selected tokens, we introduce the notion the O-coverage (see Definition D. which measures the degree to which the retained tokens cover the semantic space of the full token set.
Coverage
Random
Coarag Dnly
0.50
0.55
0.60
0.65
0.70
0.75
0.60
0.85
0.50
Figure Comparison of 0-coverage across different token pruning criteria. The experi- ments are conducted on the MME benchmark, with 64 tokens selected out of the original 576 in LLaVA 1.5 7B.
Definition 1 (0- (0-Coverage). Let V {ur E 12d i L....,n] n. denote the full of tokens extracted from an input image, and let V C V be a subset of selected tokens. For a given similarity threshold 0 E [0, 1), we say that a token v that their cosine similarity satisfies:
covered by V if there exists least one token n' V such
sim(u, 0') = |el|
2 0.
(3)
|0
The 0-coverage of V over V is then defined the proportion tokens in V that are covered by V/:
Coverage, (V*, V)
1
Σ1
max sim(u, (u. 0') o0)
(4)
V
v'eV"
DEV
where II(-) the indicator function, which equals if the condition holds and otherwise.
This definition provides semantic-aware metric to quantify how well the selected tokens set y represents the full set. A higher value of 0 imposes stricter similarity criterion, typically leading to lower coverage but ensuring that the retained tokens are more semantically representative.

---

LLM
Saliency-Coverage Oriented token Pruning for Efficient MLLMs (SCOPE)
Selecttd Set S
Updated Set S S (
Repeat
0.95
0.68
Projector
Text Tokenizer
Attention Score
Token Coverage A/ee S]
Selected Token eng Aer 42.4) 1era
SCOPE
overcast?
Prompt
Score
Retained Takens
Image
0.76
Encoder
Coeine Sinvlaniies
Taken
Unselected
Figure 3: An overview of the proposed visual token pruning framework. The left part illustrates how our method reduces the number of visual tokens before feeding them into the LLM, thereby accelerating inference in MLLMs without requiring additional model training. The right part provides a detailed view of our SCOPE method, which jointly optimizes saliency and coverage to select compact yet semantically representative subset of visual tokens.
In particular, we present the O-coverage results on the MME benchmark Fig. 2 The Saliency Only method selects dominant tokens solely based on the attention scores from the CLS token. However, consistently exhibits low coverage across different values of 0, even performing worse than the random selection baseline. This observation suggests that although the saliency- method captures dominant information, it tends to overlook a substantial amount of semantic content. In contrast, our method (detailed in 3.3 incorporates saliency scores into coverage-awe aware selection framework, striking a better balance between saliency and semantic coverage. As a result, it achieves significantly higher coverage compared to the Saliency Only method.
3.3 Saliency-Coverage Oriented Token Pruning
In contrast to saliency- pruning methods, our goal is to jointly optimize saliency and coverage in the visual token selection process. This enables the pruning algorithm to not only preserve the most informative tokens but also maximize the semantic coverage of the selected subset. As a result, the retained tokens are both highly informative and semantically diverse, thereby maintaining semantic completeness under a constrained token budget. which is an essential property for comprehensive visual understanding across wide range of multimodal tasks.
In the following, we first define the notion of coverage for selected tokens. Next, we introduce the concept of token-coverage gain, i.e., the additional coverage obtained by including a new token in the selected set 14.. Finally, we incorporate the saliency score into the token-coverage gain formulation to balance both selection criteria. The overview of the proposed method is presented in Fig. 3
Set-coverage for selected tokens. To quantify semantic coverage, measure the similarity between token vectors using cosine similarity. We first define the individual coverage score C'u S) for token E by a set of selected tokens S C V as:
Cu,, S) max sim(u, 8)
#ES
where sim(u, s) is the cosine similarity metric between token u and token s. The overall coverage of the selected subset S is defined as the sum of the maximum similarities between each token in the full set and its most similar token S:
f(S)
((,,S)= Σ max sim(u, (u, WEV
(6)
#ES
NEV
This formulation encourages the selection of tokens that semantically diverse and broadly representative of the input space. Intuitively, it ensures that each token the full set has at least one similar counterpart in the selected subset, thus preserving information while reducing the token count.
Token-coverage Gain. To quantify the contribution of each candidate token V S, we evaluate its marginal gain with respect to the current subset S [4]. The marginal gain is defined as the
5

---

Algorithm SCOPE
Require: A full set of tokens V = (11,... ..., Un C RS, number of retained token K, pairwise similarities Sue sim(u, 2) for all u, v V, attention score A. for each token and scaling factor a.
Ensure:
Selected token subset S V with |S| = K
Start with an empty selected subset 2: Initialize coverage scores: Co < O for all E b Cu tracks the best similarity between u and
1: Initialize S <- 0
any selected token so far
3: for t= to K do
4: for all V do
5: Compute marginal gain: A(u; Enev (Sue; Cu)
cul
> Compute the
additional coverage that token brings if added
6: end for
7:
Select next token: v* E arg maxie (); A
Choose the token that balances
coverage and saliency score
8:
Update selected subset: S Su {u*}
Add the selected token the subset
9:
Update coverage scores: Cu f- max Swv-) Vu E V Update coverage scores using the newly added token
10: end for
11:
return S
increase in total coverage achieved by including v, and can be formally expressed as follows:
((r; f(S (b)) f(S).
Expanding this definition using Eq. (6. we can express the marginal gain as the sum of the new coverage provided by each token that not already fully covered by S:
A(v;S) = E
max sim(u, 8) - Σ (((((())
max sim(u, 8)
#ES
WEV
NEV
(max(C(u, S), sim(u, u)) - C(u, WEV
Σ
(8)
This quantifies how much additional coverage is achieved by selecting token v, taking into account V that are not yet well-represented by the current subset S.
its ability represent other tokens
SCOPE score. While the token-coverage gain considers only the geometric coverage semantic space, it overlooks the intrinsic information carried by individual tokens. To address this limitation, we propose the SCOPE gain, which incorporates token saliency into the coverage gain better preserve visual token information. Specifically, we integrate the visual attention score into the coverage gain function as follows:
(cc, A:: A(v;)) (0:55)44.
(9)
where A denotes the attention score of the visual token v, and a is scaling factor. The token n* with the highest SCOPE gain is selected and added to the subset S:
v¹ E arg max A(v, AV; A); vEVIS
(10)
This process is iteratively repeated until the desired subset size reached. The pscudocode of the proposed pruning method is presented in Algorithm I
Integration into MLLMs. The proposed method applicable
wide range of MLLMs. In this work, we apply it to the widely adopted LLaVA and LLaVA-Next 25 models, following prior studies 7 49
Our method is integrated after the vision encoder maximize information retention post token pruning. This enables the language model to receive more complete visual signals, thereby supporting comprehensive visual understanding without compromising performance. Our method is train-free and significantly accelerates the inference of MILMS with minimal performance degradation. For example, our approach preserves over 96% of the original model's performance while reducing the number of visual tokens by factor of 8 in LLaVA 1.5 7B.

---

Table 1: Performance comparison under different vision token configurations. We evaluate the LLaVA 1.5 7B model, where the default number of visual tokens is 576. The first row for each method reports the raw accuracy across benchmarks, and the second row indicates the performance
relative to the upper bound. denotes the results adapted from
49
Method
GQA
MMB
MME POPE
SQA
Tex
SEED
MM
Avg.
Upper Bound, 576 Tokens (100%)
61.9
64.7
1862
85.9
69.5
58.2
58.6
31.1
Vanilla (CVPR (4)
100%
100%
100%
100%
100%
100%
100%
100%
100%
Retain
192
Tokens [1
7%))
52.7
61.2
1612
64.8
67.3
52.5
57.1
27.7
FastV (202V/24)
89.5%
85.1%
94.6%
86.6%
75.4%
96.8%
90.2%
97 7%
89.7%
57.6
62.5
1721
83.6
69.1
56.1
55.8
31.5
SpanseVLM (16)L-26)
96.5%
93.1%
96.6%
92.4%
97.3%
99.4%
96.4%
95.2%
101.3%
59.3
63.0
1783
85,3
68,9
57.3
56.4
31.7
VisionZip (CVPR'26)
98.0%
95.8%
97.4%
95.7%
99.3%
99.1%
98.5%
96.2%
101.9%
57.1
63.2
1766
82.3
70.2
56.1
54.7
30.5
PDrop (c)
96.2%
92.2%
97.7%
94.8%
95.8%
101.0%
96.4%
93.3%
98.1%
Ours
60.1
63.6
1804
86.4
68.8
57.7
58.7
32,5
99.5% (1 0.5%)
97.1%
98.3%
96.9%
100.6%
99.0%
99.1%
100.2%
104.5%
Retain
128
Tokens (1. 77 7%)
49.6
56.1
1490
59.6
60.2
50.6
55.9
28.1
FastV (500V124)
84.4%
80.1%
86.7%
80.0%
69.4%
86.6%
86.9%
95.4%
90.4%
56.0
60.0
1696
80.5
67.1
54.9
53.4
30.0
Sparse VL. (TcmI
93.3%
90.5%
92.7%
91.1%
93.7%
96.5%
94.3%
91.1%
96.5%
57.6
62.0
1761.7
83.2
68.9
56.8
54.9
32.6
VisionZip (0)PR:26)
96.9%
93.1%
95.8%
94.6%
96.9%
99.1%
97.6%
93.7%
104,8%
56
61.1
1664
82.3
69.9
55.1
53.3
30,8
PDrop (cVPR>25)
94.4%
90.5%
94.4%
89.4%
95.8%
100.6%
94.7%
91.0%
99.0%
59.7
62.5
1776
86.1
68.4
57.2
57.8
31.4
Ours
98.1 (L 1.9%)
96.4%
96.6%
95.4%
100.2%
98.4%
98.3%
98.6%
101.0%
Retain 64
Tokens (1 88. 9%)
46.1
48.0
1256
48
$1.1
47.8
51.9
25.8
FastV (240V124)
74.9%
74.5%
74.2%
67.5%
55.9%
73.5%
82.1 %
88.6%
83.0%
52.7
56.2
1505
75.1
62.2
51.8
51.1
23.3
Sparse VLM (IOML '25)
85.1%
85.1%
86.9%
80.8%
87.4%
89.5%
89.0%
87.2%
74.9%
55.1
60.1
1690
77.0
69.0
55.5
52.2
31.7
VisionZip (CVPR
93.5%
89.0%
92.9%
90.8%
89.6%
99.3%
95.4%
89.1%
101.9%
41.9
33.3
1092
55.9
69.2
45.9
40.0
30.7
PDrop (2)P
73.5%
67.7%
51.5%
58.6%
65.1%
99.6%
78.9%
68.3%
98.7%
58.3
61.7
1698
83,9
68.6
56.6
56.3
30.4
Ours
96.0%
(L 4.0%)
94.2%
95.4%
91.2%
97.7%
98.7%
97.3%
96.1%
97.7%
Experiment
4.1 Experiments Setup
we evaluate the effectiveness of the proposed method using set of widely adopted multimodal benchmarks. Specifically, these include GQA 13 MMBench POPE [22], ScienceQA [2)), TextVQA [36), SEEDBench
Evaluation Benchmarks and Baselines. Following prior work 49
and MMVet 45
We also compare against several tfate-of-the-ar baselines, including Fast [7].
Sparse 44). VisionZip 43). 1. and PDrop [4]. For the vidco benchmarks, we evaluate the MLLMs
on the benchmarks TGIF (15). MSVD S MSRVTT [42), and ActivityNet [46). For further details
on evaluation benchmarks and metrics, we refer the reader to the Appendix B
Implementation Details. We integrate the proposed method into LLaVA 1.5 26 and LLaVA- Next (25 for image understanding and Video-LLaVA (23 for video understanding. The pruning module is inserted after the vision encoder. The saliency score is derived from the attention weights of visual tokens with respect to the CLS token at the second-to-last layer (layer -2) of the vision encoder. The scaling factor a is set to 1.0 by default. Our inplementation is based on the Imms-evals [8 package. We conduct the experiments on 4 X A100 GPUs. The inference batch size is set to for all the evaluation results.
7

---

Table 2: Performance comparison under different vision token configurations. The evaluated model is LLaVA-Next 7B. The vanilla number of vision tokens 2,880. The first line of each method is the raw accuracy of benchmarks, and the second line is the proportion relative to the upper bound.
Method
GQA
MMB
MME
SQA
TextVQa
MMMU
Avg.
Upper Bound, 2880 Tokens (100%)
64.2
67.9
1842
70.2
61.3
35.1
Vanilla (CVPR)24)
100%
100%
100%
100%
100%
100%
100%
Retain 640
Tokens (1. 77 77.8%)
60.3
65.7
1772
67.7
57.8
34.6
SparseVI.M (1CHL)25)
96.0%
93.9%
96.8%
96.2%
96.4%
94.3%
98.6%
61.3
66.3
1787
68.1
60.2
34.7
VisionZip (cVPa/25)
97.4%
95.5%
97.6%
97.0%
97.0%
98.2%
98.9%
61.9
66.2
1842
67.8
60.1
36.9
Ours
98.9% (4 1.1%)
96.4%
97.5%
100.0%
96,6%
98.0%
105.1%
Retain 320 Tokens (L 88 (8.9%)
57.7
64.3
1694
67.3
55.9
34.4
Sparse VLM (ICHL
93.6%
89.9%
94.7%
92.0%
95.9%
91.2%
98.0%
59.3
63.1
1702
67.3
58.9
35.3
VisionZip (cVPR
95.0%
92.4%
92.9%
92.4%
95.9%
96.1%
100.6%
61.0
65.9
1789
67.7
58.4
35.6
Ours
97.1% (4 2.9%)
95.0%
97.1%
97.1%
96.4%
95.3%
101.4%
Retain 160 Tokens (1 94.4%
51.2
63.1
1542
67.5
46.4
32.8
SparscVLM (s)
86.9%
79.8%
92.9%
83.7%
96.2%
75.7%
93.4%
55.5
60.1
1630
68.3
56.2
36.1
VisionZip (2)
92.5%
86.4%
88.5%
88.5%
97.3%
91.7%
102.8%
60.0
64.3
1700
67.4
56.8
35.6
Ours
95.1% (1 4.9%)
93.5%
94.7%
92.3%
96.0%
92.7%
101,4%
4.2
Main Results
Results on LLaVA 1.5. LLaVA 1.5 one of the most representative MLLMs. We therefore apply the proposed pruning method to LLaVA 1.5 and evaluate its performance on a variety of image understanding tasks, following prior works metrics used across different benchmarks, which result in inconsistent numerical scales, we report performance as percentage of the original model's accuracy. We show the results of LLaVA 1.5 7B in [able In particular, we follow previous work 49 43 and evaluate the performance under three visual token pruning budgets (i.e., 192, 128, and 64) to evaluate the effectiveness of the proposed method. The vanilla model (i.e., LLaVA 1.5 7B with full visual tokens) serves as the upper bound (100%), representing the performance ceiling of any visual token pruning approach. Our method consistently outperforms existing approaches across all token configurations, particularly under aggressive compression settings. As shown in Tablcq when retaining only 192 tokens (a 66.7% reduction from the baseline), method achieves average accuracy of 99.5% relative to the upper bound. This surpasses state-of-the-aat baselines including FastV [7] (+6.0%), Sparse VL. (+3.0%), and VisionZip 43) (+1.5%) Under extreme compression (e.g.. 64 tokens, 88.9% reduction), our method maintains 96.0% of the original performance, significantly outperforming baselines such
43 Due the diverse evaluation
as VisionZip 43 (93.5%) and Sparse 49 (85.1%)
Surprisingly, our method preserves or even surpasses the upper bound in performance on several benchmarks. For instance, we observe relative accuracies of 100.2% and 104.5% on POPE [22] and MMVet [45). respectively, when using 192 tokens. These results suggest that visual tokens in MLLMs contain redundancy, and our method not only reduces this redundancy but also improves performance by eliminating interference from redundant information. We further evaluate our method on the larger LLaVA 1.5 13B model to validate its generalization capability in Appendix C.I
Results on VANNex. Compared to LLaVA 1.5, LLaVA-Next is more advanced MLLM that supports high-resolution image processing, thereby significantly improving vision-language understanding. VA-Nex- partitions an input image into multiple regions based on its original size. Usually, the image is divided into 4 sub-images. Both the original and partitioned images are then encoded into visual tokens, resulting total of 2,880 tokens (576x5). While effective in capturing fine-grained visual details, this strategy substantially increases the number of visual
8

---

GQA
MME
POPE
5QA
69-
Performance ancee
60-
68
50-
67
1200
Ours
Ours VislonZip
Ours VisionZip
Ours
VisionZip
65
VisienZip
1000
20
32
32
16
32
16
48
32
16
Number of lakens
Number of Tokens
Number of Tokens
Number of Tokens
Figure 4: The performance comparison under extreme token number.
tokens and reduces inference efficiency. Therefore, objective is minimize the number of visual tokens while maintaining performance as much as possible. To evaluate the proposed method on VA-Nex,,, (i.e., 640, 320, and 160). The results are presented in Table As shown, our method consistently outperforms state-of-the-an approaches under all configurations. Specifically, when retaining only 640 tokens, our approach achieves an average accuracy of 98.9% relative to the upper bound. Under extreme compression (e.g., 160 tokens, 94.4% reduction), our method maintains 95.1 15.1% performance, significantly surpassing baselines such as SparseVLM (86.9% and VisionZip (92.5%). These results further validate the effectiveness of the proposed method across different MLLM architectures. We also evaluate our method on the LLaVA- VA-Next 13B model in Appendix C.1
follow previous works 49 43 and adopt three visual token budget settings
Results on Video benchmarks. We further evaluate the effectiveness of the proposed method, and imple- ment our SCOPE based on Video- LLaVA following VisionZIP 44 The results are reported in Table As shown, our method achieves the best performance among all compared methods. Surprisingly, even with ag- gressive pruning, our method almost fully preserves the original perfor- mance. This demonstrates the strong effectiveness of our method on video- language tasks. These findings also suggest that video benchmarks contain substantial redundancy, and token pruning has great potential for accelerating video LLMs without sacrificing performance.
Table 3: Performance comparison on Video-LLaVA. The original Video-LLaVAs video token number is 2048, while method only retains the 136 tokens.
Method
TGIF
MSVD MSRVTT ActivityNet
Avg 100.0%
LLaVA
47.1
69.8
56.7
43.1
3
23.1
38.0
19,3
30.6
FastV
52.1%
49.0%
54.4%
34.0%
71.0%
44.7
68.2
31.0
42.6
SparseVLMM
86.5%
94.9%
97.7%
54.7%
98.8%
42.4
63.5
52.1
43.0
VisionZip
93.2%
90.0%
91.0%
91.9%
99.8%
47.1
69.2
55.9
44.9
Ours
100.5 %
100.0%
99.1
98.6%
104.2%
4.3 Analysis
Results under Extreme Token Reduction. Our method demonstrates superior performance stability as the number of visual tokens is progressively reduced. As shown in Fig. g even when the token count reduced few as our approach consistently outperforms VisionZip
by increasingly larger
margins. This highlights the strong capability of our framework to retain critical visual information under extreme compression. In contrast, VisionZip exhibits a sharp performance drop in low-token regimes, further validating the effectiveness of our token selection strategy and underscoring the potential of our method for aggressive visual token pruning.
Ablation Studies. As shown in Table 4
Table 4: Ablation studies of the proposed method.
our method, which jointly considers token
GQA
MMB
MME
POPE
ToxVQA
saliency and coverage, consistently outper- forms its ablated variants (saliency-only and coverage-only across all benchmarks. Both ablated models still perform better than the random baseline, indicating the individual effectiveness of each component. For instance, the coverage-only variant achieves moderate performance. However, our full method further improves these results, demonstrating that combining saliency and coverage provides complementary benefits. Explicit modeling of both saliency and coverage leads to superior performance compared to using either criterion alone selecting tokens randomly.
Random
55.5
54.0
1556
75.2
48.4
Saliency only
55.0
60.8
1665
76.8
55.4
Coverage-only
58.1
60.8
1687
82.1
56.3
Ours 58.3
61.7
1698
83.9
56.6

---

Image
Attention
Saliency-only
Coverage-only
SCOPE
Figure 5: Visualization of token pruning among different pruning strategies.
Efficiency Analysis. Table 5 compares the efficiency of our method with that of base- line pruning approach (PDrop) on LLaVA- NeXT 7B. Despite reducing the number of visual tokens from 2,880 to 160, com- pression ratio exceeding 18x, our method maintains strong performance on the POPE metric (81.3% vs. 86.4%), demonstrating that our token selection strategy effectively preserves semantic completeness. In contrast, PDrop exhibits a substantial performance drop (53.2%), likely due to its reliance on saliency- pruning, which may overlook less attended yet semantically important regions. Although our method incurs slightly higher latency than PDrop. still achieves 3.2x speedup over the full-token baseline. This oriented pruning strategy not only effective in preserving
Table 5: Efficiency analysis of our method on LLaVA-
NeXT 7B. The experiments are conducted on a system equipped with 4 Ax000. denotes the reduction ratio.
Token Namber
POPE
Latency (6)
Vanilla
2880
86.4
601.9
PDrop
184.0
3.3
Ours
160
81.3
188.8
3.2x
41
indicates that saliency-coveren performance but also computationally efficient practice.
Token Pruning Visualization. In Fig. 5 we provide a visualization of token pruning to illustrate the difference of selected tokens among different strategies. Saliency-only mainly concentrates on the most salient patch such as the cat and banana in 1st row, demonstrating object-level focus by pruning background. Coverage-only selects tokens that are spread across the image, preserving global context but potentially missing important object details. Our SCOPE maintains the high token density on salient patches (e.g., cat and banana in 1st row), while a sparse set of tokens is strategically kept for the background. This captures critical object features without discarding essential scene context.
5 Conclusion
While existing approaches predominantly rely on attention saliency to prune redundant tokens, they often neglect semantic coverage, leading to incomplete visual representations, To overcome this limitation, we propose SCOPE, a novel visual token pruning framework that jointly models both token saliency and coverage. Our method introduces set-coverage score based on pairwise token similarities and calculates a token-coverage gain for each candidate token. By incorporating saliency scores into this gain, we derive the SCOPE score, which guides an iterative token selection process. Empirical evaluations on LLaVA 1.5 and LLaVA-Next across multiple rision-languse benchmarks show that SCOPE consistently outperforms state-of-the-an pruning approaches, achieving strong performance even under aggressive token reduction. We believe that our approach offers a principled and effective framework for evaluating the value of visual tokens in MI..Mss.
10

---

Acknowledgement
This work was supported by National Natural Science Foundation of China (No.62476051 1, No.62176047, (0.82221003) and Sichuan Natural Science Foundation (No. 20022024NFFD0001) This research is supported in part by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: JISG-NMLP-2024-0033 in part by S* Career Development Fund (CDF) under C243512011, 1. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko A Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arX1w:2333308877, 2023.
[2] Saced Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity-bate based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9392-9401, 2025.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv:2309. /6609, 2023.
[4] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: VL: A frontier large Aision-language model with versatile abilities, arXiv:2308 12966, 2023.
[5] David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 900-200, 2011.
[6] Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Y Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision- lisguaelanguage pre-training, Machine Intelligence Research, 20(1):38-56, 2023.
[7] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang, An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024.
[8] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui Hc, Jiaqi Wang. Feng Zhao and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv:2311. 2023.
[9] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su. Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu. Bin Li, Ping Luo. Tong Lu, Qiao, and Jifeng Dai. VL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv:2312. 142388 2023.
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://viccna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[11] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1XXi::2010011929 2020.
[12] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306. 13394, 2023.
[13] Drew A Hudson and Christopher D Manning, GQA: A new dataset for world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[14] Rishabh Iyer. Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory, pages 722-754, PMLR, 2021.
[15] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward apatio-temport reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758 2766, 2017.

---

[16] Jacob Devlin Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota, 2019.
[17] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong. Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision- Maaion-lnnguge-acttioo model. arXiv preprint arXiv:2406.094 09246, 2024.
[18] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint a7X1v:2307.16125, 2023.
[19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B:ip-2: Bootstrapping language- -ingge pre-training with frozen image encoders and large language models. In International conference machine learning. 2023.
[20] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image worth 2 tokens in large language models, In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[21] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403. 18814, 2024.
[22] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vasion-language models. urXiv:2305./003: 2023.
[23] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv:2311.10122, 2023.
[24] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023.
[25] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 2024.
[27] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal modal model an all-around player? arXiv:2307.06288 2023.
[28] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-L Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67 67(1)222010,, 2024.
[29] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
[30] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards
detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL. 2024), 2024.
[31] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A henchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[32] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209,
2021.
[33] Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. Mobile edge intelligence for large language models: A contemporary survey, arXiv preprint arXiv:2407. 18921, 2024.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
12

---

[35] Yuzhang Shang. Mu Cai, Bingxin Xu. Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 2024.
[36] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA models that read. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anee Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302. 13971, 2023.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 11199////// /2/91, 2024.
[40] Yuxin Wen, Qingging Cao, Qichen Fu, Sachin Mehta, and Mahyar Najibi. Efficient vision-languae language models by summarizing visual tokens into compact registers, arXiv preprint arXiv:2410. 2024.
[41] Long Xing, Qidong Huang. Xiaoyi Dong. Jiajie Lu, Pan Zhang. Yuhang Zang. Yuhang Cao. Conghui He. Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large -anguuge models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17244 2024.
[42] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[43] Senqiao Yang. Yukang Chen, Zhuotao Tian, Chengyao Wang. Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04465 2024.
[44] Yuan Yao. Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li. Weilin Zhao, Zhihui Hc, et al. Minicpm-v: A gpt-4v level mllm on your phone, arXiv preprint ar 2rXiv:2408.018 2024.
[45] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR, 2024.
[46] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web vidcos via question answering. In AAAI, pages 9127-9134, 2019.
[47] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin. Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
[48] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chumyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint 240. 2024.
[49] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient languae model inference. arXiv preprint arXiv:2410.04117. 2024.
[50] Qiang Zhou, Chaohui Yu, Shaofeng Zhang, Sitong Wu, Zhibing Wang, and Fan Wang. Regionblip: A unified multi-modal pre-training framework for holistic and regional comprehension. arXiv preprint arXiv:2308.00 022999 2023.
[51] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vasion-language understanding with advanced large language models, arXiv:2304. 10592, 2023.
13

---

A Overview
This appendix provides detailed information on the experimental benchmarks, additional qualitative results, and visualizations that support the main claims of the paper. In Sectior we present compre- hensive descriptions of the benchmark datasets. Section C Cincludes supplementary experiments, such as results on larger models (LLaVA 1.5 13B and LLaVA-Ners 13B), as well as hyperparameter analysis. In Section DJ we provide additional visualization studies to further illustrate the behavior of our method. Finally, we discuss the broader impact and limitations of our work.
B Benchmarks
We conduct the experiments on several widely used visual understanding benchmarks. In the following, will give detailed description of these benchmarks.
The GQA benchmark consists of three components: scene graphs, questions, and images. The image component includes raw images, their spatial features, and the features of all objects within the images. The questions GQA are crafted to evaluate visual scene understand- ing and reasoning about various aspects of an image. Our method is evaluated on the subset of daaidee_bblanee__iiee instructions". which includes 12,578 samples.
GQA. [3].
MMBench. [27]. MMBench is a comprehensive benchmark designed to evaluate the multi-modi capabilities of large language models, covering wide range of tasks including visual question answering, image captioning, cross-modal retrieval, and creative generation. It provides a fine- grained assessment from perception to cognition, containing approximately 3,000 multiple-choic questions aggregated from diverse sources. The benchmark aims to measure whether model is true aall-around player" in multi-modal understanding and reasoning.
The MME benchmark is a comprehensive evaluation suite carefully crafted to assess multiple facets of model performance. It comprises 14 distinct subtasks targeting both perceptual and cognitive capabilities of models. By employing manually curated instruction-answer pairs and suceinct instruction formats, MME effectively reduces the risks of data leakage and ensures a fairer assessment of model abilities. We evaluate the performance on the dev split including 4,377 samples. The evaluation metric is the accuracy of the model's ' answer.
MME. 12
POPE. [22] The POPE benchmark focuses on assessing object hallucination in models by presenting them with set of targeted yes/no questions about object existence within images. This approach reframes the evaluation of hallucination, emphasizing the model's ability to correctly identify whether certain objects are present. To quantitatively analyze performance across three distinet sampling methods, the benchmark utilizes metrics such as accuracy, recall, precision, and Fl score, offering a robust measure of the model's susceptibility to hallucination. We evaluate the model's performance on the test split, including 9,000 samples. The evaluation metric is the Fl score.
ScienceQA (SQA). [29] Encompassing a wide array of fields such as natural sciences, linguistics, and social sciences, SQA structures its questions through a hierarchical framework consisting of 26 topics, 127 categories, and 379 distinct skills. This benchmark is designed to rigorously test model's proficiency in multimodal comprehension, complex reasoning across multiple steps, and interpretability. By organizing questions first by subject area, then by specific category, and finally by the required skill, SQA ensures a thorough and nuanced assessment of scientific understanding across diverse domains. This layered organization enables detailed evaluation of model's ability to handle a broad spectrum of scientific queries. The evaluation metric is the accuracy.
TextVQA is designed to assess model's capability interpret and reason over textual content embedded in images. This benchmark challenges models with visual question answering tasks that require both comprehension of image context and accurate reading of the text present within the images. To perform well, models must effectively integrate visual and textual cues, demonstrating robust understanding and reasoning skills related to text in complex visual environments. We evaluate the model's 'ss performance on the test split, including 5,000 samples. The evaluation metric is exact match (EM).
VQA. 36
SEEDBench [X] SEEDBench features a collection 19,000 multiple-choice questions curated by human annotators. Covering 12 different evaluation dimensions, this benchmark examines models'
14

---

Table 6: Performance comparison under different vision token configurations. The evaluated model LL 1.5 13B, where the default number visual tokens 576. The first row for each method reports the raw accuracy across benchmarks, and the second row indicates the performance relative to the upper bound.
Method
GQA
MMB
MME
POPE
SQA
TexiVQA
SEED-1
MMVet
Avg.
Upper Bound, 576 Tokens (100%)
63.2
67.7
1818
85.9
72.8
61.3
66.9
35.3
Vanilla (4V
100%
100%
100%
100%
100%
100%
100%
100%
100%
Retain 192
Tokens (1 66.7%)
59.1
66.9
1754
85.1
73.5
59.5
65.2
37.5
VisionZip (6))R
98.7%
93.5%
98.8%
96.5%
99.1% %
101.0%
97.1%
97.5%
106.20%
59.7
67.6
1775
86.7
73.8
60
65.5
39.4
Ours
100.2%
94.5%
99.9%
97.6%
100.9%
101.4%
97.9%
97.9%
111.6%
Retain 128
Tokens (1 77.8%)
57.9
66.7
1743
85.2
74
58.7
63.8
37.5
VisionZip (cVPR>25)
97.0%
91.6%
98.5%
95.9%
99.2%
101.6%
95.8%
95.4%
106.2%
59.3
67.2
1735
85.9
73.9
58.7
64.8
37.7
Ours
98.7%
93.8%
99.3%
95.4%
100.0%
101.5%
95.8%
96.9%
106.8%
Retain 64 Tokens (L 88.9%)
56.2
64.9
1676
76.0
74.4
57.4
60.4
33.9
VisionZip (cVPR>25)
93.7%
88. 88.9%
95.9%
92.2%
88.5%
102.2%
93.3%
90.3%
96.0%
58.7
65.5
1762
83.0
73.2
58.3
63.6
35.7
Ours
96.9%
92.9%
96.8%
96.9%
96.6%
100.5%
95.1%
95.1% %
101.1%
capabilities in identifying patterns within both images and videos, taking into account spatial as well as temporal characteristics. The evaluation metric the accuracy.
MM [45] The MMVet benchmark is constructed with the understanding that tackling complex tasks typically requires generalist model to effectively combine multiple fundamental vision- language skills. MMVet identifies six essential vision-language capabilities and systematically evaluates sixteen specific combinations arising from these core abilities, thereby assessing the model's proficiency in integrating diverse vision-lant languae functions. We evaluate the model's performance on the test split, including 218 samples. The score is evaluated by the GPT model.
C Additional Experiments
In the main paper, present experiments on LLaVA 1.5 7B and LLaVA-Next 7B. To further demonstrate the generalizability of our method across model scales, we provide additional results on LLaVA 1.5 13B and LLaVA-Next 13B. We also provide the results more MLLMs such as Qwen2-VL VL and more OCR-related benchmarks.
C.1 Results on LLaVA 1.5 13B
As shown in Table 6 our method consistently outperforms VisionZip 43 across all token budgets. With 192 tokens, our approach achieves 100.2% the upper bound's average performance, slightly higher than VisionZip [3] 7%).. The advantage becomes more evident as the token count decreases: at 64 tokens, our method retains 96.9% performance, compared to VisionZip's 93.7%. Notably, on benchmarks like MMVet [45] and POPE [22), our method even surpasses the original model's performance. These results demonstrate that our joint saliency-coverage strategy better preserves essential information under aggressive token pruning.
C.2
Results on LLaVA-Next 13B
present the results on V.eeeet 13B in Table We can observe that our method consistently outperforms VisionZip 4 under all token budgets. For example, with 640 tokens, approach
15

---

Table 7: Performance comparison under different vision token configurations. The evaluated model LLaVA-Nen 13B. The vanilla number of vision tokens 880. The first line of each method is the raw accuracy on the benchmarks, and the second line is the proportion relative to the upper bound.
Method
GQA MMB
MME
POPE
SQA
TextVQA
MMMU
SEED-1
Avg.
Upper Bound, 2880
Tokens (100%)
65,4
70.0
1901
86.2
73.5
64.3
36.2
71.9
Vanilla 13B (CVPR124)
100%
100%
100%
100%
100%
100%
100%
100%
100%
64.2
67.9
1842
86.4
70.2
61.3
35.1
70.2
Vanilla 7B (cVPR>24)
97.2%
98.2%
97.0%
96.9%
100.2%
95.5%
95.3%
97.0%
97.6%
Retain 640
Tokens (1 77.8%)
63.0
68.6
1871
85.7
71.2
62.2
36.4
68.8
VisionZip (c)
97.8%
96.3%
98.0%
98.4%
99.4%
96.9%
96.7%
100.6%
95.7%
63.6
69.3
1897
86.4
72.5
62.4
36.6
69.9
Ours
98.8%
97.2%
99.0%
99.8%
100.2%
98.6%
97.0%
101.1%
97.2%
Retain 320
Tokens () 88.9%)
60.7
67.2
1805
82.0
70.3
60.9
35.6
65.2
VisionZip (2))
94.8%
92.8%
96.0%
95.0%
95.1 1%
95.6%
94.7%
98.3%
90.7%
63.0
67.7
1830
85.1
71.7
60.8
36.3
67.9
Ours
96.9%
96.3%
96.7%
96.3%
98.7%
97.6%
94.6%
100.3%
94.4%
Retain 160
Tokens (4 94.4%)
57.8
64.9
1739
76.6
69.3
58.4
37.0
61.1
VisionZip (2V)R>26)
91.7%
88.4%
92.7%
91.5%
88.9%
94.3%
90.8%
102.2%
85.0%
61.4
66.9
1777
82.8
72.0
59.3
36.2
66.1
Ours
95.1%
93.9%
95.6%
93.5%
96.1%
98.0%
92.2%
100.0%
91.9%
achieves 98.8% of the upper bound's s's average performance, compared to VisionZip's 97.8%. As the token count decreases 160, our method still retains 95.1% performance, while VisionZip drops to 91.7%. These results further confirm the superior robustness of our method under aggressive token pruning.
C.3 Results on Qwen2-VL
To further evaluate the generalization of the proposed SCOPE, we have also evaluated our method on the Qwen2-VI. (39) model. The results summarized in Table As shown, our method achieves 94.65 4.6% of the full-model performance when retaining only 25% of the tokens. Furthermore, our method significantly outperforms prior approaches such DivPrune 22, with 3.7% improvement in average score under the 10.0% token ratio.
Table Results on Qwen2-VL. The token ratio means the ratio of retained tokens.
Method
Token Ratio
GQA
MMB
MME
POPE
Avg.
Qwen2-VL 7B
100%
61.9
77.4
2286
88.4
100%
DivPrune
25%
59.4
72
2043
85.9
93.90%
Ours
25%
59.8
72.5
2065
86.5
94.6%(+0.77%)
DivPrune
10%
54.3
63.7
1874
80.8
85.90%
Ours
10%
56.6
66.8
1953
84.3
89.6%.(+..7%
C.4 Results on more OCR Benchmarks.
In the main paper, we have already evaluated our method on several OCR-related benchmarks, such as MME [12 and Vet [45) To further demonstrate the effectiveness SCOPE, we conducted
additional experiments more OCR-specific benchmarks including DocVQA [32], ChartQA [3]
16

---

The results are shown in Table 9 As illustrated, our method consistently preserves performance and outperforms VisionZip across different token counts. This further supports the robustness of our approach for OCR tasks.
and OCRBench (28
Table 9: Results on more OCR Benchmarks. The model is LLaVA 1.57B.
Method Vanilla
#Token
DocVQA
ChariQA
OCRBench
Avg.
576
28.0
18.2
31.3
100%
VisionZip
192
26.0
31.1
95.8%
Ours
192
26.5
17.4
31.2
96.6% (+0.8%)
VisionZip
128
25.1
17.1
30.0
93.1%
Ours
128
25.9
17.3
30.7
95. 95.2%
(+2. (+2.1%)
VisionZip
64
21.1
16.0
28.2
84.4%
Ours
64
23.2
16.7
29.5
89.6%
C.5 Hyper-parameter Analysis
The hyperparameter controls the scaling of the attention scores, thereby influencing token selection in our method. As illustrated Fig. 6 the optimal performance is typically achieved when 1.0, suggesting that this setting effectively balances saliency and coverage across most benchmarks.
GQA
MMB
SQA
TextVQA
61
64
69
88
57
68
67
00
65
56
54
Figure 6: The hyperparameter analysis LLaVA 1.5 7B with 64 visual tokens.
D Visualization Results
We present additional results selected token visualization in Fig. 7 The saliency-based method selects tokens solely based on attention scores, which may overlook semantically important tokens that contribute the overall completeness of the visual representation. In contrast, our saliency- coverage oriented approach jointly considers both visual saliency and semantic coverage. As result, the selected tokens span a broader region in the embedding space.
In Fig. [ we further visualize the attention distribution selected tokens. Our method preserves the majority of high-attention tokens, demonstrating its ability to retain both informative and representa- tive visual content.
E Broader Impact
Our proposed method aims to improve both the efficiency and effectiveness multimodal large language models (MLLMs) by reducing the number of visual tokens while preserving semantic completeness. This advancement has the potential to significantly reduce the computational cost and memory footprint of MLLMs, thereby enhancing their feasibility for deployment in resource- constrained environments such as edge devices, mobile platforms, and real-time applications. By enabling more efficient inference, approach can facilitate the broader adoption of vision-languaee models across various domains, including education, healthcare, and assistive technologies.
However, as with any technology that enhances the scalability and accessibility of Al systems, there are potential societal risks. For example, more efficient MILMs could be misused to generate or disseminate misinformation, enable invasive surveillance, support other malicious activities,
17

---

Saliency- -aaency-based
Ours
Saliency baaey
Ours
Figure The selected token comparison between the saliency-based method and our saliency- coverage oriented method. The total visual token number is 576, and the selected token number is 64.
particularly when deployed at scale. It therefore essential consider these ethical implications and implement appropriate safeguards when deploying such models in practice.
F
Limitations
While SCOPE demonstrates strong performance and efficiency gains across multiple benchmarks and model architectures, several limitations remain. (1) Despite our efforts to balance saliency and coverage, aggressive token pruning may still result in the loss fine- aene-grained or rare semantic information, potentially affecting tasks that require detailed visual understanding. (2) Our experiments are primarily based on widely used vision-language benchmarks and two representative MLLMs, LLaVA 1.5 and LLaVA-Next. Therefore, the generalizability of SCOPE to other tasks model architectures has yet to be fully validated.
18

---

based
Icta acarniie: 69%
Nossedected Theans Selected Tahero
bed atterson $1.09)
seccoa Bears Morrr
0.10
0.00
0.00
Talion Rcortso icorted by atte ntton!
Taken Ra (sorted by sttont on)
154 tokene)
(04 totans:
Thta ananntee: 46 31%
Not selette Tokens
Tetal attention 42.77%
Non selecee Biers Moro
0.10
0.00
0.04
0.02
0.03
400
0.00
Thken Rank tuorted by atte itton
Toten Pankc (sorted by attent on)
Selanssydonee Issend
0.22
Selected Itkens 1841
Scectod RAces
0.30
0.01
0.00
Zetentinn
0:04
0.00
0.04
0.02
0.02
0.43
100
200
300
500
0.00
310
200
30C
405
500
tokens:
Ronselected Tocass Selected Tobans (54)
0.12
Se aeted Tainne (64)
0.30
0.02
0.00
0.06
0.92
0:02
0.00
185
Figure 8: Attention distribution visualization for selected token. The total visual token number is 576, and the selected token number is 64. Our method retained most the high attention tokens and some low attention tokens maximize the coverage.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We illustrate the contributions and scope of this paper in the abstract and
introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided the implemented details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer:
[No]
Justification:
The data is open access on the internet. We will release the code once this work is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//guu guwdes/ddee/// wwt.:Suwwisssispoooooooooooooooi
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the details in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due the complexity MLLMs, we do not report error bars following previ- ous works. The effectiveness of the proposed method is validated on various benchmarks and multiple MILMs across different scales of model size.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the GPU information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform to the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discussed the broader impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We use the public datasets for academic purposes only.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We provided the details of the dataset and implementation details of the proposed method.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
25

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
26


Reviews:
[{"review_id": "SGZlv9yt3p", "reviewer": "Reviewer_h3Fi", "summary": "This paper tackles the computational overhead in Multimodal Large Language Models (MLLMs) caused by excessive visual tokens. The authors argue that existing saliency-based pruning methods compromise semantic \"coverage.\" To address this, they propose SCOPE (Saliency-Coverage Oriented token Pruning), a novel, training-free framework that jointly optimizes for both saliency and coverage. SCOPE uses a greedy algorithm to iteratively select tokens that maximize a score combining intrinsic saliency and marginal coverage gain. Extensive experiments on LLaVA-1.5 and LLaVA-Next show that SCOPE consistently outperforms prior methods, especially under high compression rates.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe greedy algorithm, based on maximizing a marginal gain (Eq. 7 & 8), is a principled approach reminiscent of submodular optimization, making it well-suited for this diverse subset selection problem.\n\nThe SCOPE score (Eq. 9) provides a simple yet effective mechanism to balance the competing objectives of saliency and coverage.\n\nBeing training-free, the method is highly practical and can be easily integrated into existing MLLM inference pipelines.\n\n**Weaknesses**\n\nWhile the ultimate application of the paper is in MLLMs, the core technical contribution, aka the SCOPE algorithm, is a module that operates purely on the vision side. It completes all pruning operations before the visual tokens are fed into the language model. Therefore, confining the experimental comparison solely to other pruning methods designed specifically for MLLMs presents an incomplete evaluation.\n\nA more rigorous evaluation should have benchmarked SCOPE against established token pruning or merging methods designed for ViTs. These methods are the precursors and critical baselines for addressing the same fundamental problem: how to efficiently process a large number of visual tokens.\n\nIf SCOPE cannot demonstrate a clear advantage over these mature vision-side pruning methods in end-to-end MLLM tasks, its novelty and practical value are significantly diminished. In other words, a key question remains unanswered: Do we need a new, MLLM-specific visual pruning method like SCOPE, or could we simply \"plug-and-play\" existing state-of-the-art vision pruning methods into MLLMs and achieve comparable or even better results?", "was_revised": true, "final_justification": "I have reviewed the authors’ response as well as their discussion with the other reviewers. I would like to maintain my recommendation to accept the paper.", "rebuttal": "We sincerely appreciate the reviewer’s valuable and constructive feedback. Below, we address your questions as follows:\n\n> **Q1**: Comparison with methods of vision token compression for ViTs.\n\n**A1**: Thank you for the suggestion. We agree that established ViT-based token pruning methods are important baselines. Following your suggestion, we compare our method with a representative ViT-based token pruning approach, Token Merging (ToMe) [1], which reduces token redundancy while preserving visual feature quality. The results on LLaVA-1.5 7B are presented in Table A1.\n\n**Table A1: Results of SCOPE and ViT token pruning method.**\n|Method|#Token|GQA|MMB|MME|POPE|SQA|TextVQA|SEED|MMVet|Avg.|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-|\n|Vanilla|576|61.9|64.7| 1862 | 85.9| 69.5|58.2|58.6| 31.1| 100% |\n|ToMe | 192 | 54.3 | 60.5 | 1563 | 72.4 | 65.2 | 52.1 | 53.1 | 27.9| 88.9%|\n|**Ours** | 192 | 60.1 | 63.6 | 1804 | 86.4 |  68.8 | 57.7 | 58.7 | 32.5 | 99.5% (**+10.6%**)|\n|ToMe | 128 | 52.4 | 53.3 | 1343 | 62.8 | 59.6 | 49.1 | 50.9 | 27.2 | 81.9%|\n|**Ours** | 128 | 59.7 | 62.5 | 1776 | 86.1 | 68.4 |  57.2 |  57.8 |  31.4 | 98.1% (**+16.2%**) |\n|ToMe | 64 | 48.6| 43.7|1138| 52.5| 50.0|45.3|44.0|24.1| 71.1% |\n|**Ours** |64| 58.3 |  61.7|  1698|  83.9|  68.6|  56.6|  56.3|  30.4| 96.0% (**+24.9%**) |\n\nFrom Table A1, we can see that our SCOPE consistently outperforms ToMe under the same token budget. These results highlight the importance of developing vision-side token pruning strategies specifically tailored for MLLMs, and validate the effectiveness of our proposed method. This trend is also observed in prior efficient MLLM designs such as VisionZip, which adopt post-ViT pruning strategies. \n\nThe key reasons behind the performance differences are:\n- **Different semantic requirements:**\n    - ViT-based pruning methods aim to retain discriminative features for classification tasks, which primarily require category-level information.\n    - MLLMs demand more comprehensive and fine-grained semantic representations to support image understanding and reasoning.\n- **Different pruning stages:**\n    - **ViT-based methods prune tokens at intermediate layers to reduce computation within the ViT backbone**. In ViTs, as layers go deeper, the relationships among tokens evolve. Shallow layers focus primarily on local spatial structures, while deeper layers capture more global and semantic dependencies. Therefore, pruning too early may disrupt the development of critical token interactions before they are fully established.\n    - **Vision-side token pruning for MLLMs, in contrast, is applied after the full forward pass of the ViT**. This allows pruning decisions to be based on the final visual representations, which already encode rich semantic information. Such a design better aligns with the needs of MLLMs, enabling more informed token selection based on comprehensive embeddings.\n\nWe will include this comparison and discussion in the final version to provide a more comprehensive evaluation. \n\nWe thank the reviewer again for the insightful comment. Although existing ViT-based pruning methods are not directly compatible with MLLMs, we believe it is a promising future direction to design ViT-side token merging/pruning that is also aware of multimodal semantic needs.\n\n[1] Token merging: Your vit but faster, ICLR 2023."}, {"review_id": "b4rHQC6vLO", "reviewer": "Reviewer_Toag", "summary": "The paper proposes SCOPE, a visual token pruning method for VLMs that considers both token saliency and semantic coverage to improve visual token pruning efficiency. Unlike prior approaches that rely solely on attention-based saliency, SCOPE introduces a token-coverage-aware selection mechanism based on pairwise token similarities, ensuring that the selected tokens can cover the most important regions. The method is train-free, integrates seamlessly after the vision encoder and is applicable to a wide range of VLMs. It consistently outperforms state-of-the-art baselines on a range of benchmarks, especially under aggressive token pruning settings.", "questions": "1. It lacks a visualization of selected visual tokens for images to help us better understand how the SCOPE score-guided token pruning differs from saliency-guided token pruning. The provided visualization is not an actual image and thus is hard to interpret. For example, you should use a mask to highlight the chosen region of the image.\n2. How does the proposed method compare to randomly chosen tokens in terms of performance? Random selection can also enable evenly pruned tokens, thus potentially providing more complete coverage, though this is a very simple and naive heuristic.\n3. In Figure 2, the coverage for SCOPE is actually lower than for coverage only, but in the ablation study, SCOPE performs better than coverage only. Can you provide some explanation to help me understand what this means?\n4. In Figure 4, when the number of tokens is relatively large (for example, 48), it seems that VisionZip is starting to beat SCOPE. However, in Tables 1 and 2, when retaining 64 / 160 tokens, SCOPE is significantly better than VisionZip. This is inconsistent, as if Figure 4 is true, then VisionZip should be better when the number of retained tokens is 64. How do you explain this?", "limitations": "The authors can provide a breakdown of their method's latency to better understand why the significant token reduction does not translate into substantial latency reduction.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Quality**: The idea behind this work is clear and technically sound, as many vision token pruning works focus on pruning based on the attention map (saliency-oriented), and few focus on a new aspect like token similarity (called token coverage in this paper). Through extensive experiments, the authors show that incorporating token coverage into consideration can greatly improve performance, especially under extreme pruning ratios. From motivation to method to experimental results, this submission is clear and of high quality.\n\n**Clarity**: This paper is well written, with clear motivation, detailed formulas explaining their method, and neat experiment tables.\n\n**Significance**: The results are pretty good, especially for extreme pruning ratios, which can inspire other work to push further for more extreme pruning. However, though the reduction is significant, it does not translate into significant latency reduction (Table 4), which can reduce the impact of this work in real-world production. Also, from Figure 6, it seems like the difference between the proposed method and the saliency-based method is that the proposed method chooses tokens more evenly. Randomly choosing tokens could also have the same effect, but the authors do not provide a comparison with random selection to justify any performance difference.\n\n**Originality**: The idea of using set coverage to guide pruning is novel to me, but the idea of token similarity is not novel and can be found in some token merging work, such as [1].\n\nReference:\n\n[1] Token Merging: Your ViT But Faster", "was_revised": true, "final_justification": "The rebuttal addresses my concerns, but I still think that there is no large difference between coverage-oriented and saliency-oriented token selection. In other words, I think coverage-oriented token selection is just a supplement to saliency-oriented selection. Considering that the authors provides extensive experiments results to study the impact of coverage-oriented token selection and still has some contributions to the community, I would like to maintain my borderline accept.", "rebuttal": "Thank you for your recognition of our work and the constructive feedback. We would like to address your concerns individually.\n\n> Q1: The idea of using set coverage to guide pruning is novel to me, but the idea of token similarity is not novel and can be found in some token merging work, such as [1].\n\n**A1**: Thank you for recognizing the novelty of using set coverage in our pruning framework. We acknowledge that token similarity has been widely adopted in previous token compression approaches, including token merging methods such as [1]. However, our work differs fundamentally in both approach and objective:\n- **From token merging to semantic completeness**: We utilize token similarity not for merging redundant tokens, but to introduce a set coverage perspective that explicitly encourages semantic completeness among selected tokens. This represents a paradigm shift from existing methods.\n- **Individual saliency vs. collective semantic completeness**: Current saliency-based pruning methods focus on selecting individually important tokens, which creates a critical issue: semantic incompleteness. These methods optimize for individual token importance without considering the collective semantic representation.\n- **Joint saliency-Coverage optimization**: Our key contribution is to formulate token selection as a joint optimization over saliency and coverage.\n    - **Saliency**: Ensures token-level importance is preserved.\n    - **Coverage**: Guarantees selected tokens span complete semantic regions.\n\nThis dual optimization results in superior semantic completeness, which proves particularly critical for downstream reasoning tasks in MLLMs where comprehensive understanding is essential. While we share certain tools (e.g., similarity metrics) with prior works, our principal objective, and mathematical formulation represent a distinct contribution that addresses fundamental limitations in existing approaches.\n\n> **Q2**: It lacks a visualization of selected visual tokens for images to help us better understand how the SCOPE score-guided token pruning differs from saliency-guided token pruning. The provided visualization is not an actual image and thus is hard to interpret. For example, you should use a mask to highlight the chosen region of the image.\n\n**A2**: Thank you for your thoughtful suggestion. We agree that visualizing the selected tokens directly on the original image would greatly improve the interpretability of our method. \n- **Key difference in token selection beavior:** The fundamental distinction between saliency-guided pruning and our SCOPE lines in their selection strategies:\n    - **Saliency-based Pruning** focuses only on the most attended tokens (typically the main object or salient featurs), which may lead to semantic redundancy or incompleteness.\n    - **SCOPE** jointly considers both saliency and semantic coverage, aiming to select tokens that are important and semantically complete to form a more comprehensive representation.\n\n**Visualization Commitment**: We have **already prepared visualizations** that highlight the selected regions on the input images using masks. Due to the rebuttal policy, we cannot include these visualizations in this response. We commit to incorporating these detailed visual comparisons in the camera-ready version to better illustrate the behavior of our method.\n\n\n> **Q3**: How does the proposed method compare to randomly chosen tokens in terms of performance? Random selection can also enable evenly pruned tokens, thus potentially providing more complete coverage, though this is a very simple and naive heuristic.\n\n**A3**: Thank you for the valuable suggestion. The comparison between our method and random token selection is already included in the submission (Table 3), and for clarity, we extract the relevant results in Table B1 below:\n\n**Table B1: Performance comparison between Random selection and our SCOPE.**\n| Method | GQA      | MMB      | MME      | POPE     | TextVQA  |\n| :-: |:-:| :-: | :-: | :-: | :-: |\n| Random | 55.5     | 54.0     | 1556     | 75.2     | 48.4     |\n| Ours   | 58.3 | 61.7 | 1698 | 83.9 | 56.6 |\n| Gain | **+2.8**     |\t**+7.7**|\t**+142**|\t**+8.7**\t|**+8.2**|\n\nWhile random pruning may offer broad token coverage due to its uniformity, it fails to retain critical visual semantics. In contrast, our method jointly considers saliency (to select informative tokens) and semantic coverage, leading to stronger semantic preservation and consistently higher accuracy.\n\n\n> **Q4**: In Figure 2, the coverage for SCOPE is actually lower than for coverage only, but in the ablation study, SCOPE performs better than coverage only. Can you provide some explanation to help me understand what this means?\n\n**A4**: Thank you for your question. The key insight is that **maximizing coverage alone does not necessarily lead to optimal model performance**. \n\n- **Coverage-only strategy** indeed promotes semantic coverage by selecting a wide range of tokens, it often includes tokens with low saliency.\n- **Our SCOPE method formulates token selection as a joint optimization of saliency and coverage**. This means that selected tokens are not only semantically complete, but also highly informative. Though this combined objective may result in a slightly lower raw coverage score compared to the coverage-only strategy, the selected tokens better represent the essential semantics of the image, leading to improved model performance.\n\nFor better understanding, we provide an example to illustrate the difference of selected tokens among different strategies. As we cannot upload figures, we will briefly describe the visualization of our token pruning strategies on an image of a cat and a banana. \n1. **Saliency-only**: Selected tokens are mainly concentrated on the cat and banana, demonstrating object-level focus by pruning the background.\n2. **Coverage-only**: Tokens are spread across the image, preserving global context but potentially missing important object details.\n3. **Our SCOPE**: High token density is maintained on the cat and banana, while a sparse set of tokens is strategically kept for the background. This captures critical object features without discarding essential scene context.\n\nWe have **prepared visualizations to support this observation** and will include them in the final version of the paper.\n\n\n> **Q5**: In Figure 4, when the number of tokens is relatively large (for example, 48), it seems that VisionZip is starting to beat SCOPE. However, in Tables 1 and 2, when retaining 64 / 160 tokens, SCOPE is significantly better than VisionZip. This is inconsistent, as if Figure 4 is true, then VisionZip should be better when the number of retained tokens is 64. How do you explain this?\n\n**A5**: Thank you for your careful observation. We have re-checked the results and confirmed their correctness. The apparent inconsistency between Figure 4 and Tables 1 and 2 mainly arises from performance fluctuations that commonly occur when the number of retained tokens is small.\n\nIn low-token regimes (e.g., 32 or 48 tokens), token pruning methods are more sensitive to which specific tokens are preserved, and small differences in selection may lead to noticeable performance variation. For example, as shown in Figure 4 on the SQA dataset, retaining 32 tokens leads to better performance than 48 tokens.\n\nWhile performance may fluctuate at certain token counts, SCOPE generally **achieves better results than VisionZip across a wide range of benchmarks and token budgets**. We will make this clearer in the final version to ensure clarity.\n\n> **Q6**: The authors can provide a breakdown of their method's latency to better understand why the significant token reduction does not translate into substantial latency reduction.\n\n**A6**: Thank you for the helpful suggestion. To better understand the relationship between token reduction and actual latency improvement, we provide a detailed latency breakdown based on LLaVA-NeXT 7B, shown in Table B2.\n\n**Table B2: Efficient Analysis on LLaVA-NeXT 7B. The experiments are conducted on 4\\*A100 on POPE.**\n|   Method  | #Token | Total(s) | Speedup | Pre-LLM(s) | Speedup | LLM Prefilling (s) | Speedup |LLM Decoding| Speedup|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| LLaVA-NeXT | 2880  |  601.9  | 1.0$\\times$ |    58.9    | 1.0$\\times$|  487.3 | 1.0$\\times$ |55.7|1.0$\\times$|\n|SCOPE | 160|202.6| 3$\\times$ | 75.4 | 0.8 $\\times$| 79.1 | 6.2 $\\times$ |47.7|1.2$\\times$|\n\nAs shown above, SCOPE significantly reduces the number of visual tokens (from 2880 to 160), resulting in a 6.2$\\times$ speedup in the LLM prefilling stage, which is the most time-consuming part of the pipeline. However, this reduction does not linearly translate to overall latency reduction, primarily because:\n- The Pre-LLM stage (vision encoding and token pruning) becomes more dominant after pruning, as its computation is less affected by the token count.\n- The text tokens themselves also contribute non-negligible computation, and their processing cost remains constant regardless of visual token pruning."}, {"review_id": "okwhZPDxc4", "reviewer": "Reviewer_shnr", "summary": "This paper introduces SCOPE, a novel visual token pruning method that combines saliency and coverage to improve semantic completeness. By defining token-coverage gain and integrating saliency scores, SCOPE iteratively selects the most relevant tokens. Extensive experiments on LLaVA models show that SCOPE outperforms previous methods in vision-language tasks.", "questions": "Overall, this paper is well done with very few issues, but there are still some problems (see weaknesses). If the author addresses my concerns, I will consider raising the score.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: the core contribution of this paper is clear: jointly modeling both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. The proposed method achieves this goal, and both visual results and experimental findings support this claim.\n \nWeaknesses:\n1. Due to the introduction of an additional module, how much computational and time cost does SCOPE introduce compared to previous methods?\n2. Table 3 shows that the increase in the ablation experiment is limited. Could author analyze the reason for this?", "was_revised": true, "final_justification": "Based on the feedback from other reviewers and the authors' rebuttal, my concerns regarding 'computational and time cost' and 'ablation experiment' have been mainly addressed, and I maintain a positive score.", "rebuttal": "We sincerely appreciate the reviewer’s recognition of the core contribution of our work, i.e., the joint modeling of saliency and coverage to preserve semantic completeness. We are encouraged that both our visualizations and experimental results were found to effectively support this goal. We address your questions as follows:\n\n> **Q1**: Due to the introduction of an additional module, how much computational and time cost does SCOPE introduce compared to previous methods?\n\n**A1**: Thank you for your question. As shown in Table 4 of the main paper and detailed again in Table C1 below, we analyze the efficiency of our method on LLaVA-NeXT 7B. Although SCOPE introduces slightly higher latency than PDrop, it still achieves a 3.0$\\times$ speedup compared to the full-token baseline. Importantly, SCOPE maintains strong performance on the POPE metric (83.0% vs. 86.4%), showing that our token selection strategy preserves semantic completeness.\nIn contrast, PDrop exhibits a significant performance drop (53.2% POPE), likely due to its reliance on text-visual saliency-only pruning, which may overlook tokens that are not highly attended but are semantically crucial.\n\n**Table C1: Efficiency analysis of our method on LLaVA-NeXT 7B. The experiments are conducted on a system equipped with $4\\times$A100. $\\Delta$ denotes the reduction ratio.**\n| Method | Token Number | POPE | Latency (s) | $\\Delta$ |\n| :--- | :---: | :---: | :---: | :---: |\n| Vanilla | 2880 | 86.4 | 601.9 | - |\n| PDrop (CVPR'25) | 160 | 53.2 | 184.0 | $3.3\\times$ |\n| Ours | 160 | 83.0 | 202.6 | $3.0\\times$ |\n\n> **Q2**: Table 3 shows that the increase in the ablation experiment is limited. Could author analyze the reason for this?\n\n**A2**: Thank you for the question. We analyse the ablation experiments as follows:\n1. **SCOPE achieves significant gains compared to the random and saliency-only baselines**, especially on challenging benchmarks such as POPE and TextVQA. This confirms the importance of informed token selection.\n2. **The coverage-only variant achieves very strong performance**, validating our key design motivation: ensuring diverse and complete visual semantics is crucial for multimodal reasoning.\n3. **Saliency provides complementary benefits**. Although its standalone performance is weaker than coverage-only, incorporating it into our full method (SCOPE) consistently leads to further improvements. This suggests that saliency helps refine the focus toward more informative regions that coverage alone might overlook.\n\nIn summary, SCOPE effectively integrates both saliency and coverage. The relatively small gain over coverage-only reflects the strength of the coverage, rather than a limitation of our approach. The consistent improvements across all benchmarks highlight the robustness and synergy of the combined design."}, {"review_id": "opMkfASMTL", "reviewer": "Reviewer_6ZPM", "summary": "This paper proposes a saliency-coverage oriented token pruning method for efficient VLMs, which jointly model both the saliency and coverage to measure the redundancy of visual tokens as SCOPE scores. Experiments on LLaVA-1.5 and LLaVA-Next models demonstrate the validity of the proposed method.", "questions": "1. Why compute the SCOPE score with the product of visual attention score and coverage gain? Have you ever tested adding these two values? Is there an ablation study?\n2. I would like to see more results with advanced architectures (qwen2-vl) or video benchmarks, OCR tasks to see the generalizability of SCOPE for different models and tasks.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to understand.\n2. The proposed method is simple and intuitive for measuring the redundancy of visual tokens for efficient VLMs.\n3. Experimental results demonstrate the effectiveness of the proposed method to some extent.\n\nWeaknesses:\n1. The novelty of the proposed method is limited. The \"saliency\" is just the attention score which has been widely adopted in previous methods. Besides, I think the proposed \"coverage\" measure is quite similar to the diversity, which is also calculated with the cosine similarity, such as Divprune [1]. \n\n2. The proposed method should be evaluated on SOTA models to demonstrate the effectiveness, such as qwen2-vl. Besides, could the proposed method be combined with LLM-based pruning methods, such as FastV ? The proposed method should also be tested on video benchmarks to prove the validity.\n\n3. I am interested in some OCR benchmarks, which demand more advanced token pruning methods for maintaining the performance.\n\n[1] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models. CVPR2025.", "was_revised": true, "final_justification": "The rebuttal address my concerns and I raise the score to Borderline Accept.", "rebuttal": "We sincerely thank the reviewer for their positive feedback on our paper, particularly for recognizing its clarity, the simplicity and intuitiveness of our proposed method. We greatly appreciate these encouraging comments. In the following, we will address each of the reviewer’s concerns and suggestions point-by-point to further clarify and strengthen our work.\n\n> **Q1**: The novelty of the proposed method is limited. The \"saliency\" is just the attention score which has been widely adopted in previous methods. Besides, I think the proposed \"coverage\" measure is quite similar to the diversity, which is also calculated with the cosine similarity, such as Divprune [1].\n\n**A1**: Thank you for your comment. We respectfully clarify that our method is distinctive from existing works, with the following contributions:\n\n1. **Coverage is different from diversity**: While both coverage and diversity involve cosine similarity, they serve fundamentally different purposes. For example, DivPrune [1] focuses on selecting tokens that are mutually dissimilar, aiming to maximize pairwise diversity among selected tokens. In contrast, our coverage-based selection encourages selecting tokens that best represent the entire token set by maximizing set coverage gain. As illustrated in Figure D1, DivPrune would select token C to maximize diversity from already selected tokens, while our coverage strategy selects token B because it better covers the unselected tokens A and C (i.e., provides broader representativeness). This distinction is also supported by empirical results, for example, our coverage-only baseline achieves 56.3% on TextVQA and 60.8% on MMB, outperforming DivPrune’s 54.9 and 59.3%, respectively.\n\n**Figure D1: Illustration of token selection. ● and ○ selected and unselected tokens.**\n```\n  ●   ● \n●       ●    ○ A    ○ B    ○ C\n    ●               ↑      ↑\n                Coverage Diversity  \n```\n\n2. **We are the first to jointly consider saliency and coverage**: While saliency (e.g., attention scores) has been used in previous token pruning methods, these approaches often lead to semantic incompleteness by focusing only on highly attended regions. In contrast, our method, SCOPE, formulates token selection as a joint optimization over both saliency and coverage. This ensures that selected tokens are not only important (salient) but also representative of the full image (coverage), leading to improved semantic completeness and task performance.\n\nWe hope this clarifies the novelty of our approach. We will further improve the explanation and emphasize this distinction in the final version.\n\n\n> **Q2**: The proposed method should be evaluated on SOTA models to demonstrate the effectiveness, such as qwen2-vl. \n\n**A2**: Thank you for the suggestion. Following your advice, we have evaluated our method on the Qwen2-VL model. The results are summarized in Table D1. As shown, our method achieves 94.6% of the full-model performance when retaining only 25% of the tokens. Furthermore, our method significantly outperforms prior approaches such as DivPrune, with a 3.7% improvement in average score under the 10% token ratio.\n\nWe will include these results in the final version to demonstrate the robustness and generality of our approach on strong vision-language models.\n\n**Table D1: Results on Qwen2-VL. The token ratio means the ratio of retained tokens.**\n|Method|Token Ratio|GQA|MMB|MME|POPE|Avg.|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2-VL 7B | 100% | 61.9|\t77.4|\t2286|\t88.4|100%|\n|DivPrune | 25% | 59.4\t|72\t|2043\t|85.9 | 93.9%|\n|**Ours** | 25% | 59.8 |72.5| 2065 | 86.5|94.6% (**+0.7%**)|\n|DivPrune | 10% | 54.3|\t63.7|1874|80.8 |85.9%|\n|**Ours** | 10% | 56.6 | 66.8 | 1953 | 84.3 | 89.6% (**+3.7%**)|\n\n\n> **Q3**: Besides, could the proposed method be combined with LLM-based pruning methods, such as FastV? \n\n**A3**: Thank you for the valuable suggestion. We have explored the possibility of combining our SCOPE method with LLM-based pruning approaches such as FastV. In particular, we first apply SCOPE to prune visual tokens before feeding them into the LLM, and then apply FastV for further pruning within the language model. The results are provided in Table D2. These results indicate that SCOPE can be effectively combined with FastV to further reduce the number of visual tokens while maintaining reasonable performance. However, there remains significant room for improvement, and we consider this a promising direction for future exploration.\n\n**Table D2: The results of SCOPE combined with FastV. We first retain 192 tokens and then prune R ratio tokens at K-th layers of LMM using FastV.**\n|Method | GQA | MMB | MME | TextQA| Avg.|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Vanilla | 61.9\t|64.7\t|1862 |58.2| 100%|\n|SCOPE (192) | 60.1|\t63.6 |1804 |57.7| 97.8% |\n|SCOPE (192)+ FastV (K=3, R=50%) | 57.1|\t61.8| 1712| 56.2| 94.1% |\n\n> **Q4**: The proposed method should also be tested on video benchmarks to prove the validity.\n\n**A4**: Thank you for the suggestion. Following VisionZip, we conducted experiments on the Video-LLaVA. The results are reported in Table D3. As shown, our method achieves the best performance among all compared methods. Surprisingly, even with aggressive pruning, our method almost fully preserves the original performance. This demonstrates the strong effectiveness of our method on video-language tasks. These findings also suggest that video benchmarks contain substantial redundancy, and token pruning has great potential for accelerating video LLMs without sacrificing performance. We will include these results in the final version.\n\n**Table D3: Results on video benchmarks.**\n| Method       |#Token| TGIF | MSVD | MSRVTT | ActivityNet | Avg.     |\n|--------------|:-:|:-:|:-:|:-:|:-:|:-:|\n| Video-LLaVA  |256\\*8| 47.1 | 69.8 | 56.7   | 43.1        | 100.0%  |\n| FastV        |17\\*8| 23.1 | 38.0 | 19.3   | 30.6        | 52.1%   |\n| SparseVLM    |17\\*8| 44.7 | 68.2 | 31.0   | 42.6        | 86.5%   |\n| VisionZip    |17\\*8| 42.4 | 63.5 | 52.1   | 43.0        | 93.2%  |\n| **Ours**         |17\\*8| 47.1 | 69.2 | 55.9   | 44.9        | **100.5%** |\n\n\n> **Q5**: I am interested in some OCR benchmarks, which demand more advanced token pruning methods for maintaining the performance.\n\n**A5**: Thank you for your suggestion. In the main paper, we have already evaluated our method on several OCR-related benchmarks, such as MME and MMVet. To further demonstrate the effectiveness of SCOPE, we conducted additional experiments on more OCR-specific benchmarks. The results are shown in Table D4. As illustrated, our method consistently preserves performance and outperforms VisionZip across different token counts. This further supports the robustness of our approach for OCR tasks. We will include these results in the final version.\n\n**Table D4: More results on OCR benchmarks.**\n|Method|#Token|DocVQA|ChartQA|OCRBench|Avg.|\n|:-:|:-:|:-:|:-:|:-:|:-|\n|Vanilla | 576 | 28.0 | 18.2 | 31.3 |100%|\n|VisionZip| 192 | 26.0 | 17.3 | 31.1 |95.8%|\n|**Ours** | 192| 26.5 | 17.4 | 31.2 |96.6% (**+0.8%**)|\n|VisionZip|128| 25.1 | 17.1 | 30.0 |93.1%|\n|**Ours** | 128| 25.9 | 17.3 | 30.7 |95.2% (**+2.1%**)|\n|VisionZip|64|21.1| 16.0 | 28.2 |84.4%|\n|**Ours** | 64 | 23.2 | 16.7 |29.5 |89.6% (**+5.2%**)|\n\n> **Q6**: Why compute the SCOPE score with the product of visual attention score and coverage gain? Have you ever tested adding these two values? Is there an ablation study?\n\n**A6**: Thank you for the insightful question. Here we compare these two designs:\n- **product** promotes the selection of tokens that are both informative (salient) and non-redundant (bringing new coverage). This multiplicative formulation naturally down-weights tokens that are high in only one of the two dimensions.\n- **additive** could overemphasize either highly salient but redundant tokens or low-saliency tokens with marginal coverage contributions.\n\nWe conducted an ablation study to compare the product and addition strategies. As shown in Table D5, the product-based formulation consistently outperforms the additive one across all benchmarks, demonstrating its effectiveness.\n\n**Table D5: Comparison of different combinations of saliency and coverage.**\n|Method|GQA|MMB|MME|POPE|TextVQA|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Add | 57.7 | 60.1 | 1647 | 83.4 | 54.9 |\n|Product| 58.3 |  61.7|  1698| 83.9|  56.6|\n|Gain |**+0.6**|**+1.6**|**+51**|**+0.5**|**+1.7**|"}]

Author rebuttal:

Meta-review: This paper received overall positive reviews finally. Reviewers acknowledged that it introduces an efficient token pruning approach that is practical and easily integrated into existing MLLM inference pipelines. The rebuttal effectively addressed initial concerns, including incomplete comparisons and computational and time costs, clarifying the paper to an acceptance-worthy level. However, some reviewers still expressed concerns about the limited scope and overall contribution to the community. The AC concurs with the majority and considers this a solid contribution, recommending acceptance as a poster. It is advised that the clarifications and supporting evidence from the rebuttal be incorporated into the camera-ready version. We look forward to seeing this work presented at NeurIPS.
Decision: Accept (poster)