Title: SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual Token Pruning

Efficient MLLM
Abstract: Multimodal Large Language Models (MLLMs) typically process a large number of visual tokens, leading to considerable computational overhead, even though many of these tokens are redundant. Existing visual token pruning methods primarily focus on selecting the most salient tokens based on attention scores, resulting in the semantic incompleteness of the selected tokens. In this paper, we propose a novel visual token pruning strategy, called **S**aliency-**C**overage **O**riented token **P**runing for **E**fficient MLLMs (SCOPE), to jointly model both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. Specifically, we introduce a set-coverage for a given set of selected tokens, computed based on the token relationships. We then define a token-coverage gain for each unselected token, quantifying how much additional coverage would be obtained by including it. By integrating the saliency score into the token-coverage gain, we propose our SCOPE score and iteratively select the token with the highest SCOPE score. We conduct extensive experiments on multiple vision-language understanding benchmarks using the LLaVA-1.5 and LLaVA-Next models. Experimental results demonstrate that our method consistently outperforms prior approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a saliency-coverage oriented token pruning method for efficient VLMs, which jointly model both the saliency and coverage to measure the redundancy of visual tokens as SCOPE scores. Experiments on LLaVA-1.5 and LLaVA-Next models demonstrate the validity of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and easy to understand.\n2. The proposed method is simple and intuitive for measuring the redundancy of visual tokens for efficient VLMs.\n3. Experimental results demonstrate the effectiveness of the proposed method to some extent.\n\nWeaknesses:\n1. The novelty of the proposed method is limited. The \"saliency\" is just the attention score which has been widely adopted in previous methods. Besides, I think the proposed \"coverage\" measure is quite similar to the diversity, which is also calculated with the cosine similarity, such as Divprune [1]. \n\n2. The proposed method should be evaluated on SOTA models to demonstrate the effectiveness, such as qwen2-vl. Besides, could the proposed method be combined with LLM-based pruning methods, such as FastV ? The proposed method should also be tested on video benchmarks to prove the validity.\n\n3. I am interested in some OCR benchmarks, which demand more advanced token pruning methods for maintaining the performance.\n\n[1] DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models. CVPR2025.\n\nQUESTIONS:\n1. Why compute the SCOPE score with the product of visual attention score and coverage gain? Have you ever tested adding these two values? Is there an ablation study?\n2. I would like to see more results with advanced architectures (qwen2-vl) or video benchmarks, OCR tasks to see the generalizability of SCOPE for different models and tasks.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper introduces SCOPE, a novel visual token pruning method that combines saliency and coverage to improve semantic completeness. By defining token-coverage gain and integrating saliency scores, SCOPE iteratively selects the most relevant tokens. Extensive experiments on LLaVA models show that SCOPE outperforms previous methods in vision-language tasks.

STRENGTHS AND WEAKNESSES:
Strengths: the core contribution of this paper is clear: jointly modeling both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. The proposed method achieves this goal, and both visual results and experimental findings support this claim.
 
Weaknesses:
1. Due to the introduction of an additional module, how much computational and time cost does SCOPE introduce compared to previous methods?
2. Table 3 shows that the increase in the ablation experiment is limited. Could author analyze the reason for this?

QUESTIONS:
Overall, this paper is well done with very few issues, but there are still some problems (see weaknesses). If the author addresses my concerns, I will consider raising the score.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SCOPE, a visual token pruning method for VLMs that considers both token saliency and semantic coverage to improve visual token pruning efficiency. Unlike prior approaches that rely solely on attention-based saliency, SCOPE introduces a token-coverage-aware selection mechanism based on pairwise token similarities, ensuring that the selected tokens can cover the most important regions. The method is train-free, integrates seamlessly after the vision encoder and is applicable to a wide range of VLMs. It consistently outperforms state-of-the-art baselines on a range of benchmarks, especially under aggressive token pruning settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**: The idea behind this work is clear and technically sound, as many vision token pruning works focus on pruning based on the attention map (saliency-oriented), and few focus on a new aspect like token similarity (called token coverage in this paper). Through extensive experiments, the authors show that incorporating token coverage into consideration can greatly improve performance, especially under extreme pruning ratios. From motivation to method to experimental results, this submission is clear and of high quality.\n\n**Clarity**: This paper is well written, with clear motivation, detailed formulas explaining their method, and neat experiment tables.\n\n**Significance**: The results are pretty good, especially for extreme pruning ratios, which can inspire other work to push further for more extreme pruning. However, though the reduction is significant, it does not translate into significant latency reduction (Table 4), which can reduce the impact of this work in real-world production. Also, from Figure 6, it seems like the difference between the proposed method and the saliency-based method is that the proposed method chooses tokens more evenly. Randomly choosing tokens could also have the same effect, but the authors do not provide a comparison with random selection to justify any performance difference.\n\n**Originality**: The idea of using set coverage to guide pruning is novel to me, but the idea of token similarity is not novel and can be found in some token merging work, such as [1].\n\nReference:\n\n[1] Token Merging: Your ViT But Faster\n\nQUESTIONS:\n1. It lacks a visualization of selected visual tokens for images to help us better understand how the SCOPE score-guided token pruning differs from saliency-guided token pruning. The provided visualization is not an actual image and thus is hard to interpret. For example, you should use a mask to highlight the chosen region of the image.\n2. How does the proposed method compare to randomly chosen tokens in terms of performance? Random selection can also enable evenly pruned tokens, thus potentially providing more complete coverage, though this is a very simple and naive heuristic.\n3. In Figure 2, the coverage for SCOPE is actually lower than for coverage only, but in the ablation study, SCOPE performs better than coverage only. Can you provide some explanation to help me understand what this means?\n4. In Figure 4, when the number of tokens is relatively large (for example, 48), it seems that VisionZip is starting to beat SCOPE. However, in Tables 1 and 2, when retaining 64 / 160 tokens, SCOPE is significantly better than VisionZip. This is inconsistent, as if Figure 4 is true, then VisionZip should be better when the number of retained tokens is 64. How do you explain this?\n\nLIMITATIONS:\nThe authors can provide a breakdown of their method's latency to better understand why the significant token reduction does not translate into substantial latency reduction.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the computational overhead in Multimodal Large Language Models (MLLMs) caused by excessive visual tokens. The authors argue that existing saliency-based pruning methods compromise semantic \"coverage.\" To address this, they propose SCOPE (Saliency-Coverage Oriented token Pruning), a novel, training-free framework that jointly optimizes for both saliency and coverage. SCOPE uses a greedy algorithm to iteratively select tokens that maximize a score combining intrinsic saliency and marginal coverage gain. Extensive experiments on LLaVA-1.5 and LLaVA-Next show that SCOPE consistently outperforms prior methods, especially under high compression rates.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe greedy algorithm, based on maximizing a marginal gain (Eq. 7 & 8), is a principled approach reminiscent of submodular optimization, making it well-suited for this diverse subset selection problem.\n\nThe SCOPE score (Eq. 9) provides a simple yet effective mechanism to balance the competing objectives of saliency and coverage.\n\nBeing training-free, the method is highly practical and can be easily integrated into existing MLLM inference pipelines.\n\n**Weaknesses**\n\nWhile the ultimate application of the paper is in MLLMs, the core technical contribution, aka the SCOPE algorithm, is a module that operates purely on the vision side. It completes all pruning operations before the visual tokens are fed into the language model. Therefore, confining the experimental comparison solely to other pruning methods designed specifically for MLLMs presents an incomplete evaluation.\n\nA more rigorous evaluation should have benchmarked SCOPE against established token pruning or merging methods designed for ViTs. These methods are the precursors and critical baselines for addressing the same fundamental problem: how to efficiently process a large number of visual tokens.\n\nIf SCOPE cannot demonstrate a clear advantage over these mature vision-side pruning methods in end-to-end MLLM tasks, its novelty and practical value are significantly diminished. In other words, a key question remains unanswered: Do we need a new, MLLM-specific visual pruning method like SCOPE, or could we simply \"plug-and-play\" existing state-of-the-art vision pruning methods into MLLMs and achieve comparable or even better results?\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}