Title: Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention
Venue: NeurIPS 2025
Accepted: yes
Keywords: Token Pruning

MLLMs

Holistic Compression
Abstract: Despite their powerful capabilities, multimodal large language models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens. Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention or [CLS] attention to assess and discard redundant visual tokens. In this work, we identify a critical limitation of such attention-first pruning approaches, i.e., they tend to preserve semantically similar tokens, resulting in pronounced performance drops under high pruning rates. To this end, we propose HoloV, a simple yet effective, plug-and-play visual token pruning framework for efficient inference. Distinct from previous attention-first schemes, HoloV rethinks token retention from a holistic perspective. By adaptively distributing the pruning budget across different spatial crops, HoloV ensures that the retained tokens capture the global visual context rather than isolated salient features. This strategy minimizes representational collapse and maintains task-relevant information even under aggressive pruning. Experimental results demonstrate that our HoloV achieves superior performance across various tasks, MLLM architectures, and pruning ratios compared to SOTA methods. For instance, LLaVA1.5 equipped with HoloV preserves 95.8% of the original performance after pruning 88.9% of visual tokens, achieving superior efficiency-accuracy trade-offs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HoloV, a new framework designed to improve the efficiency of multimodal large language models by intelligently pruning visual tokens. The authors observe that most existing token pruning methods focus too heavily on attention scores, often retaining only locally highlighted or similar tokens. This can lead to significant information loss and reduced performance, especially when a large proportion of visual tokens are pruned.\n\nHoloV addresses this issue by adopting a holistic approach to token selection. Instead of simply keeping the most salient tokens based on attention, the framework divides the image into spatial crops and adaptively selects tokens from different regions. This ensures that both global context and diverse local details are preserved, even when aggressive pruning is applied. The method balances local saliency with overall scene understanding, aiming to maintain important semantic information that would otherwise be lost.\n\nSTRENGTHS AND WEAKNESSES:\n### Strenghts\n* The paper presents comprehensive experiments across a wide range of benchmarks and MLLM architectures, demonstrating consistent and significant improvements over state-of-the-art token pruning methods.\n* The paper provides a well-articulated motivation for holistic context retention and offers insightful analysis of the limitations of prior methods, supported by visualizations and ablation studies.\n* HoloV is easy to integrate, offering substantial computational efficiency gains with minimal performance loss.\n\n### Weaknesses\n* Key implementation details, such as: how spatial crops are defined, the choice of hyperparameters, and ablation on the crop number or allocation sharpness, are insufficiently specified. This limits reproducibility and undermines the credibility of the claimed results.\n* The process for selecting the top-k visual tokens within each crop is largely based on a straightforward score maximization, but the underlying criteria for combining local saliency and global relevance remain rather heuristic and are not thoroughly justified or compared to alternative strategies. There is no empirical evidence or ablation provided to show how sensitive the method is to the exact formulation of the selection score, or whether more sophisticated selection strategies could lead to further improvements.\n* The adaptive holistic token allocation approach in HoloV is largely heuristic and lacks principled justification. The process involves averaging token scores within each crop, discretizing quotas, and then using iterative reallocation to handle rounding issues or quota overflow. While this ensures that each region of the image is represented, the method relies on several arbitrary choices, such as the setting of the sharpness parameter $\\tau$ and the specific reallocation rules. The paper does not provide any analysis or ablation to show how sensitive the method is to these choices, nor does it offer a theoretical rationale for why this allocation procedure is optimal. As a result, the effectiveness and robustness of this adaptive allocation are uncertain, and it remains unclear whether the improvements are due to this specific mechanism or would be matched by simpler or more principled alternatives.\n* While the paper provides a theoretical analysis of computational complexity using FLOPs, it does not sufficiently address how pruning affects actual performance on real hardware. In practice, efficiency gains from pruning are highly dependent on multiply-accumulate operations (MACs), memory access patterns, and especially system latency, which can vary significantly depending on the hardware architecture. On older or resource-constrained devices, aggressive pruning may actually introduce irregular computation and reduce parallelism, potentially resulting in longer inference times or even hardware underutilization. The paper does not empirically explore how HoloV’s pruning strategy impacts different hardware setups, nor does it report detailed results on wall-clock latency or MACs. For a complete and convincing assessment of practical efficiency, future work should explicitly measure and report latency and MACs across diverse hardware platforms, since improvements in FLOPs alone do not guarantee real-world speedup or energy savings.\n* I find the section on **visual context refetching (VCR)** to be interesting in theory, but I am concerned about the lack of concrete empirical evidence supporting its utility. While the mathematical description of how pruned visual tokens are reintegrated via the feed-forward network is clearly presented, the paper does not provide experiments or detailed analysis to demonstrate the real impact of this mechanism. Specifically, the authors do not explain how the injection ratio is selected in practice, and there is no data illustrating how VCR affects uncertainty reduction or downstream performance. Without a dedicated evaluation or ablation that isolates the effect of VCR, it is challenging to judge how important or reliable this component truly is within the broader HoloV framework.\n* While the paper provides results for both LLaVA-1.5 and LLaVA-NeXT visual encoders, the experimental analysis remains limited to this family of encoders. There is no investigation into how HoloV would perform with fundamentally different visual backbones, such as other types of vision transformers (Siglip2, Dinov2, etc..), convolutional encoders (CLIP-Resnet,ConvNeXt, EfficientNet, etc..), or others models (Mamba). Since different encoders may have unique tokenization strategies and varying levels of feature redundancy, it is unclear if HoloV’s advantages and efficiency–accuracy trade-offs would generalize beyond the LLaVA architecture. A broader ablation study across diverse encoder types would strengthen the paper and provide more confidence in the general applicability of the method.\n\nQUESTIONS:\n1. Can you provide detailed information on how spatial crops, crop numbers, and related hyperparameters are selected? Additionally, could you include ablation studies that show how these choices impact the reproducibility and performance of your approach?\n\n2. What is the rationale behind your specific top-k token selection score? How does this formulation compare empirically or theoretically to alternative selection strategies or scoring methods?\n\n3. Have you evaluated the sensitivity and robustness of your adaptive holistic token allocation procedure, including the effects of discretization and the sharpness parameter? Could you discuss or provide results comparing your allocation approach to other possible methods?\n\n4. Would it be possible to use HoloV as a replacement for the Q-Former module in models such as InstructBLIP, or to integrate HoloV within the Q-Former family of architectures? If so, could you discuss any practical considerations, challenges, or potential benefits of applying HoloV in this context? It would be helpful to understand whether HoloV is compatible with Q-Former-based models and how its holistic token pruning strategy might interact with or improve these architectures.\n\n5. Could you report practical efficiency metrics such as multiply-accumulate operations (MACs), wall-clock latency, and memory usage on different hardware platforms, including less powerful or older devices and different devices (GPUs with different floating point operations)? It would be helpful to see evidence of real-world efficiency beyond theoretical FLOPs reductions.\n\n6. Can you present empirical results or ablation studies that isolate and quantify the actual contribution of visual context refetching (VCR) in your framework? In particular, how is the injection ratio chosen, and what impact does VCR have on uncertainty reduction or downstream performance?\n\n7. Do you have results or ablation studies demonstrating how HoloV performs with a wider variety of visual encoders, such as SigLIP, DINOv2, CLIP-ResNet,S2, Mamba, ConvNeXt, etc? Would you consider evaluating whether your method generalizes to models beyond the LLaVA family?\n\nLIMITATIONS:\nSee questions and weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HoloV, a novel visual token pruning framework for MLLMs, which addresses the critical limitation of attention-based methods that prioritize local saliency over global semantic context. By introducing crop-wise adaptive allocation and a hybrid scoring mechanism combining attention saliency and semantic variance, HoloV ensures holistic context retention during aggressive pruning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. This paper provides a solid motivation for addressing visual token pruning in MLLMs. Traditional attention-based approaches tend to fragment global semantics by overly focusing on locally emphasized tokens.  By incorporating insights from cognitive science and analyzing Positional Bias, the paper highlights the root causes of performance degradation in complex scenes. \n2. HoloV shows exceptional control over performance degradation, retaining 95.8% of the original performance after pruning 88.9% of visual tokens.\n3.The paper is well-organized and easy to follow.\n\n**Weaknesses:**\n\n1. The fixed spatial crop partitioning method limits the system's applicability in complex scenes where semantic regions may be non-rectangular or cross boundaries. This can result in fragmented semantic units, reducing the model’s generalizability.\n2. There is also a sensitivity to hyperparameters like the sharpness factor (τ) and scaling factor (γ), which require manual tuning. The lack of a unified optimization strategy for varying resolutions or scene complexities increases deployment costs and limits real-world applicability.\n3. While HoloV mitigates Positional Bias, it still struggles in extreme conditions such as severe lighting, occlusions, or fast-moving objects. These challenges lead to potential information loss, as seen in MSRVTT-QA experiments where pruning errors affected dynamic object understanding.\n\nQUESTIONS:\nPlease see the Major points listed under Weaknesses. \nHow does the granularity of crop partitioning (e.g., number of crops) affect pruning efficiency? For high-resolution images (e.g., LLaVA-NeXT's 2,880 tokens), should the number of crops be dynamically adjusted?\n\nLIMITATIONS:\nWhen an LLM has severe language bias, could HoloV's visual pruning indirectly exacerbate hallucinations? For example, if the language model misjudges an object's existence, might HoloV retain irrelevant visual tokens that reinforce this error?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces HoloV, a plug-and-play visual token pruning framework for MLLMs that moves beyond attention-based token selection by preserving holistic visual context. Unlike previous methods that focus on high-attention tokens, HoloV adaptively allocates pruning budgets across spatial crops, balancing local saliency with global semantic diversity. The method significantly reduces visual tokens (up to 88.9%) while retaining up to 96% of the original performance, improving inference efficiency across multiple benchmarks and models.

STRENGTHS AND WEAKNESSES:
### Strength:
1. The paper is detailed and clear, with thorough experiments, visualizations, and theoretical analysis that make the claims convincing.
2. Experiments on rich benchmarks with various retained token numbers show promising improvements, demonstrating the effectiveness of the proposed method.

### Weakness:
1. The evenly partitioned crops may not be flexible enough and could introduce potential spatial bias.

QUESTIONS:
### Question:
1. I would like to know whether HoloV can still achieve consistent strong performance on models using native resolution (e.g., Qwen2.5-VL). Adding such results would allow HoloV to cover all mainstream VLM visual encoding forms, further enhancing the persuasiveness of the results.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new visual token reduction strategy to improve the speed of MLLMs with maintained performance. Compared with prior works that directly reduces tokens with low attention values, the proposed method adaptively distributes the token reduction across spatial crops. This strategy can capture more global visual context, tackling the limitation from prior work that keeps too many semantically similar tokens in the same spatial region. Benchmark results show that the proposed method demonstrates superior efficiency-accuracy trade-off, outperforming the prior SOTA method on various benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The authors observed that current token pruning method only focuses on the attention value, which could lose the informative global semantics and pay too much attention on the local salient tokens. Thus, this paper proposes a new method to allocate tokens more evenly in a global view, but still consider the local saliency factor. The overall design is reasonable.\n2. The benchmark results show that the proposed method achieves SOTA performance on almost all benchmarks over the prior methods. In addition, there is no significant computation/latency overhead of the current method.\n3. The paper is relatively easy to understand.\n\nWeaknesses\n1. Although the overall idea is relatively easy to understand, some concepts in the paper may need revision:\nLine 51 & Line 65: There is a concept called \"semantic connectivity\". However, it seems there is no evidence showing such \"connectivity\" between semantics. The goal of the method seems more like to capture more global semantics rather than local features. \nLine 150: \"Attention dispersion\" seems a concept that a small amount of tokens take the majority of the attention. Under this concept, [CLS] attention is steeper in the curve, shows more aligned with the concept of \"attention dispersion\", but Line 157 - 159 seems to show that last-token attention and equi last attention are more aligned with such concept?\nLine 164 - 165: It introduces another concept \"global contextual cohesion\" which is only mentioned once and never used again. Seems this concept is not necessary.\n\n2. Line 134 - 135 \"As shown in Fig. 4 left, adjacent tokens with similar visual features frequently receive comparable attention scores\" It seems difficult to reach such conclusion from Fig. 4 left. Please elaborate more on it.\n\n3. Line 131 \"we focus on the attention received by visual tokens from the visual [CLS] token\" Seems there is a limitation of the only focus on attention from the CLS token. I wonder if the mutual attention between visual token needs to be considered.\n\n4. Sec 4.1 Issues in the formula (1) need to show dimension for the matrices and vectors (2) eq.2 should have $\\sum_j$ instead of $\\sum$ (3) $\\mathcal{H}^c$ is never used (4) $\\Omega_c$ is not well explained (5) In eq. 5, why minimizing global diversity $\\mathcal{V}^c$? Do we want similar patterns in each crop?\n\n5. Typo: Table 3 Avgerge  -> Average\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nThe authors have shown some reasonable limitations in the `Limitations and Future Work` section", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-09"}