Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention
Authors: Xin Zou
Di Lu
Yizhou Wang
Yibo Yan
Yuanhuiyi Lyu
Xu Zheng
Linfeng Zhang
Xuming Hu
Venue: NeurIPS 2025 poster ()
Keywords: Token Pruning
MLLMs
Holistic Compression
Primary area: deep_learning
TL;DR: HoloV prunes 88.9% visual tokens in MLLMs while retaining 95.8% performance via spatial visual holistic context retention.

Abstract: Despite their powerful capabilities, multimodal large language models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens. Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention or [CLS] attention to assess and discard redundant visual tokens. In this work, we identify a critical limitation of such attention-first pruning approaches, i.e., they tend to preserve semantically similar tokens, resulting in pronounced performance drops under high pruning rates. To this end, we propose HoloV, a simple yet effective, plug-and-play visual token pruning framework for efficient inference. Distinct from previous attention-first schemes, HoloV rethinks token retention from a holistic perspective. By adaptively distributing the pruning budget across different spatial crops, HoloV ensures that the retained tokens capture the global visual context rather than isolated salient features. This strategy minimizes representational collapse and maintains task-relevant information even under aggressive pruning. Experimental results demonstrate that our HoloV achieves superior performance across various tasks, MLLM architectures, and pruning ratios compared to SOTA methods. For instance, LLaVA1.5 equipped with HoloV preserves 95.8% of the original performance after pruning 88.9% of visual tokens, achieving superior efficiency-accuracy trade-offs.

Don't Just Chase "Highlighted Tokens" in MLLMs:
Revisiting Visual Holistic Context Retention
Xin Zou Di Lu Yizhou Wang', Yibo Yan Yuanhuiyi Lyu!??
Xu Zheng".3, Linfeng Zhang', Xuming Hu
The Hong Kong University Science and Technology (Guangzhou) The Hong Kong University of Science and Technology
3 INSAIT, Sofia University "St. Kliment Ohridski" Shanghai Jiao Tong University
https: //gptbb.///iuu....../ com/ obaoanas/Hooolo
Abstract
Despite their powerful capabilities, Multimodal Large Language Models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens. Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention or [CLS] attention to assess and discard redundant visual tokens. In this work, we identify a critical limitation of such attention-firs. pruning approaches, i.e., they tend to preserve semantically similar tokens, resulting in pronounced performance drops under high pruning ratios. To this end, we propose HoloV, a simple yet effective, plug-and-play visual token pruning framework for efficient inference. Distinct from previous attention- first schemes, Holo'V rethinks token retention from holistic perspective. By adaptively distributing the pruning budget across different spatial crops, HoloV ensures that the retained tokens capture the global visual context rather than isolated salient features. This strategy minimizes representational collapse and maintains task-relevant information even under aggressive pruning. Experimental results demonstrate that our HoloV achieves superior performance across various tasks, MILM architectures, and pruning ratios compared to SOTA methods. For instance, LLLVA ..5 equipped with HoloV preserves 95.8% of the original performance after pruning 88.9% of visual tokens, achieving superior fffiiinncyaaccuran trade-offs.
Introduction
Multimodal Large Language Models (MLLMs) have demonstrated outstanding capabilities [82, 12] in tasks such as image captioning [35, 61, 14], visual question answering [24, 99, 36), and video understanding [34, 64, 79]. However, these models [43, 78, 38] typically require converting visual inputs into long sequence representations (i.e., visual tokens), which increases the computational complexity and cost of inference [97], especially for high-resolution images [41] and multi-frame videos [57], where redundant visual information further exacerbates the computational overhead.
To address this challenge, researchers have introduced token pruning strategies [49, 13, 98, 87] that aim to retain the highlighted visual tokens as well as prune others for accelerating MLLM's inference. These methods typically define importance criteria for tokens, such as attention scores [13, 19] or gradient information [59, 58), to quantify the significance of visual tokens, and less important tokens pruned during the inference phase, which balances speed and performance, but with limitations.
*Corresponding author, fEqual contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

60-
30-
65-
performancer
performancet
80
performanee
-5
performan con
75
70-
-6
POPE
MMB
555
60
65
-0-
FastV
HoloV SparseVLM
Fasty
-HoleV
60
FastV
HHoki
30
Baseline
SparreVLL
40
0 Baseline
Baseline - SparseVV
Baseline SparsevLL
35
0.24
0.50
0.75
0.95
0.28
0.50
0.75
0:95
0.25
0.50
0.25
0.05
0.25
0.50
0.75
0.05
pruning
pruning
pruning ratio
pruning ratio
Figure 2: Relationship between performance and pruning ratios of different baseline methods. As the token pruning ratio grows, the performance of these attention-firs strategies degrades dramatically, while VoloV maintains the substantial performance even at 90% and 95% of the pruning ratios.
As shown in Fig. 1, FastV [13] is an intuitive solution that ranks visual tokens based on attention distribu- tions across different layers, and then prunes the bot- tom R% of tokens based on the computational budget. thus reducing visual token redundancy. Subsequently. more work has followed this paradigm [91, 98, 4], de- signing different strategies to prune redundant visual tokens via mrosl (i.e., text-vision) attention from LLMs. Besides, there are vision-centric prun- ing methods [77, 25, 94, 66, 88] (e.g., FasterVL.M [93]) that presume those visual tokens with low corre- lation to the [CLS] token in ViT [17], or those exhibit duplicated features tokens [20] to be redundant.
Langrage Modil
Visai
Bexavved tokuns
Quary What of porato digs th&? Reggamen Oidh
Whot
Faser
Language Mbdd ft
LoVs
Vicui Bhoodls
X
X
X
Resevvdd
Quary: What boord porato ciip is tias?
What
tolems
Lay's
Figure 1: Snapshots of FastV and our VoloV.
Although these pruning methods can recognize the inefficiency of visual tokens in MLLMs, they are not consistently effective. As shown in Fig. 2, the performance decreases significantly as the pruning ratio increases. In our argument, this occurs because these approaches implicitly assume that visual tokens with high attention correspond to higher informativeness, which disregards the spatial- semantic relations of the visual scene, i.e., they tend to retain tokens from localized salient regions where attention is drawn to, rather than those conducive to holistic semantic comprehension. Thus, at a high pruning ratio, such methods would only retain homologous tokens with higher scores. In a complex scene with multiple objects, retaining only "highlighted tokens" may sever relative positional and semantic connectivity information or lose key tokens associated with the subject, leading to a dramatic performance degradation. Besides, the attention mechanism introduces systematic biases [80, 81], i.e., the position encoding mechanism of transformer- based MLLMs may introduce spatial priors, those in upper and lower areas visual tokens usually being assigned higher attention weights as shown in Fig. right. This bias can distort the semantic contributions of the visual scene, leading the model to produce incorrect or logically contradietory inferences, even hallucinations [100, 103]. Drawing inspiration from the above discussion, we raise the following question: "How to locate and preserve those not highlighted but critical visual holistic understanding tokens?"
Cognitive science research suggests that the human visual system forms a complete semantic under- standing by integrating local features with global scene cues [70, 2, 63] (e.g.. background textures and spatial layouts). MLLMs, we analyzed the text-mapping relationships of different visual tokens through the strategy in [60]. As shown in Fig. left, the objects in scene could be represented by
Bill" "Teala"
Jacke "Honciigg
Peer" "A4
"ace"" "Iadde"
75%
50%
20%
120%
"Sel"" "
Sandle
"habling
"Woo""
75%
50%
12.9%
Figure 3: LEFT Examples of textual semantics corresponding to visual tokens from scattered crops. RIGHT Sparsification visualization examples of FastV, where retention ratios tagged in the pics.
2

---

small number of scattered tokens, and the semantic relationships between those tokens from different regions facilitate the overall understanding, e.g., "snow", "ski", "hills" are kind of self-explanatory. Motivated by this insight, we propose HoloV, which explicitly balances overall semantic connectivity and contextual attention during visual token pruning. addressing the critical limitation of redundancy in attention-fise firstt strategies. Our analysis demonstrates the importance of preserving visual holistic context, offering a new perspective on efficient visual token pruning in MLLMs. Through extensive experiments on diverse benchmarks and MLLM architectures, we demonstrate that HoloV consis- tently surpasses existing state-of-the-art token pruning approaches, achieving up to 88. 9% token reduction while preserving about 96% of the original performance. Besides, HoloV is model-aggostie and easily integrable into a wide range of MLLMs, making it well-suited for practical deployment.
2 Related Work
2.1 MLLMs and Their Challenges
The recent remarkable success of Large Language Models (LLMs) [62, 95, 72, 18, 56] has spurred the trend of applying their strong capabilities to multimodal comprehension tasks, fostering the development of MLLMs [1, 69]. Leveraging open-source LLMs such as LLaMA families [72, 73, 18], MLLMs [6. 46, 47] have demonstrated enhanced adaptability across a range of visual understanding tasks, leading more profound ability to interpret the world. While this empowers LLMs with the capability of visual perception, the incorporation of lengthy visual tokens significantly escalates the computational burdens. Moreover, studies have shown that existing MILMs still suffer from certain visual deficiencies [71, 32] and some hallucinations [29, 28]. Some work mitigates these issues by increasing the resolution of input images or videos [55, 86], but this further exacerbates the computational overhead. For example. VL-11-1 [48] encodes 336-resolution image into 576 visual tokens, while LLa [47] doubles the resolution and generates 2,880 tokens. VLaVA-One Vision [37] represents image using 7,290 visual tokens, and Video-LLaVA [44] faces even higher costs, as it must process numerous visual tokens from multiple frames during inference. These visual tokens occupy large portion of the context window of their LLMs. In this work, we conducted experiments and analysis on these representative models to verify HoloV's applicability.
2.2 Visual Redundancy Identification
In MLLMs, visual redundancy identification facilitates the distillation of visual tokens with high informativeness for faster inference. There are two main research directions: a) Vision-centrie strategies analyze the image's structure and feature distribution to discard less relevant visual tokens [13, 77]. Existing approaches include pruning based on attention scores [25, 89, 84], and using information bottleneck or entropy metrics during the prefilling stage to estimate background redundancy. b) Instruction-centricc strategies typically use cross-modal attention analysis or gradient accumulation to identify redundant tokens (49, 101, 68). Tokens with low attention or negligible gradient impact are deemed redundant [26]. Building on this, some studies explore learned importance scoring, training lightweight end-to-end model to predict each patch's is "instruction relevance," " enabling even finer- mner-ggriied pruning [31, 75, 91]. As the existence of language bias in LLM may cause hallucinations, we use vision-centric scheme.
clustering (e.g.. TokenLearner [65]), dynamic
2.3 Visual Token Compression and Pruning
The inclusion of visual information in MLLMS introduces long token sequences, leading high computation and memory costs. For example, mini-Gemini-HH [41] generates 2880 tokens from high-definition images, creating inference bottlenecks. To address this, research has focused on token compression and pruning techniques in Vision Transformers [10] and MLLMs [27]. Methods like LLaMA-VID [40] and DeCo [90] address this by modifying models and adding training, which increases computational costs. ToMe [11] reduces tokens without training but disrupts early cross- modal interactions [83]. LLaVA-PruMerge [66] selectively retains key tokens while merging less critical ones based on key similarity. FasterVLM [93] utilizes [CLS] attention scores from the visual encoder to re-rank and retain top visual tokens. FastV [13] and Sparse VLM [98] focus on token selection using attention scores or cross-modal guidance, but overlook the role of token duplication and lack -last-Attention [16, 15]. Our proposed HoloV maintains hard acceleration compatibility (e.g., Flash-Attentionn and effectively retains visual holistic context during aggressive pruning.
3

---

Preliminary and Motivation
3.1 Preliminary
Architecture of MLLMs. Given an MLLM MILLM parameterized by 0, with a general architecture consisting of text embedding layer, vision encoder, vision-text interface module, text decoder consisting of L number of transformer layers, and an affine layer which predicts the distribution of the next token. For image-grone grounded text generation task, given textual query and input image v, M M first extracts vision features of by the vision encoder, and then converts them into visual tokens Zy by MLP Q-Former [76] modules. Aligned vision tokens 2u concatenated with the query x as input to the text decoder, and finally decoded into a textual response y autoregressive, which is formulated as: Y/ Pe (-1v,, 26 Mec) & softmax( (e (,),,,,, (ct)),, where yr indicates the fth token, Ver is the token sequence generated up to the time step t. and fe is the logit distribution.
Attention mechanism. Considering the computational burden associated with the length of visual tokens in MLLMs, many studies have followed the paradigm of using attention scores to evaluate the redundancy of visual tokens. Specifically, transformer- based MLLMs typically utilize causal self- attention [5] to perform computation as: Self-attention/ QQ K, V) softmax (Q -QKK //dx) ) V, where da is the dimension of K, the result of softmax (Q K /vdx) is known as the attention matrix, In this work, we focus on the attention received by visual tokens from the visual [CLS] token.
3.2 Information Redundancy in Highlighted Tokens
When token selection is based exclusively on attention scores, the model tends to retain similar clusters, resulting in information redundancy. As shown in Fig. 4 left, adjacent tokens with similar visual features frequently receive comparable attention scores, especially in regions characterized by flat backgrounds repetitive textures. Their spatial proximity leads these tokens capture overlapping features, making it hard to distinguish those not highlighted yet informative tokens.
Fest
HoloV
Figure 4: LEFT Distribution map of visual token attention. RIGHT Visualization cases of Vast and HoloV. HoloV retains contextual tokens with rich semantics, while FastV contains much redundancy.
Positional Bias. To further investigate attention-based token pruning methods, take FastV an example and visualize the distribution of the retained visual tokens. As illustrated in Fig. 4 right, the attention scores for image tokens present consistent pattern: tokens located at the beginning and end of the sequence tend to have higher attention and are thus more likely to be preserved during pruning, leading
positional bias. We extend our analysis by conducting statistics on samples from the text-based VQA task using the VQA V2 [23] dataset. Notably, even though these samples originate from different task, the attention distributions image tokens at the same layer remain highly similar, revealing recurring patterns. While the overall shape of the distributions varies slightly across layers, the set of tokens receiving relatively high attention remains stable. We suggest that this phenomenon occurs because all visual tokens are processed with text tokens in the same manner during decoding, leading to positional bias of text shift to the visual modality, egg., boundary positions of text usually imply important information, but for images, targets are mostly located in the center.
Cummulative
Attention Dispersion. addition to positional bias, we further analyze the phenomenon of attention dispersion, i.c., small subset of similar tokens receives the majority of attention, while most tokens are assigned low attention scores [93]. Specifically, we compute the cumulative distribution of visual tokens sorted by their attention scores, as shown
Comuuviioon Atbontion
80
60
40
in Fig. 5. The curves of last-token attention [13] and equi last attn with identical position embedding are noticeably less steep than that for [CLS] attention. It is evident that compared to [CLS] attention, text-vision attention tends be dispersed over more visual tokens, e.g.,
20
Attn
Visual Token Proportion (%)
Figure 5: Cumulative distri- bution of different attentions.
the top 20% of visual tokens account for only 40% of the total attention.

---

3.3 Holistic Context Trumps Local Duplicates
Based on our previous analysis, attention-fire token pruning methods suffer from wer-locaaiizatio due to positional bias and attention dispersion, i.e., over-reliance on attention scores disrupts spatial- semantic relationships, e-g.. breaking occlusion hierarchies in multi-object interactions. Thus, our key insight is that visual token importance should be evaluated through global contextual cohesion, Le., jointly considers holistic context and local saliency rather than isolated attention magnitudes.
To further validate our hypothesis, we devised a straightforward holistic context retention strategy, i.e., pruning visual tokens through random masks to retain visual information from different regions. As shown in Fig. 6 up, compared with FastV, this random strategy out- performs more than half of the benchmarks, which demonstrates the significance preserving holistic context for visual understand- ing. On the VQA text dataset, however, the random strategy failed, possibly because random pruning discards some salient fineggriied information. This result also suggests that local saliency is indispens- able, especially for densely packed elements within small regions.
Fasti Random
36
Performance
V2
VQNrea port:
Performance (56)
milebal only muocra only 9
98
96
In addition, we conducted an exploratory experiment to investigate how holistic context contributes visual understanding in MLLMs. Specifically, we use the global thumbnail and multiple local crops as visual input separately [47]. and evaluate performance on the two settings against various benchmarks. As shown in Fig. 6 down, with only the global thumbnail yields strong results on general visual perception benchmarks such as MMBench [53], MME [21], and MM- Vet [92], highlighting the inherent role of holistic context in guiding general visual understanding. On the contrary, using only local crops leads to poor performance in these general perception tasks but excels in fine-gris fine-grained perception benchmarks such as TextVQA [67] and POPE [42], which suggests that local duplicated saliency can offer fine-grained visual information for semantic understanding.
MM4Bad KIME 506-6 VOM POPE
Figure 6: Up FastV v.s. Ran- dom strategy. DOWN Perfor- mance comparison of the thumb- nail and local crops as inputs.
4 Methodology
Building on the above analysis, we propose HoloV, which better preserves the holistic context of images for visual understanding. By removing redundant visual tokens before the LLM decoder, our approach could make MLLMs inference faster than methods that prune tokens within the LLM. An overview of our approach is depicted in Fig. 7. In what follows, we elaborate how HoloV guides overall visual token compression under a high pruning ratio to keep semantic completeness.
4.1 HoloV Framework
To address the pivotal question raised in Sec. 1 for effective and efficient visual token pruming, propose HoloV framework, which leverages crop-wise adaptive allocation decentralize attention over those non-highlighted but heterogeneous tokens. Fig. 7 illustrates the idea of Holo'.
[CLS]
[CLS)
Prioritizo
R Re-rant
iile
Reserved Tokens RoservedTokkns (Intra and Inter-Crop)
Crop-wise Tokens
Discarted Tokens with High Attention
Figure 7: Illustration of HoloV. We re-rank highlighted visual tokens for holistic context retention.
Based on our findings about the positional bias, We first rearrange visual tokens into local crops. Let the total number of image tokens be N,, which evenly partitioned into crops. This enables the model to maintain spatial granularity and gather statistics both locally and globally. Given the
5

---

normalized embeddings 2 E RM in e-th crop, we first compute intra-crop similarity matrix S as
Se = lal- o z
where @ denotes Hadamard product, and IM the identity matrix masking eeff-iimllaiiiies. Then, we capture intra-crop diversity by the variance of semantic distribution, the formula is as follows
V
M M E (S) -4)
(2)
where a high value of V indicates that i-th token has diverse connections with others, the visual semantics expressed by the informative token is essential within the crop. To obtain holistic attention, we establish a balanced scoring mechanism combining contextual diversity and attention saliency, Specifically, we merge variance y and [CLS] attention A the crop using adaptive scaling:
Hf 7c A where %o | [I]
Adaptive holistic token allocation. To preserve overall scene semantics and spatial diversity, we compute a crop-level priority score by averaging token scores within each crop. The total quota for selected image tokens T' dynamically allocated to crops according to their normalized crop-level importance. The allocation to each crop is discrete and capped, ensuring spatial coverage while preventing over-concentration on specific regions. We resolve rounding and overflow through an iterative reallocation procedure, so that crops with excess quota donate surplus tokens to those with remaining capacity, according to their crop-level scores.
We compute crop importance weights via
1 M
W.
H
(4)
where controls the sharpness of allocation. Thus, initial quota qe [w.NN.l where No denotes the number of retained tokens. When the allocated tokens overflow or fall short, we redistribute residual tokens. For overflow, the quota is changed by 9. min(ge + Ac, .M,, Ae W. (M ge). while for fall short, we allocate the remaining quota to the crop with the highest weight. In this way, HoloV adaptively adjusts its compression degree according to the informativeness of different crops. Top-k visual token selection. Within each crop, select visual tokens by maximizing:
argmaNn.c(t.....r/ Σ H. subject [2cl qe
which ensures both crop-wise local saliency and global relevance. We retain top-k visual tokens in each crop, where determined by the quota g. the allocation. By performing token pruning before the L.M decoder, we dynamically adjust the number of visual tokens as input to the language model based on the actual computational budget, thus accelerating the MILM inference.
4.1.1 Fast Visual Context Refetching
Motivated by the attention sinks [96], and information loss during visual token pruning, we further propose visual context refetching to fast supplement the visual holistic context. Specifically, we treat pruned tokens as supplementary evidence, re-injecting them into the MLLM through Feed Forward Network (FFN) as "key-value memory" at the middle trigger layer. This refeich mechanism occurs when the model exhibits high uncertainty during inference, achieving effective and efficient visual information replenishment. Limited by space, the details can be found in Appendix D.
4.2 Theoretical Analysis
To further justify the trustworthiness of our proposed HoloV, we provide a theoretical analysis of it. Under Assumption 1, for any pruned token, there exists a retained token that is sufficiently close the embedding space, with bounded context variance. By leveraging the Lipschitz continuity [8] of the transformer layer, we can bound the semantic difference between the outputs on the original and pruned token sets. The residual error introduced by the scoring threshold also controlled. Combining these components, we obtain the stated upper bound. More details are in Appendix C.

---

Table Performance comparison of various methods across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue.
Methods
GQA
MMB
MMBeN MME
POPE
SQA
VQAvz
vQArea VizWiz
Average
Upper Bound, 576 Tokens
61.9
64.7
58.1
1862
85.9
69.5
78.4
58.2
50.0
100%
LLaVA-1.5 7B
Retuin
192
Tokens [1
66.
ToMe (ICLR23)
54.3
60.5
1563
72.4
65.2
68.0
52.1
88.5%
FasTV (F00924)
52.7
61.2
57.0
1612
64.8
67.3
67.1
52.5
50.8
90.5%
MusiDrop (2624.11)
58.2
62.3
55.8
1787
82.6
69.2
76.0
$6.5
51.4
97.2%
LLaVA-PruMerge (ICCV25)
54.3
59.6
52.9
1632
71.3
67.9
70.6
54.3
91.4%
PDrop (CVPR25)
57.1
63.2
$6.8
1766
82.3
75.1
$6.1
51.1
96.7%
FiCoCo-V (2025.03)
58.5
62.3
55.3
1732
82.5
67.8
74.4
55.7
51.0
96.1%
HiRED (464125)
58.7
62.8
54.7
1737
82.8
68.4
74.9
47.4
50.1
94.6%
VisionZip (CVPR25)
59.3
64.5
57.3
1767
86.4
68.9
76.8
57.3
51.6
98.1
SpanseVL.M (5CM)25
57.6
62.5
53.7
1721
83.6
69.1
75.6
56.1
50.5
96.1%%
DART (EHNLP25)
58.9
63.6
57.0
1856
82.8
76.7
57.4
51.1
98.5%
HoloV (Ours)
65.4
58.0
1820
85.6
76.7
57.4
50.9
99.2%
LLaVA-1.5 7B
Retain 128 Tokens (L 77.8%)
ToMc (ICLR23)
52.4
53.3
1343
62.8
59.6
63.0
49.1
80.4%
FastV (F00124)
56.1
56.4
1490
59.6
61.8
$0.6
51.3
85.4%
MustDrop (2824.11)
56.9
61.1
55.2
1745
78.7
68.5
74.6
56.3
52.1
95.7%
LLaVA-PruMerge (1CCV25)
53.3
58.1
51.7
1554
67.2
67.1
68.8
54.3
50.3
89.4%
PDrop (CVPR25)
56.0
61.1
56.6
1644
82.3
68.3
72.9
55.1
51.0
94.9%
FiCoCo-V (3875.83)
57.6
61.1
$4.3
1711
82.2
68.3
73.1
55.6
49.4
94.9%
HiRED (AA4125)
57.2
61.5
53.6
1710
79.8
68.1
73.4
46.1
51.3
93.1%
VisionZip (CVPR25)
63.4
56.7
1768
84.7
75.6
56.8
52.0
97.2%
SparseVLM (ICML25)
56.0
60.0
51.1
1696
80.5
67.1
73.8
$4.9
51.4
93.8%
DART (EMNLP25)
57.9
63.2
57.0
1845
80.1
69.1
75.9
56.4
51.7
97.5%
HoloV (Ours)
57.7
63.9
56.5
1802
84.0
69.8
75.5
56.8
51.5
98.0 %
A. 7B
Retain 6: Tokens
88
(88.9%
ToMe (ICUR23)
48.6
43.7
1138
52.5
50.0
57.1
45.3
70.1%
FastV (B0CV24)
46.1
48.0
52.7
1256
48.0
51.1
55.0
47.8
50.8
76.7%
MustDrop (2874.11)
53.1
60.0
53.1
1612
68.0
63.4
69.3
54.2
51.2
90.1
LLaVA-PruMerge (ICCV25)
51.9
55.3
49.1
1549
65.3
68.1
67.4
54.0
50.1
87.7%
PDrop (CVPR25)
41.9
33.3
50.5
1092
55.9
69.2
45.9
50.7
77.5%
FiCoCo-V (2825.e3)
52.4
60.3
53.0
1591
76.0
68.1
71.3
53.6
49.8
91.5%
HiRED (MA1225)
54.6
60.2
51.4
1599
73.6
68.2
69.7
44.2
50.2
89.4%
VisionZip (CVPR25)
55.1
60.1
55.4
1690
77.0
69.0
72.4
55.5
52.9
94.5%
SpanseVLM (ICYL25)
52.7
56.2
46.1
1505
75.1
62.2
68.2
51.8
50.1
87.3%
DART (EHNLP25)
60.6
53.2
1765
73.9
69.8
72.4
54.4
51.6
93.9%
HoloV (Ous)
55.3
63.3
55.1
1715
80.3
72.8
55.4
52.8
95.8%
4.3 Computational Complexity
As language instructions are much shorter than visual tokens, we focus on the FLOPs contributed by visual tokens. Let denote the number of visual tokens, the hidden size, and the FFN intermediate size (with SwiGLU). For the prefill stage, the FLOPs per transformer layer can be approximated as an bndd endm, where a, b, and e constants. If the token count reduced by a ratio R (f (1 R)n), the FLOPs reduction ratio is:
an'd + bid + cñdm
F= an?d bnd 2 + cndm
For large n, the quadratic term dominates, slightly better than linear R. In the decode stage (with KV cache), the complexity becomes linear in n, and the FLOPs per layer are bd? + (bd edm)n, the reduction nearly proportional R. HoloV speeds up inference by pruning ahead of the LIM to avoid KV cache inefficiency.
1 R1 R². Thus, the reduction is
Experiments
5.1 Experimental Setup
Benchmarks. We conducted experiments on several widely used visual understanding benchmarks.
For image understanding task, we performed experiments on ten widely used benchmarks, including
GQA [30], MMBench (MMB) and MMB-CN [53], MME [21], POPE [42], VizWiz [9], SQA
(ScienceQA) [54], VQAv2 (VQA V2) [23], VQATeu (TextVQA) [67], and MM-Vet [92]. Video

---

performancejj
porformance/96,
60-
performance:1%
porformancee
70
55
50-
GQA)
60
(SQA)
54
52
(QAAro) )
60
TastV
-Hiido
Fasty
--oolov Basellre SparseVLM
25
Fasty
-Vbbb SparseVLM
-ooobV Baseline SparsevLM
Baseline-
SparseVLM
85-
Baseline
35
0.75
0.50
0.50 pruning ratio
pruning ratio
pruning ratio
pruning ratio
60
performancep9
perfarmanceys5
performanca
30
POPE
60-
MMB
50
MMB
28
MMVeet
50-
Tasty
55
Fasty
HHolo
FastV
Hoblo SparseVVL
24
40
50
0.50
0.75
0.36
0.25
0.50
0.75
0.95
0.25
0.50
0.75
0.95
0.25
0.50
0.75
0.55
pruning ratio
pruning ratio
pruning ratio
pruning ratio
Figure 8: Comparison of different methods across multiple benchmarks under varying pruning ratios.
QA benchmarks include MSVD-QA and MSRVTT-QA [85]. All experiments on these benchmarks follow the default settings. More details of the benchmarks are provided in Appendix A.
Comparison methods. We compare approach with several representative methods for accelerat- ing multi-modal language models (MLLMs) via token reduction, including ToMe [11], FastV [13], Sparse VLM [98]. HiRED [4], ALLVA-PruMergy [66], PDrop [83], MustDrop [49], LasterVLM [93], GlobalCom VisionZip [88]. DART [81]. These baselines employ diverse strategies such as token merging, attention- based pruning, adaptive allocation, and hierarchical retention improve efficiency by reducing redundant tokens. Each method offers a unique perspective on balancing computational cost and model performance. More details of baselines are provided in Appendix A.2.
5.2 Main Results
General-purpose benchmarks. We evaluate the performance of HoloV general-purpost datasets, i.e., GQA, Vet, MME, MMBench, SQA, and VizWiz. As shown in Tab. 1. HoloV consis- tently outperforms competing approaches at different pruning ratios, e.g., HoloV removes up to 88.9% of visual tokens with only a 4.2% performance drop, and 77.8% with just 2% on average. results under varying pruning ratios, shown in Fig. 8, the performance of FastV and Sparse VLM drops dramatically un- der high pruning ratios, while HoloV maintains robust performance with relatively minor losses at all pruning ratios on SQA and MMBench. On MMBenchew and MM-Vet, HoloV even achieves higher than baseline (unpruned) scores at pruning ratios of 25%, 50%, and 75% (MM- Vet), then the score slowly drops as the pruning ratio increases. For VizWiz evaluation. the result in Fig. 9 indicates that HoloV can consistently obtain performance improvements at different pruning ratios, even at 95%, which means HoloV effectively retains visual holistic semantics.
Further, we show
VizWiz
52
50
49
-Holov SparseVLM
48
0.25 0.50
0.75
0.95
pruning ratio
Figure 9: Performance of different methods on VizWiz under varying pruning ratios.
Hallucination benchmarks validation. We conduct the hallucination evaluations on POPE and MME benchmarks, with results on LLaVA- 1.5-7B presented in Tab. 1, where the proposed HoloV shows robust capabilities, and the performance significantly exceeds the results of the compared SOTA methods, e.g., with a pruning rate of 88 98,9% HoloV achieves 80.3% accuracy compared 76% for the second runner-up on POPE, and achieved desirable performance on MME evaluation, compared to other comparative approaches.
5.3 VoloV with Higher Resolution
For further comprehensive evaluation, we also evaluated HoloV for LLaVA-NeXT on different bench- marks mentioned above, with comparison current SOTA approaches. LLaVA-NeXT introduces new image processing method, leading to dynamic lengths of visual embeddings for various image inputs. Thus, during the evaluation, 320 visual tokens has been kept (from up 2880 raw tokens). As shown in Table 3, the evaluation results of all various benchmarks show that HoloV obtained the highest score on almost every track, and has an average of 95. 6%, much higher than the current SOTA of 93.3%.
8

---

Table 3: Performance comparison of various methods across different benchmarks. Results are shown for different pruning ratios, with accuracy and average performance highlighted. Best results in blue.
Methods
GQA
MMB
MMBeN
MME
POPE
SQA
VQAvz
vQA VizWiz
Average
Upper Bound, 2880 Tokens
64.2
67.4
60.6
1851
86.5
70.1
81.8
64,9
57.6
100%
LLaVA-NeXT 7B
Retain 320
Tokens (4 (8.9%)
FastV (FOC994)
61.6
$1.9
71.7
62.8
71.9
55.7
53.1
88.0%
LLaVA-PruMerge (I01V25)
53.6
61.3
55.3
1534
60.8
66.4
69.7
50.6
54.0
85.6%
PDrop (CVPR25)
56.4
63.4
56.2
1663
77.6
67.5
73.5
54.4
54.1
90.9%
MusiDrop (2824.11)
57.3
62.8
55.1
1641
82.1
68.0
73.7
59.9
54.0
92.2%
PasterVLM (ICCV25)
56.9
61.6
53.5
1701
83.6
66.5
74.0
$6.5
52.6
91.1%
HiRED (A44125)
59.3
64.2
55.9
1690
83.3
66.7
75.7
58.8
54.2
93.3%
SparseVT.M (5)
60.6
54.5
1533
82.4
71.5
58.4
52.0
89.7%
GlobalCom" (2025.3)
57.1
61.8
53.4
1698
83.8
76.7
$7.2
54.6
92.2%
DART (EHNLP25)
61.7
65.3
58.2
1710
84.1
79.1
58.7
56.1
93.9%
HoloV (Ours)
61.7
65.3
57.5
1738
83.9
68.9
79.5
58.7
55.3
95.6%
Table 4: Real inference comparison on POPE. Experiments adopt 66.7% and 90% pruning ratios.
Methods
Time
Prefill
Latency Mem.
Acc.
Time
Prefil
Latency Mem.
Acc.
Upper Bound, 576 Tokens
49:41
0.5ms
0.334s
19.0G
100.%
49:41
0.5ms
0.334s
19.00
100.%
LLaVA-1.5-77
Retain 192
Tokens (1 66.777 66.7%)
Retain 58 Tokens (1 90%)
FastV (ECCV24)
35:34
0.5ms
0.239s
16.0G
75.4%
30:41
0.5ms
0,206s
15.6G
66.8%
MustDrop (2824.11)
32:30
0.5ms
0.273s
15.6G
96.2%
29:40
0.6ms
0.199s
14.5G
87.1%
FasterVL.M (ICCV25)
30:09
0.5ms
0,202s
15.6G
100.%
25:08
0.5ms
0.168s
14.5G
92.5%
HiRED (MAT225)
30:08
0.6ms
0.210s
15.7G
96.4%
25:03
0.6ms
0.168s
14.5G
92.7%
SparseVLM (TCH
40:51
0.6ms
0.251s
15.8G
97.3%
31:28
0.6ms
0.212s
14.6G
92.3%
HoloV (Ours)
31:02
0.5ms
0.208s
15.6G
99.7%
27:36
0.5ms
0.176s
14.5G
95.7%
Besides, on video understanding bench- marks, HoloV maintains close to the origi- nal performance, significantly outperforming FasterVLM and FastV, shown in Table 2. This demonstrates the value HoloV when it comes to high-resolution visual input.
Table 2: Video QA Evaluations of different methods with 50% of visual tokens retained. HoloV beats SOTA.
Methods
MSVD-QA Score
MSRVT-QA
Avgerge
Acc.
Score
Cilco-ChutGPT
64.9
3.3
49.3
2.8
57.1
Video-LLaVA 7B
70.2
3.9
57.3
3.5
63.8
FastV (roz044)
71.0
3.9
55.0
3.5
63.0
3.7
FasterVLM (200v25)
70.5
3.9
56.2
63.4
DART
71.0
4.0
56.7
3.6
58.0
3.7
5.4 Efficiency Analysis
HoloV (Ous)
71.0
4.0
56.5
3.6
63.7
3.7
To assess the efficiency of HoloV, compare total inference time, prefill time, end-to-end latency, GPU memory usage, and accuracy on LLaVA-1.5-77 1.5-7 As shown in Tab, 4, under 90% pruning ratio, HoloV achieves a 42.7% reduction in inference time and 42.8% decrease in latency, with only a 4.33% drop in accuracy, similarly under 66.7% pruning ratio. Compared to FastV and Sparse HoloV uses less memory and runs faster. Although FasterVLM offers slightly quicker inference, HoloV improves accuracy by 3.0%, demonstrating better balance between efficiency and performance.
5.5 Ablation Analysis of Crop Numbers
Partition granularity does not affect pruning eff- ciency: retained visual tokens are determined by prun- ing quotas, and the quota crop, i.e., calculated dynamically via intra-crop visual token informative- ness, leaves total pruning quotas unchanged. For high-resolution images, dynamic crop number adjust- ment is beneficial: using fewer crops for high-detail areas and more for low-detail regions. Specifically, Table 5 shows results when total crops vary from to 16, where the values represent percentages relative original performance. We observe significant performance impact from varying crop numbers.
Table 5: Ablation of different crop numbers.
Methods
#=4
#=8
#= 12
#=16
Upper Bound
100%
100%
100%
100%
LLaVA-I.5-7B
Token Pruming Rate 66.7%
HoloV (Ours)
95.1%
96.7%
96.1%
94.9%
LLaVA-I.5-7B
Token Pruning Rate 77.8%
HoloV (Ours)
94.5%
95.1% %
94.6%
94.8%
L.LVA-
Token Praning Rate 98.9%
HoloV (Oun)
89.3%
89.3%
90.0% 91.2% %

---

Original image
Prane 50%
Prune 20%
Prune 87.5%
Original image
Prane -50%
Prune 200%
Prune 87.5%
Holay
Taxty
HoloV
Figure 10: The case comparison between FastV and HoloV from the GQA. It presents original images alongside their pruned versions at pruning rates of 50%, 70%, and 87.5%. The bounding boxes highlight specific regions and objects across images, where HoloV well preserves the pivotal tokens.
5.6 Visualization Analysis
Further, we visualize retained visual patches under different pruning rates. As shown in Fig. 10, black areas indicate discarded tokens, while colored regions show key semantic areas aligned with text. Compared to Fast HoloV preserves more relevant visual cues even under high pruning (e.g.. 87.5%), effectively filtering out redundant visual tokens while keeping critical objects. This supports better cross-modal alignment, allowing pivotal holistic tokens for visual overall understanding.
5.7 HoloV with Qwen Architecture
To verify the architectural generaliza- tion of HoloV beyond LLaVA-based models, we conduct experiments on the Qwen2.5-VL-7B [7] architecture. As
Table 6: Comparative Experiments Qwen2.5-VIL-775
Methods
MMB
MME
POPE
SQA
VQA
Avg.
Upper Bound
82.8
2304
86.1
84.7
84.8
100%
Qwen2,5-VL-7B
Token Pruming Rate 66.7%
shown in Tab. 6, HoloV demonstrates strong generalization capability across this architecture, consistently outper- forming the text-visual attention- -based FastV at various reduction ratios, high- lighting its robustness and adaptability to different model designs. Notably, it achieves average performance retention
FastV (192224)
75.7
2072
82.2
78.5
77.9
92.3%
HoloV (Ous)
78.3
2093
85.0
79.8
78.9
94.6%
Qwen2,5-VL-7B
Token Praming Rate
77.8%
FastV (ESCV21)
2036
80.7
78.0
69.0
89.2%
HoloV (Ours)
76.5
2043
82.3
79.8
70.3
92.7 %
Qwen2.5-VL-7B
Token Praming Rate
88.9%
FastV (F30924)
69.2
1940
78.6
77.4
60.3
84.3%
HoloV (Ours)
80.7
79.5
61.8
90.5%
rates of 94.6%, 92.7%, and 90.5% at 66.7% 77.8%, and 88.9% token pruning rates respectively, significantly higher than FastV's 92.3%, 89. 29, and 84.3% performance. These results show that our proposed holistic pruning strategy effectively generalizes across different MLLM architectures.
Conclusion
We present HoloV, a holistic token pruning framework that addresses two critical limitations of attention-based visual compression: 1) semantic fragmentation from pver-prnnin non-salient regions, and 2) static importance estimation ignoring token interdependencies. The core innovation lies in variance-modee modulated dynamic scoring and capacity-constie -onsttrained allocation, which preserve holistic context. Extensive experiments validate our method's effectiveness in maintaining both perceptual details and abstract spatial reasoning capabilities under aggressive token reduction.
A Acknowledgments and Disclosure of F Funding
This work was supported by the National Natural Science Foundation of China (Grant No. 68.62506318); Guangdong Provincial Department of Education Project (Grant 2222220KKXOOO CAAI-Ant Group Research Fund; Scientific Research Projects for the Higher-educational Institutions (Grant No.2024312096 Education Bureau of Guangzhou Municipality: Guangzhou- (Kt.ooouUUUUSTGGG) Joint Funding Program (Grant No. 2025A03333)))77) Education Burcau Guangzhou Municipality.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almcida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, ct al. Gpt-4 technical report. arXiv preprint 23Xtv:2303308774 2023.
[2] Yael Adini, Dov Sagi, and Misha Tsodyks. Context-enabled learning in the human visual system. Nature, 415(6873);790-79, 2002. 2
[3] lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 02523776-22736,, 2022. 32
[4] Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopoulos, Hans Vandicrendonck, Deepu John, and Bo Ji. Hired: Attention Attention-guided token dropping for efficient inference of high-rcsolution vision-language models in resource-constralle
environments. arXiv preprint arXiv:2408, 10945, 2024. 2, 8, 26
[5] Vaswani Ashish, Attention is all you need. Advances in neural information processing systems, 30:I,
2017. 4
[6] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: vl: A frontier large vision-languen model with versatile abilities. arXiv preprint 23000 12966, 2023. 3
[7] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Yc, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint 1rXiv:2502.13923, 2025.
[8] Louis Béthune, Thibaut Boissin, Mathieu Serrurier, Franck Mamalet, Corentin Friedrich, and Alberto Gonzalez. Sanz. Pay attention to your loss: understanding misconceptioos about lipschitz neural networks. Advances in Neural Information Processing Systems, 35:20077-20091, 2022. 6
[9] Jeffrey Bigham, Chandrika Jayant, Hanjie Ji. Greg Little, Andrew Miller, Robert Miller. Robin Miller. Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333-342, 2010. 7, 25
[10] Daniel Bolya, Cheng-Yane Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint wxXiv:2200099400 2022. 3
[11] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer. and Judy Hoffman. Token merging: Your ViT but faster. In International Conference Learning Representutions, 2023. 3, 26
[12] Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Coria,, and Rita Cucchiara. The revolution of multimodal large language models: A survey. In Findings of the Association for Computational Linguisties: ACL 2024, pages 13590-13611 13618, 2024.
[13] Liang Chen, Haozhe 7hao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. European Conference on Computer Vision, pages 35, 2024. 1, 2, 3, 26. 27
[14] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large
models with better captions. In European Conference on Computer Vision, pages 770-387.30 Springer, 2024.
[15] Tri Dao. FlashAttention-2: 2: Faster attention with better parallelism and work partitioning. International Conference on Learning Representations (ICLR), 2024.
[16] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35:1 15:16444- 16359, 2635. 3
[17) Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. G Heigold, S Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
11

---

[18] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:240..2788. 2024.
[19] Mark Endo, Xiaohan Wang, and Serena Yeung-Levy, Feather throttle: Revisiting visual token pruning for vision- -aiiuala model acceleration. arXiv preprint arXiv:4412.13100 2024.
[20] Zhanzhou Feng and Shiliang Zhang. Efficient vision transformer via token merger. IEEE Transactions on Image Processing, 32:4156 41696,441166660 2023. 2
[21] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A comprehensive evaluation benchmark for multimodal large language models. MXXi:2330613394 2023. 5, 5,7,25
[22] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key- value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1484-5495, 2021. 32
[23] Yash Goyal, Tejas Khot, Douglas Summers Stmmers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913, 2017. 4, 7,2
[24] Jiaxian Guo, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10867-10877, 2023.
[25] Yuhang Han, Xuyang Liu, Pengxiang Ding, Donglin Wang. Honggang Chen, Qingsen Yan, and Siteng Huang. Rethinking token reduction milms: Towards unified paradigm for training-free frre accelcration. arXiv preprint arXiv:241 1..441...778,,, 2024. 2,3
[26] Yefci Hc, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large lision-language models with dynamic token sparsification and kv cache compression. arXiv preprint arXiv:2410.08584 2024.
[27] Kai Huang, Hao Zou, Ye Xi, BoChen Wang. Zhen Xie, and Liang Yu. Ivip: Instruction-guider visual token pruning for large vision-language models. In European Conference on Computer Vision, pages 214-230. Springer, 2024."
[28) Lei Huang. Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
[29] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination multi-modal large language models via over-trust penalty and catrospectionnallocaaee Vision and Pattern Recognition, pages 13418-13427. 2024. 3
In Proceedings of the IEEE/CVE Conference on Computer
[30] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6700-6700 2019. 25
[31] Lei Jiang. Weizhe Huang. Tongxuan Liu, Yuting Zeng, Jing Li, Lechao Cheng, and Xiaohua Xu. Fopru: Focal pruning for efficient large vision-language models. arXiv preprint arXiv:2411.14164 2024.
[32] Yifan Jiang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, Jay Pujara, et al. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning, Advances in Neural Information Processing Systems, 37:46567 46592, 2024. 3
[33] Shibo Jie, Yehui Tang, Ning Ding, Zhi-Hong Deng. Kai Han, and Yunhe Wang. Memory-space visual prompting for efficient vision-language fine-tuning. Forty-first International Conference on Machine Learning, 2024. 32
[34] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700- 137100111000000000 2024.
12

---

[35] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinnv.. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36:21487-21506, 2023.
[36] Jiayi Kuang. Ying Shen, Jingyou Xie, Haohao Luo, Zhe Xu, Ronghao Li, Yinghui Li, Xianfeng Cheng. Xika Lin, and Yu Han. Natural language understanding and inference with milm in visual question answering: A survey. ACM Computing Surveys, 57(8))1-36, 2025.
[37] Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 2024. 3
[38] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Java-next-imterllave Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024.
[39] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- -mnaage--mmgeeae pre- pre-training for unified rision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMIR, 2022. 32
[40] Yanwei Li. Chengyao Wang, and Jiaya Jia. LLaMA-VID: An image is worth 2 tokens large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
3
[41] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. / 18814, 2024. 3,
[42] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision- laiinnla models. arXiv:2305. 10355, 2023. 5, 2,
[43] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023. 1, 26
[44] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311 10122, 2023. 3
[45] Hanxiao Liu, Andy Brock, Karen Simonyan, and Quoc Le.. Evolving caamallzation-actiiatiii layers. Advances in Neural Information Processing Systems, 33:13539-13550, 2020. 32
[46] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296 66296-26306, 2024. 3
[47] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 3, 5,26
[48] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 26
[49] Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint 1rXiv:2411.10800 2024. 1, 3, 8, 26
[50] Xiang Liu, Zhenheng Tang. Hong Chen, Peijie Dong. Zeyu Li, Xiuze Zhou, Bo Li, Xuming Hu, and Xiaowen Chu. Can llss maintain fundamental abilities under kv cache compression? arXiv preprint arXiv:2502.0194/
2025. 32
[51] Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, and Xiaowen Chu. Chunkkv:
kv cache compression for efficient long-context Ilm inference. arXiv preprint arXiv:2502.0029/ 2025. 32
[52] Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, and Honggang Chen. Compression with global guidance: Towards training-free high-resolution milmss acceleration. arXiv preprini arXiv:2501.0512 2025. 8, 26
[53] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He. Ziwei Liu, et al. Mmbench: Is your modal model an all-around player? In European Conference Computer Vision, pages 216-233, Springer, 2025. 5, 7, 25, 27
13

---

[54] Pan Lu. Swaroop Mishra. Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord. Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances Neural Information Prucessing Systems, 35:2507-2521 2022. 25
[55] Gen Luo. Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: of adaptation for multimodal large language models. arXiv preprini arX 2024.
[56] Yulin Luo, Ruichuan An, Bocheng Zou, Yiming Tang, Jiaming Liu, and Shanghang Zhang. Lim as dataset analyst: Subpopulation structure discovery with large language model. In European Conference on Computer Vision, pages 235-252. Springer, 2025.
[57] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 12585 126022, 2024. 1
[58] Junzhu Mao, Yang Shen, Jinyang Guo, Yazhou Yao, and Xiansheng Hua. Efficient token compression for vision transformer with spatial information preserved. arXiv preprint arXiv:2503.23455 2025. 1
[59] Junzhu Mao, Yang Shen, Jinyang Guo, Yazhou Yao, Xiansheng Hua, and Hengtao Shen. Prune and merge: Efficient token compression for vision transformer with spatial information preserved. IEEE Transactions on Multimedia, 2025.
[60] Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision- models. arXiv preprint urXi::2410.77144 2024. 2
[61] Thao Nguyen. Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 96:22047-22065 2023. 1
[62] Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 277444, 2022. 3
[63] Marius V Peelen, Li Fei-Fei, and Sabine Kastner. Neural mechanisms of rapid natural scene categorization in human visual cortex. Nature, 460(7251);94 97, 2009. 2
[64] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: Lime-sensitvve multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313-14323, 2024.
[65] Michael Ryoo, AJ Piergiovanni, Anurag Amab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: Adaptive space-time tokenization for videos. Advances in neural information processing systems. 34:12786-12799 2021.3 3
[66] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint NVV:24033113880 2024. 2. 3, 8,
[67] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA models that read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019. 5, 7.22
[68] Dingjie Song, Wenjun Wang. Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A simple yet effective token reduction method for efficient multi-modal 1lms. arXiv preprint arXiv:2409.10994 2024. 3
[69] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean -san-Baptistt Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 18805, 2023.
[70] Simon Thorpe. Denis Fize, and Catherine Marlot. Speed of processing in the human visual system. nature,
381 (6582))225 522, 1996. 2
[71] Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024. 3
[72) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 20X11:2302.13971, 2023.
14

---

[73] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei. Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.0928: 2023.
[74] Joel A Tropp. Greed is good: Algorithmic results for sparse approximation. IEEE Transactions on Information theory, 50(10):2231-2242, 2004. 32
[75] Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, and Panpan Xu. VI-cache: Sparsity and modality aware kv cache compression for vision-language model inference acceleration. arXiv preprint crXiv:2410.23317, 2024.
[76] Shakti N Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello. The evolution of multimodal model architectures. arXiv preprint arXiv:2405.1799 2024. 4
[77] Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [cls] token tells everything needed for training-free free efficient milms, arXiv preprint arXiv:2412.05879, 2024. 2,
[78] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-languee language model's perception of the world any resolution. arXiv preprint 240::2449.1119111 2024. 1
[79] Yi Wang, Kunehang Li, Xinhao Li, Jiashuo Yu, Yinan Hc, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Intervvido2:: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pages 396-416. Springer, 2024.
[80] Zichen Wen. Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem? arXiv preprint arXiv:2502. 115001 2025. 2
[81] Zichen Wen, Yifeng Gao, Shaobo Wang. Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplication matters more. arXiv preprint arXiv:2502. 1494, 2025. 2, 8, 26
[82] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247-2255 IEEE, 2023.
[83] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Can, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 2024. 3, 8, 26
[84] Bingxin Xu, Yuzhang Shang. Yunhao Ge. Qian Lou, and Yan Yan. freepruner: A training free approach for large multimodal model acceleration. arXiv preprint arXiv:2411. 2024. 3
[85] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan Hc, and Yucting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the ACM international conference on Multimedia, pages 1645 1653, 2017. 8, 25, 26
[86] Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: Imm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403. 2024.
[87] Yibo Yan, Guangwei Xu, Xin Zou, Shuliang Liu, James Kwok, and Xuming Hu. Docpruner: A storage- efficient framework for multi-vector arXiv preprint arXiv:2509.288. 23883, 2025.
vector visual document retrieval via adaptive patch-level embedding pruning.
[88] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arxiv:2412.04467, 2024. 2,
8, 26
[89] To Yang. Jian Jia, Xiangyu Zhu, Weisong Zhao, Bo Wang, Yanhua Cheng, Yan Li, Shengyuan Liu, Quan Chen, Peng Jiang, et al. Enhancing instruction-follooinng capability of lisuua-language models by reducing image redundancy. arXiv preprint 1rXXiv22411.1444 2024. 3
[90] Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. DeCo: Decoupling token compression from semantic abstraction in multimodal large language models. arXiv:2405.20985, 2024. 3
[91] Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training- free visual token pruning for Intelligence, volume 39, pages 22128-22136, 2025. 3.
large language models. In Proceedings of the AAAI Conference on Artificial
15

---

[92] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Machine Leurning, 2024. 2,
[93] Qizhe Zhang. Aosong Cheng. Ming Lu, Zhiyong Zhuo. Minqi Wang. Jiajun Cao. Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention you need for training-fiee visual token pruning: Make vlm inference faster. arXiv preprint urXiv:2412.0158 2024. 2, 3, 3,4.8,2 26
[94] Renshan Zhang, Yibo Lyu, Rui Shao, Gongwei Chen, Weili Guan, and Liqiang Nie. Token-level invelation-guidens compression for efficient multimodal document understanding. arXiv preprint arXiv:2407.14439 2024. 2
[95] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artelxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, ct al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 010666 2022.
[96] Xiaofeng Zhang, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang. Chaochen Gu, Hao Tang, and Jieping Yc. From redundancy to relevance: Enhancing explainability in multimodal large language models. arXiv e-prints, pages arXiv-2406, 2024. 6
[97] Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high righlresolutton large multimodal models. arXiv preprint arXiv:2406. 08487. 2024.
[98] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohci Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsification for efficient vision-language model inference. arXiv preprint (tte:2440.04417,, 2024. 1, 2, 3, 8, 26
[99] Henry Hengyuan Zhao, Pan Zhou, Difei Gao, Zechen Bai, and Mike Zheng Shou. Lova3: Learning to visual question answering, asking and assessment. Advances in Neural Information Processing Systems, 37:115146-115175 2024.
[100] Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. arXiv preprint arXiv:2488.094 0994999999 2024.
[101] Yuke Zhu, Chi Xie, Shuang Liang, Bo Zheng, and Sheng Guo. Focusllava: A coarse-to to-rec-to-fine approach for efficient and effective visual token compression. arXiv preprint arXiv:2411.14228, 2024.
[102] Xin Zou, Chang Tang, Xiao Zheng, Zhenglai Li, Xiao He, Shan An, and Xinwang Liu. Dpnet: Dynamic poly-attention network for trustworthy multi-modal classification. In Proceedings of the 31st ACM international conference on multimedia, pages 3550-3559, 2023.
[103] Xin Zou, Yizhou Wang. Yibo Yan, Yuanhuiyi Lyu, Kening Zheng, Sirui Huang, Junkai Chen, Peijie Jiang, Jia Liu, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. Forty-second International Conference on Machine Learning (ICML), 2025. 2
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are clearly stated in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The discussion on the limitations of our work is stated in the paragraph E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Our work is motivated by an interesting experimental phenomenon and pro- poses methods based on this observation, which improves the baseline by a large margin. There are no assumptions and no following proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper includes the implementation details in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide the dataset URL and code URL full submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc lublic/guides/Cdeee wtw.ommssssoooooiiooococococoooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results, See the NeurIPS code and data submission guidelines (https: //ips.cccpublli... httppuucccuuuideeess/eppp Subee sotmmmsssii wwwsssolissgoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specific experiment settings in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We don't need to conduct such an evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specific experiment settings in Section 5.4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:///pppp.itttccccsee.ee..eea
Answer: [Yes]
Justification: We conducted the research in the paper conform, every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The discussion on both potential positive societal impacts and negative societal impacts is stated in Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper swi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments or research with human subjects, so no related details included.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The research described in the paper does not involve study participants or human subjects, thus questions regarding potential risks, disclosure, or IRB approvals are not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
22

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper does mention the usage of LLMs significant or original component of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standary components.
Please refer our LLM policy htttts::/neurpps.cc/////// / for what should or should not be described.
23

---

Contents of Technical Appendices
Detailed Experiment Settings
25
A.1 Benchmarks and Metrics
25
A.2 Backbones and Baselines
26
A.3 Reproducibility
27
More Sparsification Visualization
27
B.1 MMBench Finegrained Results
27
Theoretical Analysis HoloV
D Fast Visual Context Refetching
32
D.I Preliminary: Reformulation of FFN
32
D.2 FFN with Visual Context Refetching
32
D.3 Further Efficiency Analysis
33
Impact Statement
24

---

OC Technical Appendices and Supplements
In this appendix, we first provide the details of the experimental setup, including information about the datasets, model architectures, and comparison methods. Then, we offer a more detailed computational complexity and theoretical analysis, along with more visualizations and insights.
A Detailed Experiment Settings
A.1 Benchmarks and Metries
We conducted experiments on several widely used visual understanding benchmarks. For image understanding task, we performed experiments on ten widely used benchmarks, including GQA [30], MMBench (MMB) and MMB-CN [53], MME [21], POPE [42], VizWiz [9]. SQA (ScienceQA) [54],
VQAv2 (VQA V2) [23], VQATou (TextVQA) [67], and MMVet [92].
GQA [30] The GQA benchmark is composed of three main components: scene graphs, questions, and images. The image section encompasses not only the images themselves but also their spatial features and the attributes of all objects within the images. The questions in GQA are specifically crafted to assess the model's ability to comprehend visual scenes and engage in reasoning about different aspects of the images.
MMBench [53]. MMBench provides comprehensive evaluation of model's performance across multiple dimensions. It is structured into three levels of ability dimensions. The first level (L-1) focuses on two core abilities: perception and reasoning. Building on this foundation, the second level (L-2) includes six sub-abilities, further elaborating the model's capabilities. At the third level (L-3), the evaluation becomes more granular, encompassing 20 specific ability dimensions, thus ensuring a detailed and multi-faceted analysis of the model's ' performance.
The MME benchmark is another holistic evaluation framework, designed to thoroughly assess various facets of a model's performance. It includes 14 distinct subtasks, each targeting specific perceptual and cognitive abilities of the model. By employing carefully crafted instruction-answer pairs and maintaining concise instruction designs, the benchmark minimizes issues such as data leakage and unfair evaluation, ensuring fair and reliable performance assessment.
MME [21].
POPE [42]. POPE focuses on evaluating the degree of Object Hallucination in models. It reformulates hallucination evaluation by prompting the model with specific binary questions regarding the presence of objects in images. Key metrics such as Accuracy Recall, Precision, and F1 Score are utilized to measure the hallucination level across three different sampling strategies, providing a robust and precise evaluation of the model's 's object detection and hallucination behavior.
ScienceQA [54]. ScienceQA spans many domains, including natural sciences, language sciences, and social sciences. Questions are categorized within each domain according to topics, categories, and skills, which results in 26 topics, 127 categories, and 379 skills. This hierarchical categorization facilitates a thorough and diverse range of scientific questions, enabling an in-depth evaluation of the model's 's multimodal understanding, multi-step reasoning abilities, and interpretabilityy
VQA-V2 [23]. VQA-V2 designed to evaluate a model's visual perception capabilities through open-ended questions. It consists of 265,016 images representing a wide variety of real-world scenes and objects, providing rich visual contexts for the associated questions. Each question is accompanied by 10 ground truth answers provided by human annotators, enabling a comprehensive evaluation of the model's ability to answer questions accurately and effectively.
TextVQA TextVQA focuses on the integration of text within images, evaluating the model's ability to comprehend and reason about both the visual and textual information present. The bench- mark includes a series of visual pnestioo-answeren
tasks where the model must not only interpret the visual content but also read and understand the embedded text in order to respond correctly.
MMVet [92]. MM is designed to assess a model's s ability to solve complex tasks by leveraging various core vision- vision-language language capabilities. It defines six core vision-language capabilities and examines 16 distinct integrations of these capabilities. This allows for a nuanced evaluation of how well models integrate and utilize multiple vision-language abilities to solve tasks.
MSVD-QA [85]. The MSVD-QA benchmark derived from the Microsoft Research Video Descrip- tion (MSVD) dataset and consists of 1970 video clips paired with approximately 50.5K question-
25

---

answer pairs. The questions span a wide range of topics and aspects related to the video content, making it suitable for video avestion-answeringg and video captioning tasks. The questions fall into five categories: what, who, how, when, and where, providing comprehensive set of queries for model evaluation.
MSRVTT-Q QA [85]. MSRVTT-QA includes 10,000 video clips and 243,000 question-ansee pairs. One of its primary challenges lies understanding and reasoning about video content, which involves both visual and temporal aspects. To answer questions accurately, models must effectively integrate and process these components. Similar MSVD-QA, the tasks in MSRVTT-QA are categorized into five question types: what, who, how, when, and where, allowing for detailed performance evaluation across multiple dimensions.
A.2 Backbones and Baselines
Models. We evaluate HoloV using various open-source MILMs, For image understanding tasks, experiments conducted on the LLaVA family, including LLaVA-1.5 [48] and LLaVA-NeXT [47], with the latter used to validate performance on high-resolution images. For video understanding tasks, use Video-LLa VA [43] the baseline model. Following the settings reported in their paper.
We analyze multiple representative methods for accelerating MILM inference through visual token pruning. These methods share the goal of improving efficiency by reducing redundant visual tokens.
ToMe [11] merges similar tokens visual transformer layers through lightweight matching tech- niques, achieving acceleration without requiring additional training.
LLaVA LLaVA-PruMerge [66] combines pruning and merging strategies by dynamically removing less important tokens using sparse CLS-visual attention and clustering retained tokens based on key similarity.
V [13] focuses on early-stage token pruning by leveraging attention maps, effectively reducing computational overhead in the initial layers.
HiRED [4] allocates token budgets across image partitions based on CLS token attention, followed by the selection of the most informative tokens within each partition, ensuring spatially aware token reduction.
PDrop [83] adopts a progressive token-dropping strategy across model stages, forming a pyramid-like token structure that balances efficiency and performance.
Faster VLM [93] evaluates token importance via CLS attention in the encoder and performs pruning before interaction with the language model, streamlining the overall process.
MustDrop [49] integrates multiple strategies, including spatial merging, text-guided pruning, and output-aware cache policies, to reduce tokens across various stages.
GlobalCom" 2 [52] introduces a hierarchical approach by coordinating thumbnail tokens to allocate retention ratios for high-resolution crops while preserving local details.
VparseVLM [98] ranks token importance using cross-modal attention and introduces adaptive sparsity ratios, complemented by novel token recycling mechanism.
VisionZip [88] evaluates token importance via attention in the encoder and clustering retained tokens based on key similarity.
DART [81] introduces duplication-awar token reduction method that selects a small subset of pivot tokens, calculates cosine similarity between pivot tokens and remaining tokens, retains those with the lowest duplication to pivots, achieving significant acceleration while maintaining performance and good compatibility with efficient attention operators. These methods collectively highlight diverse approaches to token reduction, ranging from attention based pruning to adaptive merging, offering complementary solutions for accelerating MLLMs.
26

---

Table Fine- -eane-gained comparison MMBench [53] between FastV and HoloV at high pruning ratios.
Vanilla Tokens)
FastV 90%
HoloV 90% (58 Tokens)
FastV J. 75%
HoloV 75%
Category (dev)
Tokens)
(144 Tokens)
(144 Tokens)
Action Recognition
90.7
85.2
85.3
87.0
89.7
Attribute Comparison
50.0
50.0
53.9
52.3
48.7
Attribute Recognition
79.7
68.9
71.7
77.0
79.7
Celebrity Recognition
79.8
74.7
78.8
78.8
Function Reasoning
75.9
72.2
83.9
75.9
83.9
Future Prediction
45.0
30.0
58.3
40.0
58.3
Identity Reasoning
93.3
86.7
97.5
95.6
97.7
Image Emotion
78.0
58.0
68.7
78.0
76.0
Image Quality
35.8
22.6
38,8
28.3
40.1
Image Scene
96.2
90.4
91.5
96.2
97.1
Image Style
77.4
73.6
71.7
77.4
77.4
Image Topic
83.3
80.6
92.9
83.3
83.3
Nature Relation
41.7
39.6
49.4
37.5
37.5
Object Localization
39.5
35.8
23.3
37.0
38.3
OCR
$9.0
59.0
84.4
Physical Property Reasoning
50.7
60.3
49.3
53.3
58.0
Physical Relation
33.3
41.7
32.7
41.7
41.7
Social Relation
88.4
53.5
75.8
72.1
75.7
Spatial Relationship
17.8
17.8
18.5
17.8
18.5
Structured Image- Text Understanding
26.9
30.8
21.8
28.2
21.9
A.3
Reproducibility
Implementaion Details. All of our experiments are conducted on Nvidia A800-80G GPU. The implementation carried out in Python 3.10, utilizing PyTorch 2.1.2, and CUDA 11.8. All baseline settings follow the original paper. We set numerop = /0024/N], where N denotes the number of retained visual tokens, thus the smaller the quota, the more crops there will be for visual holistic context retention.
B More Sparsification Visualization
We conduct a detailed visualization retained visual patches across varying pruning rates to illustrate the effectiveness of HoloV. As depicted in Fig. 11, 12, 13, the black regions represent discarded visual tokens, whereas the colored areas highlight key semantic zones that align with textual descriptions, demonstrating how HoloV strategically preserves informative content. Compared pruning method, HoloV exhibits superior capability in extremely high pruning ratios, such 87.5%. This is achieved through its holistic pruning strategy, which prioritizes spatial-semantic diversity over isolated attention scores. By dynamically allocating pruning budgets across different image crops, HoloV effectively filters out redundant tokens while safeguarding critical objects and their contextual relationships. For instance, in complex scenes with multiple interacting elements, HoloV ensures that tokens corresponding to both focal objects and their surrounding environmental cues are preserved, whereas Fast'V tends to over-concentrate high-attention regions, leading to loss of contextual coherence. This enhanced preservation of visual holistic understanding facilitates more accurate cross-modal aligmment between visual features and language tokens, enabling MLLMs to maintain robust semantic reasoning capabilities even under aggressive token reduction. The visualization not only validates the superiority of HoloV's design philosophy but also provides empirical evidence of its ability to balance efficiency and semantic integrity in visual token pruning.
to FastV, a representative retaining relevant visual cues
B.1 MMBench Finegrained Results
As shown in Table 7, in the MMBench [53] fine- fine-grained comparison between FastV [13] and HoloV at 90% and 75% pruning ratios, significant performance improvements are evident with HoloV V in several categories. Specifically, HoloV shows enhanced outcomes Action Recognition, Attribute Recognition, Future Prediction, Identity Reasoning, Image Emotion, Image Quality, and Image Scene. These results underline HoloV's ability retain crucial visual information for complex understanding and response capabilities within dynamic environments.
27

---

Original image
Prune 25%
Prune 55%
Prune =0%
Prune =87.5%
FastV
HoloV
FastV
HoloV
FastV
HoloV
Fast
HoloV
FastV
HoloV
Figure 11: The case comparison between FastV and HoloV from the GQA. It presents original images alongside their pruned versions at pruning rates of 25%. 50%, 70%, and 87.5%. The bounding boxes highlight specific regions and objects across images, where HoloV well preserves the pivotal tokens.
28

---

Original image
Prune =25%
Prune 50%
Prune -0%
Prune =87.5%
FastV
HoloV
FastV
HoloV
FastV
HoloV
FastV
HoloV
FastV
HoloV
Figure 12: The case comparison between F and HoloV from the GQA. It presents original images alongside their pruned versions at pruning rates of 25% 50%, 70%, and 87.5%. The bounding boxes highlight specific regions and objects across images, where HoloV well preserves the pivotal tokens.
29

---

Original image
Prune =25%
Prune 50%
Prune -0%
Prune =87.5%
HoloV
FastV
HoloV
FastV
HoloV
FastV
HoloV
FastV
HoloV
Figure 13: The case comparison between F and HoloV from the GQA. It presents original images alongside their pruned versions at pruning rates of 25% 50% 70%, and 87.5%. The bounding boxes highlight specific regions and objects across images, where HoloV well preserves the pivotal tokens.
30

---

C Theoretical Analysis of HoloV
To further justify the trustworthiness of our proposed HoloV, we provide theoretical analysis of it.
Assumption (Contextual Stability) Let Ny be the original visual tokens set, and Ry Ny the retained visual tokens subset, We assume the following:
(C1). For any pruned visual token 2 E Ne Rw, there exists 2, Ry with:
d));; 2 e and (ar(d();,NNz;)) 5
where d means distance function like cosine similarity, (x;) denotes xz 's local context neighbors.
(C2). For H(x,) -V(x;;) + A(x;) satisfies H(xx)) 2 for all retained tokens E R.
Lemma C.1 (Token Coverage Guarantee) Under (A1), for any pruned token F there exists R. such that:
|| - 2(1 -)11,,) + v8
Proof C.1 From the cosine similarity bound, there have x. æ; 2 < e||xx|l|<=g)) Using the variance constraint:
Bi I - 5 VIR (Nzz)
where J = B[rr zx). Combining via the triangle inequality:
||,, x = ||=;|| 2 || || 2x, zj
< 2B² - 22 + V6
2 vo
The lemma shows that pruned tokens can be approximated by retained tokens in Euclidean space.
Theorem C.1 (Semantic Preservation) Let be a transformer layer with Lipschitz constant For input embeddings X₂ and pruned set R., satisfying (CI)-(C2):
Il((x)) - S(R))l T (2 + v8 (B,,7)
where m1B,r) = ( (B² /n) is the residual error from the scoring threshold.
Proof C.2 Decompose the error into three components: 1) Geometric distortion: Bounded by Lemma C.1 2) Context variance: Controlled by Vo 3) Scoring residual:
For any x E X, \ RR with S(x:) < <;
ve + A <7= V(x) < A(x)
Using Cauchy-Schwarz inequality:
n S 1 Σ ||Wv
CB²
aRR.
Combining terms via the triangle inequality completes the proof.
This theorem guarantees that, even after pruning, the semantic difference between the outputs of the transformer for the original.
Corollary (Dynamic Allocation Optimality) The token allocation in Section achieves:
kp
max (R
log
Spt
S.L.
Naaget
t=1
with approximation ratio - 1/e when using greedy selection.
31

---

Proof C.3 The allocation problem is equivalent to maximizing a monotone submodular function.
Greedy algorithms provide (1 - approxaapoiinntio guarantees [74] such problems.
This corollary shows that your token allocation strategy is not only efficient but also theoretically near-optimal.
This theoretical framework demonstrates that HoloV: 1) Preserves semantic relationships through bounded geometric distortion. 2) Context variance is controlled via stability-aware pruning. 3) Token allocation is provably near-optimal, balancing efficiency and effectiveness.
D Fast Visual Context Refetching
D.I Preliminary: Reformulation of FFN
Vanilla FFN comprises two fully connected layers with non-linear activation in between. We suppose a E R2 as an input token of the FFN, and FFN function can be formulated as
FFN o (x W
(7)
where is activation function like ReLU SiLU [45], and W,, W2 Rd are the weight matrices, in usual D = 4d. Peculiarly, W and Wa 2 can be rewritten as
W = (k1, le......k.pp........ W₂ (w), 00,,002... U2,...... UD):
(8)
where k o, E Rd denote entries of key and value, respectively. As a result, the FFN can be reformulated as
FFN(#) = Σ Φ ((x, k;))
Thus, the FFN function can be construed using input query measure similarity with keys, find matching values, and gather values by similarity, which works like a key-value memory storing the factual knowledge as found in previous studies (22, 33].
D.2 FFN with Visual Context Refetching
We propose visual context refetching (VCR), i.e., reinjecting pruned visual information into the middle layer of the text decoder during elevated uncertainty during reasoning. This strategy treats pruned visual tokens as anchors to recalibrate off-target predictions and reduces uncertainties in object, attribute, relationship tokens. The reason call this pattern of reinjecting visual evidence VCR is that the model finds and refreshes key visual memories based on the hidden states in this process. In particular, inspired by the fact that FFN executes analogous retrieval from its key-value memory, we consider VCR to serve as a simplified and efficient information re-retrieval process. Given a hidden token æ Idd and dimension-aligma vision tokens z.. FFN with visual context refetching at (-th layer can be written as follows
FFN (xc 2.) aA (1 a:) (FN (z).
(10)
where 2, (..,.l...,;,,,,,, N.. RdxN. 2,, denotes execute VCR A from to visual features z. and a E [0, 1] denotes injection ratio of visual memory through the FFN layer which proportional to image complexity. Specifically, instead of performing retrieval via cross-attention layers as in previous approaches [39, 3, 102], we consider a simple retrieval process for VCR as,
No. - Ex 1=1 ((x2, Zo.i)) Zu,i.
A ②))
(11)
From the perspective of FFN, VCR works by treating value entries (visual evidence) to supplement vision-related information in the hidden states. In this information re-retrieval process, MemVCR does introduce any parameters that need be trained. Notably, since the size of key-value memory D in FFN typically far exceeds the number of visual tokens N, (for instance, D 11008 in ILaMA-7B and N, 256 for ViT-L/14, N. & D), the computation of VCR is negligible. Thus, VCR operation is more efficient than the cross-attention mechanism with quadratic complexity [50, 51].
query, and (Zu,z Ze,i) new key-
32

---

D.3 Further Efficiency Analysis
As shown in Fig. 14, we conduct efficiency evaluation on LLaVA- NeXT 7B at 95% pruning ratio, where we also introduce baseline (unpruned Vanilla) and FastV (95% pruned) for comparison. We evaluate these approaches using QA pairs from GQA, and the output length has been set to During evaluation, an A800 80GB GPU has been used, and the average FLOPs, VMemory usage and throughput has been calculated, shown Fig. 14. HoloV reduces over 90% of FLOPs requirement, 37% lower than FastV, and its VMemmryy usage at the lowest level, while keeping throughput at 5.2 per second, 2.16x and 1.13x faster than baseline and FastV, respectively.
PTVanilla Fastv Holov
30
20
17
10
6.1
3.8
4.6.2
0
FLOis(T) Momory( Throughput
Figure 14: Inference effi- ciency comparison between FastV and HoloV,
E Impact Statement
This paper presents HoloV, a visual token pruning framework for MLLMs, and discusses its potential societal impacts. On the positive side, HoloV enhances the accessibility of multimodal technologies by reducing computational overhead, making advanced applications like medical image analysis and autonomous driving more feasible in resource-constrained environments such edge devices or underserved regions, Its efficiency also contributes to energy sustainability by lowering the energy consumption of MLLM inference, aligning with global efforts mitigate the environmental impact of AI. Additionally, by preserving holistic visual context instead of relying solely on attention-base "highlighted tokens," HoloV may reduce biases in model outputs, improving fairness in diverse scenarios like visual reasoning involving underrepresented communities. The framework's 's plug- and-play design further accelerates its integration into real-world systems, driving innovations in education, accessibility tools, and emergency response to enhance societal resilience.
However, the work also carries potential negative implications. The reduced computational barriers enabled by HoloV could facilitate misuse, such as the creation of deepfakes or misinformation, particularly regions with limited regulatory oversight. While aiming to mitigate attention-based biases, the framework's 's crop-wise token allocation might inadvertently reinforce other biases if training data lacks diversity, potentially disadvantaging underrepresented groups. Morcover, the focus on inference efficiency might lead developers to prioritize speed over model interpretability. raising concerns about accountability in "black-box deployments for high-stakes tasks like healthcare diagnostics. Lastly, over-reliance on post-hoc pruning could deter investments in more equitable training data or architectural improvements, potentially accumulating technical debt and masking foundational issues in MLLM development.
Limitations and Future Work. HoloV demonstrates robust performance in preserving holistic visual context but faces two key limitations: its dependence on fixed spatial crop partitioning may hinder fine-grained semantic capture in complex scenes, and minor accuracy declines persist even at high pruning ratios (c.g., 4.2% drop when pruning 88. 88.9% visual tokens). To address these, future work could prioritize adaptive crop, sparse attention, multi-modality extensions (e.g., 3D data), and integration with hallucination mitigation, while optimizing for edge computing energy efficiency.
33


Reviews:
[{"review_id": "dgLTrqc63Q", "reviewer": "Reviewer_MYQs", "summary": "This paper proposes a new visual token reduction strategy to improve the speed of MLLMs with maintained performance. Compared with prior works that directly reduces tokens with low attention values, the proposed method adaptively distributes the token reduction across spatial crops. This strategy can capture more global visual context, tackling the limitation from prior work that keeps too many semantically similar tokens in the same spatial region. Benchmark results show that the proposed method demonstrates superior efficiency-accuracy trade-off, outperforming the prior SOTA method on various benchmarks.", "questions": "See the weaknesses section.", "limitations": "The authors have shown some reasonable limitations in the `Limitations and Future Work` section", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The authors observed that current token pruning method only focuses on the attention value, which could lose the informative global semantics and pay too much attention on the local salient tokens. Thus, this paper proposes a new method to allocate tokens more evenly in a global view, but still consider the local saliency factor. The overall design is reasonable.\n2. The benchmark results show that the proposed method achieves SOTA performance on almost all benchmarks over the prior methods. In addition, there is no significant computation/latency overhead of the current method.\n3. The paper is relatively easy to understand.\n\nWeaknesses\n1. Although the overall idea is relatively easy to understand, some concepts in the paper may need revision:\nLine 51 & Line 65: There is a concept called \"semantic connectivity\". However, it seems there is no evidence showing such \"connectivity\" between semantics. The goal of the method seems more like to capture more global semantics rather than local features. \nLine 150: \"Attention dispersion\" seems a concept that a small amount of tokens take the majority of the attention. Under this concept, [CLS] attention is steeper in the curve, shows more aligned with the concept of \"attention dispersion\", but Line 157 - 159 seems to show that last-token attention and equi last attention are more aligned with such concept?\nLine 164 - 165: It introduces another concept \"global contextual cohesion\" which is only mentioned once and never used again. Seems this concept is not necessary.\n\n2. Line 134 - 135 \"As shown in Fig. 4 left, adjacent tokens with similar visual features frequently receive comparable attention scores\" It seems difficult to reach such conclusion from Fig. 4 left. Please elaborate more on it.\n\n3. Line 131 \"we focus on the attention received by visual tokens from the visual [CLS] token\" Seems there is a limitation of the only focus on attention from the CLS token. I wonder if the mutual attention between visual token needs to be considered.\n\n4. Sec 4.1 Issues in the formula (1) need to show dimension for the matrices and vectors (2) eq.2 should have $\\sum_j$ instead of $\\sum$ (3) $\\mathcal{H}^c$ is never used (4) $\\Omega_c$ is not well explained (5) In eq. 5, why minimizing global diversity $\\mathcal{V}^c$? Do we want similar patterns in each crop?\n\n5. Typo: Table 3 Avgerge  -> Average", "was_revised": true, "final_justification": "After reading the authors' rebuttal, I think most of my concerns have been addressed. I would like to update my rating to borderline accept.", "rebuttal": "We sincerely thank Reviewer MYQs for the constructive comments on our work. We promise to revise the paper.\n\n---\n\n> Q1: some concepts in the paper may need revision.\n﻿\n- Line 51 & Line 65: “Semantic connectivity”\n﻿\nThe concept of  “semantic connectivity” means the cross-region interactions in semantic space, e.g., A on/under/in B, such object relations, retaining only \"highlighted tokens\" would lose this connectivity. We have clarified in the revision paper.\n﻿\nTo further substantiate this concept, we have added to Figure 3 heatmaps in the revision paper for two representative images (pedestrian and vehicle), showing pairwise attention strengths between adjacent and distant visual tokens.\n﻿\n- Line 150 & Line 157-159: Inconsistency in “attention dispersion” presentation.\n﻿\nSorry for miswriting, \"attention dispersion\" is a counter concept that more visual tokens take the majority of the attention. Under this concept, compared to [CLS] attention (the top 20% of visual tokens account for 80% of the total attention), text-vision attention tends to be dispersed over more visual tokens, e.g., the top 20% of visual tokens account for only 40% of the total attention, consistent with \"attention dispersion\". We have revised this mistake in the revision.\n﻿\n- Line 164 - 165: the concept of \"global contextual cohesion\" is not necessary.neng\n﻿\nThanks for your nice suggestion. We agree, our initial idea means not over-reliance on attention scores, as it disrupts spatial semantic relationships, considering global semantics could mitigate this issue. We have revised this concept in the revision. \n\n---\n\n> Q2: Line 134 - 135 \"As shown in Fig. 4 left, adjacent tokens with similar visual features frequently receive comparable attention scores\" It seems difficult to reach such conclusion from Fig. 4 left. Please elaborate more on it.\n\nTake FastV's distribution map of visual token attention as an example, many tokens in regions characterized by flat backgrounds, such as sky, grass, and trees with similar visual features, receive comparable attention scores. When using the top-k strategy to select retained visual tokens, it leads to semantic redundancy, i.e., retaining many visual tokens with semantics of the same objects.\n﻿\nFurther, we added more cases and a statistical analysis to elaborate on this redundancy in the revised Appendix.\n\n---\n\n> Q3: Line 131 \"we focus on the attention received by visual tokens from the visual [CLS] token\" Seems there is a limitation of the only focus on attention from the CLS token. I wonder if the mutual attention between visual token needs to be considered.\n﻿\nWe clarified that our primary goal was to study how [CLS] aggregates global visual information, while mutual attention between visual tokens was included in self-attention at ViT, and we don't need this mutual attention between visual tokens to select key visual tokens.\n\n---\n\n> Q4: Sec 4.1 Issues in the formula (1) need to show dimension for the matrices and vectors (2) eq.2 should have $\\sum_j$ instead of $\\sum$ (3) $\\mathcal{H}^c$ is never used (4) $\\Omega_c$ is not well explained (5) In eq. 5, why minimizing global diversity $\\mathcal{V}^c$? Do we want similar patterns in each crop?\n\nR(1), we actually have listed dimension of $\\mathbf{Z}_v^c \\in \\mathbb{R}^{M\\times d}$, further we follow your comment to show clear dimension, in the formula (1) $\\mathbf{S}^c = (\\mathbf{1}-\\mathbf{I}_M)\\odot\\mathbf{Z}_v^c {\\mathbf{Z}_v^c}^\\top$, where $\\mathbf{Z}_v^c {\\mathbf{Z}_v^c}^\\top \\in \\mathbb{R}^{M\\times M}$, and $(\\mathbf{1}-\\mathbf{I}_M) \\in \\mathbb{R}^{M\\times M}$, thus $\\mathbf{S}^c  \\in \\mathbb{R}^{M\\times M}$.\n\nR(2), thanks for your kind suggestion, we have revised from $\\sum$ to $\\sum_j$.\n\nR(3), $\\mathcal{H}^c=\\gamma_c \\mathcal{V}^c + \\mathcal{A}^c$ indicates our holistic attention that is calculated through a balanced scoring mechanism combining contextual diversity and attention saliency, in the code we actually use it to calculate the crop importance weights in formula (4), as there is a writting mistake, we have revised it to $w_c = \\frac{(\\frac{1}{M}\\sum_{t=1}^M \\mathcal{H}_t^c)^\\tau}{\\sum^{\\mathcal{C}} (\\frac{1}{M} \\hat{\\mathcal{H}}^{c'})^\\tau}$, \n\nwhere $\\hat{\\mathcal{H}}^{c'}=\\sum_{t=1}^M \\mathcal{H}_{t}^{c'}$.\n\nR(4), $\\Omega_c$ means the objective has a quota constrain, i.e., $|\\Omega_c| = q_c$\n\nR(5), $\\mathcal{V}_i^c$ indicates that $i$-th token has diverse connections with others, our objective is to retain those with rich semantics tokens within a crop. To avoid ambiguity, we revised formula (5) to more clearly express the idea of the paper and conform to the actual implementation of the code.\n\n---\n\n> Q5: Typo: Table 3 Avgerge -> Average\n\nWe have revised our paper accordingly, the typo \"Avgerge\" is revised to \"Average\" in Table 3.\n\nWe sincerely appreciate your valuable suggestions again."}, {"review_id": "p9IS0y58b1", "reviewer": "Reviewer_4HaK", "summary": "This paper introduces HoloV, a plug-and-play visual token pruning framework for MLLMs that moves beyond attention-based token selection by preserving holistic visual context. Unlike previous methods that focus on high-attention tokens, HoloV adaptively allocates pruning budgets across spatial crops, balancing local saliency with global semantic diversity. The method significantly reduces visual tokens (up to 88.9%) while retaining up to 96% of the original performance, improving inference efficiency across multiple benchmarks and models.", "questions": "### Question:\n1. I would like to know whether HoloV can still achieve consistent strong performance on models using native resolution (e.g., Qwen2.5-VL). Adding such results would allow HoloV to cover all mainstream VLM visual encoding forms, further enhancing the persuasiveness of the results.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength:\n1. The paper is detailed and clear, with thorough experiments, visualizations, and theoretical analysis that make the claims convincing.\n2. Experiments on rich benchmarks with various retained token numbers show promising improvements, demonstrating the effectiveness of the proposed method.\n\n### Weakness:\n1. The evenly partitioned crops may not be flexible enough and could introduce potential spatial bias.", "was_revised": true, "final_justification": "My question has been answered, and the author has provided more comprehensive results to respond. I'll keep my score to support their work.", "rebuttal": "We sincerely thank Reviewer 4HaK for the constructive comments on our work. We are very grateful to the reviewer for recognising the novelty of our idea and the richness and rationality of our experiments.\n\n---\n\n> Q1: The evenly partitioned crops may not be flexible enough and could introduce potential spatial bias.\n\nThanks. Although crops are partitioned evenly, the quota of a crop is dynamically allocated, thus, this would not affect flexible allocation based on the density or complexity of the content in the image, which could reduce fragmentation and improve model performance in more intricate scenes. \n\nTo further show the advantages of our method, we further present our results on MM-Vet benchmark as follows,\n\n| Method | Retain 192 tokens | Retain 128 tokens |\tRetain 64 tokens |\n| :-----: |  :-----: |  :-----: |  :-----: | \n|*Vanilla (576 tokens)*\t|*31.0*|*31.0*\t|*31.0*|\n|FastV\t|28.9\t|26.7\t|26.1|\n|SparseVLM\t|31.5\t|30.0\t|23.3|\n|VisionZip\t|31.7\t|32.6\t|30.2|\n|HoloV (ours)\t|32.9\t|32.0\t|30.2|\n\n---\n\n> Q2: I would like to know whether HoloV can still achieve consistent strong performance on models using native resolution (e.g., Qwen2.5-VL). Adding such results would allow HoloV to cover all mainstream VLM visual encoding forms, further enhancing the persuasiveness of the results.\n\nWe agree with the importance of having HoloV cover all major VLMs, and based on your suggestion we have additionally included experiments with Qwen2.5-VL, and the results on multiple benchmarks demonstrate HoloV's effectiveness.\n\n|Method\t|MME|\tPOPE|\tMMB-EN|\tSQA|\tAverage|\n| :-----: |  :-----: |  :-----: |  :-----: |  :-----: |  :-----: | \n|Qwen2.5-VL-7B\t|2304\t|86.1\t|82.8\t|84.7\t|100%|\n|FastV (prune 66.7%)\t|2072\t|82.2\t|75.7\t|78.5\t|92.37%|\n|HoloV (prune 66.7%)\t|2093\t|85\t|78.3\t|79.8\t|94.58%|\n|FastV (prune 77.8%)\t|2036\t|80.7\t|74.9\t|78\t|91.16%|\n|HoloV (prune 77.8%)\t|2043\t|82.3\t|76.5\t|79.8\t|92.71%|\n|FastV (prune 88.9%)\t|1940\t|78.6\t|69.2\t|77.4\t|87.96%|\n|HoloV (prune 88.9%)\t|2006\t|80.7\t|72.4\t|79.5\t|90.52%|\n\nThese results demonstrate that HoloV achieves superior performance."}, {"review_id": "0QuI2rohzW", "reviewer": "Reviewer_h9EW", "summary": "This paper presents HoloV, a novel visual token pruning framework for MLLMs, which addresses the critical limitation of attention-based methods that prioritize local saliency over global semantic context. By introducing crop-wise adaptive allocation and a hybrid scoring mechanism combining attention saliency and semantic variance, HoloV ensures holistic context retention during aggressive pruning.", "questions": "Please see the Major points listed under Weaknesses. \nHow does the granularity of crop partitioning (e.g., number of crops) affect pruning efficiency? For high-resolution images (e.g., LLaVA-NeXT's 2,880 tokens), should the number of crops be dynamically adjusted?", "limitations": "When an LLM has severe language bias, could HoloV's visual pruning indirectly exacerbate hallucinations? For example, if the language model misjudges an object's existence, might HoloV retain irrelevant visual tokens that reinforce this error?", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. This paper provides a solid motivation for addressing visual token pruning in MLLMs. Traditional attention-based approaches tend to fragment global semantics by overly focusing on locally emphasized tokens.  By incorporating insights from cognitive science and analyzing Positional Bias, the paper highlights the root causes of performance degradation in complex scenes. \n2. HoloV shows exceptional control over performance degradation, retaining 95.8% of the original performance after pruning 88.9% of visual tokens.\n3.The paper is well-organized and easy to follow.\n\n**Weaknesses:**\n\n1. The fixed spatial crop partitioning method limits the system's applicability in complex scenes where semantic regions may be non-rectangular or cross boundaries. This can result in fragmented semantic units, reducing the model’s generalizability.\n2. There is also a sensitivity to hyperparameters like the sharpness factor (τ) and scaling factor (γ), which require manual tuning. The lack of a unified optimization strategy for varying resolutions or scene complexities increases deployment costs and limits real-world applicability.\n3. While HoloV mitigates Positional Bias, it still struggles in extreme conditions such as severe lighting, occlusions, or fast-moving objects. These challenges lead to potential information loss, as seen in MSRVTT-QA experiments where pruning errors affected dynamic object understanding.", "was_revised": true, "final_justification": "The authors has addressed most of my concerns. While some minor reservations remain, I believe the paper's contribution is significant and meets the bar for acceptance. Therefore, I have raised my score to Accept.", "rebuttal": "We sincerely thank Reviewer h9EW for the constructive comments on our work. Below, we will address each of your questions and concerns.\n\n---\n\n> Q1: The fixed spatial crop partitioning method limits the system's applicability in complex scenes where semantic regions may be non-rectangular or cross boundaries. This can result in fragmented semantic units, reducing the model’s generalizability.\n\nActually, the quota of a crop is dynamically allocated, thus, this would not affect flexible allocation based on the density or complexity of the content in the image, which could reduce fragmentation and improve model performance in more intricate scenes.  \n\n---\n\n> Q2: There is also a sensitivity to hyperparameters like the sharpness factor (τ) and scaling factor (γ), which require manual tuning. The lack of a unified optimization strategy for varying resolutions or scene complexities increases deployment costs and limits real-world applicability.\n\nWe acknowledge that *τ* (sharpness factor) requires manual setting, but in practice, these hyperparameters do not have a significant effect on the results. For example, the sharpness factor (τ) is always set to 1. For $\\gamma_c$ (scaling factor), it is determined by $\\mathbb{E}[\\|\\mathcal{A}^c\\|]/\\mathbb{E}[\\|\\mathcal{V}^c\\|]$.\n\n---\n\n> Q3: While HoloV mitigates Positional Bias, it still struggles in extreme conditions such as severe lighting, occlusions, or fast-moving objects. These challenges lead to potential information loss, as seen in MSRVTT-QA experiments where pruning errors affected dynamic object understanding.\n\nThank you for your recognition of our work. We agree that extreme conditions like severe lighting, occlusions, and fast-moving objects pose significant challenges for HoloV and can lead to information loss. In response, we are considering temporal pruning, i.e., tracking **dynamic objects** across frames, pruning **static duplicates**, to better capture dynamic changes and reduce pruning errors in such conditions. This strategy would improve the robustness of HoloV in challenging scenarios and dynamic object understanding, where traditional image models may struggle to capture the full range of information.\n\n---\n\n> Q4: How does the granularity of crop partitioning (e.g., number of crops) affect pruning efficiency? For high-resolution images (e.g., LLaVA-NeXT's 2,880 tokens), should the number of crops be dynamically adjusted?\n\nIn fact, the granularity of pruning partitions does not affect pruning efficiency, as the retained visual markers are determined by pruning quotas, and the quota in a crop is calculated by the informativeness of intra-crop visual tokens, which is dynamic, the total pruning quotas would not be changed. For high-resolution images, we agree that adjusting the number of crops dynamically could be beneficial. For instance, using a lower number of crops for highly detailed areas (such as objects or regions of interest) and a higher number of crops for less detailed regions (such as background areas) could improve pruning performance.\n\nFurther, we show performance when we vary the total number of crops from 4 to 16, and we observe that varying the number of crops does not have a significant effect on performance.\n\n|Patch_num\t|4\t|8\t|12\t|16|\n| :---: |  :-----: |  :-----: |  :-----: |  :-----: | \n|*Vanilla (576 tokens)*\t|*100%*\t|*100%*\t|*100%*\t|*100%*\t|\n|Retain 64 tokens\t|89.3%\t|89.3%\t|90.0%\t|91.2%|\n|Retain 128 tokens\t|94.5%\t|95.1%\t|94.6%\t|94.8%|\n|Retain 192 tokens\t|95.1%\t|96.7%\t|96.1%\t|94.9%|\n\n*where the numbers indicate the percentage of performance relative to the raw performance.*\n\n---\n\n> Q5: When an LLM has severe language bias, could HoloV's visual pruning indirectly exacerbate hallucinations? For example, if the language model misjudges an object's existence, might HoloV retain irrelevant visual tokens that reinforce this error?\n\nWe recognize that when an LLM has severe language bias (e.g., in tasks involving object recognition or reasoning), it may hallucinate or misjudge the existence of certain objects. If this occurs, visual pruning could indeed exacerbate the issue by retaining irrelevant visual tokens. However, in the experiments, especially in POPE, which is a hallucination-related benchmark, compared with SOTA methods, our HoloV achieves the best performance, as shown in the table.\n\n|Method\t|Retain 192 tokens\t|Retain 128 tokens\t|Retain 64 tokens|\n| :---: | :---: | :---: | :---: |\n|Vanilla (576 tokens)\t|85.9\t|85.9\t|85.9|\n|ToMe\t|72.4\t|62.8\t|52.5|\n|FastV\t|64.8\t|59.6\t|48.0|\n|LLaVA-PruMerge\t|71.3\t|67.2\t|65.3|\n|PDrop\t|82.3\t|82.3\t|55.9|\n|FiCoCo-V\t|82.5\t|82.2\t|76.0|\n|MustDrop\t|82.6\t|78.7\t|68.0|\n|HiRED\t|82.8\t|79.8\t|73.6|\n|SparseVLM\t|83.6\t|80.5\t|75.1|\n|HoloV (ours)\t|85.6\t|84.0\t|80.3|\n\nThe results demonstrate that HoloV's visual pruning has minimal impact on exacerbating hallucinations.\n\nIn the future, we aim to explore cross-modal consistency checks, where visual tokens are only retained if they align with the language model's predictions. If the LLM's prediction is highly uncertain, we would retain more visual tokens associated with that object, potentially reducing hallucinations.\n\nWe appreciate the reviewer’s thoughtful feedback on these critical aspects of HoloV. We look forward to continuing to refine HoloV for real-world applicability."}, {"review_id": "JSSUIxLTks", "reviewer": "Reviewer_cqzx", "summary": "This paper introduces HoloV, a new framework designed to improve the efficiency of multimodal large language models by intelligently pruning visual tokens. The authors observe that most existing token pruning methods focus too heavily on attention scores, often retaining only locally highlighted or similar tokens. This can lead to significant information loss and reduced performance, especially when a large proportion of visual tokens are pruned.\n\nHoloV addresses this issue by adopting a holistic approach to token selection. Instead of simply keeping the most salient tokens based on attention, the framework divides the image into spatial crops and adaptively selects tokens from different regions. This ensures that both global context and diverse local details are preserved, even when aggressive pruning is applied. The method balances local saliency with overall scene understanding, aiming to maintain important semantic information that would otherwise be lost.", "questions": "1. Can you provide detailed information on how spatial crops, crop numbers, and related hyperparameters are selected? Additionally, could you include ablation studies that show how these choices impact the reproducibility and performance of your approach?\n\n2. What is the rationale behind your specific top-k token selection score? How does this formulation compare empirically or theoretically to alternative selection strategies or scoring methods?\n\n3. Have you evaluated the sensitivity and robustness of your adaptive holistic token allocation procedure, including the effects of discretization and the sharpness parameter? Could you discuss or provide results comparing your allocation approach to other possible methods?\n\n4. Would it be possible to use HoloV as a replacement for the Q-Former module in models such as InstructBLIP, or to integrate HoloV within the Q-Former family of architectures? If so, could you discuss any practical considerations, challenges, or potential benefits of applying HoloV in this context? It would be helpful to understand whether HoloV is compatible with Q-Former-based models and how its holistic token pruning strategy might interact with or improve these architectures.\n\n5. Could you report practical efficiency metrics such as multiply-accumulate operations (MACs), wall-clock latency, and memory usage on different hardware platforms, including less powerful or older devices and different devices (GPUs with different floating point operations)? It would be helpful to see evidence of real-world efficiency beyond theoretical FLOPs reductions.\n\n6. Can you present empirical results or ablation studies that isolate and quantify the actual contribution of visual context refetching (VCR) in your framework? In particular, how is the injection ratio chosen, and what impact does VCR have on uncertainty reduction or downstream performance?\n\n7. Do you have results or ablation studies demonstrating how HoloV performs with a wider variety of visual encoders, such as SigLIP, DINOv2, CLIP-ResNet,S2, Mamba, ConvNeXt, etc? Would you consider evaluating whether your method generalizes to models beyond the LLaVA family?", "limitations": "See questions and weakness.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strenghts\n* The paper presents comprehensive experiments across a wide range of benchmarks and MLLM architectures, demonstrating consistent and significant improvements over state-of-the-art token pruning methods.\n* The paper provides a well-articulated motivation for holistic context retention and offers insightful analysis of the limitations of prior methods, supported by visualizations and ablation studies.\n* HoloV is easy to integrate, offering substantial computational efficiency gains with minimal performance loss.\n\n### Weaknesses\n* Key implementation details, such as: how spatial crops are defined, the choice of hyperparameters, and ablation on the crop number or allocation sharpness, are insufficiently specified. This limits reproducibility and undermines the credibility of the claimed results.\n* The process for selecting the top-k visual tokens within each crop is largely based on a straightforward score maximization, but the underlying criteria for combining local saliency and global relevance remain rather heuristic and are not thoroughly justified or compared to alternative strategies. There is no empirical evidence or ablation provided to show how sensitive the method is to the exact formulation of the selection score, or whether more sophisticated selection strategies could lead to further improvements.\n* The adaptive holistic token allocation approach in HoloV is largely heuristic and lacks principled justification. The process involves averaging token scores within each crop, discretizing quotas, and then using iterative reallocation to handle rounding issues or quota overflow. While this ensures that each region of the image is represented, the method relies on several arbitrary choices, such as the setting of the sharpness parameter $\\tau$ and the specific reallocation rules. The paper does not provide any analysis or ablation to show how sensitive the method is to these choices, nor does it offer a theoretical rationale for why this allocation procedure is optimal. As a result, the effectiveness and robustness of this adaptive allocation are uncertain, and it remains unclear whether the improvements are due to this specific mechanism or would be matched by simpler or more principled alternatives.\n* While the paper provides a theoretical analysis of computational complexity using FLOPs, it does not sufficiently address how pruning affects actual performance on real hardware. In practice, efficiency gains from pruning are highly dependent on multiply-accumulate operations (MACs), memory access patterns, and especially system latency, which can vary significantly depending on the hardware architecture. On older or resource-constrained devices, aggressive pruning may actually introduce irregular computation and reduce parallelism, potentially resulting in longer inference times or even hardware underutilization. The paper does not empirically explore how HoloV’s pruning strategy impacts different hardware setups, nor does it report detailed results on wall-clock latency or MACs. For a complete and convincing assessment of practical efficiency, future work should explicitly measure and report latency and MACs across diverse hardware platforms, since improvements in FLOPs alone do not guarantee real-world speedup or energy savings.\n* I find the section on **visual context refetching (VCR)** to be interesting in theory, but I am concerned about the lack of concrete empirical evidence supporting its utility. While the mathematical description of how pruned visual tokens are reintegrated via the feed-forward network is clearly presented, the paper does not provide experiments or detailed analysis to demonstrate the real impact of this mechanism. Specifically, the authors do not explain how the injection ratio is selected in practice, and there is no data illustrating how VCR affects uncertainty reduction or downstream performance. Without a dedicated evaluation or ablation that isolates the effect of VCR, it is challenging to judge how important or reliable this component truly is within the broader HoloV framework.\n* While the paper provides results for both LLaVA-1.5 and LLaVA-NeXT visual encoders, the experimental analysis remains limited to this family of encoders. There is no investigation into how HoloV would perform with fundamentally different visual backbones, such as other types of vision transformers (Siglip2, Dinov2, etc..), convolutional encoders (CLIP-Resnet,ConvNeXt, EfficientNet, etc..), or others models (Mamba). Since different encoders may have unique tokenization strategies and varying levels of feature redundancy, it is unclear if HoloV’s advantages and efficiency–accuracy trade-offs would generalize beyond the LLaVA architecture. A broader ablation study across diverse encoder types would strengthen the paper and provide more confidence in the general applicability of the method.", "was_revised": true, "final_justification": "Given what was discussed during the rebuttal, some of the questions were resolved. I encourage the authors to edit the final document by incorporating all the suggestions presented, with special emphasis on the analysis of compute overhead.", "rebuttal": "We sincerely thank Reviewer cqzx for the insightful comments.\n\n---\n\n> Q1: Can you provide detailed information on how spatial crops, crop numbers, and related hyperparameters are selected? Additionally, could you include ablation studies that show how these choices impact the reproducibility and performance of your approach?\n\nWe show performance when we vary the total number of crops from 4 to 24, and we observe that varying the number of crops in the range of 4-16 does not have a significant effect on performance, but if the number of patches is too large (>20), it causes a decrease in performance.\n\n|Patch_num\t|4\t|8\t|12\t|16\t|20\t|24|\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|Retain 64 tokens\t|89.3%\t|89.3%|\t90.0%|\t91.2%|\t91.3%|\t90.2%|\n|Retain 128 tokens\t|94.5%\t|95.1%|\t94.6%|\t94.8%|\t95.6%|\t95.9%|\n|Retain 192 tokens\t|95.1%\t|96.7%|\t96.1%|\t94.9%|\t92.4%|\t91.3%|\n\n*where the numbers indicate the percentage of performance relative to the raw performance.*\n\nFor hyperparameters, we usually set allocation sharpness $\\tau=1$, and set the same value in all experiments.\n\n---\n\n> Q2: What is the rationale behind your specific top-k token selection score? How does this formulation compare empirically or theoretically to alternative selection strategies or scoring methods?\n\nOur objective is $\\mathcal{H}^c =  \\gamma_c \\mathcal{V}^c + \\mathcal{A}^c$. Following your suggestion, we divided the objective into local saliency $\\mathcal{V}$ and global relevance $\\mathcal{A}$, and we compare against pure local saliency, global relevance, and our HoloV. The results when retaining 64 tokens are as follows,\n\n| Strategy | MME | POPE | SQA | MMB-EN|\n| :---: |  :---: |  :---: |  :---: |  :---: | \n|local saliency|1604|68.9|65.3|60.1|\n|global relevance|1687|75.8|68.9|60.6|\n|HoloV|1715|80.3|69.5|63.3|\n\nThe results demonstrate that HoloV's strategy is reasonable. About whether more sophisticated selection strategies could lead to further improvements, we are researching a text-based approach to further improve the performance of HoloV.\n\n---\n\n> Q3: Have you evaluated the sensitivity and robustness of your adaptive holistic token allocation procedure, including the effects of discretization and the sharpness parameter? Could you discuss or provide results comparing your allocation approach to other possible methods?\n\nWhile it needs setting of the sharpness parameter, we set $\\tau=1$ in all experiments, and we also tried setting other values, and the results show that the model is not sensitive to this hyperparameter. For the effectiveness and robustness of this adaptive allocation, we vary the total number of crops from 4 to 16, and we observe that varying the number of crops does not have a significant effect on performance.\n\n|Patch_num\t|4\t|8\t|12\t|16\t|\n| :---: | :---: | :---: | :---: | :---: | \n|Retain 64 tokens\t|89.3%\t|89.3%|\t90.0%|\t91.2%|\n|Retain 128 tokens\t|94.5%\t|95.1%|\t94.6%|\t94.8%|\n|Retain 192 tokens\t|95.1%\t|96.7%|\t96.1%|\t94.9%|\n\n---\n\n> Q4: Would it be possible to use HoloV as a replacement for the Q-Former module in models such as InstructBLIP, or to integrate HoloV within the Q-Former family of architectures? If so, could you discuss any practical considerations, challenges, or potential benefits of applying HoloV in this context? It would be helpful to understand whether HoloV is compatible with Q-Former-based models and how its holistic token pruning strategy might interact with or improve these architectures.\n\nTo integrate HoloV within the Q-Former family of architectures, we conducted experiments on Qwen2.5-VL.\n\n|Method\t|MME|\tPOPE|\tMMB-EN|\tSQA|\tAverage|\n| :-----: |  :-----: |  :-----: |  :-----: |  :-----: |  :-----: | \n|Qwen2.5-VL-7B\t|2304\t|86.1\t|82.8\t|84.7\t|100%|\n|FastV (prune 66.7%)\t|2072\t|82.2\t|75.7\t|78.5\t|92.37%|\n|HoloV (prune 66.7%)\t|2093\t|85\t|78.3\t|79.8\t|94.58%|\n|FastV (prune 77.8%)\t|2036\t|80.7\t|74.9\t|78\t|91.16%|\n|HoloV (prune 77.8%)\t|2043\t|82.3\t|76.5\t|79.8\t|92.71%|\n|FastV (prune 88.9%)\t|1940\t|78.6\t|69.2\t|77.4\t|87.96%|\n|HoloV (prune 88.9%)\t|2006\t|80.7\t|72.4\t|79.5\t|90.52%|\n\nThese results demonstrate that HoloV achieves superior performance.\n\n---\n\n> Q5: Could you report practical efficiency metrics such as multiply-accumulate operations (MACs), wall-clock latency, and memory usage on different hardware platforms, including less powerful or older devices and different devices (GPUs with different floating point operations)? It would be helpful to see evidence of real-world efficiency beyond theoretical FLOPs reductions.\n\nWe reported  practical efficiency metrics on 4060Ti GPU (16GB), the results are as follows,\n\n|Method| Time| Prefill|Latency(s)|Memory| Average Peformance.|\n| :---:| :---:| :---:| :---:| :---:| :---:|\n|*Vanilla (576 tokens)*|49:41|0.5ms|0.334|19.0GB|100%|\n|FasterVLM (192 tokens)|30:09|0.6ms|0.202|15.6GB|98.5%|\n|FastV (192 tokens)|35:34|0.5ms|0.239|16.0GB|90.5%|\n|HiRED (192 tokens)|30:08|0.6ms|0.202|15.6GB|94.6%|\n|MustDrop (192 tokens)|32:30|0.5ms|0.218|15.6GB|97.2%|\n|SparseVLM  (192 tokens)|37:20|0.5ms|0.251|15.6GB|96.1%|\n|HoloV  (192 tokens)|31:02|0.6ms|0.208|15.6GB|99.2%|\n|FasterVLM (58 tokens)|25:08|0.6ms|0.168|14.5GB|92.5%|\n|FastV (58 tokens)|30:41|0.5ms|0.206|15.6GB|76.7%|\n|HiRED (58 tokens)|25:03|0.6ms|0.168|14.5GB|89.4%|\n|MustDrop (58 tokens)|29:40|0.5ms|0.199|14.5GB|90.1%|\n|SparseVLM  (58 tokens)|31:28|0.5ms|0.212|14.6GB|87.3%|\n|HoloV  (58 tokens)|28:28|0.5ms|0.176|14.5GB|95.7%|\n\n---\n\n> Q6: Can you present empirical results or ablation studies that isolate and quantify the actual contribution of visual context refetching (VCR) in your framework? In particular, how is the injection ratio chosen, and what impact does VCR have on uncertainty reduction or downstream performance?\n\nSure. We added experiments about the isolated contribution of visual context refetching (VCR) in our framework and under different injection ratios.\n\n| Strategy | MME | POPE | SQA | MMB-EN|\n| :---: |  :---: |  :---: |  :---: |  :---: | \n|HoloV w/o VCR|1700|78.3|69.2|62.0|\n|HoloV|1715|80.3|69.5|63.3|\n\n| Strategy | MME | POPE | SQA | MMB-EN|\n| :---: |  :---: |  :---: |  :---: |  :---: | \n|HoloV (0.05) |1702|78.5|69.2|62.3|\n|HoloV (0.10)|1708|79.9|69.3|62.8|\n|HoloV (0.15)|1715|80.3|69.5|63.3|\n|HoloV (0.20)|1711|80.3|69.4|63.0|\n|HoloV (0.25)|1708|80.1|69.5|63.2|\n|HoloV (0.30)|1705|79.7|69.3|62.9|\n\nFor the injection ratio, we often set it to 0.15, and this strategy can effectively contribute to the performance improvements.\n\n---\n\n> Q7: Do you have results or ablation studies demonstrating how HoloV performs with a wider variety of visual encoders, such as SigLIP, DINOv2, CLIP-ResNet,S2, Mamba, ConvNeXt, etc? Would you consider evaluating whether your method generalizes to models beyond the LLaVA family?\n\nWe have not conducted ablation studies with a wider variety of visual encoders as time is too short and this needs pretraining these visual encoders (most MLLMs' visual encoders are ViT framework), in the future we will take time to research such influence. Following your suggestion, we have additionally included experiments with Qwen2.5-VL (beyond the LLaVA family), and the results on multiple benchmarks demonstrate HoloV's effectiveness. (*Table in response to Q4*)\n\n---\n\n> Q8: Paper Formatting Concerns: Several tables and figures in the paper do not follow the official NeurIPS template and formatting guidelines. In particular, Figure 5, Figure 6, Figure 9, Table 3, and Table 4 stand out as not being consistent with the expected style. I recommend revising these figures and tables to conform with the NeurIPS template.\n\nWe have reformatted **all figures and tables** to match the NeurIPS style guidelines—single-column width for tables, consistent font sizes, correct caption placement, and vector‐format graphics."}]

Author rebuttal:

Meta-review: The paper presents a practical and well-substantiated approach to visual token pruning that improves high-ratio pruning robustness and real-world efficiency. The authors addressed most reviewer concerns with additional experiments and clarifications. Remaining issues (grid-induced fragmentation, heuristic justifications, wider encoder/hardware coverage) are important but do not undermine the main contribution and can be addressed in the camera-ready and future work.

Overall, reviewers converged on accepting or borderline-accepting after the rebuttal, with key conceptual and empirical concerns addressed.
Decision: Accept (poster)