Streamline Without Sacrifice - Squeeze out Computation Redundancy in LMM
Authors: Penghao Wu
Lewei Lu
Ziwei Liu
Venue: ICML 2025 poster (2025)
Keywords: Large Multimodal Model
Multimodal Large Language Model
Large Multimodal Model Acceleration
Primary area: deep_learning->large_language_models

Abstract: Large multimodal models excel in multimodal tasks but face significant computational challenges due to excessive visual tokens. Unlike token reduction methods that focus on token-level redundancy, we identify and study the computation-level redundancy on vision tokens to ensure no information loss. Our key insight is that vision tokens from the pretrained vision encoder do not necessarily require all the heavy operations (e.g., self-attention, FFNs) in decoder-only LMMs and could be processed more lightly with proper designs. We designed a series of experiments to discover and progressively squeeze out the vision-related computation redundancy. Based on our findings, we propose ProxyV, a novel approach that utilizes proxy vision tokens to alleviate the computational burden on original vision tokens. ProxyV enhances efficiency without compromising performance and can even yield notable performance gains in scenarios with more moderate efficiency improvements. Furthermore, the flexibility of ProxyV is demonstrated through its combination with token reduction methods to boost efficiency further.

Streamline Without Sacrifice - Squeeze out Computation Redundancy in LMM
Penghao Wu² Lewei Lu
Ziwei Liu'
Abstract
00.00
Large multimodal models excel in multimodal tasks but face significant computational chal- lenges due to excessive computation on visual tokens. Unlike token reduction methods that fo- cus on token-level redundancy, we identify and study the computation-lewe redundancy on vi- sion tokens to ensure no information loss. Our key insight is that vision tokens from the pretrained vision encoder do not necessarily require all the heavy operations (e.g., self-attention, FFNs) in decoder-only LMMs and could be processed more lightly with proper designs. We designed se- ries of experiments to discover and progressively squeeze out the vision-related computation re- findings, propose
Vanilla LMM
100%
100%
Pruxy//L12 L212
58 00.00-
ProxY/-L16
75%
80%
Perfommacc
58.00
50.
Compunaiion (%)
55 16
34%
-60%
54.
$4.00
32.00
20%
50.00
0%
Finengaaieed Avg Performance Prefilin TLOPs
Prefllig Time
Figure 1. ProxyV retains increases the fine-grained benchmark performance while effectively reducing the computational cost. ProxyV-L112 and ProxyV-L.16 denote applying ProxyV from layers 12 and 16, respectively.
dundancy. Based ProxyV, novel approach that utilizes proxy vi- sion tokens to alleviate the computational burden original vision tokens. ProxyV enhances effi- ciency without compromising performance and can even yield notable performance gains in sce- narios with more moderate efficiency improve- ments. Furthermore, the flexibility of ProxyV demonstrated through its combination with token reduction methods to boost efficiency further. The code will be made public here.
computation-level redundancy exists in the processing of vi- sion tokens and can be squeezed out through proper designs without sacrificing performance.
Current mainstream LMMs adopt typical decoder-only architecture, exemplified by LLaVA-style (Liu al., 2024b) pipelines, usually comprising: 1) A pretrained vision coder extracting vision features, 2) A lightweight projection module projecting vision features into LLMs' input space 3) An L.M jointly processing the concatenated vision and text tokens through multiple layers of self-attention and feed-forward networks (FFNs). Though simple and effec- tive, this structure faces significant challenge: the high computational cost incurred by vision tokens, which typi- cally far outnumber text tokens. Since self-attention scales quadratically with sequence length, the problem becomes even more severe as top-performing LMMs (I.i et al., 2024b; Wang et al., 2024; Chen et al., 2024b) usually process high- resolution images, resulting in thousands of tokens per im- age. Moreover, LMMs are now being extended process video frames (Lin al., 2023a; Maaz et al., 2024; Zhang et al., 2024a; Xue et al., 2024) or multiple images (Li et al., 2024c; Jiang et al., 2024a) within the same structure, further increasing the length of visual token sequences.
1. Introduction
Large multimodal models (LMMs) have demonstrated pow- erful capabilities by combining visual information with large language models (LLMs), but their computational overhead can be immense due to the large number of vision tokens. To mitigate this, most existing efforts address token-level redundancy by pruning or merging vision tokens with the risk of discarding fine-grained details. In this paper, we instead tackle the compufation-level redundancy on vi- sion tokens, ofven-overlooked dimension. We find that
'S-Lab, Nanyang Technological University "Sense Research. Correspondence to: Penghao Wu <peng- hao001 www.tt..u..ug........ Ziwci Liu <ziwei.liu@ edu.edu.sg>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A popular and intuitive solution to this problem to re- duce the number of vision tokens via pruning and merging

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
(Chen et al., 2025b; Shang et al., 2024; Zhang et al., 2024b; Xing et al., 2024; Yang al., 2024b). While effective in certain scenarios, this approach is problematic as it involves non-recoverable operations that risk discarding fine- tne-ggriied information or contextual details. For instance, in the case of a dense document image, pruning or merging tokens is highly likely to result in the loss of critical information. Sev- eral methods (Huang et al., 2025; Xing et al., 2024) use the textual question as guidance for the token pruning or selec- tion process. However, this approach can fail in multi-turn conversations, where later questions might require visual information not retained based on the guidance of the initial question. Additionally, it can be challenging to identify all critical visual information for complex or indirect questions. Furthermore, many token reduction methods rely on text- to-image attention scores, which makes them incompatible with time and memory-efficien attention implementations (Dao et al., 2022: Dao, 2023).
igate the negative impact of skipping all heavy operations vision tokens?
We then propose a better solution, ProxyV, that introduces group of proxy vision tokens to relieve the full vision tokens from the heavy computation burden. As shown in Figure I, ProxyV achieves 101 % performance with prefill- ing FLOPs and time reduced by 43% and 40% respectively, and achieves 102.4% performance with FLOPs and time reduced by 36% and 33% respectively on benchmarks re- quiring fine-grained visual understanding when applied to Vicunal.5-7B We compare ProxyV against token reduc- tion methods and highlight the information loss problem for them. Furthermore, we explore a non-spatial variant of Proxy which can be seamlessly integrated with token reduction methods to further enhance efficiency.
Overall, our contributions are threefold:
We systematically study the computation level redun- dancy on vision tokens in decoder-only LMMs and explore ways to progressively reduce it.
How can we reduce the computation cost brought by long vision sequences while always preserving all vision tokens to avoid any possible information loss? Note that the vision tokens for most LMMs come from a pretrained vision en- coder, so the vision tokens are already highly semantic. This raises the question: Is it still necessary to perform all the heavy operations (e.g., sision-to-visien attention and FFNs) on them within the LLM? Is there any computation-level redundancy on vision tokens in LMMs? Cross-attention- based LMMs (A (Alayrac et al., 2022: Laurencon et al., 2023; Dubey et al., 2024; Dai et al., 2024) already offer some promising insights into these questions. These methods treat vision features context, injecting them into the LLM via additional cross-attention modules, thereby avoiding unrolling all vision tokens in the LLM and improving com- putational efficiency. However, these models also have drawbacks such as requiring significantly more pre-training data, bringing a large number of additional parameters, and still slightly under-performing the decoder-only counter- parts. The computation-level redundancy on vision tokens is likely to exist, but can we reduce it while retaining the simplicity and efficiency of the decoder-only structure?
We propose ProxyV, a novel design that introduces proxy tokens to carry out heavy computations, effec- tively reducing computation while ensuring perfor- mance.
We extensively validate the effectiveness of ProxyV with different L.LMs and show its flexibility by propos- ing a non-spatial variant that can be directly combined with token reduction methods.
2. Discover Computation-level Redundancy
To validate our hypothesis regarding computation-level re- dundancy in decoder-only LMMs, we first design series of exploratory experiments to investigate the presence of such redundancy in self-attention operations among vision tokens. Specifically, we train a set of LMMs based on the LLaVA-Next (Liu et al., 2024a) structure with differ- ent LLM backbones, including Vicumal.5-7B (Zheng et al., 2023), Vicunal.5-133 (Zheng al., 2023), ILLama-8BB (Dubey et al., 2024), Qwen2-7B (Yang et al., 2024a), Phi3- 3B (Abdin et al., 2024), and InternLM2.5-78 (Cai et al., 2024). Detailed experimental settings are provided in Sec- tion 4. During inference, we masked the attention within vision tokens to disable inter-token interactions, applying this attention masking at different positions within the LLM (i.e., across various portions of the decoder layers). To evaluate the impact of this masking on the performance of LMMs, we select set of OCR-extensive benchmarks VDoc (Mathew et al., 2021), ChartQA (Masry et al., 2022), InfoVQA (Mathew et al., 2022), OCRBench (Liu et al., 2024c). TextVQA (Singh et al., 2019)) that require fine-grained visual information, thus being highly sensitive
Motivated by this, we first conduct experiments to verify the existence of computation-level redundancy. Our find- ings confirm that attention-related computation redundancy on vision tokens does exist, with different LLMs exhibiting varying degrees of redundancy. We then explore the possibil- ity of skipping both attention and FFN operations on vision tokens by replacing them with lightweight MLP modules. Interestingly, the newly added lightweight modules, which are vision-specific bring additional performance gains. And the final performance can be understood as the performance gain from the lightweight modules minus the performance drop caused by skipping attention and FFN operations vision tokens. But can we have better design to further mit-
2

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
stage in Table 1. Notably, even when vision attention is skipped across all layers, the reduction in FLOPs is limited. This is primarily due to the computationally intensive nature of the heavy FFN operations on vision tokens. This leads to another question: is it possible to also skip the heavy FFNs or replace them with lightweight alternatives?
100
Score _ine Relative (%)
LLMs
Vicura115-7B
80
Llama3-88
Vouur1155133
Table I. Results of finetuning LMMs with vision-to-al all attention skipped. TF denotes the training free approach by masking vision attention during inference and FT denotes finetuning with vision attention skipped. LO, L12, and L16 indicate the layer index where the masking skipping starts be applied. Finetuning can further improve the performance, but FLOPs reduction is limited.
Qwen2-73
PhiB-38
InterLM22 5-7B
1.0
0.8
0.6
0.4
0.2
0.0
Attention Masking Ratio
Scoreaie Rel
FLOPPs
Time
Figure The relative Scoregne with different vision attention masking ratios for different LLMs. The computation redundancy begins in the middle to rear part of the LLMs as masking the vision attention does not affect the performance.
L.- -O-TF
62.7%
LO-FT
87.2%
+ 18%
4 6.60
L12 TF
94.5%
99.3%
11%
4040%
L16 TF
100.2%
to potential information loss in vision tokens, and denote the average performance on them as Scoregne-
L16 FT
99.9%
4 1.9%
4 3.%
As shown in Figure 2, directly masking vision token atten- tion across the entire L.LM leads a significant performance drop, while masking it from the middle or later layers has minimal or effect on performance. Furthermore, while the general trend is consistent across all LLMs, different models exhibit different patterns, indicating varying degrees of redundancy. For instance, Vicumal.5-7B .B57B and Vicunal.5 1.5- 13B maintain full performance when the masking is applied from the middle layers, while Qwen2-7B achieves 100% per- formance when only the latter 25% layers are masked. The different patterns of vision attention across LLMs present an intriguing research direction, but we do not explore them in this paper and focus on the general trend. Based on these observations, we conclude that the atfention-related com- putation redundancy on vision tokens does exist in the middle and later layers of LMMs, with varying degrees of redundancy across different LLMs.
Table 2. Results replacing the original attentions and FFNs on vision tokens with lightweight MLPs. ATN represents the previous approach that only skips the attention, while ATN-FFN represents skipping both attention and FFNs. Skipping FFNs reduces the FLOPs, and adding lightweight MLPs brings performance gain.
Scoreane Rel
FLOPPs
Time
LO ATN
87.2%
4 18%
60%
LO- ATN+FEN
65.1%
$ 80%
77%
L12 - ATN
99.3%
+ 11%
40%
L12 ATN+FFN
99.3%
4 5,5%
4.%
L16 ATN
99.9%
4. 1.9%
32%
L16 ATN+FFN
100.4%
4 4.%%
1
Our initial attempt reveals that directly skipping FFNs and keeping vision tokens constant across decoder layers severely degrades the performance, we use lightweight MLPs to replace the attention operations and FFNs for the vision token update. The corresponding results, shown in Table 2, indicate that this approach significantly reduces FLOPs and further improves the speed. Compared to skipping attention operations alone, additionally skipping the FFNs degrades the performance for the layer-O case, achieves similar performance for the layer- 12 case, and even improves the performance for the layer- case. This interesting performance improvement arises because the addition of lightweight MLPs introduces decoupled vision- specific modules, which benefit overall performance. This finding also partially aligns with findings from Libra (Xu
Since the above experiments are directly conducted in the training-free free approach, natural question arises: Can finetune the model with the vision-to-de attention operations skipped to further reduce the performance gap? In practice, skipping the vision-to-all attention operation means that the vision tokens only act as the keys and values in the atten- tion operation and are updated directly with v-projection and o-projection, which is also the implementation stud- ied in EE-MLLM (Ma et al., 2024). We then conduct this experiment with the Vicunal .5-7B LLM and skip the vision- to-all attention from layers 0, 12, and 16, respectively. As shown in Table 1, finetuning with the vision-to-all atten- tion skipped mitigates the performance drop. We also report the reduced FL.OPs and time cost for the prefilling
3

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
al., 2024), which demonstrated that the decoupled vision- specific modeling can better process the vision-specific in- formation without distorting the original knowledge in the LLM. However, unlike Libra, which employs a substantial number of vision-specific parameters, we achieve this with lightweight MLPs (9.44M parameters per layer). The final performance can thus be understood as the performance gain from the newly added vision-specific parameters minus the performance drop caused by skipping the orig- inal heavy operations on vision tokens. Based on these observations, instead of this simple implementation, can we have a better design to eliminate performance loss further or even enhance performance while maintaining computa- tional efficiency?
could also include some local attention layer or convolution layer to further promote the fine-grained inter-token inter- actions in each local r window, and we leave this for future work. We validate the ProxyV design in the same setting as Section 2 and the results are shown in Table 3. With ProxyV, the negative effect of skipping attention and FFNs on full vision tokens is effectively mitigated, and now the performance gain brought by decoupled vision-specific modules becomes more pronounced, leading to better over- all performance at the layer 12 and 16 cases.
Table 3. Results of applying ProxyV from different layers. ProxyV effectively mitigates the performance drop caused by skipping operations on vision tokens and brings additional performance gain with vision-specific modules.
3. A Better Solution with Proxy Vision Tokens
Scoreßne Rel
FLOPs
Time
We now introduce our algorithm ProxyV, with its overall framework shown in Figure 3. The core idea of ProxyV is to employ a small group of proxy vision tokens as sub- stitutes for the original vision tokens in compute-intensiy operations. These proxy tokens then guide the updates of the original vision tokens through lightweight modules.
L0 ProxyV V L12 - ProxyV V L16 Prooyy
88.9%
73%
I 68%
101.0%
1 4,6
+ 4,41 %
102.4%
4.36%
311%
We further validate the effectiveness of our ProxyV algo- rithms with different LLM backbones in Table 4. The results indicate that applying ProxyV from the middle layers can achieve no performance loss or a small performance gain (100% 101%) with moderate efficiency improvement. Ap- plying from the middle and rear part of the LLM achieves notable performance improvement (101 % 102%) with a smaller efficiency gain. We provide the evaluation results on additional general LMM benchmarks in the Supplementary.
3.1. Proxy Tokens with Spatial Priors
As the original full vision tokens still preserve the 2D spatial structure, direct approach is downsample them into smaller vision feature map to serve as proxy tokens. Specif- ically, given N x N original vision tokens, we downsample them by a factor r to get a thumbnail version of size M x M where M- = N/r. In the LLM decoder layer, the proxy vi- sion tokens and text tokens serve as queries, while the values and keys consist of the proxy vision tokens, original full vision tokens, and text tokens. After the attention operation, only the proxy tokens and text tokens are processed by the FFNs. In this way, proxy vision tokens replace full vision tokens in the compute-intensive operations, significantly reducing computational cost. After the proxy tokens obtain useful information through these operations, each proxy token guides the spatially corresponding r x r full vision tokens for an update through a lightweight guided-update module. Through this design, the important information in the heavy computation of the decoder layer could be ef- fectively obtained and transferred to the full vision tokens with proxy vision tokens as an intermediary. As a result, the negative effect of skipping all extensive operations is effectively mitigated without sacrificing much efficiency.
We also conduct a preliminary study to reveal the possible internal mechanisms underlying the performance improve- ment. Specifically, we measured the MIR (Modality Inte- gration Rate) score (Huang et al., 2024), which quantifies the degree of alignment between visual and textual tokens within LMMs (lower scores indicate better alignment). We randomly sample 100 instances from the dense captioning dataset DetailCaps-4870 (Dong et al., 2024) and measure the MIR scores of the baseline and the variant where the vision-specifie MLPs replace the original vision operations from layer O. ProxyV with vision- speciiiic MLPs reduces the MIR scores from 3.62 to 3.10. This indicates that the setion-peccffic parameters achieve better alignment between text and vision tokens, leading to performance improvement. We leave further study about this direction as future work.
3.2. Comparison with Token Reduction Methods
In our implementation of the guided-update module, we first down-project the full and proxy vision tokens with linear layers and then directly concatenate the full vision tokens with their spatially corresponding proxy vision tokens and process them with a lightweight two-layer MLP to update the full vision tokens. Note that this guided module
Here we compare ProxyV with two state-of-the-e token reduction methods: VisionZip (Yang et al., 2024b) and Pyra- midDrop (Xing et al., 2024) with the (icunal.5-77 LLM setting. VisionZip performs token reduction before the LLM by selecting group of dominant vision tokens and merging
4

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
interpolate
Q Attention K & V
FFN
Q Attention
K & V
Proxy vision tokens init
Relieve full vision tokens from heavy ops
Original vision token
MLP
cat
Proxy vision token
FFN
repeat
Text token
Guided-update with spatial correspondence
Vanilla LMM
ProxyV
Figure 3. Left: the vanilla LMM structure where full vision tokens cause significant computation. Right: the overall pipeline of the proposed ProxyV algorithm. The full vision tokens are first downsampled to obtain much smaller version that works as proxy vision tokens. The proxy vision tokens participate in the original operations in the decoder layer including the self-attention and the FFNs to obtain useful information at much lower cost. After this, each original vision token guided by its spatially corresponding proxy vision token for an update through a lightweight MLP.
the remaining as contextual tokens. PrramidDrop progres- sively reduces the number of vision tokens inside the LLM,, utilizing the attention scores. We configure these two meth- ods so that the efficiency improvement similar to ours, and finetune the LMMs for fair comparison. We acknowledge that token-level redundancy does exist in general scenarios, but our goal is to ensure vision information loss for all cases even when the images contain very dense visual in- formation or require accurate visual grounding. Therefore, we also evaluate the models on a grounding benchmark Ref- COCO (Kazemzadeh et al., 2014) and additionally include the document parsing task to simulate this scenario. For the document parsing task, we continue to train the models the 1M document parsing data from DocStruct4M (Hu et al., 2024) dataset and evaluate them on the CCpdf (Turski et al., 2023) dateset in the validation split. We measure the BLEU (Papineni et al., 2002) and edit distance for this task.
3.3. A ProxyV Variant without Spatial Constraints
Though we directly compare ProxyV V with token reduction methods in the previous section, our goal is to diminish computation-level redundancy, which is theoretically or- thogonal the objective of token reduction methods that focus on removing token-level redundancy. This raises the question: Is it possible to combine ProxyV with these to- ken reduction methods? The primary challenge in directly combining the two approaches lies in the fact that ProxyV V relies on the 2D spatial structure of vision tokens for gen- crating proxy tokens and establishing correspondence in the mnideddupdate module. However, after applying token reduction methods, the spatial structure of the vision tokens is longer preserved, making integration nontrivial.
To resolve this problem, we propose non-spatial variant of the original ProxyV algorithm to remove the require- ment of a spatial prior so that this alternative can be flexibly combined with token reduction methods or non-spatial vi- sion features. Specifically, we initialize set of learnable embeddings Q mxd where m is the number of de- sired proxy tokens and d is hidden dimension typically set smaller than the hidden dimension of the LLM. A linear layer projects the full vision tokens to K E 1 where n is the total number full vision tokens. And we also define values V E gaxdniadoo to be directly equal to the full vision tokens, where hhidden is the dimension of the full vision tokens.
As shown in Table 5, VisionZip and Prrp achieve nearly performance drop selected benchmarks but have notable degradation and grounding benchmark and much worse performance on the document parsing task, highlighting the issue of visual information loss inherent to token reduction methods. We also provide qualitative examples in Figure to show the failure of token reduction methods where dense or structured visual information needs to be extracted or the image contains dense information and visual details, while V retains all visual information.
A vanilla attention operation is then applied on Q, K, and
5

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
Table The results of applying ProxyV different L.MMs. Applying ProxvV from the middle layers ensures no performance drop, and applying it from the middle to rear layers achieves notable performance improvement.
ScorCfime
ScorCfine Relative
FLOPs reduced
Time reduced
Vicuna-7B
Baseline
54.64
100.0%
ProxyV Layer 12
55.16
101.0%
46%
41 %
ProxyV Layer 16
55.94
102.4%
36%
31 %
Vicuna-13B
Baseline
58.58
100.0%
ProxyV Layer 16
58.95
100.6%
43%
40%
ProxyV Layer 20
59.22
101.1%
36%
33%
Llama3-8B
Baseline
56.60
100.0%
ProxyV Layer 16
56.87
100.5%
42%
34%
ProxyV Layer 20
57.49
101.6%
32%
25%
Qwen2-7B
Baseline
60.15
100.0%
ProxyV Layer 16
60.55
100.7%
37%
29%
ProxyV Layer 20
61.44
102.1% %
25%
14%
Phi3-3B
Baseline
49.89
100.0%
ProxyV Layer ProxyV Layer 20
50.28
100.8%
35%
34%
50.72
101.7%
27%
25%
InternL.M2.5-77
Baseline
58.33
100.0%
ProxyV Layer ProxyV Layer 20
58.68
100.6%
39%
33%
59.08
101.3%
30%
24%
V to get the attention logits (before softmax) A Rmxx
4. Experiment Details
As result, the proxy tokens are the direct weighted com-
For all experiments, we use the widely adopted 2-stage training pipeline. For stage 1, we pretrain the multi-modal projector and the newly added vision-specific modules using 1.2M captioning data from ShareGPT4V (Chen al., 2025a) for I epoch. For the finetuning stage, train the model for epoch using the 779K instruction tuning data in LLava- Next (Liu et al., 2024a) and unfreeze the L.M. in this stage.
bination of the full vision tokens softmax( A, dim -1)V. During the guided update process for the full vi- sion tokens, the previous ProxyV algorithm with spatial constraints mapped each full vision token to a proxy token based on spatial correspondence. In this non-spatial variant, we re-use the attention logits matrix A, transpose it, apply the softmax along the proxy token dimension, and multiply with the proxy tokens to get the guidance for all full vision tokens the weighted combination of proxy tokens. The process is illustrated in Figure 4.
For image encoding, we adopt the AnyRes strategy (Liu et al., 2024a) with a maximal grids per image, including the thumbnail one. Each grid with resolution x36 encoded by S----ViT---336- 6pxxx (Radford al., 2021) 24x24 image feature. The image feature is further projected by 2-layer MLP projector and flattened in raster order within each grid, and concatenated grid by grid, sim- ilar to the UniRes strategy (Zhang et al., 2024a). We also append one separator token after each grid.
First, we validate the feasibility of this non-spatial variant version of ProxyV. Subsequently, we combine it with Vi- sionZip to explore the possibility combining our approach with token reduction methods. As shown in Table 6, the non-spatial ProxyV variant attains similar performance the original one, and combining it with VisioZip achieves the desired performance with further increased efficiency.
For our ProxyV implementation, we choose the downsam- pling factor that 576 full vision tokens are com-
6

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
Table Comparison between ProxyV and token reduction methods. Token reduction methods significantly underperform on the document parsing task, indicating the inherent information loss problem.
Methods
Scorerine
RefCOCO
DocParsing-Ble BLEU f DocParsing-ED
FLOPs
Time
Baseline
54.64
82.96
0.7991
0.1551
ProxyV Layer 12
55.16
84.41
0.7923
0.1566
4 4.46
+ 41%%
VisionZip
54.58
79.11
0.7218
0,1734
1 32%
40%
ProamidDrop 54.63
80.75
0.7261
0.1702
1 42%
4.66
Table The results of the non-spatial ProxyV V variant (the ProxyV- entry) and the combination of non-spatial ProxyV and VisionZip (the Combination entry). The non-spatial version achieves simi- lar performance and combining with token reduction methods further boosts the efficiency while maintaining the performance.
Learnable Query
Linear
K
Full Vision Tokens
A
Scorefine
FLOPs
Time
Proxy vision tokens generation
V
Proxy Vision Tokens
Baseline
54.64
Proxy
VisionZip
54.58
$ 32%
I 40%
Vision Tokens
ProxyV L2yer12
55.16
$ 46%
41%
Guidance
ProxyV-ns Layer12
54.85
1 44%
44%
Combination
54.83
4 62%
655
cat
MLP
A Full vision tokens guided-update
Full Vision Tokens
pressed to 36 proxy vision tokens, and each proxy token corresponds to 16 full vision tokens in the guided-update process. For the non-spatial ProxyV version, we set the num- ber of learnable queries to be the same as the spatial version. The hidden dimension in the guided-updated updated MLP module set to be of the hidden dimension the LLM. The number of parameters of the newly added guided-update module for each layer is 14.68M for the Vicunal.5-78 case. For the VisionZip baseline, we use 360 dominant tokens and 40 contextual tokens. For the PyramidDrop baseline, the vision token is reduced by 50% after layers 12, 20, and 26.
Figure 4. The illustration of the non-spatial ProxyV. Upper part: proxy vision tokens are generated as weighted combination of full vision tokens through simple attention operation. Lower part: The previous attention score is reused to splat the proxy vision tokens into guidance
full vision tokens update. The softmax operations are skipped in the figure.
into the LLMs.
Current LMMs can be broadly categorized into two groups: decoder-only LMMs and cross-attentionn-aas based LMMs. Decoder-only LMMs (Liu et al., 2024b; Wang et al., 2024; Chen al., 2024b) directly concatenate the projected visual tokens with textual tokens before LLM and process them equally text tokens in the LLM through self-attention de- coder layers. This approach has validated its simplicity and effectiveness, but incurs high computational costs with long vision sequences. Cross-attentie based L.MMs (Alayrac et al., 2022; Laurencon et al., 2023; Dubey et al., 2024; Dai et al., 2024), on the contrary, treat the visual information as context and introduce additional cross-attention layers to let text tokens extract visual information. These meth- ods avoid the computationally intensive vision-to-viss to-vision attention. But they also bring additional training complex- ity and a non-negligible amount of parameters compared with decoder-only ones and usually require a significantly larger amount of data for pertaining cross-atted cross-attention mod-
The reported FLOPs and time for all experiments are mea- sured during the prefilling stage, using fixed configuration of five image grids (2880 tokens) and 50 text tokens, with eager attention implementation on single H100 GPU. The full evaluation results are provided in Appendix B.
5. Related Works
5.1. Large Multimodal Models
As large language models achieved unprecedented success, large multimodal models (Liu et al., 2024b; Dai et al., 2023; Liu et al., 2024a; Lin et al., 2023b; Tong et al., 2024; Li et al., 2024b; Wang et al., 2024: Chen et al., 2024b) have emerged, leveraging LLMs as a foundation while incor- porating multimodal capabilities by injecting multimodal information extracted from pretrained multimodal encoders

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
nuck o
Follow
B
When is the article about China's s space exploration posted?
Question:
zuck Same updatee
Innovation
ettams, king sem visiin to peners
New Cortd-19's symptoore have
spe responabiy, make widelly avalible every to can rejper ai reseoren effart, (PAjl and CenAll saar together support mesi-gen model Liano and werte reaa compute
ProxyV: day ago
VisionZip: : 13 hours ago
futaro
roudimsp, including 500k inDil aimost 0008 vilb0s equledere compusee you rnclude other GPUS, Allo
PyramidDrop days ago
Viar
bu dl centiicc cor nouknng soories like the Bun
using glasses
V
A
Question: What is the exact content of the tweet?
Proxy:: Some updates on our AI efforts. Our long term vision is to build general intelligence, open source it responsibly, and make it widely available everyone can benefit. We'r bringing our two major AI research efforts (FAIR and GenAT) closer together to support this. We're currently training our next-gen model Llama 3, and we' we're building massive compute infrastructure to support roadmap, including 350k H100s by the end of this year and almost 600k H1e0s equivalents of compute if you include other GPUs. Also really excited about our progress building new AI-centric computing devices like Ray Ban Meta smart glasses. Lots more to come soon.
future
VisionZip: Some updates on our Al efforts. Our long term vision is to build general intelligence, open source it responsibly, and make it widely available so everyone can benefit. We're ee'r bringing our two major Al research efforts (FAIR and GenAI) closer together to support our future roadmap. Including 350k H180s by the end of this year and overall almost 600k H100s equivalents of compute if you include other GPUs Also really excited about our progress building new Al-Centric computing devices like Ray Ban Meta smart glasses. Lots more to come soon.
PyramidDrop: Some updates on Al. Our long term vision is to build general intelligence, open source it, make it readily available so everyone can benefit. We're re bringing our two- GenAI closer together to support our future roadmap, including 350k H100s by the end of this year - and overall almost 600k H1e0s equivalents of compute if you include other GPUs. Also really excited about our progress building new AI-centric computing devices like Ray Ban Meta smart glasses. Lots more to come soon.
Figure 5. Cases where token reduction methods fail. Left: Token reduction methods fail to extract the complete dense information accurately. Right: Token reduction methods fail to retain critical visual information when the image contains diverse and dense visual details. In these cases, ProxyV retains all the visual information and successfully extracts the important visual details.
ules. Besides, cross-attention based methods still slightly underperform the decoder-only ones, especially on tasks requiring fine-gre ene-grained visual information.
Xue et al., 2024) and multi-image LMMs (Li et al., 2024c; Jiang et al., 2024a), which introduce significantly more vi- sion tokens, further exacerbating the computational load.
Early LMMs typically processed low-resolution images 336), which significantly constrained their per-
5.2. Token Reduction in LMMs
(c.g.. 336 336 formance due to the limited input resolution. Recent ad- vancements in LMMs have adopted multi-grid image en- coding schemes (Liu et al., 2024a) directly processed native high-resolution images (Wang al., 2024), leading to substantial performance improvements and unlocking their potential for a wide range of applications. However, this capability comes at a cost, as the computational burden increases sharply due to the quadratic scaling of attention computation with the number of vision tokens. This chal- lenge becomes even more pronounced with video LMMs (Lin et al., 2023a; Maaz et al., 2024; Zhang et al., 2024a;
As LMMs face significant computational costs, especially with long vision sequences, extensive research has focused on reducing these costs through vision token reduction. The stages at which vision token reduction occurs can be grouped into three categories: (1) before the L.MM, (2) dur- ing the prefilling stage, and (3) during the decoding stage.
Token Reduction Before the LLM: Methods in this cat- egory directly reduce the number of vision tokens either within the vision encoder or from its outputs, utilizing the attention scores in the vision encoders (Shang et al., 2024;
8

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
Jiang et al., 2024b; Yang et al., 2024b). LLaVA-Prumeny (Shang et al., 2024) utilizes the attention scores between CLS token and other vision tokens from the vision encoder to choose crucial tokens to retain which are then further grouped and merged. FoPru (Jiang et al., 2024b) adopts two strategies to calculate the significance of vision tokens in global and local views respectively. VisionZip (Yang et al., 2024b) also uses the vision attention scores select informative tokens but they merge the remaining tokens to reduce information loss.
However, the reduction also inevitably introduces informa- tion loss when dense information is contained in the image and little redundancy exists. And the instruction/text guided pruning method does not consider the multi-turn conver- sation cases where later questions might require different vision information. Also, for complex or indirect questions, it is hard to accurately select all the critical vision tokens. On the contrary, our method takes another perspective to reduce the computation-leva redundancy to avoid informa- tion loss and can also be flexibly combined with these token reduction methods.
Token Reduction During the Prefilling Stage: Methods that conduct token reduction during the prefilling stage within the LLM often use the attention scores in the LLM self-attention for token reduction (Chen et al., 2025b; He et al., 2024; Zhang et al., 2024b; Xing et al., 2024; Endo et al., 2024; Zhong et al., 2024). Some methods (He al., 2024; Zhang et al., 2024b; Xing et al., 2024; Zhong et al.,
6. Conclusion
In this paper, we reveal the computation-level redundancy with vision tokens in LMMs. We further explore gradu- ally skipping the heavy attention and FFNs operations and find that using lightweight vision-specific MLPs as re- placement is able to compensate for the performance drop. We then propose a better solution ProxyV which utilizes proxy vision tokens to reduce the negative influence brought by skipping heavy operations on full vision tokens. Ex- periments on different LLMs validate the effectiveness of ProxyV. We also design a non-spatial variant of ProxyV which can be seamlessly combined with token reduction methods for better efficiency.
2024) also adopt adaptive strategies to progressively prune or merge the vision tokens along the layers.
FastV (Chen et al., 2025b) uses the attention patterns in early layers to prune vision tokens in the subsequent layers. ZipVL(He et al., 2024) designs a dynamic ratio allocation strategy to adaptively determine important vision tokens layer-specific attention statistics. SparseVLM (Zhang et al., 2024b) selects visual-relevant text tokens to rate the signifi- cance of vision tokens using the LLM self-attention scores and progressively prunes the vision tokens with adaptive scarification ratios across layers. PyramidDrop (Xing et al., 2024) uses the text-to-vision attention scores to progres- sively prune vision tokens in multiple stages across lay- ers. G-search and P-Sigmoid are proposed in (Zhao et al., 2024) to search for the optimal reduction ratio per layer. FEATHER (Endo et al., 2024) identifies the positional bias problem in previous methods and proposes to incorporate uniform sampling to ensure uniform coverage. AIM (Zhong et al., 2024) first merges the vision tokens before LLM based on embedding similarity and then adopts the page rank algorithm to progressively prune vision tokens in the LLM.
Acknowledgments
This study is supported by the Ministry of Education, Sin- gapore, under its MOE AcRF Tier 2 (MOE-T2HP2022)- 0012, MOE-T2EP20023-0002 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
and under the RIE2020
Impact Statement
This work aims to address the computational challenges of LMMs while maintaining enhancing their performance. The proposed approach reduces computational costs, mak- ing LMMs more accessible and environmentally sustainable by decreasing energy consumption. However, like other LMMs and LLMss ProxyV may inherit ethical considera- tions related bias, misuse, or privacy concerns depending the datasets and tasks it is applied to. Utilization of ProxyV V should focus on these aspects to ensure responsible and equitable deployment of such technologies.
KV-Cache Compression in Decoding: KV-Cache com-
pression during the decoding stage is widely studied in the LLM field (Xiao et al., 2023; Ge et al., 2023; Zhang et al., 2023). For LMMs, LOOK-M (Wan et al., 2024) designs text-prior KV pairs eviction strategy for multimodal KV cache pruning and provides different strategies to merge KV pairs. ElasticCache (Liu et al., 2024d) uses important key/value vectors as anchors and merges the less impor- tant ones into them. CSP (Pei et al., 2024) achieves more precise KV cache pruning by independently managing the mnter-modality and intra-modality attention.
Token reduction methods effectively remove token-level re- dundancy when it exists in the image in certain scenarios.
9

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A.. Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Endo, M., Wang, X., and Yeung-Levy, S. Feather the throt- tle: Revisiting visual token pruning for vision-language model acceleration. arXiv preprint arXiv:2412. 2024.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I,, Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M.. et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems, 35:23716 16-1116-26666 2022.
Fu, C.,, Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.1333
2023.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z.,, Chu, P., et al. Internlm2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for Ilms, arXiv preprint arXiv:2310.01801, 2023.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J.. Qiao, Y., Lin, D., al. Are we on the right way for evaluating large vision- assion-language models? arXiv preprint
He, Y., Chen, F,, Liu, J., Shao, W., Zhou, H., Zhang, K., and Zhuang, B. Zipvl: Efficient large ussionlanguaaee models with dynamic token sparsification and kv cache compression. arXiv preprint 2rXiv:2410008884 2024.
2024a.
Chen, L., Li, Dong,
X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi- models with better captions. ECCV, 2025a.
Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L.,, Zhang, B., Li, C., Zhang, J., Jin, Q., Huang, F,, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document under- standing. arXiv preprint arXiv:2403. 12895, 2024.
Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In ECCV, 2025b.
Huang, K., Zou, H., Xi, Y., Wang, B., Xie, Z., and Yu, L. Ivtp: Instruction Instruction-guided visual token pruning for large vision-language models. In ECCV, 2025.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, I.., al. Internvi: Scaling up vision foundation models and aligning for generic visual-linguisti linguistic tasks. In CVPR, 2024b.
Huang, Q., Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang. J., Lin, D., Zhang, W. and Yu, N. Deciphering cross-modal alignment in large vision- models with modality integration rate. arXiv preprint arXiv:2410.07167, 2024.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Dai, W., Lee, N., Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvim: Open frontier-class multimodal lims. arXiv preprint arXiv:2409.1 1402 2024.
Jiang, D., He, X., Zeng. H., Wei, C., Ku, M., Liu, Q. and Chen, W. Mantis: Interleaved multi-image instruction tuning. arXiv preprint 2rXiv:2405.0144 2024a.
Jiang, L., Huang, W., Liu, T., Zeng. Y., Li, J., Cheng, L., and Xu, X. Fopru: Focal pruning for efficient large vision- language models. arXiv preprint arXiv:2411.1444 2024b.
Dao. T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.08661, 2023.
Dao. T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficien exact attention with io-awareness. NeurIPS, 2022.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. "ReferltGame: Referring to objects in photographs of natural scenes". In EMNLP, 2014.
Dong, H., Li, J., Wu, B., Wang, J., Zhang, Y., and Guo, H. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405. 0..40059999 /9092, 2024.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In ECCV, 2016.
10

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
Laurencon, H., Strien, D., Bekman, S., Tronchon, L.,, Saulnier, I.., Wang. T., Karamcheti, S., Singh, A., Pis- tilli, G., Jernite, Y., et al. Introducing idefics: An open reproduction of state-of-the tatt--of-the-at visual language model. 2023.
Liu, Z., Liu, B., Wang, J., Dong, Y., Chen, G., Rao, Y., Krishna, R., and Lu, J. Efficient inference of vision instruction-following models with elastic cache. In ECCV, 2024d.
Lu, P., Mishra, S., Xia, T., Qiu, L.,, Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 2022.
Li, B., Wang, R., Wang, G., Ge, Y.. Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with gener- ative comprehension. arXiv preprint 1Xiv:2307.16125, 2023a.
Ma, F., Zhou, Y., Li, H., He, Z., Wu, S., Rao, F., Zhang, Y., and Sun, X. Fe-milm: A data-efficient and compute- efficient multimodal large language model. arXiv preprint arXiv:2408. 11795, 2024.
Li, B., Zhang, P., Zhang, K., Pu, F., Du, X., Dong, Y., Liu, H. Zhang, Y. Zhang, G., Li, C., and Liu, Z. Lmms- eval: Accelerating the development of large multimodal models, March 2024a. URL https: :ttpa:///iiiee //ggg/gtt././///////////....hh com/ EvolvingIMMs Lab/ 1mms eval.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In ACL, 2024.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint 23X1:2408..33366, 2024b.
Masry, A., Long, D., Tan, J. Q., Joty, S., and Hoque, E. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL Findings, 2022.
Li, F, Zhang, R., Zhang, H., Zhang, Y., Li, W., Ma, Z., and Li, C. intttreeiin Tackling multi- image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407. 407907895,... 2024c.
Mathew, M., Karatzas, D., and Jawahar, C. Doevqa: A dataset for vqa document images. In WACV, 2021.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305. 10355, 2023b.
Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. Infographievqa. WACV, 2022.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. Bleu: method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceed- ings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002.
Lin, B., Ye, Y.. Zhu, B., Cui, J., Ning, M. Jin, P., and Yuan, L. Video-Ilava: Learning united visual represen- tation by alignment before projection. arXiv preprint 1rXiv2231..10122 2023a.
Pei, X., Huang, T., and Xu, C. Cross-self kv cache pruning for efficient arXiv:2412.04653 2024.
Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models, 2023b.
inference. arXiv preprint
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Liu, H. Li, C., Li, Y. Li, B., Zhang, Y, Shen, S., and Lee, Y. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https://llava www.tgitththhhbbee 2024-013
Shang, Y., Cai, M., Xu, B., Lee, Y. J., and Yan, Y. Llava- prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403 248.. 2024.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tuning. NeurlPS, 2024b.
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, Li, C., Yin, X.-C., Liu, C.-L., Jin, L., and Bai, X. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 2024c.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, 2019.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all- all-around player? In ECCV, 2025.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: 1: A fully open, vision-centric exploration of multimodal lmms. In NeurIPS, 2024.

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
Turski, M., Stanistawek, T., Kaczmarek, K., Dyda, P., and Graliñski, F. Cepdf: Building high quality corpus for visually rich documents from web crawl data. In ICDAR, 2023.
Zhang, Y., Fan, C.-K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D., Okuno, T., Nakata, Y., Keutzer, K., et al. Sparsevim: Visual token sparsification for effi- cient vision-language model inference. arXiv preprint arXi::2410.04417, 2024b.
Wan, Z.. Wu, Z., Liu. C., Huang. J., Zhu, Z., Jin, P., Wang, L., and Yuan, L. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. arXiv preprint ar wxXi::2406118153 2024.
Zhang, Z.., Sheng, Y., Zhou, T., Chen, T., Zheng, I.., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. NeurIPS, 2023.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409.12199 2024.
Zhao, S., Wang, Z., Juefei-Xu, F., Xia, X., Liu, M., Wang, X., Liang, M.. Zhang, N., Metaxas, D. N., and Yu, L. Accelerating multimodel large language models by searching optimal vision token reduction. arXiv preprint arXiv:2412.00556. 2024.
XAL grok, 2024. URL / /oLhhttps//x.aalllo////r/ 5v.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z.,, Li, D., Xing, E,, Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In NeurIPS, 2023.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv, 2023.
Zhong, Y., Liu, Z., Li, Y., and Wang, L. Aim: Adaptive inference of multi-modal llms via token merging and pruning. arXiv preprint arXiv:2412.0322 2024.
Xing. L., Huang, Q., Dong, X., Lu, Zhang, P., Zang, Y., Cao, Y., He, C., Wang, J., Wu, F., et al. Pyramid- drop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprinl arXiv:2410. 2024.
Xu, Y., Yang, X., Song, Y., and Xu, C. Libra: Building decoupled vision system on large language models. In Proceedings of the st International Conference on Ma- chine Learning, 2024.
Xue, F,, Chen, Y., Li. D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., et al. Longvila: Scaling long- context visual language models for long videos. arXiv preprint Xiv: 10188, 2024.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., al. Qwen2 technical report. arXiv preprint arXiv:2407.1067 2024a.
Yang. S., Chen, Y. Tian, Z.. Wang, C., Li, J., Yu, B., and Jia, J. Visionzip: Longer is better but not necessary in vi- sion language models. arXiv preprint arXiv:2412.04467 2024b.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., and Liu, Z. Long con- text transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024a.
12

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
A. Evaluation on More Benchmarks
Besides the fine-grained benchmarks, here we evaluate models on a wide range of general multimodal benchmarks including MMBench (Liu et al., 2025), SEED-Bench (Li et al., 2023a), RefCOCO (Kazemzadeh al., 2014), MMStar (Chen et al., 2024a), GQA (Hudson & Manning, 2019), MME (Fu et al., 2023). MMMU (Yue al., 2024), POPE (Li et al., 2023b), ScienceQA (Lu et al., 2022), A12D (Kembhavi al., 2016), and RealWorldQA (xAI, 2024). We use the en-dev split of MMBench, the image-split of SEED-Bench, the test-A and test-B splits of RefCOCo, the ssstde-ballacee split of GQA, the perception split of MME, the validation split of MMMU, and the image split of ScienceQA.
We observe that on such general benchmarks where less fine-grained visual information is needed, the redundancy of visual information appears in earlier layers and the performance loss from skipping operations smaller. Consequently, applying ProxyV from middle-to-late layers achieves similar or better overall performance and better efficiency than later layers, it introduces vision-specific modules in more layers, leading on different LLMs together with the two token reduction baselines in Table 7. We can observe that ProxyV consistently achieves no performance loss even improvements on general multimodal benchmarks.
larger gain. We provide results of applying ProxyV
Table 7. Evaluation on comprehensive set of general multimodal benchmarks.
MMBench
SEED-Img
RefCOCO
MMStar
GQA
MME-P
MMMU
POPE
SQA
A12D
RealWorldQA
Vicuna-7B
Baseline
65.03
68.60
75.39
37.70
63.36
1428.52
36.56
86.91
68.52
66.71
56.73
Vr1.1
67.61
70.02
76.76
38.35
64.02
1478.61
35.44
86.55
68.07
69.11
59.08
VisionZip
65.37
68.74
71.63
37.38
63.93
1437.50
34.33
87.44
69.51
68.30
57.52
ProamidDoop
65.37
68.45
72.53
36.66
63.84
1451.33
35.56
87.10
67.63
67.94
58.04
Vicuna- 13B
Baseline
67.78
70.45
81.68
41.6
65.19
1617.57
34.56
86,58
71.05
69.88
58.95
V-L16
69.15
71.71
83.30
42.40
65.59
1602.
36.00
87.16
73.53
71.50
58.82
Llama3-8B
Baseline
70.01
72.11
71.8
44.04
64.63
1470.08
37.56
87.81
77.64
72.54
59.61
ProxyV-L116
70.53
72.44
73.32
44.61
64.28
1480.87
36.67
87.17
74.22
72.93
59.74
Qwen2-7B
Baseline 73.71
72.62
84.28
48.73
64.77
1579.88
42.67
87.19
77.59
74.13
62.61
V-1.16
75.51
73.58
90.37
51.38
64.28
1543.9
44.67
87.5
77.24
74.94
63.53
Phi3-3B
Baseline
64.69 67.95
68.10 16
52.63
38.36
61.27
1477.93
38.56
85.57
70.
67.68
56.73
Pr0x
68.77
56.22
40.60
61.64
1437.92
39.78
85.76
70.85
68.17
57.52
IntemLM2.5544
Baseline
74. 14
74.55
66.36 77.4
48.07
64.52
1455.53
42.56
87.78
77.24
75.61
63.52
Proxy V-L16
76.20
75.07
49.52
65.39
1465.65
43.00
87.13
77.98
74.22
65.35
B. Fine-grained Benchmark Details and Full Results
For all evaluations, we use the validation splits of DoA (Mathew et al., 2021), Info (Mathew et al., 2022), and TextVQA (Singh et al., 2019). We use the English dev split for MMBench (Liu et al., 2025) and the perception split for MME (Fu et al., 2023) (the score is normalized to 0-100 when calculating Score For the grounding benchmark ReiCOCO, we calculate the average of the testA and testB splits. All the evaluations are conducted using the Imms-eval framework (Li et al., 2024a).
The full benchmark results of Tables I to 3 are shown in Table 8 and the full results of Table 4 are provided in Table 10.
13

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
Table 8. Full results of the explorative experiments and ProxyV the fine-grained benchmarks.
Voc
ChartQA
InfoVQA
TextVQA
OCRBench
Scorefine
Baseline
68.03
59.64
33.60
62.12
49.80
54.64
-O-TF TF
35.82
31.92
25.74
45.85
31.90
34.25
LO-FI FT
57.50
48.48
31.38
58.29
42.70
47.67
LO ATN+FFN
40.19
36.60
25.60
45.88
29.60
35.57
LO ProxyV
57.99
51.24
30.88
59.07
43.70
48.58
L12 TF
63.20
55.40
31.59
60.53
47.40
51.62
L12 FT
66.55
58.48
34.27
61.36
50.70
54.27
L12 ATN+FFN
67.45
59.48
34.77
60.72
49.00
54.28
L12 ProxyV
68.18
60.16
34.77
61.69
51.00
55.16
L16 TF
68.46
59.20
33.55
61.80
50.70
54.74
L16
68.09
59.40
33.35
62.07
50.00
54.58
L16 ATN+FFN
68.95
59.32
33.49
61.29
51.20
54.85
L16 ProxyV
69.90
61.48
34.24
62.28
51.80
55.94
Table Full results of the token reduction methods, the non-spatial Proxy'V, and the combination of Proxvy-ns and VisionZip.
DocVQA
ChartQA
InfoVQA
TextVQA
OCRBench
Scoresue
Baseline
68.03
59.64
33.60
62.12
49.80
54.64
ProxyV Layer12
68.18
60.16
34.77
61.69
51.00
55.16
VisionZip
68.84
58.04
33.35
62.26
50.40
54.58
PyramidDrop
68.50
58.88
34.43
61.84
49.50
54.63
Proxy V-ns Layer12
68.16
60.28
34.27
61.56
50.00
54.85
ProxyV-ns + VisionZip
68.31
58.76
34.17
62.29
50.60
54.83
14

---

Streamline Without Sacrifice Squeeze out Computation Redundancy in LMM
Table 10. Full results ProxyV with different LLMs on the fine-grained benchmarks.
DocVQA
ChartA
InfoVQA
TextVQa
OCRBench
ScorOgne
Vicuna-7B
Baseline ProxyV Layer 12
68.03
59.64
33.60
62.12
49.80
54.64
68.18
60.16
34.77
61.69
51.00
55.16
ProxyV Layer 16
69.90
61.48
34.24
62.28
51.80
55.94
Vicuna- 1 13
Baseline
72.11
63.64
37.52
65.35
54.30
58.58
ProxyV Layer 16
73.69
65.04
38.09
64.72
53.20
58.95
ProxyV Layer 20
73.83
65.32
37.52
65.65
53.80
59.22
Llama3-8B
Baseline
71.87
63.64
32.20
63.08
52.20
56.60
ProxyV Layer 16
73.08
63.00
33.81
62.86
51.60
56.87
ProxyV Layer 20
72.89
63.32
35.09
62.84
53.30
57.49
Qwen2-7B
Baseline ProxyV Layer 16
76.59
65.76
42.31
63.69
52.40
60.15
76.54
65,80
42.31
63.99
54.10
60.55
ProxyV Layer 20
78.29
68.12 18.12
42.30
63.60
54.90
61.44
Phi3-3B
Baseline
62.98
53.00
33.88
56.89
42.70
49.89
Proxy'V Layer 16
62.55
53.72
34.67
56.98
43.50
50.28
ProxyV Layer 20
64.78
54.04
33.60
56.48
44.70
50.72
InternI.M2.5-77
Baseline
72.46
64.80
38.38
63.40
52.60
58.33
ProxyV Layer 16
74.00
65.20
39.41
62.70
52.10
58.68
ProxyV Layer 20
74.24
66.04
38.66
63.98
52.50
59.08
15


Reviews:
[{"review_id": "szM150K8OD", "reviewer": "Reviewer_JDVo", "summary": "Summary:\nThis paper explores the computational redundancy inherent in vision tokens within multimodal large language models. This paper reveals significant computational redundancy exists in vision token processing, particularly in the middle and later layers of such models. To address this inefficiency, this paper proposes ProxyV, a lightweight approach that optimizes vision token computation by introducing compressed proxy tokens and efficient update mechanisms.\n\nStrength:\n1.\tThis paper explores computational redundancy in vision tokens through extensive experiments.\n2.\tThe proposed ProxyV method successfully reduces the computation overhead and inference time.\n\nWeakness:\n1.\tThe article needs to be better organized. The font in the figure is too small and difficult to read.\n2.\tThe article lacks formal expression and the details are difficult to confirm.\n3.\tThe structure of the article needs further organization. The author gives a lot of experimental results in the method section, but does not explain the specific experimental settings.\n4.\tThe author's experimental and comparison methods should be aligned with [1][2]. The current experimental results lack comparison on a wide range of MLLM benchmarks.\n\n[1] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction\n[2] VisionZip: Longer is Better but Not Necessary in Vision Language Models", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have reviewed your suggestions thoroughly and made corresponding revisions to address each of your concerns as outlined below.\n- *Paper Organization and Readability*  \n  We appreciate your suggestion on improving readability. We have modified our figures, and the revised figures can be found in this [link](https://anonymous.4open.science/r/ICML2025_Paper2984_Rebuttal_Figures-B873). We will reorganize the paper for clarity. We will add clearer captions, make expressions more consistent, and provide a more formal, detailed explanation of our approach to ensure better understanding.\n- *Experimental Details*  \n  We have provided our experimental setting and details (data, training pipeline, model structure, image encoding scheme) for all experiments in Section 4 and provide more model hyperparameter details in Section B of the Supplementary. We will add more experimental and evaluation details in our revised version.\n- *Comparison on more MLLM benchmarks*  \n  In response to your valuable comment on broader comparisons, we have expanded our evaluation to include additional standard benchmarks for MLLMs, as summarized in the table below. Our revised manuscript will include extensive evaluations across different LLM variants. From this table, we can see our method ensures no performance loss or achieves performance gain on most benchmarks, while token reduction methods have inferior performance on grounding benchmarks like RefCOCO, which requires finer visual information. Also, we clarify that we initially mainly focused on fine-grained benchmarks as they are good indicators for possible visual information loss, which is critical for MLLM acceleration methods.\n\n| Model Group   | Model         | Avg     | MMBench | SEED-Img | RefCOCO | MMStar | GQA   | MME-P   | MMMU  | POPE  | SQA   | AI2D  | RealWorldQA |\n|---------------|---------------|---------|---------|----------|---------|--------|-------|---------|--------|-------|--------|-------|--------------|\n| **Vicuna-7B** | Baseline      | 63.35 | 65.03   | 68.60    | 75.39   | 37.70  | 63.36 | 1428.52 | 36.56  | 86.91 | 68.52 | 66.71 | 56.73        |\n|               | VisionZip     | 63.27 | 65.37   | 68.74    | 71.63   | 37.38  | 63.93 | 1437.50 | 34.33  | 87.44 | 69.51 | 68.30 | 57.52        |\n|               | Pdrop         | 63.24 | 65.37   | 68.45    | 72.53   | 36.66  | 63.84 | 1451.33 | 35.56  | 87.10 | 67.63 | 67.94 | 58.04        |\n|               | **ProxyV**         | **64.44** | 67.61   | 70.02    | 76.76   | 38.35  | 64.02 | 1478.61 | 35.44  | 86.55 | 68.07 | 69.11 | 59.08        |\n| **Vicuna-13B**| Baseline      | 66.23 | 67.78   | 70.45    | 81.69   | 41.60  | 65.19 | 1617.57 | 34.56  | 86.58 | 71.05 | 69.88 | 58.95        |\n|               | **ProxyV**         | **67.20** | 69.15   | 71.71    | 83.30   | 42.40  | 65.59 | 1602.15 | 36.00  | 87.16 | 73.53 | 71.50 | 58.82        |", "claims_and_evidence": "See summary", "methods_and_evaluation_criteria": "See summary", "theoretical_claims": "See summary", "experimental_designs_or_analyses": "See summary", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "See summary", "other_strengths_and_weaknesses": "See summary", "other_comments_or_suggestions": "See summary", "questions_for_authors": "See summary", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SQ6vqG64oO", "reviewer": "Reviewer_nPTg", "summary": "This paper focuses on the token acceleration of LMM. It understands the hierarchical redundancy on visual tokens through validation experiments, and finds that visual tokens from LMM visual encoders do not necessarily require all heavy operations in the decoder-only LMM. ProxyV is designed to reduce computational burden with slight loss of information, on some VQA benchmarks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback. We have carefully considered your comments and made detailed revisions to address each point as follows:\n- *Generalizability of Claims*  \nThe exploratory experiments presented in Section 2 are designed to identify computational redundancy patterns rather than propose universal claims or laws. Through our experiments on a wide range of LLMs, we can observe clear signals of computational redundancy on visual tokens, and we also explicitly state that \"different models exhibit different patterns\" in L130. Our intent was to highlight potential redundancy, motivating further focused studies. We also additionally add experiments on a deeper model, Qwen-2.5-32B (using LLaVA-1.5 style image encoding), and observed similar computational redundancy patterns (See the updated [Figure here](https://anonymous.4open.science/r/ICML2025_Paper2984_Rebuttal_Figures-B873/Figure2_revised.png)). We will revise our paper to make the claims more clear.\n- *Comprehensive Benchmarking and Evaluation*  \nWe appreciate your suggestion regarding the comprehensiveness of evaluation benchmarks. Accordingly, we have expanded our evaluation to include multiple popular MLLM benchmarks. Our extended evaluation (detailed below) clearly shows that our method consistently achieves equal or superior performance compared to the baseline, while other token reduction methods show a notable gap in **fine-grained visual grounding tasks like RefCOCO**. We will provide additional comprehensive evaluations of all LLM variants in our next revision. Also, we clarify that we initially mainly focused on fine-grained benchmarks as they are good indicators for possible visual information loss, which is critical for MLLM acceleration methods.\n\n| Model Group| Model| Avg| MMBench |SEED-Img|RefCOCO|MMStar|GQA|MME-P|MMMU|POPE|SQA|AI2D|RealWorldQA|\n|---------------|---------------|---------|---------|----------|---------|--------|-------|---------|--------|-------|--------|-------|--------------|\n| **Vicuna-7B** | Baseline      | 63.35 | 65.03   | 68.60    | 75.39   | 37.70  | 63.36 | 1428.52 | 36.56  | 86.91 | 68.52 | 66.71 | 56.73        |\n| |VisionZip| 63.27 |65.37|68.74|71.63| 37.38|63.93|1437.50| 34.33  | 87.44 | 69.51 | 68.30| 57.52  |\n||Pdrop| 63.24 |65.37| 68.45|72.53|36.66|63.84| 1451.33|35.56|87.10|67.63|67.94|58.04|\n||**ProxyV**|**64.44**| 67.61|70.02|76.76|38.35| 64.02 | 1478.61 | 35.44  | 86.55 | 68.07 | 69.11 | 59.08 |\n|**Vicuna-13B**| Baseline|66.23|67.78|70.45| 81.69   | 41.60  | 65.19 | 1617.57 | 34.56  | 86.58 |71.05 |69.88 | 58.95 |\n|               | **ProxyV**         | **67.20** | 69.15   | 71.71| 83.30|42.40 |65.59|1602.15|36.00 | 87.16 | 73.53 | 71.50 | 58.82        |\n- *Unclear experimental details*   \nWe have provided our experimental setting and details (data, training pipeline, model structure, image encoding scheme) for all experiments in Section 4 and provide more model hyperparameter details in Section B of the Supplementary. We will add more experimental and evaluation details in our revised version.\n- *Absolute values for FLOPs & Time, ProxyV’s additional costs*  \nThank you for your valuable suggestion. We provide the absolute FLOPs & time values for Vicuna-7B/13B based models below, and we will add the results for all models in our revised version. All the FLOPs and times are directly measured in the same setting for all models, so the additional operations in ProxyV are already included. We will add clarification about this in the paper.\n\n| Model Group   | Model         | Time (s)     | FLOPS (T) |\n|---------------|---------------|---------|---------|\n| **Vicuna-7B** | Baseline      | 0.252 | 42.47   |\n|               | ProxyV-L12     | 0.148 | 23.13   |\n|               | ProxyV-L16     | 0.173 | 27.00   |\n| **Vicuna-13B**| Baseline      | 0.411 | 81.42    |\n|               | ProxyV-L16        | 0.244 | 46.01   |\n|               | ProxyV-L20        | 0.274 | 51.91   |  \n- *Missing Essential References & Only compared with two non-SOTA competitors*  \nThank you for recommending additional related papers. According to the ICML 2025 review instructions, \"Authors cannot expect to discuss other papers that have only been made publicly available within **four months** of the submission deadline.\" PyramidDrop (CVPR 2025) is the most recent token reduction method that has been peer-reviewed to the best of our knowledge and can well represent the SOTA performance of token reduction methods. Besides, we emphasize that our proposed ProxyV approach is orthogonal and complementary to token reduction methods, with combined effectiveness demonstrated in Section 3.3.\n- *Figure font size and detailed results in tables*  \nWe appreciate your suggestion on improving readability. We have modified our figures in this [link](https://anonymous.4open.science/r/ICML2025_Paper2984_Rebuttal_Figures-B873). The detailed benchmark results for all experiments are provided in Section A of the Supplementary.", "claims_and_evidence": "This manuscript was inspired by experiments to develop specific designs, thus many priors may not necessarily have generalizability. For example, in Line 106-129, \"directly masking vision token attention across the entire LMM leads to a significant performance drop while masking it from the middle or later layer has minimal or no effect on performance.\" Does the network only need to be divided into three levels? If the network is deeper, does the conclusion still hold?", "methods_and_evaluation_criteria": "Overall, the evaluation criteria and benchmark are insufficient, which makes the work's contribution to the community somewhat minor. As LMM has wide application scenarios, only using DocVQA\\ChartQA\\InfoVQA\\TextVQA\\OCRBench for evaluation is not convincing enough. \nn fact, if evaluated on more comprehensive LMM benchmarks (such as fine-grained vision, retrieval, detection, and segmentation), it would be better to assess the multi-modal understanding and reasoning capability.", "theoretical_claims": "The paper does not propose any theoretical claims.", "experimental_designs_or_analyses": "After reviewing all the experimental analyses, the reviewer found many results of this paper somewhat unconvincing. For reduced FLOPs/Time, there is no specific value, only the change proportion (In fact, there is a lot of blank space in the manuscript). For comparisons with existing methods, there are only two competitors (and not SOTA).", "relation_to_broader_scientific_literature": "Compared to mainstream token compression ideas, this paper focuses on using some proxy tokens to compensate for performance, which makes it somewhat similar to one prompt-tuning approach (learnable tokens) applied to the acceleration of LMM. \n\n[1]  Learning to prompt for vision-language models. IJCV 2022\n\n[2] Prefix-tuning: Optimizing continuous prompts for generation. ACL 2021", "essential_references_not_discussed": "There are several methods of token acceleration are missing, which have proposing some similar experimental findings.\n\n[1] Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration. ArXiv 2024\n\n[2] Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for Training-Free Acceleration. ArXiv 2024\n\n[3] FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance. ArXiv 2024", "other_strengths_and_weaknesses": "[-] Insufficient Comparisons. This paper only compares with VisionZIP and PyramidDrop, which is unconvincing. And in terms of benchmarks, there are only so-called fine-grained VQA, without comprehensive evaluation for the LMM's performance.\n\n[-] Unclear Details. Many of results and validation experiments do not provide specific details, which makes the reviewer a bit confused about the validity of the conclusions.\n\n[-] Unfair Comparisons. To squeeze out computation redundancy in LMM, this paper introduces ProxyV tokens with additional costs. How does Tab. 4-11 consider these additional costs?", "other_comments_or_suggestions": "Please consider the font size and information content of the table/image. Most figures are difficult to see in terms of text size, and most tables have results that are not detailed enough.", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eZs1XnuQoa", "reviewer": "Reviewer_iBst", "summary": "This paper focuses on the computational redundancy of visual tokens in large multimodal models (LMMs). It is found that there is computational-level redundancy in visual tokens within LMMs, and different LLMs exhibit varying degrees of redundancy. The ProxyV algorithm is proposed. By introducing proxy visual tokens, it alleviates the computational burden of original visual tokens, improves efficiency without sacrificing performance, and can even enhance performance in some scenarios. Experiments verify the effectiveness of ProxyV on different LLMs. Moreover, a non-spatial variant is designed, which can be combined with token reduction methods to further boost efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive and insightful feedback. We carefully considered your reviews and revised paper accordingly. Below, we directly address each point you raised:\n- *Incomplete evidence for the existence of computational redundancy*  \n  Thanks for your suggestion to validate our observations on larger-scale models. Initially, we explored redundancy patterns using six widely used LLMs to broadly cover commonly studied architectures. Following your recommendation, we additionally conducted experiments on a larger-scale model, Qwen-2.5-32B (using LLaVA-1.5 style image encoding), and observed similar computational redundancy patterns (See updated [Figure here](https://anonymous.4open.science/r/ICML2025_Paper2984_Rebuttal_Figures-B873/Figure2_revised.png)). This confirms that our earlier findings generalize to larger models.\n- *Lack of in-depth analysis for the performance improvement*  \n   Your suggestion to clarify the internal mechanisms underlying performance improvements is very valuable. To elucidate this, we measured the MIR (Modality Integration Rate) score [1], which quantifies alignment between visual and textual tokens (lower scores indicate better alignment). Evaluating 100 randomly sampled instances from the DetailCaps-4870 benchmark [2], we found that applying ProxyV-L12 to Vicuna-7B decreased the MIR score from 3.62 to 3.10, reflecting improved modality alignment due to decoupled vision modules. We will expand upon this insightful analysis in our revised manuscript.\n- *Internal relationship between task characteristics and algorithm performance*  \nThanks for your constructive suggestion. Instead of defining tasks as fine-grained and coarse-grained based on human priors, we will provide quantitative metrics to represent task characteristics. Here we take ChartQA and GQA as two representative tasks, and we downscale the input image to 336*336 and measure the relative performance drop. ChartQA's performance drops by 28.7%, while GQA's performance drops by only 0.1%, indicating ChartQA requires much more fine-grained information than GQA. Then, we conduct experiments with the ATN+FFN (skipping both visual attention and FFN) variant to study the appearance of visual computation redundacy. We find that to retain the original performance, the skipping operation has to be applied after layer-17 for ChartQA and only layer-6 for GQA. This indicates that the visual computation redundancy appears from early layers on less fine-grained dataset GQA. We will thoroughly detail these results and analyses in the revised paper.\n- *Evaluation criteria may not be comprehensive enough*    \n  Following your advice, we expanded our evaluations across additional MLLM benchmarks (see our response to Reviewer nPTg). We will provide results for all model variations in our revision.\n-  *The results of applying ProxyV to different layers could be presented with more fine-grained details.*  \nWe provide the full evaluation results on each benchmark in Section A of the Supplementary.\n- *Validate ProxyV with larger scale data & models*  \nThanks for your valuable advice. We further increase the data scale to 3M and validate our method with Vicuna-7B (results shown in the table below) and observe a consistent conclusion. Due to computational constraints, we currently do not have results for even larger datasets or models such as 32B-scale models, but we plan to explore this in future work.\n\n|SFT Data|**Avg**|DocVQA|ChartQA|InfoVQA|TextVQA|OCRBench|\n|---------------|---------|---------|----------|---------|--------|-------|\n|Baseline (779K)|**54.64**|68.03|59.64|33.60|62.12|49.80|\n|ProxyV-L16 (779K)|**55.94**|69.90|61.48|34.24|62.28|51.80|\n|Baseline (3M)|**64.89**|79.45|65.32|48.73|67.84|63.1|\n|ProxyV-L16 (3M)|**65.97**|80.3|67.04|50.20|69.01|63.3||\n\n- *Results of ATN+FFN*  \n  In Table 2, the performance of ATN+FFN is much worse than ATN for the layer-0 case, achieves similar performance for the layer-12 case, and improves the performance for the layer-16 case. We have analyzed the reason in the last part of Section 2 and attribute the performance gain to the additional vision specific modules in ATN+FFN and the negative impact of additionally skipping FFNs becomes smaller in later layers. Our focus is to ensure no performance drop after the acceleration. Note that ATN+FFN still incurs performance degradion when applying from Layer 12 ($Score_{fine}$ = 54.28) and only achieves no performance loss applying from ($Score_{fine}$ = 54.85), while ProxyV-L12 achieves performance gain ($Score_{fine}$ = 55.16). In this case, ProxyV-L12 also has better efficiency than ATN-FFN-L16. All experiments are conducted in the same setting for fair comparison.\n\n[1] Huang, Qidong, et al. \"Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate.\" arXiv preprint arXiv:2410.07167 (2024).\n\n[2] Dong, Hongyuan, et al. \"Benchmarking and improving detail image caption.\" arXiv preprint arXiv:2405.19092 (2024).", "claims_and_evidence": "The evidence is partially insufficient. Although a series of experiments in the paper verify the existence of computational redundancy and the effectiveness of ProxyV, the evidence for some key conclusions is incomplete. For example, when analyzing the redundancy patterns of visual attention in different LLMs, only the experimental results of a limited number of models (such as Vicuna1.5-7B, Qwen2-7B, etc.) are presented, which is difficult to represent all LLMs. We would like to see results on larger models (32B).\nThe analysis of the reasons for performance improvement is inadequate. The ProxyV algorithm brings performance improvement. The paper attributes it to the decoupled vision-specific modules. However, there is a lack of in-depth analysis and evidence on how these modules specifically act on different tasks and model structures, making it difficult for readers to clearly understand the internal mechanism of performance improvement.", "methods_and_evaluation_criteria": "Rationality of the method: The ProxyV algorithm, aiming at the computational redundancy of visual tokens, uses proxy tokens. Theoretically, it can effectively reduce the computational load. The method is reasonably designed. Compared with traditional token reduction methods, it avoids information loss and is innovative.\nEvaluation criteria: The selected OCR-extensive benchmark tests and other related datasets, such as DocVQA and ChartQA, are suitable for evaluating the performance of models in processing visual information tasks. They can effectively test the model's ability to understand and process fine-grained visual information. However, for more complex tasks in real-world scenarios, such as dynamic scenarios of multimodal information fusion, the evaluation criteria may not be comprehensive enough.", "theoretical_claims": "There is no theoretical content.", "experimental_designs_or_analyses": "Comprehensiveness of experimental design: The overall experimental design is relatively comprehensive. By comparing different experimental settings (such as different attention masking positions and different model structures), the computational redundancy problem and the performance of the ProxyV algorithm are systematically studied. However, in the comparison experiments, for some experiments combining with other methods (such as the combination of non-spatial ProxyV and VisionZip), the experimental design could be more detailed. For example, experimental results under different combination ratios could be added to more comprehensively evaluate the combination effect.\nDepth of experimental analysis: The experimental analysis mainly focuses on the comparison of performance indicators (such as Score, FLOPs, Time). The analysis of the reasons behind the experimental results is not in-depth enough. For example, when ProxyV performs differently in different tasks (fine-grained and coarse-grained tasks), only the early or late appearance of computational redundancy is simply mentioned, and the internal relationship between task characteristics and algorithm performance is not deeply explored.", "relation_to_broader_scientific_literature": "Relationship with existing models: The work of this paper is closely related to current mainstream research on large multimodal models. In terms of improving computational efficiency, it contrasts with traditional token reduction methods, pointing out that token reduction methods have the problem of information loss, while the ProxyV algorithm of this paper avoids this problem by reducing computational redundancy. Compared with cross-attention-based LMMs, ProxyV reduces computational costs while maintaining the simplicity and efficiency of the decoder structure.\nExpansion of research direction: Based on the existing research's focus on visual information processing and computational efficiency, this paper further explores the computational redundancy of visual tokens, providing new ideas and methods for improving the efficiency of large multimodal models and expanding the research direction in this field.", "essential_references_not_discussed": "No more referecens needs to be discurssed.", "other_strengths_and_weaknesses": "The results of applying ProxyV to different layers could be presented with more fine-grained details.", "other_comments_or_suggestions": "Supplement more theoretical analysis.\nValidate on more benchmarks.", "questions_for_authors": "In the experiments, only a limited number of LLMs were tested. How can you ensure that the ProxyV algorithm is equally effective on larger-scale LLMs (32B)? And what is the performance under larger data volumes (such as Cambrian-10M or Llava-onevision)? If you can provide experimental results or theoretical analysis on more models, it will enhance the universality of the paper's conclusions, and my evaluation of the paper will be more positive.\nThe results of ATN+FFN in the paper seem to be good enough. Please compare the experiments of ATN+FFN and ProxyV in detail under fair settings.\nIn Figure 2, the results of ATN+FFN are better than those of ATN, and it achieves better performance with less computational effort. Please analyze the reasons.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for token pruning that introduces a few proxy visual tokens to reduce the burden from the original tokens.  The authors provide both a study of the redundancy pattern of tokens, motivating the proposed approach.  The majority of reviewers commended rejecting the paper in its current form.  However, one reviewer did not update their review post-rebuttal, and the ACs feel their comments were addressed.  The other reviewer requested experiments comparing to another task, prompt-tuning, which the reviewer argues is conceptually similar.  While the AC agrees that there can be parallels drawn between these approaches, the goals of the two tasks are different.  Specifically, prompt-tuning typically aims to reduce training resources (typically without evaluating test-time inference time), whereas token pruning focuses on test-time inference.  Thus, even if they are conceptually similar, it does not diminish their contributions as the two tasks have wildly different applications.  The AC does agree with the comment made by all reviewers that the experimental details are lacking, especially the in-depth comparisons to prior work (e.g., comparing against methods like PruMerge or FastV, even if they are expected to perform worse than other baselines).  The results in the rebuttal only partly address these concerns, and the authors are encouraged to provide several computational budgets on a similarly rigorous evaluation comparing against prior work in their next version.
Decision: Accept (poster)