VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration
Authors: Hanxun Yu
Wentong Li
Xuan Qu
Song Wang
Junbo Chen
Jianke Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Alignment
Vision Language Model
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A unified training-free framework for MLLM acceleration through comprehensive vision token compression.

Abstract: Multimodal large language models (MLLMs) suffer from high computational costs due to excessive visual tokens, particularly in high-resolution and video-based scenarios. Existing token reduction methods typically focus on isolated pipeline components and often neglect textual alignment, leading to performance degradation. In this paper, we propose VisionTrim, a unified framework for training-free MLLM acceleration, integrating two effective plug-and-play modules: 1) the Dominant Vision Token Selection (DVTS) module, which preserves essential visual tokens via global-local view, and 2) the Text-Guided Vision Complement (TGVC) module, which facilitates context-aware token merging guided by textual cues. Extensive experiments across diverse image and video multimodal benchmarks demonstrate the performance superiority of our VisionTrim, advancing practical MLLM deployment in real-world applications. The code is available at: https://github.com/hanxunyu/VisionTrim.

Published as a conference paper at ICLR 2026
VISIONTRIM: UNIFIED VISION TOKEN COMPRESSION FOR TRAINING-FREE MLLM ACCELERATION
Hanxun Yu' Wentong Li Xuan Qu Song Wang', Junbo Chen"t,
Jianke Zhu" 1.4
State Key Lab of CAD & CG, Zhejiang University *Shenzhen Loop Area Institute
NUAA Udderaai
"Equal contribution. 'Corresponding authors.
ABSTRACT
Multimodal large language models (MLLMs) suffer from high computational costs due to excessive visual tokens, particularly in high-resolution and video- based scenarios. Existing token reduction methods typically focus on isolated pipeline components and often neglect textual alignment, leading to performance degradation. In this paper, we propose VisionTrin Trim, a unified framework for training-free free MLLM acceleration, integrating two effective plug-and-play mod- ules: 1) the Dominant Vision Token Selection (DVTS) module, which pre- serves essential visual tokens via global-local view, and 2) the Text-Guided Vision Complement (TGVC) module, which facilitates context-aware token merg- ing guided by textual cues. Extensive experiments across diverse image and video multimodal benchmarks demonstrate the performance superiority of our Vi- srom,rrm, advancing practical MILM deployment in real-world applications. The code is available at: http://gthhub.com/aanxuuu wawaaannunnnunii/iiionoo VisionTrim.
INTRODUCTION
With the recent advancements large language models (LLMs) (Vicuna, 2023; Touvron et al., 2023; Bai et al., 2023a; Achiam et al., 2023), significant efforts (Bai et al., 2023b; Chen et al., 2024c; Reid et al., 2024) have been devoted to extending their impressive reasoning and interaction capabilities to vision- langualan tasks. Current multimodal large language models (MLLMs) typically integrate visual signals as sequential tokens, which are processed by an LLM to enable visual perception of the world.
Despite their promising performance, the extensive use of visual tokens, which dominate the input sequence of LLMs, substantially increases the computational complexity and cost associated with inference in MLLMs. This issue is particularly pronounced in high-resolution methods (Liu et al., 2024a;b; Chen al., 2024e; Li et al., 2024b) and video-based models (Zhang al., 2024b; Cheng et al., 2024; Shen et al., 2024), where the increased token length exacerbates computational overhead and severely restricts the practical deployment potential of VLMs (Jin al., 2024b).
Recent studies (Wang et al., 2024a; 2025a; Ye et al., 2025; Zhong et al., 2024: Jin et al., 2025) have focused on accelerating the inference of MLLMs by reducing visual tokens while preserving essential information. For instance, FasterVLM (Zhang et al., 2024a) and VisionZip (Yang et al., 2025) perform global dominant visual token selection after vision encoding, whereas FastV (Chen et al., 2024a) and Sparse (Zhang et al., 2025b) prune tokens based on attention weights during LLM decoding. While these methods yield promising results, they tend to focus primarily on spe- cific individual components of the MILM framework, typically either the vision encoding or LLM decoding phases. Though the concurrent work VScan (Zhang et al., 2025a) adopts a two-stage prun- ing approach, it overlooks the essential role of the text query in aiding visual token selection during the vision encoding stage and directly uses the attention distribution between all visual tokens and the final instruction token for pruning during the LLM decoding stage, causing the potential loss of crucial text-related visual tokens. Furthermore, existing text-agnostic approaches like Pyramid- Drop (Xing et al., 2024) frequently overlook the necessity of aligning visual token selection with

---

Published as a conference paper at ICLR 2026
JCLS)
Large Longunge Model
Trandtrmee Blnck E-
Large Longuige Model
Virim
Pruse
DVIS
DVTS
Cettmnee Block
or
Prune
type und pake tue seanit
Prompe Wast
bye anl coke
Trampt: What
TGVC
TGVC
Tokenizer
imags?
Vision Encialig Stage
ILM Decodling Stage
Previous metheds
Trim (ours)
Figure I: Comparison of previous methods with VisionTrim. (a) Previous methods focus solely on specific part of the MILM framework, typically the vision encoding L.LM decoding stages. (b) In contrast, VisionTrri optimizes the entire MILM pipeline by introducing two plug-and-play modules, Dominant Vision Token Selection (DVTS) and Text-Guided Vision Complement (TGVC), to effectively reduce visual tokens in both the vision encoding and LLM decoding phases.
textual information. This oversight can result in the loss of textual context, which essential for accurate LLM decoding, ultimately leading substantial degradation in performance.
To tackle these issues, we propose a unified vision token compression framework, named Vi- sionTrim, for training-free acceleration of MLLMs. As illustrated in Figure 1, in contrast to pre- vious methods that focus exclusively on visual token compression during either vision encoding or LLM decoding, our approach considers the entire forward propagation of the MILM. We introduce two plug-and-play modules that effectively accelerate both the vision encoding and LLM decoding processes, which can be seamlessly inserted between any two layers of the vision encoder and the LLM. Specifically, our proposed method primarily consists of two key components: the Dominant Vision Token Selection (DVTS) and the Text-Guided Vision Complement (TGVC) modules.
Firstly, within the DVTS module, we consider both global semantics and local spatial continuity to filter visual tokens that convey essential visual information. Beyond utilizing [CLS] token's 's attention scores for global semantic importance, (LTAM) algorithm to simultancously capture feature similarity and spatial proximity among visual tokens. This approach ensures that critical visual details are retained while reducing redundancy. Secondly, in the TGVC module, we leverage textual information to guide the clustering and merging of pruned visual tokens relevant the input text instructions. These tokens then employed to complement the dominant visual tokens from the DVTS module. By integrating textual context into the visual token reduction process, our method enhances the implicit alignment between visual and textual representations, thereby improving the overall efficiency and performance of the pruned MLLM. As shown in Figure 2, our approach consistently surpasses previous techniques across range of reduction ratios, offering significant advantages in both efficiency and accuracy for various image- and video-based MI.LMs. In summary, the contributions our work are threefold:
develop the Local Token Affinity Measurement
We introduce VisionTrim, unified framework for vision token compression that enables training-free MILMM acceleration, optimizing the entire MILM pipeline.
We present two effective plug-and-play modules, DVTS and TGVC, designed to acceler- ate the forward processes of both the vision encoder and the LLM backbone, seamlessly integrable between any two layers.
Extensive experiments conducted on a variety of multimodal benchmarks, spanning both standard and high-resolution strate the superiority of our VisionTrim over previous state-of-the-art counterparts.
well image- and video-based MILMs, clearly demon-
2 RELATED WORK
2.1 MULTIMODAL LARGE LANGUAGE MODELS
Large Language Models (LLMs) (Vicuna, 2023; Touvron et al., 2023; Bai et al., 2023a; Team, 2023; Achiam et al., 2023) have garnered significant attention due to their powerful capabilities in natural language processing tasks such text understanding, generation, and question answering. Nonetheless, the reliance on purely textual data limits their applicability, as human perception is inherently multimodal. This has spurred the development of Multimodal LLMs (MLLMs) (Liu
2

---

Published as a conference paper at ICLR 2026
SprrVVMM
MINE
Sinraae Accuracy
Imap
VideoTp
YoWa
WVision Tokens/Image
Wision Tokens/laage
Comparison
benehmorks
(b) Comparison with other training-free methods
Comparison with other fine-tuning methods
Figure 2: Performance of VisionTrim. (a) Comparison across 10 benchmarks using the standard LLa (Liu et al., 2024a), with an 88.9% reduction in visual tokens. (b) & (c) Performance vs. efficiency of various methods with a range of visual tokens, in both training-free free and fine-tuning scenarios, respectively. The fine-tuning VisionTrim (Oursf) demonstrates superior performance over previous image- and video-based MLLMs.
al., 2023; Bai et al., 2023b; Chen et al., 2024c: Reid al., 2024; Yu et al., 2025; Lin al., 2024; Li et al., 2024a; Liu et al., 2024c; Wang et al., 2025b), which integrate LLMs with visual encoders to augment performance multimodal tasks. The typical image- and video-based MLLMs (Liu al., 2024a; Cheng et al., 2024; Lin et al., 2023) utilize an MLP to project visual information encoded by a Vision Transformer (ViT) (Dosovitskiy, 2020) into a space interpretable by LLMs, improving performance visual-language tasks through visual instruction tuning. However, this paradigm requires a large number of visual tokens to represent visual information, particularly with high- resolution images and long-context video inputs, which further exacerbates the issue. The resulting increase in computational demands and inference times poses significant challenges, hindering the practical deployment of MILMs in real-world applications.
2.2 VISION TOKEN COMPRESSION FOR MLLMS
The quadratic complexity inherent in Transformer networks (Vaswani et al., 2017), which scales with the sequence length of input tokens in MLLMs, remains widely acknowledged challenge. To address this issue, several methods (Li et al., 2023a; Bai et al., 2023b; Cha et al., 2024; Li et al., 2024b; Yao al., 2024; Hu al., 2024; Chu al., 2023; 2024) explore efficient visual projectors that enable compact visual representations using fewer visual tokens before feeding them into the LLM. While these approaches have demonstrated promising performance, they often necessitate architectural modifications and extensive training. Alternatively, recent works (Shang et al., 2024; Chen al., 2024a; Zhang et al., 2024a; 2025b; Yang et al., 2025) aim to reduce visual tokens in training-frre manner and mainly focus either the vision encoding LLM decoding stages. LLaVA-PruMerge (Shang et al., 2024) utilizes class-spatial similarity for pruning, and Faster- VLM (Zhang al., 2024a) evaluates token importance via attention scores between the [CLS] token and image tokens, both operating before sending the vision tokens to the LLM. Fast (Chen et al., 2024a) and Sparse (Zhang et al., 2025b) prune redundant tokens at specific layer of LLM based on attention scores solely during the LLM decoding stage. Additionally, most existing approaches overlook the alignment between visual token selection and textual information. While CrossGET (Shi et al., 2024) and Turbo (Ju et al., 2024) directly leverage text-visual attention to aid token selection, they place excessive emphasis on text tokens, which can lead hallucinations and disrupt multi-round interactions. In contrast, our approach considers the entire MLLM pipeline and simultaneously integrates both global semantic significance and local spatial continuity preserve visual integrity. Furthermore, we introduce a text- -ut-dd visual complement mechanism to ensure alignment with textual instructions, offering a more comprehensive and effective solution to the challenge of vision token compression.
METHODOLOGY
As illustrated in Figure 3, approach comprehensively considers the entire pipeline of MLLM, comprising two key components that simultaneously accelerate the vision encoder and LLM forward processes. The first component, Dominant Vision Token Selection (DVTTS) module, meticulously fil-
and
3

---

Published as a conference paper at ICLR 2026
Deminant Vision Token Selection (DVTS)
Large Language Model
Text-Suided Vision Complement
Z
(TBVC)
[CLS]
(CLS) Atteniin
Token Rank
* Encoder
Affaany
Chister
Vision
Tup-K
Transf oorfrre Block
DVTS
Transfor ormeer Block
Pro jector
**...*
Tent-to- Sirilee----gggee SimiirryyAAterennn
TGVC
Top Canter
Prompt: What
Tokenizer
and color
T6VC
Imagesee Similanrre
the image?
M
Desalet Tilcce
Toleet
(a) Overall training-free framework
(b) Strueture of TGVC module
Figure 3: (a) Overview of VisionTrim featuring the detailed DVTS module, and (b) the structure of the TGVC module. Both DVTS and TGVC modules can be generally utilized in both the vision encoding stage and the LLM decoding stage.
ters tokens preserve vital visual information, focusing particularly their significance for global semantics and local spatial continuity. The second component, Text-Guided Vision Complement (TGVC) module, leverages textual context to guide the clustering and merging of discarded visual tokens relevant to the input text instructions. This process complements the dominant visual tokens by integrating critical visual details. Both DVTS and TGVC are designed plug-and-ples modules that can be seamlessly integrated between any two layers of either the vision encoder or the LLM.
3.1 DOMINANT VISION TOKEN SELECTION (DVTS)
To preserve visual integrity during visual token compression, we introduce a novel scoring mecha- nism for selecting dominant vision tokens. This mechanism thoroughly incorporates both global se- mantic significance and local spatial continuity. Initially, we utilize [CLS] token's 's attention scores relative to other visual tokens to assess global semantic importance. Then, we develop the Local Token Affinity Measurement (LTAM) algorithm, which employs a dual-kernel method to capture feature similarity and spatial proximity, thereby ensuring local spatial continuity. These comple- mentary metrics are subsequently integrated using an adaptive variance-based weighting scheme to prioritize the reliable visual tokens.
Global Semantic Importance. Motivated by previous methods (Yang et al., 2025; Zhang et al., 2024a), the [CLS] token's attention distribution across all image tokens serves as natural measure of global semantic significance. We extract the attention weights from the penultimate layer of the CLiP-based vision encoder and leverage the attention patterns from the [CLS] token. The self- attention computation for the [CLS]] token is expressed as follows:
QiCLs: Wo
K, W
4-11 = softmax Q / de), i E [1, [
(2)
Here, x I--1 [CLS] and x denote the hidden states of the [CLS] token and the i-th visual token at the (L 1)-th layer, respectively. We and Wk are learnable projection matrices, and de is the dimension of key vector. N represents the total number of visual tokens. The global importance score 5 for the i-th visual token is the average attention score across all H heads:
H
S = H
Σ
L-1 L
E
This formulation effectively assesses each visual token's contribution to the global semantic rep- resentation of the image based on the [CLS] token's attention mechanism. The computed global T==1 are then normalized to yield a probability distribution over all visual tokens, i.e. $ exp(S))) Σx= ex
scores (S N
Local Spatial Continuity. Inspired by (Ru et al., 2022; Li et al., 2023b), we introduce the Local Token Affinity Measurement (L.TM)) algorithm effectively capture the local spatial continuity of
4

---

Published as a conference paper at ICLR 2026
visual tokens. LTAM utilizes a dual-kemel affinity mechanism to simultancously account for feature similarity and positional proximity. For the i-th token at position (x, y). its local importance S is determined by computing the affinity with neighboring tokens within a local kernel W(x, y) of size x k. For tokens positioned at (z, y) and (u, v). the affinity kernel x² is defined as a weighted combination of feature-based based term feat and a position based term Kposi
Fay Fuell
P Puol WoOp
aeguuo feat
zy.uv Pspos
(4)
feat
kow
+ W3g8ps
(5)
where Fay IR and Pzy E IR2 denote the feature vector and spatial coordinates of the token at (I, y). respectively. of and op represent the standard deviations of the feature and positional dif- ferences. The pair (h, w) sampled from the neighborhood set (xx y), and W1, W2, and Wy are balancing parameters. The local importance S of the i-th token at (x, y) is then computed by aver- aging the affinity scores over all neighboring tokens and converting to a probability distribution.
Adaptive Variance-based Weighting. To integrate global and local importance scores, we present an adaptive basedd weighting mechanism:
= + (1 a) where a off)
o and of denote the variances of the global and local importance scores, respectively. This adap- tive weighting scheme automatically prioritizes more reliable signals based on their consistency, ensuring robust token selection. The final importance scores, { are used to select the top- K' informative tokens Vdom E R xd from the complete set V E Rxd. This selection process ensures the preservation of both semantic relevance and spatial continuity.
3.2 TEXT-GUIDED VISION COMPLEMENT (TGVC)
Selected dominant tokens, while capturing primary visual information, may not fully reflect their relevance to the input instructions, potentially leading to misalignment with the textual information and loss of crucial visual elements. To address this issue, we introduce the Text-Guided Vision Complement (TGVC) module, which utilizes text instructions to complement the selected dominant vision tokens. By leveraging CLIP's text encoder, we calculate the similarity between the remaining visual tokens and text tokens, identifying the top- tokens as clustering centers. These centers then direct the allocation of remaining visual tokens to R clusters. Each cluster is merged to yield the final visual tokens most relevant to the text, which we term the vision complement tokens.
Clustering Centers. Given the remaining visual tokens V, E [(N-K)xd after dominant token selection, begin by calculating their similarity Si20 E R (X(xXN-)) with the text features T I.xd to identify potential clustering centers:
Si20 coftmax(TV?
Next, token-level importance scores s E R-N-K are obtained by averaging the similarity scores across all text tokens, expressed as s = 1 E Size.- The top-R tokens are then selected as clustering centers, denoted as C' = (c.....,CR)
Token Assignment. For each remaining token o E V, C. we compute its assignment score for each clustering center using text-guided similarity. Specifically, for center cj, the similarity scores are calculated as follows:
S softmax(e;21" /Vd), Size = sT /Va).
(8)
The assignment score a,y is then determined by asj S St Each token is assigned to the clustering center with the highest similarity score:
cluster(v:) arg max asj-
Cluster Aggregation. For each cluster centered cj. we aggregate the assigned tokens through weighted averaging based on their texl-guided similarities:
aij
Cj
+
Vi-
(10)
Ev Ecluster(y) akj
Eclusterr()
5

---

Published as a conference paper at ICLR 2026
Table I: Comparison with other methods on LLaVA-1.5-7B. The vanilla visual token count is 576.
The best results in each setting are bolded, and the second-bes are underlined.
Method
GQA
MMB
MME
POPE SQA VQA"
VQA
SEED
MMVet
VizWiz
Avg.
Upper Bound, 576 Tokens (100%)
Vanilla
61.9
64.7
1862
85.9
69.5
78.5
58.2
58.6
31.1
50.1
100.0%
Retain Averaged 192 Tokens
(1 66.7%)
SparseVLM
57.6
62.5
1721
83.6
69.1
75.6
56.1
55.8
30.2
50.0
96.4%
VisionZip
59.3
63.0
1783
85.3
68.9
76.8
57.3
56.4
31.7
50.5
98.3%
PDrop
57.3
63.3
1797
84.8
69.2
76.4
56.5
57.2
30.8
50.0
97.6%
VScan
60.6
63.9
1806
86.2
68.6
77.8
57.7
50.4
98.9%
Ours
61.0
64.4
1798
86.8
70.8
78.4
58.4
58.3
33.2
51.3
100.6 %
Retain Averaged 128
Tokens
(L 77.8%)
VL.
56.0
60.0
1696
80.5
67.1
73.8
54.9
53.4
30.0
51.0
94.2%
VisionZip
57.6
62.0
1762
83.2
68.9
75.6
56.8
54.9
32.6
50.0
97.2%
PDrop
57.1
61.6
1761
82.6
68.4
76.0
56.6
56.2
31.0
49.6
96.5%
VSean
59.8
63.0
1792
86.1
68.9
77.1
57.3
51.7
98.6%
Ours
60.3
64.0
1788
86.6
69.7
78.2
58.2
57.5
32.7
51.4
Retain Averaged 64 Tokens
() 88.9%)
VLM
52.7
56.2
1505
75.1
62.2
51.8
51.1
23.3
49.6
86.7%
VisionZip
55.1
60.1
1690
77.0
69.0
72.4
52.2
31.7
51.9
94.4%
PDrop
47.5
58.8
1561
76.2
69.0
73.3
50.6
53.0
30.5
50.2
90.8%
VScan
58.3
62.1
1698
85.0
69.1
75.4
55.6
51.8
96.8%
Ours
58.8
63.0
1780
86.2
71.0
76.8
56.8
54.8
32.2
52.3
98.8 %
This process is repeated for T iterations to refine the clusters. The final vision complement to- kens V com {ofom, 22am .... uR voom are then concatenated with the dominant tokens to form the complete visual representation:
final
V dom' V Vcom]
E R (K+))xd
(11)
This text-guided complement mechanism ensures that visual tokens effectively capture key visual details of the image while remaining aligned with the textual instruction.
3.3 MULTI-STAGE PRUNING STRATEGY
Our Dominant Vision Token Selection (DVTS) and Text-Guided Vision Complement (TGVC) mod- ules provide a versatile approach to token reduction that can be effectively applied at two stages of the MILM pipeline.
1) Vision Encoding Stage: Before LLM processing, DVTS and TGVC can reduce the initial visual token sequence = (V1, V2. VN more compact representation V where K + R N.
{v' Vi,..., v VK+R)
2) LLM Decoding Stage: DVTS and TGVC can be integrated between any two transformer layers during LLM decoding, enabling dynamic token pruning while preserving cross-modal alignment. Specifically, instead of using the [CLS] ] token, we leverage the attention distribution of the first generated token as a natural measure of the global semantic significance over all image tokens. At layer 7, global semantic scores S for DVTS and cross cross-modal attention scores A between visual and textual tokens for TGVC are computed as follows:
H VD
8 = softmax
gen
VD
E TR
IXN. A = softmax
NoN.
OR
N.
A
(12)
where Hi E IRIXD H E N.N. and H E NR represent the first generated token, visual tokens, and textual tokens at layer I, respectively. a; denotes the average cross-modal attention score for the i-th visual token. Using these scores along with the local spatial affinity scores S' from the LTAM mechanism, we first select the top-K tokens V dom DVTS and then perform top-R token complement Voom. in TGVC. Finally, obtain V final IV dom' V co The multi- stage application of our proposed DVTS and TGVC modules refines the visual representation, while ensuring both computational efficiency and effective cross-modal alignment.

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparisons across vari- ous token counts on LLaVA-NeXT-7B.
Table 3: Comparison with previous state-of-the- art methods on Video-LLaVA-7B.
Method
GQA MMB MME SQA VQA' POPE Avg.
Method
TGIF
MIVD
MISVITT
AetiviNNet
Ang.
Upper Bound, 2880 Tokens (100%)
Scors
Score
Vanilla
69.2
67.9
1842
70.2 61.3 86.3
100%
Vamis
43.1
3.55
09.5
3.9
56.7
3.65
4%11
3.35
100DS
+0.00
Retain 640
Tokens
(2 71.8%)
SpaseVLM
94.2%
0.06
97.7%
0.02
54.78
2.08
0.80
96.8%
0.02
8655
00.33
Sp.rseVI.M
93.9%
60.3
96.8%
65.7
96.2%
1772
96.4%
67.7
94.3%
57.8 85.2
98.7%
96.1%
VidinZip
42.0
-0.19
3.16
91.088
63.
4.54
3.58
87.5%
4.14
3.44
97.4%
-0.44
321
91.3%
40.20
VisionZip
95.5%
61.3
97.6%
66.3
97.0%
1787
97.0%
68.1
98.25
00.2
8177
69%
97.8%
Ours
96:09
45.2
3.32
98.39
69.6
3.0
96.39
54.9
3.42
100.9%
43.5
0.04
3,31
90.0%
(6.75)
-083
(017))
PDrop
62.9
66.5
1733
69.4
58.3
86.4
97.4%
Table 4: Ablation of pruning at vision encoding and LLM decoding stages.
98.0%
97.9%
94.1%
98.9%
95.1%
100.1%:
63.2
67.2
1825
70.7
61.0
88.5
99.9%
Ours
98.4%
99.0%
99.1%
190.7%999.5%
102.5%
(2.15%)
Retain 320
Takons
88.9%)
Stuges
Tokemss
GQA MMB POPE VQA
KV Cuche (MI)
Sp
$7.7
64.3
1694
67.0
55.9
78.6
92.4%
LJaVA-1 5-7B
S76
51.9
64.7
85.9
76.5
303.6
89.9%
94.7%
92.0%
95.4%
91.2%
91.1%
Only In VIT w/ DVTS w/ DVTS-TGVC
VisionZip
92.4%
59.3
92.9%
63.1
92.4%
1702
95.9%
67.3
96.1%
58.9
95.1%
94.1%
64
55.6
52.8
60.2
56.9
76.1
80.2
68.6
72.2
25.4 (1 91.8%)
58.5
63.2
1667
66.8
58.2
81.9
Only in LLM w/ DVTS DVTS-TGVI
PDrop
93.3%
64
52.0
57.4
75.8
70.2
43.5 (3 85.75))
91.1%
93.1%
90.5%
95,25
94.9%
94.9%
73.6
Ours
61.7
64.8
1795
69.6
59.6
83.6
97.0%
Both in ViT and LEM
64
58.8
63.0
56.2
76.8
502 (2 90/39)
96.1%
95.4%
97.4%
99.1%
97.2%
96.9%
(2.9%)
Retain 100
Tokens
(L 94.44%)
Table 5: Ablation study of various ensemble strategies in the DVTS module.
Sp
51.2
63.1
1542
67.5
46.4
77.3
86.3%
79.8%
92.9%
83.7%
96.2%
75.7%
89.6%
VisionZip
55.5
60.1
1630
68.3
56.2
79.4
90.7%
Ensemble Strategy
GQA MMB MME POPE
SQA
VQA Tou
86.4%
88.5%
88.5%
97.3%
91.7%
92.0%
PDrop
87.4%
56.1
88.8%
60.3
83.9%
1545
96.0%
67.4
89.2%
54.7
90.4%
89.3%
Element wise Maximum
Only [CLS] loken
55.3
58.7
1536
1702
74.2
77.6
67.9
70.0
51.2
52.4
57.2
63.3
1702
70.2
58.3
81.1
94.0%
Geometric Mean
56.5
1631
80.2
Ours
89.1%
93.2%
92.4%
100.0%95.1%%
94.0%
(3.3%)
Adaptive Weighting
58.8
63.0
1780
86.2
71.0
56.8
EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Datasets and Benchmarks. We conduct a comprehensive evaluation across 10 widely-used image-based benchmarks to assess the multimodal understanding and reasoning capabilities of our proposed approach. These benchmarks include common visual question answering tasks, like GQA (Hudson & Manning, 2019), VQA V2 (Goyal et al., 2017) and VizWiz (Gurari et al., 2018), as well as other multimodal benchmarks such as POPE (Li et al., 2023c), MMBench (Liu et al., 2025), MME (Fu et al., 2023) and MM- Vet (Yu et al., 2023). Additionally, we experiment with 4 widely used video-based multimodal understanding tasks: TGIF-QA (Jang et al., 2017). MSVD-QA (Xu et al., 2017), MSRVTT-QA (Xu et al., 2017), and ActivityNet-QA (Yu et al., 2019).
Implementaion Details. We apply our approach to various open-source MLLMs, including the classic LLaVA-1.5 (Liu et al., 2024a) model for normal-nesolution images. LLaVA-NeXT (Liu et al., 2024b) for high-resolution images, Video-LLaVA (Lin et al., 2023) for video- based tasks, and Qwen2-V1 (Wang et al., 2024b) and Qwen2.5-VL. (Bai et al., 2025) for broader validation. To ensure fair comparison, we adopt the default settings and evaluation metrics reported in their re- spective papers. We compare our approach with Sparse (Zhang et al., 2025b), VisionZip (Yang al., 2025), PyramidDrop (Xing et al., 2024). and VSean (Zhang et al., 2025a). Following the same spirit, we design different algorithms for multiple stages of the MLLM pipeline.
4.2 MAIN RESULTS
Normal Resolution. As shown in Table 1, we first evaluate our approach on L.LVVA-1.57B (Liu et al., 2024a) under the normal-resolution setting. VisionTrim consistently surpasses previous meth- ods across all token configurations (192, 128, and 64). In benchmarks, such as POPE, SQA, and TextVQA, Vision not only maintains its performance without degradation but also achieves improvements, highlighting the severe redundancy present in visual tokens fed to the LLM.
High Resolution. Our approach minimizes token count while maintaining performance NB (Liu et al., 2024b) with high-resolution inputs. As demonstrated in Table 2, Vision retains 99.9% of the original performance using only 22, 2..%% visual tokens. With nearly 95% token
LLaVA-
7

---

Published as a conference paper at ICLR 2026
Table 6: Experiment results of deploying Vision on Qwen2-VL-7B and Qwen2.5-VL-7B over several benchmarks. For both models, approximately 1/3 of the original input tokens are used.
Method
MMB
MMStar
VQA'
Method
MMB
MMB
MMMU
SEED
RefCOCO
Owen2-V
80.7
60.7
2322
84.3
S-V1
83.4
38,3
70.4
89.5
FastV
77.8
57.3
1859
77.6
FastV
77.5
76.3
34.6
64.8
73.6
SparseVLM
79.0
57.9
2019
79.2
84.2
SparseVI.M
79.5
80.3
36,0
67.5
78.4
PDrop
80.6
58.6
2053
80.2
82.5
PDrop
78.2
77.0
35,8
67.9
77.5
Ours
82.8
60.6
2310
83.5
86.3
Ours
83.2
81.8
37,9
70.2
86.8
Table Ablation study on TGVC module. This experiment, conducted before inputting data into the LLM, evaluates the effectiveness of the TGVC in reducing noise within text-visual at- tention during the LLM's forward pass.
1ayer2
Laper 18
Eayer 16
Syaue
Takcer
(1) Vanilla
Layer
Layer
Lo
Benchmack
Number of Tokens
Arg.
Smacn
Uier
POPE
81.0
73.9
78.4
(a) Token Componcens
(c) VisionTrim (ours)
w/ WTTVV
3.11
(1.7
4.317
753 440
12 3.71
MMBBrcch
614
58.4
Figure 4: Comparison of attention maps during LLM forward processing, with and without our proposed VisionTriim.
TGVC
63./1123)
(29/122))
333
19
61
55.3
54A
526
5012
53.1
TGVe
57.821255
5601 3.6
$4.11 40)
36 36
reduction, it achieves 94.0% performance without training, surpassing the previous sate-of-the-ar method, VisionZip (Yang et al., 2025), by 3.3%. These results validate the superior efficacy of VisionTrim for high rigouuesolution inputs.
Video. To assess the generalization of our approach on rideo-based scenarios, we apply it to Video- VL (Lin et al., 2023), which processes 8 frames from a video and generates 2048 visual tokens. Following Sparse VL.M (Zhang et al., 2025b), we reduce the visual tokens to 136. As shown in Table 3, Vision'rrim achieves 98.0% of the original performance with a 93.4% pruning ratio, outperforming all other methods across four benchmarks. Furthermore, VisionTrim consistently exceeds 96.0% in performance, demonstrating its effectiveness and robustness. Our method excels even with high pruming ratios, effectively balancing inference speed and accuracy in video tasks.
Broader Validation. To further evaluate the effectiveness of VisionTrim, we deploy to the state- of-the-an open-source MLLMs, Qwen2-VL-7B (Wang et al., 2024b) and Qwen2.5-VL-7B (Bai et al., 2025), using approximately 1/3 of the original input tokens. As shown in Table 6, VisionTrim exhibits competitive performance with only about 0.1 % performance loss across several cases, and even occasionally outperforms the baseline MILM. Notably, VisionTrim exceeds the vanilla Qwen2-VL (Wang al., 2024b) by 2.1% on the MMBench dataset, confirming its effectiveness in reducing visual redundancy. Please refer to the Appendix for more experiments on other tasks to further assess VisionTrim's generalization capabilities.
4.3 ABLATION STUDY
Component-wise Analysis. We conduct a thorough ablation study evaluate our approach for both vision encoding and LLM decoding stages, as presented in Table 4. We reduce image tokens to 64 for an 88.9% reduction (the same below). Initially, applying DVTS and TGVC modules solely in the vision encoder improves multimodal processing and reduces KV cache memory by 91.6%. For a fair comparison, we also implement the DVTS and TGVC modules only in LLM's decoding stage, yielding performance gains of 5.4% and 4.9% over Sparse VLM (Zhang et al., 2025b) on VQA v2 and MMBech datasets, respectively. When applied to both vision encoding and LLM decoding stages, VisionTrim outperforms approaches that target only speeific stages and achieves higher performance with 90.1 % reduction in memory usage, significantly surpassing existing state-of-the the-art methods. Moreover, Figure shows attention maps with and without VisionTrim. highlighting that the vanilla LLM exhibits high redundancy and suboptimal cross- modal alignment. In contrast, VisionTrim improves cross-modal alignment and reduces visual redundancy without performance compromise. Please refer to the Appendix for more ablation studies and visualization results.
Ensemble Strategy in DVTS. In the DVTS module, we employ various ensemble strategies to com- bine global semantic information from the [CLS] token attention and local spatial affinity captured by the LTAM algorithm, as shown in Table 5. Specifically, we explore three ensemble methods: element-wist maximum, geometric mean, and adaptive variance-hased weighting. Compared the
8

---

Published as a conference paper at ICLR 2026
N/o TGVC
TGVCx1
TGVCx5
"What Resgenes '64""
Mask Visualization of retained visual patches with and without the dual-attention mechanism in the DVTS module. Black-masked areas indicate discarded visual tokens.
Original Attcutiin Duel-Atication
Figure 5:
Figure 6: Visualization of retained visual patches with and without TGVC module. We show the correspondence between the salient vi- sual regions and text in different colors.
baseline that solely uses [CLS] token attention. incorporating both global semantic and local spa- tial continuity enhances performance significantly. Furthermore, as depicted in Figure 5, relying exclusively on [CLS] token results in the loss of crucial semantic information, while considering local spatial continuity helps retain better visual token coverage. Consequently, our proposed DVTS with dual-attention filtering mechanism offers more holistic approach to attention integration.
Visual Token Complement of TGVC. As shown in Table 7, incorporating the TGVC module sig- nificantly boosts performance across three multimodal tasks: POPE, MMBench, and Text-VQA. Notably, as the compression ratio increases and token count decreases, the TGVC module's s impact becomes more pronounced, resulting in performance gains exceeding 4%. Additionally, Figure 6 illustrates that the TGVC module retains essential visual tokens related to textual instructions, ensur- ing critical visual information is not pruned away. It can also be applied multiple times for enhanced visual completion, allowing textual tokens to better align with the corresponding visual information in the subsequent LLM decoding stage.
The Usage of Textual Prompts.
In VisionTrim, token compression has two stages: dominant visual tokens are selected via dual-attention mechanism, and discarded tokens are utilized with text- guided cues. (1) Without textual prompts, VisionTrrim runs in text-agnostic mode using DVTS, which relies on global [CLS] attention and local affinity; as shown in Table 10 in Appendix D.3, this maintains most accuracy while improving efficiency. (2) When prompts are irrelevant or misleading, text-image similarities become uniformly low, making textual initialization effectively random. Due to the semantic redundancy of visual tokens, the system behaves like unsupervised visual clustering (similar to ToMe (Bolya et al., 2022)), naturally grouping meaningful tokens. Since VisionTrim merges rather than prunes tokens, essential semantics are preserved even under poor textual cues.
4.4 EFFICIENCY ANALYSIS
We evaluate the efficiency of our method by measuring CUDA time, FLOPs, and storage memory, and compare it with vanilla LLaVA- NeXT-7B (Liu et al., 2024b) and other techniques, as shown in Ta- ble 8. At an 88.9% reduction ratio, our method reduces CUDA time by 61.4%, FLOPs by 91.7%, and storage memory by 93.3%, while maintaining 99.1% accuracy on SQA. Notably, when retaining the same token count, our method is 44.3% faster in inference time compared to VLM (Zhang et al., 2025b) and requires 50.0% less computational budget than VisionZip (Yang et al., 2025), while also minimizing KV cache memory usage. These results demonstrate the high efficiency of our approach. Additionally, further efficiency analyses on the POPE benchmark, including both the overall end-to-end latency and prefill time, demonstrating that VisionTrim is highly effective at accelerating MILM inference.
Table 8: Efficiency analysis of our method on LLaVA- NeXT-7B. The detailed metric includes latency (CUDA time), computation (FLOPs), and storage (cache memory).
Methods
PTokens
(%)
SQA
CUDA Time (Min & Sec)
FLOPS
Cache
Vanilla
2880
70.2
20034
9.6
1512.1
SpareMLM
520
67.0
18:26
30,66
1.5
$4.4%
168.0
85.95
Visenziip
320
67.3
17:33
32.7%
63.3%
1801
88
Ours
69.6
61.4%
101.
shown in Table 9, provide

---

Published as a conference paper at ICLR 2026
Table 9: Additional efficiency results on the POPE benchmark. We further report the end-to-end latency, prefill time, and the corresponding accuracy on single NVIDIA A100 GPU.
Methods
#Tokens
Total Inference Time
Prefill Time 4
FLOPs 4
Accuracy t
LLaVA-1.5-BB
576
1303 (1.00))
494 (1.0x)
3.8 T
85.9
+ SL.asse
1068 (1.22x)
377 (1.31x)
1.3
+ OOus
64
685 8 (1.90x)
235 (2.10x)
0.8
86.2
LLaVA-NeXI-7B
2880
2284 s (1.00x)
1062 s (1.00x)
12.6
86.3
+ LLM
320
1872 (1.22x)
644 s (1.65x)
2.5T
78.5
+ Ours
320
921 (2.48x)
360 s (2.95x)
1.2T
84.8
CONCLUSION AND LIMITATIONS
In this paper, proposed VisionTrim, unified training-free framework for MLLM acceleration through comprehensive vision token compression. We presented two effective plug-and-play mod- ules that accelerated both vision encoding and LLM decoding stages. By integrating the DVTS module, which selects tokens based on global semantics and local spatial continuity, with the TGVC module, which performs text-guided visual token complement, our approach consistently surpassed previous methods across various reduction ratios in both image and video understanding tasks.
Limitations. Although VisionTrim achieves 98.8% of the original performance with an 88.9% re- duction ratio in token count without additional training costs, it is not entirely without loss. We are committed to advancing our research to further explore the redundancy of visual tokens and developing lossless methods to enhance the efficiency of visual understanding with MI..Ms.
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China under Grant No.62376244. It is also supported by the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXivv2303.08771 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint MxXiv:2309.16609 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vasion-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023b.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2rXiv:2502.13923 2025.
Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.0946). 2022.
10

---

Published as a conference paper at ICLR 2026
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungscok Roh. Honeybee: Locality-enhanced enhanced projector for multimodal 1lm. In CVPR, pp. 13817-13827, 2024.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In ECCV, pp. 19-35. Springer, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-souree suites. arXiv preprint arXiv:2404. 16821, 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms, arXiv preprint XxXv:2406.00476. 2024.
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang. Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXiv:2312. 16886, 2023.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03700
2024.
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang. Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Ste Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6904-6913, 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pp. 18688-3612 2018.
Wenbo Hu, Zi-Yi Dou, Liunian Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. NeurIPS, 37:50168-555 18550188--088,,, 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In CVPR, pp. 2758-2766. 2017.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, pp. 13700-137 10, 2024a.
Xinqi Jin, Hanxun Yu, Bohan Yu, Kebin Liu, Jian Liu, Keda Tao, Yixuan Pei, Huan Wang, Fan Dang, Jiangchuan Liu, et al. Streamingassistant Efficient visual token pruning for accelerating online video understanding, arXiv preprint arXiv:2512. 2025.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey, arXiv preprint 2rXiv:2405.1073 2024b.
11

---

Published as a conference paper at ICLR 2026
Chen Ju, Haicheng Wang, Haozhe Cheng, Xu Chen, Zhonghua Zhai, Weilin Huang, Jinsong Lan, Shuai Xiao, and Bo Zheng. Turbo: Informativityy acceleration plug-in in for vision-lanee Assoonlanguaae large models. In ECCV, pp. 436-455. Springer, 2024.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742. PMLR, 2023a.
Wentong Li, Yuqian Yuan, Song Wang. Wenyu Liu, Dongqi Tang, Jianke Zhu, Lei Zhang, et al. Label-efficient segmentation via affinity propagation. Advances in Neural Information Processing Systems, 012299011229111 2023b.
Wentong Li, Yuqian Yuan, Jian Liu, Donggi Tang. Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang- Tokenpacker: Efficient visual projector for multimodal Ilm. arXiv preprint arXiv:2407.02399, 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint S.tv:2200..0353. 2023c.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, pp. 26689-26699, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuming. In NeurlPS, 2023.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In CVPR, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li. Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee, Llava-next: Improved reasoning, ocr, and world knowledge, 2024b. URL https: www.ggitth....oo.oooooooooooooee
next/-
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around Springer, 2025.
player? In ECCV, pp.
Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx milm: On- demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.1290 2024c.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.00 05424, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077993 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8788 8763. PmLR, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat. Julian Schrittwieser. et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0553 055330 2024.
12

---

Published as a conference paper at ICLR 2026
Lixiang Ru, Yibing Zhan, Baosheng Yu, and Bo Du. Learning affinity from attention: End-to-end weakly-supervie semantic segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16846-16855, 2022.
Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporaa adap- tive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024.
Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers. In ICML., 2024.
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://gitti com/ InternIM/ InternIM, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrIPS, 30, 2017.
Vicuna. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https: / //vicuna. 1msys. / 2023.
Ao Wang, Fengyuan Sun, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. [cls] token tells everything needed for training-free efficient milms. arXiv preprint arXiv:2412.05879, 2024a.
Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Quétu, Shuai Xiao, and Enzo Tartaglione. Folder: Accelerating multi-modal large language models with enhanced perfor- mance. arXiv preprint arXiv:2501.02433 2025a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Yuxin Wang. Lei Ke, Boqiang Zhang, Tianyuan Qu, Hanxun Yu, Zhenpeng Huang, Meng Yu, Dan Xu, and Dong Yu. N3d-vim: Native 3d grounding enables accurate spatial reasoning in vision- language models. arXiv preprint arXiv:2512. 2025b.
Hui Wei, Hanxun Yu, Kewei Zhang, Zhixiang Wang, Jianke Zhu, and Zheng Wang. Moiré backdoor attack (mba): A novel trigger for pedestrian detectors in the physical world. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 8828 8838, 2023.
Hui Wei, Hao Tang, Xuemei Jia, Zhixiang Wang, Hanxun Yu, Zhubo Li, Shin'ichi Satoh, Luc Van Gool, and Zheng Wang. Physical adversarial attack meets computer vision: A decade survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9797-9811 2024. doi: 10. (009/TPAM..2024.3430000
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-languag models via pyramid visual redundancy reduction. arXiv preprint 1aki::2410.172444 2024.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM, pp. 1645-1653, 2017.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In CVPR, 2025.
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decou- pling token compression from semantic abstraction in multimodal large language models. arXiv preprint ar 2it::2205.29988. 2024.
13

---

Published as a conference paper at ICLR 2026
Xubing Ye, Yukang Gan, Xiaoke Huang. Yixiao Ge, and Yansong Tang. Voco-Ilama: Towards vision compression with large language models. In CVPR, 2025.
Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-Imm: Instance-aware 3d scene understanding with multi-modal instruction tuning. arXiv preprint 00:::25333005333 2025.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang- Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.022 02490,, 2023.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. AAAI,, volume 33, pp. 9127-91111 9934, 2019.
Ce Zhang, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang. Yaqi Xie, Katia Sycara, Haitao Mi, and Dong Yu. Vscan: Rethinking visual token reduction for efficient large vision-language models. arXiv preprint arXiv:2505.22654. 2025a.
Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo Minqi Wang. Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token prun- ing: Make vlm inference faster. arXiv preprint arXiv:2412.0160 0280188 2024a.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevlm: Visual token sparsifi- cation for efficient vision-language model inference. In ICML. PMLR, 2025b.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL. https: / //wl///alaaaaaeeee g. thub. io/ii//llll 20011l//222400000000000
Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. Aim: Adaptive inference of multi-modal Iims via token merging and pruning. arXiv preprint arXiv:2412.03248 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2026
TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
In this part, we provide additional details and more experimental results on our approach. The supplementary material structured follows:
A: Individual case performance analysis;
C: Additional implementation details;
D: More experimental results;
Broader impacts;
Asset license and consent.
A INDIVIDUAL CASE PERFORMANCE ANALYSIS
Image:
Instruction: What color is wood bench?
Image:
Instruction: How long has the drink on the left been aged?
LLaVA-1:5: Grey.
Vision Token: 576
LLaVA-1.5: 12 yeurs.
Vision Token: 576
VisionZip: Brown.
VisionZip: 18 years.
Ours (w/ DVTS): Brown.
Vision Taken: 64
Ours (w/ WwVVVS): 18 years.
Vision Taken:
(DVTS + TGVC): Grey.
Vision Token:
(DVTS TGVC): 12 years.
Vision Taken:
(a) Individual Case 1.
(b) Individual Case 2.
Image:
Instruction: What is the lady doing?
Image:
Instruction: What brand soda on the bottom shelf on the left side?
LLaVA-1.5: Cooking.
Vision Taken: 576
LLaVA-1.5: Coca-Cola.
Vision Toker: 576
VisionZip: Cooking.
VisionZip: Drink.
Ours (w/ DVTS): Cooking.
Vision Taken: 64
Ours (w/ DVTS): Drink.
Vision Taker:
(DVTS TGVC): Cooking.
Vision Token:
(DVTS TGVC): Drink.
Vision Token:
(c) Individual Case 3.
(d) Individual Case 4.
Figure 7: Visualized individual-caat -cse performance analysis concerning the phenomenon of knowl- edge boundary drift. Outputs highlighted in red indicate factual errors.
During the discussion phase, we conducted a detailed case study under an 88.9% token reduction ratio on LLaVA-1.5-7BB progressively applying the DVTS and TGVC modules, as shown in Fig- 7. We observed an interesting pattern from the failure cases: when using only DYTS, knowl- edge boundary drift occurs in certain cases-i.e., the original model gives a correct answer, but after token pruning, the response becomes incorrect. However, after incorporating the TGVC module, the discarded tokens from DVTS are leveraged, and text-related visual tokens explicitly comple- mented. This prevents the loss critical information in the input to the model, thereby correcting the responses and mitigating knowledge boundary drift some extent.
For example, in Figure 7(a) and (b), the original model provides correct answers, but applying Vi- sionZip the DVTS module alone leads to emrrrs due the loss of key visual cues. In contrast, com- bining DVTS with TGVC enables the model to recover the correct responses, highlighting TGVC's role in restoring essential text-related tokens. We attribute this behavior to VisionTrim's two-stage
15

---

Published as a conference paper at ICLR 2026
Table 10: Additional performance-cfficinney experiments on different components of VisionTrim in LLaVA-1.557B (under an 88.9% token reduction ratio).
Methods
# Tokens
Total Inference Time + Prefill Time J
Accuracy 1
Vanilla
576
1303 (1.00x)
494 s (1.00x)
85.9
w/ DVTS (only)
64
670 (1.94x)
227 (2.18x)
84.6
w/ TGVC (only)
64
741 8 (1.76x)
263 s (1.88x)
85.0
DVTS TGVC (full)
64
685 s (1.90x)
235 s (2.10x)
86.2
token compression process: DVTS first extracts dominant visual tokens via a dual-attention mech- anism that captures global semantics and local spatial continuity, while TGVC complements dis- carded tokens using text guidance to mitigate the potential loss of critical visual information. Al- though DVTS pruning may remove some visual information, TGVC clusters and merges these to- kens to preserve critical text-related features. This design allows VisionTrim to maintain the original model's performance while minimizing knowledge boundary drift.
In future work, we plan to perform targeted, case-driven tests on critical instances prior de- ployment, including rigorous stability and reliability assessments for applications where stability paramount, such as Al-based disease diagnosis. This will ensure that models processed with Vision Trim are reliable and trustworthy for real-world deployment.
B THE USE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation of this paper, we utilized large language models (LLMs), specifically Ope- nAI's GPT-4 (Achiam al., 2023), assist with grammar refinement, sentence restructuring, and enhancing the overall readability of the manuscript. All scientific contributions, ideas, and evalu- ations presented in this work were independently developed and validated by the authors, without any involvement of LLMs.
C ADDITIONAL IMPLEMENTATION DETAILS
C.I COMPUTATIONAL BUDGET ESTIMATION
In typical MILMM framework (Liu et al., 2023; 2024a; Zhu et al., 2023), the LLM decoder utilizes the causal self-attention mechanism (Vaswani et al., 2017), where each token attends only to the the length of the input sequence, is
past tokens. The self-attention matrix A E 4xL, where computed model global dependence relationships among tokens follows:
A Attention Q, K) Softmax (Q. (Q.KT/V ///K/Vd
(13)
where Q E LR and K E Lx represent the query and key matrices, respectively, and the sealar dy the dimension these matrices. The computational complexity of each decoder layer includes the computation of the multi-head attention (MHA) mechanism and feed-forward network (FFN)
module. For the entire LLM, the computational metric FLOPs after the K-th layer is estimated as:
FLOPS
K (4nd2 + 2n 2 2ndm),
(14)
where n denotes the token number, d is the hidden state size, and is the intermediate size of FFN.
Therefore, the computational cost of MLLM exhibits a quadratic increase with respect to the quantity of input tokens. This highlights the significance of reducing the number of input tokens, particularly visual tokens (normally 20 times more numerous than system and question prompt tokens), while preserving the model's pioneering performance.
C.2 THEORETICAL COMPUTATION REDUCTION
In analyzing computational efficiency, we focus on the FLOPs associated with visual tokens in the MLLMSs Here, N denotes the initial number of visual tokens, and K + R represents the number
16

---

Published as a conference paper at ICLR 2026
Table 11: Token number settings for VisionTrim in LLaVA-1.5.
Retain tokens
Retain tokens
Retain 128 tokens
Retain 192 tokens
L.VVA-1.5
DVTS module TGVC module
DVTS module
TGVC module
DVTS module
TGVC module DVTS module
TGVC module
#Tokens
24
48
16
96
32
144
48
Table 12: Token number settings for VisionTrim in LLaVA-NeXT.
Retain 80 tokens
Retain 160 tokens
Retain 320 tokens
Retain 640 tokens
MaVA-NeXT
DVTS module
TGVC module
DVTS module TGVC module
DVTS module TGVC module
DVTS module
TGVC module
140
of tokens retained after our TGVC module (K tokens for the main visual feature and R tokens for the complement). For single transformer layer with hidden size d and intermediate size m, the theoretical FLOPs reduction ratio F, given token reduction rate = (K + R)/N, is calculated as:
Ny + 4(+:N)dd Ny
F= 1
(15)
8Nd + 4NVd + 6Ndm
The theoretical reduction formula above clearly illustrates that the computational cost of LLMs scales quadratically with the number of input visual tokens. This observation further underscores the importance of pruning and compressing redundant visual tokens during the forward propagation process of MLLMs.
D MORE EXPERIMENTAL RESULTS
D.1 TOKEN NUMBER CONFIGURATIONS
In our method, the DVTS module captures the primary visual information, while the TGVC module utilizes the textual information to further achieve vision complementarity. Here, we provide our detailed token count configurations in the DVTS and TGVC modules, for normal resolution and high-resolution image-based MLLMs and video-based MLLMs.
For LLaVA-1.5 (Liu et al., 2024a), we detail our four configurations: 32 tokens, 64 tokens, 128 tokens, and 192 tokens, summarized in Table 11. Regarding LLaVA-NeXT (Liu al., 2024b), which is capable handling high-resolution images, we specify the token counts as shown in Ta- ble 12. the case of Video-LLaVA (Lin al., 2023), which processes 8 frames per video with 256 tokens per frame, we perform token pruning to reduce the count to 17 tokens per frame, resulting in a total of 136 tokens for the entire video.
D.2 MORE QUANTITATIVE RESULTS
D.2.1 EXTREME TOKEN COUNT SETTING
Ultra-low token compression presents significant challenge. To evaluate the effectiveness of our proposed method, we present quantitative experimental results under extreme token count settings. As shown in Table 13, VisionTrim applied directly during inference without the need for additional training. while Vision'rrinğ involves efficient fine-tuning of the cross-modality attention module in TGVC and the projector, using only 1/10 of the LLaVA-1.5 training dataset. Notably, even with just 16 tokens, Vision Trim retains 90% of the original performance in training-free manner, demonstrating the robustness and efficacy of our approach.
Furthermore, we explore the extreme case of retaining only 1 token resulting in 99.8% reduction token count. Surprisingly, VrimonTrim still preserves approximately 82.8% of the original perfor- mance without any additional training. When fine-tuned with only 1/10 of the VA- training dataset, VisionTrimţ achieves 86.5% of the original performance. We also observe that as the num- ber of retained tokens decreases and the compression ratio increases, the performance gains from fine-tuning become more pronounced. These experimental results further emphasize the potential of our approach for flexible deployment and broader applicability in real-world scenarios.
17

---

Published as a conference paper at ICLR 2026
Table 13: Results of VisionTrim across different configurations of token counts on LLaVA-1..- 7B.
Method
GQA
MMB
MME
POPE
SQA
VQAYS
VQA Text
SEED
MMVet
VizWiz
Avg.
Upper Bound, S76 Tokens (100%)
61.9
64.7
1862
85.9
78.5
58.2
31.1
50.1
Vanilla
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
Retain Averaged 16 Tokens (L 97.2%)
51.5
57.7
1510
72.6
69.4
68.4
53.0
53.4
28.9
50.0
VisionTrim
90.0%
83.2%
89.2%
81.1%
84.5%
99.9%
87.1%
91.1%
91.1%
92.9%
99.8%
55.0
61.3
1612
78.1
70.2
72.3
55.4
55.9
30.2
50.5
94.3%
VisionTrim
88.9%
94.7%
86.6%
90.9%
101.0%
92.1%
95.2%
95.4%
97.1%
100.8%
(4.3%)
Retain Averaged Tokens
(L 98.6%)
49.7
53.0
1491
70.1
68.2
64.4
52.7
53.3
27.9
49.5
VisionTrim
87.4%
80.3%
81.9%
80.1%
81.6%
98.1%
82.0%
90.5%
90.9%
89.8%
98.8%
51.7
58.3
1525
75.2
69.9
70.2
55.3
54.4
30.4
91.8%
VisionTrim
83.5%
90.1%
81.9%
87.5%
100.6%
89.4%
95.0%
92.8%
97.7%
99.6%
1 (4.4%)
Retain Averaged 4 Tokens () 99.3%)
46.9
47.0
1460
64.0
67.8
66.6
$1.6
52.2
25.7
50.5
VisionTrim
84.5%
75.8%
72.5%
78.4%
74.5%
97.6%
84.9%
88.6%
89.1%
82.5%
100.8%
48.9
48.2
1496
65.4
68.6
72.5
52.0
53.6
29.6
51.4
88.0%
VisionTrim
79.0%
74.5%
80.3%
76.1%
98.7%
92.4%
89.3%
91.5%
95.2%
102.6%
(3.5%)
Retain Averaged
Tokens
(L (9.8%)
45.3
46.0
1441
64.1
66.7
65.7
50.4
51.7
24.5
49.4
VisionTrimm
82.8%
73.2%
71.1%
77.4%
74.6%
96.0%
83.7%
86.6%
88.2%
78.7%
98.7%
47.3
47.4
1435
64.6
68.5
70.2
51.3
53.3
29.3
50.7
86.5%
VisionTrim
76.4%
73.3%
77.1%
75.2%
98.6%
89.4%
88.1%
91.0%
94.2%
101.2%
(3.7%)
Table 15: Ablation experiment different in- serted LLM layers across various datasets on -LaVA-L.5-7B (a1 is set to 50%).
Table 14: Performance comparison with other methods on Intern VL2-2B (70.0% token reduc- tion ratio).
Settings
GQA
MMB
POPE
Method
OCRBench
OK-VQA
SQA
POPE
2, 77.3%
60.4
64.2
86.2
Vanilla
75.6
43.6
94.2
88.4
34,5
$ 8, a2 66.77%
60.2
64.0
86.0
68.8
43.3
93.7
87.9
12,
60.0%
60.5
63.8
85.5
TopV
72.9
43.0
94.2
88.0
34.6
16,
50.0%
58.4
62.9
84,4
20, 33.3%
57.3
63.3
84.1
k 24, 09 0.0%
56.5
60.5
83.2
D.2.2 ADDITIONAL TASK GENERALIZATION
In addition to the evaluation on classical MLLM benchmarks presented in the manuscript, we conduct more comprehensive experiments across various tasks to assess the generalization of Vi- sionTrrmm These additional evaluations encompass generative image captioning (as detailed in Table 17), fine-grained visual classification (as shown in Table 18), and multi-modal retrieval (as presented in Table 19). VisionTrim exhibits consistently superior performance across all bench- marks, effectively preserving or enhancing the original model's capabilities while outperforming ex- isting SOTA methods by significant margin. Notably, it achieves marked reduction in inference time alongside a considerable boost in model throughput, thus advancing the practical deployment of MLLMS in real- sol-world scenarios.
18

---

Published as a conference paper at ICLR 2026
Table 16: Experiment results of deploying VisionTrim on LLaVA-OneVision-77 and Qwen2-VL- 7B over single-/multi-meen and video benchmarks. For LLaVA-One Vision, we utilize 243 tokens per image (down from the original 729 tokens) and 66 tokens per video frame (reduced from the original 196 tokens). Similarly, for Qwen2-VL, approximately 1/3 of the original tokens are used.
Single-mmage Benchmarks
Mutts-inage Benchmarks
Video based Benchmarks
MMMU
MMStar
MMlllench
MMVet
Muirllench
Mantis
Q-ltench
Video MMI
Perceptiio Test Eposchema MY-Bench
ELaVA-OncVision
41.5
58.288155
Viioosraan
(3,51).2) (1.6 11 803 (1 39.0 1.5
41.4 (1
65.4 L.) 143(20.0)
59.5
1.33 (1
$7.0
623
56.5 (1.22)
QweeB-VL
54.1
60.7
80.7
62.0
63.3/89.
62.3
56.7
67.0
0.35 (.. (120) 6180 02)
(2 (.20000999
(2.0
(6.5())2))
68.8
Table 17: Evaluation the COCO Caption dataset with BLIP2.
Table 18: Evaluation Referring Object Clas- sification (ROC) Task with LLaVA-1.5-7BB
Method
BLEU@4
CIDEr
FLOPS//bhroghputt"
Method
Box
Mask
Scribble
Point
BLIP2
42.8
145.6
/37..3/28.5
Vanilla 55.4
56.2
54.7
53.0
Turbo
42.1
142.2
989.7 50.9
VisionZip
40.4
42.2
36.8
37.2
Ours
43.2
145.4
752.4/ 67.0
Ours
55.1
54.8
55.2
50,5
Table 19: Evaluation Multi-modal Retrieval task across the COCO dataset with BLIP2.
Method
Retrieval R@ R@5 Text-to-Image
FLOPP//Thrrouhhutt
A
Image-to-Text
BLIP2
85.4
68.3 87.7
717.5/ 10.8
Turbo
84.2 / 96.2
67.1 87.1
396.5 18.6
44.9% / 72.2%
Ours
85.1 / 97.0
68.0 87.5
265.2 27.55
63.0% 154.6%
D.2.3
MORE EFFICIENCY ANALYSIS
We conduct further experiments incorporating additional efficiency metrics related to hardware- specific optimizations, shown in Table 20. These experiments are performed an NVIDIA A40 GPU, aiming to simulate real- wellwwoldd deployment scenarios. The additional efficiency metrics clude prefilling time, total inference time, and memory usage. The results reveal that VisionTrim significantly accelerates MLLMs across various metries, outperforming the previous SOTA method, VisionZip, by substantial margin in both efficiency and accuracy. Notably, one particularly excit- ing finding is that with VriionTrim using 320 tokens, the 13B model achieves faster inference than the 7B model while maintaining superior performance.
D.2.4 VARIOUS SIZES AND MODEL FAMILIES
In the main paper, we provided evaluation experiments on Owen2-VL-7B and Qwen2.5-VL-7B across hoggle/mmuti-imagee and video benchmarks. Additionally, due to the page limit, conduct more efficiency analysis on LLaVA-NeXT-I3B in Table 20. Here we conducted more evaluation ex- periments comparing our VisionTrim against other methods on Intern (Chen al., 2024b), as shown Table 14, and LLaVA-OneVision different MILMS of various sizes and model families, VisionTrim consistently maintains strong performance and efficiency, and also significantly outperforms existing baselines across multiple benchmarks, even under substantial token compression. This further highlights VisionTrim's su- perior ability to preserve critical visual details through our proposed token pruning strategy and modules.
(Li et al., 2024a), as shown in Table 16. Across
D.3
MORE ABLATION STUDY
D.3.1 ABLATION STUDIES ON LAYER SELECTION.
In our default settings and the provided experiments, the proposed modules are inserted only be- tween the second and third layers of the LLM, as presented in Table 26. To further clarify the implementation details, conducted additional ablation experiments with different LLM layer lections, as shown in Table 15. Here, & denotes the inserted LLM layer, and a represents the token retention rate after the inserted layer. To maintain the same overall token retention ratio of 75.0%
19

---

Published as a conference paper at ICLR 2026
Table 20: More efficiency analysis on LLaVA-NeXT-7B/13B
Method
#Tokens
Prefilling Time (ms)
Total Inference Time (s)
Memory Usage (MB). Avg. Acct
Vanilla-7B
2880
112.4
2080
20874
96.2%
VisionZip-78
160
39.8 (3.2x)
1224 (2.1x)
17012 (- (-18.5%)
91.3%
Ours-7B
160
15.2 (7.4x)
547 (3.8x)
12482 (-40.2% %)
94.8%
Vanilla- 13BB
2880
156.5
2814
36840
100%
VisionZip-13BB
320
57.3 (2.7x)
1585 (1.8x)
30862 (-16.2%)
93.3%
Ours-138
320
28.0 (5.6x)
826 (3.4x)
25130 (-31.8%
97.4%
Table 21: Evaluation results on OCR-Heavy datasets with LLaVA-1.5-13B (32 tokens).
Method
OCRBench
OCRVQA
DocVQA
TextVQA
Avg.
Inference Time
Vanilla
54.9
45.2
59.4
100%
2394
VisionZip
42.3
31.0
53.6
71.4%
Ours
55.2
44.3
57.6
96.3%
Table 22: Comparison results on Video-LLaVA-75 between our VisionTrim for inter-frame token compression and the uniform sampling method (136 tokens).
Method
TGIF (Acc)
MSVD (Acc)
MSRVTT (acc)
ActivityNet (Acc)
Vanilla
47.1
69.8
56.7
43.1
Uniform Sampling
45.2
68.6
54.9
43.5
47.4
46.0
throughout the full LLM decoding stage, we adaptively adjust a based the inserted layer k. We observed that inserting the proposed modules into the shallow layers typically results in better per- formance.
D.3.2 TRADE-OFFS FOR OCR-HEAVT TASKS
To strengthen the practical applicability of VisionTrrim, we further include experiments on OCR- heavy benchmarks, shown in Table 21. The results indicate that VisionTrim surpasses the previ- ous SOTA method VisionZip, delivering 96.3% of the original MLLM performance while achieving 3.4x inference speedup. This is accomplished with 94.4% token reduction ratio, effectively bal- ancing accuracy and inference efficiency.
D.3.3 INTER-FRAME TOKEN COMPRESSION FOR VIDEO-BASED MLLMS
Most video-based MLLMs often rely on uniform sampling of a fixed number of frames to achieve video understanding, which overlooks inter-frame token correlations. We leverage our VisionTrim to address temporal redundancy in videos through inter-frame token compression. Specifically. inspired by Chat-UniVi (Jin al., 2024a) and LongVU (Shen al., 2024), we first calculate the average similarity Sim' for each frame relative to all others using visual features extracted from the vision encoder like CLIP (Radford et al., 2021). We then select top- frames with the lowest Sim' values as cluster centers and assign the remaining N - K frames to their nearest cluster. Similar to fer-frame token merging, we consolidate non-essential frames in each cluster by merging K K-nearest neighbor tokens across frame features. The comparison results are reported in Table 22.
D.3.4 COMPARISON WITH DIRECT TEXT-GUIDED METHODS
Direct text-guided methods (Ju et al., 2024; Shi et al., 2024) can hinder multi-round interactions in MLLMs and potentially lead to hallucinations. Our two-stage approach addresses these issues by initially preserving core visual tokens through the DVTS module and subsequently aggregating text- related tokens using the TGVC module. As shown in Table 23, VisionTrim outperforms the direct text-guided method, particularly in benchmarks related to hallucinations and OCR-related tasks.
20

---

Published as a conference paper at ICLR 2026
D.3.5 COMPARISON WITH RANDOM PRUNING
We also provide the experiments to compare VisionTrim with indiscriminate random pruning, as shown in Table 24. In contrast to the random pruning, our approach achieves significant performance improvement across all benchmarks.
D.3.6 GENERALIZATION TO NON-CLIP VISION ENCODERS
To further assess the generalizability of the VisionTrim framework, test it on non-CLIP vision coders, such as DINOv2 (Oquab et al., 2023). Specifically, we first replace the CLIP vision encoder in LLaVA with DINOv2 and fine-tune LLaVA accordingly. Subsequently, to the fine-tuned MLLM to evaluate its generalizability. The experimental results are presented in Table 25.
apply VisionTrim
D.3.7 HYPERPARAMETER CONFIGURATIONS FOR REPRODUCTION
In our proposed DVTS and TGVC modules, we maintain consistency in hyperparameters across different MLLMs and benchmarks. To facilitate the reproduction of our VisionTrrm'ss experimental results, we provide detailed hyperparameter settings in Table 26. For MILM parameters, we use the default settings each base MI.LM, as our method applied during inference on pre-trained models.
D.3.8 DIFFERENT COMPONENTS IN VISIONTIRM
We conduct additional ceaturmance-fffccienne experiments on different components added to the final model, shown in Table 10. We observe that using only the DVTS module further improves inference efficiency, but leads to a larger performance drop because text-related information is not considered. In contrast, using only the TGVC module preserves performance but yields smaller improvement in inference efficiency due to the increased token-level similarity computations involved in clustering. By combining both modules, the final model achieves better trade-off between performance and efficiency.
effectively,
D.4 MORE CASE STUDIES
Figure 8 showcases several examples of VisionTrrim's application in both image- and video-based understanding tasks. Unlike other methods that typically rely on a larger number of visual tokens, VisionTrim effectively captures visual details with a minimal token set. For instance, VriionTrim successfully distinguishes between different characters in an image and accurately identifies the license plate number on bus.
In the context of video understanding, Video-ChatGPP (Maaz et al., 2023) utilizes fixed number of visual token representations across the entire video, which limits its ability generate compre- hensive descriptions that capture temporal dynamics. Video-LLaVA (Lin et al., 2023) processes eight frames per video, allocating 256 tokens per frame. Nonetheless, this approach inherently con- tains substantial visual redundancy, potentially leading to hallucinations. For example, in the of a soccer game video, Video-LLaVA misinterprets the player's footwork and interactions with defenders. In contrast, VisionTrim effectively retains crucial visual information while significantly reducing redundancy. Our approach not only enhances visual comprehension but also optimizes efficiency, making more practical and viable solution for efficient multimodal interaction in real-world applications.
D.S MORE VISUALIZATION RESULTS
D.5.1 TOKEN REDUNDANCY VISION ENCODING STAGE
MLLMs typically leverage the encoded features from the penultimate layer of the vision encoder. To further illustrate the redundancy within vision encoders, present additional visualization results. As shown in Figure only a small subset of tokens receives high attention to capture the key visual information, while the majority of visual tokens allocated minimal attention and contribute little
21

---

Published as a conference paper at ICLR 2026
Table 23: Experimental results comparing our VisionTrim with the direct text-guided method on LLaVA-1.5-13B (32 tokens).
Method
POPE
MME
OCRBench
DocVQA
OCRVQA
TextVQA
Vanilla
86.2
1875
331
45.2
54.9
59.4
Direct Text Guidance
59.5
1240
154
28.4
37.5
47.4
Ours
84.4
1685
297
44.3
55.2
57.6
Table 24: Comparison of Vision vs. Ran- dom Pruning on LLaVA-1.5-78 (32 tokens).
Table 25: Evaluation results on LLaVA-1.5-77
using DINOv2 as the vision encoder (32 tokens).
Method
GQA
MMB
POPE
SQA
VQA
Method
SEED
MMB
MME
POPE
SQA
Random Pruning
40.8
51,4
62.8
60.5
56.7
Vanilla
60.6
64.2
1404
86.7
70.0
Ours
55.3
60.2
77.5
70.1
72.6
Ours
58.5
63.0
1368
83.4
70.7
Table 26: Detailed hyperparameter configurations in VisionTrri.
Parameter
Value
Kernel-size in LTAM
3 3 (0.3,0.3,0.5)
(101 W2, W3) in LTAM
Inserted layer ViT (pre-LLM) Inserted layer in LLM (within-LLM) Token count retained in DVTS ys. TGVC
23rd (total 24 layers)
2nd (total 32 layers)
3:1
Token retention rate 01 in pre-LLM stage
50%
Token retention rate Q2 in within-LLMy stage
Dynamically adjusted
to the overall visual content. This observation underscores the significant redundancy among visual tokens, emphasizing the necessity for their compression.
D.5.2 TOKEN REDUNDANCY IN LLM DECODING STAGE
To examine the token redundancy in the LLM decoding stage, we provide visualizations of the attention scores at the 16th and 32nd layers of the LLM as illustrated Figure 10 and Figure 11. Consistent with the findings from the vision encoding stage, only a limited number of tokens receive high attention, while most visual tokens are largely ignored. This suggests that many of these less relevant visual tokens should be pruned during the LLM propagation phase.
D.5.3 ANALYSIS LAYER-WISE ATTENTION DISTRIBUTION
Based on the observations discussed above, a natural question arises: What are the underlying causes of redundancy in visual tokens? In this section, present comprehensive analysis of the attention distribution of layers within both vision encoders and LLMs.
As illustrated in Figure 12, attention in the shallow layers is broadly distributed across the entire image. However, as the model progresses to the middle layers, attention rapidly consolidates around smaller subset of tokens. In the deeper layers. it becomes increasingly concentrated on limited number of dominant tokens, reaching its peak concentration by the 23rd layer, which is then utilized for visual token extraction for the LLM. A similar trend is observed during the LLM decoding phase. As shown in Figure 13, attention initially distributes relatively evenly across all visual tokens. However, as the number of layers increases, the LLM's attention progressively narrows focus on only few tokens, while neglecting the majority of visual tokens.
To assess the effectiveness of our proposed DVTS and TGVC modules, we visualize the attention maps across all 32 layers during the LLLM decoding stage, both with and without VisionTrim. We set VisionTrim to reduce the vanilla 576 visual tokens to 128 tokens in the vision encoding stage before being input to the LLM. For comparison, we randomly select 128 visual tokens for the base- line LLM. As shown in Figure 14, the vanilla LLaVA-115-78 model exhibits the aforementioned redundancy in visual tokens, particularly after the shallow layers (i.e., the first two layers), where attention concentrated on only a small subset of tokens, largely disregarding the majority of visual tokens. In contrast, depicted in Figure 15, Vision enhances the cross-modal alignment be-
22

---

Published as a conference paper at ICLR 2026
Image:
Instruction: What color is the hat worn by the man who is wearing white top and holding a frishee?
GPT-4o: Yellow and white mixed.
LLaVA-1.5: White.
Vision Token: 576 c 110 mg
Instruction:
Pleuse describe the content the video.
11s Video
VisionZip: Yellow.
Vision Tokes: 126 80 me
Ours: Yellow and white mixed.
Vision Taken:
ms
The video shows match between the blue and yellow teams, with the blue ream's number 10 player dribbling past severnl defenders and scoring brilliant goal.
Video- Giteo-ChatGiT:
Image:
Instruction:
What license plate number of the bus in the picture?
Token: 256-100-330
O 0.6
Video-L LLa The video shows intense foothall match between the blue team and the yellow team. A player from the blue team. wearing the number 10 shirt, receives pass from his teammate, dribbles the past defenders, displaying powerful skills, finishes with remarkable shot on goal.
GPT-40: SBS506SD.
Taken: 286-8-2048
LLaVA-1.5: $BS50660.
Vision Tokeni 576 100
The video features intense soceer match between blue and yellow teams. The player dribbling the ball, wearing the number 10 jersey for the blue leam, showcases his exceptional skill by smoothly bypassing deffenders from the yellow team players number and
Ours:
VisionZip: SHS50661.
Visten Token: 128 3 ms
Ours: SBS506SD.
Vision Taken:
O
before delivering brilliant shot on goal.
Taken: 0.4
(a) Case image understanding
(b) Case of video understanding
Figure 8: Visual examples showcasing VisionTrim's ability to accurately capture detailed visual information in both images and video. Outputs highlighted in red indicate factual errors.
Table 27: Open-source resources utilized in this paper.
Name
License
URL
VextVQ
hotper /tteee/////////////oo// orgg/
VQA Dataset
BSD License
hotps: wtw.. www//ooo//////eee Ctnp//IVViss a on Lono-onn/V
ScsenceUA Datset
Licease
hotpa: h/tpt//c....bcco//////o con/ Jupannced SciencegA
POPE Dataser
MIT License
br: http:///ggl //t.t////......../// con/ brwwgn//rPee
LLaVA-1.5
Apache License
hot / h//gttacbbbb.... con/ /haot lar- LLLVV
LIaVA-NeXT
Apache License 2.0
hrtt:w://gg www.....bbb..................... con/ LLAVA VL/Litaa Neke
Video-LLaV
Apache License
hotper sttpee//ggttet b.c.g/t....ooooooooo.ooooooo www.coo//////ee Srt-Y wwtpo//////ddddeeeeoeeee LLaV
MMBench Dutaset
Apache License
hotpo: httl://///ttt con/ oppn- compas /MMBBnch
MME
Apache Licesse 2.0
hitps: : //cc. con/ /Badyy (/adyy////ae Awassaa- Mul Lmodall- Large- Lanquage aaaag---ooddaaa
MM-Ver
Apache Licesse
hudgges
uun// yoo Lhwe/ Vell
SEED Bench Dutaset
Apache Licesse 2.0
hitps: //4 tlach con/ boiDzaccn/PO0
COCD Dataser
Creative Commons Amiibution 4.D
gus /uusdda assel oorg/ Hhoee
Vi,Wir Duiseet
Creative Commons Arsibution 4.0
hottpe //vi mat it org/tt wwseg/tkkeeeeeeeee ske- ende--nd---n ded--ettee set set /urge/
GQA
hotpe: // www.torrd...ooo.....oo...o...... wwwp//oppppceecooooooooooooo wtt.t///ooooroooeeooeeeeo pgp wtt.///w/.//////////////////eeta hami
OCR-VQA
hctps:/ / woo.-vaa........oo...o..e....... githhbb.aa 11/
tween visual and textual tokens, enabling the LLM to more effectively focus on the retained image tokens while preserving the most salient visual information. This improvement not only maintains model performance but also significantly boosts inference speed and efficiency.
E BROADER IMPACTS
This paper introduces training-free method, named VisionTrim, designed to accelerate Multimodal Large Language Models (MLLMs) through two plug-and-pla modules. These modules effectively filter the essential visual tokens, maintaining both global semantics and local continuity, while lever- aging textual information to guide token merging, thereby enhancing the visual-textual alignment. On the positive side, our approach has the potential to significantly benefit the efficient deploy- ment of MLLMs for real-world image and video understanding tasks, offering a clear reduction in training and inference costs while maintaining competitive performance. However, due to the inherent robustness challenges of large multimodal models, some erroneous outputs may result in misinformation safety concerns (Wei et al., 2023; 2024). To mitigate these risks, recommend
23

---

Published as a conference paper at ICLR 2026
implementing stringent security protocol to address potential failures of our approach in practical multimodal applications.
ASSET LICENSE AND CONSENT
We conduct an extensive evaluation across 10 widely-used image-hased benchmarks and 4 popular video-based tasks to assess the multi-modal understanding and reasoning capability of our proposed Vision Additionally, we utilize 66K mixture nnsruuttoo-follloinng data (a subset of the original 665K VLa dataset) for efficient instruction tuning. All datasets publicly available and free for academic research. Table 27 lists the resources used in this research work along with their associated licenses.
24

---

Published as a conference paper at ICLR 2026
Attention Score
06
004
VT FAMS
CINS
6S9
Figure 9: Visualization of redundancy in the penultimate layer of the vision encoder (CLIP Model).
25

---

Published as a conference paper at ICLR 2026
RI O PANS
Attention Score
-04
-02
CALIFORNIA A
Figure 10: Visualization of redundancy in the 16th layer of LLM.
26

---

Published as a conference paper at ICLR 2026
10
VI FANS
Attention Score
CALIFORNIA
Figure 11: Visualization of redundancy in the 32nd (final) layer of L.M.M
27

---

Published as a conference paper at ICLR 2026
Layerl
Layer2
Layer3
Layer4
100
Attention Score
16
Layer5
Layeró
Layer7
Layer8
-12
Layer9
Layer10
Layerl1
Layer12
L 3
L
Layerl5
Layerl6
Layerl7 7
Layerl8
Layer19
Layer20
Layer21
Layer22
Layer23
Layer24
Figure 12: Visualization of attention distribution change in vision encoder (CLIP Model),
28

---

Published as a conference paper at ICLR 2026
Layerl
Layer2
Layer3
Layer4
Layer5
Layeró
Layer7
Layer8
Layer9
Layer10
Layerl
Layer12
Attention Score
Layerl3 3
Layerl4 14
Layer15
Layerl
Layer 7
Layer 8
Layer19
Layer20
Layer21
Layer22
Layer23
Layer24
Layer25
Layer26
Layer27
Layer28
Layer29
Layer30
Layer31 |
Layer32
Figure 13: Visualization changes in attention distribution across all 32 layers in LLM..
29

---

Published as a conference paper at ICLR 2026
Layerl
Layer2
Layer3
Layer4
Layer5
Laycró
Layer7
Layer8
Layer9
Layer10 0
Layer
Layer 1 2
Attention Score
02
L
Layerl4
Laycr15
Layerl
Layer17
Layer18
L
Layer20
Layer2
Layer22
Layer23
Layer24
Layer25
Layer26
Layer27
Layer28
Layer29
Layer30
Layer31
Layer32
Figure 14: Visualization of attention maps across all layers vanilla LLM processing.
30

---

Published as a conference paper at ICLR 2026
Layerl
Layer2
Layer3
Layer4
Layer5
Layeró
Layer7
Layer8
Layer9
Layer10
Layerl
Layer12 2
006
Attention Score
004
Layer 4
Layer15 5
Layerl 6
Layer17 7
Layer18
Layer19 9
Layer20
Layer21
Layer22
Layer23
Layer24
Layer25
Layer26
Layer27
Layer28
Layer29
Layer30
Layer31
Layer32
Figure 15: Visualization of attention maps across all 32 layers in LLM processing with our proposed
UisionTrim
31


Reviews:
[{"review_id": "A32hkxUbPh", "reviewer": "Reviewer_RMDZ", "summary": "The manuscript identifies two drawbacks of existing token reduction methods:\n\n(i) they focus on only one of visual encoding or LLM decoding processes,\n\n(ii) they neglect the role of text queries during the token pruning process.\n\nThe authors propose VisionTrim to overcome these drawbacks, where tokens are selected based on both visual and textual information (using DVTS and TGVC respctively), and the proposed modules are applied both after visual encoding and during LLM decoding. \n\nThe proposed solutions for both drawbacks are validated in the experiment section. Both text-guided token selection and multi-stage application of token selection strategy are shown to be effective in improving the performance of low-token scenario. Finally, compared with existing approaches, VisionTrim demonstrates competitive performance.", "questions": "1. I am not sure about the term CUDA time, is it the same as latency, in terms of the time spent on processing the samples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is well-motivated. Two key drawbacks of existing token pruning methods are identified, respectively the neglection of the role of text in visual token selection, and the application of token pruning on only one stage of visual encoding or language decoding. This provides a good review of existing methods and provides guidance to future research on token pruning for training-free mllm acceleration. \n\n2. The component designs are reasonable and thoroughly ablated in Table 4 and 5, including the local affinity in visual token selection, text-guided selection, and multi-stage application of token pruning, showing sound effectiveness of these components. \n\n3. The performance of VisionTrim is competitive against existing token pruning methods.", "weaknesses": "1. Despite the effectiveness of the proposed strategies resolving the two drawbacks of token pruning methods, in Table 5, it seems the ensemble strategy is playing a vital role in the strong performance. This is intriguing and I believe it deserves more explanation, both in terms of how the adaptive weighting mechanism is designed, as well as why it is so effective. \n\n2. Different components added to the final model may have different latencies, which the manuscript fails to provide. \n\n3. The usage of the first generated token as the natural measure of global semantic significance over all image tokens in LLM decoding stage does not make sense to me. Why is this strategy chosen?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wz7eQxnGE4", "reviewer": "Reviewer_r8MU", "summary": "The paper proposed to prune the visual tokens to accelerate MLLM computation, referred as VisionTrim, including two training-free plug-and-play modules:\n\n1)\tDonimant Vision Token Selection (DVTS), which defines a global importance score that is the average attention scores w.r.t. the [CLS] token, and a local importance score that is the average affinity score within its neighboring tokens. These two importance scores are averaged by their variance to select the top K informative tokens.\n\n2)\tText-Guided Vision Complement (TGVC), which assumes an associated textual descriptions/prompts related to the image and uses the CLIP score between the visual and text tokens to select top-R tokens as clustering centers.\n\nThese modules can be plugged in to the vision encoding stage or the LLM decoding stage. Experiments on LLaVA-NeXT-7B, Video-LLaVA-7B, and Qwen2.5-VL-7B models show that the proposed VisionTrim achieve competitive performance on multiple tasks after pruning 78% to 94% visual tokens.", "questions": "How to choose top-K in DVTS  and top-R in TGVC?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The proposed DVTS and DVTS models are easy to implement and reproduce, which empirically show competitive performance even after pruning a large portion of visual tokens. The experiments are thorough and convincing. This may be inspiring to practical MLLM systems.", "weaknesses": "What if these is no associated textual prompts available? Or the textual clues are noisy and misleading?\n\nGiven the same image, if the textual prompts are varying in different tasks, the image has to be processed per textual prompt since the TGVC results may be quite different, right?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J31JK9pMB3", "reviewer": "Reviewer_M569", "summary": "-   The paper introduces VisionTrim, a training-free framework for reducing vision tokens in MLLMs are a way of accelerating inference.\n-   VisionTrim proposes two modules: Dominant Vision Token Selection (DVTS) and Text-Guided Vision Complement (TGVC).\n\t-   DVTS: Aims to preserve visual integrity during visual token compression by first identifying dominant visual tokens.\n\t-   TGVC: Utilize text prompts to cluster remaining vision tokens through CLIP embeddings. Each cluster is then merged.\n-   Experimental results show that VisionTrim improves on existing methods by around 1-2% averaged across benchmarks.\n-   Furthermore, there is also a significant reduction in CUDA Time, FLOPS, and KV cache memory usage.", "questions": "-   The paper describes TGVC as a training-free plug-and-play module. However, Figure 3(a) shows a [fire] symbol over TGVC, while a [snowflake] symbol over other modules, such as the vision encoder and projector, which are not explained. It is unclear whether this was an intended design or an oversight. Could the authors clarify this? (Fig. 1 as well)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-   The redundancy in vision tokens in MLLMs is definitely an area of concern.\n-   Unlike existing methods, which compress vision tokens in either the vision encoder or the LLM decoder, VisionTrim proposes a unified framework that can be applied to both modules.\n-   Experimental results show general improvement over existing methods by around 1-2% on average score across benchmarks, with more pronounced improvements at a lower number of vision tokens per image, showing its effectiveness at preserving visual information.\n-   Paper is generally easy to read (except for some equations that lack elaborations). The diagrams and tables illustrating the methods are clear and well-drawn.", "weaknesses": "-   The paper positions itself as improving inference latency of MLLMs and reports on CUDA time for decoding, which is an odd metric to use for inference latency (most papers report end-to-end inference latency or time to first token). However, to my knowledge, clustering and pairwise similarity between tokens can be slow, given the high dimensions in both the vision token count and the embedding size. The authors did not mention whether clustering is performed on the CPU or GPU. The authors did not disclose the overall end-to-end latency; with only the CUDA time reported, I remain sceptical that this method is an effective approach at reducing inference latency.\n-   The proposed VisionTrim is a unified framework that can be plugged into both the vision encoder and the LLM decoder. However, there is a lack of experiments exploring the configuration: what percentage of vision tokens are pruned in the vision encoder / LLM decoder, respectively? At what stages are they pruned? What are the performance/efficiency trade-offs in pruning in the vision encoder vs in the LLM decoder? The lack of details would make it difficult to reproduce experiments for future work.\n-   Minor presentation issues: some equations, especially those in local spatial continuity, are not well explained. The intuition behind these could be further elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vk1VtUvErL", "reviewer": "Reviewer_8p5d", "summary": "This work proposes a training-free token pruning method for model acceleration. Specifically, the authors designed a two-stage plug-and-play module. First, DVTS uses a pre-trained CLIP-type encoder to prune vision tokens. Then, the TGVC module clusters and merge the pruning vision tokens for ensure semantically relevant details are preserved. Result shows that the proposed method can still perform well even when pruning 88% of the tokens.", "questions": "1. Does the proposed pulg-in module have the same weights in the encoder and decoder stages?\n2. Are these components plugged into each layer?\n3. Is top-k selection also mandatory during training? If so, how can author ensure it can be effectively generalized to different selected scales during inference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The experiments are comprehensive, as the authors tested the proposed method on more than just a single model. Furthermore, the authors evaluated the use of a key-value cache. Diagrams and qualitative results are clear and intuitive. The findings of this paper are highly instructive for industrial applications, particularly the key proposal to tailor both the encoder and decoder.", "weaknesses": "1. Academic writing requires standardization. It appears that the author does not use citation formatting well. Most citations are in the form of \"name et al (year),\" which makes them difficult to read.\n\n2. To my limit knowledge, token pruning or quantization methods can ensure that overall performance does not drop significantly. However, they may introduce knowledge boundary drift, meaning that answers that were previously correct may now be incorrect, and vice versa [1]. This possibility could have negative consequences in scenarios where critical questions must be answered correctly. If possible, could the authors conduct a simple analysis to see whether the proposed method leads to changes in individual case performance?\n\n[1] Sun, Yizheng, et al. \"Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study.\" arXiv preprint arXiv:2503.06794 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)