iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
Authors: Lianyu Hu
Liqing Gao
Fanhua Shang
Liang Wan
Wei Feng
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models
Model efficiency
Token merging
Primary area: foundation or frontier models, including LLMs
TL;DR: Efficient Large Vision-Language Models with two-stage visual token merging strategies in the image encoder and LLM.

Abstract: Recent methods have made notable progress in accelerating Large Vision-Language Models (LVLMs) by exploiting the inherent redundancy in visual inputs. Most existing approaches, however, focus narrowly on reducing image tokens before or within the Large Language Model (LLM) stage to lower computational cost. This overlooks other major bottlenecks, particularly the image encoder, which itself requires substantial computation. As a result, these methods fall short of achieving true end-to-end acceleration. Importantly, the image encoder is the primary contributor of input tokens to the LLM. Thus, reducing visual redundancy at the encoder stage not only speeds up the encoder itself but also significantly lightens the workload for the subsequent LLM. Motivated by this, we investigate how to jointly optimize the image encoder and the LLM along with other LVLM components for comprehensive acceleration. To mitigate the risk of performance degradation from token reduction, we propose a novel token merging strategy that recycles useful information from otherwise discarded tokens. Our approach, iLLaVA, delivers consistent improvements across both image and video understanding tasks, achieving up to a 2$\times$ throughput boost and a 4$\times$ reduction in prefilling time. Notably, iLLaVA enables a larger model (e.g., InternVL-2.5 26B) to surpass a smaller counterpart (e.g., InternVL-2.5 8B) in both accuracy and efficiency. Extensive comparisons with state-of-the-art token pruning and merging techniques demonstrate the clear superiority of our method. Finally, we provide detailed visualizations for the merging steps of iLLaVA , offering deeper insights into how different LVLM components contribute to efficient computation.

Published as a conference paper at ICLR 2026
ILLAVA: AN IMAGE IS WORTH FEWER THAN 1 INPUT TOKENS IN LARGE MULTIMODAL MODELS
3
Lianyu Hu', Liqing Gao", Fanhua Shang', Liang Wan Wei Feng 1.3
School of Computer Science and Technology. Tianjin University,
2 Schooo of Computer Science and Technology. Tiangong University,
3 "ey Research Center for Surface Monitoring and Analysis of Relics, State Administration of Cultural Heritage
[hly2021] @tju.edu.cn, {Iggao) wittegng.ndduu.ooooooo (fhshang, lwan, wfeng) ) @ @tju.edu.cn.
O Code: https://github wttp://...oooohaaaaaaaaaaeaaaaaa http://ggggttta.ocooooaaaaaaaaat
ABSTRACT
Recent methods have made notable progress in accelerating Large Vision- Language Models (LVLMs) by exploiting the inherent redundancy in visual in- puts. Most existing approaches, however, focus narrowly reducing image to- kens before or within the Large Language Model (LLM) stage to lower com- putational cost. This overlooks other major bottlenecks, particularly the image encoder, which itself requires substantial computation. As a result, these meth- ods fall short of achieving true end-to-end acceleration. Importantly. the image encoder is the primary contributor of input tokens to the LLM. Thus, reducing vi- sual redundancy at the encoder stage not only speeds up the encoder itself but also significantly lightens the workload for the subsequent LLM. Motivated by this, we investigate how jointly optimize the image encoder and the LLM along with other LVLM components for comprehensive acceleration. To mitigate the risk of performance degradation from token reduction, we propose a novel token merg- ing strategy that recycles useful information from otherwise discarded tokens. Our approach, iLL delivers consistent improvements across both image and video understanding tasks, achieving up 2x throughput boost and 4x reduction in prefilling time. Notably, iLLaVV enables larger model (e.g.. Intern VL-2.5 26B) surpass smaller counterpart (e.g., InterVVL-2.. 8B) in both accuracy and ef- ficiency. Extensive comparisons with state-of-the-art token pruning and merging techniques demonstrate the clear superiority of our method. Finally, we provide detailed visualizations for the merging steps of iLLa offering deeper insights into how different LVLM components contribute to efficient computation.
INTRODUCTION
Over the past several years, Large Vision Dision-Language Models (LVLMs) Li et al. (222a): Bai et al. (2023): Chen et al. (2024c) have demonstrated tremendous progress and enabled promising appli-
cations in various downstream tasks including cross-model retrieval Alayrac al. (2022); Li et al.
(2023), medical image analysis Li et al.] (2024a); [aab et al.] (2024), and video understanding Maaz al. (2023): Wang al. (2024c). Within these impressive models, images are usually first divided into patches and then transformed into token sequences, finally fed into a Large Language Model (LLM) and concatenated with text instructions to generate the desired textual outputs. The direct combination of image tokens and textual contexts already shows impressive performance over broad range of tasks, which enables the emergence of powerful AI tools like GPT-4o, Gemini 1.5 pro, E3 and Kling.
Despite their remarkable success, LVLMs face severe challenges in terms of computational com- plexity and resource demands. These models typically rely on self/crros--ateention mechanisms to attention operations makes
capture input information. However, the inherent (nn2) complexity computation scale quadratically with the number of tokens. For image and video inputs, which can easily expand into thousands or even tens thousands of tokens, this burden becomes especially
"Liang Wan is the Corresponding Author

---

Published as a conference paper at ICLR 2026
prohibitive. In practice, running inference with 144B-parmeeter LVLM can require around 80GB of memory Xu et al. 2024), far exceeding the capacity of many institutions with limited computing in- frastructure. Moreover, the slow inference speed of LVLMs raises serious concerns for applications demanding real-time or near-real-time responses Chen et al. (2024a); Yang et al. (2025). Therefore, improving the inference efficiency of L.V.Ms is not only technical necessity but also critical step for enabling their widespread deployment in real-world scenarios.
Several works Chen et al. (2024a); Xing et al. (2025); Yang et al. (2025; Zhang et al. (022aa)
Alvar et al. (2025) have explored reducing image tokens for inference acceleration, motivated by the observation that visual features often contain substantial redundancy. While these approaches have achieved notable progress, they narrwly focus on pruning or compressing tokens before or within the LLM stage to reduce its computational overhead. However, this overlooks another criti- cal bottleneck: the image encoder. As we demonstrate in this paper, the image encoder constitutes a non-negligible portion of the overall computation in LVLMs, and its cost cannot be ignored. More importantly, the encoder is also the largest contributor of tokens to the LLM. Thus, reducing dundancy at the encoder stage not only accelerates the encoder itself but also yields compounded efficiency gains by substantially reducing the input load for the subsequent LLM.
To enable comprehensive acceleration of LVLMs, we propose iLLaVA, which jointly optimizes the image encoder and the LLM which constitute two of the most computationally demanding compo- nents in LVLMs. Our design is built on two key innovations. First, unlike prior methods that only reduce tokens within the LLM, VL performs token reduction in both the image encoder and the LLM. This dual-stage reduction substantially decreases the overall computational budget of the entire model. Second, to mitigate the risk of performance degradation from token reduction, we in- troduce a novel token merging strategy that recycles potentially informative content from discarded tokens, thereby preserving critical input information. Experiments upon over 10+ image and video understanding tasks show that iLLaVA successfully maintains >95% performance over various to- ken reduction ratios and achieves significantly better results compared to state-of-the-art methods. Furthermore, iL.LaVA could 2 the throughput and reduce the prefilling time by 4x, and enables larger model (c.g.. Intern VL- 26B) to outperform a smaller model (c.g.. Intern VL- 8B) with both better performance and higher throughput. Comparison of iLLaVA with recent competitive approaches demonstrates the superiority of iLLaVA LA over computing efficiency and model accuracy.
2
RELATED WORK
2.1 LARGE VISION-LANGUAGE MODELS
Our work is closely related to the surge of LVLMs. Traditional methods Ramachandram & Taylor (017)): Xu et al. (2023b); Ochoa et al. (2017) usually collected large vision-language datasets and learned joint representations between vision and language from scratch to handle different tasks. These methods usually worked well in in-domain data but performed inferiorly in out-domain data that requires common sense and world knowledge.
Later, powered by the abundance of high-text data, LVLMs Li et al. (2024a) ; Wang et al. (2024b);
Bai et al. (2025): y: Chen et al. (2024c) shows impressive performance across a wide range of image understanding Yue et al. (2024a); Liu et al. (2024); Masry et al. (2022); Mathew et al. 2022 2021) and reasoning tasks Li et al. (2024b); Chen al. (2024b). These methods usually first transform the input images patches, and then feed them into a vision-transformer-
image encoder. The extracted features are sent into a projector for dimension projection, whose outputs are further con- catenated with the system prompts and user instructions to serve inputs for the language model to generate textual outputs. As the attention mechanism with computational complexity O(z²) is both used in the image encoder and language model in LVLMs, they have consume high computational resources and own high inference latency when faced with long input sequences. Our method is de- ployed upon the state-of-the-art Qwen2.5-VL Bai al. (2025). Intern Intern VL-2.5 Chen et al. (2024c), Minicpm-v2.6 Yao et al. (2024) and LLa LLaVA- WA--AAA-Oiiiione Li ct al. (022a) series to accelerate models.
2.2 TOKEN REDUCTION
Token reduction has been widely explored in both computer vision Meng et al. (022);; Chen et al. 2023): Pan et al. (202); Ryoo et al. (202T); Rao al. (202T) and natural language processing
2

---

Published as a conference paper at ICLR 2026
Raw image
Layer
Layer
Image enooder
Projector
ai Language model
45%
1%
54%%
Multi-Image lask
24%
1%
75%
Layer
Layer
Single-image lask
0%
20%
40%
60%
80%
100%
(a) Visualizations for attention scores in VIT
(b) Inference time for each component
Figure 1: (a) Visualizations for the attention scores of different layers in the image encoder. (b)
Proportion of inference time for each component in LVLMs across different tasks.
(NLP) Goyal et al. 2020): Kim & Cho (2020); Lassance et al. (022)) However, these methods
usually require training, while our method can done a traming-free free manner. In multimodal learning, series of methods tried to prune the tokens of intermediate layers for model acceleration. FastV Chen et al.] (2024a) first proposes to select the TopK-activated tokens within the language model to accelerate the forward pass in pioneering manner. Sparse Zhang al. (2025b) proposes to select relevant text tokens to rate the significance of visual tokens using self-attention matrices, and then prunes visual tokens using the proposed strategy maximize sparsity. Faster- VLM Zhang et al. 2025a) utilizes the attention scores between the [CLS] token and image tokens from the visual encoder conduct token pruning. VisionZip Yang al. (2025) selects set of informative tokens before the language model as inputs and merges beneficial information from the discarded tokens. PyramidDrop Xing et al. (2025) partitions the LVLM into several stages and drops part of the image tokens at the end of each stage with a pre-defined ratio. DivPrune /Alvar al.) (2025) first formulates token pruning as Max-Min Diversity Problem (MMDP) and then solves the MMDP to obtain the selected subset and prunes the rest. AdaFV Han et al. (2025) introduces self-adaptive cross-modality attention mixture mechanism to dynamically leverage visual saliency and text-to-image similarity to select informative visual tokens. AIM Zhong et al. (2024) first con- ducts token merging before the LLM to decrease the input length, and then gradually performs token pruning for the intermediate LLM layers to reduce image tokens. Despite considerable progress on model efficiency, these methods typically just reduce image tokens within or before the LLM, and overlook other computationally heavy components in LVLMs. In this work, go beyond this lim- itation by jointly accelerating both the image encoder and the LLM, enabling more comprehensive and effective efficiency improvements.
ILLAVA
3.1 INSIGHTS
Recent LVLMMs Li et al. (2024a): Wang et al. (2024b); Bai et al. (2025): Chen et al. (2024c) are always consisted of an image encoder to encode input images, whose output features are transformed by projector and are further combined with textual tokens inputs into LLM for processing. Previous efficient methods Chen et al. (2024a); Xing et al. (0255; ; Yang et al. 2025): Zhang et al.
(2025):: Alvar et al. (025)) have made considerable progress in accelerating LVLMs, which usually try to reduce image tokens during or before the LLM to accelerate LVLMs. However, the massive redundancy in other network stages (e.g., image encoder) overlooked by recent methods, which fail to fully leverage the potential of inherent feature redundancy to accelerate LVLMs. To validate the feature redundancy within the image encoder, we first visualize a raw image and four spatial attention maps from different intermediate layers of the image encoder in Fig. [Ta). One can see that the encoder only pays major attention to small part of areas in the image (the bird) and overlooks others, which demonstrates that there exists considerable visual redundancy in the image encoder. We also observe that visual redundancy spreads unevenly across different layers, as the distribution of spatial attention maps varies across different layers. In Fig. [bb, we further plot the inference time distribution for components in LVLMs including the image encoder, projector and L.M. It's observed that the image encoder and the LLM dominate (>99%) the inference time in LVLMs and a large part of the inference time is consumed by the image encoder. The inference time proportion of the image encoder consistently increases task inputs vary from single-image, multi-image
3

---

Published as a conference paper at ICLR 2026
(a) Visualization the merging process across different blocks
Text outputs
Large
Transformer block K+1
Language
Model
(b) The method overvlew
Attembion
Token merging
tolems
Albention
Praderfred weights
A
A
T Transformer block K
Step Sellct tokens ottention
Step 2: Nuutppl selected
tolkrss with commspondng weights wergang
System
Image tokens
User instruttions
Image(s) 6)
blocks
A ttrntton
Token merging
Feed Forward Netwock
Projector
image encoder
Figure 2: (a) Visualizations for the token merging process across different blocks in the image encoder. The red region denotes selected tokens and the black regions represent discarded tokens. (b) The framework overview for iL.. We apply token merging at the intermediate layers of both the image encoder and the language model accelerate the forward pass.
to videos, owning up to 45%. Besides, as the image encoder provides quite large quantity of input tokens for the subsequent LLM, reducing tokens the image encoder not only accelerates itself. but also drastically improves the computing efficiency for the LLM. We also observe that when reducing the same number of tokens. performing token reduction in the image encoder owns +25.3% throughput boost and -21.2% memory consumption than pruning tokens within the LLM. However, accelerating the image encoder within LVLMS is few explored by recent works. In this paper, explore how to incorporate the image encoder for acceleration and coordinate it with the LLM to perform thorough acceleration with higher efficiency.
3.2 METHOD
As introduced above, in this paper, we try to overcome the limitation of previous methods that only reduce tokens during or within the L.M in LVL.Ms, which inevitably can't reach higher efficiency. In practice, we observe that aggressively pruning tokens in the image encoder leads to substantial performance degradation, as excessive token removal at early stages discards critical information. This motivates to develop more elegant way to intelligently allocates the computational budget across both the image encoder and the LLM. Besides, to avoid beneficial information loss caused by token reduction operations and maintain model performance, we design new token merging strategy to recycle potentially beneficial information from those redundant tokens. Specifically, as shown in Fig. Bb) our proposed method, termed as iLLaVA, aims accelerate the forward pass of both image encoders and LLM to achieve high efficiency. Within the image encoder, it performs token merging between the attention module and the Feed-Forward Network of several encoder blocks to decrease visual tokens. Within the LLM,, it performs token merging operations after several specific LLM blocks. Each token merging module will reduce constant number of image tokens to decrease the required computational burden. By incorporating the image encoder into model acceleration, iLLaVa could notably decrease the image tokens early network stages with high computing efficiency, and simultaneously lower the computational costs of the subsequent LLM. As reflected in Fig. [ a), iLLaVA can accurately preserve the informative tokens in both the image encoder and LLM to keep beneficial information. We next introduce each design in detail.
3.2.1 TWO-STAGE TOKEN MERGING
We first introduce the token merging process in the image encoder and LLM. respectively. In the im- age encoder, we insert a token merging module after the attention module of several encoder blocks to reduce image tokens. Given encoder block consisting of Multi-Head Attention module
4

---

Published as a conference paper at ICLR 2026
(MHA) and Reed-Forward Network (FFN), the forward pass with token merging operations can be represented as:
Fout FToeeT Merge (M
Token_Merge denotes the token merging operation, which reduces constant number of image tokens for each block. After several token merging operations, the image tokens drastically decrease, largely reducing the computational demand both the subsequent encoder blocks and LLM.
In the LLM, we insert token merging module between several specific LLM blocks to reduce image tokens. Given one LLM block K; that is expected to conduct token merging operations, the forward pass can be represented as:
F out Token (ergee
After the token merging operations, the number of image tokens gradually decreases, which largely diminishes the computational requirements of parameter-intensee LLM.
We apply the same token merging strategy with different merging tokens for the image encoder and language model, respectively. the image encoder, we select B, blocks and merge tokens by Rg per block. In the language model, we perform token merging operations of B, selected blocks and reduce Rg image tokens per block. Thus, given N input tokens, we finally keep N R. x B, R2 x B, tokens. The computational budget can be dynamically adjusted by modulating the number of reduced tokens in each stage.
3.2.2 TOKEN MERGING STRATEGY
To effectively reduce redundancy while retaining the most informative visual tokens, it is crucial to accurately identify which tokens contribute most the model's predictions. As we have concluded from the observations in Sec. 3.1 attention scores from the attention module in the image encoder could well reveal the location of informative tokens. We also observe that attention scores in the LLM serve as good indicator for the importance of each token. We thus use the attention score from the LVLM as our metries for measuring token importance.
Specifically, the attention scores are calculated as Eq. 3
Q
S Softmax
(3)
VD,
where S₁ E NNN is the attention score of each head and we omit the batch dimension here. D is the head dimension, and Qn and K₁, represent query and key, respectively.
We take the image encoder as example to introduce token merging strategy. Given N input image tokens, the token merging module reduces R2 tokens to decrease required computational costs, only keeping P. N - Ry image tokens. The preserved tokens P, consists of two subsets with different focuses: pi informative tokens, which preserve the most important information of inputs, and P recycled tokens, which try recycle beneficial information from the less important tokens. Accordingly, we have P + P; = P...
We begin by first selecting the informative tokens P from inputs that are most important to the model. To verify which tokens are critical, we average S₁, across the head dimension over all tokens to obtain an attention map Savg E R.N Savg indicates the importance of each token to other tokens. Based on Savg, we select top Pi tokens with the highest attention scores as pi informative tokens, which are expected to keep the most informative information of visual inputs.
In the above token selection procedure, other tokens are identified as less important ones and will not be preserved in this process. However, these tokens may still contain beneficial information for the model, and directly discarding them may cause beneficial information loss and lead to perfor- mance degradation. To avoid performance drop and preserve potentially critical information from these tokens as much as possible, we propose to recycle beneficial information from these tokens by condensing them into some representative tokens. To form robust and information-intensiee rep- resentations, we assign P tokens with the highest attention scores based on Savg (excluding Pi informative tokens) recycled tokens, which experted to serve as clusters to aggregate po- tentially beneficial information from other discarded tokens. To obtain the relationships between recycled tokens Pe and others, we compute the attention scores between them following Eq. 3 and
5

---

Published as a conference paper at ICLR 2026
Table 1: Results on image henchmarks. RealWoridQA
grok 1.5v
2024 is abbreviated as RWQA.
MMMU MMMU Pro MMBench MMBench VI.1
MME MMStar MM MuirBench RWQA
Method
Avg
(Val)
Overall
(EN)
(EN)
(sum)
(Test)
(Test)
Vanilla
58.6
38.3
83.5
82.6
2347
63.9
67.1
59.6
68.5
65.3
Upper hound)
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
Roduce about 66.
66.7%
image tokens
LM
$7.9
37.7
81.2
80.8
2258
61.1
63.9
57.8
66.4
63.5
(ICCV25)
98.8%
98.4%
97.2%
97.8%
96.2%
95.6%
95.2%
96.8%
97.0%
97.0%
Prop
57.9
37.8
81.6
80.5
2267
60.9
64.3
58.7
66.5
63.7
(CVPR25)
98.8%
98.6%
97.7%
97.5%
96.6%
95.4%
95.8%
98.2%
97.1%
97.3%
VLM
58.4
38,3
82.7
2299
61.9
64.6
58.1
67.2
64.2
(ICML.25)
99.6%
100.0%
99.0%
98.4%
98.0%
96.8%
96.4%
97.4%
98.2%
98.2%
VisionZip
58.3
38.2
82.5
82.0
2299
62.1
64.8
58.2
67.4
64.4
(CVPR25)
100.0%
99.6%
98.8%
99.2%
98.0%
97.2%
96.8%
97.6%
98.3%
98.4%
58.2
38.1
83.0
81.7
2328
62.8
66.0
59.1
68.2
64.8
iLLaVA (Ours)
99.2%
99.5%
99.4%
99.1%
99.2%%
98.3%
98.4%
99.2%
99.6%
99.2%
Reduce about 77 8% tmage tokens
Drop
55.3
36.4
78,5
77.7
2178
58.6
61.7
55.3
64.1
60.6
(CVPR25)
94.3%
95.1%
94.0%
93.6%
92.8%
91.7%
91.9%
92.8%
93.4%
93.2%
SLM
56.7
36.6
80.3
78.3
2208
60.1
62.9
56.4
64.7
62.2
(CCML25)
96.8%
95.6%
96.2%
95.0%
94.1%
93.9%
93.7%
94.5%
95.0%
95.1%
F
56.8
36.5
79.6
79.6
2210
60.5
63.7
56.3
66.0
62.9
(ICCV25)
97.0%
96.5%
95.9%
97.1%
94.2%
94.7%
95.0%
94.3%
96.5%
95.8%
VisionZip
56.7
37.5
79.3
80.7
2280
61.3
63.1
57.6
64.8
63.6
(CVPR25)
96.6%
98.0%
95.8%
97.9%
97.3%
96.0%
94.9%
96.6%
95.2%
96.4%
$7.9
37,8
82,9
79.8
2266
61.6
65.2
37.2
67.7
64.3
iLLAVA (Ours)
99.3%
98.7%
99.3%
96.7%
96.9%
96.2%
97.3%
96.0%
99.1%
97.6%
Reduce about 88
9% image tokens
Drop
52.3
34.0
74.4
71.9
2039
55.1
57.3
53.0
59.8
57.2
(CVPR25)
89.2%
88.7%
89.1%
87.3%
86.9%
86.2%
85.4%
89.0%
87.4%
87.5%
LLM
55.6
36.2
78.2
75.8
2128
58.0
60.8
34.6
64.7
60.0
(CCML25)
94.3%
94.5%
93.2%
92.0%
91.1%
90.9%
90.8%
91.7%
94.3%
92.8%
L
55.9
36.4
78.7
77.1
2131
58.5
62.2
55.2
63.7
61.0
(ICVV25)
94.8%
95.0%
93.6%
94.1%
91.1%
91.5%
92.7%
92.6%
93.1%
93.2%
VisionZip
56.0
35.9
79.2
76.1
2159
58.8
62.0
54.8
65.4
61.4
(CVPR25)
95.2%
94.0%
94.2%
93.1%
92.0%
91.9%
92.4%
91.8%
95.3%
93.7%
56.9
36.6
79.7
79.7
2207
60.5
63.0
56.3
65.3
62.8
iLLaVA (Ours)
96.9%
97.0%
95.5%
96.5%
93.6%
93.4%
93.2%
94.3%
94.8%
95.2%
average the results across the head dimension to obtain Saub E (n.,+ We then aggregate complementary features from other closely correlated tokens into these PS recycled tokens to recy- cle beneficial information. For each recycled token, we assign M (usually M = 5) tokens with the highest similarity as its group members, and merge tokens within the same group by weighted sum according to their normalized attention scores. Finally, the token recycling operation would keep Ps informative tokens and Pe cluster tokens, which are concatenated the outputs of the token merging module.
3.3 EFFICIENCY ANALYSIS
Compatibility with Flash-Attention. For the vanilla attention, can obtain pre-calculated attention scores Sn to compute Savg. However, Flash-Attention can't return the full atten- tion matrix S₁ during inference. In practice, pass the parameter return_attn.pi aasshaatt_wvrren.
to the function to enable returning cumsum attention weights Scumsum E RN, which can be transformed to Sang via simple transformation, eliminating extra computations.
Computational Complexity. For either vanilla attention or Flash-Attemtion, we can obtain Savg without incurring extra computations. The additional computations are brought by the calculation of Saub- Theoretically, the only computational costs incurred by this procedure is Ó (RR.). x (B.) + O(R.) x (Bt). As R., and RG are rather small (tens of tokens) compared to the input tokens (usually thousands of tokens) in practice, the incurred extra computations are few and can be ignored.
EXPERIMENTAL RESULTS
4.1 MODEL SETTINGS
By default, we use Qwen2.5-VL. 7B Bai et al. (2025) as the base model. We allocate 40% of merged tokens to the image encoder, distributing them evenly across Layers 5, 9, and 13. The remaining 60% of merged tokens are assigned to the LLM, evenly distributed across Layers 2, 8, and 14.

---

Published as a conference paper at ICLR 2026
Table 2: Results on video benchmarks with Qwen2.5-VI. 7B over different token reduction ratios.
EgoSchema
Mangalam et al. (2023) is abbreviated as EgoS here.
VideoMME
EgoS (Test)
MLVU
VideoMME
EgoS
MLVU
Method
MVBench
Avg
MVBench
Avg
(w/wo sub))
(M-Avg)
(w/wo sub))
(Test)
(M-Avg)
Vanilla
71.6/65.1
69.6
65.0
70.2
68.3
71.6/ 65.1
69.6
65.0
70.2
68.3
(Upper bound)
100.0%/100000%
100.0%
100.0%
100.0%
100.0%
100.0%//100.0%
100.0%
100.0%
100.0%
100.0%
Reduce about 90%
video tokens
Reduce about
95%
video tokens
SLarscVLM
67.1 60.3
66.7
61.1
66.9
64.0
66.4 / 59.6
65.9
60.5
65.7
63.1
(ICML25)
93.7%
95.8%
94.0%
95.3%
94.5%
92.8%%
94.7%
93.0%
93.5%
93.2%
Prop
68.6 61,4
68.0
62.2
68.4
66.5
67.6 / 60.2
67.5
61.5
67.0
64,9
(CVPR25)
95.8%
97.2%
95.8%
96.9%
96.1 96.1%
94.5%//93.2%
96.8%
94.6%
95.1%
94.8%
FasterVLM
68.6 61.6 61.6
68.3
62.3
68.2
66.6
67.8 60.9
67.6
61.8
67.1
65.3
(ICCV25)
95.8 95.8%/94.75% 7%%
97.6%
95.9%
96.6%
96.2%
94.8%/99388%
96.9%
95.0%
95.5%
95.3%
VisionZip
69.2 62.0
68.4
62.5
69.0
67.0
67.9 60.9
67.2
61.7
67.1
65.5
(CVPR25)
96.6%/95.39
97. %%
96.2%
97.3%
96.7%
94, 94.99//93.6%
96.4%
94.9%
95.4%
95.1%
69.9 63.0
68.9
63.3
69.8
68.2
68.6 62.3
67.9
62.7
69.3
66.9
iLLaVA (Ours)
97.7%
98.5%
97.4%
98.6%
97.8%
96.6%/955.%%
97.3%
96.4%
97.8%
96.8 %
MMMU
MMStar
MMMU
MMStar
61.0
65,5
50.0
39.0
60.0
56.0
67.0
+4.2%
64.5
63.5
65
64
63
+2.2%
35 6.570
56.0
55
55.0
+5.6%
64
60
+8.1 1/6
36.0
65.0 1.0
62.5
52.0
$3.0
3B mx3ill
54
56
3B model
1.4H
1.5x
2.2x
2,6
1.0x
1,4x
1.8x
2.24
20x
L0K
1.44
1.8x
2.2
2.6x
10k
1.4x
1,8x
2:2x
2.6
Theouglipt boost
Throughput boost
Thoougppot boost
Throughut boxsd
(4) VnterVI1. 2.5 26D (w/ iLLaVA) v.s. InternVL. 2.5 8B
(b) QWL 7B (w/ iLLaVA) vss Qwen2. V.V.
Figure 3: (a): deploying VA upon In2ernVL2.5 26B and comparing it with InternVL2. 8B. (b): deploying iLLaVA upon Qwen2.5-VL. 7B and comparing it with Qwen2.5-VL. 3B.
4.2 EFFECTIVENESS ON IMAGE UNDERSTANDING
We compare our method with Sparse VLM Zhang et al. (025b)), Faster Zhang et al. (2025a),
PyramidDrop Xing et al. (2025) and VisionZip Yang al. (2025) over wide range of image under- standing benchmarks Liu et al. )2 Xu et al. (2023a): Yue et al.] (2024a b): Chen et al. (2024b); Yu al. ( Wang al. (2024a); grok 1.5v 2024) to validate its effectiveness on image under- standing. To comprehensively assess performance, we present the results in percentage format for comparative analysis, with the vanilla model's accuracy serving as the 100% upper limit. We use three vision token reduction configurations (66.7%. 77.8% and 88. 38.9% to evaluate the advantage of our VA. As shown in Tab. [ our method demonstrates evident performance advantages over all token reduction ratios compared to other methods. When employing 66.7% token reduction ra- tios, our method could retain 99.2% performance, owning only 0.8% decrease compared to the vanilla model. When increasing the token reduction ratios to 77.8% and 88.9%, the advantage of our method is more obvious, outperforming other methods with larger margins upon performance. When reducing 88.9% image tokens, our iLLa VA could still keep 95.2% performance, representing only a 4.8% decrease compared to the vanilla method using 10 times the visual tokens.
4.3
EFFECTIVENESS ON VIDEO UNDERSTANDING
We evaluate iLLaVV on four representative video benchmarks including VideoMME Fu et al. MVBench Li et al. (2024c), Egoschema Mangalam al. (2023) and MLVU Zhou et al.
2024
We compare our method with Sparse VLM et al. (2025b). Faster Zhang el al. PyramidDrop Xing et al. 2025) and VisionZip Yang al. (2025) upon two heavy token reduction configurations (90% and 95%) We use Qwen2.5-VL. 7B as the backbone and set its per- formance on corresponding video benchmarks as the 100% upper limit. As shown in Tab. 2 our method demonstrates strong superiority compared to existing approaches with either 90% or 95% token reduction ratios. When reducing 90% visual tokens, iI. VA outperforms the state-of-the-art VisionZip Yang al. (2025) by 1.2%. When increasing the token reduction ratio to 95%, the perfor- mance gap between our method and VisionZip advances to 1.7% validating its efficacy in handling high reduction ratios.
2024
2025a
7

---

Published as a conference paper at ICLR 2026
4.4 COMPARISON WITH SMALLER MODELS
An advantage of VA is that it could enable larger model own both better performance and higher efficiency than a smaller model. In Fig. 3 we deploy iLLaVA upon InternVL--2.5 26B Chen al. (2024c) and Qwen2.5-V
7B Bai et al. (225), and compare them with smaller models includ- ing InternVL-2.5 8B and Qwen2.5-VI VL 3B over both accuracy and throughput to validate its effec- (a). equipped with iLLa ILtemn 26B could achieve similar or better throughputs than Intern 2L. 8B, while gaining +4.2% and +2.2% performance advantages on the MMMU and MMStar benchmarks.
tiveness. As shown in Fig. (a),
Fig. (b), when equipped with iLLaVA, Qwen2.5- VL 7B could achieve higher throughputs than Qwen2.5-VL 3B and obtain +5.6% and +8.1 18.1% % perfor- mance boost on the MMMU and MMStar benchmarks. This reflects that iLLa VA could overcome the computing challenges of large models by enabling larger models to run faster and perform better than smaller models, thus significantly broadening the application scenarios of large models.
4.5 FLEXIBILITY OF ILLAVA
Except for Qwen2.5-VI. Bai et al. 2025), We deploy iLLaVA upon dif- ferent LVLMs to verify its flexibil- ity to different model architectures. We adopt LLaVA-Onevision Li et al. (222a)) Intern Chen et al. 2024cp and Minicpm-V 2.6 Yao et al.
Table 3: Flexibility of VLLAV upon various VLMs. We
abbreviate RealWorldOA RWQA here.
MMMU MMBench MMStar RWQA
Avg
(Val)
(EN Dev)
(Test)
(Test)
LLavA Onevision 7B
48.8
80.8
61.7
66.3
64.4
PyramidDrop (CVPR25)
46.5
76.8
59.4
62.7
61.3
Sparse (ICML.25)
46.8
77.6
59.2
63.6
61.8
2024) as our backbones conduct comprehensive evaluation. We com- pare iLLaVA with previous meth- ods upon four representative image benchmarks including MMMU Yue et al.] (202a)). MMBench Liu et al. (2024), MMStar [hen et al. (2024b)
FasterVLM (ICCV25)
47.6
78.8
60.4
64.6
62,8
VisionZip (CVPR25)
48.0
79.2
60.2
65.1
63.2
iLLaVA (Ours)
48.4
79.7
60.2
65.8
63.7
InterVV2.55 8B
56.0
84.6
62.8
70.1
68.4
ProamidDrop (CVPR25)
52.6
80.6
59.8
65.2
64.3
VLM (ICML255
53.1
81.3
59.2
65.6
64.8
Faster VLM (ICCV25)
54.2
82.5
60.2
66.8
65.9
and RealWorldQA grok 1.5v
(2024)
VisionZip (CVPR25)
54.8
83.2
61.3
68.7
67.0
and set the token reduction ratio as 77.8%. Tab. 3 shows iLLaVA con- sistently demonstrates better averaged accuracy compared to previous state- of-the-art methods when deployed upon different LVLMs, notably vali- dating its flexibility.
iLLaVA (Ours)
55.4
83.1
61.8
69.4
67.5
MiniCPM-V2.6 8B
49.8
81.5
57.5
65.0
63.5
SparseVL.M (ICML.25)
47.6
77.8
54.9
62.1
60.6
PrramidDrop (CVPR25)
48.0
78.6
54.7
62.7
61.0
FasterVLM (ICCV25)
48.6
79.5
56.1
63.4
61.9
VisionZip (CVPR25)
48.9
80.2
56.5
64.0
62.4
iLLaVA (Ours)
49.3
81.0
56.3
64.3
62.8
4.6 EFFICIENCY ANALYSIS
We validate the efficiency of iLLaVA by comparing it with state-of-the-art methods including Spar- se ZeVLM et al. (2025b) FasterVLM Zhang et al.] (2025a). PyramidDrop Xing et al. (2025) and VisionZip Yang et al.) (2025). ) These methods are evaluated over various metrics including memory usage, throughput, prefilling time and accuracy across different visual token reduction ratios on the MMMU benchmark. Here, the prefilling time denotes the time required to generate the first out- put token. As shown in Fig. 4 iLLaVA largely reduces the memory usage by 1.59x, improves the model throughput by 2.12x and reduces the prefilling time by 4.46x. When evaluated upon the same token reduction ratio, iLLaVA not only achieves higher model efficiency than previous meth- ods with lower memory usage and better throughputs, but also obtains better model performance. This indicates that iLLaVA could simultaneously deliver enhanced model accuracy and efficiency than previous methods. This is because iLLaVA leverages the visual redundancy in both the im- age encoder and LLM accelerate LVLMs, thus further improving the upper limits. Additionally, we observe that as token reduction ratios gradually increase from 50% to 90%, the advantage of iLLaVA on improving model efficiency and maintaining model accuracy is more obvious. iLLaVA could save more memory usage and achieve higher network throughput than previous methods, while better preserving the model accuracy.
4.7 ABLATION STUDY
Effectiveness of the two-stage design. In the ablation study, we report the averaged accuracy for iLLaVVA and other methods over image understanding benchmarks Liu al. (2024): Xu et al.
8

---

Published as a conference paper at ICLR 2026
(e) Meroryy v.s. lolon reduction ralo Sp
O9 Throughys tokon redustion ndlo LM
(c) Pretiling time takes rnduction
(50 ccuumay u.s. bikzn reducton mioo
Spaney
SpannVi
Spors
Pyp
- Preeeammp
Re
- -yneriiiiig
Viandiip
Viter2p
V:swnzip
090
Thers (aggggg
3.00
(42mm))
Crede
1123
12:22
1.8
58
0.8
0.1
03
Tokee EwG.x 1250
Tokee DOSUctin 7220
Toltn rodactco 2003
Toker reductionnonnn
Figure Comparison over memory usage. throughput, prefilling time and accuracy over the MMMU benchmark. Prefilling time denotes the latency of generating the first output token. VA owns notably lower memory usage and throughputs and retains better accuracy than other methods, while achieving comparative prefilling time.
: Wang et al. (2024a); grok 1.5v 2024) by reducing 88.9% visual tokens. We first validate the effectiveness of accelerating LVLMs in the image encoder and L.M stage, respectively. As shown in Tab. 4 We observe that compared to the vanilla model, conducting token reduction in the LLM stage could notably improve the model throughput and decrease the memory usage. When incorporating the image encoder into model acceleration, could further improve the model throughput by 2. 2.12x, decrease the memory usage to 0.64x while achieving better accuracy. This reflects that conducting token reduction in both the image encoder and LLM is more effective than just reducing tokens in the LLM.
(2023a) Yue et al. (2024a b): Chen et al. (2024b); Yu et al. (2023);
Effectiveness of our proposed token merging strategy. We evaluate the effectiveness of our pro- posed token merging strategy by replacing with other token reduction strategies and keeping other network configurations unchanged to conduct fair evaluation. The methods used for comparison include token pruning strategies Xing et al. 2025) and token merging Arratggies/hhang et al. (20256); Yang et al. (2025); Shang et al. (2024) from recent representative methods. Except for approaches specially designed for LVLMs, we notice that previous representative method named ToMe Bolya et al. (2022) has explored reducing tokens in the image encoder, and incorporate it in our experi- ments. As shown in Tab. 5 compared to other approaches, our proposed strategy achieves better averaged accuracy, which validates the design of recycling beneficial information from discarded tokens. Besides, our strategy is also efficient in technical implementations, which gains comparable or better throughputs compared to state-of-He sate--of-thearte methods.
Table 4: Validation of the two-stage design. Memory
Table 6: Comparison with more training- training-free to- ken reduction methods.
Encoder LLM Acc(%) Throughput
x
65.3
1.86
32.1G
Style
Methods
Reduction ratio
x
62.1
3.46
23.1G
77.8%
88.9%
FastV
58.2
52.4
62.8
3.94
(2.12x) 20.6G (0.64x)
PyramidDrop
62.2
57.2
Token
AdaFV
62.4
59.8
Table 5: Comparing different token reduction ways.
pruning
FasterVLM
62.9
61.0
Style
Methods
Acc(%)
Throughput
DivPrune
63.1
60.8
Pruning
PrromidDrop
61.0
4.63
FEATHER
63.4
61.0
Bipartite matching
56.4
4.50
LLa PruMerge
61.8
59.6
merging
Token
LLaVA PruMerge
Sparse VLM
VisionZip
61.7
61.1
62.1
4.60
4.61
4.52
merging
Token
SparseVLM
VisionZip
AIM
63.6
63.8
62.2
60.0
61.2
61.4
iLLaVA
64.3
62.8
iLLaVA 62.8
4.62
Comparison with more token reduction methods. We compare our iA with more token
reduction methods Chen et al. (2024a); Xing et al. (2025): Han al. 2025) : Zhang et al. (2025)); Alvar et al. (2025); ; Endo et al. (2024): Shang et al. (2024) Zhang et al. (2025b): Yang et al. (2025) Zhong et al. (2024) designed for LVLMs to confirm its superiority. We perform the experiments under two token reduction ratios including 77.8% and 88.9% and show the results in Tab. [ We ob- serve that while some token pruning methods own promising performance, token merging methods generally achieve better results compared to token pruning methods. Compared to these state-of-the art approaches, iLLaVA achieves the best performance across both token reduction ratios, validating its superiority among these strong competitors.

---

Published as a conference paper at ICLR 2026
JavaScript
bjects
Single-image banchmarks
Jonn
Walk tmaue
Salwer sul
Selee yieess! LLM
wiforrm VtV
Saa
LLM
Mulli-image benchmarks
Sdlcced
Souddu LLM
bakars
Edlided 1134
Solided L1.22
ofV/t
Video benchmarks
Vidurham
Iunes
Slavad Winw
Figure 5: Visualizations for selected tokens of iLLAV over different benchmarks. We select several representative images for each benchmark, and display the raw image, selected tokens from the image encoder and selected tokens from the LLM in each row in order.
4.8 VISUALIZATIONS
To better understand the token merging process in iLLaVA, we visualize the selected image tokens for single-image benchmarks, multi-image benchmarks and video benchmarks in Fig. 4.7 respec- tively. On each benchmark, we select three images and visualize the raw image, selected tokens from the image encoder and selected tokens from the LLM in order. We observe that both the image encoder and LLM can well attend to the important information from raw images, like the words in the first row and the person in the last row. We also notice that the LLM tends to allocate more tokens to images closer to the output texts, and pay less attention to images which are input earlier into the LLM. This may affect the behavior LVLMs in memorying long contexts.
4.9 COMPUTATION DISTRIBUTION IN IMAGE ENCODER AND LLM
To accelerate LVLMs by reducing computations in both the image encoder and LLM, a key chal- lenge allocating the computation reduction be- tween the two components. Fig. 6 shows how performance and throughput scale with the pro- portion of computation reduced in the image en- coder. When we reduce 0-40% of computations in the image encoder (and (00-60% in the LLM),
Thoughput intaee
propartion computations
650
L.L.V otlected
42.0
performance only drops slightly while through- put improves significantly (1.8x to 2.1 How- ever, further increasing this proportion to 40- 90% leads rapid performance decline, even as throughput grows that moderate reduction of image tokens (0-40%) Figure 6: Performance and Throughput boost v.s. has little impact performance, but excessive the proportion of reduced computations in the im- reduction degrades it substantially by stripping age encoder. the fine- lnnegrained spatial information needed by the model. We also find reducing image tokens the encoder boosts throughput more effectively than reducing tokens in the LIM,, since the encoder con- tributes most of the input tokens to the LLM. Based on these observations, we select 40% reduction in the image encoder and 60% reduction in the LLM to balance performance and efficiency.
$200
sharply. This indicates
Propertion reduced computation: imagee cucoddrr
CONCLUSION
In this paper, we review and validate that there exists considerable feature redundancy the image encoder of LVLMs. We propose utilize and combine it with the visual redundancy in other LVLM components (e.g.. LLM) to accelerate models for further efficiency. Moreover, to avoid performance token merging strategy recycle the potentially
loss caused by token reduction, propose beneficial information from discarded tokens. Results on 10+ image and video understanding bench- marks across various token reduction ratios validate the effectiveness of method.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
Our work was supported by the following projects: National Natural Science Foundation China (Grant No. U2574216); Emerging Frontiers Cultivation Program of Tianjin University Interdisci- plinary Center; National Natural Science Foundation of China (No. 62276182 and 62572349).
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics and has been conducted with careful consideration of ethical implications. The research does not involve human subjects or personally identifiable data, and no new datasets requiring special release protocols were created or used, All methodologies and applications presented are intended for constructive, non-harmful purposes, and we have taken steps to identify and mitigate potential biases in model design and training data. No conflicts of interest exist, and the research received industry sponsorship that could influence outcomes. Privacy, security, and legal compliance were prioritized throughout the experimental process. We uphold principles of research integrity through transparent documentation, reproducible experiments, and responsible dissemination of results.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we have taken comprehensive steps to document and share all necessary components of our work. The model architecture, training procedures, and hyperparameter settings are fully described the implementation details of the main paper. All datasets used in our experiments are publicly available. The code of our work will be made publicly available if this paper receives acceptance by ICLR2026.
REFERENCES
Jean- Ban-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-
23736, 2022.
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2rXiv:2502.1392. 2025.
Daniel Bolya, theng--Yng Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09460 2022.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer Plug-and aad--andpplay inference acceleration for large vision- language models. In European Conference Computer Vision, pp. 19-35. Springer, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403 200.03300 2024b.
Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. Sparsevit: Revis- iting activation sparsity for efficient high-resolution vision transformer. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2061-2070, 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of opun-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c.
11

---

Published as a conference paper at ICLR 2026
Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token pruning for vision-language model accelcration. arXiv preprint arXiv:2412.13180, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075 2024.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Confereence on Machine Learning, pp. 3690-3699, PMLR, 2020.
grok 1.5v, Realworldqa benchmark, January 2024. URL
1.nkk-1144 5v
Jayy Han, Liang Du, Yiwen Wu, Xiangguo Zhou, Hongwei Du, and Weibo Zheng. Adafv: Acceler- ating vlms with self-adaptive cross-modality attention mixture. arXiv preprint arXiv:2501.09532, 2025.
Gyuwan Kim and Kyunghyun Cho. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:20/0.07003, 2020.
Carlos Lassance, Maroua Maachou, Joohee Park, and Stéphane Clinchant. A study on token pruning for colbert. arXiv preprint arXiv:21 12.02152. 2021.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang. Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint (rXiv:2408.03322 2024a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings the IEÉF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13308, 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 2023.
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22200 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pp. 216-233. Springer, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint 2:Xiv:2306605424, 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. Advances in Neural Information Process- 2023.
ing Systems,
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint 2IXiv:2203.1024: 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Emest Valveny, and CV Jawahar. Infographicvga. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706. 2022.
12

---

Published as a conference paper at ICLR 2026
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12309-12318, 2022.
Xavier Ochoa, AWDG Charles Lang, and George Siemens. Multimodal learning analytics. The handbook of learning analytics, 1-119-11,, 2017.
Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, and Jianfei Cai. Less is more: Pay less attention in vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2043, 2022.
Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE signal processing magazine, 34(6);96-108, 2017.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:1 13937-13949, 2021.
Michael Ryoo, AJ Piergiovanni, Anurag Arna,, Mostafa Dehghani, and Anelia Angelova. Token- learner: Adaptive space-time tokenization for videos. Advances in neural information processing systems, 34: 14411266-12997, 2021.
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Flahe Vedadi, et al. Capabilities gemini models medicine. arXiv preprint arXhv:2404.18416 2024.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388, 2024.
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint w:Xh::2406.09111 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lisoun-language model's perception of the world at any resolution. arXiv preprint arXiv: 12191, 2024b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang. Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pp. 396-416. Springer, 2024c.
Long Xing, Qidong Huang, Xiaoyi Dong. Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2025.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong trining-free baseline for video large language models. arXiv preprint arXiv:2407.15841 2024.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. LvIm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint urXiv:2306.09265. 2023a.
Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey.
IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(10)::2113-12132 2023b.
Sengiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2025.
13

---

Published as a conference paper at ICLR 2026
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0 01800, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308 20X0:22008.02490, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.028/1 02811,, 2024b.
Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang. Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token prun- ing: Make vlm inference faster. International Conference on Computer Vision, 2025a.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevlm: Visual token spar- sification for efficient vision-language model inference. International Conference on Machine Learning, 2025b.
Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. Aim: Adaptive inference of multi-modal Iims via token merging and pruning. arXiv preprint arXiv:2412.03228, 2024.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint 2.X14:2400.04244, 2024.
A APPENDIX
A.1 LIMITATIONS
While we validate that iLLaVA performs well across wide range of benchmarks, we find that upon benchmarks like Doc,QA, InfoVQA and ChartQA which require large quantity of image tokens to perform detailed document understanding, conducting token merging may lead to more evident performance drop. Moreover, image images may contain small objects that are difficult for the models to capture and build fine-grained features, which are easily missed during the token reduction process. In these scenarios, the model could demonstrate inferior performance compared to the vanilla model as they lack the critical information contained in the small objects for understanding.
A.2 LLM USAGE
In this work, we employed large language models (LLMs) solely for grammatical refinement and language polishing. No substantive content, arguments, or technical formulations were generated or altered by the LLM; its use was strictly limited correcting syntax, improving sentence fluency, and ensuring grammatical accuracy in the final manuscript. All intellectual contributions, including conceptual development, analysis, and writing, remain entirely the authors' own.
14


Reviews:
[{"review_id": "REN74WydLI", "reviewer": "Reviewer_hqY7", "summary": "The paper proposes iLLaVA, achieves a balance between efficiency and accuracy by jointly merging tokens at both the image encoder and LLM ends and introducing \"recycled tokens\" to aggregate the information of discarded tokens. Despite a 88.9% reduction in visual tokens, it still maintains 95.2% performance, enabling the 26B model to comprehensively outperform the 8B model in terms of speed and efficiency. It offers 2× throughput, 4× first-token latency and 1.59× memory savings, and is suitable for a wide range of LVLMS", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method extends token reduction from LLM to image encoders, achieving dual acceleration and significantly reducing overall computational and memory overhead.\n\n- It aggregates discarded information with \"recycled tokens\", maintaining an accuracy of over 95% even at extremely high compression rates, balancing speed and performance.", "weaknesses": "- Can this method be adapted to flash attention? And, can it be adapted to vllm and sglang inference frameworks? It seems the adaption on mainstream frameworks somehow has difficulties.\n\n- The performance drops significantly for tasks that require fine spatial information, such as DocVQA and ChartQA. Small targets or dense text are prone to losing key details due to token merging.\n\n- The reduction ratio of tokens and the insertion layer positions need to be manually optimized, lacking an adaptive mechanism. When changing models or tasks, it may be necessary to re-search for the optimal configuration, resulting in high migration costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yp84EoXOke", "reviewer": "Reviewer_AUQc", "summary": "This paper proposed a two-stage method for reducing visual tokens in VLMs. In particular, it tries to reduce tokens both at visual encoder stage and LLM decoding stage. In each stage, a token merging module is inserted in between the self-attention layer and FFN every N blocks. Within the token merging module, tokens that received highest attention scores are deemed to be important and kept, and the rest of tokens are merged into clusters called recycled tokens. Both important and recycled tokens are sent to the next layer for processing. Each token merging module reduces the overall number of tokens by a fixed amount, and the overall model can reduce the visual tokens to the target budget. Experiments on a set of image and video understanding benchmarks with several popular VLMs show that the proposed method outperforms several baselines, both in terms of the performance retention and efficiency.", "questions": "Can authors discuss how the proposed method differs from existing two-stage visual token pruning methods, and showcase the advantage of iLLaVA in experiments?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Unlike many previous works in the area that focus on single image task only. This paper presented comprehensive experiments on multi-image and video benchmarks and shows stronger performance compared to several baselines. \n\nThe experiments with four different VLMs show the effectiveness of the iLLaVA. \n\nThe results on memory usage, prefilling time and thoughput provide interesting insights on the impact of visual token pruning from different angles, which is often neglected in previous works.", "weaknesses": "My main concern is on the novelty of this work. The paper claims the two-stage pruning method and token merging as novel contributions. However this idea has been done by previous works. For example, VScan (Zhang et al. 2025) adopt very similar idea to prune tokens at both visual encoder stage and llm decoder stage. Furthermore, VScan also proposed to merge pruned tokens instead of discarding them. The only difference might be the specific layer index where the token pruning/merging happens. \n\nZhang, Ce, Kaixin Ma, Tianqing Fang, Wenhao Yu, Hongming Zhang, Zhisong Zhang, Yaqi Xie, Katia P. Sycara, Haitao Mi and Dong Yu. “VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models.” ArXiv abs/2505.22654 (2025): n. pag.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nvLkR6iY8W", "reviewer": "Reviewer_jbVu", "summary": "The paper proposes iLLaVA to address a key limitation of existing LVLM acceleration methods: most only reduce image tokens in the LLM stage, overlooking the image encoder—a critical bottleneck accounting for 99% of inference time (45% in video tasks)—and failing to achieve end-to-end acceleration. To tackle this, iLLaVA enables comprehensive acceleration through two core innovations: two-stage token merging (inserting modules at critical positions in both the image encoder and LLM to jointly optimize the two computationally intensive components) and a novel token merging strategy that recycles useful information from discarded tokens, compatible with Flash-Attention with negligible extra computation. Experiments demonstrate iLLaVA retains over 95% performance while reducing 66.7%-88.9% of image tokens, outperforms SOTA methods, supports multiple LVLM architectures, and delivers significant improvements in memory usage, inference speed, and throughput.", "questions": "- Flexibility of insertion position. The token merging module can be inserted into any layer of ViT and LLM. Do different VLMs require distinct insertion positions of the token merging module to achieve optimal performance? Additionally, are there differences in the optimal insertion positions across various VLMs?\n- In Table 4, reducing tokens in both the encoder and LLM achieves better accuracy than only reducing tokens in LLM. Why does reducing more tokens in ViT further improve performance? Moreover, the paper lacks an ablation study focusing on applying the token merging module solely in ViT.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple training-free approach. The paper adopts a training-free token merging method for ViT and LLM. Compared with merely merging or pruning tokens on MLLMs, the proposed method achieves better performance, lower memory overhead, and higher throughput.\n- Extensive experiments. The paper conducts extensive experiments on two benchmark suites to verify the effectiveness of the proposed method across 9 image tasks and 8 video tasks. Under fair comparison settings, the proposed method achieves better performance than four existing methods.\n- High generalization. It proves adaptable to four MLLMs, demonstrating its strong generalization capability.", "weaknesses": "- Lack of novelty. The paper adopts the existing token merging method and extends its application from LLM to ViT.\n- Simplified visualization in Figure 2. Figure 1 presents the attention scores in ViT for a single object against a simple, plain background. This straightforward case fails to convince readers. What would the attention map look like when the image scene is complex?\n- Limited experimental benchmarks. The paper only conducts experiments on general QA benchmarks. How does it perform on benchmarks focusing on capturing key objects in dense information? For instance, text-centric benchmarks (TextVQA, ChartVQA, DocVQA, InfoVQA, OCRBench) and high-resolution image benchmarks (V*, etc.)?\n- Typos: Figure 3(b) contains a typo: \"QWen2.5-VL 7B\" should be formatted as \"Qwen2.5-VL\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6KdA1AEVWl", "reviewer": "Reviewer_MrAM", "summary": "This work proposes iLLaVA, a method that accelerates VLMs with token reduction. Different from most prioir work that focues on reducing in only LLMs, this work considers both the image encocer and the LM. Moreover, a merging strategy is further introduced to include potentially useful information. The proposed approach is evaluated on a variety of tasks and models, and the results illustrate the its effectiveness: it can obtain over 95% of the performance over various token reduction ratios and perform better than other methods.", "questions": "(* Please refer to the weakness part.)", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The direction of accelerating VLMs is important for the real-world deployment.\n- The idea of joint optimization of both image encoder and LLM is intuitive.\n- The results illustrate the effectiveness of the proposed approach.", "weaknesses": "- Using the attention score as an indicator of token importance is not quite a novel idea; in addition to applying the reduction to both vision encoder and LM, the methodological differences to previous work should be further illustrated.\n- There seems to be a lack of detailed analyses on why the proposed method works well? It will be better to provide more detailed analyses to explain the better performance compared to other methods.\n- It seems not quite straight-forward for me to understand the token merging process. There should be more detailed illustration for this process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)