HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
Authors: Hao Wu
Yingqi Fan
Dai Jinyang
Junlong Tong
Yunpu Ma
Xiaoyu Shen
Venue: ICLR 2026 Poster (2026)
Keywords: MLLMs
Vision Token Pruning
Efficiency and Compression
Interpretability and Analysis
Primary area: foundation or frontier models, including LLMs

Abstract: The quadratic computational cost of processing vision tokens in Multimodal Large Language Models (MLLMs) hinders their widespread adoption. While progressive vision token pruning offers a promising solution, current methods misinterpret shallow layer functions and use rigid schedules, which fail to unlock the full efficiency potential. To address these issues, we propose HiDrop, a framework that aligns token pruning with the true hierarchical function of MLLM layers. HiDrop features two key innovations: (1) Late Injection, which bypasses passive shallow layers to introduce visual tokens exactly where active fusion begins; and (2) Concave Pyramid Pruning with an Early Exit mechanism to dynamically adjust pruning rates across middle and deep layers. This process is optimized via an inter-layer similarity measure and a differentiable top-$k$ operator.  To ensure practical efficiency, HiDrop further incorporates persistent positional encoding, FlashAttention-compatible token selection, and parallel decoupling of vision computation to eliminate hidden overhead associated with dynamic token reduction. Extensive experiments show that HiDrop compresses $\sim$90\% visual tokens while matching the original performance and accelerating training by 1.72$\times$. Our work not only sets a new state-of-the-art for efficient MLLM training and inference but also provides valuable insights into the hierarchical nature of multimodal fusion. The code is released at https://github.com/EIT-NLP/HiDrop.

Published as a conference paper at ICLR 2026
HIDROP: HIERARCHICAL VISION TOKEN REDUCTION IN MLLMS VIA LATE INJECTION, CONCAVE PYRAMID PRUNING, AND EARLY EXIT
Hao Wu Yingqi Fan Jinyang Dai Junlong Tong1.244 Yunpu Ma Xiaoyu Shen Institute of Digital Twin, Eastern Institute of Technology, Ningbo
"Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative
Unniversity of Science and Technology of China "Shanghai Jiao Tong University
Munich Center for Machine Learning, LMU Munich haowu. ai. resea ressearch@gaaiiii 1 com
xyshen@eitech www.eeehhhhc....e..eeeeeee edu.cn
ABSTRACT
The quadratic computational cost processing vision tokens Multimodal Large Language Models (MLLMs) hinders their widespread adoption. While progres- sive vision token pruning offers a promising solution, current methods misinter- pret shallow layer functions and rigid schedules, which fail to unlock the full efficiency potential. To address these issues, we propose HiDrop, a framework that aligns token pruning with the true hierarchical function of MILM layers. HiDrop features two key innovations: (1) Late Injection, which bypasses passive shallow layers introduce visual tokens exactly where active fusion begins; and (2) Concave Pyramid Pruning with an Early Exit mechanism to dynamically ad- just pruning rates across middle and deep layers. This process is optimized via an inter-layer similarity measure and differentiable top-k operator. To ensure practical efficiency, HiDrop further incorporates persistent positional encoding. SethhAttentinnommaatiii token selection, and parallel decoupling of vision com- putation to eliminate hidden overhead associated with dynamic token reduction. Extensive experiments show that HiDrop compresses ~90% visual tokens while matching the original performance and accelerating training by 1.72x. Our work not only sets new state-of-the tfrt for efficient MLLM training and inference but also provides valuable insights into the hierarchical nature of multimodal fusion. The code is released at https//gtthub.com//TTNPPPPPPPP
INTRODUCTION
Multimodal Large Language Models (MLLMs) have driven rapid progress in fields ranging from visual question answering to embodied Al by seamlessly bridging vision and language (OpenAl,
2023 2024; Lu et al.) 2024; Bai et al. 2025 Achiam et al.] 2023). Most modern MLLMs rely on a
connector-based architecture (Liu at. 2023a 2024a Bai et al. 2023 Wang al. 2024 wherein a lightweight module projects visual features directly into the LLM's 's embedding space, allowing a pre-trained text backbone to process multimodal inputs without retraining from scratch (Li et al. 2023ah. However, visual encoders typically generate substantially more tokens than text due to their higher information density (Dosovitskiy al. 2020). With token count scaling quadratically with image resolution and self-attention quadratic in token number, computation quickly becomes prohibitive (Lin et al. 2024 Cui et al. 2024).
To address this issue, common strategy involves progressive vision token pruning (Chen et al., 2024b Xing et al., 2024,] Yao et al., 2025). where less informative visual tokens are gradually elimi- nated as they move through the model. The underlying intuition is that early layers should maintain dense visual representations to preserve fine-grained details, whereas deeper layers can function ef- fectively with reduced token set concentrated semantically significant content (Fan et al. 2025
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
System Tolom
Layer 4 + 1-1.
Layer 31-/
Layer 4 + 1-1.
LUIVA-d.5 76
Vision Token
98.3%
Accuracy
Tex: Token
JAVZI
BOC RCD0
Visual Tokens
576
Layer +
Layer + 1-6
Layer 1-2(
Layer +1-4
89%
TFlops
3.02
41%
s4
Training
Layer 1-f
Layer 1-1
Layer 1-1
Layer 1-4
PORt
hours
B (b) TwigVLM
Fasty
(e) PDrup
Ours: HiDrop
(e) Performance
Figure 1: Comparison of progressive vision token pruning methods. (a) FastV conducts single-stage pruning an early layer. (b) TwigVLM performs early pruning and removes remaining vision tokens deeper layers. (c) PDrop applies progressive pruning with uniform ratios and intervals. (d) HiDrop introduces vision tokens only at the end of shallow layers, prunes them in a non- onn-onfform progressive manner in middle layers, and removes remaining vision tokens before deep layers. (e) HiDrop prunes vision tokens by about 4.8x more aggressively than state-of-the-art progressive pruning method with negligible performance drop.
However, after a closer examination of the internal dynamics of such models, we identify that exist- ing pruning approaches are constrained by two core misunderstandings regarding how multimodal large language models (MLLMs) process visual information across different layers.
First, shallow layers are misinterpreted. Prior work observes that removing early layers degrades performance and thus concludes that these layers are critical for multimodal integration (Xing al.) 2024 Zhang et al., 2025) Wu et al. 2025). Our analysis shows otherwise: vision tokens, already deeply processed by the vision encoder, undergo almost no transformation in the initial LLM layers. Both intra-modal evolution and cross- cross-modal influence are negligible. These layers primarily act as propagators and attention sinks, not true integrators (Fan et al. 2025).
Second, pruming schedules are overly rigid. Existing approaches often adopt fixed-ratio, pyramid- like schemes such as FastV (Chen et al.) 2024b), TwigVI.M (Shao et al., 2025), and PDrop (Xing et al. 2024). However, we find that visual information flow is highly non-uniform: redundancy can be removed more aggressively in middle layers where fusion dominates, while visual tokens can be safely discarded altogether in the deep layers once integration is complete. Uniform schedules miss this structure and thus lead to suboptimal efficiency-accuraccy trade-offs.
Motivated by these findings, we propose HiDrop, a hierarchical vision token dropping framework that coordinates token pruning and computation skipping in accordance with the hierarchical pro- cessing dynamics of MLLMs across different layers.
To address the shallow-layer misconception, we adopt Late Injection strategy: rather than pruning in shallow layers, bypass them altogether and inject the full set of vision tokens only at the onset of the true fusion stage. This approach perfectly reflects the functional redundancy of the early layers without prematurely discarding potentially valuable information, marking the first attempt to deliberately delay, rather than simply prune, visual input for greater efficiency in MLLMs.
To address the limitations of rigid schedules, we propose Concave Pyramid Praning scheme, which accelerates token reduction early in the fusion stage and slows it later, together with an Early Exit mechanism that fully discards vision tokens before the language-dominant layers. When apply- ing this schedule, we identify reliable pruning layers using an Inter-Layer Visual Attention Similarity (ILVAS) measure, and select the most informative tokens with a differentiable top-k operator (Liu et al. 2024b). These mechanisms jointly enable precise and adaptive pruning decisions.
Beyond algorithmic design, HiDrop employs persistent position identifiers to preserve positional consistency under dynamic token activation, decouples token selection from the main attention computation to remain compatible with efficient kernels such as FlashAttention, and exploits late injection to parallclize vision-related computation with text-only prefill. These designs ensure that dynamic token management does not incur hidden overheads and translate the theoretical efficiency gains into real-world acceleration.
Extensive experiments on LLaVA-1.5-7B show that HiDrop compresses ~90% of visual tokens while matching the original performance, accelerating training by up 1.72x and substantially improving inference throughput. Our contributions are threefold: (1) we diagnose two fundamental weaknesses of existing pruning methods related to shallow-layer interpretation and pruning sched-
2

---

Published as a conference paper at ICLR 2026
Intra-nodality Similarty
1.00
Cross-medallty Siuillarit
1.000
LLaVA-v1.5 7B
0.95
0,975
0.90
0.950
System
0.85
Visual
0.925
Textual
0.80
25
32
Embedding
17 Layer
32
Layer
Figure 2:
Layer-wise representational dynamics, with the left panel showing intra-modal refinement, and the right panel highlighting cross-modal interaction intensity.
ules; (2) we introduce HiDrop, featuring the novel Late Injection strategy, Concave Pyramid Prun- ing with Early Exit, and optimized layer- and token-selection mechanisms; and (3) we empirically demonstrate that HiDrop achieves state-of-the-an efficiency-accuracy trade-offs.
2 UNMASKING THE PROCESSING DYNAMICS IN MLLMS
A Multimodal Large Language Model (MLLM) processes a unified sequence of text and vision embeddings. ho E E,). through its Transformer layers. The text embeddings E, E NN come from a standard tokenizer, while the vision embeddings E, E IR N, originate from a vision encoder that partitions an image into No patches and projects their features into the LLM's s hidden dimension d. The primary computational bottleneck in this architecture is self-attention, whose cost scales quadratically with the number of vision tokens, O(Ndd), as typically N. >> N..
To mitigate this computational burden, common solution is progressive token pruning, which iteratively reduces the number of vision tokens across the model's layers. However, most existing strategies rely predetermined and static pruning schedules, such linear convex decay, applied uniformly across layers without accounting for the distinct processing dynamics at different stages of the model. This raises critical question: what constitutes effective pruning strategy? We argue that token pruning should be grounded in the model's actual behavior rather than handcrafted heuristics. Achieving this requires deeper understanding of how MLLMs internally process and integrate visual information. Therefore, we conduct an in-depth analysis of their internal dynamics, revealing that different layers play fundamentally distinct roles in multimodal fusion, which in turn motivates more principled pruning approach.
prevalent assumption in progressive pruning is that shallow layers are essential for early cross-modal fusion and must be preserved (Xing al. 2024 Zhang et al. 2025). To scrutinize this belief, we perform a training-free layer-wise probe on LA- vI.5-7B, feeding GQA
Shallow Layers:
Propagators
pairs through the network and recording hidden states at all layers. Our analysis, however, reveals that these layers function not as active integrators but as simple propagators. We demonstrate this by examining their contributions from two perspectives.
First, we analyze intra-modal refinement by measuring how token representations evolve across layers for each modality M E (System, Visual, Textual). Concretely, we compute the modality- specific cosine similarity (S/M) between the outputs of consecutive layers:
Namppe Σ
S M intra
Σ
N Nample
NM
BETM | |
where denotes the layer index, Nasmll is the number of samples, TM is the set of tokens belonging to modality M with NM [Txl,, and af,, e is the representation of token t in sample at layer i.
As shown in the left panel of Fig. visual token representations in the shallow layers exhibit remark- ably high self-similarity undergoing only very minor changes across consecutive layers, indicating that the LLM backbone performs negligible processing on them in this stage.
Second, measure cross-modal influence by how much text embeddings for fixed instruction change when paired with different images, and define the resulting cross-modal similarity as Slns
Nauupe
(h ins
(l.mis) (L.eef)
S
i,in
cross
Nsample
l
(Lmis))
[|2
((.eef)
||
iin
kiins
3

---

Published as a conference paper at ICLR 2026
where h (f.mis) iins is the layer-/ instruction embedding for sample paired with mismatched image,
and h (1.ref) is the counterpart paired with a fixed reference image.
Lins
Contrary to common belief, the right panel of Fig. shows that, in shallow layers, text embeddings for a fixed instruction are nearly invariant to the accompanying image, indicating that cross-modal influence is still negligible and meaningful fusion has not yet occurred. Combined with the intra- modal analysis above, these results suggest that shallow layers primarily act as passive conduits, simply passing visual information deeper layers where substantive processing begins.
Middle Layers: Sparse Fusion Hubs In stark contrast to the passive shallow layers, the mid- dle layers emerge as the primary hubs for cross-modal fusion. At this stage, the model ac- tively integrates visual information, causing textual representations to vary significantly in re- sponse to visual input (Fig. 2). This fusion, however, is highly sparse: small subset of key visual tokens grounds the textual embeddings, rendering the vast majority other vi- sual tokens redundant. This dual characteristic, being both the center of fusion and the peak of redundancy, makes the middle layers the natural bottleneck for multimodal processing. We further substantiate this redun- dancy with training-based pruning experiments. On LLaVA-v1.5-7BB
Yeur
Token Numoee
500
we applied an aggressive middle- layer schedules parameterized by ex-
(ED
9.3)
ponential decay (ED) and generalized exponential decay (GED). In GED, an exponent p controls the decay shape, and when < p < 1 the keep ratio drops much faster in early lay- ers, enabling extremely early prun- ing. Under extreme GED schedule that reduces visual tokens from 576 just 1 across the middle layers, the model still retains 99.6% of its original GQA performance. Moreover, this robustness is not an ar- tifact of single schedule. As shown in Fig. 3 various alternative pruning strategies also maintain near- pear-perfect accuracy. Such invariance demonstrates that high visual redundancy is stable, inherent property of the middle layers, making them the ideal location for aggressive token compression.
51
52
355
50
4806 CR MOMDY AOsMBl EUIEO
Stage
Figure 3: Left: Vision token reduction curves under different values, where lower p enforces stronger pruning. Right: Model performance remains stable even under high compression rates, demonstrating robustness of our pruning strategy.
Deep Layers: Language-Doming Dominant Reasoning Once cross-modal fusion is completed in the middle layers, the net- work transitions into its final stage, which is dominated by abstract, language-centric reasoning. The direct influence of visual tokens steadily diminishes until their role becomes neg- ligible, as seen Fig. 2 We validate this with behavior on LLaVA-v1.5-7B with a training-free free "early exit" experiment, where we discard all visual tokens specific layer and ob- serve the impact on performance. As shown in Fig. 4 remov- ing visual tokens in the shallow or middle layers causes catas- trophic performance drop. However, removing them after the main fusion stage (c.g., beyond layer 24) results in almost no degradation. This finding provides strong evidence that the deep layers can operate effectively without direct access to visual information, relying instead on the fused multimodal representations formed in the middle layers. At this point, the network transi- tions fully into a language-dominal regime to refine semantics and generate the final output.
0000
60
55
50
45
doep thalto-hallls. 100% casked
35
exp-tosballeettit musket
tupsssshitacchh rudied
10
30
Layer
Figure 4: Early vision exit analysis under different masking ratios.
3 HIDROP
Building on the insights above, we propose HiDrop (Hierarchical Vision Token Dropping), frame- work that adapts pruning to the hierarchical dynamics of MI.LMs. As illustrated in Fig. we exploit hierarchical redundancy by partitioning the LLM's layers into shallow, middle, and deep stages: handle the shallow and deep stages with Late Injection and Early Exit, and apply Concave Pyramid Dropping in the middle stage to progressively reduce vision tokens.
4

---

Published as a conference paper at ICLR 2026
MAkRe
Viloo pennndar lesseing
Prograla "Shas Franig
Early (Sice Est
Allentinn Vep
DO000 COC
Differmniabee Tap- (Oura)
BeBBCC 3C 3
Deooder Layer 0
Decoder
Decoder
Bibretable Selicten
Decader Layer
Diferetabee deeceeectnnn
Decoder Layer
Dasoder Layer
Deroder
Hani Topsk
..e
Soh Conandor
Siaeeggly -x))
Und di
Ohelked Tepacced Dilinessike Topa
Figure 5: Overview of HiDrop. (a) Framework illustration, shallow layers focus on vision-independeut rea- soning, middle layers progressively prune redundant tokens through differentiable top-k selection, and deep layers enable early vision exit. (b) Comparison between hard top-k and differentiable top-k, which achieves adaptive selection and better information preservation.
3.1 SHALLOW AND DEEP: JOINT VISUAL LAYER REDUCTION
As shown in Sec. 2 visual tokens are redundant in both shallow and deep stages. We therefore combine Late Vision Injection, which delays their introduction until fusion begins, with Early Vision Exit, which discards them once language-dominn dominant reasoning takes over.
Late Vision Injection Knowing that shallow layers act as passive conduits (Sec. 2), our approach avoids wasteful computation by employing a Late Vision Injection strategy. Instead of processing visual tokens from the first layer, HiDrop bypasses the initial Luj 1 layers for the visual stream entirely. The text-only forward pass proceeds until the injection layer Lanj- where the vision tokens first introduced and concatenated with the text representations: h (hy... h J- This injection point is strategically chosen the onset of the active fusion stage, which we identify by local minimum in the visual layer-wis similarity curve (layer 9 in our experiments, Fig, 2
Early Vision Exit Our analysis in Sec. shows that deep layers transition a language-dominant regime where direct visual input is no longer required for reasoning. Therefore, HiDrop incorporates an Early Vision Exit strategy after specific exit layer Lext. all remaining vision tokens are discarded, and the forward pass continues with only the text stream. We determine this exit point by identifying where model performance plateaus in our deep-to-shallon masking analysis, indicating that visual tokens are no longer contributing (layer 25, Fig. 4.
Together, Late Injection and Early Exit create a focused "vision processing window," restricting all vision tokens to only middle layers. This targeted approach significantly accelerates both training and inference, all while preserving the model's predictive accuracy.
3.2 MIDDLE: AGGRESSIVE CONCAVE PYRAMID PRUNING
Within the core vision processing window, propose Concave Pyramid Pruning, aggressive yet adaptive strategy to manage the high redundancy found in the middle layers (Sec. 2 This ap- proach is designed to prune tokens rapidly at the start of the fusion stage and then more gradually, preserving essential information while maximizing computational savings. Implementing this strat- egy requires answering two key questions: (1) Where in the middle layers should pruning occur? and (2) Which specific tokens should be pruned at these locations?
Am Siiilaiiy
Where to Prune: Identifying Filtering Layers with ILVAS To determine the optimal layers for pruning, we introduce the Inter-Layer Visual Attention Similarity (ILVAS) metric. The core idea is to identify layers where the model has formed stable assessment of token importance, making them ideal 'filtering" points. ILVAS measures how consistently the most attended to visual tokens at one layer remain important in sub- sequent layers. Specifically, we compare the top-K attention distributions for vision tokens between layer 1 and a future
2: 4.
Layer
Figure 6: ILVAS curves for different window sizes, extended results in Ap- pendix. G.5
5

---

Published as a conference paper at ICLR 2026
layer 1 n
A!tm)
I),
1+n
where A! is the head-wise attention vector for vision token i. A high ILVAS score indicates stable filtering capacity. We compute its curve across the middle layers and select the local maxima to form our of filtering layers F (e.g., layers {10, 14, 16, 18) Fig. 6
Which Tokens to Prune: Adaptive Selection with Differentiable Top-K Once the filtering lay- are identified, the next challenge is to select which specific tokens prune. Previous methods often rely on non-differentiable Hard Top-K selection, which may lead to suboptimal token selec- tion. To overcome this, employ Differentiable Top-K (DTop-K) operator (Liu al. 2024b), which provides a continuous relaxation of the selection process..
Given vector of importance scores R-V for N tokens, the DTop-K operator first computes normalized rank score c' for each token: &4 = - E; [F 2 c)). This maps the scores to [0, 1]
range. Next, a soft mask is generated using sigmoid function with a learnable pruning ratio a:
Mask(c, a) Sigmoid - -)) 1 e --c
This soft mask produces smooth importance score for each token, enabling more fine-grained lection than hard cutoff. For the forward pass, a hard threshold is applied to the mask to make discrete token selection. By combining ILVAS to determine where to prune and DTTpp- to deter- mine which tokens to prune, our method dynamically and efficiently compresses visual information. detailed comparison with Hard Top-K is provided Sec.
3.3 SOLUTIONS TO IMPLEMENTATION CHALLENGES
Persistent Position Encoding HiDrop dynamically changes which visual tokens are active across layers because of late injection, progressive dropping, and early exit. Naively reindexing tokens under this dynamic behavior can misalign positional encodings. To avoid this, each visual token assigned persistent positional identifier input: although the shallow layers contain no visual tokens, their indices are reserved, activated upon injection, and preserved through subsequent drop- ping or exit. For RoPE, queries and keys are always rotated using these fixed identifiers, ensuring consistent relative geometry across the model.
Efficient Attention Compatibility To remain compatible with efficient attention kernels such as FlashAttention, the original attention computation is left intact over the full sequence. Token selec- tion is handled separately by a lightweight auxiliary attention pass, restricted to interactions between the final text token and visual tokens. Since this auxiliary step involves only single query, its over- head is negligible, and the efficiency benefits HiDrop are fully preserved.
Parallel Decoupling of Vision-related Operations Late injection theoretically allows us to shorten the critical-path prefill time by decoupling vision-related computation from the main at- tention stack. Before the injection layer, all transformer layers operate purely on text tokens, while in parallel we run the vision encoder once, apply the projector to obtain visual KV tensors, and cache them. At the injection layer, these cached visual KV tensors are concatenated with the text KV ten- sors, and subsequent layers attend over the combined set. During HiDrop's 's multi- pruning, we only update indices over the cached visual KV tensors instead of recomputing projections. This par- allel decoupling removes visual KV projection from the prefill bottleneck and remains compatible with Flash Atenttinnnoo kernels.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Models Within the LL architecture (Liu et al. 2023a). verify the effectiveness of the proposed HiDrop with three different LLM backbones:
MobilelLaMAA2277 (Wu et al. 2024).
Vicuna-7B-vl.5, and Vicuna-13B-v1.. (Zheng et al. 2023
The details are provided in Appendix
Benchmarks To thoroughly evaluate the HiDrop. we conduct experiments on 11 mainstream
benchmarks, including MME (Fu et al. 2023),
MMB, MMBCN (Liu et al. 2025), GQA (Hudson &
Manning, 2019). VQA v2 Goyal et al. 2017)
2 SQA! (Lu et al. 2022). VizWiz (Gurrr et al. 2018)
TextVQA (Singh et al. 2019),, POPE (Li et al. 2023b), SEED' eLi al. 2024a), and MMStar (Chen
et al. 2024c). Notably, MMStar (Chen et al. 2024c) a multimodal benchmark characterized by
strong visual dependency and minimal data leakage. See Appendix D for details.
Efficiency Evaluation We consider the efficiency in both training and inference following PDrop (Xing et al. 2024). For training, we report real GPU hours the same device; for in- ference, we report FLOPS for vision token part. Specifically, for a Transformer block, the FLOOPs from MHA and FFN are 4nd2 + 2n2d + 3ndm, where n is the number of vision tokens, is the hidden size, and m is the FFN intermediate dimension. Aggregating across layers (with ne denoting the number of vision tokens at layer E), the total FLOPs are:
FLOPs Σ (4 + +2n + 3 ngdm)
Implementation Details For DTop-K operation, we set the temperate À = N.. which means the number of the visual candidate vision tokens. For LLaVA-1.5-7B,
adopt late injection layer
Lijj = 9. early exit layer Lexit = 25, and filtering layers F = { {10, 14. 16, 18). For LLaVA-1.5- MobileLLaMA-21.
we ues Linj 15, Lexit = 28, and F {16. 19, 22, 25). All experiments are conducted on 8 NVIDIA A100 40 GB GPUs Unless otherwise stated, we follow LLaVA's default training (pretrain and instruction finetuning) and evaluation settings for benchmarks included in its suite. The evalution of the MMStar is done via LMMS-Eval (Zhang et al. 2024a) toolkit.
4.2 MAIN RESULTS
Comparison with State-of-the-art Methods To ensure a fair comparison, we conduct controlled- budget experiment under three different compression ratio. As shown in Table using LLa 7B as the base LMM, we compare HiDrop against state-of-the-art in-LLM vision token compression methods across eleven widely used benchmarks. HiDrop consistently and markedly outperforms all counterparts at all pruning ratios. Notably, it retains 98.3% and 96.5% of the baseline performance while pruning 88.9% and 91.7% of vision tokens, respectively. Compared with the most similar progressive token pruning approach, PDrop (Xing et al. 2024), HiDrop achieves higher performance on nearly all benchmarks under the 88.9% pruning ratio, with gap of 4.1 1 average performance. At even more aggressive compression, HiDrop still retains 96.5% of the bascline at 91.7% pruning, whereas PDrop cannot reach this pruning level under the same protocol.
Efficiency of HiDrop in Training & Inference As shown in Table HiDrop reduces the training time (including both pretraining and finetuning stages) of LLaVA-1.5-7B from 159.3 to 94.4 GPU hours, resulting in an impressive 40.7% reduction in overall time. In addition to the training effi- ciency improvement, HiDrop also reduces the inference FLOPs from 3.82T to 0.42T, achieving an 88.9% reduction. Moreover, HiDrop lowers the prefill latency from 63.6 ms to 32.6 ms, and can be further reduced to 31.8 ms and 28.8 ms through parallelly decoupled visual KV projection and fewer dropping stages. Notably, compared to PDrop's ' pruning ratio of 46.9%, HiDrop achieves much higher pruning ratio of 89.0%, which 4.8 times more aggressive, while the performance drop is only 1.6%, demonstrating HiDrop's superior efficiency and minimal accuracy trade-off. Similar trends are observed on and larger backbones, HiDrop consistently delivers substantial reductions in training time, FLOPs, and prefill latency under much stronger pruning ratios, while incurring only a slight degradation compared to the vanilla models.
and LLaVA- 135:-13: across both smaller
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparisons with three pruning ratios benchmarks. methods are applied on the same base model LLaVA-1.5-7B. The best result for each benchmark and pruning ratio is bolded. Dashed lines separate training- taining-free (above) and vaining-based (below) methods within each block. The denotes results reproduced using the official checkpoints; denotes training-based methods evaluated under training- free settings; denotes training-free methods evaluated under training-hased settings.
TextVQA
MMB
MMStta
MME
MMMB
GQA
VQA
SQA
Vi2Wiz
POPE
SEED
Avg %
Method
Upper Bound, 576
Tokens (100%)
LLaVA-1.5-7B LLaVA-1.5-7B*
1510.7
64.3
58.3
62.0
78.5
66.8
50.0
58.2
85.9
66.1
1506.5
64.7
58,1
61.9
78.5
69.5
50.1
58.2
86.8
66.2
33.7
100.0
Retain Tokens Average (L 86.1%)
FastV
1214.4
57.3
47.8
51.3
66.6
68.8
51.3
52.1
73.7
52.9
31.1
87.9
PDrop'
1133.1
53.6
50.8
67.2
46.9
51.5
72.0
50.8
30.8
84.5
FastVf
1348.2
62.3
53.1
55.4
68.9
68.8
43.4
49.1
80.6
55.3
34.7
91.3
PDrop
1412.1
64.6
54.7
57.9
74.3
69.8
52.4
54.3
83.7
59.3
35.0
96.8
VoCo-LLaMA
1307.0
58.0
44.5
58.7
74.2
66.7
52.1
50.4
83,9
54.7
32.2
91.2
TwigVLM
1471.5
62.8
56.4
59.5
76.8
69.7
51.5
56,9
85.0
60.7
34.0
97.9
HiDrop (Ours)
1467.0
63.7
56.3
61.3
76.6
67.5
51.4
54.9
86.6
65.3
31.2
98.4
Retain 64 Tokens Average (L 88, 88.9%) 9%)
FastV PDrop
1086.6
53.3
42.7
48.8
61.6
68.9
50.5
49.9
67.7
49.1
29.6
82.8
962.0
45.6
32.7
45.4
58.3
68.2
45.9
48.2
64.0
47.3
29.0
76.5
Fastvt
1303.8
61.7
52.7
56.2
70.7
70.0
43.8
51.0
83.1
55.6
33.8
91.8
PDrop
1350.7
63.1
54.3
56.6
71.8
70.3
51.8
51.7
82.6
57.9
32.7
94.2
VoCo-LLaMA
1256.5
55.4
44.2
58.1
73.9
66.2
51.8
49.5
83.2
54.7
33.3
90.4
TwigVLM
1404.0
60.4
53.6
58.8
75.6
70.0
51.2
55.8
82.7
56.9
33.1
95.3
HiDropp (Ours)
1473.3
63.2
58.0
60.5
76.5
52.6
55.2
86.4
64.5
32.0
98.3
Retain 48 Tokens in Average (L 91.7%)
FastV
816.9
37.3
68.7
47.6
46.3
56.1
42.4
25.6
70.2
F
1327.4
61.3
53.6
54.4
68.4
69.0
45.8
49.6
82.2
54.6
34.4
91.4
VoCo-LLaMA
1321.9
56.2
46.2
58.6
74.1
68.1
51.8
50.9
83.9
54.9
32.3
91.6
TwigVLM
1199.9
53.1
42.6
55.0
71.8
69.0
49.4
53.6
75.7
48.6
31.6
87.3
HiDrop (Ours)
1446.4
63.7
55.5
59.8
75.6
67.7
49.5
54.4
85.8
61.8
32.7
96.5
Table 2: Efficiency comparison across three LLM backbones within the VL- framework. Prefill latency (ms) is reported as actual / decoupled visual-KV fewer dropping stages.
Model
Method
Avg. Vis. Tokeus
Trinn
Infer TFlpps
Prefill Latency (ms))
MME:
MMB
GQA
VizWiz
VQA
Avg %
LLAVA-155
Vanilla
PDrop
270
576
108.4
50.3
0.70
1.52
35.3
18.7
1258.2
1231.1
54.3
57.0
59.4
57.0
30.9
32.6
47.5
48.6
100.0
96.3
ours
206
25/1/25.1/2.0
J
Vanilla
576
159.3
3.82
63.6
1506.5
61.7
61.9
50.1
58.2
100.0
LLaVA-L.5-BB
PDrop
270
107.3
L.78
43.7
1490.1
63,9
61.7
52.4
57.7
100,2
ours
206
32.6/31
8/2888
1474.3
98.66
Vanilla
576
297.2
7.44
122.8
1529.9
68.5
63.5
$3.6
61.2
100.0
LL-VA-L.5-13B
PDmp
270
213.7
3.47
74.9
1555.2
68.8
63.1
53.7
100.3
ours
2016
175.8
48.6/46.6/4355
1497.2
62.1
56.3
98.76
4.3
ABLATION STUDIES
To better understand the proposed HiDrop, we conduct three group ablation studies to investigate key attributes of several critical components: (1) Late injrection and early exit, assessed indepen- dently on the base model; (2) The effect of differentiable top-k and token importance calculation, examined within the progressive dropping setup, where vision tokens are pruned in stages (576 - 64 - + 1) at evenly spaced intervals; and (3) Position encoding and filter layer selection, analyzed within the complete shallow-miee silloo--iddldddeeyy compression structure.
8

---

Published as a conference paper at ICLR 2026
Table Performance comparison of LLaVA variants with Hard vs. Differentiable Top- Operators. PT and FT denote pretrain and finetune, respectively.
Model
Train
Top-K
MME
MMB
GQA
VQA
VizWix
VQA
MMStar
Avg(
-LaVA-1.5-7B
1506.5
64.7
61.9
78.5
50.1
58.2
33.7
100.0
Hard
1436.9
64.2
59.7
76.4
50.1
55.7
33.9
97.7
PT+FT
LLaVA-1.5-7B
Diff.
1484.7
65.5
60.2
76.3
52.7
56.2
34.3
99.7
Top-K
Hard Diff.
1482.7
65.0
60.3
76.5
46.8
55.9
33.4
97.5
FT
1471.7
65.2
59.9
76.5
47.1
56.2
34.8
98.1
Table 4: Effect of different strategies for estimating vision token saliency.
Model
MME"
MMB
GQA
VQA
VizWiz
VQA
MMStar
Avg(% %)
L.VVA-I.5-BB
1506.5
64.7
61.9
78.5
50.1
58.2
33.7
100.0
Last token (I-rounds)
1424.7
65.3
59.6
75.6
49.0
55.5
33.2
97.1
Last token (n-rounds)
1484.7
65.5
60.2
76.3
52.7
56.2
34.3
99.7
Last token (n-rounds, norm)
1447.0
65.2
59.7
76.3
48.8
55.9
34.0
97.9
token
1414.8
65.0
59.0
74.8
51.4
56.6
34.3
token (L2 norm)
1424.0
65.5
59.9
75.2
53.2
56.6
35.5
99.6
Late Injection and Early Exit Our late in- jection and early exit guided by two diag- nostics: layer aligns with a local minimum in the visual layer-wise similarities (Fig. 2). and accuracy plateaus around layer 25 under deep- to-shallow masking (Fig. Ap. We validate these choices with three sweeps (Fig. 7 In the late entry sweep, varying the injection layer with the exit fixed shows a clear peak at layer 9; in- jecting earlier adds cost with little gain, and in- jecting later degrades accuracy. In the fixed en- try span sweep, fixing injection at layer and varying the exit peaks around layers 25 to 26; later exits add cost and earlier exits hurt accu- racy. In the equal depth window sweep, sliding a constant-length window confirms 8-24 and 9-25 as near-optimal, while 10-26 underperforms. Notably, in the deep-to-shallow diagnostic, performance matches the baseline layer 26 and is only slightly lower at layer 25; we therefore choose 25 as the exit, expecting training to recover the small gap, and the sweeps verify that the 25 window strong choice.
fejetion
Exad-Extty
Equi-Deyth
Augg
96
6
Layer
Figure 7: Ablation across visual perception lay- ers comparing Late Injection, Fixed-Eni Entry Spen, and Equal- Depth Window, confirming that setting is the most efficient. The full per-henchmark results are pro- vided Appendix G.T Table
Differentiable Top-K We study hard top-k and differentable top-k under a progressive pruning schedule. As shown in Table replacing hard top-k with differentable top-k lifts the average perfor- mance from 97.7% 99.7% with two-stage training (pretraining then finetuning) and from 97.5% to 98.1% % with one-stage training (finetuning only), indicating more faithful token selection under the same training setting. Since the gain larger with two-stage training, we adopt this recipe the default in our experiments. See Appendix G.2 for additional token decay schedules.
Token Weighting Strategies We compare training-time strategies for estimating vision token im- portance. As shown in Table aggregating attention from all text tokens with L.2norm weighting performs comparably to the multi-round last-token variant, and is in fact 0.3% worse on average across 11 benchmarks (Table [0 in Appendix G.3). Given the extra cost from the eager attention used for importance calculation, we default to the multi-round last-token scheme.
Position Encoding Conceptually, similar to the "position-ID mismatch" in streaming LLMs (Tong et al. 2025b a Lin al. 2026),, but distinct in cause: ours arises from cross-layer changes in the set of surviving vision tokens due to late injection (insertion), progressive dropping (pruming), and early exit (removal). We therefore compare three positional encoding (PE) schemes: (1) Persistent PE: assign fixed RoPE indices at input and never update them: (2) Compacted PE (PDrop-style): start with preset indices and, at pruning stages, reset indices compact surviving vision tokens and

---

Published as a conference paper at ICLR 2026
Table 5: Effect of position encoding (PE) schemes under shallow-middle- compression.
Model
MME"
MMB
GQA
VQA"
VizWiz
VQA"
MMStar
Avg(%
LLaVA-1.5-72
1506.5
64.7
61.9
78.5
50.1
58.2
33.7
100.0
Persistent
1414.4
63.7
61.3
76.6
52.1
55.6
32.0
97.6
Compacted PE
1452.3
64.6
61.1
76.8
48.9
55.1
30.3
96.4
Group PE
1442.2
63.9
60.4
76.2
51.2
55.5
31.1
97.0
Table 6: Effect of instruction fine-tuning data scale (HiDrop retains 48 visual tokens in average),
Model
Data Scale
MME
MMB
GQA
VQA
VizWiz
VQA
MMStar
Avg(%) (%)
LLaVA-1.5-7B
665k
1506.5
64.7
61.9
78.5
54.4
58.2
33.7
100.0
HiDrop
IM
1446.4
63.7
59.8
75.6
56.3
54.4
32.7
97.0
LLaVA-L.5-7B
665k
1526.1
68.7
62.7
79.2
61.2
58.8
38.2
100.0
HiDrop
IM
1453,9
66.2
59.5
76.1
60.7
55.4
36.9
96.3
fill gaps; and (3) Group PE: allocate disjoint RoPE index ranges for instruction and vision tokens, with no in-place updates during injection, pruning, exit. As summarized in Table Persistent PE achieves the best average performance, Group PE is close, and Compacted PE performs worst, con- sistent with the hypothesis that resetting indices exacerbates cross-layer position mismatch. Given its accuracy and zero overhead, we adopt Persistent PE by default. More benchmark results appear in Appendix G.4
Filtering Layer Selection We first compute the ILVAS curve over the middle layers on a model configured with late injection and early exit, and select its local maxima as the filtering layers, yielding {10, 14, 16, 18} (Fig. 6) token decen- schedule that follows the concave pyramid dropping policy and sweep the filtering layers (Fig. 8). Compared with a control schedule (12, 15, 18, 21), the ILVAS-based set achieves higher average accuracy. Fixing {10, 16, 18} and sweeping the remaining slot produces clear peak at 14, whereas 12 or 13 degrades performance. Jointly sweeping the middle pair further confirms {14, 18) as the best combination; nearby alternatives {13, 18). {13, 19), and (14, 19} underperform. We therefore adopt (10, 14, 16, 18) in all main experiments.
Caccui
(AA...AA
100
To validate this choice, we
(12) (1N
(14)
(19
Figure 8: Ablation across filter layers, con- firming that our setting is the most efficient. The full per-benchmark results are provided
in Appendix G.5 Table T2
Training Data Scale The HiDrop variant evaluated in Table retaans only 48 visual tokens across all settings. We compare the base L.L and its HiDrop--quipped counterpart under two instruction fine-tuning data scales (665k vs. 1M). As the data scale increases, both the base model and HiDrop consistently improve most benchmarks (c.g.. MMB, MMB-CN, SEED-IMG, MM- Star), indicating that HiDrop benefits from additional instruction data rather than being bottlenecked by compression. At the same time, the compressed model remains close to the base model, with erage performance drops of only 3.0% (665k) and 3.7% (1M) despite operating under a much more aggressive visual-token budget. These results show that HiDrop tracks the gains of the base model as data scale grows, supporting that our layer-wise compression design is compatible with stronger instruction tuning and that the observed improvements not artifacts of under-training.
CONCLUSION
In summary, our study challenges prevailing assumptions about visual processing MLLMs and demonstrates that shallow layers only act passive propagators for visual tokens. By introducing HiDivDrop with Late Injection, Concave Pyramid Pruning, and Early Exit, we align pruning with the true hierarchical dynamics of multimodal integration. Our findings not only achieve state-of-the of-the-art efficiency-aae aificiency-acuurcyy trade-offs, but also provide new insights into how MLLMs allocate computation across layers, paving the way for more principled and scalable multimodal architectures.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work does not present any ethical concerns. Our research focuses on methodological contribu- tions and efficiency analysis without involving sensitive data, human subjects, applications that could raise ethical risks.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility cluding dataset descriptions, training details, and hyperparameter selections, are clearly documented in the main text and appendix. We further provide extensive ablation studies to justify our design choices. Upon acceptance, we will release the full codebase and scripts to facilitate replication of our results.
work. All experimental settings, in-
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.055 08774, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 1.X.::2308.1266.. 1(2):3, 2023.
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.138 13923, 2025.
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhancee projector for multimodal 1lm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13817-13827, 2024.
Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Efficient large multi-modal models via visual context compression. Advances in Neural Information Pro- cessing Systems, 37.73986-74007, 2024a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang.
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403. 20330, 2024c.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF winter conference applications of com- puter vision. pp. 958-979. 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929, 2020.
Yingqi Fan, Anhao Zhao, Jinlan Fu, Junlong Tong, Hui Su, Yijie Pan, Wei Zhang, and Xiaoyu Shen. Visipruner: Decoding discontinuous cross-mode dynamics for efficient multimodal Ilms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 18896-18913, 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.

---

Published as a conference paper at ICLR 2026
Yash Goyal, Tejas Khot, Douglas Summers- Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6904-6913, July 2017.
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 2018.
Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Ma- tryoshka query transformer for large vision-language models. In Advances in Neural Information Processing Systems, pp. 50168-50188, 2024.
Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. Dynamic-llava: Efficient multimodal large language models via dynamic vision- language context sparsification. arXiv preprint arXiv:2412.0088 2024.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real- worlwwrlld visual reasoning and compositional question answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6700-6709, 2019.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13299-13308, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal 11m. arXiv preprint arXiv:2407.02339 2024b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hal- lucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, 2023b.
Junyan Lin, Haoran Chen, Dawei Zhu, and Xiaoyu Shen. To preserve or to compress: An in-depth study of connector selection in multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5666-5680, 2024.
Junyan Lin, Junlong Tong, Hao Wu, Jialiang Zhang, Jinming Liu, Xin Jin, and Xiaoyu Shen. Speak while watching: Unleashing true real-time video understanding capability of multimodal large language models. arXiv preprint arXiv:2601. 066443, 2026.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. nextt-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL httpps://
lavavvl. gill.gittueee thub. wtt.uu..oobblloooo httpio/o//oo//////oo// 01-30-0000--00 1lava-next_
Kai Liu, Ruohui Wang, Jianfei Gao, and Kai Chen. Differentiable model scaling using differentiable topk. In Proceedings of the 41st International Conference Machine Learning, pp. 31994- 32014, 2024b.
Ting Liu, Liangtao Shi, Richang Hong. Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint 1XXv:2411.10800 2024c.
12

---

Published as a conference paper at ICLR 2026
Wenjie Liu, Hao Wu, Xin Qiu, Yingqi Fan, Yihan Zhang, Anhao Zhao. Yunpu Ma, and Xi- aoyu Shen. Vica: Efficient multimodal Iims with vision-only cross cross-attentinn attention. arXiv preprint arXiv:2602.07574, 2026.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233, 2025.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. 2024.
arXiv preprint arXiv:2403.0555
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang. Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, volume 35, pp. 2507-2521, 2022.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
OpenAl. Gpt-ov(ision) system card. Card.ppF 2023.
OpenAL. Hello got-4o, May 2024. Available: http:://opena..coo//dddxxxlllppp
Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang, Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing a twig to accelerate large vision-language models. arXiv preprint arXiv:2503. 14075, 2025.
Amanprect Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8317-8326, 2019.
Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael X. Guan, and Benyou Wang. Less is more: A simple yet effective token reduction method for efficient multi-modal llms. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 7614-7623, 2025.
Junlong Tong, Yingqi Fan, Anhao Zhao, Yunpu Ma, and Xiaoyu Shen. Streamingthinker: Large language models can think while reading. arXiv preprint arXiv:2510.17235. 2025a.
Junlong Tong, Jinlan Fu, Zixuan Lin, Yingqi Fan, Anhao Zhao, Hui Su, and Xiaoyu Shen. Llm as ef- fective streaming processor: Bridging streaming- batch mismatches with group position encoding. arXiv preprini arXiv:2505.1 250., 2025b.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv: /2191, 2024.
Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. MobilevIm: A langunel model for better intra-and inter-ui understanding, arXiv preprint arXiv:2409. 14891 2024.
Qiong Wu, Wenhao Lin, Yiyi Zhou, Weihao Ye, Zhanpeng Zen, Xiaoshuai Sun, and Rongrong Ji. Accelerating multimodal large language models via dynamic visual-token exit and the empirical findings, 2025. URL httppp/:/wxi..oo////apaaaa
19628
Long Xing. Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui Hc, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision- lsiounlanguage models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410. 172477. 2024.
Sengiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang. Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-19802. 2025.
13

---

Published as a conference paper at ICLR 2026
Linli Yao, Long Xing, Yang Shi, Sida Li, Yuanxin Liu, Yuhao Dong, Yi-Fan Zhang, Lei Li, Qingxiu Dong, Xiaoyi Dong, et al. Towards efficient multimodal large language models: A survey on token compression. Authorea Preprints, 2025.
Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Aip-llava: Adaptive token pruning for large vision language models. arXiv preprini arXiv:2412.00447 2024a.
Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, and Yansong Tang. Voco-Ilama: Towards vision compression with large language models. arXiv preprint arXiv:2406. 12275, 2024b.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check the evaluation of large multimodal models, 2024a.
12772
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.088 00..29900.0895,, 2025.
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevlm: Visual token sparsifi- cation for efficient vision-language model inference. arXiv preprint 2rX1v:2410.04417, 202444
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Yuke Zhu, Chi Xie, Shuang Liang, Bo Zheng, and Sheng Guo. Focusllava: A coarse-to to-fine ap- proach for efficient and effective visual token compression. arXiv preprint arXi::241..14228, 2024.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
We employed large language models (LLMs) solely as general-purpose writing assistants for lan- guage refinement, including improving clarity, grammar, and style. Importantly, LLM was in- volved research ideation, methodological design, analysis, or result interpretation; the role of the LLM limited linguistic polishing. All substantive contributions originated from the thors. This ensured that the scientific content remained entirely authored by the researchers, while benefiting from improved academic writing quality.
B RELATED WORK
distinctive property of MILMs is that vision tokens are far more numerous yet information-sparye compared to text tokens (Mam 20TO), making them the primary source of redundancy and motivat- ing research on token compression. Most prior work is training-free, pruning vision tokens during inference via heuristic rules (Chen al., 2024b; Zhang et al. 2024b| Yang al. 2025 Liu al., 2024c). While effective in reducing computation, these methods introduce train -nne-neffeenee mis- match. To address this issue, training-basse -bsed approaches learn token reduction end-to-end, achieving alignment between training and inference and enhancing adaptability.
Among training-based methods, previous studies can be grouped into Pre-LLM, In-LLMM, and joint approaches, according to where the reduction is applied. (1) Pre-L.MM approaches compress to-
kens before the L.LM via compact projectors (Cha al.] 2024 Li al. 2024b) or encoder-side modules (Hu et al. 2024 Song al. 2025 Zhang al. 2025). Such approaches remain discon-
nected from the LLM's internal reasoning, preventing compression from adapting to cross-modal interactions. (2) In-LLM approaches integrate compression into the LLM, enabling strategies for token selection, aggregation, or reduction. Some methods perform representation compression by replacing vision tokens with latent tokens (Ye al., (202b)) or by pooling operations (Chen al. 2024a), while others adopt selection-basee -baee pruning, either through heuristic schedules (Xing et al,, 2024 Shao al. 2025 Liu al. 2026) or adaptive strategies (Ye al. 2024a). However, most pruning approaches rely on non con-differentiable Top-k operators. hindering end-to-end optmization. DLnaMic-LLaVA (Huang al. 2024) relaxes this with soft gating but still provides only approxi- mate gradients, whereas our differentiable Top-k yields a continuous relaxation with stable gradient flow. (3) Joint approaches combine the strengths of both Pre-LLM and In-LLM strategies, e.g., Fo- CusLLaVA (Zhu et al., (2024), which applies vision-guided pre-LLM compression and text-guided pruning inside the LLM. While such hybrid designs demonstrate the potential combining both perspectives, their two-stage pipeline increases architectural complexity and prevents unified end- to-end optimization. Our work instead focuses on the In-LLM setting, aiming to achieve effective compression with fully differentiable and text-aware token selection strategy.
C LLM BACKBONES
Table 7: Detailed settings of LLM backbones.
Model
Blocks
Heads
Hidden Dim
FFN Dim
MobileLLaMA 2.7B
32
32
2560
6912
Vicuna-7B-v1..
32
32
4096
11008
Vicuna-13B-v1..
40
40
5120
13824
We use two decoder-only LLM backbones within the LLa VA-15 framework: MobileLLaMA
2.7B (Wu et al. 2024) and Vicuna-7B-v1.5 (Zheng et al. 2023). As shown in Table 7 both have 32 transformer blocks and 32 attention heads. MobileLLaMA 2.7B uses a hidden size of 2560 with an FFN dimension of 6912, while Vicuna-7B-v uses 4096 and 11008, respectively. Unless other- wise noted, all other architectural and training settings follow LLaVA defaults; our method changes only the vision token schedule and leaves the tokenizer, projector, and attention kernels untouched.
15

---

Published as a conference paper at ICLR 2026
D BENCHMARKS
We conduct experiments on 11 mainstream benchmarks, including MME-Perception (Fu et al. 2023), MMBench, MMBench-CN (Liu et al., 2025), GQA (Hudson & Manning 2019).
VQAv2 (Goyal et al., 2017), ScienceQA-lange (Lu et al. 2022). VizWiz (Gurari al., 2018
TextVQA (Singh et al. 22199. POPE (Li et al. 2023b). BEED-Image (Li et al. 2024a), and MM-
Star (Chen et al. 2024c).
MME- MME-Perccption (Fu et al. 2023). A subset of tasks within the MME benchmark that focuses on answer pairs to ensure that the model must genuinely "understand" the image comprehend the text respond, rather than relying on memory or data leakage.
evaluating a model's 's perception abilities. It relies on manually constructed mnstuertoo-answer
Comprehensively measures model's performance across different ability dimensions. It not only assesses whether the model can "understand" images or text but also evaluates its reasoning ability, knowledge integration, and more refined cognitive performance.
MMBench (Liu et al. 2025
GQA (Hudson Manning, 2019). Used to evaluate a model's understanding and reasoning abilities on real images. It emphasizes scene understanding and logical reasoning, not just the recognition of individual objects.
VQAv2 Gooaa et al., 2017). Evaluates model's visual perception ability through open-ended questions. Its core objective is to test whether the model can understand the content of an image and provide reasonable answers based on the questions.
ScienceQA-lange (Lu et al. 2022). Aims to evaluate a model's multimodal understanding, com- plex reasoning, and explainability abilities, covering multiple domains including natural sciences, language sciences, and social sciences.
VizWiz (Gurari et al. 2018). Used to evaluate model's visual understanding under real-world, non-ideal image conditions. Its goal is to test whether the model can provide accurate answers in low-quality images and real-world question scenarios.
Tex (Singh et al. 2019). Focuses on evaluating model's ability to understand textual infor- mation images. It requires the model to recognize, read, and reason about the text the image, and then generate correct answers by integrating visual information.
POPE (Li et al., 2023b). Used to evaluate the degree of object hallucination in models. Its core objective is to quantify the extent to which model produces hallucinations, helping researchers understand the model's reliability in visual perception and generation.
SEED-Image (Li et al. 2024a). Evaluates a multimodal large model's ability to understand and generate image content. Its goal is to test the model's comprehensive multimodal abilities visual perception, spatial reasoning, and image-texy text interaction tasks.
MMStar (Chen et al. 2024c). Aims to address insufficient visual dependency and data Icakage issues in current multimodal evaluations. It defines 6 core visual-language (VL) abilities and con- structs 18 detailed evaluation dimensions based on them, covering multiple aspects from coarse perception to fine-gr fine-grained reasoning.
Protocol. Unless otherwise noted, we follow the official LLaVA evaluation protocol for all bench- marks above; MMStar is evaluated via LMMS-Eval.
E INTRODUCTION TO BASELINES
We conduct comparisons under the LLa (Liu et al. 2023a) framework to ensure consistency and fairness across different approaches. Specifically, we evaluate our method alongside several representative vision compression techniques, including FastV (Chen al, 2024b), PDrop (Xing
et al. 2024), VoCo-LLaMA Ye et al. 2024b) and TwigVLM (Shao et al. (022)..
V (Chen et al. 2024b). A general plug-and-ple method that prunes unnecessary visual to- kens in the early filtering layer according to attention score ranking, thereby significantly reducing inference cost without sacrificing performance.
16

---

Published as a conference paper at ICLR 2026
10t Inbamddllly Saniuiity
Similait
minammodaltyy.... Sanitaity
Similarity
6.0-
8.00
0.44
Syewn
Sydwe
835
Vixad
035
Vixud
Tecuall
0.55
Testal
660
17
850
17
Layer
Layer
Layer
Layer
(a) LLaVAw1.--obbllLLaMMM222
(b) LLaVA-v (LLaVA-v1.5-7B)
1.00
Cooer mo0lll Siuularyy
Imzamddally Sinilrity
Iitammdiliy
Synre
8:5-
Synee
0.900
LLANN-1S LaNAAA
0.875
Layarr
(c) LLLVAAIS-LLMAA.....
batamndaliy Sinileit
Sieilerity
modalty Simllaity
Sinilare
8.05
2.55
0:958
603
Syewn Vinua Tocual
2.54
ea3
Syaee Mead
0.322
LLAW Narta
Taxnal
0.900
11a VA
Layer
(c) LLaVA-
(f LLAVANNeTTHLLMAAA5
Figure 9: Layer-wise representational dynamics, where each subfigure consists of left panel show-
ing intra-modal refinement and a right panel highlighting cross modsl interaction intensity
PDrop (Xing al., 2024). An approach dividing the LVLM into several stages, discarding part of the image tokens at the end each stage based on lightweight similarity computation with a predefined ratio, with negligible time overhead.
VoCo-LLaMA (Chen et al. 2024b). The first method to compress visual information using LLMs, distilling the LLM's understanding of visual tokens into compact representations, compressing hun- dreds of visual tokens into a single VoCo token while minimizing information loss.
TwigVLM (Shao al. 2025). A method that trains lightweight twig block on the early layers of the base VLM, and through twig-guided token pruning (TTP) strategy and self-speculative decoding (SSD) strategy, achieves better accuracy and faster generation.
F EXTENDED ANALYSIS
F.1 LAYER-WISH REPRESENTATIONAL DYNAMICS ANAYLSIS
To demonstrate that the phenomena observed in this paper are universal, we conducted layer-wise representational dynamics analysis on various LLM backbones and model sizes within the LLaVA- v1.5 framework, including MobileLLaMM-2.77B Vicuna-7B, -LaMA3.1-8B and Vicuna-13B. As shown in Figs. Pa- 9d. all these L.MMs exhibit similar trends and behaviors: (1) the intra-modality similarity in shallow layers starts at relatively high level, then decreases and remains low for while, before gradually increasing again and stabilizing at a higher level; and (2) the cross-modality similarity is also relatively high in the shallow layers. Besides, we also performed the same analysis under the LLa VA-NeXT framework using Vicuna-7B and LLaMA3-8B as backbones, as shown in Figs. 9e and 99 and observed highly consistent patterns in both intra-modality and cross- modality similarity across layers. This further supports the universality of the identified phenomena across different L.M architectures and multimodal training pipelines.
17

---

Published as a conference paper at ICLR 2026
Table Complete berbhenchmark results corresponding Fig. i, E; denotes early visual exit at layer
denotes visual injection at layer
MME
MMB
MMB
GQA
VQA
SQA!
VieWiz
TextVQS QA
POPE
SEED
MMSSta
Avg. %
Model
Baseline
1506.5
64.7
58.1
61.9
78.5
69.5
50.1
58.2
86.8
66.2
33.7
100.0
Late Injection
1441.9
66.0
62.4
69.5
55.9
65.6
100.1
1442.5
65.8
62.4
69.3
56.5
66.1
99.7
I7
1413.8
66.2
58.8
62.2
78.3
69.8
48.6
56.6
86.4
65.3
33.1
99.0
Is
1424.1
65.1
58.2
62.7
78.3
69.1
50.7
57.3
87.1
65.9
31.9
99.1
I
1444.4
65.4
57.9
61.5
77.9
68.9
53.0
56.1
86.5
65.3
32.7
99.3
7
110
1402.3
63.5
54.4
62.0
77.8
68.9
50.6
56.8
87.0
63.77
32.7
97.7
8
In
1392.8
63.1
56.4
61.9
77.7
68.6
50.9
57.6
86.8
62.8
33.3
97.7
Fixed Free--njjetion Span
Ig & Fiz
1456.2
63.8
56.4
61.9
78.1
68.1
50.9
57.6
86.9
64.3
31.8
98.4
10
1g & Ezs
1438.2
63.9
56.6
61.9
78.1
66.7
51.9
58.0
86.7
65.0
32.9
98.7
Ig & Ez4
1461.5
63.9
57.4
61.7
78.1
68.7
51.2
57.8
86.7
64.1
31.9
98.7
1o Eas
1436.6
65.8
56.4
62.5
77.9
67.1
51.5
57.3
87.1
65.3
33.5
99.2
1. Eas
1460.8
65.4
57.9
62.2
78.1
68.8
50.9
57.4
87.1
65.0
35.2
100.0
1g Fer
1435.9
65.2
58.2
62.4
78.0
68.5
48.3
56.7
86.9
64.9
33.1
98.6
Ezs
1467.2
65.2
62.4
68.3
56.2
65.0
99.2
Window
16
Ig & E24
1441.5
64.7
56.2
61.7
78.1
68.0
50.1
57.7
87.2
65.0
34.7
99.1
17
Io & Eas
1436.6
65.8
56.4
62.5
77.9
67.1
51.5
57.3
87.1
65.3
33.5
99.2
18
I10 & Eas
1383.4
62.4
53.3
61.6
77.8
68.1
51.3
56.8
86.7
63.1
30.6
96.6
G EXTENDED EXPERIMENTAL RESULTS
G.1 LATE INJECTION AND EARLY EXIT
Our design of late injection and early exit is guided by two key diagnostics. First, layer coincides with a local minimum the visual layer-wise similarity curve (Fig. (2). suggesting a natural entry point for visual tokens. Second, accuracy plateaus around layer 25 under the deep-to-shallow mask- ing experiment (Fig. indicating a reasonable cutoff for discarding vision tokens. We validate these choices through three sets of sweeps (Fig.
(1) Late injection sweep. Varying the injection layer while fixing the exit depth reveals a clear peak at layer 9. Injecting earlier increases computation with negligible gains, whereas injecting later leads to accuracy degradation.
(2) Fixed-entry span sweep. With injection fixed at layer 9, varying the exit depth yiclds optimum around layers 25-26. Exiting later adds cost, while exiting earlier reduces accuracy.
(3) Equal- window sweep. Sliding constant- window confirms 8 24 and 9-25 near- optimal spans, while 10-26 underperforms.
Notably, in the deep-to-shallow diagnostic, accuracy at layer 26 matches the baseline and at layer 25 is only marginally lower. We therefore select 25 as the exit depth, expecting training to recover the small gap. Taken together, these ablations validate the 9-25 window strong design choice for balancing efficiency and accuracy.
G.2 DIFFERENTIABLE TOP-k
Here present detailed results on the advantages brought by differentiable Top-k op- erator. In the main text, we compared hard and differentiable Top-k under a progressive pruning schedule (Table 3 showing that replacing hard Top-k with differentiable Top-k improves the
18

---

Published as a conference paper at ICLR 2026
erage score from 97.7% to 99.7% with two-stage training (PT+FT) and from 97.5% to 98.1% with one-stage training (FT only).
Appendix Table 9 further demonstrates that the gain of differentiable Top-k is most pronounced under high compression ratios. For example, when the number of visual tokens reduced from the original 576 to few as 72, hard Top-k suffers clear degradation, whereas our differentiable Top-k consistently preserves accuracy across benchmarks. The improvement is especially evident in vision-heavy tasks such as MMBench, SQA-I, and VizWiz, where more faithful token retention plays a critical role.
These results confirm that differentiable Top-k provides smoother selection mechanism that adapts to training signals, making it particularly effective in aggressive pruning regimes. We therefore adopt PT+FT with differentiable Top-k as the default configuration in all main experiments.
Table 9: Performance comparison of LLaVA variants with Hard vs. Differentiable top-k Operators. PT and FT denote pretrain and finetune, respectively.
TarVQA
MMStar
MMR
MMB
MMB
GQA
VQ4
SQA
POPE
SEXAN
Avgl
Model
Topk
LLaVA-1.5-BB
1506.5
64.7
58.1
61.9
78.5
69.5
50.1
58.2
86.8
66.2
33.7
100.0
S76
64
PT+FT
Hard
1436.9
64.2
59.7
76.4
70.4
55.7
86.5
63.1
33.6
98.0
LLaVA-I.5-7B
Dift.
1484.7
65.5
56.3
60.2
76.3
71.5
52.7
56.2
86.2
63.3
34.3
99.3
TopK
Hard
1482.7
65.0
54.9
60.3
76.5
69.9
46.8
55.9
86.0
63.5
33.4
97.5
Diff.
1471.7
65.2
56.6
59.9
76.5
70.7
47.1
56.2
85.9
63.2
34.8
98.2
G.3 TOKEN WEIGHTING STRATEGIES.
Table TO reports the detailed results of different strategies for scoring visual tokens during training. We evaluate both last- last-token based methods, which compute importance by repeatedly attending from the last text token across multiple rounds, and all-token based methods, which aggregate attention from all text tokens to vision tokens. For each family, we also test variants that incorporate L2-norm weighting.
The results show that while all-token strategies slightly improve performance on some individual benchmarks, their overall average is not better than the multi-round last-token baseline. For exam- ple, the best all-token variant achieves 99.6% average, compared to 99.9% for the last-token (n-R) variant. Given the additional computational cost of eager attention required by all-token approaches, we conclude that the multi-round last-token scheme provides the best trade-off' between efficiency and performance.
G.4 POSITION ENCODING
Table II provides the detailed benchmark results of the three positional encoding (PE) schemes com- pared under the -iilloowddddleddeepp compression setting. As discussed in the main text, the un- derlying challenge is conceptually similar to the "position-ID mismatch" in streaming LLMs (Tong al., 2025b). but arises here from dynamic changes in the set of surviving vision tokens across layers due to late injection, progressive dropping, and early exit.
We evaluate three PE strategies: 1) Persistent PE: fixed RoPE indices assigned at input and never updated across layers. 2) Compacted PE PE (PDrop-style): indices are reset after pruning to compact surviving tokens and fill gaps. 3) Group PE: disjoint RoPE index ranges are allocated for text and vision tokens, avoiding in-place updates during token injection removal.
As shown in Table II Persistent PE achieves the highest average performance (97.8%), supporting the hypothesis that stable positional assignments mitigate cross-layer mismatch. Group PE performs slightly worse (97.1%), suggesting that disjoint indexing viable but not superior. By contrast, Compacted PE yields the lowest accuracy (96.9 9%)) confirming that index resets exacerbate position inconsistency. Given both its accuracy and zero additional overhead, we adopt Persistent PE as the default in all main experiments.
19

---

Published as a conference paper at ICLR 2026
Table 10: Different strategies for scoring visual tokens. Last-token variants are computed using
repeated attention from the last text token, while all-token variants aggregate attention from all text tokens, with or without L2-norm weighting.
MME
MMB
MMB
GQA
VQA
SQA
VizWiz
TexVQQ
POPE
SEED
MMsttr
Avg %)
Model
LLaVA-1.5-7B
1506.5
64.7
58.1
61.9
78.5
69.5
50.1
58.2
86.8
66.2
33.7
100.0
Last token (1-R)
1424.7
65.3
56.9
59.6
75.6
71.0
49.0
55.5
86.2
63.0
33.2
97.7
Last token (n-R)
1484.7
65.5
56.3
60.2
76.3
71.5
52.7
56.2
86.2
63.3
34.3
99.3
Last token (n-R. L2)
1447.0
65.2
56.8
59.7
76.3
70.6
48.8
55.9
86.5
63.5
34.0
98.2
All token
1414.8
59.2
59.0
74.8
70.3
51.4
86.4
63.4
34.3
98.6
(L2)
1424.0
65.5
58.7
75.2
68.9
56.6
87.0
35.5
Table 11: Effect of position encoding (PE) schemes under shallow-middle-dey compression. Per-
sistent PE with fixed RoPE indices performs best overall, while resetting indices (Compacted PE) leads to accuracy degradation.
MME!
MMB
MMB
GQA
VQA
SQA
VieWie
TeaTVQA
POPE
SEED
MMStar
Avg(
Model
LLaVA-1.5-7B
1506.5
64.7
58.1
61.9
78.5
69.5
50.1
58.2
86.8
66.2
33.7
100.0
Persistent PE
1414.4
63.7
56.7
61.3
76.6
67.0
52.1
55.6
86.9
65.2
32.0
97.8
Compacted PE
1452.3
64.6
56.1
61.1
76.8
67.9
48.9
55.1
86.5
64.6
30.3
96.9
Group PE
1442.2
63.9
55.4
60.4
76.2
67.6
51.2
55.5
86.9
63.6
31.1
97.1
G.5 FILTERING LAYER SELECTION
Table reports the detailed per-benchmark results for the selection of filtering layers. We first com- pute the ILVAS curve the middle layers on model configured with late injection and early exit. As shown in Figure 10 the ILVAS profiles are consistent across Top- E {5, 10, 20, 50, 100, 200} and window sizes E {4, 8). with local maxima occurring at layers 10, 14, 16, 18. We therefore select {10, 14, 16, 18} as the filtering layer set F.
To validate this choice, we fix the concave pyramid token decay schedule and sweep different layer configurations. Compared with a control schedule {12, 15, 18, 21}. the ILVAS-basee selec- tion achieves consistently higher average accuracy. Fixing (10, 16, 18) and sweeping the remaining slot yields a clear peak at 14, whereas 12 or 13 lead to noticeable degradation. Similarly, joint sweeps of the middle pair confirm {14, 18) as the strongest combination, while nearby alternatives such as {13, 18). {13, 19). and (14, 19) underperform. These ablations confirm {10, 14, 16, 18) as our final filtering-layer configuration for all main experiments, balancing efficiency and accuracy across tasks.
Table 12: Per-benchmark results for different filtering layer configurations under the concave pyra- mid dropping policy. The LLVAS-based set (10, 14, 16, 18} achieves the best trade-off.
MME?
MMR
MMB
GQA
VOA"
SQA"
VizWzz
TORVQQ
POPE
85220
MMStar
Avgr %
#
Model
1 Baseline
1506.5
61.7
61.9
78.5
69.5
58.2
86.8
33.7
100.0
2
(12.15.18.21)
1431.0
64.6
59.5
61.4
77.4
67.4
46.5
56.2
86.7
65.3
32.0
97.7
3
116112.16,18)
1452.9
61.3
55.2
60.7
76.8
67.6
49.3
54.1
86.4
64.5
31.7
96.5
4
[10113.16.18]
1459,3
64.8
56.8
60.0
76.3
68.1
49,3
55.3
86.6
64.5
29.9
96.9
5
166,14,16,18)
1469.5
65.0
56.2
60.9
76.7
69.0
50.8
55.1
86.1
64.7
33.1
98.3
6
16,11)
1468.9
77.2
56.2
86.8
64.5
30.6
7
16,11
1459.3
64,8
56.8
60.0
76.3
68.1
49.3
55.3
86.6
64.5
29.9
96.9
8
10.
1460.9
63.6
60.8
76.6
54.8
86.6
31.8
97.5
9
118
1469.5
65.0
56,2
60.9
76.7
69.0
50.8
55.1
86.1
64.7
33.1
98.3
110,1411.19 10
1472.6
64.0
$7.2
60.5
76.8
68.5
47.5
55.1
86.2
64.6
31.5
97.2
20

---

Published as a conference paper at ICLR 2026
S-V end IEAt vi and
M.WArr end
S.V en
Imuu Imunlayy Aut Streltantt
It. vi 5:76
1E vi end
42-
1%
11
16
11
19
11
15
17
Tayer
Tayar
Tayer
(a) T
(b) Top-10
Top-20
0.5
0.5
base Layee
Tayer
Layor
Jayer
(d) Top-50
(e) Top-100
(l) Top-200
Figure 10: ILVAS curve over the middle layers on model configured with the late injection and early exit. (f)-(f) sweep top-k {5, 10, 20, 50, 100, 200), and each curve compares observation windows n = 4 and n = 8. Consistent valleys across indicate layers with strong filtering ability, i... candidates for the pruning set F.
These results support our final choice of {10, 14, 16, 18) as the filtering layers for all main experi- ments, balancing efficiency and accuracy across tasks.
21


Reviews:
[{"review_id": "xqKAHdnTaH", "reviewer": "Reviewer_gcbi", "summary": "This paper studies the visual token reduction problem for the multimodal large language models (MLLMs). The author analyzes the diverse impact of the visual tokens in shallow, middle, and deep layers for LLaVA-1.5-3B and LLaVA-1.5-7B, and proposes a hierarchical division-based vision token dropping (HiDivDrop) method where the shallow layers are handled with Late Injection and the deep layers are handled with Early Exit, and apply the Concave Pyramid Dropping in the middle layers to progressively reduce vision tokens. Experiments are conducted on the LlaVA-1.5 architecture with different LLM backbones and 11 mainstream benchmarks, verifying the effectiveness of the proposed method.", "questions": "Please refer to the Weaknesses section for details.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is effective in reducing a large amount of visual tokens compared to the state-of-the-art methods on extensive benchmarks.", "weaknesses": "1. Generalizability of the analysis in Section 2. The analysis in Section 2 serves as the primary motivation for the proposed method. However, the details about the training and evaluation datasets, training configuration, and the exact model type (e.g., the LLM backbones) are missing. It is challenging to convince the reviewer that the empirical observation in Section 2 is generally applicable to various pretrained datasets, evaluation tasks, training configurations, and model types for MLLM. The author should provide more context for the empirical observation to let the reader understand the generalizability of the observation.\n\n2. The applicability of the Joint Visual Layer Reduction. In Section 3.1, lines 227-228 and lines 232-234, the author selects the specific layers for Late Vision Injection and Early Vision Exit based on the empirical observations in Section 2. However, as mentioned in Weakness 1, the pretrained datasets, evaluation tasks, training configurations, and model types for MLLM are not clarified in the main paper, making it hard for the reviewer to understand whether this layer selection is also applicable when we change to other pretraining datasets, evaluation tasks, and model types. \n\nMoreover, the author does not provide a concrete description of how to identify those in practice; for example, should we evaluate the MLLM performance, as in Section 2, every time we are given new datasets, evaluation tasks, and model types? Should we analyze the training dataset or the validation dataset to identify the optimal layers? How can we guarantee that the selection is optimal when we change the model training recipe? \n\n3. The choice of MLLM architecture. In Section 4, the author only evaluates the LLaVA framework as the MLLM. However, it is not clear whether the proposed method can be applied to other MLLM frameworks. \n\n4. Technical Contribution is limited. Most of the methods used in Section 3 have existed previously, and the overall novelty of the proposed method is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2PoxcBJScS", "reviewer": "Reviewer_8C26", "summary": "The manuscript unveils several key mechanisms as to how the visual information are processed in MLLMs, and use these findings to guide the design of HiDivDrop, a new visual token reduction method for MLLMs. The first finding is that shallow layers are merely propagators of visual contents into middle layers where true vision language fusion happens. Hence, HiDevDrop only inserts visual tokens at the beginning of middle layers. The second finding is that the fusion process happening in the middle layers is highly sparse, leading to the drastic reduction in the number of visual tokens during the middle layers in HiDevDrop. The final finding is that deep layers does not possess the capability of vision-language interaction. Thus, HiDevDrop drops all tokens in deep layers. Experiments show that HiDevDrop can retain strong performance under low visual token number regime.", "questions": "1. What is the difference between the two figures in Figure 2? \n\n2. Further related to Figure 2, how is the statement in L151~L152 reflected? ('Fig. 2 shows that text embeddings are nearly invariant to the visual input in shallow layers')\n\n3. What does p value mean in Figure 3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The complete method is built on close observation of the vision-language fusion process in MLLMs, which not only produces reasonable model structures, but also provides important insights for future vision-language model research. \n\n2. The proposed method achieves a strong performance under low token number regime, retaining 96.5% of the performance using 48 tokens compared to original 576 tokens.", "weaknesses": "1. While the identified internal mechanisms are potentially insightful, the presentation and explanation of these mechanisms is of limited quality. Key concepts lack sufficient clarification, making readers incapable of following the reasoning process that helps produce the conclusion. This is especially the case for Figure 2 and 3, (questions detailed in the Questions section below). \n\n2. The analysis seems to be limited to a single type of language model, which lowers the credibility of the generality of the observed underlying vision-language fusion mechanism.\n\n3. Under a similar training budget, it seems the proposed method achieves lower performance than PDrop. Though at a lower inference computation budget, the important metric is the inference latency, which the manuscript fail to provide. It is also confusing why the method, reducing so many visual tokens (64 compared to 270 in PDrop), requires similar training budget. \n\n4. The experimental analysis on the proposed method is not sufficient. For example, it would be crucial to know how the the training/inference cost and performances change when the injection becomes later (starting from first-layer injection). \n\n5. It seems costly to determine the injection layer and the exit layer for each given MLLM, how can the hyperparameter be efficiently determined?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYahGkXD5o", "reviewer": "Reviewer_VaQT", "summary": "This paper introduces a novel visual token pruning methodology for VLMs, aimed at enhancing their inference efficiency. The proposed approach is methodologically straightforward: it involves skipping visual tokens in the shallow and deep layers while employing a 'learnable top-k selection mechanism' for tokens in the middle layers. Through experimental evaluation, the paper demonstrates that the proposed method outperforms baseline approaches under identical settings. Furthermore, the finding that visual tokens in the shallow layers play a limited role is an insightful observation. This insight holds significant heuristic value for the VLM research community.", "questions": "1.  Is the pruning strategy (e.g., skipping policy, selection parameters) uniform across different tasks? \n2.  How general is the proposed method? Could it, for example, be applied during the pre-training phase rather than just as a post-hoc optimization or fine-tuning strategy?\n3.  The experiments are conducted on LLaVA-1.5, which may be undertrained. Do these conclusions consistently hold for more recent, powerful models (e.g., Qwen2.5-VL, Gemma3-VL, Qwen3-VL) and across a broader spectrum of tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The finding that visual tokens in most shallow layers are also dispensable is a novel insight that holds significant heuristic value for the VLM research community.\n2.  The methodology itself is sound; skipping non-essential layers combined with a learnable token selection in the intermediate layers allows for maximal visual token compression, a claim that is substantiated by the experimental results.", "weaknesses": "1.  The paper lacks a detailed justification for *why* visual tokens in shallow layers are non-essential. This finding serves as a critical premise for the proposed method, yet the supporting argumentation provided is neither detailed nor sufficient.\n2.  The experimental validation relies on relatively older, and arguably undertrained, VLM models (e.g., LLaVA-1.5). It remains unclear whether the conclusions generalize to more recent, powerful models (e.g., Qwen2.5-VL, Gemma3-VL, Qwen3-VL) and across a broader spectrum of tasks.\n3.  The description of the training strategy is unclear and requires polishing. It is not specified at which stage the learnable token selection mechanism is trained (e.g., during pre-training, SFT, or task-specific fine-tuning). This ambiguity hinders a clear understanding of the methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "COccyfvWOL", "reviewer": "Reviewer_doEa", "summary": "This work proposes HiDivDrop, a token pruning methods for vlm. The author claim there two main challenging in exist works. They are shallow layers are misinterpreted and pruning schedules are overly rigid. The author combined late Injection, concave pyramid pruning in the middle layers, and early exit to sort these challenges. The experiment shows the author proposed methods performance well.", "questions": "1.The authors only tested two backbones. If possible, could they test different sizes or other backbones to observe the generalization of the proposed method?\n\n2.Does the proposed method exhibit knowledge drift? That is, in individual cases, answers that were previously correct may now be incorrect, and vice versa.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments are comprehensive and validated the effectiveness and acceleration performance of the proposed method.\n\nThe proposed method is well-designed, and the experiments further validate the claims.", "weaknesses": "1. The writing of this paper needs improvement. There are too many paragraphs. The purpose of Chapter 2 is unclear. Could it serve as the first section of the methodology, validated experimentally? As a separate chapter, it feels redundant with Chapters 1 and 3.\n\n2. Tables 1 and 2 are too far removed from the analysis of the results to be easily readable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)