Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Authors: Qiong Wu
Wenhao Lin
Yiyi Zhou
Weihao Ye
Zhanpeng Zeng
Xiaoshuai Sun
Rongrong Ji
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Modal
Token Prune
Primary area: applications

Abstract: In this paper, we study the visual redundancy problem of multimodal large language models (MLLMs) from the perspective of attention behaviors. 
Via extensive empirical experiments, we observe and conclude three main inference stages of MLLMs:
(i) Early fusion between tokens is first accomplished quickly. 
(ii) Intra-modality modeling then comes to play. 
(iii) Multimodal reasoning resumes and lasts until the end of inference. 
In particular, we reveal that visual tokens will stop contributing to reasoning when the text tokens receive enough image information.
Based on this observation, we propose an effective method to improve the efficiency of MLLMs, termed dynamic visual-token exit (DyVTE), which is orthogonal but collaborative to previous token-wise visual compression methods.
To validate the efficacy of DyVTE, we apply it to a set of MLLMs, including LLaVA, VILA, EAGLE and InternVL.
The experimental results not only show the effectiveness of our DyVTE in improving MLLMs' efficiency, e.g., DyVTE reduces the computation overhead of LLaVA-1.5 by up to 45.7% without performance drop, but also reveal a general pattern across multiple MLLMs, well facilitating the in-depth analysis of MLLMs. 
Our code is anonymously released at https://anonymous.4open.science/r/AnonymousDyVTE-26AB/.

Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and Empirical Findings
Qiong Wu Wenhao Lin 12, Yiyi Zhou Weihao Ye', Zhanpeng Zeng', Xiaoshuai Sun Rongrong Ji
1 Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. 2 Institute of Artificial Intelligence, Xiamen University. 361005, P.R. China. Iqiong, (anhaolin]@ass xmu. en, zhouyiyi 0xmu edu. en, cn, weihaoye@stu. xmu, edu. en, (zzeng, xssun, rrji)0xmu edu.
Abstract
In this paper, study the visual redundancy problem of nultimodal large language models (MLLMs) from the perspective of attention behaviors. Via extensive empirical experiments, we observe and conclude three main inference stages MLLMs: (i) Early fusion between tokens is first accomplished quickly. (ii) intra-modality modeling then comes play. (iii) Multimodal reasoning resumes and lasts until the end of inference. In particular, we reveal that visual tokens will stop contributing to reasoning when the text tokens receive enough image information. Based on this observation, we propose an effective method to improve the efficiency of MLLMs, termed dynamic visual-token exit (DyVTE), which is orthogonal but collaborative to previous token-wiig wise visual compression methods. To validate the efficacy of DyVTE, we apply to a set of MLLMs, including LLaVA, VILA, EAGLE and InternVL The experimental results not only show the effectiveness of our VTE in improving MILMs' efficiency, e.g. VV reduces the computation overhead of LLaVA-115 by up to 45.7% without performance drop, but also reveal general pattern across multiple MLLMs, well facilitating the in-depth analysis of MLLMs. Our code is released at https: //gi thub. com/ DoubtedSteem/DyyVT
Introduction
Recently, the rapid development of vision-language learning has been witnessed with the great success of large language models (LLMs) 2
53 (4). Numerous efforts are devoted to equip LLMs with multimodal capability, i.e., multimodal large language models (MLLMs) M1
6
16
46
52
29
36 40 (6). To overcome visual shortcoming (34 49] in extreme high computation overhead, recent MLLMs often resort to higher-resolution images as input, which often accompanied with a
31
multitude of visual tokens
IS 29 43
However, the multitude of visual tokens lead prohibitively expensive computation. For instance, compared with LLa VA 1.5 [40), LLaVA-NeXT 29 adopts about 1728 more visual tokens, resulting in about times more FLOPs. Despite the performance gains, recent works 62 37 also show that these large amount of tokens are obviously redundant, leading to great waste of computation. For instance, Ye al. (62) show that randomly dropping half visual tokens barely affects performance on common VL tasks, such as MMB [4] and SQA 42). In this case, the efficient use of visual tokens has recently become a research hot-spot, and attracts an influx attention
13 55 57 However,
CCorresponiing Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mutimodil Reaseeingg
Layer 16
Layer 24
Full Attention Marrix
Aitn.
Text Att.
Keep
Farly Inrr-moddllty
1 Visual Takend
Fusion Medeling
If Similar
Cross Attn
Remove
PAA
Image Self Attin
IV Visual Tokenst
Text Self Attn
Kayst
<ingg
<imga
<img
Figure I: Left: Illustration of three main stages observed in MILMs. During inference, MLLM will go through three main stages, i.e., early fusion between all tokens, intra modality modeling of the same-mon model tokens, and multimodal reasoning between all tokens again. Right: The impact of visual tokens on the text self attention at the multimodal reasoning stage. Removing visual tokens at an appropriate time barely changes the text attention patterns, indicating that visual tokens contribute little to multi-modal reasoning.
existing research mainly focuses evaluating token-wits redundancy, e.g., token pruning 7 62 and token merging [32 47], lacking in-depth exploration of the intrinsic behaviors MILMs.
In this paper, we are dedicated to understand how MLLMs use visual tokens and how they behave during multimodal reasoning. To approach this target, we conduct extensive empirical studies about the attention behaviors of MLLMs, and observe and summarize three main stages of their inference, as illustrated in Fig Left. At the first stage, an MILM quickly accomplishes the exchange of multimodal information at its shallow layers, which we term (i) early fusion. Afterwards, this information exchange decreases, and the tokens primarily engage intra-modal interactions, termed (ii) intra-modalic modeling. At the last stage, visual tokens will resume the information propagation to the text ones, and this process will continue until a certain layer, and we term (ii)) multimodal reasoning. Through extensive comparisons, we reveal that these observed behaviors of MLLMs are common across different models [12 36 40 48)
D
and tasks 22 63].
Behind these shared behaviors of MLLMs, we observe a key finding in terms of visual redundancy, that is visual tokens will continue to propagate their semantics the text ones at the multimodal reasoning stage, while their impact can be very limited. As shown in the right sub-figure of Fig I removing visual tokens at a suitable layer of MILMs does not produce significant changes to the text self self-attention distributions. This case suggests that visual tokens actually contribute little to multimodal reasoning at the last inference stage. In other words, multimodal reasoning only happens within text tokens after they receive enough visual semantics. This finding is also confirmed in some very recent works (37] where the manual removal of visual tokens does not decline the performance of MLLMs some tasks. But in this paper, also found that the optimal time to remove visual tokens is distinct for different examples, tasks and even MLLMs. And the manual exploration is experimentally expensive and hard to meet the global optimum. Therefore, when and how to automatically remove visual tokens is still a thorny challenge.
Motivated by these observations, we propose novel and effective method to improve MLLMs' efficiency, termed dynamic risual-token exit (DyVTE). In particular, we apply lightweight hyper- networks to perceive text token status and then dynamically decide the exit of all visual tokens, thereby speeding up inference. As discussed above, VTE mainly focuses on the overall contribution of all visual tokens during multimodal inference, which is orthogonal but collaborative to token-wise solutions like token pruning 27 (62). In our experiments, we show that these two paradigms can be casily combined to boost the efficiency of MLLMs. In addition, VYE also differs from previous dynamic early exiting methods [9) 21 51] which often refer to an inference break i.e., skipping rest layers for direct predictions. In contrast, text tokens continue to transform in DyVTE, which better meets our empirical findings. Moreover, we also introduce an efficient training regime for VyE independent to the global gradient computation of MLLMs, achieving effective and dynamic visual-token exit with very cheap training expenditure.
To validate VTE, we apply it to set of advanced MILMS with varying scales, including LLaVA- 1.5 VILAA (36]. EAGLE (88 and Intern VI. [22, and conduct experiments on bunch of
widely-used VL and MLLM benchmarks [20] 22 23
49 63 The experimental results show

---

that our DyVTE can greatly reduce the computation overhead of MLLMs, while retaining their competitive performance on various benchmarks. For instance, our DyVTE reduces the computation overhead of L.-.V--1.5 by up to 45.7% without performance drop. When combined with token pruning methods like FastV 77, DyVTE can help LLaVA-1.5 reduce up to 51.5% computation with only 0.7% performance drop on average.
Overall, our contributions are three-fold:
We study the problem of visual redundancy from the perspective of MLLMs' behaviors, and reveal the dependency between text and visual tokens.
Based on the empirical findings, we propose a novel and effective approach to reduce visual redundancy of MLLMs, termed dynamic visual-token exit (DyVTE), which can dynamically evaluate and schedule the contributions of visual tokens to multimodal reasoning.
The extensive experiments on a set of MLLMs well validate the motivation and effectiveness of DyVTE, also providing insights into the principle of MLLMs.
2
Related Work
Based on the successful large language models (LLMs) [ 16 2
14
25
numerous multimodal
and achieved terms of methodology, most MLLMs often project the extracted visual features onto the semantic space of LLM for both uses a projection layer
large language models (MLLMs) have been recently proposed 31
40
3
29
remarkable progresses on various vision-language tasks 22 20
23
60)
multimodal fusion and reasoning [4 56 48 12. For instance, LLaVA
40
to transform visual features into input tokens for LLaMA [54]. BLIP-2
31 introduces QFormer to
aggregate query-related visual features as the input visual tokens. Similarly, QWen- also adopts learnable tokens as queries to obtain limited number of visual tokens B
To improve the ability in fine-grained tasks 49 45), some recent MILMS resort to increasing the resolution of input
images [29] 56 10 for better visual understanding. For instance, LLaVA-NeXT [39] concatenates the tokens of each crop of the image as the input of the LLM. InternVL2.3 [0] and Qwen2-VL VL [4] introduce dynamic resolution designs to match different images and preserve their visual details. Despite success, these high-resolution settings also inevitably increase the number of visual tokens, leading to excessive computation overhead 62]
Moreover, recent studies also show that the excessive use of visual tokens brings in obvious redun- dancy in both information and computation 35 58 59. In this case, the efficiency study of MLLMs
also becomes a research hotspot, of which focus ranges from structure design [35 58 59), token
pruning (27 18 5 and token merging [7 5) The research of structure design mainly aim to build
a new and lightweight MLLMs [35] 65], and our work is closer to the token efficiency researches [62] 77. For instance, FastV ? applies the averaged attention scores evaluate the importance of each visual token and then drop the less important ones to reduce computation. FitPrune [62] resort to statistic principle to quickly produces a token pruning regime for MLLMs based on the metrics of visual and cross attentions. PruMerge [47 speeds up the inference of MLLMs by merging visual tokens that have similar semanties. G- G-Prune [26] proposes an graph graph-based algorithm to select the most representative visual tokens to avoid redundant semantics and computations. Compared with these token-wise approaches, our DyV focuses more on the overall contributions of visual tokens in MILMs, providing an alternative but orthogonal way for efficient MILMSs In terms of dynamic inference, our works are also related to the dynamic early-exit studies for L.MSs [17 9. However, these methods focus on the skipping of redundant layers for early prediction akin to previous dynamic methods [5] 21). Recently, VA-Mini (64] introduces pre-fusion module through full pre-training and SFT, reducing visual tokens to just one. And our Dy is to conduct the early removal of visual tokens while keeping the original inference pattern. Moreover, this paper also involves quantitative analyses about the attention behaviors of MLLMs, providing insights into the mechanisms for their multimodal reasoning.
3 Inference Pattern Analysis of MLLMS
We first quantitatively investigate the attention behaviors of MILMMs, and then examine the impacts of visual tokens during multimodal inference.
3

---

LLaVA-1.5 7B
LLaVA-1.5 13B
Aftention Score
53
Layer Index
Layer Index
EAGLE-X5 7B
InternVL 7B
Aftcation: Score
21
Layer Index
Layer Index
Cross A Attention
Visual Self Attention
Text Self Attention
Figure 2: The averaged attention scores of four MLLMs terms of cross, visual and text attentions. The background colors denote the three summarized stages of MLLMs, i.e,, early fusion (green), intra-modality modeling (purple) and multimodal reasoning (blue).
LLa 7B GQA
LLaVA-1.5 TexVQA
LLaVA-L.5 7BSQA-
1LaVA-1 55 7B MM-VVt
32.6
GQA
LLaVA-13
436
632
Baseiine
203
01
253
T2A
46.
12
24
24
32
12tt Layer
bxtt Legee
Enit Lager
Ebot Laper
Figure 3: The relationship between manual visual-token exit and performance on LLaVA- 7B and 13B. We show the results of removing all visual tokens at O-th, 8-th, 16-th, 24-th and 32-th layers. "Baseline" represents the original performance of the MILM. After certain layer, the removal of all visual tokens barely impedes performance, but the layers for removal are different for different tasks.
Text Solf Attention
Text Self Attention
Text Attentiooo
Text Self Attention
10
10
10
10
Orignal
Origial
Original
Orianal
Enropy
Entopy
Eneopy
Exi
Entoppy
50
28
24
14 12
Figure 4: The entropy of text self-attention distribution with different layer to remove all visual tokens. We visualize the entropy of text self-attention on LLaVA-1.5 7B on GQA benchmark with different with different layer remove all visual tokens, which shows that the removal of visual tokens after a certain layer barely affect the text self-attention.
Attention behaviors of MILMs. We first visualize the attention patterns of four representative MLLMs 400 48 [2] of two scales 2 These attention data includes visual self-attention, the hoium summarized attention scores on the LLaVA-splir [40]. From these plots, we can first observe that four MILMS share similar patterns for three types of attentions. At the shallow layers, all MLLMs show high cross-attention values between visual and text tokens, suggesting the intensive interactions between two modalities. This corresponds the early fusion stage we termed above, and it denotes the quick exchange of multimodal information. But earlyfusion declines quickly, and can observe that the self-attention between the tokens of the same modality becomes the main activity at the second stage, i.e., the modeling mainly involved in each modality. In this case, we can assume that MLLM starts the intra- modaatty modeling stage. At the last stage, the visual tokens will
visual-text cross-attention and text self-attention?] Here, the values in these distributions
TThe visualization details provided in appendix A.1

---

resumee information propagation to the text tokens, and this process will continue to oscillate until the end of inference. Here. we term this process as the multimodal reasoning stage. Notably, we can find these patterns exist in MILMSs of different families and scales.
The impact of dropping all visual tokens. Next, we quantitatively measure the importance of visual tokens. In Fig 3 we present the results of removing visual tokens of LLa 7B and 13B on four benchmarks at different layers. From these plots, we can first observe that visual tokens are not always required throughout the entire process, especially the last stage. For instance, on GQA, removing all visual tokens at 24-th layer only has 0.5% and 0.6% performance drops for LLaVA-1.5 7B and 13B, receptively. This finding is also consistent with some recent efforts [37] that visual tokens receive little attention in deep layers. Another finding is that the removal of visual tokens have little impact on the modeling among text tokens. As shown in Fig the removal of visual tokens that does not harm the performance and also has no impact on the modeling pattern among the text tokens, e.g., after 24th layer on GQA. The only difference lies in the slight change of attention intensity due to the shorter token sequence. More importantly, we can find that the exit time of visual tokens is different for different examples and tasks. For instance, the optimal exit layer of SQA is earlier than that of TexiVQA, i.e., 16 v.s. 25, suggesting the need of dynamic early-exist modeling. Overall, these results confirm our argument that visual tokens stop contributing to prediction after certain layer, and also shows the need of exploring dynamic and automatic strategies for effective visual token removal.
Discussion. Based on the above observations and analyses, we can categorize the inference of MLLMs into three main stages. (i) Early Fusion. In the initial stage, the MLLM quickly accomplish the exchange of multimodal information from visual to text tokens. (ii) Intra-modality Modeling. Next, the self-attention intra the modality becomes the main activity, enhancing vision and language understanding. (iii) Multimodal Reasoning. Finally, the visual tokens will resume the information propagation to the text tokens, though this transfer demonstrates limited impact on final response generation. In addition to these three stages, we can also conclude one common behavior of visual tokens in MLLMs, i.e., the contribution of visual tokens to multimodal prediction diminish after some layers. Moreover, these results also show that the exist times for different types of VL examples are different, well motivating our exploration of dynamic visual-token exit.
4 Dynamic Visual-token Exiting
4.1 Method
In this paper, we propose novel and effective method to reduce the visual redundancy of MLLMs, termed dynamic Eisual-token exit VTE). VYE uses lightweight hyper- to perceive the text token status of MLLMs and then adaptively judge the right time remove all visual tokens.
Concretely, given the text tokens (Kk) at the k-th layer of an MLLM, we use a simple MLP as the hyper-network learn their status and decide whether to remove all visual tokens, defined by
p = Softmaz (GLUUa//GGLLL(( (k) I T T() W:))
(1)
where p E R2 is the binary prediction, W1 E R2dxh and Wa Rhx2 are weight matrices, and is the dimension of the hidden states.
In particular, the hyper-network of Eq] uses two text token representations, i.e., aug( (T: and Tr The former is the averaged text tokens representing their overall state. The latter Te is the last token, which often plays an important role for decoding under the uni-directional self-attention setting of MLLMs [38 (5). With these two types of text representations, Dy VTE can well perceive the state of MLLMS and automatically decide whether additional visual information is still needed for reasoning.
When the prediction p is exit, VTE will remove all visual tokens after this layer, while the text ones are kept in the rest layers of MLLMs. This process can be denoted by
P' = (T+1:(T(")
where (-) denote the remaining layers the MILM. Compared with previous dynamic exit methods that skip layers for early prediction 17 9. the principle of our DTE is to make the removal of visual tokens based on the text token status. Thus, its implementation requires no structure tweaks, e.g., adding new prediction layers, and also no the update of MLLMs.
5

---

4.2 Optimization
In particular, given sufficient examples of visual-token exit, Dy can learn to judge the text token status at each layer of the MLLM. Thus, its accurate removals of visual tokens will greatly speed up the inference in the rest layers of the MLLM, while retaining similar predictions.
Thus the objective of DyVTE can be defined by
argmin d(P, P(),
(3)
0n
where 02 denotes the weights of heper-setworks in DyVTE. d(:;-) -) is KL-Divergence 28 And P/ and P are the predictions of an MLLM with and without early visual token exit, respectively.
To optimize E a direct approach is to merge the predictions of hyper-networks with visual tokens of MLLMs, e.g. implementing attention masks based Gumbel softmax (24], thus using the next token prediction loss to indirectly update VyV.E. Although feasible, this approach still requires the computation of the model's all gradients, which is still inefficient and expensive.
In this case, we propose an efficient training regime independent to the gradient pack-propagation of MLLMs. Concretely, we can compare the discrete outputs of the MLLM with and without Dy VTE, e.g. the answer strings A. If the answers are exactly the same, we can give positive feedback to hyper-networks, and vice verse. By comparing it with the default output A, we can judge that whether the visual tokens should be exited at I-th layer:
1,
A A,
y
0,
A/
Here, A denotes the answer predicted with visual-token exit the 1-th layer. Besides, to make this supervision more robust, we also consider the prediction uncertainty as a regularization term, thereby making the MILM behaviors closer to the default inference:
1, A AA Pla <T
(5)
0, otherwise.
Here, Pl. denotes the prediction uncertainty valued by cress-entropy and multiplied by a scaling factor, and 7 denotes the threshold.
With this supervision, the hyper-networks in VTE can be optimized by the cross-entropy loss:
Lp = - log(p:) + (1 y) loggpo)).
Compared with the indirect optimization using next token prediction [6]]. Eq can provide more
effective supervisions to Dy VTE. We randomly sample exit layers at each training step. and compute the labels y according to E Finally, the hyper-networks The overall expenditure of Dy VTE training will be much cheaper than tuning MLLMs or learning new prediction layers in previous dynamic exit approaches 61).
optimized based loss defined in Eq 6
5 Experiment
5.1
Datasets and Metries
The benchmarks used in this paper consist of four conventional vision-language (VL) benchmarks and five newly introduced MILM benchmarks. The traditional VL benchmarks include VQAv2 GQA 23 ScienceQA 42 and TextVQA 49] The MLLM benchmarks comprise POPE (34).
22
MME 20), MMB 41, SEED 30 and MM- Vet (63) Different from general VL evaluation, the
MLLM benchmarks focus more on the evaluations of MLLMs, such as fine fine-grained reasoning 63
and visual hallucination [34].
5.2
Implementation Details
We apply VY to five popular MLLMs, namely EAGLE-X5 7B 48 VILA 7B 36,, Intern VL 7B [1], LLaVAA 1.5 7B 40) and LLaVA- 13B 40). The scale for p' in Eq 5 is set to 1.03. For all MLLMs, the hidden dimension of the hyper-network is set to 2,048. During training, all MILMs are frozen, and the training rate of hyper-networks is set to 4 x 10-5, And we randomly sample % examples of the 1LaVA-605k instruction set 440 train VTE for epoch. More details can refer
to our code project.

---

Table 1: Results of MLLMS with and without DyVTE on five MLLM benchmarks. The accuracy (higher is better) and TFLOPs (lower is better) are reported. The relative percentage change from the baseline model to Dy VTE is also shown in parentheses.
Method
SEED Accuracy
MME
MMB TFLOPs...
POPE TFLOPs
MM-Vet Accuracy
IFLOPs L
Score
TFLOPs J Accuracy
Accuracy
TFLOPs &
EAGLE-XS EAGLE.DyVTE
73,9
47.8
1528.0
27.8
68.4
29.6
88.8
27.7
37,4
27.6
73.6
43.0 c100
1581.7 (4589) 20.3 (2705)
68.8 68.8))05)
23.7 c1909)
cas4)
(-18
23,5 (3109)
VILA 7B
61.7
9.2
1489.2
8.9
69.9
9.5
86.3
8.8
36,3
8.7
VILA-DyVTH 7B
61.8
5.9 (3599)
1503.1 (401)) 4.6 (4839)
69.8.
6.0 63689)
85.6
(45.9)
36.7
6-119)
6.6 (3419)
InternVI. 78
59.2
16.0
1525.1
15.5
64.6
16.2
86.4
15.4
31.2
15.4
LolenVL-DyVTE 78
6929)
11.9
(25
1474,]
10.9
(207))
(439)
12.0
(3509)
6533)
(392%))
(544)
13.0 (4569)
LLaVA-1.5 7B
58.6
9.2
1510.7
8,9
64.3
9.6
85.9
8.8
30,5
8.7
LLaVA-DyVTE 7B
58.6 mori
5.0
(44.779
1491.
4.3 (3128)
64.7 (400))
5.4 643.50
(1.6
4.1 (32,90)
(-409)
6.3 (137005)
LLaVA-L5 13B
61.6
17.6
1531.3
16.9
67.7
85.9
36.1
16.7
LLaVA-DyVTE 13B
(973)
7.1 (59779)
1546.4 (41.0))
7.2 (17.5)
7.8 estooo
84.8
7.6 (5138))
34.8 (s0R)
10.6 (36.5))
Table 2: Results of MILMS with and without VTE on four VL benchmarks. The accuracy (higher is better) and TFLOPs (lower is better) are reported. The relative percentage change from the baseline model to VT is also shown in parentheses.
Method
Accurncy 1 TFLOPs
GQA
Score 1 TFLOPs
VQA
Accuracy 1 TFLOPs
TextVQA
SQA-I Accurncy
TFLOPs I Accuracy TFLOPs I.
Average
EAGLE-X57B
64.9
27.8
83.4
27.8
71.2
29.5
69.8
29.2
72.3
28.6
EAGLE-DyVTE 7B
62.4 (s99; 21.7 (110))
82.6
(4.9))
c16
71.7 (-275)
(25.3))))))
71.7 (a0r))
VILA 7B
63.1
8.8
80.3
8.8
62.6
9.5
69.5
9.8
68.8
9.2
VILA-DyVTE
61.9
(199)
5.5 (33.99)
79.2 (144) 5.4 (33509)
(122
7.2 (2424)
69.5 aLos)
6.1 (33188)
67.9 (133))
6.0 (3448))
InterVL 7B
62.9
15.4
79.3
15.4
$7.0
16.1
66.2
16.4
66.4
15.8
IntenVL-DyVTE 7B
(2.5)
11.8
(22.9))
(3.19)
11.7 (2409)
(2.19)
13.5 (9)
66.2 m.0%)
12.1 (2629)
65.2 (1.89)
12.3 (22229)
LLAVA-1.5 7B
62.0
8.8
78.5
8.8
58.2
9.5
69.4
9.8
67.0
9,2
11aVA-DyVTR
(300)))))))
76.6 (1/0))
62.6))
6.7 10)
(-36
5.5 (-199)
65.7 (199))
5.6 19)
1LaVA-1.5 13B
63.3
16.8
80.0
16.8
61.3
18.1
72.9
18.6
69.4
17.6
LLVVA-DyVTB
62.3 (1.6%)
(45.4))
78.8 (1.5%)
8.9 (4706)
(3.9)
10.8
(.039)
72.3 (456)
8.2 63509)
68.1 (1199)
(47.78)
5.3
Quantitative Analysis
Results of DyEEE on different MLLMs. In Tab and Tab we report the results of ap- plying VTE to a set of MILMs of different families and sizes, including EAGLE [48]. VILA [36]. IntervLL [12), and LLaVA-115 [4). From these tables, we can first observe that VT significantly reduces the computational overhead of existing MLLMs. For ex- ample, DyVTE reduces FLOPs of LLaVA 7B by 45.7% on SEED without dropping perfor- mance. Additionally, the actual reduction of FLOPs are distinct for different tasks. For instance, on SQA with multiple-choice questions, VTE reduces computational overhead by up to 43.9%, and on MM-Vet about granular answers, the reduction achieved is 27.6%. These results confirm our intuition that the optimal removal of visual tokens are different for differentt tasks. Another observation is that DyVTS's effects vary across these MLLMs. Specifically, VTE can reduce the computa- tional overhead of VILA 7B by 48.3% on the MME benchmark with no performance drop, whereas Intern VTE 7B only achieve a reduction of 29.7%. When applying VTE to larger MLLMs, such VA. 13B, we can observe a more significant efficiency im- provement. For instance, VTE reduces the computational overhead of LLa VA-155 13B by 55.9% on SQA with only -0.8% on per- formance. Furthermore, DyVTE is more ef- fective for MLLMs that uses stronger visual representations. For instance, EAGLE-X5 is a SOTA MLLM with strong and hybrid visual backbones, and it works very well with our DyVTE VTE can reduce its computation by up to 6.2 TFL.Pss with only 0.8% performance drop on the VL benchmarks. Overall, these results not only validate the effectiveness of our VTE but also confirm motivations and findings about MLLMs.
Table 3: Comparison of real inference budget be- tween token pruning methods and vanilla decoding. "Acc." "
denotes the accuracy.
SQA-I
MMB
Method
Latency 1
Acc. 1
Latency
Acc. 1
LLavA 13B
0.237s
72.9
0.236s
67.7
FasV ToMe
0.171s (21.9) 73.1 (4039)
0.1748 (2829)
68.6 (11.9)
0.175s (2822)
73.2 (40.55)
0.179%
(22:24)
67.4 (444)
DyVTE
0.161s (33.8) 72.3 c4u9:: 0.1638 (3199:
66.0
DyVTE+FasV 0.1144 (34.9) 73.0 (4025)) 0.155s (-34.44; 68.5 (+129)
7

---

Text-Image Cross Attention
Text Self Attention
Toxt-Image Cross Attention
Text Self Attention
Enropy
10
DyVTE
10
DyVTE
Adomu
10
DyNTE
10
DyVTE
12
1 8 12
12
(1) SQA-1
TextVQA
Text- Lost-Image Cross Attention
Self Attention
Text- Imate Cross Attention
Sell' Attention
wi DyVIz
byViE
byvil
by
Entrogy
10-
DyVTE
Introyy
10-
D-VIE
10.
DDVTE
TIN 10.
ByVIE
12 16
12 16 24
12
12 20 28
(c) GQA
(d) Met
The entropy of two attention distributions. We visualize the entropy of cross-attention matrices and text self-attention on LLaVA-1.5 7B with and without our Dy which shows that the removal of visual tokens barely affect the text self-attention.
Figure 5:
Table 4: Ablation study of different token statuses for VTE. "Mean Visual" denotes the average of all visual tokens except the last one, similar with "Mean Text". "Last Visual" and "Last Text" denotes the last visual or text token. "Exit Layer" denotes the averaged layer numbers selected to remove by VTE "Acc." denotes the accuracy. The default setting of VTE is the last row. The best and second best results are marked in bold and underline, receptively.
State
GQA
TestVQA
MM-Vet
SQA-I
Average
Meun Visual
Visual
Mean
Text
Last Text
Acc.
Layer
Acc.
Exit Layer
Asc.
Layer
Acc. I
Layer
Acc.
Layer
Exit
61.4
21.6
57.2
21.0
31.5
21.4
69.7
21.7
55.0
21.4
61.2
21.3
57.1
21.1
30.0
21.0
199.7
21.4
54.5
21.2
00.1
12.4
57.6
22.1
31.1
22.1
09.2
20.4
$4.6
20.5
59.8
16.3
56.3
19.1
29.9
17.6
61.1
21.2
57.3
21.3
31.6
21.3
69.7
21.0
54.9
21.2
58.7
16.5
57.6
22.2
32,7
22.5
69.7
21.3
54.7
20.6
59.4
16.3
56.5
19.9
30.9
21.8
69.7
14.4
54.1
18.1
59.4
16.7
21.8
31.3
22.0
69.6
20.3
$4.4
20.2
59.3
16.4
19.9
69.6
14.4
18.1
60.0
16.4
19.7
31.9
69.6
13.4
17.6
Inference Latency. In Tab. we compare the actual inference latency of Dy with the token prune and the vanilla decoding methods. From the table, we can first observe that token prune methods can significantly reduce the inference latency. For instance, FastV can reduce the inference time by -27.7% 7% and -26.2% on SQA and MMB benchmarks, receptively. Compared with these token prune methods, our DT further enhances inference efficiency. Specifically, Dy improve the inference efficiency by 32.1% % and 30.9% on SQA and MMB benchmarks with competitive performance. Moreover, the proposed DyE can work together with existing token prune methods and achieve better efficiency. For example, when combined with FastV, DyVTT achieves even greater reductions inference latency, i.e., 34.9% and 34.4% on SQA and MMB benchmarks. Overall, these results validate our DYV can accelerate the inference.
The attention entropy with visual tokens exit. In Fig 5 we visualize the entropy distributions of tross-attention matrices and text self-attention matrices of VLaVA-1.5 7B. And we compare the results with and without VTE across four benchmarks. We first observe that there are differences between the multimodal reasoning procedures across different benchmarks. Specifically, in SQA and Tex the changes in entropy during text modeling are more obvious than those in GQA and MM-Vet. This phenomenon fully illustrates the necessity of adopting dynamic approach to exit visual tokens. Another observation is that VYE can effectively remove all visual tokens at a specific layer according to the status of multimodal reasoning. Specifically, after removing all visual tokens by DyVTE, the entropy of text self self-attention remains unaffected. In this way, the exiting determined by DyV only blocks visual information's 's cross-mdaaity propagation while preserving text tokens' modeling. Overall, these findings validate that visual tokens are not always necessary in the reasoning process, and also confirm the effectiveness of the proposed VTE
A Ablation study, In Tab[4 we perform set of experiments to analyze the effectiveness of different token status for Dy In this table, various tokens are used as the token statuses in Eq As shown Tab[4] when using only image-related tokens, the model tends to remove all visual tokens at a later stage. Specifically, the average exit layer are 21 .4 and 21.2. Using the "Mean Text" token leads to an earlier exit from the layers, but still retains the computation redundancy. When only the "Last Text" token is used VYE removes all visual tokens too carly, i.e., at the 17-th layer, resulting
8

---

Table 5: Comparison between DYV and token pruning methods. The best and second best results are marked in bold and underline.
Method
SQA-I
MM-Vet
SEED
MMB
Average Accuracy
Accuracy TFLOPS
Score
TFLOPS Aecuracy TELOPS J Accuracy TFLOPS
TFLOPs J
LLaVA 7B
69.4
9.8
30,5
8.7
58.6
9.2
64.3
9.6
55.7
9.3
ToMe [5
69.6 (4)
5.9 c3855)
30.6 (4019)) 4.9 (45.99)
6148
5.5 (-1025)
63.7 (499)
5.7 (-005))
55.4 (4.5%)
5.5 (-1095)
FastV
69.0c
6.2
(3675)
31.3
(e306)) 5.2 63588)
58.2 (678)
5.8 (37,5))
64.4 (-029)
6.0 (3758)
55.7
5.8 (37.8))
69.6 640.9%
5.5 (-n-99)
31.9 (4105))
6.3 637.88)
58.6 0.00))
5.0 (-57775)
64.7 (-008)
5.4 (-118%)
56.2 (4099)
5.5 (-10959)
DayTE+FasV
68.9 (0.7%)
(31,0%) 29.8 :2 (2.5%) 4.0 (54.0%
58.2 (4.7%)
4.6 (30005)
64.4 (-025)
(52.19)
55.3 (4.7%)
4.5 (31.5%)
Table 6: Results of MLLMS with Dy that trained on themselves and others on five benchmarks. "Trained" denotes the MLLM that DyVTE trained on. "Exit Layer" denotes the averaged layer numbers selected to remove by DyVTE. "Ace." " denotes the accuracy.
GQA
VQAY2
SEED
MMB
MME
MLLM
Trained
Exit Layer
Exit Layer"
Exit Layer
Exit Layer
Exit Layer
Acc.
Acc.
Acc.
Acc.
Acc.
63.1
80.3
61.7
69.9
1489.2
VILA
VILA 7B
61.9
17.4
16.7
61.8
17.6
16.2
1503.1
LLavA 7B
61.3
17.1
79.1
17.1
61.8
15.8
69.8
16.5
1504.9
14.6
62.9
79.3
59.2
64.6
1525.1
IntemVI. 7B
61.3
16.1
77.6
15.8
59.1
13.9
64.4
13.6
1474.1
12.1
LLaVA 7B
61.3
17.0
77.7
16.8
59.2
18.1
64.9
17.8
1516.9
14.1
in obvious performance drop, i.e., performance drops by 1.0% on average. For combinations of two representations, we can observe that the "Mean Text + Last Text" is the most effective way. Specifically, it removes visual tokens at about 17-th layer, while causing only 0.5% performance drop. On the other hand, using a combination of visual token statuses, i.e., "Mean Visual + Last Visual", yields results similar to using single visual representation. In this way. the visual tokens are retained until the 21-st layer. Overall, these results confirm that the key to judging whether visual tokens have a necessary impact on the prediction lies in the status of text tokens.
Generalization to different MLLMs. To evaluate the generalization capability of DyVTE across different MILMs. conduct experiments where Dy is trained on one MLLM and directly applicd to another without additional fine-tuning As shown in Tab we compare the performance of DyVTE trained on the same MLLM versus trained on LLaVA 7B and applied to VILA 7B and InternVL 7B. The results demonstrate that VTE exhibits strong cross-model generalization. For instance, when applying DYE trained on LLaVA to VILA, the model achieves 61.3% accuracy on GQA with an average exit layer of 17.1, which is comparable to the specifically trained VTE (61.9% accuracy with exit layer 17.4). Another notable observation is that the optimal exit layers identified by cross-model VTE are close to those of specifically tuned VTE. On the MME benchmark. both VILA with LLaVA-trained VTE (exit layer 14.6) and VILA with self-trained DyVTE (exit layer 13.1) achieve similar efficiency gains. These results indicate that Dy has learned generalized patterns of visual token importance across different MLLM architectures. This strong generalization capability stems from the task definition of Dy VTE as a simple binary prediction problem, which makes the optimization independent of specific MILM gradient updates and enables direct supervision via binary labels based on MILMs' predictions. Overall, the cross- model experiments validate that VTE can be trained once and effectively deployed across different MLLMs with minimal performance degradation.
Comparison with token pruning methods. In Tab 5 we compare the performance and efficiency of DyVTE with the visual token pruning methods on LL-1.- 7B. From the table, we can observe that VTE outperforms token pruning methods in both performance and efficiency, For example, when compared with ToMe on SQA, VTE not only maintains the performance but also further reduces FLOPs by 4.1 1%. When compared with FastV, the advantage of Dy is more significant. Specifically. VTE improves performance by 0.4% while reducing computation overhead by 8.7% on the SEED. Another observation that VTE can significantly improve performance in complex tasks. For example, in the MM- Vet benchmark, which evaluates multiple functions including mathematies and OCR, DYE outperforms ToMe by 4.3 4.3Additionally, VTE shows excellent compatibility with existing methods. When combined with V DYE reduces the computational overhead by 51.5% while only decreasing the performance by 0.7% on average. Overall, these results validate the effectiveness and efficiency of Dy VTE.
9

---

5.4 Qualitative Analysis
To gain insight into the proposed DyVTE, we visualize the effect of applying it on LLaVA- 7B in Fig. 6 We can first observe that DyVTE maintains consistent predictions with the default LLaVA. As shown Figure (a), DyVTE can correctly identify the object in question even when the foreground complex. Additionally, DyVTE provides much faster inference than the default VL. Specifically, in the two given examples. Dy VTE can improve inference speed by up to 10.2% compared to the default LLaVA. Another observation is that removing redundant shown in (b). applying Dy can help sign. Overall, these results confirm that VTE enhances inference efficiency without harming the reasoning process of the default model, aligning well with our motivation.
(e) Quesdion: Whit's But Shing in the upper right come??
VAV la the
DyVTE (Ours) thane padiuge
FostV) corve there packaze of broccull.
right cxenzz than is packags
the
raspberiis.
eapperrio..
34.5
32.3
Question:
street sighs?
signe
number and
mamberrs and 6637
on Bom
617
31.7 bokeess
sakees
32.3 sakenss
Figure 6: Examples on LLa with VTE.
Our Dy can help the MLLM answer the ques- tions as accurately as default, while being faster. visual information may enhance performance. As LLaVA to better recognize the numbers on the
6 Limitation
While Dy VTE shows promising efficiency gains across multiple MLLMs and benchmarks, Dy currently performs full visual-token exit, which may not be optimal for cases where partial visual information still beneficial later layers.
Conclusion
In this paper, we investigate the visual redundancy problem of MLLMs via analyzing their inference behaviors, and summarize three key stages MILLSs during multimodal inference, ie., early fusion, inter-modality modeling, and multimodal reasoning. Moreover, we also reveal that visual tokens stop contributing to multimodal reasoning after some layers. Motivated by this insight, we propose a simple yet effective method for MLLMs, termed Dynamic Visual-Token Exit (DyVTE) which optimally determines the layer at which visual tokens can be removed, thereby reducing computational overhead without compromising performance. Extensive experiments on nine benchmarks demonstrate that VTE significantly enhances the efficiency of MLLMs while maintaining their performance.
Acknowledgment
This work was supported by the National Science Fund for Distinguished Young Scholars (No. No.662025633, the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383. No. U21A20472, No. 62176222, No. 62176223, No. 62176226. No. 62072386. No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No.2022106001.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXivv:303.00777 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966. 2023.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 20t.:25002.3925.. 2025.
[5] Daniel Bolya, Cheng Cheng-Yangal Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmman. Token merging: Your vit but faster. arXiv preprint 21X0022210.09461, 2022.
10

---

[6] Tom B Brown. Language models are few-shot learners. arXiv preprint 1rXii:2005.44160 2020.
[7] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and -nd--al inference acceleration for large vision-lanuues models, arXiv preprint (iXXxx2203.06735
2024.
[8] Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17164-17174, 2023.
[9] Yanxi Chen, Xuchen Pan, Yaliang Li. Bolin Ding, and Jingren Zhou. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. arXiv preprint (xXi::3312.049// 2023.
[10] The Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.0527), 2024.
[11] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo. Zheng Ma, et al. How far are we gpt-4v? closing the gap commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024.
[12] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo. Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14255 2023.
[13] Peiyan Dong, Mengshu Sun, Alec Lu, Yanyue Xie, Kenneth Liu, Zhenglun Kong, Xin Meng. Zhengang Li, Xue Lin, Zhenman Fang, et al. Heatvit: Hardware-efficient adaptive token pruning for vision transformers. In 2023 IEEE International Symposium on
Computer Architecture (HPCA), pages
442 455. IEEE, 2023.
[14] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. cttcm/m-xcompose22: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401. 16420, 2024.
[15] Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang. Haodong Duan, Wenwei Zhang, Yining Li. et al. Internlm xcomposer2 language model handling resolutions from 336 pixels to 4k hd. Advances in Neural Information Processing Systems, 37:42566-42592 2025.
4khd: A pioneering large vision-
[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
[17] Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang. and Zhongyuan Wang. Not all layers of llms are necessary during inference. arXiv preprint arXiv:2403.0216 2024.
[18] Mohsen Fayyaz, Soroush Abbasi Koohpaycgani, Farnoush Rezaci Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Jürgen Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision pages 396-414. Springer, 2022.
[19] Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian. Deecap: Dynamic early exiting for efficient image captioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12216-12226, 2022.
[20] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. Computing Research Repository (CoRR), 2023.
[21] Amir Ghodrati, Babak Ehteshami Bejnordi, and Amirhossein Habibian. Framcexit: Conditional carly exiting for efficient vidco recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 156188 2021.
[22] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh, Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9904-6913, 2017. 201..
[23] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.

---

[24] Eric Jang. Shixiang Gu. and Ben Poole. Categorical reparameteriiztion with gumbel-softmax. arXiv preprini arXiv:161..0144. 2016.
[25] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04086 2024.
[26] Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, and Yiyi Zhou. What kind of visual tokens do we need? training- free visual token pruning for multi molli--modl large language models from the perspective of graph. arXiv preprint (tl::2501.022666 2025.
[27] Zhenglun Kong. Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pages 620-640. Springer, 2022.
[28] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22)1))79-86, 1951.
[29] Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
[30] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench Benchmarking multimodal ilms with generative comprehension. Computing Research Repository (CoRR), 2023.
[31] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imann -iaaaggee-gageaaae pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-199999 19742. PMLR, 2023.
[32] Wentong Li, Yugian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal Ilm. arXiv preprint arXiv:2407.02.88 2024.
[33] Xudong Li, Mengdan Zhang, Peixian Chen, Xiawu Zheng. Yan Zhang, Jingyuan Zheng, Yunhang Shen, Ke Li. Chaoyou Fu, Xing Sun, et al. Zooming from context to cue: Hierarchical preference optimization for multi-image milmss arXiv preprint (rxXii:2505.22999 2025.
[34] Yifan Li. Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ro-g Wen. Evaluating object hallucination in large vision-language models. Computing Research Repository (CoRR), 2023.
[35] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Pong Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning. et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 15947, 2024.
[36] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu,, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint 233.:3331. 2023.
[37] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. arXiv preprint 2005::2405.05803, 2024.
[38] Daria Lioubashevski, Tomer Schlank, Gabriel Stanovsky, and Ariel Goldstein. Looking beyond the top-1: Transformers determine top tokens in order. arXiv preprint arXiv:2410.20210, 2024.
[39] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[41] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? Computing Research Repository (CoRR), 2023.
[42] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Wei Chang, Song-Chum Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 2022.
[43] Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: of adaptation for multimodal large language models. arXiv preprint 233.03240..03003 2024.
12

---

[44] Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvga. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706, 2022.
[45] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEF/CVF winter conference on applications of computer vision, pages 2200-2209,
2021.
[46] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024.
[47] Yuzhang Shang. Mu Cai, Bingxin Xu. Yong Jae Lee, and Yan Yan. prameerru Adaptive token reduction for efficient large multimodal models. arXiv preprint 2tt:::4003.138883 2024.
[48] Min Shi, Fuxiao Liu. Shihao Wang, Shijia Liao, Subhashree Radhakrishnan. De-An Huang. Hongxu Yin. Karan Sapra. Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. arXiv preprint arXiv:2408. 1,598, 2024.
[49] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[50] Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan, A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. arXiv preprint 2.d5::2305.50554 2023.
[51] Shengkun Tang, Yaqing Wang. Zhenglun Kong, Tianchi Zhang, Yao Li, Caiwen Ding. Yanzhi Wang. Yi Liang, and Dongkuan Xu. You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10781-10791, 2023.
[52] Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[54] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307,09920 092888 2023.
[55] Hongjie Wang, Bhishma Dedhia, and Niraj K Jha. Zero-tprune: Zero-shot token pruning through leveraging of the attention graph pre- pre-trained transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16070-16079, 2024.
[56] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- languale model's perception the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[57] Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[58] Qiong Wu, Zhaoxi Ke, Yiyi Zhou, Gen Luo, Xiaoshuai Sun, and Rongrong Ji. Routing experts: Learning to route dynamic experts in multi-modal large language models. arXiv preprint 2rXiv:2407.1409. 2024,
[59] Qiong Wu, Wei Yu, Yiyi Zhou, Shubin Huang. Xiaoshuai Sun, and Rongrong Ji. Parameter and computation efficient transfer learning for vision-language pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
[60] Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-rl: Mitigating shortcuts in visual reasoning with reinforcement learning, arXiv preprint 2tl.::2555..4777 2025.
[61] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang. Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu. et al. Pyramiddrop: Accelerating your large vision- models via pyramid visual redundancy reduction. arXiv preprint 2rXiv:2410.17247, 2024.
[62] Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and training maining-free visual token pruning for multi-modal large language models. arXiv preprint arXiv:2409.1019 2024.
13

---

[63] Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. Computing Research Repository (CoRR), 2023.
[64] Shaolei Zhang, Qingkai Fang. Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. The Thirteenth International Conference on Learning Representations, 2025.
[65] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Wu, and Lei Huang. Tinyllava: framework small-scale large multimodal models. arXiv preprint 24Xiv:2402.142222 14289, 2024.
14

---

A Appendix
In this paper, we first conduct extensive empirical studies on the attention behaviors MLLMs, and summarize three main inference stages in MLLMs: (i) Early fusion between tokens is first accomplished quickly. (ii) Intru-modality modeling then comes to play. (iii Multimodal reasoning resumes and lasts until the end of inference. To better confirm findings, conduct further experiments on more MLLMs in the supplementary materials.
A.1 Attention Visualization
In Fig 2 we visualize the attention weight of each part in attention matrix, and the attention weights are calculated as follow: We first calculate the attention matrix with system prompt S E RR*d visual tokens V Ruxd and text tokens T Rtxd
V. TJW, of ((S (S v)
A Softmar
M),
Vd
where M represents the attention mask for causal reasoning. Then, the weight of each part be represent as
strit sit
stotister
W. = ΣΣ Aij/., W.
ΣAyh s=ste==st
Σ An/l
(8)
(=
where W., We and W, represent the attention weights of visual self attention, cross attention and text self attention, respectively. In addition, we combine the attention matrices of different heads in MHA through the max pooling.
A.2 Ablation Study
In Tab and Tab perform set of experiments analyze the impact of different representation selections on VTE As shown in Tab when relying on more than two representations, we can find out that their performance similar to "mean text last text". For instance, the additional representation, i.e., "last visual" and "mean visual", has no noticeable effect on the results. We can also observe that the missing of "mean text" "last text" will lead to inaccurate exit. For example. the missing of "mean text" leads to later exit, i.e., layers latter, while the absence of "last text" does the opposite, Lee., 2 layers earlier. As for taking attention scores be the representations, we find that hyper-network can not make the correct judgment. Specifically, the exit layers for all benchmarks can be much earlier. Overall, these results well confirm the use of text tokens status for dynamic visual token exiting in MLLMs.
Table 7: Ablation study of the token status selection for VTE. "Mean Visual" denotes the average of all visual tokens except the last one, similar with "Mean Text". "Last Visual" and "Last Text" denotes the last visual text token. "Exit Layer" denotes the averaged layer numbers selected to remove by VTE. "Acc." denotes the accuracy. The default setting of VY is the last row. The best and second best results are marked in bold and underline, receptively.
GQA
TexVQA
MMG-Ve
SQA-1
Average
Visual
Mean
Visual
Last
Mean
Text
Last
Text
Azc.
Exit Layer
Acc.
Exit Layer
Acc.
Exit Layer
Acc. ?
Pxit Layer
Azc.
Fxit Layer
61.4
21.6
$7.3
21.0
31.5
69.2
21.7
55.0
21.4
61.2
21.3
21.1
30.0
21.0
69.7
21.4
54.5
21.2
60.1
17.4
57.6
22. 1
31.1
22.1
69.7
201.4
54.6
20.5
59.8
16.3
$6.3
19.1
29,9
21.0
69.8
13.8
54.0
17.6
21.2
$7.3
21.3
31.6
21.0
54.9
21.2
58.7
16.5
57.6
22.2
32.7
22.5
69.7
21.3
56.7
20.6
50.4
16.3
$6.5
19.9
30.9
21.8
69.7
14.4
54.1
18.1
4
59.4
16.7
$7.4
21.8
31.3
22.0
69.6
20.3
$4.4
20.2
59.3
16.4
56.5
19.9
31,2
54.1
18.1
60.0
31.9
69.6
60.1
16.4
$7.0
20.2
30.8
21.3
69.5
14.1
54.4
18.0
4
60.2
16.7
$7.1
20.3
31.0
21.6
69.7
14.3
54,5
18.2
31.7
4.
15.2
56.2
30.0
11.7
60.5
168
57.1
20.5
31.1
21.8
60.8
14.1
54.6
18.3
LLaVA-1.5 1.5 7B
62.0
58.2
30.5
69.4
55.0
15

---

Table 8: Ablation study of the attention status selection for DyVTE. "Visual Self" denotes the attention score from visual modality of each visual tokens, similar with "Text Self". "Cross" denotes the attention score from visual the text of each visual tokens. For attention representations whose dimensions are not 576, we use interpolation to align the dimensions. "Exil Layer" denotes the averaged layer numbers selected remove by DyVTE. "Acc." " denotes the accuracy. The default setting of VTE is the last row. The best and second best results are marked in bold and underline.
State
GQA
ToaVOA
MM-Vet
SQA-I
Average
Visual
Self
Cross
Self
Acc.
Layer
Acc.
Layer
Acc.
Layer
Acc.
Layer
Acc. 1
Layer
38.3
38.8
1.3
43.1
1.3
11.5
1.2
65.2
1.2
39.6
1.2
39.6
1.7
42.7
1.1
11.9
3.5
64.7
0.6
39.7
1.7
39.4
2.7
43.1
3.0
13.1
2.6
65.4
3.8
40.3
3.0
40.7
3.0
43.0
1.7
13.3
5.9
65.1
1.0
40.5
2.9
40.7
3.0
43.1
1.6
13,4
6.0
65.2
40.6
49.5
10.9
45.6
5.6
15.6
7.1
65.6
2,5
44.1
6,5
LLAVA-1.5 7B
62.0
58.2
30.5
69.4
55.0
A.3
Distribution of exit layers
In Fig. 7 we present the distributions of the visual token exit layers for LLa 7B and 13B. From the figure, we observe that VTE dynamically selects the exit layer based on the specific task requirements. For example, on the TextVQA dataset, which demands fine-grained information. the exit occurs later compared to other datasets. In contrast, on the GQA dataset, which involves open-ended word questions and simple truc/false judgments, the exit layers are more evenly distributed across early and late layers. Another notable observation is that the timing of the exit layer shows similar distributions across MILMs of different scales and architectures. For instance, both LLa LLaVA-DyVTE 7B and 13B remove all visual tokens at the same layer early in the process on the SQA dataset. Similarly, for Intern VL and LLaVA peaks in the exit layer distributions occur at comparable layers. Overall, these results support our motivation and the proposed VTE.
HL
HLaVA-DVVTE
26
010
24.33
DLWVE 198
DVVTE
DL 15B
17.4 1L-VA-DWWT 1W
Phopgtaes
Proponterr
Proputtes
Properter
30-
2.00
0.00
13 30 31 12 10.0
8:00
28 24
36.0
16 20
10.00
0.00
5t 28 31
36
DoYTV 18
DWVTE 7B
LneaVL BY
D.VTE
4.48-
Progertion
Proprtion
Urrgertion
820
24
Tatt Lague
Rich Layer (s) Test/Q
Edtt Lager
Toch Layer
90A-1
(c) GQA
(8) MLN-Vet
Figure 7: Statistics of the exiting layers decided by DYE on LLaVA-1.5 7B, LLaVA-1.5 13B and InterVVL. 7B. The horizontal axis represents the exit layer, and vertical axis represents the proportion. The exit time by is different for different tasks, but similar for MLLMs with the same sizes.
A.4 Distribution Attention
In Fig we further visualize the attention distributions on different MLLMs for different datasets. We can first observe that datasets produce similar attention patterns on the same MLLM. Another observation is that attention patterns across different MILMs have similar trends and the distributions can generally be categorized into three distinct stages. These findings emphasize the universality of attention behavior in MILMs, further validating our approach. Overall, the experiment results confirm that different MLLMs have similar attention patterns the different data.
16

---

GQA
TextVQA
SQA-1
MM-Vat
Attention Weight
2
24
32
16
24
32
16
24
32
16
32
Layer
Layer Index
Layer Index
Layer
Cross Attention
-
Visual Self' Alteation
Text Self Attention
Figure 8: Distributions of averaged attention scores image self -off-attenioon, cross-attention, and text self-attention of LLaVA- 7B. We visualize the mean and variance of the attention weight of each part GQA, TextVQA, SQA-I, MM- Vet datasets. From these distributions, we can summarize three main stages of MLLMs as introduced in the main paper, which are then marked by red lines.
GQA
TextVQA
SQA-I
MM-Ve
Attention Weight
2
8
24 Layer Index
16
32
40
24 Layer Index
16
32
40
8
16 Layer Index
24
32
40
8
16 Layer Index
24
32
40
Cross Attention
Visual Self Attention
Text Self Attention
Figure 9: Distributions of averaged attention scores of image self-attention, cross-attention and text self-attention LLaVA-1.5 13B. We visualize mean and variance of the attention weight each part GQA, TextVQA, SQA-I, MM- datasets. From these distributions, main stages of MLLMs as introduced in the main paper, which are then marked by red lines.
summarize three
GQA
TextVQA
SQAd
MM- Vet
Attention Weight
w
32
16 Layer Index
32
Layer Indox
Layer Index
Layer Index
Cross A Attention
-
Visual Self Attention
Text Self Attention
Figure 10: Distributions of averaged attention scores of image self- aifffattention, cross-attention and text self-attention of Vn 7B. We visualize the mean and variance of the attention weight of each part on GQA, TexIVQA, SQA-I, MM- Ret datasets. From these distributions, we can summarize three main stages of MILMS introduced in the main paper, which are then marked by red lines.
GQA
TextVQA
SQA4
MM- Ves
Ancation Weight
32
16
24
Layer Index
Lay Index
Layer Index
Layer Index
Cross Attention
Visual Atteation
Text Self' Atteation
Figure 11: Distributions of averaged attention scores of image self-attention, cross-attention and text self-attention of VILA 7B. We visualize the mean and variance of the attention weight of each part on GQA, TextVQA, SQA-I, MM- Vet datasets. From these distributions, we can summarize three main stages of MLLMs as introduced in the main paper, which are then marked by red lines.
17

---

GQA
TexVVQA
SQA-
MM- Vet
Astemtion Weight
24
32
32
16
24
32
Layer Index
Layer Index
Layer Index
Layer Index
Cross Attention
Visual Sell Attemsion
Tex: Self Attention
Figure 12: Distributions of averaged attention scores of image attffattention, cross-attention and text self-attention EAGLE-X5 7B. We visualize the mean and variance of the attention weight of each part on GQA, Tex SQA-I, MM- Vet datasets. From these distributions, we can summarize three main stages of MLLMs as introduced in the main paper, which are then marked by red lines.
A.5 Entropy of Attention
In L3 we visualize the entropy distributions of cross-attention matrices and text self-attention matrices. For all these MLLMs and datasets, can observe that removing all visual tokens an appropriate time will not have a significant impact on the answer modeling process. Specifically, for all MILMs and datasets, the removing of visual tokens only makes the entropy of cross-attention disappear, while the entropy of text self-attention maintains the same distribution. Overall, these results well confirm that cross-modal interaction in fact barely contributes to multimodal reasoning after some layers.
Text-Image Cross Artention
Text Self Attention
Text-Image Cross Attention
Text Self Artension
wis D:VTE
wi DyVTE
DYAT
we DyVTE VTE
Entropy
DyYTE
()) SQA-1
(b) TexVVA
Text- Text-Imapee Cross Atlention
Text Self Allention
Text-Inage Cross Atlention
Text Self Aliention
D-VTE DyYTE
Dy
DyVTE TEE
Entrppy
10
10
VTE
Nilonng
DyVTE
Entoopy
10
4 8 12
32
12 16 20
24
28 32
4
12
16
20
24
28
16 20
GQA
(d) MM-Vet
Figure 13: The entropy of two attention distributions. We visualize the entropy of cross-attention matrices and text celf-attention on LL. 7B with and without our Dy VTE, which shows that the removal of visual tokens barely affect the text self-attention.
Text- Text-Image Cross Atlention
Text Self Atlention
Text-Inage Cross Alleetion
Text Self Allention
DyVTI
DyVTE D.YTE
DyVTI
Ennopy
DyYTE
DyVTE
Aloana
2216202422833
46
12 20 30
2216100423333664
T4 28 123660
(a) SQA-I
(b) TanvQA
Text Image Cross Attention
Text Self Attention
Text-Image Cross Attention
Text Sclf Attention
wis DyVTI:
wo DyVTE DyVTE
wio DyATI
DyVTE
Eutropy
10
w: DyyTE
Adouse
v DyVTE
1
DyVTE
0
4 8 1620
1640
12 R 16 24 3640
14 121620 32
14
(6) GQA
(d) MM-Ver
Figure 14: The entropy of two attention distributions. We visualize the entropy of cross-attention matrices and text self attentinnn on LLLVA-115 13B with and without our DyT,, which shows that the removal of visual tokens barely affect the text self-attention.
18

---

Text-Image Cross A Attentoon
Text Selt Attention
Text-Image Croas Atrention
Text Self Attention
DyVTB DyYTE
Ty DyYE
wik TyTT DAVTE
Ennopy
10.
DyVTE
4
5
12
15
20
24
28
32
4
8
12
16
20
24
28 32
4 12 16 20
12 16 20
24
28
(0) SQA-1
(b) TexVQQA
Text Eoatt-mage Cross Attention
Text Attenttniion
Text- Image Cross Attention
Text Self Attention
119/ VT
VVV DyYTS
wes DYVTE DyVTE
DyVT Dy
Ensopy
10.
10 bdnpay
4 K 12
24
28
52
12 16 20
16
20
26
(c) GQA
(d)
MM-Vet
Figure 15: The entropy of two attention distributions. We visualize the entropy of cross-attention
matrices and text self-attention on InternVI. 7B with and without our DyVTE, which shows that the removal of visual tokens barely affect the text self-attention.
Text-Image Cross Attention
Text Self Attention
Text-Image Cross Attention
Text Selt Attention
Dyfitt
DyVTI
wobs DyVTE
DyVT
Ennopy
10
Adoonn
DyWTE
Edhamgy
01
4
8
12
16
20
24
28
12
4
8
12
16
20
52
12
16
20
24
28
28 (b) TeavQA
12
16
20
21
(a) SQA-I
Text-Image Cross A Attention
Text Attentttoioo
Text-Image Cross Atention
Text Self Astention
DyVIE DyYTE
we DyVIE
wib DyVTE
DyVTE DyVTE
Enzogy
10
DyVT
kdoong
12 28 32
12 16
24
28
12
16
20
24
28
12 16 20 24 28
(c) GQA
(c) MM-Ver
Figure 16: The entropy of two attention distributions. We visualize the entropy of cross-attention matrices and text self-attention on VILA 7B with and without our DYV,E, which shows that the removal of visual tokens barely affect the text self-attention.
Text-Image Cross Atlention
Text Self Attention
Text-Image Cross Altention
Text Sell Atlention
20
DyVTI
20
wi VYTI
20 1
wes DyVTE
20
DYVTS
Enropy
Eanropy
Ensopy
01
12
15
24
28
32
12 16
12 16 20 24
28
12
12
16 20 24 28
(4) SQA-I
(b)
TexVQQ
Text-Image Cross Attention
Text Atlentttntion
Text- Text-Image Cross Attention
Text S Attention
20
DyVIl
20
VV
veb DYVTE DyVTE
20
DyVIE
Entopy
Eatropy
0. 12
28
12 16
12 16
20
24
12
28
(c) GQA
(d)
MM-Vet
Figure 17: The entropy of two attention distributions. We visualize the entropy of artss-attention matrices and text self-attention on EAGLE-X5 7B with and without our VYE, which shows that the removal of visual tokens barely affect the text self-attention.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
19

---

You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often
nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction explicitly outline the main contributions, includ- ing an empirical findings in the inference MI.LM and novel and effective Dy method exit the visual tokens in early stage.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
The limitation have been provided in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
20

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include any new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
We provide all the details in our paper, and our code is anonymously released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed
21

---

instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is anonymously released and all datasets used is public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
public//uiiess/ddeeluliiiiss wttpuuuiiiiiiioooooooooo.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//t////ooeeeeeeeeeeeeeeeeee soooccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the details in our paper, and our code is anonymously released.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not include experiments involving random processes that would require the reporting of error bars or statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: have provide these information our experiment details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have strictly adhered to the guidelines.
23

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper does not explicitly discuss the societal impacts of the work per- formed, as primarily focuses on the technical development of multimodal large language models.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work is based on public data and models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
24

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited their works.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our code is produced in our anonymous project.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
25

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: LLM, as a part of MLLM, is the object of our study.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should be described.
26


Reviews:
[{"review_id": "mWH3h6bPb8", "reviewer": "Reviewer_3PHf", "summary": "The paper proposes DyVTE, an early-exit method for efficient MLLM inference. It first looks into the MLLM inference behavior by analyzing the attention scores, and categorizes the inference process into three stages. By examining the importance of visual tokens in the three stages, a learned hyper-network is used to remove the visual tokens adaptively. Extensive experiments over multiple benchmarks demonstrate the effectiveness of DyVTE.", "questions": "1. Please see the Weaknesses.\n2. In Table 5, combining DyVTE with FastV decreases performance, while in Table 3, improves accuracy. As there is only a change in LLaVa model size, what might be the reasons behind this?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe in-depth analysis of token importance is interesting and novel.\nExtensive experiments are done, justifying the proposed method’s effectiveness.\nThe paper is well written, and the proposed findings are well explained and justified.\nWeaknesses:\nThere seems to be a weak connection between MLLM inference behavior and the proposed method, where though DyVTE is motivated by the findings on the changing importance of visual tokens, the categorization of MLLM inference did not lead to any technical contribution. It seems that it is only an empirical and rather intuitive finding, which weakens the overall contribution of the paper.\nThere is a lack of ablation studies on the computation cost of training the hyper-network. While many token pruning/merging techniques adopt a training-free scheme, it is unclear whether DyVTE can be more efficiently deployed.\nCan a universal DyVTE be used, i.e. a same DyVTE be shared by different MLLMs?\nDyVTE adopts an all-or-nothing visual token removal strategy which either retains all or discards all visual tokens at a chosen layer. While effective for efficiency, this coarse-grained approach lacks flexibility and may overlook intermediate solutions where only some visual tokens (e.g., those with low attention weights) can be removed. Fine-grained or per-region exit strategies may offer better trade-offs.\nMinor Weaknesses:\nL245 “it work very well with our DyVTE” should be “it works very well with our DyVTE”\nReferences [5] and [6] are duplicate citations of the same work “Token Merging”\nL225 “DyVTE can reduce 45.7% FLOPs of LLaVA” should be “DyVTE reduces the FLOPs of LLaVA by 45.7%”", "was_revised": false, "final_justification": "", "rebuttal": "## Comment to Reviewer 3PHf\n\nWe highly appreciate your time and effort in reviewing this paper, as well as your encouraging and constructive comments on our work. Below, we response to your key concerns point by point.\n\n**Comment 1:** There seems to be a weak connection between MLLM inference behavior and the proposed method, where though DyVTE is motivated by the findings on the changing importance of visual tokens, the categorization of MLLM inference did not lead to any technical contribution. It seems that it is only an empirical and rather intuitive finding, which weakens the overall contribution of the paper. \n\n**Response:** Thanks for your comment. We think that the observed and summarized inference behaviors of MLLMS give sufficient supports for the design of DyVTE in terms of both theoretical and methodological aspects.\n\nFirst, our findings explain when and why visual tokens become less important in MLLMs. \nAs discussed in the paper, several existing works [8, 47, 61] have revealed the visual redundancy problem in MLLMs, but they tend to rely on attention metrics to measure the importance of individual tokens, and lack of the further exploration about the changing role of visual tokens in the middle or shallow layers of MLLMs. In this case, our findings well explain the patterns of visual tokens throughout the entire inference of MLLMs, and provide a direct basis for the early-token exit principle of  DyVTE.\n\nSecondly, our findings also indicate the necessity of dynamic modeling and its advantages to the hand-crafted solutions. Although most MLLMs share similar three-stage behaviors, their explicit categorizations are slightly different, especially the last multimodal reasoning stages, the optimal time for dropping all visual tokens at the multimodal reasoning stage varies across models, tasks and even examples. This inspires us to explore the dynamic modeling for DyVTE rather than hard-metric based ones in terms of methodology.\n\nMoreover, these connections are also reflected in the quantitative and qualitative evaluations of DyVTE. For instance, DyVTE only drops the visual tokens at the last inference stage of MLLMs and its overall performance is better than previous metric-based pruning methods, as shown in Fig.7 (In appendix) and Tab.1,2, respectively.\n\nOverall, we think that our findings about MLLMs' behaviors do contribute a lot to the design of DyVTE, which are also valuable insights to the community.\n\n**Comment 2:** There is a lack of ablation studies on the computation cost of training the hyper-network. While many token pruning/merging techniques adopt a training-free scheme, it is unclear whether DyVTE can be more efficiently deployed. \n\n**Response:** Thanks for your comment. Following your suggestion, we report the detailed training cost of DyVTE in the following table. It can be seen that compared to the common SFT tuning, its training cost is much cheaper, which only takes about half an hour on less than 2% data. As discussed in Sec.4.2, the training of DyVTE is independent to MLLMs and requires no the complete gradient updates, thus its expenditure is extremely cheap. \n\n|Method|Device|Data|Time|\n|-|-|-|-|\n|SFT|8xA800|665k|12hours|\n|DyVTE|4xA800|~8k|0.5hours|\n\nWe also acknowledge the training-free advantages of existing token pruning/merging methods [8, 47]. As far as we know, although these methods do not require training-based optimizations, they still need hyper-parameter tuning, *e.g.*, threshold of attention score and ratio of pruned tokens, which is still time-consuming as commented in [61]. For instance, conducting the an evaluation on LLaVA for VQAv2 benchmark also requires about 120 mins with 4xA800.\n\nIn this case, we think that our DyVTE is still a cost-effective method.\n\n**Comment 3:** Can a universal DyVTE be used, i.e. a same DyVTE be shared by different MLLMs? \n\n**Response:** Thanks for your constructive comment. Following your suggestion, we apply a DyVTE trained on LLaVA (LLaVA + DyVTE) to the other two MLLMs, *i.e.*,  InternVL and VILA, and also compare it with the specifically tuned DyVTEs on these two MLLMs.\n\n|Method|TextVQA|Exit Layer|GQA|Exit Layer|SQA|Exit Layer|\n|-|-|-|-|-|-|-|\n|InternVL+DyVTE|55.80|19.50|62.9|15.18|66.20|13.46|\n|InternVL+DyVTE-LLaVA|55.22|18.91|61.32|15.74|66.39|17.86|\n|VILA+DyVTE|62.60|21.70|63.10|16.17|69.5|15.38|\n|VILA+DyVTE-LLaVA|60.90|20.08|61.18|15.86|69.86|16.68|\n\nThe results are a bit astonishing. \nWe can see that the DyVTE trained on LLaVA can be directly applied to other MLLMs, although encountering slight performance drops. And even the optimal existing layers are close to the ones of the specifically tuned DyVTEs. These results may imply that DyVTE master generalized patterns across MLLMs. This finding also yields a direction for our future research of DyVTE. Thanks again for your comment.\n\n**Comment 4:** DyVTE adopts an all-or-nothing visual token removal strategy which either retains all or discards all visual tokens at a chosen layer. While effective for efficiency, this coarse-grained approach lacks flexibility and may overlook intermediate solutions where only some visual tokens (e.g., those with low attention weights) can be removed. Fine-grained or per-region exit strategies may offer better trade-offs. \n\n**Response:** Thanks for your comment. We think that the most valuable part of our work is its orthogonal contributions to existing research. As discussed above, our study focuses more on the overall status of all visual tokens in MLLMs, based on which DyVTE is proposed. This property makes DyVTE can well supplement the existing token-wise studies [8, 47, 61]. As shown in Tab.5, the visual token removal of DyVTE can be easily combined with the token-wise pruning methods like FastV to achieve better efficiency. \n\n**Comment 5:** L245 \\`\\`it work very well with our DyVTE'' should be \\`\\`it works very well with our DyVTE'' References [5] and [6] are duplicate citations of the same work \\`\\`Token Merging'' L225 \\`\\`DyVTE can reduce 45.7% FLOPs of LLaVA'' should be \\`\\`DyVTE reduces the FLOPs of LLaVA by 45.7%''\n\n**Response:**  Many thanks for your careful review. We will revise these typos and keep on polishing our paper until the final submission."}, {"review_id": "9CHqzSuP7q", "reviewer": "Reviewer_9P8t", "summary": "The paper shows that multimodal LLMs only need image tokens early on, once text tokens have fused image information, visual tokens can be removed. Exploiting this, the authors propose Dynamic Visual-Token Exit, which is a tiny hyper-network to decide when to discard all image tokens while letting text tokens keep processing.  Experimental results on five popular MLLMs show that DyVTE can reduce the inference cost without significant performance drop.", "questions": "1. At which specific layer is the early exit applied? Although line 164 mentions that DyVTE is applied at the k-th layer of an MLLM, the Implementation Details section lacks clarification about the exact layers or criteria used for selecting these layers in practice.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly written and easy to follow, effectively guiding readers through the key insights and contributions.\n2. The experiments conducted are extensive, covering multiple models and benchmarks, which provides robust empirical validation of the proposed method.\n\nWeaknesses:\n1. The issue of visual token redundancy has already been extensively studied in prior works, such as LLaVA-Mini and FastV. Therefore, the empirical findings presented in this paper regarding redundancy contribute limited novelty.\n2. The paper lacks novelty in terms of methodology. Specifically, important related early-exit approaches, like LLaVA-Mini—which discards all visual tokens after fusing visual information into text tokens within just four LLM layers—are not discussed or compared against. Including comparisons or at least a detailed discussion of such highly relevant work would significantly strengthen the paper.\n3. In the efficiency comparisons, the authors only evaluate theoretical FLops differences. However, in practical scenarios, MLLM inference typically involves optimization techniques like flash attention and KV cache. The reliability and impact of the proposed approach would be better assessed through actual inference latency rather than theoretical computational cost alone.", "was_revised": true, "final_justification": "The author's rebuttal has addressed my concerns, and I will raise the rating.", "rebuttal": "## Comment to Reviewer 9P8t\n\nWe highly appreciate your time and effort in reviewing this paper, and also thanks a lot for your constructive comments on our work. Below, we response to your key concerns point by point.\n\n**Comment 1:** The issue of visual token redundancy has already been extensively studied in prior works, such as LLaVA-Mini and FastV. Therefore, the empirical findings presented in this paper regarding redundancy contribute limited novelty.\n\n**Response:** Thanks for your comment. We think that our findings are orthogonal and supplementary to prior works about the study of visual redundancy in MLLMs.\n\nFristly, the research focuses are different. As discussed in Lines 104–108, most prior works[8, 47, 61], focus on token-level redundancy, while ours is keen to investigating the overall status of visual tokens in MLLMs. \nIn particular, prior works like FastV and Llava-prumerge aim to improve efficiency via token pruning, merging or compression based on the token-wise estimation, *e.g.*, the attention-based metrics. In contrast, our DyVTE is to perceive the status of all visual tokens during the inference of MLLMs. This great difference also yields a distinct and novel methodology of our DyVTE, *i.e.*, using lightweight hyper-neworks to dynamically predict the visual token exit.\n\nSecondly, our findings are good supplementary to prior studies. As discussed above, quite a lot of prior works [8, 27, 33, 47, 61] reveal the visual redundancy problem of existing MLLMs, and also LLaVA-mini and FastV show that visual tokens are more important in the shallow layers of MLLMs. However, to our best knowledge, there still lacks of a systematic study to explain when and why visual tokens become less important during the inference of MLLMs. In this case, our work not only summarize three main behaviors of MLLMs but also in-depthly depict the changing roles of visual tokens in MLLMs via extensive empirical studies. Thus, our findings are good supplementary to prior works about the visual redundancy problem of MLLMs. \n\nOverall, we think that our findings are novel enough and give timely contribution to the community, as highly recognized by other reviewers.\n\n**Comment 2:** The paper lacks novelty in terms of methodology. Specifically, important related early-exit approaches, like LLaVA-Mini—which discards all visual tokens after fusing visual information into text tokens within just four LLM layers—are not discussed or compared against. Including comparisons or at least a detailed discussion of such highly relevant work would significantly strengthen the paper.\n\n**Response:** Thanks for your comment. We highly recognize the contribution of LLaVA-Mini as an excellent prior work in this field, and will actively cite and compare with it in our new version.\n\nHowever, we think that our DyVTE is greatly different from LLaVA-Mini in terms of both principle and methodology. In principle, DyVTE aims to adaptively perceive the status of visual tokens in MLLMs, thereby determining the optimal time of all visual-token exist. In contrast, LLaVA-mini focus on compressing all visual information before the MLLMs. Although the two methods share a similar intuition, *i.e.*, visual tokens are more important at the shallow layers, the detailed observations and adopted principles are quite different, as discussed above.\n\nIn terms of methodology, the difference between two methods become more obvious. DyVTE applies lightweight hyper-networks to make the prediction of exist layers based on the token status in MLLMs, while LLaVA-mini introduce a query module to compress all visual information into text tokens before MLLMs. Moreover, the training of DyVTE is independent to MLLMs, which only takes a little time to optimize the lightweight hyper-networks, *e.g.*, about 30 mins on 8k VL examples. Instead, LLaVA-mini requires the traditional pretraining and SFT tuning stages like most MLLMs. \n\nBased on the above discussions, we think that the contributions of DyVTE are orthogonal to LLaVA-mini, and our novelty is indeed significant.\n\n**Comment 3:** In the efficiency comparisons, the authors only evaluate theoretical FLops differences. However, in practical scenarios, MLLM inference typically involves optimization techniques like flash attention and KV cache. The reliability and impact of the proposed approach would be better assessed through actual inference latency rather than theoretical computational cost alone.\n\n**Response:** Thanks for your comment. Your mentioned latency comparison has been given in Tab.3 of the paper, where our DyVTE shows lower inference latency than the pruning methods. Moreover, without the requirement of attention score, our DyVTE is fully compatible of acceleration techniques like FlashAttention.\n\n**Comment 4:** At which specific layer is the early exit applied? Although line 164 mentions that DyVTE is applied at the k-th layer of an MLLM, the Implementation Details section lacks clarification about the exact layers or criteria used for selecting these layers in practice.\n\n**Response:** Thanks for your comment. As introduced above, DyVTE is a dynamic approach, which will adaptively determine the exit time of visual tokens' status.Thus, $k$ is not a fixed number in DyVTE."}, {"review_id": "MJT1ldK6sM", "reviewer": "Reviewer_SwWi", "summary": "The paper investigates the problem of visual redundancy in Multimodal Large Language Models (MLLMs) by analyzing their attention behaviors during inference. Through extensive empirical studies, the authors identify three primary inference stages in MLLMs: (i) early fusion, where visual and text tokens exchange information rapidly in shallow layers; (ii) intra-modality modeling, where tokens primarily interact within their respective modalities; and (iii) multimodal reasoning, where visual tokens resume propagating information to text tokens until a certain point, after which their contribution diminishes. A key finding is that visual tokens often become redundant in the multimodal reasoning stage once text tokens have received sufficient visual information. Based on this, the authors propose a novel method called Dynamic Visual-Token Exit (DyVTE), which uses lightweight hyper-networks to dynamically determine when to remove visual tokens during inference, thereby improving computational efficiency.", "questions": "- The paper mentions that DyVTE uses lightweight hyper-networks, but it does not quantify their computational or memory overhead. How significant is this overhead compared to the savings achieved by removing visual tokens?\n- The experiments focus on standard vision-language benchmarks (e.g., MMB, SQA, QQA). Have the authors tested DyVTE on more complex or real-world tasks, such as open-ended visual question answering or tasks requiring fine-grained visual understanding?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper presents a robust empirical analysis of MLLM attention behaviors, supported by quantitative experiments across multiple models.\n- The work addresses a critical challenge in MLLMs: the computational inefficiency caused by excessive visual tokens.\n\nWeaknesses:\n- The analysis and methods lack novelty. Analysis techniques based on attention score mechanisms have been applied in many previous works (e.g., FastV and most related studies). The conclusions derived from attention entropy and \"The impact of dropping all visual tokens\" have also been presented in some prior works (e.g., llava-mini). It seems that the authors should explore some specific analysis methods tailored to early-exit, which would make the work more compelling.\n- The connection between the analysis results in Section 3 and the proposed method is not strong. Clarifying why the method design can be inspired by these analysis results is crucial for establishing the method’s validity.", "was_revised": true, "final_justification": "Thank you for your response. After carefully considering both the content of the paper and the rebuttal, I have decided to keep my score unchanged.", "rebuttal": "## Comment to Reviewer SwWi\n\nWe highly appreciate your time and effort in reviewing this paper, as well as your encouraging and constructive comments on our work. Below, we response to your key concerns point by point.\n\n**Comment 1:** The analysis and methods lack novelty. Analysis techniques based on attention score mechanisms have been applied in many previous works (e.g., FastV and most related studies). The conclusions derived from attention entropy and \"The impact of dropping all visual tokens\" have also been presented in some prior works (e.g., llava-mini). It seems that the authors should explore some specific analysis methods tailored to early-exit, which would make the work more compelling.\n\n**Response:** Thanks for your comment. We think that our analysis and conclusions are orthogonal and complementary to prior attention-based studies.\n\nFirstly, we present a different aspect of studing visual redundancy problem in MLLMs, i.e., the overall behaviors of MLLMs and visual tokens. As discussed in Lines 104–108 of our paper, most existing works use attention scores primarily for token-level redundancy estimation, thereby providing basis for the proposed token compressing solutions [8, 47, 61]. In contrast, our work realize the first attempt of systematically studying the overall status of visual tokens in MLLMs. Specifically, we do not simply reuse attention score as a redundancy metric. Instead, we analyze how attention distributions changes across layers and how these dynamics inform the answer modeling.\n\nSecondly, our findings are good supplementary to prior studies. As discussed in the paper, many prior works including LLaVA-mini and FastV reveal that the importance of visual tokens will become much less significant after the shallow layers of MLLMs. However, there are still not sufficient findings to explain when and why they will become not important during the inference. In contrast, our analyses and findings well mitigate this defect, portraying the changing role of visual tokens in MLLMs via much more extensive observations. \n\nIn terms of LLaVA-mini, we acknowledge it as an excellent prior work. However, our principle and methodology is greatly different from it. For instance, DyVTE is a dynamic and plug-in design, of which optimization is independent to MLLMs. Besides, LLaVA-mini is more like an efficient MLLM. And it contributes more to the compression of all visual tokens to one text input before MLLM, rather than directly dropping all visual information during inference.\n\nOverall, we think that our findings are orthogonal and supplementary to prior works. And the proposed DyVTE is innovative and different enough in terms of both principle and methodology. \n\n**Comment 2:** The connection between the analysis results in Section 3 and the proposed method is not strong. Clarifying why the method design can be inspired by these analysis results is crucial for establishing the method's validity.\n\n**Response:** Thanks for your comment. In fact, our observed and summarized findings give sufficient supports for the proposed DyVTE in terms of both principle and methodology.\n\nAs discussed above, our findings explain when and why visual tokens become less important in MLLMs. \nThrough the systematic analysis, we observe that the contribution of visual tokens dynamically decreases during inference. Moreover, the optimal exit layer varies across  tasks and even examples, as shown in Fig.3. In this case, our findings well explain the patterns of visual tokens throughout the entire inference of MLLMs, and  provides a direct basis for the early-token exit principle of DyVTE.  In addition, these findings also guideline us to explore the dynamic modeling of DyVTE instead of the hard-metric based ones. \n\nMoreover, these connections are also reflected in the quantitative and qualitative evaluations of DyVTE. Specifically, DyVTE only drops the visual tokens at the last inference stage of MLLMs as shown in Fig.7 (In appendix). And the overall performance of DyVTE is better than previous metric-based pruning methods as shwon in Tab.1,2.\n\nOverall, we think that our findings about MLLMs' behaviors greatly inspire and support the design of our DyVTE, which are also valuable insights for the community.\n\n**Comment 3:** The paper mentions that DyVTE uses lightweight hyper-networks, but it does not quantify their computational or memory overhead. How significant is this overhead compared to the savings achieved by removing visual tokens?\n\n**Response:** Thanks for your suggestion. We give the computational and memory overhead of our DyVTE below:\n\n||Parameters|FLOPs|Memory|\n|-|-|-|-|\n|DyVTE|8.39M|8.39M|72.19MB|\n|LLaVA|7B|9.6T|15.7G|\n|LLaVA + DyVTE|7B|5.4T|15.8G|\n\nWe can see that the additional parameters and computation introduced by DyVTE are minimal mainly due to its lightweight designs. Moreover, DyVTE can significantly reduce the overall computation cost via dynamically removal all visual tokens at the optimal time, and its designs are fully compatible with acceleration techniques like Flash Attention and KVCcahe.\nCompare to the original model, DyVTE can achieve significant improvement in inference efficiency. The negligible overhead from the hyper-networks is far outweighed by the computational savings from token reduction.\n\nIn summary, DyVTE's lightweight hyper-networks incur only a minimal overhead, while enabling much greater savings in computation and inference time by removing redundant visual tokens.\n\n**Comment 4:** The experiments focus on standard vision-language benchmarks (e.g., MMB, SQA, GQA). Have the authors tested DyVTE on more complex or real-world tasks, such as open-ended visual question answering or tasks requiring fine-grained visual understanding?\n\n**Response:** Thanks for your constructive comment. Following your suggestion, we further conduct experiments on two additional VL tasks, namely visual grounding (RefCOCO) and image captioning (COCO), which have stronger requirements about fine-grained visual understanding. The results are reported in the following table. It can be seen that on these two granular VL tasks, the advantages of our DyVTE are much more obvious, which can adaptively determine the removal of visual tokens according to task difficulty. In a stark contrast, token pruning methods like FastV meets obvious performance drops, which might be due to its hard and fixed pruning strategy.\n\n|Method|RefCOCO REC Val Acc@0.5|exit layer|COCO2014 Cap Val CIDEr|exit layer|\n|-|-|-|-|-|\n|LLaVA-7B|50.03|-|108.74|-|\n|LLaVA-7B+FastV|37.94|-|108.03|-|\n|LLaVA-7B+DyVTE|51.86|18.37|107.62|22.37|\n|InternVL-7B|55.34|-|113.70|-|\n|InternVL-7B+FastV|55.34|-|113.95|\n|InternVL-7B+DyVTE|55.29|17.16|112.42|23.32|\n|LLaVA-13B|66.56|-|113.79|-|\n|LLaVA-13B+FastV|61.47|-|113.95|-|\n|LLaVA-13B+DyVTE|67.98|20.82|111.27|25.04|"}, {"review_id": "2GDosJHPd0", "reviewer": "Reviewer_9aDZ", "summary": "The paper \"Accelerating Multimodal Large Language Models via Token Reduction\" presents a method to enhance the efficiency of Multimodal Large Language Models (MLLMs) by reducing the number of visual tokens processed during inference. The core contribution is a token reduction framework that dynamically selects and preserves only the most informative visual tokens while discarding redundant ones, significantly improving inference speed without sacrificing accuracy.\n\nThe authors introduce a trainable token selection mechanism that leverages importance scores to prune the input token set. This method is evaluated across several MLLMs and vision-language benchmarks (e.g., VQA, image captioning), demonstrating that token reduction leads to substantial latency savings with minimal performance degradation. Additionally, the paper explores both plug-and-play and end-to-end training approaches for integrating the token reduction module into existing MLLM pipelines.\n\nOverall, the paper contributes a practical and generalizable framework for making MLLMs more computationally efficient, which is especially valuable for deployment in resource-constrained or real-time settings.", "questions": "1. The proposed token reduction strategy shows strong results across standard benchmarks, but how does it perform in cases involving dense or fine-grained visual information (e.g., images with small, critical details)? Could you provide either qualitative examples or empirical results for such edge cases? Clarifying this would help assess whether the method consistently preserves semantically crucial content across varying data complexities. A strong showing on these examples would positively affect the significance and robustness assessment.\n2. Since token pruning alters the visual input sequence, how does this affect cross-modal attention alignment in models like Flamingo or BLIP-2 that heavily rely on interaction between visual and text tokens? Have the authors observed any degradation or shifts in attention patterns, and if so, are there mechanisms to preserve alignment fidelity? Addressing this woul", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper presents a high-quality and timely contribution to the field of multimodal large language models (MLLMs), specifically addressing the challenge of computational inefficiency caused by long visual token sequences. The proposed token reduction framework is well-motivated, methodologically sound, and broadly applicable across multiple MLLM architectures. The authors introduce a token selection mechanism that is both trainable and compatible with existing models in a plug-and-play fashion, offering flexibility in deployment scenarios. Their extensive experiments across vision-language tasks—including VQA, image captioning, and visual grounding—demonstrate that substantial reductions in inference cost (e.g., 30–50% token reduction) can be achieved with minimal or negligible impact on task accuracy.\n\nIn terms of originality, while token reduction and pruning are not entirely new ideas in the vision or NLP domains, this work distinguishes itself by developing a general and efficient method tailored for MLLMs, a relatively underexplored area in terms of inference-time optimization. The clarity of presentation is strong; the paper is well-organized with clear explanations of the proposed modules, training objectives, and ablation studies. Diagrams and figures effectively illustrate the architecture and token retention behavior, aiding comprehension.\n\nHowever, a few limitations are worth noting. First, the paper could benefit from more discussion around the potential trade-offs introduced by aggressive token pruning in edge-case scenarios, such as images with dense or fine-grained visual detail. Second, the token scoring and selection mechanism, while effective, could be explained in more depth, especially regarding its training stability and generalization to different MLLMs. Lastly, the work does not explicitly address how token reduction may interact with downstream decoding strategies or cross-modal attention patterns—areas that could present challenges in real-world deployment.", "was_revised": false, "final_justification": "", "rebuttal": "## Comment to Reviewer 9aDZ\n\nWe highly appreciate your time and effort in reviewing this paper, as well as your encouraging and constructive comments on our work. Below, we response to your key concerns point by point.\n\n**Comment 1:** The paper could benefit from more discussion around the potential trade-offs introduced by aggressive token pruning in edge-case scenarios, such as images with dense or fine-grained visual detail. \n\n**Response:** Thanks for your insightful suggestion. Following your suggestion, we further conduct experiments on two additional VL tasks, *i.e.*, visual grounding (RefCOCO) and image captioning (COCO). These two tasks and benchmarks are related to edge-case scenarios with dense and rich image details. \n\nThe results are reported in the following table. We can observe that the advantages of our DyVTE are much more obvious on these two granular VL tasks. Meanwhile, token pruning methods like FastV meets obvious performance drops due to dropping necessary visual information too early. And these results also show the necessity of dynamic visual-token exist in DyVTE, which can better handle more diverse VL tasks.\n\n|Method|RefCOCO REC Val Acc@0.5|exit layer|COCO2014 Cap Val CIDEr|exit layer|\n|-|-|-|-|-|\n|LLaVA-7B|50.03|-|108.74|-|\n|LLaVA-7B+FastV|37.94|-|108.03|-|\n|LLaVA-7B+DyVTE|51.86|18.37|107.62|22.37|\n|InternVL-7B|55.34|-|113.70|-|\n|InternVL-7B+FastV|55.34|-|113.95|\n|InternVL-7B+DyVTE|55.29|17.16|112.42|23.32|\n|LLaVA-13B|66.56|-|113.79|-|\n|LLaVA-13B+FastV|61.47|-|113.95|-|\n|LLaVA-13B+DyVTE|67.98|20.82|111.27|25.04|\n\n**Comment 2:** The token scoring and selection mechanism, while effective, could be explained in more depth, especially regarding its training stability and generalization to different MLLMs.\n\n**Response:** Thanks for your insightful question. Firstly, the training stability is well ensured by the task definition of DyVTE. As discussed in Sec.4.1, we define the visual-token exit as a problem of simple binary prediction. This definition can help the optimization of DyVTE independent to gradient updates of MLLMs, and DyVTE can be also directly supervised via the binary labels based on MLLMs' prediction, as introduced in Sec.4.2. In this case, DyVTE is easy to optimize, and the training expenditure is cheap, e.g., about 30 mins with about 8,000 VL examples.  \n\nBesides, its generalization is well confirmed in our work. As shown in Tab.1 and 2, DyVTE can achieve much better efficiency while retaining performance on 5 MLLMs of different families. Notably, as suggested by other reviewers, we also test its direct generalization to different MLLMs, *i.e.*, training one DyVTE on LLaVA (the third and last rows) and validating on InternVL and VILA.\n\n|Method|TextVQA|Exit Layer|GQA|Exit Layer|SQA|Exit Layer|\n|-|-|-|-|-|-|-|\n|InternVL+DyVTE|55.80|19.50|62.9|15.18|66.20|13.46|\n|InternVL+DyVTE-LLaVA|55.22|18.91|61.32|15.74|66.39|17.86|\n|VILA+DyVTE|62.60|21.70|63.10|16.17|69.5|15.38|\n|VILA+DyVTE-LLaVA|60.90|20.08|61.18|15.86|69.86|16.68|\n\nFrom the above experiments, the first observation is that the DyVTE trained on LLaVA can be directly applied to other MLLMs. Despite suffering a slight performance degradation, the optimal existing layers are close to the ones of the specifically tuned DyVTEs. These results show that DyVTE has learned generalized patterns across MLLMs.\n\n**Comment 3:** The work does not explicitly address how token reduction may interact with downstream decoding strategies or cross-modal attention patterns—areas that could present challenges in real-world deployment.\n\n**Response:** Thanks for your comment. In terms of principle and experimental results, we think that DyVTE barely affects the downstream decoding strategies and attention patterns of MLLMs. \n\nFirstly, the removal of all visual tokens only plays at the pre-filling stage of MLLMs. After that, the input tokens of each layers of MLLMs are fixed for the following decoding steps. In this case, either the popular KV caching or the recent multi-token decoding strategies can well work with our DyVTE. Moreover, without using attention-based metrics, DyVTE can also well support the acceleration techniques like FlashAttention.  \n\nIn terms of the attention patterns, the removal of visual tokens at the optimal layer barely impact the text token attentions according to our findings. As shown in Fig.4 and discussed in Line 141-144, the modeling process of the remaining text tokens are almost the same after the visual token removal, and the only difference is the decreased attention intensity due to the shorter token sequence.\n\nOverall, our DyVTE is applicable to existing downstream decoding strategies and has little impact on the default attention patterns.\n\n**Comment 4:** The proposed token reduction strategy shows strong results across standard benchmarks, but how does it perform in cases involving dense or fine-grained visual information (e.g., images with small, critical details)? Could you provide either qualitative examples or empirical results for such edge cases? Clarifying this would help assess whether the method consistently preserves semantically crucial content across varying data complexities. A strong showing on these examples would positively affect the significance and robustness assessment.\n\n**Respones:** Thanks for your comment. Following your suggestion, we conduct additional experiments on two vision-and-language tasks, namely visual grounding (RefCOCO) and image captioning (COCO). These two tasks all requires the dense or fine-grained visual understanding. \nAs shown in the table below, our DyVTE method demonstrates more obvious advantages on these challenging tasks. \nIn contrast, metric-based pruning strategies such as FastV encounter obvious performance drops, due to the premature removal of semantically critical visual information.\n\n|Method|RefCOCO REC Val Acc@0.5|exit layer|COCO2014 Cap Val CIDEr|exit layer|\n|-|-|-|-|-|\n|LLaVA-7B|50.03|-|108.74|-|\n|LLaVA-7B+FastV|37.94|-|108.03|-|\n|LLaVA-7B+DyVTE|51.86|18.37|107.62|22.37|\n|InternVL-7B|55.34|-|113.70|-|\n|InternVL-7B+FastV|55.34|-|113.95|\n|InternVL-7B+DyVTE|55.29|17.16|112.42|23.32|\n|LLaVA-13B|66.56|-|113.79|-|\n|LLaVA-13B+FastV|61.47|-|113.95|-|\n|LLaVA-13B+DyVTE|67.98|20.82|111.27|25.04|\n\n**Comment 5:** Since token pruning alters the visual input sequence, how does this affect cross-modal attention alignment in models like Flamingo or BLIP-2 that heavily rely on interaction between visual and text tokens? Have the authors observed any degradation or shifts in attention patterns, and if so, are there mechanisms to preserve alignment fidelity?\n\n**Respones:** Thanks for your comment. In fact, your mentioned case may be an advantage of our DyVTE compared to previous token pruning methods. \n\nSpecifically, DyVTE is to directly remove all visual tokens during the inference of MLLMs, when these visual tokens have well interacted with the text ones and no longer contributes to multimodal reasoning. Thus, it will not alters the visual input sequence, while some token pruning methods does. In terms of Flamingo and BLIP-2, we think the principle of DyVTE is also applicable to them, *i.e.*, stopping the involvement of visual tokens when they not longer work for multimodal reasoning. \n\nIn terms of the attention patterns, we can see that DyVTE barely changes the attention patterns of text tokens after it removing all visual tokens, as shown in Fig.4 and disccssed in Line 141-144. The only difference is the slight changes of attention intensity due to shorter token sequences. This observation is also confirmed in quantitative experiments, where the correctly removal of visual tokens will not decline performance."}]

Author rebuttal:

Meta-review: The paper proposes Dynamic Visual-Token Exit (DyVTE), a lightweight hyper-network that learns when to drop all visual tokens during MLLM inference after observing three distinct attention stages. Strengths include extensive experiments on five MLLMs showing up to 45.7 % FLOP reduction without accuracy loss, strong rebuttal addressing training cost, universal transferability, and latency gains. Weaknesses center on limited novelty—similar insights exist in LLaVA-Mini/FastV—and coarse all-or-nothing pruning; reviewers also wanted richer related-work discussion. After the discussion, all four reviewers raised or kept positive ratings. Especially,  Reviewer SwWi claimed that most of the earlier concerns have been addressed, indicating the reviewer has acknowledged the novelty of this paper. Based on the above reason, this paper is recommended to be accepted.
Decision: Accept (poster)