Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
Authors: kaiyuan Li
Xiaoyue Chen
Chen Gao
Yong Li
Xinlei Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision Language Models; Efficiency; Token pruning
Primary area: deep_learning

Abstract: Large Vision-Language Models (LVLMs) have shown impressive performance across multi-modal tasks by encoding images into thousands of tokens. However, the large number of image tokens results in significant computational overhead, and the use of dynamic high-resolution inputs further increases this burden. Previous approaches have attempted to reduce the number of image tokens through token pruning, typically by selecting tokens based on attention scores or image token diversity. Through empirical studies, we observe that existing methods often overlook the joint impact of pruning on both the current layer's output (local) and the outputs of subsequent layers (global), leading to suboptimal pruning decisions. To address this challenge, we propose Balanced Token Pruning (BTP), a plug-and-play method for pruning vision tokens. Specifically, our method utilizes a small calibration set to divide the pruning process into multiple stages. In the early stages, our method emphasizes the impact of pruning on subsequent layers, whereas in the deeper stages, the focus shifts toward preserving the consistency of local outputs. Extensive experiments across various LVLMs demonstrate the broad effectiveness of our approach on multiple benchmarks. Our method achieves a 78\% compression rate while preserving 96.7\% of the original models' performance on average. Our code is available at https://github.com/EmbodiedCity/NeurIPS2025-Balanced-Token-Pruning.

have substantially Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
Kaiyuan Li Xiaoyue Chen'r Chen Gao Yong Li?, Xinlei Chen
Tsinghua Shenzhen International Graduate School
BNRist, Tsinghua University
(likaiyua23, (tt.:a/uaa222cccriiac2244223211 chenxiac24)0maile www.amaaa......nnaaammmmm
{chgao96, 1iyoog07) edu. en, chen. xinlei@sz. tsinghua, eduu cn
Abstract
Large Vision-Language Models (LVLMs) have shown impressive performance across multi-modal tasks by encoding images into thousands of tokens. However, the large number of image tokens results in significant computational overhead, and the use of dynamic high-resolution inputs further increases this burden. Previous approaches have attempted reduce the number of image tokens through token pruning, typically by selecting tokens based on attention scores or image token diversity. Through empirical studies, we observe that cxisting methods often overlook the joint impact of pruning on both the current layer's output (local) and the outputs of subsequent layers (global), Icading to suboptimal pruning decisions. propose Balanced Token Pruning (BTP), plug-
To address this challenge, and-play method for pruning vision tokens. Specifically, our method utilizes a small calibration set divide the pruning process into multiple stages. In the early stages, our method emphasizes the impact of pruning on subsequent layers, whereas in the deeper stages, the focus shifts toward preserving the consistency of local outputs. Extensive experiments across various LVLMs demonstrate the broad effectiveness of approach multiple benchmarks. Our method achieves 78% compression rate while preserving 96.7% of the original models' performance on average. Our code available at Balanced- Torend-Toooo---rnnnn
Introduction
Recent advances in Large Vision-Language Models (LVLMs) & 15
4R
improved visual understanding. These models typically employ visual encoder to convert images into discrete tokens, which are then processed jointly with textual tokens by a large language model backbone. The incorporation of visual information significantly increases the total number of input tokens 2 28 59]. a problem further amplified when handling high-resolution images. In edge applications such as emergency monitoring 7 44], logistics (6 [56]. and smart homes [51). models are typically deployed on devices like drones and unmanned vehicles [23 [0]. which are constrained by limited memory and striet latency requirements. The excessive number of image tokens poses major bottleneck for deployment, drawing increasing research interest in accelerating edge inference
38
Prior studies [T] have demonstrated that visual tokens often exhibit significant redundancy [5] 26). Consequently, visual token pruning has been proposed as effective strategy to reduce input redundancy and enhance computational efficiency (4
53 48]. Visual token pruning faces
"Equal Contribution. Coneesonnding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

two fundamental challenges: identifying the most important visual tokens and determining the appropriate layers for pruning. Existing token pruning strategies can be broadly classified into two categories: attention-basee based methods that leverage text-image interactions 5
46 32 and diversity-
based methods that exploit the heterogeneity of visual representations [0 However, the impact of their distinct optimization objectives on overall model performance remains underexplored, and systematic comparison between them is largely absent. Moreover, when comes to pruning layer selection, existing methods rely heavily on validation performance and manually defined settings, lacking principled guidance based on the model's intrinsic properties.
What animal is in the picture?
Attention Area
Variation
Variation
Original
Layer
Layer 11
Layer 18
There is cat in the picture.
Figure 1: Layer-wise visualization of attention in LVLMs.
To address these problems, first explore the nature of image token pruning from an intuitive perspective: its impact on the current layer's (local) output and its influence on the outputs of subsequent pruning layers (global). We begin by visualizing the spatial distribution of image tokens that receive higher attention from text tokens across different layers. As shown in Figure [] observe that the image tokens attended by text tokens vary across different layers. This indicates that pruming solely based on the current layer tends to overlook its impact on subsequent layers. Then we further investigate the impact of different pruning methods the model outputs. Specifically, we compare the hidden states of output tokens at different decoding positions under two pruning methods with those of the original model.
Decoded Taken
Decoded Token
Cosine Similarity
0.99
Attantonn Sanec
Jote rton Baaed
0.97
Layer
Layer
Figure 2: Impact of different pruning strategies on layer-wise representations.
It can be found in Figure 2 that attention-based methods preserve output similarity well at early pruning layers, but the error accumulates in deeper layers. contrast, diversity- methods do not maintain output similarity at the initial layers, but achieve better consistency in later pruning stages. This implies that cecnttoo-based pruning methods focus solely on optimizing the current pruning layer while ignoring their impact subsequent layers, whereas diversity-bai -based methods overlook the preservation of output quality at the current layer.
Motivated by the above observation, we aim to tackle a fundamental challenge: how to prune with joint consideration of the current and subsequent layers achieve global optimality. To address this challenge, we propose Balanced Token Pruning (BTP), a visual token pruning method that balances local objectives (current layer) with global objectives (subsequent layers). We begin by analyzing and formulating local-global objective for image token pruning. Based on this objective, BTP first partitions the pruning process into multiple stages using small calibration set [35] 20), leveraging the way LVLMs process images, as illustrated in Figure 4 In early stages, where more image tokens are retained, BTP emphasizes a diversity-based objective to preserve the quality of downstream representations. In later stages, where fewer tokens are retained, it prioritizes an attention- based objective to maintain the consistency of local outputs. With this design, we preserve token diversity in the early layers while focusing task-relevant tokens in the later layers.

---

Extensive experiments demonstrate the effectiveness of our proposed BTP method. We evaluate BTP across models of varying sizes and architectures, consistently achieving superior performance under higher compression ratios. Notably, our approach retains only 22% of the original image tokens on average while preserving 98% of the model's original performance. Furthermore, end-to-end efficiency evaluations confirm that BTP significantly reduces both inference latency and memory usage in practice.
2 Related work
2.1 Large Vision- Language Models
Recent progress in large vision language models (LVLMs) has been substantially accelerated by the open-sourcing of foundation models like LLaMA AT and Vicuna [60] Representative models,
including LLaVA 27 28 29), Qwen-VL 2 43 and InternVL g [5] leverage vision encoders
to encode images into visual tokens, which are then integrated into the language model unified multimodal representation and understanding [14]. For example, LLaVA-115 encodes image into 576 visual tokens using single-scale encoder. As these models increasingly support high-resolution visual inputs 2 28), the number of visual tokens grows. Using a multi-resolution encoding strategy, LLaVA-NeXT can generate up to 2,880 tokens per image. Multimodal large models have been widely applied various scenarios, including embodied agent [77] The large number of image tokens limits its applicability in scenarios such real-time applications [39].
34
25 9
2.2 Visual Token Pruning
Early efforts reduce visual token redundancy primarily focus on attention-based pruning 4. 19 (4, 32). For example, FastV (5) prunes visual tokens with low attention scores after the filtering layer, with subsequent layers processing only the remaining token. Another approach, VTW 226. adopts complete token elimination strategy, removing all visual tokens after specified layer. PyramidDrop [66 introduces more sophisticated approach, performing stage-wise pruning through- out the transformer. ranking visual tokens by their attention scores to the instruction token at each stage and progressively discarding the least informative ones. Compared to attention-based methods, diversity-based methods prioritize retaining a richer variety of semantic information. For instance, DivPrune ① formulate token pruning as a Max-Min Diversity Problem [33 37 Additionally, some methods fuse remaining tokens into retained tokens through token fusion such as LLLVA--PuMert 40) and VisionZip [47). Different from prior methods, our method jointly considers the impact of pruning on both the current layer and subsequent layers.
3 Preliminary
3.1 Visual token processing
In the prefilling stage, images and texts are first encoded into embedding vectors (tokens), which are then processed by LVLM. We denote the input token sequence as which consists of the system prompt Xs, the image tokens X, and text tokens Xy, X (Xx)), X is then fed into the LLM backbone composed of N decoder layers. For the I-th decoder layer, we denote the input as (!) and the layer output (1+1) is:
((+1) (9) Atten ① (LN(X())) MLP (aN (1)
x())):
(1)
where (1 is the attention block, LN is the layer normalization and MLP(() is the projector layer. It can be observed that the outputs of the attention block and the MLP block are closely tied to the attention mechanism (4). Formally, the attention mechanism can be represent as:
autro Softmar Q((Ki) M
Vi.
(2)
VD
where Q₁, Ki, Vi are calculated by Query projector, Key projector and Value projector. Dr is hidden state dimension. M is the casual mask which imposes constraint such that each token is permitted to incorporate information only from tokens at earlier positions. K₁, Vi are stored the KV cache for further decoding stage.
3

---

3.2 Visual token pruning formulations
Prior works on image token pruning can be broadly categorized into attention-basse methods 46 and sveesstty-bssed methods []] Attention based methods utilize text- text-image attention score to select important image tokens specific layers. For input sample with m text tokens, we can denote the importance score Sing of image tokens at 1-th layer as:
m Atten ( (X,, XX)
$0 kmg
Σ m
(3)
After obtaining the importance scores of the image tokens, these methods select a pruned image token set Pet Potten C X with the highest scores. In contrast to attention score-based methods, diversity- approaches focus on maximizing the diversity among selected image tokens. These methods are typically based either on the spatial diversity of the selected image token set or on the semantic diversity of the selected images. Formally, given a diversity metric C {Fopa F identify a pruned set of image tokens Fdiv C X₁ that maximizes the objective function Ldiv:
Feem. ) our goal is to
Live max F(Paso))-
Methodology
4.1 Limitations of existing methods
Attention-based methods pursue local optima We analyze the impact of pruning image tokens on the subsequent text and response tokens. From Equations [ we can see that pruning image tokens at 1-th layer mainly affects the layer output (1+1) by changing the attention output, which is weighted sum of the value vectors Vi. If the norms of the Vi are similar, selecting image tokens with high importance scores defined in effectively reduces the difference between the layer output before and after pruning. We provide supporting evidence for this assumption the Appendix 7.1 Formally, given original I-th layer output X (I+1) origin and pruned 1-th layer output x (1-1)
praned distance
metric function D(,,), we can define the objective function Latten of intention-basee methods 5 46
as:
Latten min D (1+1) origin
((+1)
(5)
F
However, iitention-basee methods locally optimize the output error at individual layers. For instance, pruning is conducted at the 1-th layer and (1+ k)-th layers, with Py and Pi+ denoting the respective optimal sets of selected image tokens. As shown in Figure T PP Pi. So, attention-based selection results in a globally suboptimal pruning strategy.
Diversity-based methods ignore local constraints The diversity- baased approach aims to maximize the diversity of the selected tokens, thereby partially mitigating the issues encountered by attention- based methods tokens with maximally different semantic information. However it can be observed in Figure that diversity-based approaches ineffective in maintaining local output consistency, which can lead to a failure preserving local output consistency during deep-layer pruning, resulting in degraded performance.
in Figure I Because diversity- based methods tend to select
Layer selection for pruning Current approaches typically rely on manually predefined pruning layers utilize small validation select pruning layers based the observed performance. However, these methods require extensive trial-and-error and dataset-specific calibration. As de- scribed in Section due the presence of the causal mask M, the encoding an image token in the LLM backbone is independent of the input question. Therefore, we aim to determine the pruning layers from the perspective image token encoding.
4.2 Balanced token pruning with joint local and global objectives
Local-global objective Based on the above analysis, we argue that an effective token pruning strategy should achieve local optimization by preserving the current layer's output, while also considering the global impact of pruning on subsequent layers. As shown in Equation []] the model's output depends on both the outputs of previous layers and the attention module of the current

---

Pruning Stage Selection
Balanced Token Pruning
Spatial rultializacion
Global Pruning! !
28 Wright
Layors
Pruning Layers
Shallow Layer Focus Diversity
Select
weighe 0.5
Seletion
Medium layer Balance
Deep Layer Focus Attn
Easape
Image Triken
Processing
Image Ereader
MLLM
Image
2t Weight 6.31
2t
Waight 0.5
Encoder
Dreade
Text
describe...
Local Pruning
Calbration Image Set
Image
Important Image
Layer
Figure 3: Overview of BTP: We first use a calibration set to determine the pruning layers. In the early layers, we emphasize diversity-bast pruning preserve the output of subsequent layers. In the deeper layers, attention-based pruning is prioritized to maintain the output of the pruning layers. Due the pruning strategy, we achieve overall optimal pruning balance.
layer. Therefore, to ensure that the final output of the pruned model remains similar to that of the original model, we should maintain the similarity between the output of each pruned layer and its corresponding original output. Firstly,
formulate global objective function. Suppose token pruning is performed at layers 1 < l2 < lg. For each pruned layer E (21,2,44 we aim to select a subset tokens iP; such that the difference between the pruned outputs X-+1 prrnndd and original outputs X Xrrigin minimized. To quantify hidden vectors' difference, we use a unified distance function D(..)) to measure the discrepancy between the outputs before and after pruning. Then our objective is to minimize the total output discrepancy across all pruned layers:
Cglobal =
Σ D(Xtt origin' x(+1).
(6)
According to Equation 5] we can get optimal pruned token set Pf based on attention. However, since the attention distribution varies across input samples and Ptg C Pig C Pt. it is difficult to predict which tokens will be important for deeper layers (e.g.. 12, 13) when pruning at layer 4,. To address this issue, we propose to optimize local-global objective approximate the optimal token set P Building upon the local attention-bare to approximate the token preferences of later layers. Assume weight coefficient À (0. 1), measure diversity by computing the sum of distance F.ie(.) among elements within a set:
selection objective, we introduce a diversity term
Ltocal global
Σ
(A
Σ JEP:
Atten ( (x())
Xx) + (1 F.)
(7)
The first term of Equation 7 ensures that the output of the pruned layer remains close to the original, while the second term encourages the selected tokens at previous layer l to also include those important for deeper layers such as l2 and lg.
Balanced token pruning (BTP) Building upon the proposed local-global objective, introduce our method. Figure 3 our approach divides token pruning into multiple stages denoted as S (si..... s,). Under a predefined pruning ratio a, each stage retains fixed fraction of image tokens from the previous stage. As shown in Appendix 7.2, we can observe that retaining only attention objectives. Since early pruning
small number of image tokens sufficient to optimize stages retain more tokens and influence the pruning decisions of later stages, their objectives need to emphasize token diversity. In contrast, deeper stages preserve fewer tokens and have less impact on subsequent stages. Therefore, set the hyperparameter A; to gradually increase across stages.
5

---

Attention optimization: We optimize the attention objective by selecting the top-k image tokens with the highest importance scores defined in Equation| To efficiently computing the importance scores, we only use the last token of the input prompt as Xr, which reduces the computational complexity to (n)). We observe that the attention scores are influenced by positional encoding, which leads to a tendency to favor tokens located toward the end of the sequence. We apply a re- re-balancing operation to alleviate the influence of positional encoding. Assume that at /-th layer, we aim to prune the image tokens by selecting k indices I out of N candidates based on the attention scores A,. Instead of directly selecting the top-k tokens, we first over-select the top-k' tokens indices /g,, where k' > To mitigate positional bias, rebalance the selection by first retaining tokens from earlier positions, followed by selecting additional tokens from later positions:
N
Ipre = -w < b
(8)
N
post = Ix(TK 2 2 l[: prrell
(9)
I = Concat (Ipres Ipost).
(10)
Through the rebalancing operation, we are able to preserve the attention objective while selecting more informative tokens.
For optimizing the second objective related to diversity, we follow the formulation used DivPrune by modeling it as Max-Min Diversity Problem (MMDP). However, solving the MMDP objective requires (n²²) computational complexity and cannot be efficiently accelerated by GPUs, resulting significant computational latency. This issue becomes more pronounced in high-resolution multimodal models with larger number of image tokens. To address this challenge, we propose an initialization strategy based on spatial position information. We observe that image patches with large spatial distances tend to exhibit greater semantic differences, while spatially adjacent patches are often semantically similar. Based on this intuition, we initialize the set of selected image tokens by solving an MMDP problem over their spatial positions. Formally, given N image tokens X,, which are originally obtained by flattening 2D image, we first formulate a 2D grid of size VN vN. For any two tokens y and we from the N tokens, their distance defined as the Manhattan distance d(..-) between their positions in the 2D grid. Based this distance metric, we construct the initial token set Einitial:
Diversity optimization:
Einitial argmaa min (d(y, w) VS C X,).
(1I)
y.wES
4.3 Pruning layer selection
We propose that determining which layers to prune is closely related to encoding process of image tokens. Specifically, pruning should occur either before or after the layers where the meaning of image tokens changes significantly, since it is difficult to identify truly important tokens in such layers. We compute the cosine similarity between image token hidden states x/,x/+ x/+1 before and after each layer. For each layer, we plot the number of tokens with similarity below threshold alongside the total attention allocated to image tokens. As shown in Figure 4 it can be observed that LVLMs tends to allocate more attention to image tokens layers following those where the representations of image tokens undergo significant changes. Based on these insights, we propose task-independent layer selection strategy for pruning. Using a fixed set 64 samples across all datasets, we identify layers immediately before and after major shifts in image token semantics. As shown in Figure 3 we
perform pruning at selection layers, which enhances the effectiveness of our pruning strategy.
Dheakii-10 Us
Attedies Sam
0.20
W
auc-
13 11
2B
15
Lawer Iodes
Lig Todis
Image Procsesing
Atter ton Allocation
Image Processingg
Attention Allocation
Figure 4: Layer-wise image token hidden state dynamics and attention allocation in LVLMs.

---

Table Comparison of BTW with VTW, PDrop, FastV, and DivPrune across different models and datasets. For models using dynamic resolution, we report the token retention ratio instead of the absolute token count.
Method
Token
TFLOPS
GQA
MME MMB
POPE SQA MMVET Avg.
LLaVA-1.5-7B
Original
576
3,82
62.0
1510.7
64.3
85,8
69.4
29.0
100%
VTW (AAAI25)
26
236
1.67
51.3
1475.0
63.4
82.1
68.8
17.8
89%
PDrop (CVPR25)
46
192
1.30
57.1
1399.0
61.6
83.6
68.4
25.8
94%
FastV (ECCV24)
S
1.65
57.6
1465.0
61.6
81.0
68.9
96%
DivPrune (CVPR25)
128
0.83
58.8
1405.4
62.1
85.1
68.4
27.4
96%
(Tu(s)
128
0.85
59.0
1487.0
62.7
85.6
69.1
29.1
98%
LL.a LLa VA-1.5-13B
Original
576
7.44
63.2
1521.7
68.8
87.0
72.7
37.4
100%
VTW (AAAI25)
26
236
2.97
55.6
1517.1
67.7
79.0
72.2
22.6
89%
PDrop (CVPR25)
46
192
2.46
60.5
1493.0
67.3
85.1
73.7
32.8
96%
FastV (ECCV24)
172
2.25
60.0
1473.0
67.0
83.6
72.9
31.9
95%
DivPrune (CVPR25)
128
1.63
58.8
1461.0
65.8
86.5
72.6
34.0
96%
BTu())rs)
128
1.68
62.2
1519.7
68.0
86,9
72.7
34,5
98%
LLaVA- VA-1.6-7B
Original
100%
20.82
64.2
1519.3
67.1
86.4
73.6
37.5
100%
VTW (AAAI25)
26
40%
9.11
53.3
1472.8
65.6
84.1
68.3
16.3
85%
PDrop (CVPR25)
46
25%
6.77
60.4
1462.6
65.1
86.4
68.3
27.4
92%
FastV (ECCV24)
S
22%
5.76
60.3
1469.1
64.3
85.5
68.2
32.3
94%
DivPrune (CVPR25)
22%
4,20
61.4
1467.9
65.4
86.2
92%
(Tu()
22%
4.52
60.6
1490.8
65.8
86.7
68.4
30.3
94%
Qwen2.5- VL-72.5-VL-7B
Original
100%
5.48
60.4
1690.8
82.5
87.4
76.7
100%
VTW (AAAI25)
26
40%
2.38
40.2
1129.8
58.7
61.5
69.7
4.5
65%
PDrop (CVPR25)
46
30%
1.81
49.9
1462.5
70.6
76.8
72.6
9.58
82%
FastV (ECCV24)
S
30%
1.79
52.6
1595.5
73.4
83.9
74.0
16.2
96%
DivPrune (CVPR25)
25%
1.34
50.1
1639.2
76.9
85.4
73.0
17.5
96%
BTP()urs)
25%
1.67
57.2
1651.5
75.2
86.2
74.1
16.8
97%
5
Experiment
Baselines and models To rigorously assess the generalizability of proposed image token compression method, we integrate it into several state-of-the-arr multimodal large models and conduct extensive experiments diverse benchmark tasks. Specifically, evaluate approach four representative models: VLaVA-v 1..5 LLaVA-v1.5-1333 LLaVA-v1.6-77 and Qwen2.5- VL-
We select several plug- aad---ndd-lyyy compression baselines that support inference-tine time token pruning: FastV S and PyramidDrop [46), which select informative tokens via attention mechanisms; DivPrune [1. which filters tokens based visual diversity and VTW (26), which discards image tokens at a specific transformer layer determined by validation performance.
7B-Instruct 27 28
43
Benchmarks and evaluation We conduct comprehensive experiments standard visual under- standing tasks using models of different sizes, model families, and compression ratios. We report the results GQA, MMB, MME, POPE, SQA and MM-VeT [33 22 30 49 (5)). All experiments are carried out using the LMMs-Eval
3 24 framework. In addition to accuracy on each dataset, evaluate all methods in terms of FLOPs, inference latency, and KV cache memory usage. For inference throughout, we follow the PyramidDrop. Specifically, we calculate the FLOPs of the 1-th layer's attention and MLP modules through 4nd 2n + 3ndm. is the number of tokens, d is the hidden state size, and m is the intermediate size of the FFN.
Implementation details All pruning experiments are conducted on 8 NVIDIA A800 GPUs using the HuggingFace Transformers library. To determine pruning stages, we randomly sample 64
instances from the LLaVA-655k 227 28 29 dataset and use the same set across all models and

---

benchmarks, thus avoiding separate calibration for each benchmark. We gradually reduce the number of image tokens each stage. In the early layers, we use larger À value focus more on global information, while in the deeper layers, we use smaller lambda to emphasize local details. More implementation details for different models are provided in the see Appendix [7.3] Similar to the implementation of PyramidDrop, we compute the required attention scores separately within the FlashAttn itn module at the specified pruning layers, achieving full compatibility with FlashAttn 12 It is worth noting that all our experiments are conducted with FlashAttention acceleration enabled.
5.1 Main results
BTP outperforms SOTA methods across LVLMS As shown in Table we conduct extensive experiments across different model families and parameter scales. Empirical results demonstrate that our approach consistently surpasses state-of- saaee-of-he-aar f--th--art art methods on most benchmark tasks. Our method achieves 98 % of the original average performance under 22 compression rate across LLaVA models of different sizes. Moreover, our method consistently outperforms all models, achieving better results than both attention-based and diversity-basee based approaches. We also visualize the impact of different methods on layer outputs Figure 5 our method preserves consistency with the original outputs at both local and global levels. The Appendix 7.5 further provides visualizations of the spatial distribution of image tokens selected by various methods. Our method yields more effective token selection in deeper layers.
BTP maintains stable performance across different compression ratios We assess the performance of our
Dhamth Exsd
method across range of compression ratios to verify its effectiveness. We find that FLOPs account only for the computational cost of the attention and MLP modules, while ignoring the overhead introduced by additional com- ponents. As result, FLOPs alone fail to accurately reflect actual inference latency. Therefore, as shown in Ta- we compare the performance and average inference
cartal
Costire Sinalsrity
0,590
0.005
ble 2 time different methods under varying compression ra- tios. In can be observed that although DivPrune achieves lower theoretical FLOPs, its end to end latency even ex- ceeds that of the original uncompressed model. In contrast, method leverages spatial division for initialization, significantly reducing the actual inference time. Across various compression ratios, our method consistently achieves better performance than tate-offthhear approaches on most datasets, without incurring additional computational overhead.
Figure 5: Effect of various pruned meth- ods on the output of decoder layers.
Table 2: Performance comparison with FastV and DivPrune across varying compression ratios. We
report the results on Va-V1.
Method
Average Token
TFLOPS
Latency
GQA
MME
MMB
SQA
I.I.VA-1.5-BB
576
3.82
0.145s
62.0
1510.7
64.3
69.4
FastV
128
0.86
0.122s(15% L)
49.6
1388.6
56.1
60.2
DivPrune
128
0.83
0.224 1)
58.8
1405.4
62.1
68.4
BTP(ours)
128
0.85
0.134s(7%%
59.0
1487.0
62.7
69.1
Fast'V
0.42
0.118s (18%
46.1
801.3
48.0
51.1
DivPrune
0.41
0.150s(0.5%%))
57.5
1350.0
58.5
67.6
BTP(()urs)
64
0.42
0.120s( (17% L)
55.0
1364.1
58.6
68.3
5.2 Efficiency analysis
The additional overhead introduced by our method primarily arises from the attention computation and the selection of the diversity set. Since we compute attention only between the final token and the image tokens, the added attention complexity is O(n). For the selection of the diversity set, our proposed spatial initialization strategy and progressive weight decay allow us to select only a small number of additional tokens. In this section, we compare the efficiency of our method with other

---

approaches, evaluating from multiple perspectives including theoretical FLOPs, inference latency, KV cache size, and corresponding benchmark performance. For inference latency, we report the average inference time per sample. For KV cache memory usage, we report the average GPU memory consumption after compression. We conduct experiments using I.LVVA-v1.5 and L.LVA-v Notably, LLaVA-v1.6 processes images at a higher resolution, resulting in larger number of image tokens.
Table 3: Evaluation of compression efficiency different models
Method
Averge token
Cache Size
TFLOPS
Latency
LLaVA-COCO
LLaVA-1.5-7B
576
0.34GB (100%)
3.82
2.24s
90.8
FastV
172
0.15GB (55.8% J)
1.65
2.11s (5% ()
80.6
DivPrune
128
0. D.B (67.6) ()
0.83
2.33s (4% f)
80.3
BTP(ours)
128
0.1 (67.6%
0.85
2.13s (4% 4)
80.9
LLaVA-1.6-7B
2880
1.11GB(10%%
20.82
4.24s
106.6
FastV
860
0.37GB (66.6% ()
6,45
3.77s (1 (119%L
92.6
DivPrune
633
0.28GB (74.7% J)
4.20
5.00s (17%1)
99.1
BTP(ours)
633
0.28GB (74,7% !)
4.52
3.91s(7%.)
98.9
As shown Table our method achieves the best performance while maintaining practical efficiency.
5.3
Ablation study
Choice of balance factor value: We first analyze the effect of À in the local-global objective functions. This factor determines the trade-off each layer between preserving local outputs and contributing to the global output. To thoroughly analyze the contribution of each pruning layer, we perform comprehensive ablation experiments on the LLaVA model. Our method includes three pruning layers, and we evaluate three configurations by fixing the À parameters of two layers while varying the remaining one: (I) tuning the shallow layer while fixing the middle and deep layers, (2) tuning the middle layer while fixing the shallow and deep layers, and (3) tuning the deep layer while fixing the shallow and middle layers. We define the ratio between the performance of the pruned model and that of the base model on the target task as the performance gain. The computation of performace performance gain is detailed the Appendix 7.4
145
VOA
1BC
PVB
205
2.00
10/0
1051
1.103.08.3.0
11105.06.1.1
10000.1 2.1
1-105.10.11
Ablation Layar
Ablaablttoon Layer
(o)Ablation Deep Layer
Figure 6: Ablation study on balance factor.
As shown in Figure 6 we can observe an early preference for the diversity objective in the shallow layers results in performance degradation. The middle layers should still retain moderate degree of diversity, whereas the deeper layers, due to the limited number of remaining tokens, should prioritize the attention objective. This highlights the importance of our method effectively balancing the two objectives.
Effectiveness of rebalanced attention and spatial diversity initialization: We then per- form ablation studies on the attention rebalance module and the spatial initialization module. We experimented with various combinations of the two modules. The results are presented in Table It can be observed that removing the attention rebalance module results in significant degradation in model performance. This degradation arises from the inherent bias in attention mechanisms, where positional encodings tend to shift at- tention disproportionately toward later tokens, leading to suboptimal token selection. On the other hand, omitting
Table 4: Ablation study on attention re- balance module and spatial initialization module.
RA 7
SI
Latency
MME
GOA
POPE
0.2324
1496.5
57.9
$6.4
0.14
1464.6
57.4
85.1

---

the spatial initialization module causes a marked increase in inference latency, in some cases even surpassing that of the original unpruned model. This suggests that while pruning reduces token count, naive initialization can introduce computational overhead that negates the benefits of pruning, thereby limiting the method's I' applicability in latency-sensitive real-world scenarios (55). This demonstrates the effectiveness of the proposed module in improving both model performance and inference speed. We also conducted an ablation study on the distance definitions used in the spatial diversity initial- ization module. As shown in the Appendix 7.7 we found that the Euclidean distance models the diversity of image tokens more effectively than the Manhattan distance.
Effectiveness of calibration-baset pruning stage selection: To evaluate the effectiveness of our proposed salibration-baser pruning stage selection, we compare with a base- line that uniformly divides the pruning stages according to the total number of decoder layers, under the same compression rate. Experimental results are shown in Table 5 We observe that our pruning layer selection method out- performs uniform selection. This is especially evident on Qwen2.5-VL, where uniform selection leads to signif- icant performance drop. We attribute this to differences in how Qwen2.5-VL processes image tokens as shown in Figure 4 We also conduct an ablation study on the size and composition of the calibration set. Specifically, we expanded the calibration set by incorporating images from multiple datasets, including GQA, V*Bench [45], and SQA and Urban Video-Bench [58]. We then repeated the experiment shown in Figure 4 using calibration set sizes of 64, 128, and 256. The results are presented in Appendix [7.6] we can see that the varia- tion patterns of image tokens remain consistent across different calibration sizes and content, demonstrating the robustness of our pruning layer selection method.
Table 5: Ablation study on layer selec- tion strategy.
Method
Stuge Selection
MMF
MMR
LL.
Avarged Ours
1483.2
1457.0
62.7
ULaa-v166
Averaged
1480.1
64.7
Ours
1490,8
65.8
Qwen2.svi
Avengee
1551.6
73.8
Ours
1641.5
75.2
In our method, the im- portance score of each image token is obtained by using the attention assigned to it by the last text token in the prefilling stage. To verify the robustness of this design, conduct ablation study comparing different ways of computing the importance score: 1. Averaging attention weights from all text tokens to each image token. 2. Following the approach in [57], where image-text similarity is first computed and the most similar text tokens are then selected to calculate the importance score.
Ablation on the Computation of Attention-
Importance Scores:
Table 6: Ablation study on importance score calculation method.
Method
MME
MMB
POPE
GQA
SQA
last-token(ours)
1497
63.4
85.6
59.1
69.1
averaged-tokens
1490
62.8
84.7
57.3
69.4
similarity-based
1485
63.1
84.7
57.9
69.7
The results are shown in Table We can see that last token efficiently modeling the importance score. We believe that the last token in the input prompt is suitable choice for computing the importance score because it typically decoded as the first output token during the decoding stage. This allows it to effectively capture the model's focus.
6 Conclusion
In this work, we conduct initial studies to investigate and verify the limitations of existing image token pruning methods. We further analyze the impact of two pruning strategies on model performance from the perspective of the objective function, and formulate local-globar pruning optimization objective. To reduce information loss during pruning, we propose Balanced Token Pruning (BTP), a multi-stage pruning method. We first determine the pruning stages using a calibration set. In the early layers, we focus on a diversity-oriented objective to account for the influence of pruning on deeper layers, while in the later layers, we adopt an allention-based objective to better preserve local information. In future work, we will further investigate the lightweight deployment on real devices [61] and explore its potential applications in multi-agent collaboration [36] 16].
10

---

Acknowledgments
This paper was supported by the Natural Science Foundation of China under Grant 62371269 and 62272262, Shenzhen Low-Altituee Airspace Strategic Program Portfolio Z253061, Guangdong Innovative, Entrepreneurial Research Team Program (202IZT09L197), Meituan Academy of Roboties Shenzhen and Tsinghua University- Toyoot Research Center.
References
[1] Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Di- vprune: Diversity-baseds visual token pruning for large multimodal models. arXiv preprint arXiv:2503.02115. 2025.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijic Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
[3] Li Bo, Zhang* Peiyuan, Zhang* Kaichen, Pu* Fanyi, Du Xinrum, Dong Yuhao, Liu Haotian, Zhang Yuanhan, Zhang Gc, Li Chunyuan, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024.
[4] Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, and Alan Yuille. Effi- cient large multi-modal models via visual context compression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[5] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. European Conference on Computer Vision, pages 19-35. Springer, 2024.
[6] Xuecheng Chen, Haoyang Wang, Yuhan Cheng, Haohao Fu, Yuxuan Liu, Fan Dang, Yunhao Liu, Jinqiang Cui, and Xinlei Chen. Ddl: Empowering delivery drones with large-scale urban sensing capability. IEEE Journal of Selected Topics in Signal Processing, 2024.
[7] Xuecheng Chen, Zijian Xiao, Yuhan Cheng, Chen-Chun Hsia, Haoyang Wang, Jingao Xu, Susu Xu, Fan Dang, Xiao-Ping Zhang. Yunhao Liu, and Xinlei Chen. Soscheduler: Toward proactive and adaptive wildfire suppression via multi-uav collaborative scheduling. IEEE Internet of Things Journal, 11(14):24858-2453 2024.
[8] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
[9] Moulik Choraria, Xinbo Wu, Sourya Basu, Nitesh Sekhar, Yue Wu, Xu Zhang, Prateek Singhal, and Lav R. Varshney. Semantically grounded gformer for efficient vision language understand- ing. 2024.
[10] Can Cui, Zichong Yang. Yupeng Zhou, Juntong Peng, Sung-Yeon Park, Cong Zhang, Yunsheng Ma, Xu Cao, Wenqian Ye, Yiheng Feng, Jitesh Panchal, Lingxi Li, Yaobin Chen, and Ziran Wang. On-board vision-language models for personalized autonomous vehicle motion control: System design and real-world validation, 2024.
[11] Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In International Conference Learning Representations (ICLR), 2024.
[12] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[13] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint 3339 2023.
11

---

[14] Chen Gao, Baining Zhao, Weichen Zhang, Jinzhu Mao, Jun Zhang, Zhiheng Zheng, Fanhang Man, Jianjie Fang, Zile Zhou, Jinqiang Cui, Xinlei Chen, and Yong Li. Embodiedeity; A benchmark platform for embodied agent in real- city environment, 2024.
[15] Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internyl:: a flexible-transfer pocket multi- modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1-17, 2024.
[16] Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, and Qing Li. Multi-modal agent tuning: Building vlm-driven agent for efficient tool usage, 2025.
[17] Qiuyi Gu, Zhaocheng Ye, Jincheng Yu, Jiahao Tang, Tinghao Yi, Yuhan Dong. Jian Wang, Jinqiang Cui, Xinlei Chen, and Yu Wang. Mr-cographs: Communication--fficcee multi-robot open-vocabulary mapping system via 3d scene graphs. IEEE Robotics and Automation Letters, 10(6):5713-5720, 2025.
[18] Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, and Shaohui Lin. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.00877 2024.
[19] Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, and Shaohui Lin. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification, 2025.
[20] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference Machine Learning, pages 4466-4475. PMLR, 2021.
[21] Drew A Hudson and Christopher D Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[22] Mohit lyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for sequential question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1 1821-1831, 2017.
[23] Zhuozhu Jian, Zejia Liu, Haoyu Shao, Xueqian Wang, Xinlei Chen, and Bin Liang. Path generation for wheeled robots autonomous navigation on vegetated terrain, 2023.
[24] Zhang Kaichen, Li Bo, Zhang Peiyuan. Pu Fanyi, Cahyono Joshua-Adrian, Hu Kairui, Liu Shuai, Zhang Yuanhan, Yang Jingkang, Li Chumyuan, and Liu Ziwei. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024.
[25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation, 2022.
[26] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 39, pages 5334-5342, 2025.
[27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023.
[28) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[29] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[30] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? In European conference computer vision, pages 216-233. Springer, 2024.
12

---

[31] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-frre free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.
[32] Yu Meng, Kaiyuan Li, Chenran Huang, Chen Gao, Xinlei Chen, Yong Li, and Xiaoping Zhang. Piphp: Per-layer per-head vision token pruning for efficient large vision-language models, 2025.
[33] Daniel Cosmin Porumbel, Jan Hao, and Fred Glover. A simple and effective algorithm for the maxmin diversity problem. Annals of Operations Research, 186:275- 2011.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[35] Charles C Ragin. Fuzzy sets: Calibration versus measurement. Methodology volume of Oxford handbooks of political science, 2, 2007.
[36] Jiyuan Ren, Yanggang Xu, Zuxin Li, Chaopeng Hong, Xiao-Ping Zhang, and Xinlei Chen. Scheduling uav swarm with attention-based graph reinforcement learning for ground-to-ai heterogeneous data communication. In Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, UbiComp/ISWC '23 Adjunct, page 670-675, New York, NY, USA, 2023. Association for Computing Machinery.
[37] Mauricio GC Resende, Rafael Martí, Micael Gallego, and Abraham Duarte. Grasp and path relinking for the max-min diversity problem. Computers & Operations Research, 37(3):498- 508, 2010.
[38] Ciyu Ruan, Zihang Gong, Ruishan Guo, Jingao Xu, and Xinlei Chen. Edmamba: Rethinking efficient event denoising with spatiotemporal decoupled ssms, 2025.
[39] Ciyu Ruan, Ruishan Guo, Zihang Gong, Jingao Xu, Wenhan Yang, and Xinlei Chen. Pre-mamba: 4d state space model for ultra-high-frequen event camera deraining, 2025.
[40] Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
[41] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[43] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- Language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[44] Jiang Wu, Sichao Wu, Yinsong Ma, Guangyuan Yu, Haoyuan Xu, Lifang Zheng, and Jingliang Duan. Monitorvim:a vision language framework for safety violation detection in mining operations, 2025.
[45] Penghao Wu and Saining Xie. V*; Guided visual search as core mechanism in multimodal 1lms, 2023.
[46] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajic Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, al. Pyramiddrop: Accelerating your large lisiunelanguage models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410. 17247, 2024.
[47] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models, arXiv preprint 2rX 2024.
13

---

[48] Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models. arXiv preprint arXiv:2412.00447, 2024.
[49] Li Yifan, Du Yifan, Zhou Kun, Wang Jinpeng, Zhao Wayne-Xin, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[50] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR, 2024.
[51] Sojeong Yun and Youn-kyung Lim. What if smart homes could see our homes?: Exploring diy smart home building experiences with vlm-based camera sensors. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI '25, New York, NY, USA, 2025. Association for Computing Machinery.
[52] Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen, How to enable IIm with 3d capacity? survey of spatial reasoning in Ilm, 2025.
[53] Shaolei Zhang, Qingkai Fang. Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.0388 2025.
[54] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token, 2025.
[55] Weichen Zhang, Chen Gao, Shiquan Yu, Ruiying Peng, Baining Zhao, Qian Zhang, Jinqiang Cui, Xinlei Chen, and Yong Li. Citynavagent: Aerial vision-and-land lnn-lagg navigation with hierarchical semantic planning and global memory, 2025.
[56] Xinyuan Zhang, Yonglin Tian, Fei Lin, Yue Liu, Jing Ma, Kornélia Sára Szatmáry, and Fei- Yui Wang. Logisticsvln: Vision-language navigation for low-altitude terminal delivery based on agentic uavs, 2025.
[57] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Sparsevim: Visual token sparsification for efficient vision-language model inference, 2025.
[58] Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, and Yong Li. Urbanvideo-bench: Benchmarking vision-language models on embodied intelligence with video data in urban spaces, 2025.
[59] Baining Zhao, Ziyou Wang, Jianjie Fang, Chen Gao, Fanhang Man, Jinqiang Cui, Xin Wang, Xinlei Chen, Yong Li, and Wenwu Zhu. Embodied-r: Collaborative framework for activating embodied spatial reasoning in foundation models via reinforcement learning, 2025.
[60] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging ilm-as-a-judge with mt- methbench and chatbot arena, 2023.
[61] Yue Zheng, Yuhao Chen, Bin Qian, Xiufang Shi, Yuanchao Shu, and Jiming Chen, A review on edge large language models: Design, execution, and applications, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Sec. for the main claims and Sec. for the detailed contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA -answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Sec. for the discussion on the limitations
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
See Sec. aand Sec. for assumptions and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yen Yes]
Justification: We present our experiment results in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide our code and data at the anonymous link in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We present our experiment settings and implementation details in Sec. 5 and Appendix 77.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the the statistical significance of the experiments Sec. 5.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We introduce the compute resources need for our work in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discuss broader impacts in Sec Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe the safeguards in Sec.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited and cited licenses of existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We provided proper documents our released code and data.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
This study is based on vision- lsionnlanguage large models, and the use of such models is discussed detail Sec. S
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy (https://net) /ttp://///e.p....eee.. wwteerrppssccooooeeeeeeee /t//oonferenncs//2225 for what should or should be described.
21

---

Appendix
7.1 Key and Value of LVLMs
Following previous works token quantization KIVI 31 LVLMs, the results are shown below:
visualize the K2 and V of different
4
10
2
5
100 Column
100 Column
0
0
200 400
50
0
200
50
To
600 Token
Token 600
400
(a) LLaVA key
(b) LLaVA value
Figure 7: Visualization of key and value of LLaVA-v1.5
4
10
2
5
0
0
Column
100
0
00
100 Column
10000
50
1000
50
Token 2000 0
Token 2000 0
(a) VL. key
(b) LLaVA-v1.6 value
Figure 8: Visualization of key and value of LLa
20
3
2
10
1
50 100
50 100 Column
0
0
Column
250 To Token 750
250 500
500
7 To
750
(a) Qwen2.5-vl key
(b) Qwen2.5-vl value
Figure 9: Visualization of key and value of Qwen2.5-vl
22

---

7.2 Top-k Importance Image Token Received Attention Ratio
We calculate the ratio between the attention scores received by the top-k most text-attended image tokens and the total attention scores received by all image tokens:
Layer
Layer 8
0.7
Jchendtn 6.-
73% to DEPS
100% Tpp
50%
50.5%% %
Baak Image Teeen
mage Toen
Layer 4 Attention Ratio
(b) Layer Attention Ratio
Figure 10: Visualization Top-k Importance Image Token Received Attention Ratio
Layer32
Laver 16
1.0
244
Atbendon
19 15
29h 37h
70t ta 18th
10% 22.5% m230%
15% op 2008
(a) Layer 12 Attention Ratio
(b) Layer Attention Ratio
Figure 11: Visualization Top-k Importance Image Token Received Attention Ratio
7.3 Experiment Settings
For LLaVA-v1.5-1 5-BB, LLaVA-v1.5 5-13B, and LLaVA-v1. ---.V, divide the pruning process into five stages based on the image token handling pipeline described the Appendix. In each stage, except for the last one, retain 50% of the tokens from the previous stage. In the final stage, all tokens are discarded to maximize inference speed. For Qwen2. -eV,, since its image token processing be clearly divided into stages, retain 25% of the tokens in the fourth stage and 12.5% in the final stage to preserve model performance. The À used for different models are shown below:
Table 7: À settings in different models
Model
À
llava-v1.5-70
(0.6,0.8.1.0)
llava-v 1.5-33b (0.6,0.8,1.0)
llava-v1.6-14
(0.4,0.7.1100)
qwen-2.5-vl-7b (0.2,0.5,088,...0))
7.4 Calculation of model gain
Since evaluation metrics vary across tasks and the difficulty levels differ significantly. it is not reasonable present all task results directly in unified format. For example, the original L.aVV- v1.5 model scores 1510 the MME benchmark but only 62 on GQA. To address this, we define model gain metric as:
Prunedscore
Gain Normalize
(12)
Original
23

---

7.5 Visualization of token selection under different pruning strategies
Is there cat in the image?
Layer 8
Layer 16
Our method
Attn Based method
Diversity Based method
Figure 12: Visualization of Image Token Selection Across Different Methods
7.6 Ablation Study on Calibration Set
According to Equation 2 in the paper, in multimodal large language models, M denotes the causal mask, which constrains each token to attend only to preceding tokens. In our input format, image tokens always precede the text prompts (i.e., the input follows the structure: prefix + image + question). As result, the model processes the image tokens before it receives the specific question which is unrelated to the input question. To validate our hypothesis, we posed different types of questions on the same image. We then conducted the experiment presented in Figure computed the number of image tokens whose cosine similarity between adjacent layers falls below 0.93. The resulting trends shown follows:
4 using L.L We
Table Ablation
Calibration Set Size.
Set Size
layerl
layer5
layer9
layer13
layer17
layer21
layer25
64
0
325
141
155
45
128
325
141
155
45
256
0
325
141
155
45
We observe that varying the question type for the same image does not lead significant differences in the results. In the following analysis, we investigate how image content and the size of the calibration set affect our method. We then repeated the experiment shown in Figure 4 uuiin calibration set sizes of 64, 128, and 256. Specifically, we computed the number of image tokens whose cosine similarity between adjacent layers falls below 0.93 using LLaVA-v1.5. The results are presented below:
Table
Ablation on Calibration Set Size.
Set Size
layerl
layer5
layer9
layer13
layer17
layer21
layer25
64
325
141
155
45
128
0
350
166
127
36
5
256
332
174
145
32
7.7 Ablation Study on Distance
To evaluate the impact of this choice, we conducted an ablation study comparing two different distance metrics using 1... The results summarized in the table below:

---

Table 10: Ablation Study on Distance
Method
MME
MMB
POPE
GQA
SQA
Manhattan
1497.0
63.4
85.6
59.1
69.1
Euclidean
1506.0
64.0
85.6
58.9
69.3
25


Reviews:
[{"review_id": "od7cdnkHO7", "reviewer": "Reviewer_Gsmq", "summary": "This paper proposes Balanced Token Pruning (BTP), a novel plug-and-play method for accelerating large vision-language models (LVLMs) by reducing visual token redundancy. Unlike prior methods that focus on either local (layer-wise) or global (multi-layer) optimization, BTP introduces a local-global pruning objective that balances preserving immediate output fidelity with maintaining downstream feature diversity. BTP is extensively evaluated on several SOTA LVLMs (e.g., LLaVA-7B/13B, Qwen-VL), achieving significant reductions in token count, latency, and memory, while maintaining ~98% of original performance across multiple benchmarks.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths.\n\n- The local-global pruning objective is conceptually well-motivated and addresses a clear limitation in previous token pruning strategies. The balance factor $\\lambda$ and stage-wise optimization strategy are intuitive and theoretically grounded.\n- Across multiple models and datasets (GQA, MME, POPE, SQA, MM-VeT), BTP consistently outperforms state-of-the-art approaches like FastV, DivPrune, and VTW in both performance retention and efficiency.\n- The authors show that BTP is effective across different model architectures and scales (LLaVA-7B/13B, Qwen-VL), increasing its impact and generalization potential.\n\nWeaknesses.\n\n- The method relies on a calibration set to determine pruning layers and balance factors. While the authors state that the same calibration set is reused across experiments, further discussion is needed on how sensitive the method is to the size and content of this set.\n- It is not clear how Figure 2 is depicted, e.g., which data and features are used to measure similarity? It would be good to provide more details.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We appreciate the opportunity to address the concerns you raised. Below, we provide our responses to the weaknesses:\n\n---\n\n**W1**: The method relies on a calibration set to determine pruning layers and balance factors. How sensitive the method is to the size and content of this set?  \n**R1**: Thank you for your constructive comment. In our approach, the selection of pruned layers is guided by the calibration set. The trade-off parameter $ \\lambda $ is a fixed hyperparameter and is not determined by the calibration set. Our method is insensitive to the content and size of the calibration set.  \n\n**Insensitive to the text content**: According to Equation (2) in the paper, in multimodal large language models, *M* denotes the causal mask, which constrains each token to attend only to preceding tokens. In our input format, image tokens always precede the text prompts (i.e., the input follows the structure: prefix + image + question). As a result, the model processes the image tokens before it receives the specific question which is unrelated to the input question. To validate our hypothesis, we posed different types of questions on the same image. We then conducted the experiment presented in Figure 4 using LLaVA-v1.5.  We computed the number of image tokens whose cosine similarity between adjacent layers falls below 0.93. The resulting trends are shown as follows:  \n| Question_type | layer1 | layer3 | layer5 | layer7 | layer9 | layer11 | layer13 | layer15 | layer17 | layer19 | layer21 | layer23 | layer25 | layer27 | layer29 |\n|------|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n|  Caption    |   0   |    0  |   0   |   151   |  325    |   359   |   141   |   136   |   155    |  38     |   45    |    11   |    1   |   0    |    3   |\n|   POPE   |   0   |    0  |   0   |   151   |  325    |   359   |   141   |   136   |   155    |  38     |   45    |    11   |    1   |   0    |    3   |\n|   SpatialVQA   | 0   |    0  |   0   |   151   |  325    |   359   |   141   |   136   |   155    |  38     |   45    |    11   |    1   |   0    |    3   |  \n\nWe observe that varying the question type for the same image does not lead to significant differences in the results. In the following analysis, we investigate how image content and the size of the calibration set affect our method.  \n\n**Insensitive to the image content and size**: We expanded the calibration set by incorporating images from multiple datasets, including GQA, V*Bench[1], and SQA. We then repeated the experiment shown in Figure 4 using calibration set sizes of 64, 128, and 256. Specifically, we computed the number of image tokens whose cosine similarity between adjacent layers falls below 0.93 using LLaVA-v1.5. The results are presented below:  \n| Size | layer1 | layer3 | layer5 | layer7 | layer9 | layer11 | layer13 | layer15 | layer17 | layer19 | layer21 | layer23 | layer25 | layer27 | layer29 |\n|------|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n|  64    |   0   |    0  |   0   |   151   |  325    |   359   |   141   |   136   |   155    |  38     |   45    |    11   |    1   |   0    |    3   |\n|   128   |   0   |    0  |   0   |   164   |  350   |   385  |   166  |   139  |   127    |  65     |   36    |    16   |    5   |   0    |    2   |\n|   256   | 0   |    0  |   0   |   132  |  332  |   368 |   174  |   155   |   145    |    61   |   32    |    11  |    3   |   0    |    1  |  \n\nWe observe that the patterns remain consistent across different image contents and calibration set sizes, indicating that these factors do not affect the behavior of our method. We will include this additional analysis in the Ablation Study section (Section 5.3) of the revised paper.\n\n---\n\n**W2**: It is not clear how Figure 2 is depicted, e.g., which data and features are used to measure similarity? It would be good to provide more details.  \n**R2**: Thank you for your constructive feedback. We conducted experiments on the captioning task. In Figure 2, our goal is to investigate the impact of pruning during the prefilling stage on the model’s output. In multimodal large language models, the first token generated during the decoding stage is directly influenced by the final token from the prefilling stage. This final token is also the only point during prefilling where the model produces an output. Therefore, we evaluate the effect of different pruning strategies by analyzing their impact on the hidden state of the last token in the prefilling stage.  \n\n**Effect of Different Pruning Strategies on the Full Output**: In the captioning task, multimodal large language models generate not just a single token, but a sequence of tokens in an autoregressive manner. Therefore, we further investigated how different pruning strategies affect tokens at various positions in the output sequence. Specifically, we analyzed the impact on the 1st, 4th, and 20th tokens. We use llava-v1.5-7b to conduct this experiment. The results are summarized below:  \n| Method-Position | Layer4 | Layer8 | Layer12 | Layer16 | Layer20 | Layer24 |\n|------|:------:|:------:|:------:|:------:|:------:|:------:|\n|   Attn(1st)   |   1   |    1  |  **0.996**    |   **0.994**   |  0.979    |   0.973   |\n|   Div(1st)   |    1  |    1  |  0.992    |   0.991   |    **0.985**  |   **0.982**   |\n|   Attn(4th)   |   1   |   **0.998**  |   **0.993**   |   **0.991**   |   0.954   |  0.949    |\n|   Div(4th)   |   1   |   0.995 |    0.989  |    0.989  |   **0.975**   |  **0.972**    |\n|   Attn(15th)   |   **0.369**   |  **0.311**  |  **0.375**   |   0.279   |  0.303 |   0.352   |\n|   Div(15th)  |   0.299   |   0.204   |  0.274   |   **0.312**  |  **0.423**   |    **0.496**  |\n|   Attn(20th)   |   **0.297**   |  0.213  |  0.141   |   0.161   |  0.168 |   0.166   |\n|   Div(20th)  |   0.292   |   **0.255**   |  **0.162**    |   **0.171**   |  **0.169**    |    **0.235**  |\n\nWe observe that tokens appearing later in the output sequence are influenced by previously generated tokens, leading to a significant drop in similarity with the corresponding tokens from the original (non-pruned) output. Moreover, **tokens at different positions exhibit similar patterns under both pruning strategies**: attention based methods focus on local goal and diversity based methods focus on global goal.These findings further support the validity of our observations. We will include this discussion in the Introduction section of the revised manuscript.  \n\n[1] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs."}, {"review_id": "yfKqGWcusp", "reviewer": "Reviewer_W7Hn", "summary": "This paper aims to address the computational overhead in Large Vision-Language Models (LVLMs) caused by the large number of image tokens. The authors begin by empirically analyzing existing token pruning methods, categorizing them into attention-based and diversity-based approaches. They find that attention-based methods excel at optimizing the output of the current layer (a local objective) but neglect the impact on subsequent layers (a global objective). Conversely, diversity-based methods better preserve global information but sacrifice local output consistency.\n\nTo address this local-global trade-off, the authors propose Balanced Token Pruning (BTP), a plug-and-play method for visual token pruning. The core idea of BTP is to divide the pruning process into multiple stages and dynamically adjust the weight between local and global objectives. Specifically, in the early layers of the model, the method emphasizes a diversity-based criterion to preserve rich information that has a significant impact on subsequent layers. In the deeper layers, it shifts focus to an attention-based criterion to ensure the accuracy of the immediate layer's output. Additionally, the method incorporates several supporting techniques: 1) using a small calibration set to automatically select the pruning layers; 2) an attention rebalancing operation to correct for positional biases; and 3) a spatial initialization strategy to accelerate diversity calculations.", "questions": "Refer to weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1.  The authors clearly point out the inherent contradiction between \"local optimization\" and \"global impact\" through an in-depth analysis of existing methods. The visualization analysis of Figures 1 and 2 intuitively shows the difference in behavior of different pruning strategies at different depths of the model, providing a solid foundation and convincing motivation for the \"balance\" idea proposed in the paper.\n\n2. The core idea of ​​BTP - dynamically balancing local (attention) and global (diversity) goals according to the network depth where the pruning is located - is very novel and intuitive. Rather than simply mixing the two methods, it proposes a principled, staged strategy, that is, protecting information diversity at shallow levels to benefit the global, and focusing on task relevance at deep levels to ensure local. This idea is very elegant.\n\n3. Exhaustive Experimental Validation: The experimental part of the paper is very solid. The author tested on multiple models (LLaVA-v1.5/v1.6, Qwen2.5-VL) and multiple sizes (7B, 13B), covering a wide range of visual understanding benchmarks. The ablation study is very comprehensive and systematically verifies the effectiveness of each component such as balancing factor, rebalancing module, spatial initialization and layering strategy.\n\n**Weaknesses**\n1. The paper makes completely unclear statements when describing how the core balance factor $\\lambda$ changes with the depth of the network.\n- In Section 5.3, the author describes: \"In the early layers, we use a larger $\\lambda$ value to focus more on global information, while in the deeper layers, we use a smaller lambda to emphasize local details.\"\nHowever, in the title of Figure 6 on the same page, the description is: \"Therefore, we use smaller $\\lambda$ values ​​(0.5-0.7) in the shallow layers and larger values ​​(0.8-1.0) in the deeper layers.\" According to Formula 7, $\\lambda$ is the weight of the attention term, so a larger $\\lambda$ means more attention to local targets (attention). These two descriptions are completely contradictory. This serious contradiction makes me doubt the implementation details of the paper and the rigor of the author.\n\n2. Heuristic Treatment of \"Global Objective\": The paper formalizes the \"global objective\" as the sum of the output errors of all pruned layers (Formula 6), and proposes to use \"diversity\" as its proxy. This connection is intuitive, but lacks a formal proof. The final optimized objective function (Formula 7) is a weighted sum of attention and diversity scores, rather than a direct optimization of Formula 6. This jump from the true objective to the proxy objective is based on heuristics and lacks stronger theoretical support.\n\n3. Although the ablation experiment in Figure 6 shows the impact of a fixed $\\lambda$ value on performance, the paper does not detail how the $\\lambda$ schedule across different stages in the final experiment is determined. For example, what is the specific strategy for transitioning from 0.5/0.7 (shallow layers) to 0.8/1.0 (deep layers)? Is this strategy fixed or does it need to be adjusted for different models? This makes the reproducibility of the method and its sensitivity to hyperparameters unclear.", "was_revised": true, "final_justification": "I thank the authors for their detailed and timely response. After reading the authors' feedback, I believe most of my concerns are addressed, and I decide to keep my original rating, leaning toward acceptance.", "rebuttal": "We sincerely thank you for your careful review and insightful feedback. We appreciate the opportunity to address the concerns you raised. Below, we provide our responses to the weaknesses:\n\n---\n\n**W1**: The paper makes completely unclear statements when describing how the core balance factor $ \\lambda $ changes with the depth of the network.  \n**R1**: Thank you for pointing out the unclear description of the balance factor $ \\lambda $ in Section 5.3. We acknowledge a typographical error, which incorrectly states: “in the early layers, we use a larger $\\lambda$ value to focus more on global information, while in the deeper layers, we use a smaller lambda to emphasize local details.” In our method, $ \\lambda $ increases with network depth. This strategy is correctly reflected in Formula (7), Figure 3, and Section 4.2, and has shown strong empirical performance. We will revise the text for consistency and thank you again for your helpful feedback.\n\n---\n\n**W2**: Heuristic Treatment of \"Global Objective\": The final optimized objective function (Formula (7)) is a weighted sum of attention and diversity scores, rather than a direct optimization of Formula (6). This jump from the true objective to the proxy objective is based on heuristics and lacks stronger theoretical support.  \n**R2**: Thank you for pointing out this issue which is critical for improving the quality of our paper. We will provide both theoretical and empirical justifications to explain why optimizing Formula (7) effectively leads to the optimization of Formula (6). Suppose token pruning is performed at layers $l_1$ < $l_2$ < $l_3$. Since the tokens retained in deeper layers are a subset of those retained in earlier layers, the preserved token sets satisfy the relationship: $ P_3 \\subseteq P_2 \\subseteq P_1 $.  \n\n**Relation between attention scores and global objective**: The global objective aims to minimize the discrepancy between the hidden states of pruned layers $ l_1, l_2, l_3 $ and those from the original (unpruned) model. According to Formula (1), the hidden state at each pruned layer primarily depends on the outputs of both the attention and MLP modules. Since the MLP parameters are fixed and its input includes the output of the attention module, **we attribute the changes in the pruned layer's hidden states primarily to variations in the attention module's output**. According to Formula (2), the attention module's output is affected by attention score and $V_l$. In Appendix A.1, we observe that different image tokens exhibit similar magnitudes in their value vectors (V). Therefore, maximizing the sum of attention scores contributes to improving the local output similarity, as reflected in the first term of Formula (7). This helps reduce the deviation between the outputs of shadow layers ($l_1$) and the original outputs, thereby contributing to the optimization of the global objective.  \n\n**Relation between diversity scores and global objective**: As shown in Figure 1, different layers attend to different regions of the image tokens—that is, the regions with high attention scores vary across layers. Therefore, selecting only high-attention tokens in the shallow layers may lead to local output deviations in subsequent layers. As the diversity objective is strengthened, the overlap between the tokens retained in deeper layers $l_2,l_3$ and the optimal retained token set $ P^* $ increases: $|P^* \\cap P^{attn}| < |P^* \\cap P^{div}|$. This helps reduce the deviation between the outputs of deeper layers ($l_2,l_3$) and the original outputs, thereby contributing to the optimization of the global objective.  \n\nWe also observe that the attention from output tokens to image tokens is sparse. To quantify this, we compute the proportion of total attention assigned to image tokens that is captured by the top-k% highest-attention image tokens:  \n| Layer | top&nbsp;1% | top&nbsp;5% | top&nbsp;10% | top&nbsp;25% | top&nbsp;50% | top&nbsp;75% | top&nbsp;100% |\n|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n|   4   |   0.62   |   0.76   |  0.81    |   0.89   |   0.95   |   0.98   |   1   |\n|   8   |   0.38   |    0.60  |   0.71   |   0.83   |   0.93   |    0.98  |   1   |\n|   12   |   0.32   |   0.56   |   0.69   |    0.84  |   0.93   |   0.98   |   1   |\n|   16   |   0.21   |   0.59   |   0.76   |    0.90  |   0.96   |  0.98    |    1  |\n\nWe can see that as the number of retained tokens increases, the gain from optimizing the first term of Formula (7) diminishes, while the benefit of promoting diversity to support subsequent layers increases. In summary, we believe that optimizing Equation (7) serves as a reasonable approximation to optimizing Equation (6). We will include this discussion in Section 4.2 of the revised paper. A more rigorous theoretical justification will be explored in future work.  \n\n---\n\n**W3**: How the schedule across different stages in the final experiment is determined? Is this strategy fixed or does it need to be adjusted for different models? This makes the reproducibility of the method and its sensitivity to hyperparameters unclear.  \n**R3**: Thank you for your constructive comment. In our experiments, the $\\lambda$ parameter was manually selected. Its value varies across models due to differences in decoding depth and the patterns of image token variation. However, for a given model, the $\\lambda$ value remains fixed across different benchmarks. The table below summarizes the $\\lambda$ values used for each model at pruning layers:  \n| Model | $\\lambda$ of different layers |\n|:------:|:------:|\n|   llava-v1.5-7b   |   (0.6,0.8,1.0)  |\n|   llava-v1.5-13b   |   (0.6,0.8,1.0)   |\n|   llava-v1.6-7b  |   (0.4,0.7,1.0)   |\n|   Qwen-2.5vl-7b   |   (0.2,0.5,0.8,1.0)   |\n\nWe will include this information in the experimental setup section of Appendix A.2. Once the pruned layers are determined, the $\\lambda$ parameter is also fixed. We included preliminary ablation results in Figure 6. To further investigate the sensitivity of our method to the choice of $\\lambda$, we conduct an ablation study on LLaVA-v1.5-7B by varying $\\lambda$ at different stages. The $\\lambda$ values are used in pruning layers $(l_1,l_2,l_3)$. The results are shown below:  \n**Experiments in the shallow layer**  \n| $\\lambda$ | MMB| MME | GQA | POPE| SQA |\n|------|:------:|:------:|:------:|:------:|:------:|\n|   (0.3,0.8,1.0)   |   **63.4**   |   1497   |   **58.9**   |  **85.3**    |   69.1   |\n|   (0.5,0.8,1.0)   |   63.1   |   **1500**   |   58.3   |  84.7    |    **69.6**  |\n|   (0.7,0.8,1.0)   |  _62.7_    |   _1490_   |   _57.9_   |  _84.5_    |   _68.6_   | \n\n**Experiments in the middle layer**\n| $\\lambda$ | MMB| MME | GQA | POPE| SQA |\n|------|:------:|:------:|:------:|:------:|:------:|\n|   (0.5,0.6,1.0)   |   **63.2**   |   1486   |   **59.0**   |   **85.2**   |   **69.4**   |\n|   (0.5,0.8,1.0)   |   63.1   |   **1500**   |   58.3   |  84.7    |    69.6  |\n|   (0.5,1.0,1.0)   |   _59.3_   |   _1428_  |  _55.8_    |   _82.9_   |  _54.4_    |  \n\n**Experiments in the deep layer**\n| $\\lambda$ | MMB| MME | GQA | POPE| SQA |\n|------|:------:|:------:|:------:|:------:|:------:|\n|   (0.3,0.8,1.0)   |   **63.4**   |   **1497**   |   **58.9**   |  **85.3**    |   **69.1**   |\n|   (0.3,0.6,0.8)   |   63.2   |   1491   |   58.5   |  84.3    |    68.9  |  \n\nWe observe that focusing too early on attention-based pruning in shallow or middle layers leads to performance degradation. Similarly, emphasizing diversity-based pruning in the deeper layers also results in a drop in performance. We will include this discussion in the ablation study section of the revised paper."}, {"review_id": "gWVhOa309u", "reviewer": "Reviewer_wCTc", "summary": "This paper introduces Balanced Token Pruning (BTP), a plug-and-play method to reduce the computational cost of  LVLMs by more effectively pruning image tokens. Unlike prior approaches that rely solely on local attention or token diversity, BTP balances both local and global impacts of token pruning across model layers using a multi-stage strategy informed by a small calibration set. Experiments across several LVLMs show that BTP consistently improves efficiency with minimal performance loss.", "questions": "- Line 41 - 42: Is this phenomenon of different image tokens being attended by the text tokens consistently observed across the dataset? It seems counter-intuitive to me. At least in vision transformers, we do observe the same tokens being attended to by other tokens throughout the layers.\n- Lines 44 - 56: Why was the hidden state of only last word considered for this experiment? What dataset and model was used for this experiment? Overall, I think it would be insightful to add some more explanation (if any) on the observations shown in figure 1 and 2. But this idea of creating a new local-global objective is quite interesting.\n- Having different number of tokens pruned for different input sample breaks the regularity in processing batched input. Does it not deteriorate the throughput?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "- The paper gives a very interesting insight on how token pruning affects the subsequent blocks which is often overlooked in the existing approaches.\n- The approach is convincing, in that, a combination of local-global consideration for pruning would be more optimal than the exisiting pruning approaches.\n- BTP has been evalulated extensively and compared with many recent token pruning approaches in VLMs.\n- Some empirical observations, such as token attention patterns across layers, are counter-intuitive and insufficiently explained, with missing experimental details and unclear generality across the dataset.\n- The method’s use of input-dependent token pruning may affect batching efficiency and inference throughput, but this practical concern is not addressed.", "was_revised": true, "final_justification": "The authors have provided a rebuttal that directly addresses the key suggestion: clarifying CLAdapter's effectiveness across tasks with varying amounts of training data. \nGiven the technical soundness, novelty, results, and other reviewers’ comments, I would like to raise my rating.", "rebuttal": "Thank you for your thorough review and for the positive feedback on our work. We appreciate the opportunity to address your questions. Below, we provide detailed responses to the concerns you raised.\n\n---  \n\n**Q1**: Line 41 - 42: Is this phenomenon of different image tokens being attended by the text tokens consistently observed across the dataset? It seems counter-intuitive to me. At least in vision transformers, we do observe the same tokens being attended to by other tokens throughout the layers.  \n**R1**: We are very pleased to have the opportunity to discuss this phenomenon with the reviewer. Yes, we have observed the pattern shown in Figure 1 across different types of datasets, including POPE, GQA, and COCO.  \n\nWe believe that the reviewer’s observation and our findings are not contradictory. As shown in Figure 1, certain image tokens are indeed attended by multiple layers. Our analysis reveals that the overlap of highly attended image tokens is high between adjacent layers, but this overlap significantly decreases between non-adjacent layers. In the table below, we report the overlap of the top 144 most attended image tokens across different layers:  \n| layer1-layer2 | layer2-layer3 | layer8-layer9 | layer10-layer11 | layer16-layer17 | layer18-layer19 | layer24-layer25 | layer26-layer27 |\n|----|----|----|----|----|----|----|----|\n|  0.604  |  0.784  |  0.881  |  0.875  |  0.895  |  0.875  |  0.861  |  0.854  |  \n\n| layer1-layer8 | layer2-layer10 | layer8-layer16 | layer10-layer18 | layer16-layer24 | layer18-layer26 |\n|----|----|----|----|----|----|\n|  0.486  |  0.576  |  0.687  |  0.694  |  0.555  |  0.641  |  \n\nWe observe that the overlap of highly attended image tokens is relatively high between adjacent layers, while it significantly decreases between layers that are farther apart. This suggests that adjacent layers tend to focus on similar regions of the input image, whereas distant layers shift their attention to different regions. Moreover, this trend becomes even more pronounced when computing the overlap using a smaller subset of top-attended image tokens.  \n\n---\n\n**Q2**: Lines 44 - 56: Why was the hidden state of only last word considered for this experiment? What dataset and model was used for this experiment? Overall, I think it would be insightful to add some more explanation (if any) on the observations shown in figure 1 and 2. But this idea of creating a new local-global objective is quite interesting.  \n**R2**: We use coco dataset to conduct this experiment with llava-v1.5-7b. We have also observed similar patterns across different datasets and model architectures, suggesting that this phenomenon is consistent and not specific to a particular setting. As shown in Figure 1, diversity-based pruning tends to preserve image tokens that are important for deeper layers, while attention-based pruning primarily retains tokens that are important for the current layer. This discrepancy leads to the results observed in Figure 2.  \n\nIn Figure 2, our goal is to investigate the impact of pruning during the prefilling stage on the model’s output. Due to the autoregressive nature of large language models, the last token in the prefilling stage is decoded as the first token in the decoding stage. It is also the only “model output” that we can access during the prefilling phase. Therefore, we select the hidden state of the last token in the prefilling stage for analysis.  \n\n**Effect of Different Pruning Strategies on the Full Output**: However, in captioning tasks, the model generates multiple tokens in an autoregressive manner. Therefore, we further investigate how output tokens at different positions are affected by different pruning strategies. Specifically, we analyzed the impact on the 1st, 4th, and 20th generated tokens with llava-v1.5-7b. The results are summarized below:  \n| Method-Position | Layer4 | Layer8 | Layer12 | Layer16 | Layer20 | Layer24 |\n|------|:------:|:------:|:------:|:------:|:------:|:------:|\n|   Attn(1st)   |   1   |    1  |  **0.996**    |   **0.994**   |  0.979    |   0.973   |\n|   Div(1st)   |    1  |    1  |  0.992    |   0.991   |    **0.985**  |   **0.982**   |\n|   Attn(4th)   |   1   |   **0.998**  |   **0.993**   |   **0.991**   |   0.954   |  0.949    |\n|   Div(4th)   |   1   |   0.995 |    0.989  |    0.989  |   **0.975**   |  **0.972**    |\n|   Attn(15th)   |   **0.369**   |  **0.311**  |  **0.375**   |   0.279   |  0.303 |   0.352   |\n|   Div(15th)  |   0.299   |   0.204   |  0.274   |   **0.312**  |  **0.423**   |    **0.496**  |\n|   Attn(20th)   |   **0.297**   |  0.213  |  0.141   |   0.161   |  0.168 |   0.166   |\n|   Div(20th)  |   0.292   |   **0.255**   |  **0.162**    |   **0.171**   |  **0.169**    |    **0.235**  | \n\nWe observe that tokens generated at later positions are more influenced by previously autoregressively generated tokens, resulting in lower similarity to their original hidden states. However, across all positions, the impact trends of the two pruning strategies remain consistent: attention-based pruning preserves local information, while diversity-based pruning better maintains global information.  \n**Further Explanation of Figure 1**: We attempt to provide a deeper explanation of the phenomenon shown in Figure 1. Since the multimodal large language model cannot access future questions during image processing due to the causal mask, the image token processing pipeline remains the same across different types of questions. This implies that different layers may encode different aspects of image information. As illustrated in Figure 4, image tokens gradually align to the language space in deeper layers. Moreover, MLLMs tend to assign uniform attention scores to all image tokens in early layers, and only focus on semantically relevant different regions in deeper layers. We plan to further explore this interesting phenomenon in future work.  \n\n---\n\n**Q3**: Having different number of tokens pruned for different input sample breaks the regularity in processing batched input. Does it not deteriorate the throughput?  \n**R3**: Thank you for the insightful question. Our method supports batch inference for models that encode images into a fixed number of image tokens. This is because, under fixed pruning layers and pruning ratios, the number of retained tokens before each layer remains consistent across samples, avoiding dimensional mismatches during inference.  \n\nHowever, for models such as LLaVA-v1.6 and Qwen-2.5-VL that employ dynamic resolution and encode different images into varying numbers of image tokens, our approach does pose challenges for batch inference. This presents an interesting trade-off. For example, one potential solution is to determine the number of image tokens to discard based on the sample with the fewest tokens in a batch, thereby ensuring input dimensional consistency across the batch. We plan to explore this further in future work, investigating whether pruning and token reuse strategies can be adapted to support efficient batch inference under such dynamic settings."}, {"review_id": "RpcaNWZddJ", "reviewer": "Reviewer_jfqF", "summary": "This paper proposes a novel Balanced Token Pruning (BTP) method to reduce the computational cost of large Vision-Language models. Through empirical analysis, the authors show that attention-based pruning mainly focuses on locality, while diversity-based pruning considers the impact on subsequent layers. BTP combines the strengths of both approaches by applying a diversity-based objective in the early layers to select semantically diverse tokens, and an attention-based objective in the later layers to locally optimize each layer and minimize output error. To support this framework, the paper introduces an attention rebalance module and a spatial initialization strategy. It also proposes a layer selection mechanism using a small calibration image set. Experiments applying the proposed method to various models (LLaVA-1.5-7B, LLaVA-1.5-13B, LLaVA-1.6-7B, and Qwen2.5-VL-7B) show that BTP achieves better performance not only compared to attention-based pruning, but also to diversity-based pruning methods. Ablation studies further validate the effectiveness of each component.", "questions": "**1. Spatial Initialization**\n\nCould you explain why you chose Manhattan distance instead of Euclidean distance? For example, given a source token at (2, 2), both (6, 6) and (2, 10) have the same Manhattan distance, but different Euclidean distances. In many spatial contexts, Euclidean distance may provide a more accurate measure of proximity.\n\n\n**2. Pruning Layer Selection**\n\nIn Qwen2.5-VL, significant changes in image token representations are observed in layers 10 and 24, yet relatively fewer image tokens are selected in those layers. This appears inconsistent with the assumption that image tokens receive more attention in layers with significant feature changes. LLaVA-v1.5 shows a similar pattern—although layers 8–12 exhibit large representational shifts, more image tokens are selected in layers 17, 19, and 20. Further clarification would be helpful.\n\n**3. Attention Optimization**\n\nThe method uses the last token in the input prompt ($X_T$) to compute importance scores. It is unclear whether relying on a single token is sufficient to capture the full contextual relevance of the prompt. Further explanation of this choice would improve the understanding of its effectiveness.\n\n**4. $\\lambda_i$ Values**\n\nThe $\\lambda_i$ values are critical hyperparameters that influence the performance of the proposed framework. In Section 5.3, the authors apply smaller lambda values (0.5–0.7) to shallow layers and larger ones (0.8–1.0) to deeper layers. However, it is not clearly explained how the $\\lambda_i$ values are determined for each layer. It would be beneficial to provide a more concrete rule or conduct an ablation study on the impact of $\\lambda_i$ settings.\n\n**5. Inference Latency and Spatial Initialization**\nIn Table 4, when the attention rebalance (RA) module is used, adding spatial initialization significantly improves inference latency. However, in cases where RA is not used, the latency difference with or without spatial initialization becomes marginal. An explanation for this discrepancy would be appreciated.\n\n\n**6. Appendix A.1**\n\nIn Section 4.1, the authors present the assumption that when $V_l$ values are similar, selecting image tokens with high importance scores can effectively reduce the difference in outputs before and after pruning. However, the only supporting evidence is a figure, and no explanation is provided. This lack of explanation makes the assumption difficult to evaluate.\n\n\n**7. Equation (1)**\n\nThe equation for $\\text{Atten}^{(l)}$ is not explicitly defined in the paper. Including a clear mathematical formulation would help readers better understand how attention scores are computed at each layer.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "***Strenghts***\n\n1. The paper is well-motivated, supported by empirical analysis comparing attention-based and diversity-based pruning.\n2. The proposed method demonstrates superior performance over both attention-based and diversity-based pruning across multiple large-scale VL models.\n\n***Weaknesses***\n\n1. Some experimental results are missing or not fully detailed, limiting the understanding of the method’s generalizability.\n2. Certain components lack sufficient explanation.", "was_revised": true, "final_justification": "While I greatly appreciate efforts and the additional insights provided, I believe that my current score remains appropriate based on my overall assessment of the work, and I have therefore chosen to maintain it.", "rebuttal": "Thank you for your insightful feedback. We appreciate the opportunity to address the concerns you raised. Below, we provide our responses to the problems:\n\n---  \n\n**Q1**: Spatial Initialization: Could you explain why you chose Manhattan distance instead of Euclidean distance?  \n**R1**: Thank you for your constructive feedback. We did not apply any special design when choosing the distance metric. To evaluate the impact of this choice, we conducted an ablation study comparing two different distance metrics using llava-v1.5. The results are summarized in the table below:  \n| Method | MME | MMB | POPE | GQA | SQA |\n|----|:----:|:----:|:----:|:----:|:----:|\n| Manhattan | 1497 | 63.4 | **85.6** | **59.1** | 69.1 |\n| Euclidean | **1506** | **64.0** | **85.6** | 58.9 | **69.3** |  \n\nWe found that Euclidean distance indeed performs better for spatial initialization. We will incorporate this finding into Section 5.3 of the revised paper and update our implementation to adopt Euclidean distance in the next version.  \n\n---  \n\n**Q2**: Pruning Layer Selection: In Qwen2.5-VL, significant changes in image token representations are observed in layers 10 and 24, yet relatively fewer image tokens are selected in those layers. This appears inconsistent with the assumption that image tokens receive more attention in layers with significant feature changes. LLaVA-v1.5 shows a similar pattern—although layers 8–12 exhibit large representational shifts, more image tokens are selected in layers 17, 19, and 20. Further clarification would be helpful.  \n**R2**: Thank you very much for your question—we appreciate the opportunity to clarify this point. Firstly, once the pruning layers are determined, our method retains more tokens in earlier layers than in deeper ones. As described in Section 4.2, the token set preserved in deeper layers is a subset of that from earlier layers. This design is consistent with prior work [1], and therefore, it is not the case that more image tokens are retained in deeper layers.  \n\nSecondly, our hypothesis in Section 4.3 is: *\"LVLMs tend to allocate more attention to image tokens in layers following those where the representations of image tokens **undergo** significant changes,\"* which is different from *\"image tokens receive more attention in layers with significant feature changes.\"*   \n\nTo illustrate, we only prune a selected subset of layers. The tokens retained in a pruned layer are propagated through subsequent layers. Our method favors diversity in early layers and attention-based selection in deeper layers. For example, in LLaVA-v1.5, we retain more tokens in layers 8–12 and fewer in layers 17, 19, and 20. If we have misunderstood your concern, we would be happy to engage in further discussion.  \n\n---  \n\n**Q3**: Attention Optimization: The method uses the last token in the input prompt $X_T$ to compute importance scores. It is unclear whether relying on a single token is sufficient to capture the full contextual relevance of the prompt. Further explanation of this choice would improve the understanding of its effectiveness.  \n**R3**: Thank you very much for your helpful suggestion. We agree that further exploring the computation of the importance score is valuable.  \n\nTo this end, we compared with two more approaches: (1) computing the importance score by averaging over all text tokens following the image tokens, and (2) the method used in [2], which first computes image-text similarity and then selects the most similar text tokens to compute the importance score. The results are shown in the table below:  \n| Method | MME | MMB | POPE | GQA | SQA |\n|----|:----:|:----:|:----:|:----:|:----:|\n| last-token(ours) | **1497** | **63.4** | **85.6** | **59.1** | 69.1 |\n| averaged-tokens | 1490 | 62.8 | 84.7 | 57.3 | 69.4 |\n| similarity-based | 1485 | 63.1 | 84.7 | 57.9 | **69.7** |  \n\nWe believe that the last token in the input prompt is a suitable choice for computing the importance score because it is typically decoded as the first output token during the decoding stage. This allows it to effectively capture the model's focus. We will include this discussion in the ablation study section of the revised paper.  \n\n---\n\n**Q4**: The values $\\lambda_i$ are critical hyperparameters that influence the performance of the proposed framework. It would be beneficial to provide a more concrete rule or conduct an ablation study on the impact of $\\lambda_i$ settings.  \n**R4**: Thank you for your constructive comment. In our experiments, the $\\lambda_i$ parameter was manually selected. For a given model, the $\\lambda_i$ value remains fixed across different benchmarks. We only set $\\lambda_i$ values at pruning layers. The table below summarizes the $\\lambda_i$ values used for each model at pruning layers:  \n| Model | $\\lambda$ of different layers |\n|:------:|:------:|\n|   llava-v1.5-7b   |   (0.6,0.8,1.0)  |\n|   llava-v1.5-13b   |   (0.6,0.8,1.0)   |\n|   llava-v1.6-7b  |   (0.4,0.7,1.0)   |\n|   Qwen-2.5vl-7b   |   (0.4,0.6,0.8,1.0)   |\n\nOnce the pruned layers are determined, the $\\lambda$ parameter is also fixed. To further investigate the sensitivity of our method to the choice of $\\lambda$, we conduct an ablation study on LLaVA-v1.5-7B by varying $\\lambda$ at different stages. The results are shown below:  \n**Experiments in the shallow layer**  \n| $\\lambda$ | MMB| MME | GQA | POPE| SQA |\n|------|:------:|:------:|:------:|:------:|:------:|\n|   (0.3,0.8,1.0)   |   **63.4**   |   1497   |   **58.9**   |  **85.3**    |   69.1   |\n|   (0.5,0.8,1.0)   |   63.1   |   **1500**   |   58.3   |  84.7    |    **69.6**  |\n|   (0.7,0.8,1.0)   |  _62.7_    |   _1490_   |   _57.9_   |  _84.5_    |   _68.6_   | \n\n**Experiments in the middle layer**\n| $\\lambda$ | MMB| MME | GQA | POPE| SQA |\n|------|:------:|:------:|:------:|:------:|:------:|\n|   (0.5,0.6,1.0)   |   **63.2**   |   1486   |   **59.0**   |   **85.2**   |   **69.4**   |\n|   (0.5,0.8,1.0)   |   63.1   |   **1500**   |   58.3   |  84.7    |    69.6  |\n|   (0.5,1.0,1.0)   |   _59.3_   |   _1428_  |  _55.8_    |   _82.9_   |  _54.4_    |  \n\n**Experiments in the deep layer**\n| $\\lambda$ | MMB| MME | GQA | POPE| SQA |\n|------|:------:|:------:|:------:|:------:|:------:|\n|   (0.3,0.8,1.0)   |   **63.4**   |   **1497**   |   **58.9**   |  **85.3**    |   **69.1**   |\n|   (0.3,0.6,0.8)   |   63.2   |   1491   |   58.5   |  84.3    |    68.9  |  \n\nWe found that using a smaller $\\lambda$ value in earlier layers encourages the pruning strategy to focus more on the global objective, while employing a larger $\\lambda$ value in deeper layers helps preserve local outputs. This observation further supports the effectiveness of our proposed method. We will include this discussion in the ablation study section of the revised paper.  \n\n---  \n\n**Q5**: Inference Latency and Spatial Initialization In Table 4, when the attention rebalance (RA) module is used, adding spatial initialization significantly improves inference latency. However, in cases where RA is not used, the latency difference with or without spatial initialization becomes marginal. An explanation for this discrepancy would be appreciated.  \n**R5**: We sincerely apologize for the confusion caused by the writing error. After reviewing our experiment logs and re-running the experiment across three datasets, we found that the averaged latency in the last row of Table 4 should be **0.231s**. We will correct this in the revised version of the paper. Thank you again for your careful review—your feedback has been very helpful to us.  \n\n---\n\n**Q6**: In Section 4.1, the authors present the assumption that when values are similar, selecting image tokens with high importance scores can effectively reduce the difference in outputs before and after pruning. This lack of explanation makes the assumption difficult to evaluate.  \n**R6**: We appreciate the opportunity to clarify this point. As shown in Equation (1), the hidden states at each pruned layer are primarily influenced by the outputs of both the attention and MLP modules. Since the MLP parameters remain fixed and its inputs include the outputs from the attention module, we attribute the majority of the observed changes in the hidden states to variations in the output of the attention module.  \n\nFurther, according to Equation (2), the attention output is determined by the attention scores and the value matrix $V_l$. As detailed in Appendix A.1, we observe that the value vectors across different image tokens exhibit similar magnitudes. Therefore, maximizing the sum of attention scores effectively enhances the similarity of local outputs, which aligns with the assumption.  \n\n---\n\n**Q7**: The equation for $Attn^l$ is not explicitly defined in the paper. Including a clear mathematical formulation would help readers better understand how attention scores are computed at each layer.  \n**R7**: Thank you very much for your suggestion. We denote $\\mathrm{Attn}^l$ as the attention module at layer $l$, with its formulation defined as:\n\n$$\\mathrm{Attn}^l = softmax({\\frac {Q_lK_l^T+M}{\\sqrt d_k}V_l}).$$  \n\nwhere M is the casual mask. We will include this definition in Section 3 Preliminaries of the next version of the paper.  \n\n[1] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction. CVPR2025.  \n[2] SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. ICML2025."}]

Author rebuttal:

Meta-review: All four reviewers initially rated the submission as borderline accept (4), and the discussion and rebuttal phase led to one reviewer raising their score to 5. The paper presents a novel and well-motivated method—Balanced Token Pruning (BTP)—to reduce computational costs in LVLMs by balancing local and global pruning objectives across layers. Reviewers found the approach intuitive, technically sound, and empirically well-supported, especially appreciating the extensive ablation studies and thorough rebuttal. While minor concerns remain (e.g., hyperparameter tuning clarity and practical batching implications), the authors addressed these with detailed analysis. Given the overall positive assessments and the clarity brought through rebuttal and discussion, I recommend acceptance.
Decision: Accept (poster)