Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning
Authors: Yihong Huang
Fei Ma
Yihua Shao
Jingcai Guo
Zitong YU
Laizhong Cui
Qi Tian
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Vision-Language Models; Vision Token Pruning; Inference Acceleration; Visual Grounding
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Improving model performance for efficient VLM under token pruning settings by restoring spatial integrity, particularly for visual grounding tasks.

Abstract: Vision token pruning has proven to be an effective acceleration technique for the Efficient Vision Language Model (VLM). However, existing pruning methods demonstrate excellent performance preservation in visual question answering (VQA) and suffer substantial degradation on visual grounding (VG) tasks. Our analysis of the VLM’s processing pipeline reveals that strategies utilizing global semantic similarity and attention scores lose the global spatial reference frame, which is derived from the interactions of tokens' positional information. Motivated by these findings, we propose N\"uwa, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity. In the first stage, after the vision encoder, we apply three operations, namely separation, alignment, and aggregation, which are inspired by swarm intelligence algorithms to retain information-rich global spatial anchors. In the second stage, within the LLM, we perform text-guided pruning to retain task-relevant visual tokens. Extensive experiments demonstrate that N\"uwa achieves state-of-the-art performance on multiple VQA benchmarks (from 94\% to 95\%) and yields substantial improvements on visual grounding tasks (from 7\% to 47\%). Code is released.

Published as a conference paper at ICLR 2026
NÜWA: MENDING THE SPATIAL INTEGRITY TORN BY VLM TOKEN PRUNING
Yihong Huang Fei Ma Yihua Shao Jingeai Guo Zitong Yu', Laizhong Cui Qi Tian Guanggong Laboratory of Artificial Intelligence and Digital Economy (SZ)
Schooo of Artificial Intelligence, Xidian University
"The Hong Kong Polytechnic University 'Great Bay University
Shennzhnn University 'Huawei
mafeiiimmmm. Sangyihong@stuu xidian. edu, en, mafei @mml www.gmaa....oo...oo
ABSTRACT
Vision token pruning has proven be effective acceleration technique for the efficient Vision Language Model (VLM). However, existing pruning meth- ods demonstrate excellent performance preservation in visual question answering (VQA) and suffer substantial degradation on visual grounding (VG) tasks. Our analysis of the VLM's processing pipeline reveals that strategies utilizing global semantic similarity and attention scores lose the global spatial reference frame. which is derived from the interactions of tokens' positional information. Moti- vated by these findings, we propose Nüwa, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity. In the first stage, after the vision encoder, apply three operations, namely separation, alignment, and aggregation, which are inspired by swarm intelligence algorithms to retain information-rich global spatial anchors. In the second stage, within the LLM, we perform text- text-guided pruming to retain task-relevant visual tokens. Exten- sive experiments demonstrate that Nüwa achieves SOTA performance multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% 47%)
Code: https://github. wtt.tuu..oocoemaao hetp://anapPpeeReeeeeeee///////e
Reforcool
56.4
Vasilla
35.3
Navo 125
Vaiilla
Vidcest
SparsVVL
Fasty
Helcoorg Ind.
Bekoco ked/
FaadV
Relroco+
Helroro+
bstB
Figure I: Nüwa Performance On VQA and VG tasks, preserving 95% and 47% under 88.9% reduc- tion of vision tokens. (a) Our Nüwa outperforms current efficient VLMs 10 VQA benchmarks; (b) On visual grounding benchmarks, Nüwa also achieves SOTA results.
"Corresponding Author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
VLMs [Liu et al. 2024a Bai et al. 2025 Zhu
2025
exhibit strong multimodal capabili-
ties through pre-training massive image-text pairs. However, the large number of vision tokens generated during inference leads to substantial computational overhead and reduced throughput. Re- cent visual token pruning methods aim to accelerate inference while preserving model performance.
These include approaches based on visual-semantic similarity (Yng et al. 2024 Li & Shin 2024
Jeddi et al. 2025), textual semantic filtering (Chen et al.
2024 Xing et al. 2024 Endo al. 2024), and mului-staee pruning (Zhang et al. 2025a] Liu et al. put. Additionally, token merging (Bolya al., 2023), technique within token pruning, increases token sparsity in VLMs during inference. Fundamentally, reducing the number of tokens enhances sparsity and thereby improves VLM's inference throughput.
2025b), which improve inference through-
Vision-Basse
Fg. Videoodp
Text-Based
Egg
Nüwa
on Cobeskon
Lagss
[51.]]
RAllmmeent
Semartik Prusint
(a)
(b)
(c)
Figure 2: The left panel contrasts our Nüwa framework with prior token pruning methods. (a) Pruning at the vision encoder stage; (b) Text-guided pruning within the LLM; (c) Our two-stage approach: initial spatial-aware pruning via local aggregation that preserves global anchors in the vision encoder, followed by text-guided refinement in the LLM.
Nevertheless, recent studies (Wen et al. 2025 Endo et al. 2024) have questioned the effectiveness of existing pruning methods (Chen al. 2024] Zhang et al. 20256). particular, random pruning and pooling-based merging can achieve competitive performance, yet these methods exhibit substantial degradation on visual grounding (VG) tasks compared with visual question and answering (VQA) tasks (Long et al., 2025; (haao et al., 2025a). To assess whether these issues are widespread, we systematically categorize existing pruning methods and compare them with simpler baselines across multiple datasets. Our experiments confirm that these limitations persist. These findings raise funda- mental questions:
Why do existing pruning methods exhibit significant task rusk-dependet degra- dation? @ How is vision information encoded and utilized within the VLM's processing pipeline? How to Mend grounding performance gaps in VLM's token pruning setting?
Through systematic experimental analysis, we uncover that VLMs employ multi stage visual pro- cessing pipeline that progresses from global to fine-grained integration, with task-specific require- ments. In particular, grounding tasks depend preserving global spatial reference frames, which are constructed from token position information and can be disrupted by token pruning. Informed by these insights, we introduce Nüwa, shown in Figure 2 two-stage spatial- token prun- ing framework, patching up the torn spatial integrity. The first stage operates in the visual semantic space to reduce token redundancy while maintaining spatial topology. employs a Boids-inspired algorithm (Reynolds, 1998) with three operations: (1) Separation: partitioning the token map into localized regions; (2) Alignment: selecting representative tokens based on their alignment with the global context and information density; and (3) Aggregation: merging features of neighboring tokens around representatives using semantic similarity. The second stage performs text- eoxt-guided re- finement in the intermediate layers of the LLM after multimodal feature alignment, using textual semantics to guide further pruning.
Nüwa demonstrates significant improvements, shown in Figure I (7% 447 %,, 18%-77 48%-75%% across multiple pruning configurations in LLaVA-1.55 alongside en- hancements in VQA benchmarks, including image reasoning and understanding performance (44,,, and validates its effectiveness across additional models.
VG benchmarks
Our contributions are as follows:
2

---

Published as a conference paper at ICLR 2026
1. Task-specific Analysis:
We systematically examine VLM's processing pipelines and show that current pruning methods fail on grounding tasks by overlooking task- specific require- ments and disrupting spatial structure. Position reconstruction experiments confirm that spatial perception arises from the integrity of the global reference frame.
2. Nüwa Framework: We propose a two-stage spatial-aware pruning framework that retains global spatial anchors through separation and adaptive region aggregation, thereby preserv- ing both spatial and semantic integrity. It further leverages textual information in the LLM for multimodal alignment-basee pruning.
3. Performance Validation: Our approach yields superior results across 13 datasets and mul- tiple VLMs, establishing new SOTA on VQA (95% performance retention) and VG (47.2% performance retention) tasks while achieving 89% reduction TFLOPs and 62% reduction in prefill time with 88.9% tokens reduction.
2 DISSECTING THE VISUAL PROCESSING PIPELINE: FROM SEMANTIC FLOW TO SPATIAL INTEGRITY
In this section, we first perform a systematic analysis (Sec 2.1 of existing pruning methods to ad- dress two key questions. We then examine the visual information processing pipeline (Sec 2.2) in VLMs through two analytical experiments, tracing the progression from glohal attention mecha- nisms to local processing paradigms. Finally, position reconstruction experiments (Sec uncover the root causes of performance degradation in grounding tasks, thereby providing insights for the design of pruning methods.
2.1 EVALUATING COMPETITIVE ADVANTAGES: SIMPLE BASELINES VERSUS ADVANCED PRUNING METHODS
Recent research Wen et al. 2025 Endo et al. 2024) has questioned the effectiveness of existing visual token pruning methods. To investigate two key aspects - (1) Generalization: whether advanced methods consistently outperform simple baselines, and (2) Robustness: whether perfor- mance remains stable across tasks with diverse requirements, we conduct a comprehensive cross- task evaluation.
Experimental Setup We conduct comprehensive evaluation across 12 datasets, covering a broad spectrum of capabilities including image grounding, fine-grainet understanding, and complex rea- soning. To facilitate systematic comparison, we categorize mainstream visual token pruning meth- ods into three distinct families based on their architectural placement and operation stage: Vision Encoder-Side Pruning. which focuses on reducing redundancy within or at the output of the vi- sion encoder to save memory early on (c.g., VisionZip (Yang et al. 2024), PruMerge (Shang et al,, 2024)); LLM Single-Layer Pruning. which applies one-time, fixed-ratio pruning operation at specific layers within the LLM (e.g.. FastV (Chen et al., 2024)); ); and LLM Multi-Layer Pruning, which dynamically identifies and removes non-essential vision tokens across consecutive LLM lay- ers (e.g.. PyramidDrop (Xing et al., [2024], Sparse VLM (Zhang et al. 2025b)). To ensure a fair and rigorous assessment, benchmark these sophisticated methods against two simple yet effec- tive baselines, random sampling and average pooling, to determine the value of complex pruning designs.
Table 1: Performance comparison of various vision token pruning methods on LLAVAI.5-7B. In- cluding LLM Langle-Layer Pruning, LLM Multi Pruning, and Vision Encoder-Side Pruning.
GUA
VQJtest
SUA
Arg(
UVPR222
09.5
31.1
1000
PantV
HXN
66.1 174 9163
48.0 (34.53)
$4.00 (90009370 9953
1288
087 415)
56.0 115)
47.8 (87.)
40.5 (90 410-
68.7 (96 8756
23.8(24.993)
Ronderr
$1.2 (83.666)
45.5 164 9953
34.1 (94.0955
1351 (72 655)
65-4 (834-133)) (35))
44.9 (77 (1.)
61.1 156)
66 (.0 1%6
15.9 143
75.1
Pooling
53.3 (54.4%)
45.7 (75.3%)
35,0 (93.796)
1380
(74.1%)
09.3
(38.016)
65.3 (77.555)
07.8 (78,595)
(97.797)7
15.3
(52.4%)
80.3
SparseYLM
1CMB
(90.
Bandhers
$1.5 (61.766 266
05.0 (210)111.55 395
14.1 (94.995)
142 (12.11
085
(10)
21.5 (81.655)
18.1
711 (74.14)
87.5
VidrenZip
$5.1 (190-095)
68.3 (M7 5953
36.3 (M9) 715.
6460 (103 (11))
72.4 (97 710)
85.8 195 415
77.0 (30) (16)
31.7 S78)
MA,5
PruMergee
10CV
$5.8 (80.51)) 516)
99.5 (92.
35.8 (05.655)
1516 (56 2153
71
(00.8 (0)
$2.0 (50 (15)
75.77 (881119)
69.5 (100L0R))
28.0 (00L05%)
91.7
Ranshers
$4.%
(95)
(15)
100
(16)
7117 (66 (26)
51.5 (83.119)
(08.6/))
32.1 (85.4%))
(61.299)
(56.316)
(13
64.7 (93.1%5)
15.7 (60L115)
77.2

---

Published as a conference paper at ICLR 2026
Our results reveal key pat-
Table Performance comparison on RefCOCO series datasets.
terns across task types.
On general-purpose VQA
Avg
Method
Refcoco-test
Refcocut-tesLA
Refcocot-lestB
Refcocog-lest
benchmarks (Table sim-
576
LLaVA
58.30
59.43
38.88
48.50
FastV
10.34
ple haselines achieve com-
SparseVLM
6.27
5.79
4.22
6.35
petitive performance, often matching advanced prun- ing methods. In contrast, on object-centric ground- ing tasks (Table 2 all methods show systematic performance degradation, regardless of design complexity. Notably, average pooling yields the best results among pruning approaches, likely because it partially preserves spatial structural features.
128
VisionZip
4.49
4.86
Pooling
23.01
24.37
15.04
19.69
FastV
2.73
1.17
1.02
2.19
Sparse M
1.04
0.96
1.28
0.61
64
VisionZip
Pooling
12.01
12.20
7.55
11.40
Finding Advanced pruning methods provide limited benefits over simple baselines on VQA tasks, whereas all methods suffer systematic degradation grounding tasks, with average pool- ing achieving the best performance.
2.2 UNVEILING MASK-DEPENDENT VISUAL PROCESSING PIPELINE
Building on the task-dependent performance degradation observed in Sec. 2.1 prior explainability studies on LLMs and VLMs (Selvaraju et al., 2016 Ding et al. 2017 Zhang et al. 2024 Yin al.
2025) have not sufficiently explored how visual processing adapts to shifts in task focus, such as from VQA to VG. To address this, we conduct flows from the final token to vision tokens during decoding, and applying bution methods to trace critical visual information pathways across tasks. Additionally, we evaluate the model's object-centric perception at different stages using two fine-grained metrics.
analytical experiments: visualizing attention
weilhhte attri-
Last-to-Vision
Vislon-to-Viiie
Text-to- Vision
Last-to-Text
Groundia
VOA
Attn Flow
Grad
(4)
Figure 3: (a) to (d) show different types of attention flows (First row) and gradient-weighter atten- tion flows (Second row), where A-to-B means the degree of attention A pays to B. (e) shows the differences in Last-to- Vision attention maps across different tasks. VLMs exhibit two-stage visual processing pipeline, with task-independent multimodal interactions in early layers and task-specific processing middle layers.
Figure 3 depicts task-dependent characteristics of visual processing in VLMs. Panels (a) and (b) at the first row show that attention flows exhibit distinct early and mid-stage phases. However, gradient- weighted analysis (Second row) reveals pronounced task-dependent divergence in the mid-stage, underscoring the model's sensitivity task requirements during visual integration with VG tasks showing greater reliance vision tokens. Panel (c) highlights task- insk-independent aspect: early multimodal interactions, suggesting universal visual processing initial stages. Panel (d) illustrates task-varying differences in text information handling. Further experiments on attention blocking (Appendix [B.5) indicate that, in VG tasks, textual cues extract critical visual details, resulting in unique last-to-text attention patterns.
Visual Attention Entropy And Object-Centric Cohesion: Attention flows offer insights into the model's information processing dynamics. To further quantify the multi-stage visual processing
4

---

Published as a conference paper at ICLR 2026
pipeline identified in the prior analysis and its task-dependent characteristics, introduce two fine- grained metrics: Visual Attention Entropy (VAE) and Object-Centne Centric Cohesion (OCC). VAE mea- sures the distribution of information in the visual self-attention mechanism by computing the average Shannon entropy across visual tokens (Eq. ① High VAE values indicate diffuse, global attention patterns, whereas low values reflect concentrated, local focus. Complementing this, OCC assesses object-level feature cohesion by calculating the Intersection over Union (IoU) between ground-truth object tokens and the top-k tokens most similar to the object's center token (Eq. (2)) Higher OCC scores denote stronger localization of features to relevant objects, capturing theeggained processing.
H(v,) = Σ p(aylo:) log2 p(uy/oe),
VAE
H(u))
(1)
N
1=2
Vinuell Vol
oCc(O) = m Vol
(2)
VIT-Lape11
VIT-Laye 6
VIH-
VIT-baee
Vir-aae 2t
W
Válan Encder Shar
LLM Decodar Stage
L
LLM 18
LLM 24
6
(h)
Figure 4: Visualization of VLM's Two-Stage Vision Tokens Processing: (a) Layer-wise Analysis of VAE and OCC Metries; (b) Layer-wise Instance Heatmap Visualization. Both demonstrate fine- grained feature extraction the mid-stage.
As shown in Figure the VAE of the ViT encoder exhibits decreasing trend in the middle stage, in- dicating gradual shift from global context integration to fine- tineggraneed feature extraction. In contrast, the VAE of the LLM decoder fluctuates after sharp initial increase, suggesting a more complex pro- cess of reorganizing visual features and integrating them into the textual semantic space. The OCC scores provide a clearer explanation - they peak in the middle stage of both ViT and LLM, sig- nifying the formation of object-level representations. This phenomenon also effectively explains the earlier observation: why grounding tasks demand such high levels visual information at this stage.
Finding 2 Visual processing in VLMMs unfolds through multi-stage pipeline, progressing from global semantic integration to fine-grained object-centric focus, with task-specific reliance on vision tokens. Grounding tasks require heightened visual integration during middle stages for spatial reasoning, in contrast to the reduced demands in image understanding tasks.
2.3 SPATIAL INTEGRITY: RECONSTRUCTING THE GLOBAL REFERENCE FRAME
Building on the mid-stage visual integration demands in Sec. 2.2 where pruning disrupts task- specific vision reliance, hypothesize that spatial integrity Frame from position embeddings - is essential for spatial perception, as pooling methods' superior grounding performance indicates. To validate this, we design experiments restoring integrity through modified position embedding strategies.
the Global Spatial Reference
2.3.1
A TAXONOMY POSITION EMBEDDING STRATEGIES
To rigorously test our hypothesis, we first deconstruct the implicit position embedding (PE) handling strategies within existing pruning methods, shown in Figure abstracting them into three distinct paradigms:

---

Published as a conference paper at ICLR 2026
Position Embedding Range Compression (PERC): Compresses the PE of pruned tokens into tiny range, missing the global reference frame, like Visionzip.
PERC
Vausa
Viuad
Spatial
Position Embedding Sparse Preservation (PESP): Retains the original PE for each pruned token, forming sparse subset within an incomplete spatial frame, like FastV.
RPME:
Visuad
Spatial
Pend talons
Relative Position Mapping Extension (RPME): Preserves the relative spatial dis- tance of the pruned tokens and extends their PE via linear mapping, to span the entire original range and retain the spatial integrity.
Figure 5: Sketch of different Position Embedding Strategies. RPME retains the spatial integrity.
Experiment Setup We select two represen-
tative methods, VisionZip (PERC) and FastV (PESP), replacing their PE strategy with RPME, and then evaluate the performance of these "fixed" models on visual grounding benchmarks.
Table 3: Position Reconstruction Experiment on Refcoco series and VQA Benchmarks. The sym- bols '4 and indicate changes relative to the pre-reconstruction values showed in Table 2.
Methed
Refcoco
Refowes
Refeace* -sslB
Refcocor
4pd
rral
MYR
VQM2
MMT.
867
(1.37
M.27 ced Saj
R.2 (e 12)
8.10 (+4.11)
KM I0)
M8(-17)
1000-110
Rist- fis
(.5 19)
411 10)
3.4 (44 2.67) 12.30
(175
4.31 (.390
6 1900
22220220
462 (10.)
475(02)
561 (09)
(247
Pooling
9,85
F38 Tolors
MisooZigtth ths
16.900116.00
(6
19.00 (-15
13 c-8 oug
16, 10 12 221
15 19)
15.00 15 1212
(999
7630133
1751
13.41 (43.07)
(.3)))
14.58 (3.3))
(2.2()3..5)
(+345)
(1.3
527
Pooling
24.39
Results in Table 3) show that RPME yields notable improvements across benchmarks: VisionZip achieves gains of 5.6 % and 13.4% in two settings, while FastV sees more modest increases of 1.8% and 3.2%. These differences confirm our analysis: PERC in VisionZip eliminates positional information, whereas PESP in FastV preserves absolute coordinates but disrupts spatial continuity. Gains grow with larger token budgets, underscoring the increasing importance of complete spa- tial frameworks for richer visual organization. Pooling methods outperform others consistently by aggregating features on coarse grids that implicitly maintain global topology, reinforeing that recon- structing continuous spatial coordinates is vital for grounding tasks. This strategy has a negligible impact on image understanding and reasoning benchmarks, indicating broad applicability.
Finding The degradation VLMs on grounding tasks is principally driven by the loss Global Spatial Reference Frame within token pruning strategies, which can be restored by pre- serving global position embedding.
3
METHODOLOGY
Our analysis reveals that existing token pruning methods fail on spatial localization tasks by disrupt- ing the glohal spatial reference frame. This motivates three core design principles for effective visual token compression: (1) preserving spatial uniformity to ensure consistent coverage; (2) aggregating redundant information in vision-centric, cohesive manner while retaining local salience (Stagel). and (3) applying text-modulated fine-grained filtering to select tesk-relevant tokens based on textual We apply these principles in Nüwa, two-stage pruning framework.
semantics Stage2).
3.1 STAGE 1: SPATIAL COHESION PRUNING IN THE VISION ENCODER
This stage reduces the initial 22 visual tokens in the vision encoder to a dense, spatial-preserving sequence via three sequential operations.

---

Published as a conference paper at ICLR 2026
3.1.1 SEPARATION VIA GRID PARTITIONING
To maintain spatial integrity, partition the input token grid 7 { ...... fya) into M M non-overlapping local regions Rij. Subsequent selection and aggregation occur at the region level, enabling a complete global coordinate system.
3.1.2 ALIGNMENT VIA SALIENCE IDENTIFICATION
Within each region R, we select representative benchmark tokens as aggregation centers. These tokens should exhibit high global salience; we initially use attention scores from the [CLS] token. However, analysis indicates sparse distributions in deeper vision encoder layers. To mitigate this, incorporate information capacity, defined as the L2-norm of the token's 's key vector (l]k; ll2). as a secondary criterion. The resulting salience score (t;;) for token t; is the product of its global attention score and information capacity:
St4) C'os.. |
where Oclsa is the attention weight from the [CLS] token. In each local region Rk, we select the k tokens with the highest salience scores to form the Benchmark Token set TB.
Stage
Separation Alignment
Region
Stage
Pruning
Profilling Time
capacity +
+ V Visual
Similarty
Nüwa
57
SALIENCE
Vanilla
124
Vanilia 138
(A)
[c]
lignment
ampa
Layer 32
Visian
V@A
Decode
VQa Tusk:
Layer
Graunding
Layer-wise Decode
Locate the cabcher
1044.
Figure 6: The Framework Nüwa: (a) Stage 1 Pruning regarding Separation, Alignment and Cohe- sion; (b) Layer-wise 2D visualization of text-visual token similarity during LLM; (c) Stage2 pruning based on text semantics at LLM mid-stage; (c) prefill time of Nüwa across different scales.
3.1.3 AGGREGATION VIA SPATIAL PROXIMITY
This operation merges features from other tokens into the benchmark set TB, guided by role assign- ment and spatial proximity, yielding semantically rich and spatially complete token sequence.
Pillars and Collectors. We differentiate benchmark tokens in TB by informa- tion capacity. Recent works (Darcet et al.] 2024 Lappe & Giese 2025) identify high-norm tokens ViTs as registers frequently attended during decoding and often task-agnostic. Modifications to these can shift feature distributions and affect predictions. Thus, we classify tokens with |k; ||2 in the top quartile Pillar Tokens (Tp), whose features remain unmodified. The rest are Collector Tokens (Tc). which aggregate from spatial neighbors.
Role Assignment:
Tp = (t. E TB k 2 ((,nii(((((lll,,eeaaa 0.75)
To = TB \Tp
(4)
Weighted Aggregation. High semantic similarity does not imply aggregability; relying solely on it for global features is inadequate, risks disrupting object-centric representations by merging

---

Published as a conference paper at ICLR 2026
spatially distant tokens. Thus, balance it with spatial proximity to form weight matrix W I x where [TBl. combining semantic and proximity matrices.
Semantic Similarity Matrix (A): We consider only positively correlated semantic information.
Element As defined as Eq. 5):
V V vvlly,
Ag = ReLU (sim(v,, v5)) = ReLU
(5)
Spatial Proximity Matrix (P): To penalize long range aggregation, we define proximity ma- trix allowing each benchmark token to aggregate features within an extended local neighborhood. enabling limited cross-region interaction. Element Piy is computed as Eq. (6):
Poj = max (1. d(pi.P))
(6)
ditress
where d(p,, Pу) is the Euclidean distance between ps and Py, and dinessa a predefined threshold. Based on role assignment, the final aggregation weight Wiz is defined as:
dig A.g.Ps Poj
if 4 E Tp (Pillar Token)
Wi
i4 E To (Collector Token)
where 6ig the Kronecker delta, ensuring Pillar Tokens only aggregate from themselves.
The weight W is row-normalized from W, the original feature matrix is V E N= The updated feature matrix for benchmark tokens, V B E RRXD, is computed as V = WV.
3.2 STAGE 2: TEXT-MODULATED PRUNING IN THE LLM
Following Stage I, the aggregated vision tokens V are fed into the LLM for multimodal feature interaction. We apply a second round of task-oriented pruning at an intermediate layer, after initial multimodal aligmment (Shukor& & Cord 2024). where textual and visual features converge in a shared space. To guide this pruning, first derive a holistic textual query vector q by average-por werage-pooling the embeddings {qi...., } of text tokens:
1
q
q:
We calculate relevance score R3 for each visual token 1 (with updated feature vector v/. the i-th token of VBB) " by measuring its cosine similarity to the query vector in the shared embedding space:
proj(v v() q
R2 = sim(proj(v?) (p)pooj((v)))))
proj v() 1q
where proj(-) denotes the multimodal projection layer mapping visual features into the common text-vision embedding space. Finally, we retain only the top-Kinal visual tokens with the highest relevance scores R2 passed to subsequent LLM layers for final reasoning and response generation.
EXPERIMENT
Experimental Setup: To validate the generality and effectiveness of our method, we conduct ex- periments on multiple VLMs and diverse benchmarks for image understanding and visual ground- ing tasks. The evaluated models are LLLVA- LLaVA-NeXT. We use VQA benchmarks (e.g., GQA, TextVQA) and 3 VG benchmarks (RefCOCO, etc.). All experiments are run on NVIDIA A A100-40G GPUs. Detailed configurations are the Appendix B.2
4.1 MAIN RESULT
Performance VQA Tasks: We apply Nüwa during the inference stage of LLaVA-1.5-7B. More results in Table 5 demonstrate that Nüwa achieves optimal performance across nearly all bench- marks, with average performance further improving upon existing SOTA methods. On more VLM
8

---

Published as a conference paper at ICLR 2026
models with different scales, such LLaVA-NeXT-7B. Nüwa consistently demonstrates perfor- mance gains, establishing its strong generalizability. Results can be found in Appendix B.2 Perfor- mance Visual Grounding Tasks: Visual grounding tasks are highly sensitive to spatial informa- tion tokens, constituting a critical evaluation dimension for compression methods. On RefCOCO series visual grounding benchmarks, shown in T our method substantially outperforms al- ternative approaches, achieving approximately 35% performance improvement over previous meth- ods under 64 average tokens configuration. When retaining 192 tokens, our method maintains 79% of the original model's performance.
Efficiency Analysis: As shown in Table
Table Comparison of Model Efficiency. "main" and "metric" mean the standard Transformer pipeline and the additional computational load of pruning metric.
we evaluate efficiency from two dimensions: theoretical computational complexity and ac- tual prefill latency. Nüwa introduces negligi- ble computational overhead, with TFLOPs creasing by only 0.01 and prefill stage latency increasing by ms, compared with previous SOTA. Nüwa's ' design requires executing atten- tion computation only once on tokens from the final layer of the vision encoder, enabling seam- less FlashAttention compatibility through sim- ple code modifications.
Method
Avg Token
main (TFLOPs)
metric (MFLOPs)
Prefill-
(ms)
Vanilla
5,9730
FistV VisionZip Nowa
0.8341
4.7185
26%
5.50%0
104
0.6461
8,9128
45
6.3%
64
0.6476
17.5633
46
6.%
4.2 ABLATION STUDY
Ablation on Spatial Proximity Threshold To enable aggregation based on spatial proximity, we define local neighborhoods via distance threshold Empirical evaluation (Table shows that per- formance peaks at r 26% of the maximum distance. Smaller values restrict aggregation scope, leading suboptimal results, while larger values incorporate noise from distant regions, also de- grading performance. These results confirm the effectiveness of localized aggregation in preserving spatial integrity. Ablation on Key Components Experimental results in Table g show that region partitioning is essential for grounding tasks, it implements more precise RPME strategy, but has negligible effects on VQA tasks. The L2-norm criterion positively enhances baseline token selection across all tasks, consistent with analysis in Sec. 3.1.3 For stagstage pruning, gains over random pruning remain modest. Notably, combining random pruning with region partitioning substantially degrades performance, as the partitioning introduces potentially task- iiseiirrleeaant tokens that random selection may retain.
Table 5: VQA performance comparison On LLava-1.5 7B. Best and second-best results are high- lighted.
Method
Source
GQA
MMB
MMMU
MME
VQÁv2
VQAtext
POPE
SQA
SEED
MMVeet
Vanilla
CVPR'24
61.9
64.7
36.3
1862
78.5
58.2
85.9
69.5
58.6
31.1
100%
Average Token
66.7%
FastV
BCCV"24
52.7
61.2
34.3
1612
67.1
52.5
64,8
67.3
57.1
89 53%
PDro
CVPR'25
57.1
63.2
34.1
1766
74.9
56.1
82.3
70.2
54.7
30.5
95. 87%
VLM
ICML'25
57.6
62.5
33.8
1721
75.6
56.
83.6
69.1
55.8
11%
VisionZip
CVPR'25
59.3
63.0
36.6
1782
76.8
$7.3
85.3
68.9
56.4
31.7
98 26%
Nüwa
60.9
64.3
35.5
1834
75.9
57.4
86.4
68.2
59.7
30.5
98.80%
Average Token 128
77.8%
FastV
BCCY"24
49.6
56.1
34.9
1490
61.8
50.6
59.6
60.2
55.9
28.1
85 04.00%
PDrop
CVPR-25
56.0
61.1
34.2
1664
73.5
55.1
82.3
69.9
53.3
30.8
94.32%
SLM
ICML'25
56.0
60.0
33.8
1696
73.8
54.9
80.5
67.1
53.4
30.0
93.36%
VisionZip
CVPR-25
57.6
62.0
37.9
1761
75.6
56.8
83.2
68.9
54.9
32.6
97.633%
PruMerge
ICCV'25
57.8
59.6
36.2
1712
74.7
54.3
81.5
67.6
30.4
95 06%
Nüwa
60.2
63.4
35.8
1828
75.1
57.0
85.5
67.8
58.7
29.8
97.87 %
Average Token 64 88.9%
ECCV"24
34.0
55.0
51.1
51.9
PDrop
CVPR'25
41.9
33.3
26.5
1092
57.3
45.9
55.9
69.2
40.0
24.9
71.56%
SLM
ICML/25
53.8
60.1
68.2
77.5
51.1
VisionZip
CVPR'25
55.1
60.1
36.2
1690
72.4
$5.5
77.0
69.0
52.2
31.7
93.99%
PruMerge
ICCV'25
55.4
59.6
35.8
1616
71.3
52.0
75.7
69.5
28.0
91.71%
Nowa
58.3
62.0
36.4
1706
72.8
34.9
83,0
67.5
56.44
28.2
94.91% %

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison on the RefCOCO series benchmark On LLava-1.5 7B. Best and second-best results are highlighted.
Method
Source
Refewcn-frst
Refeocs-val
Refeocos-testA
Refoocosdesst
Refeeens-sal
Refeoeog-lest
Refeocog-val
avg
Vanilla
CVPR
$8.30
36.42
39.43
58.88
46.32
48.50
48.82
Average
FEATHER*
2CCV'25
27.7
24.7
27.2
48.35 18.38%
Nuwa
47.91
46.12
43.18
31.86
37.68
37.64
37.90
79.29%
Average Tokens 128
77.8%
Fusty
ECCV
10:34
10.13
9.83
8.16
9.10
VipionZip
CVPR 25
4.49
4.06
4.86
738
3.48
8.14
Nuwa
45.09
43.69
42.63
28.98
35,32
36.39
36,00
75.20%
Average Tokenss 64 88.9%
Fasty
BCCV-224
2.23
2,01
1.17
1,12
2.41
2:19
2.01
1.1%
Sp Visin/Zip Nuwa
CVPR'25
29.33
28.00
17,47
22:22
21.11
21,42
47.19%
Table 7: Ablation Study On cohesion distance. The best pest-performing result in each column is bolded, and the second-best is underlined.
Table 8: Ablation Study on each design. Include Pillar-token selecting, Stage2 Random Pruming and Region Separation.
MMH
-csL3
-4ctBB
-Jest
pilae mundees
Refescn
Refenmos
Refcness
dathk
0.57544
64 2953
1695.5
1.2753
0.28118
0.16.5
012011
51
0.1330
6.72
0.48
3.25
62:0275
6:35
6:17
5.00
da556
0.1811
0.5810
0.185
0.3888
621134
62:1775
61.9416
1311.112
1706
1750 985
1696.3
0.2879
0.2834
0,2901
017718
0.2500
013637
0.1836
0.1655
011651
113135
012120
0.2033
38.43
57:34
57.15
61.71
56.10
$8.68
1771
1742
1724
43.50
45.37
1301
38.94
59.85
8.81
26.10
23.56
4.42
34.30
35.54
4.00
00.08
63.88
1828
65.89
02.63
38.86
16.99
0.1630
36.33
62.14
1791
$8.35
51.2
11.30
35.90
CONCLUSION
In this paper, we identify limitations in existing token pruning methods for visual grounding tasks and perform a systematic analysis of VLMs' multi-stage visual processing pipelines. Results reveal task-specific demands, where grounding relies on global spatial reference frames disrupted by prun- ing. To mitigate this, we propose Nüwa, a two-stage framework with Boids-inspired aggregation and text-guided refinement to preserve spatial integrity. Extensive experiments across 13 datasets and multiple VLMs show state-of-the-art performance on VQA (95% retention) and VG (47.2% retention), with 89% TFLOPs and 62% prefill reductions via 88 9% token pruning.
REPRODUCIBILIIY STATEMENT
To ensure the reproducibility of the results presented in this paper, we take the following steps. For the experiment of Sec. [.1 we set up simple baselines for each category of methods to con- duct comparative experiments. Implementation details are provided in the Appendix B33 For the analytical experiments in Sec. [2.2] no additional configurations are applied. For the Experi- ment of Sec 2.3 we conduct position re-estimation experiments. The algorithm implementation for RPME is provided in the Appendix [B.4] All experiments are based on LLAVA-1.5 7B, with the same environment configuration, dataset, and model weights claimed in Appendix B.T For the Main Experiment, we provide the algorithm implementation of the complex Stage-1 pruning in the Appendix B.2
The Stage-2 implementation is analogous to FASTV.
REFERENCES
Saced Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. 2025 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 2392-94011 2025. URL hhttps://api.
Demanticschnn www.shoola...o wttp:or///oorpus:::77770077
Kazi Hasan lbn Arif, JinYi Yoon, Dimitrios S. Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vasion-language models, 2024. htRhhtts:: arxiv.
10945
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani S. Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsey, Simon Korn-
10

---

Published as a conference paper at ICLR 2026
blith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Open- flamingo: An open-source framework for training large autoregressive vision-languae mod- cls. ArXiv, 0tss/238.00190, 2023. URL https: /ppi..p..... semanticscholar org/ CDrpusTD:221043333
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL httpttsss//a.ii...oot
/tss//502.. .0923.1399 13923
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster, 2023. URL Ittpps:/azziv..rgg: /bs/
22100.00 09461
Mu Cai, Jianwei Yang. Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal mod- els. ArXiv, 174330 2024. URL http::////pp.eeeattcccccchoooolo
CorpusID:2700635::
Junbum Cha, Wooyoung Kang. Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal 1lm. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13817-13827, 2023. URL
org/CozpusID: 2261:2166612222
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang.
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. European Conference on Computer Vision, 2024. URL https://api. http:/ooopusDD:22333500
semanticschol
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305 06500, 2023. URL https:
semnt seaaniishmiicc orq wtqgCorpus... ID: : 258615866520 626666666
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partition-
ing. ArXiv, htts/2377.0861,, 2023. URL https: //api.//ppp... www.ssstcaactcccolla... orr
CorpusID:259936733
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher R'e. Flashattention: Fast and memory-efficient exact attention with io-awareness. ArXiv, 14sp/22205114135,,. 2022. htthttss:
//api. htt.://aarttccccllaaaa...ooooppa
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2024.
/209 16588
Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. Visualizing and understanding neural machine translation. In Annual Meeting of the Association for Computational Linguistics, 2017.
URL httttt/sp//pppp www.app....ammm................e www.niicschoooooooooo.....
org/ http:/oouul1::22300000.
Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token
pruning for vision-language model acceleration. ArXiv, (ttp/2412.11188 2024. URL https:
7/api. www.ssematticciic www.ssaannnccccola...oooo htt.://choooaa...oooooopppaaaaaa
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306. 10096..334,,,. 2023. URL
https://api semant www.eonntcchoooooooo wr......rgrggooo.... wwwpr/////roooooooooooooooo ID: : 259243923
Yash Goyal, Tejas Khot, Douglas Summers-Stre -tay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398 - 414, 2016. URL httpps://aa
semant cschol hrr.ccooaarrrr//rrrpptst:::::

---

Published as a conference paper at ICLR 2026
Lianyu Hu, Fanhua Shang, Wei Feng, and Liang Wan. Lightvlm: Acceleraing large multimodal models with pyramid token merging and kv cache compression, 2025. URL https: //apiii.//axxv..
org/abst abs/ 2509. 00419
Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaoshen Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, and Shaohui Lin. Dynamic- Ilava: Efficient multimodal large language models via dy- namic lisiun- context sparsification. ArXiv, abs/2412.00876, 2024. URL https:
7/ 7/api. www.seetttttcchceeeeeeeee ott.aaaattiscccooooooooooororrpa
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. 2019 IEEE/CVF Conference on Computer Vision and Pat- temn Recognition (CVPR), pp. 6693 2019. URL http:::://////m httpps://appp.oo
oxq/CozpusID: 1522822655
Ahmadreza Jeddi, Negin Baghbanzadeh, Elham Dolatabadi, and Babak Taati. Similarity-aware token pruning: Your vim but faster. ArXiv, abs/2503. 1549, 2025. URL https://api,
Semanticscholat
Samir Khaki, Junxian Guo, Jiaming Tang, Shang Yang. Yukang Chen, Konstantinos N. Plataniotis, Yao Lu, Song Han, and Zhijian Liu. Sparsevila: Decoupling visual sparsity for efficient vlm inference. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 23784-237944 October 2025.
Alexander Lappe and Martin A. Giese. Register and cls tokems yield a of local and global features in large vits, 2025. URL J http:s///aa......rrr////////5550 05892)
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench-
marking multimodal lims with generative comprehension, 2023a. URL https://arxix:
org/abs/ 2307. 16125
Dylan Li and Gyungin Shin. Promerge: Prompt and merge for unsupervised instance segmentation. In European Conference Computer Vision (ECCV), 2024.
Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang. Jianke Zhu, and Lei Zhang. To- kenpacker: Efficient visual projector for multimodal 1lm. ArXiv, fis/2407.02399) 2024. URL
pettss//aapi semant icscholar org/ Org/CorpusID: : 270878717
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and rong Wen. Evaluat- ing object hallucination in large vision-language models. In Conference on Empirical Methods in Natural Language Processing, 2023b. URL lattps://appl semanticscholleee orgg
Corpus :58:11::258.00 25022069777777
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp-
26286-26296, 2024a. doi: 10.1000/VV552233.2002 20.240444,
Jizhihui Liu, Feiyi Du, Guangdao Zhu, Niu Lian, Jun Li, and Bin Chen. Hiprune: Training free visual token pruning via hierarchical attention in vasion-language models, 2025a. URL https:
00553
Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model, 2024b. URL https:
//arxiv.org/ wttp:///g/aaaa///////// 10803
Xuyang Liu, Ziming Wang, Junjie Chen, Yuhang Han, Yingyao Wang. Jiale Yuan, Jun Song, Linfeng Zhang, Siteng Huang, and Honggang Chen. Global compression commander: Plug-and-play inference acceleration for high-resolution large vision- language models. 2025b. URL https:
//api semant www.antttcccolllooo
Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? ArXiv, 0ttp//337.06688. 2023. URL https:
//api. semant cascolaa wetchhlrr..rgg//// crtp://orrrsTTT::5557000000
12

---

Published as a conference paper at ICLR 2026
Xinwei Long, Kai Tian, Peng Xu, Guoli Jia, Jingxuan Li, Sa Yang. Yihua Shao, Kaiyan Zhang, Che Jiang, Hao Xu, et al. Adsqa: Towards advertisement video understanding. arXiv preprint DrXX::2500.08621, 2025.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. ArXiv, 0aa/2209.09513. 2022. URL https://api
Craig W. Reynolds. Flocks, herds, and schools: distributed behavioral model, pp. 273-282. Association for Computing Machinery, New York, NY, USA, 1998. ISBN 1581 13052X. URL
Stttp::/ddo.org//10 wtt.:o/...../......... 1ee111115221111 221008
Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based lo- calization. International Journal of Computer Vision, 128:336 359, 2016. URL https:
seapl..ematticcchooo.... org. wwtgg/CoopusDDcco 15019293999
Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge Adaptive token reduction for efficient large multimodal models. ArXiv, abs/2403. 15388, 2024. h
//api..emant i cachol ar watg//orruuss
268666288
Yihua Shao, Haojin He, Sijie Li, Siyu Chen, Xinwei Long, Fanhu Zeng, Yuxuan Fan, Muyang Zhang, Ziyang Yan, Ao Ma, et al. Eventvad: Training-free event-aware video anomaly detection. arXiv preprint arXiv:2504. /3092, 2025a.
Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang. Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing a twig to accelerate large vision-language models. ArXiv, 140225553.14045... 2025b. URL htcpppppe//ppp..emttttcccccooooo oore/
Mustafa Shukor and Matthieu Cord. Implicit multimodal alignment: On the generalization of frozen Ilms to multimodal inputs, 2024. /tt/:ttteee/aarrxv..orr//aa//a/a
16700
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2019. httpttss://app
semanticscholas wwtp:aa.....acoooooaooaeoao : 85553602]
Jintao Tong, Wenwei Jin, Pengda Qin, Anqi Li, Yixiong Zou, Yuhong Li, Yuhua Li, and Ruixuan Li. Flowcut: Rethinking redundancy via information flow for efficient vision-language models,
2025. RRLLittpss//arxii/ wwwpe/ii..rgggaaa///aaaaa http:aab//25555.99 19536
Pavan Kumar Anasosalu Vasu, Fartash Faghri. Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gregory Gabriel, Peter Grasch, Oncel Tuzel, and Hadi Pouransari. Fastvlm: Efficient vision encoding for vision language models. 2025 IEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19769-19788 2024. URL https:
//api. www.etticccoola..ccooo...... http:c/cch..aac....ooooooaaaaaaa
Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem? In Wanxiang Che. Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Com- putational Linguistics: ACL 2025, pp. 15537-15549, Vienna, Austria, July 2025. Association for doi: 10. 11153/1113222 findings-ack ac2. org/ 2025. (.1.5..inn.... findings-aci 802/]
Computational Linguistics. ISBN 8779-8-89176--8 19176-5565
httph: http:::///laaaahhhooooe
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. ArXiv, abs/2309.1745 2023. URL httpps//api.
semanticscholar Dtwpeoooaar.or//oruuusDD..... 2I33110022633383
Long Xing, Qidong Huang, Xiao wen Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. Pyramiddrop: Accelerating your large vision- language models via pyramid visual redundancy reduction. ArXiv, abs/2410. 17247, 2024. URL https: semant inchooo ar. wttp://rrpss::22350777...
13

---

Published as a conference paper at ICLR 2026
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. 2025 IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 19792-19802, 2024. URL
Sttpps://aee semantt
semant ceschol ar .rg. wwwooorgrroorre Corpus ID D:224514545
Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adap- tive token pruning for large vision language models. 2025 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 24972-24982, 2024. URL https://api. semanti cscho ar org ot..oo//oooppsstT.... ID: ID:234333117
Hao Yin, Guangzong Si, and Zilei Wang. Lifting the veil on visual information flow in milms: Un- locking pathways to faster inference. 2025 IEEE/CVE Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 9382 9391, 2025. URL https::/appp. htt.:/ppp..emmtticcchooo.a..
org/CorpusID:2771c4.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Model- ing context in referring expressions. ArXiv, abs/1608.00272. 2016. URL https://api.
Gemanticscholle
wtt ID: 1:0::668555
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capa-
bilities. ArXiv, abs/2308.002499 2023. URL htts:://pp.semnatttccss .org//
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 9556-9567, 2023.
URL https:/ http::::/aap..emmattiscssss www.sseemanisss.eeee semanticchaoi /
CorpusiD:2654555
Ce Zhang, Kaixin Ma, Tianqing Fang. Wenhao Yu, Hongming Zhang, Zhisong Zhang, Yaqi Xie, Katia P. Sycara, Haitao Mi, and Dong Yu. Vscan: Rethinking visual token reduction for ef- ficient large vision-language models. ArXiv, abs/2505 222654, 2025a. URL hhtpss://api.
semant Commantcschol.. ar. org/Corpus ID: I 278955999
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsi- fication for efficient vision-language model inference. In International Conference on Machine Learning. 2025b.
Zhenyu (Allen) Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language mod- els. ArXiv, abs/2306. 14048, 2023. URL https::/api. http://ppp..mmmttccccccoo.......
Corpus 1D:
Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. Cross-modal information flow in multimodal large language models. 2025 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19781-19791, 2024. URL http://aaappssmmaaiciccccocooooa
ID:733362327
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjic Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang. Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng. Jiaye Ge. Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. In- ternv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https: : // arxiv...oo org/ abs 20004.044.47
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1 EFFICIENT LARGE VISION-LANGUAGE MODELS
LLMs and VLMs face significant computational efficiency challenges, particularly with extended sequences. LLMs grapple with the growing key-value (KV) cache during autoregressive infer- ence, leading to the development of token reduction strategies like StreamingLLM (Xiao et al.) 2023) and H2O (Zhang et al. 2023). However, VLMs confront amplified complexity due to the quadratic growth of visual tokens with image resolution or video frames, making their compu- tational costs prohibitive and necessitating modality-speecifie optimizations. Two main architec- tural approaches address these computational constraints. One involves architectural compression, where modules like Q-Former (InstructBLIP (Dai al.) (023)), perceiver resampler (OpenFlamingo Abstractor (Honeybee (Cha et al., (023)) distill high-dimensional visual inputs into compact representations, reducing the sequence length pro- cessed by expensive attention mechanisms. The other pathway utilizes hardware-awe
(Awadalla et al. 2)23)), and
optimization
strategies, such as FlashAteentin (Dao et al., 2022, Dao 2023), which optimize memory access patterns for accelerated self-attention computation without altering token quantities, achieving per- formance gains through algorithmic refinements and efficient resource utilization.
A.2 TOKEN PRUNING IN LARGE VISION-LANGUAGE MODELS
complementary approach to VLM efficiency focuses on reducing computational overhead through token sequence optimization. The quadratic computational complexity of Transformer attention mechanisms becomes particularly problematic when processing the extensive visual token se- quences typical in VLMs. Consequently, vision token pruning has emerged as critical research direction, which can be systematically categorized along multiple dimensions. Token reduction ap- proaches can be classified based their training requirements into training-free and training- -ased methods. Regarding implementation stages, these techniques operate across four primary phases: (I) visual encoder preprocessing, (2) LLM internal processing, (3) KV cache optimization, and (4) hybrid multi-stage approaches. Each pruning strategy involves two fundamental decisions: identi- fying which tokens to retain and aggregating useful features from discarded tokens.
Token Pruning At Vision Encoder ToME (Bolya al. 2023) establishes the foundation for
training-free token merging at the visual encoder stage, demonstrating effective feature-based token consolidation that influences subsequent works, including VisionZip (Yang et al. 2024). DivPrune (Alvar et al. 2025), ALaVA-PruMergy (Shang et al., 2024), and so on (Tong et al. [025 Liu et al.) 2025ap. These methods leverage visual feature similarity merge redundant tokens before they enter the language model, thereby reducing the computational burden on downstream processing stages.
Token Pruning Within LLM FastV (Chen et al. 2024) pioneers attention score- based token prun- ing within the L.M processing pipeline, establishing a training-free paradigm that guides later de- velopments such as VLM (Zhang et al. 2025b). PyramidDrop (Xing et al.) (2024). FastVLM (Vasu et al., 2024), and so on (Liu et al.) 2025a Ami et al., 2024 Khaki et al. 2025). ). These ap- proaches dynamically identify and remove less informative tokens based on attention patterns during inference, maintaining model performance while significantly reducing computational requirements.
Multi-Stage Optimization Strategies Comprehensive efficiency improvements have been achieved through multi-stage approaches that simultaneously optimize visual encoding, LLM pre- fill, and KV cache management during decoding. Representative methods include MustDrop (Liu et al. 2024b), LightLMM (Hu et al. 2025), and GlobalCom (Liu et al. 2025b). which coordinate token reduction across multiple pipeline stages maximize computational savings while preserving model capabilities.
Training-Bane Based Methods While training- rraiein-bbased approaches may exhibit reduced generalizability compared to training- free methods, they demonstrate superior performance preservation through aruning-aware optimization. Methods such as MD (Cai et al., 2024). ATP-LLaVA (Ye et al. 2024). Dynamic-LLL LLLVA VA (Huang al. 2024), TokenPacker (Li et al. 2024), and Twig VLM (Shao al.
15

---

Published as a conference paper at ICLR 2026
2025b) achieve competitive superior performance compared their full-token baselines through specialized training procedures that adapt the model to operate effectively with reduced token se- quences.
B DETAILED EXPERIMENT SETUP
In this section, we provide detailed experimental setups and algorithm implementations, along with supplementary experiments. These include the main experiment, position reconstruction experi- ments, and attention blocking experiments.
B.1 IMPLEMENT DETAILS
To ensure the reproducibility the results presented this paper, we have provided reproducible explanations for the key experiments in the paper.
Table Important packages in the Conda Environment.
Name
Version
datasets
4.0.0
1lava
1.2.2. post1
Imms-eval
0.3.4
qwen-vl-utils
0.0.14
sentencepiece
0.2.0
tokenizers
0.21.4
terch
2.6.0 cu124
torchaudio
2.6.0 + cu124
torchvision
0.21.0 cu124
transformers
4.54.0.dev0
The model weights used sourced from the Hugging Face community, specifically as follows:
Models
1.
LL5 7B
2. LLAVA-NexI 7B
3.
QWEN-2.5 VL
Datesets
The dataset used originates from the Imm-lab datasets.
B.2 MAIN EXPERIMENT
Nüwa's stage-1 pruning algorithm: To more clearly illustrate the proposed method, particularly the complex first-stage cropping, we provide pseudocode for the algorithm implementation here Algorithm
A detailed setup for different VLMs: Based on different models, adjust Nüwa's 's framework configuration, which still revolves around a two-stage process. The configuration calculation for Average Token (R,) is follows:
Lu+hi-
R =
(10)
L
i=b
where LI is the number of LLM layers and ro is the number of visual tokens entering each LLM
layer for processing. Our setting is shown in Table 10
Lext 7B Performance evaluation of our method LLAVA-Nex 7B, including VG task at Table II and VQA task at Table [2 Nüwa achieves ofate-of-the-art performance across multiple datasets.
16

---

Published as a conference paper at ICLR 2026
Algorithm Nüwa Stage-I Pruning
Require:
Input images X E BBCCxxX
Vision tower model (use CLIP Vrr-Large here)
Penalty threshold percentile = 0.25
Region configuration: top-n tokens per g x g region
Target tokens per image
Ensure:
Aggregated tokens T' <: Bxxxx (selected and aggregated features) Benchmark indices E Bxc (sorted positions of selected tokens)
1: B < X|: N < H W; 9, 9,n, k Pruncefy
Get setting based configuration
2: Hg, W t H/g.W/g
3: Ha, A Visiontower
Output hidden states and attentions
4: + + H.[L]
Ar + A[ 5: M M M<-EAll: :,
D Select layer L = -2 hidden states, shape B x (1 + NN x D
Metric map from CLS attention, shape
N
6: M2D < Reshape M to 2D
b based on grid separation Hg x W
7: Step1: Region-based Candidate Selection:
8: Unfold M2D into regions of patches
Shape B x Hg W x (g g)
9: for each region (i.j) do
10:
Cappend opn (Map]:, i, Select top-n indices of Local coordinates within region 11: end for
12: Sc <- Gather scores for C from M
13: Select top-k from Sc: +- Benchmark indices
> Sorted
Step2: Spatial Proximity And Similarity Construction:
& Patch grid Spatial Proximity Matrix based on distance threshold 16: H Average hidden states from mid-stage
15: Create PE HR xW,
Shape B (1 N) D > Patch tokens, normalized Similarity matrix, B N N b Aggregation weights with distance penalty
17: + +
Sim + P.
19: W + RReLU(Sim) @ P
Step3: Select Pillar Token:
21: S₁ + Gathee scores for I from L2ndrm
22: Ipillar +- --Quaniile((,,
23: W -SetV alues (W, Ipillar: Lpittar 0)
>> for Pillar Tokens, set 0 value for aggregation Weight W
24: Step4: Aggregation:
25: W, t- Gather W for benchmark indices I
Shape B x k N
26: Normalize W,
> Sum to per row, self-weight Aggregate to benchmark tokens
27: T t- W,- H[;, 1
28: return T,1
Table 10: Nüwa two-stage pruning setting on different VLMs.
Stage
Stage
Stage
Stage 2
Stage
Stage
Average Token 64
Average Token
Average Token
ILAVA-1.
112
224
32
48
Average Token
6%%)
Average Token 320 (11.15)
Average Token 640 (22.2%)
LLAVA-Neat
9%
1%
16%
2%
32%
4%
Average Token 25%
Average Token 50%
Average Token 75%
Qwen2.5
35%
42%
60%
66%
85%
70%
17

---

Published as a conference paper at ICLR 2026
Table 11: Refcoco series Benchmarks performance comparison On LLaVA-Next 7B, Best and second-best results are highlighted.
Method
Refcoco-test
Reftoco+-testA
Refcoco-testB
Refcocog-test
avg
Vanilla
77.73
76.34
57.25
71.05
100%
Average Tokens 94.4%
Nuwa
18.67
15.44
10:45
17.04
21.62%
Average Tokens 320 J. 88.9 %
Nüwa
38.67
33.86
25.30 36
31.49
45.68%
Average Tokens 640 J. 77.8 %
Nüwa
68.17
66.91
49.54
59.72
86.48
Table 12: VQA Benchmarks performance comparison On LLaVA-Next 7B. Best and second-best results are highlighted.
Methods
GQA
MMB
MME
POPE
SQA
TextVQa
avg
Vanilla
64.2
67.9
1846
86.4
73.2
61.3
100%
Average Tokens 160 94.4%
Sparse
51.2
52.1
1542
72.7
67.5
46.4
79.80%
VisionZip
55.5
60.1
1628
74.8
68.3
56.2
87.60%
Niiwa
60.0
60.4
1684
83
67.5
56.3
92.29 %
Average Tokens 320 J. 88.9% %
VL.
$7.7
63.2
1685
82.2
67.3
91.20%
VisionZip
59.2
63.1
1702
82.1
67.3
58.9
93.00%
Niiwa
62.3
63.2
1813
86.0
68.2
58.5
96.10%
Average Tokens 640 77.8%
VLM
60.3
65.8
1773
84.3 2
67.7
$7.8
95.30%
VisionZip
61.3
66.2
1787
85.9
68.1
60.2
96 96.90%
Nüwa
63.4
65.4
1879
87.2
68.6
98.10%
18

---

Published as a conference paper at ICLR 2026
QWEN-2.5-VL 7B Performance evaluation of our method on QWEN-2.5 VL 7B, including VG task at Table 13 and VQA task at Table 14 Nüwa achieves state-of ofae--of-the-rt performance across multiple datasets.
Table 13: Refcoco series Benchmarks performance comparison On QWEN-2.5 VL 7B. Best and second-best results are highlighted.
Method
Refcoco-testA
Refcoco-testB
Refcocos-testA
Refoncos-testi
Refencog-fest
Vanilla
92.56
85.16
89,02
79.15
87.24
100%
Tokens 75%
Nüwa 91.76
84.37
87.98
77.18
$6.87
98.8%
Averuge Tokens 50%
86.74
96.4%
Averuge Tokeus 25%
73:57
Table 14: VQA Benchmarks performance comparison On QWEN-2.5 VL 7B. Best and second-best results are highlighted.
Methods
GQA
POPE
SQAimg
MMB-en
MME
VQA.fext
avg
Vanilla
61.9
87.9
77.8
83,5
2347
82.2
100.0% %
Average Tokens 75%
Nüwa
60.41
87.52
77.98
83.13
2340
77.35
98.5%
Average Tokens
Nüwa
59.93
87.46
78.82
83,02
2330
76.03
98.1%
Average Tokens
Nüwa
58.4
87.06
78.58
82.47
2313
73.81
96.9%
B.3 SIMPLE BASELINE COMPARISON EXPERIMENT
To provide a more accurate comparison, we make minor modifications to the baseline settings for each category of methods. Specifically, only perform simple replacements on the pruning part. For random, we use torch. randprem with a random seed set to seed (44). For pooling, we employ adaptive pooling as shown in Algorithm accommodate dynamic token inputs and cropping rates. The pseudocode is as follows:
Algorithm 2 Adaptive Token Pooling Compress
Require:
Input visual token sequence X NWd
Original grid dimensions Hi,, W..) (such that Hi, Win N
Token retention ratio E [0, 1]
Ensure:
Pooled visual token sequence V' E Nowa
ktarget Nn X p.
Calculate target token
2: a = H Hin/Wa W
3: Wout = round( / /Kitarget/ 0);; Hoat round(a Woat)
D Calculate New grid based original
aspect ratio
4: Nout Hour x W
Final number of output tokens > Reshape to 2D feature map
5: Fgrid = Reshape X, (1,d, d, Hin, Win))
6: F Fpooted Pooling( F grid, output.size = (Houts Wout)) & This operation automatically handles
scaling and pooling
7: X' Flatten( (Fpooted)
8: return X"
19

---

Published as a conference paper at ICLR 2026
B.4 POSITION RECONSTRUCTION EXPERIMENT
Here, we conduct a detailed exploration of the positional reconstruction experiment. For methods like Visionzip that employ PERC, the positional embedding regenerated for the pruned complete sequence. Assuming ideal pruning, where the target is perfectly extracted, PERC effectively crops the image target and feeds into an LLM for grounding. This approach naturally cannot output bounding box based on the original image coordinate system. For PESP used in methods like Fast and Sparse VL.M, the original position embedding is employed, which performs relatively well but still compromises spatial integrity unless the target is located at the bottom-right corner of the image. This explains why such methods excel in grounding tasks. In contrast, RPME effectively scales the target to restore its original coordinate system, aiding localization of larger objects. However, it still fails with small objects, explaining why Visionzip's 's localization accuracy improves marginally after adopting RPME. Here, provide the pseudocode for RPME:
Algorithm 3 Relative Position Mapping Extension
Require:
Pruned visual token Indices E R (sorted ascending. k is the number pruned tokens)
Original Vision Position ID (s, (Start and End indices of full visual range)
Ensure:
New Position Embedding V E R (remapped embeddings for pruned tokens)
1: max, < [[: 1]
>. Last index as reference span Anchor first pruned token to start
2: new-pos [s]
3: for to 1
4: scaled + ceil (1bi) x e-s + mazz new-ppss.ppeed (scaled)
5:
6: end for
V' < get get.position.embeeddue
Retrieve embeddings for new positions; shape
k x d
8: return V"
B.5 ATTENTION BLOCK EXPERIMENT
To better understand the multimodal information interaction process within VLMs, we conduct at- tention blocking experiments three datasets, shown in Table We divid tokens into four main categories: (1) System tokens, (2) Visual tokens, (3) Text tokens, and (4) Last tokens, which repre- sent the token that will be used to predict. The attention-blocking experiment is conducted based on LLAVA-1.5 7B. The model is divided into four equal phases according to the decoder layers, with attention blocking applied to each phase.
Table 15: Attention Blocking Experiments LLAVA-1.5 7B.
Vision to Vision
Text to Vision
Last to Vision
Blocked
GQA
MMB
Refcoco-test
GQA
MMB
Refcoco-test
GQA
MMB
Refenco-test
Layer
Original
61.9
64.7
58.30
61.9
64.7
58.30
61.9
64.7
58.30
0-7
60.5
61.62
18.17
39.38
$1.63
54.75
62.55
64.26
57.85
8-15
55.2
57.98
2.64
53.32
46.39
14.11
62.47
64.08
2.01
16-23
58.44
62.37
19.27
61.04
62.45
15.94
61.32
64.26
64.52
24-31
58.81
62.37
19.31
61.12
63.05
15.08
62.12
64.17
59.10
B.5.1 BLOCKING ATTENTION FROM VISION TO VISION
In this setting, we block self self-attention between visual tokens in each phase. This disrupts the model's ability to construct global and local visual contexts within the LLM decoder.
Impact QA Tasks (GQA, MMB); Performance shows certain degree of decline, but it is not catastrophic. For example, on GQA, even when blocking the most critical layers 8-15 (perfor- mance drops from 61.9 to 55.2), the model retains most of its performance. Wayer-wise Sensitivity:
20

---

Published as a conference paper at ICLR 2026
Disabling intermediate layers (8-15) causes the most significant performance degradation. This in- dicates that intermediate layers the visual encoder are crucial for integrating low-level features extracted by earlier layers (e.g., edges, textures) into semantically meaningful object-level repre- sentations. Disabling early layers (0-7) or deep layers (16-31) has relatively minor impacts, likely because early features contain redundancy while deep features highly abstracted.
Impact on grounding tasks (Refcoco): Performance crashes dramatically. Particularly when blocking layers 8-15, performance plummets from 58.30 to 2.64. Layer sensitivity: Similar to VQA tasks, intermediate layers (8-15) are absolutely critical, validating our previous experimental observations
Summary: In stark contrast VQA tasks, localization tasks are extremely dependent spatial and contextual relationships between visual features. This can be explained by our spatial integrity hypothesis and the global coordinate reference system. For instance, to understand "the cup the left of the table", the model must first establish spatial adjacency between "table" and "eeup" via vision-to-vision attention. Blocking vision-to-vision attention reduces the image collection of disjointed, unconnected visual patches, preventing the model from forming a coherent understanding of the scene structure thus causing the grounding task fail completely.
B.5.2 BLOCKING ATTENTION FROM LAST TO VISION
In this setting, we block cross-attention from last tokens to vision tokens in each phase. This directly prevents the model from extracting visual information during decoding.
Impact on VQA tasks (GQA, MMB): Minimal impact on VQA Tasks. GQA performance varies slightly from 61.9 to 62.55 (within error margins, with a marginal improvement), and MMB de- creased marginally from 64.7 to 64.26.
Impact on grounding tasks (Refcoco): Catastrophic failure on VG tasks, particularly when blocking layers 8-15, where performance drops to 2.01. Moreover, blocking attention in subsequent stages actually improves accuracy.
Summary: The results demonstrate that answer generation in VQA tasks depends minimally on vision, disconnecting visual inputs any stage has a negligible impact on performance. VQA tasks primarily leverage global semantic information extracted from early multimodal interactions. In contrast, the result of grounding tasks contrasts with initial expectations. Integrating Text-to- Vision attention and gradient-weighted flows from Sec. 2.2 (last-to-text), hypothesize that visual grounding tasks rely more on spatially aware visual information processed in the model's mid- stage. The process then continually leverages textual cues to extract features, enabling integration of complex visual semantics within the visual modality (as analyzed in Sec. 2.2 Furthermore, blocking the Last-to- Vision attention in subsequent stages actually improves accuracy. We attribute this to the fact that subsequent visual information requires higher-level semantic integration to be transferred the output space, which disrupts the spatial information in later stages.
B.5.3 BLOCKING ATTENTION FROM TEXT TO VISION
In this setting, we block cross-attention from text tokens to vision tokens in each phase. This blocks multimodal information extraction.
Impact on VQA tasks (GQA, MMB): Performance degrades substantially, especially when blocking early layers (0-7), with GQA dropping from 61.9 to 39.38. Recovery occurs gradually as the blocked layer depth increases.
Impact on grounding tasks (Refcoco): Performance suffers severely across all blocked layers, particularly in mid-to-late stages (8-15, 16-23). This intriguing result highlights that grounding requires sustained multimodal interactions, yet early ones have minimal influence - consistent with anomalies in text-to-vision attention observed in Sec. 2.2
21

---

Published as a conference paper at ICLR 2026
Summary: These findings indicate that VQA tasks rely on text extracting global abstract features from visual inputs during initial processing for comprehension, with subsequent response genera- tion depending little on vision. In contrast, visual grounding tasks exhibit different reliance: they depend more on ongoing visual feature extraction in later stages, particularly when spatial under- standing emerges, rather than early features, which lack enough spatial information.
Finding Attention blocking experiments further reveal that grounding tasks primarily rely on two sets of information: 1. Abstract semantic information continuously extracted from text tokens visual tokens; Precise positional information was extracted from the model's 's mid- stage. Meanwhile, most VQA tasks predominantly depend on abstract semantic information extracted from text tokens to visual tokens during the model's early to mid-stages.
B.6 BENCIMARRS
We utilize several benchmarks that evaluate a model's ''s ability to understand static images, including their content, context, and associated textual queries.
GQA (Hudson & Manning, 2019) The GQA benchmark is structured around three core compo- nents: scene graphs, questions, and images. It enriches visual content with comprehensive spatial information and object-level attributes. Questions specifically designed to evaluate models' abil- ity to comprehend visual scenes and reason about diverse aspects of images.
The MMBench Benchmark evaluates models through three hier- archical levels of abilities. The first level (L-1) focuses on fundamental perception and reasoning capabilities. The second level (L-2) expands into six distinct sub- aub-abilities, while the third level (L-3) further refines these into 20 specific dimensions. This structure enables granular and comprehen- sive assessment of model's various capabilities. Additionally, MMB-CN the Chinese version of the benchmark.
MMBench (Liu al., 2023)
MME (Fu al. 2023) MME comprehensively evaluates models' perceptual and cognitive abil- ities across 14 subtasks. By employing manually constructed instruction-answer pairs and concise instructions, effectively minimizes data leakage and ensures fair assessment model perfor- mance.
MMMU Yuue et al., 2023) MMMU evaluates multimodal models on complex tasks requiring college-level knowledge and reasoning. includes 11.5K curated questions from exams, quizzes, and textbooks, spanning six disciplines: Art & Design, Business. Science, Health & Medicine, Hu- manities Social Science, and Tech & Engineering. Covering 30 image types like charts, diagrams, and chemical structures, MMMU challenges models with advanced perception and Nomain-specinn reasoning.
MM defines six siion-and-languaal (VL) capabilities: recog- nition, OCR, knowledge, language generation, spatial awareness, and math. These capabilities inte- grate to address range of complex multimodal tasks. MMVet evaluates 16 specific integrations of these capabilities through quantitative assessments.
MMVet (Yu et al., 2023)
POPE (Li al. 2023b) The POPE benchmark systematically evaluates object hallucination in models through series of binary questions about object presence in images. Using accuracy, re- call, precision, and Fl score as metrics, precisely measures hallucination levels across different sampling strategies.
ScienceQA (Lu et al. 2022) ScienceQA encompasses a wide array of domains, including natural, language, and social sciences, with questions hierarchically organized into 26 topics, 127 categories, and 379 skills. Through this comprchensive structure, it offers diverse scientific questions that fectively evaluate multimodal understanding, multi-step reasoning capabilities, and interpretability.
22

---

Published as a conference paper at ICLR 2026
SEEDBench (Li et al., 2023a SEEDBench comprises 19,000 buman-annotatee multiple-choie questions evaluating models across 12 distinct aspects. It comprehensively assesses capabilities in recognizing spatial and temporal patterns within both images and videos.
Tex (Singh al., 2019) TextVQA focuses on the integration of textual information within images. It evaluates model's 's ability to interpret visual elements and embedded text in images through tasks, requiring both visual and textual comprehension answer questions accurately.
VQA-V2 (Goyal et al. 2016) VQA-V2 evaluates models' visual perception capabilities through open-ended questions about 265,016 real-world scene images. Fach question contains 10 human- annotated ground truth answers, enabling thorough assessment of a model's ' ability to interpret and respond to visual queries.
These benchmarks are specifically designed to evaluate a model's ability to locate textual descrip- tions of specific objects or regions within image.
RefCOCO, RefCOCO+, RefCOCOg Yu al. 2016) These datasets are standard benchmarks for evaluating referring expression comprehension.
RefCOCO and RefCOCO+ provide large number of images annotated with referring expressions for objects. RefCOCO+ expands on RefCOCO by increasing the diversity of expressions and objects.
RerCOCOg is an extension that includes more natural, grammatically complex, and longer referring expressions, challenging models to understand more nuanced linguistic descrip- tions and their grounding in complex scenes. These datasets typically evaluate models on their ability to accurately locate the target object described by the expression.
C DETAILED FLOPS CALCULATION
We provide detailed derivation of the Floating Point Operations (FLOPs) for VLM with flexible, layer-wise token pruning mechanism.
C.I PRELIMINARIES AND NOTATIONS
We first define the key hyperparameters of the underlying LLM, which follows standard Trans- former architecture, These parameters, summarized in Table [6 form the basis of calculations.
Table 16: Model architecture parameters used for FLOPs calculation. Example of LLAVA-1.5 7B.
Symbol
Value
Description
C
32
Total number of Transformer layers
H
4096
Hidden dimension the model
1
11008
Intermediate dimension of the Feed-Forward Network (FFN)
(()
Variable
Sequence length (number of tokens) layer
Our calculation assumes that one FLOP corresponds to one which involves two floating-poit -oitt operations (a multiplication and an addition). This is standard convention in analyzing the computational cost of neural networks.
(MAC) operation,
C.2 FLOPS A STANDARD TRANSFORMER LAYER
The computational cost of a single Transformer layer is dominated by components: the Multi- Head Self-Attention (MHA) block and the Feed-Forward Network (FFN). We formulate the FLOPs for each, given sequence of length S.
Multi-Hee Head Self-Attention (MHA). The MHA mechanism involves four primary matrix multipli- cation steps:
23

---

Published as a conference paper at ICLR 2026
1. Q, K, V Projection:
Projecting the imput sequence of shape (S, H) into Query, Key, and Value matrices. This involves three separate weight matrices of size (H, H)
FLOPSOKY 3 S H H) 6S
2. Attention Score Calculation: Computing the dot product between Query and Key matri- ces, (S, SH
FL.OPSSerrs 2 S
3. Value Aggregation: Multiplying the attention scores (after softmax) with the Value matrix, (8,9) x H).
FLOPSyalues 2 5
4. Output Projection: Projecting the aggregated value matrix back to the hidden dimension via weight matrix of size (H,
FLOPSpmi = 2 S H 2SH
The total FLOPs for MHA block is the sum of these components:
FLOPsamu (S) 8 + 4
(11)
Feed-Forward Network (FFN). The FFN block consists of two linear transformations with a non- linear activation in between.
1. Up-Projection: Expanding the hidden dimension from H to I.
Ft.OPspp S -
2. Down-Projection: Reducing the dimension from back to H.
FLOP Sdown 2 -
The total FLOPs for one FFN block is:
FLOPS 4SHI
(12)
Thus, the total computational cost for single Transformer layer is: FLOPs (S) FLOPS (S) FLOPPsm(SS)
(13)
C.3 GENERALIZING TO DYNAMIC TOKEN PRUNING
Token pruning methods dynamically alter the sequence length S as data propagates through the model's layers. A given pruning strategy can be formally described by pruning schedule, which specifies the sequence length each layer. Let S() denote the number of tokens present at the input of layer I, where I {0, 1,............. 10.,,..,, 1}.
For baseline model without pruning, 5(1) constant for all layers. For a model with pruning, (()) becomes piece-wer wise constant function that is non-increasing with 1. For instance, method that prunes tokens before layer Ip would have 5(0) Smiial for / < 1p and (1)) Sprmne for I 2 lp-
To calculate the total FLOPPs for the entire LLM under specific pruning schedule, we sum the FLOPs of each layer using its corresponding input sequence length. The total computational cost is given by:
FTOP
FLOP Sayer (501)
(14)
This generalized Eq. 14 provides robust framework to accurately quantify the computational sav- ings of any token pruning method, regardless of where or how many times pruning is applied. Our analysis primarily accounts for matrix multiplications, which dominate the computational cost in large Transformers. The cost other operations, such as LayerNorm, activation functions, and soft- max, is considered negligible, consistent with prior work in the field (c.g., Kaplan al., 2020). By applying this methodology, we ensure a fair and insightful comparison across all evaluated methods in our experiments.
24

---

additional, non- Published as a conference paper at ICLR 2026
C.4 COMPUTATIONAL OVERHEAD OF PRUNING METRICS
A comprehensive analysis of token pruning methods must account for the computational overhead incurred by the pruning decision logic itself. While pruning reduces the overall FLOPs of the main computational graph, the process of selecting which tokens prune introduces negligible cost. This overhead is critical for holistic comparison, as a method with a computa- tionally expensive metric might offset the gains from pruning. this subsection, formalize the FLOPs required for various pruning metrics.
First, we define the computational cost for common metrics used by the methods under evaluation. Let S. be the number of vision tokens and Sy be the number of query tokens (e.g.. CLS or text token, where S 1).
Attention Score-based Metrie: This involves computing the dot product between Sy query vectors and S. key vectors, both of dimension H. The cost is:
PLtOSaat.soord (S..
Sq S.
(15)
Cosine Similarity Metric: This metric requires computing the pairwise cosine similarity between S. vision tokens. The dominant cost is the dot product for each of the (8.--1) pairs. The cost is approximated by:
2
FLOPScosine (S = (S.)² H
(16)
L2-Norm Metrie: Calculating the L2-norm for S. vectors of dimension requires ap- proximately:
FLOPSpomm (S,) = S.
(17)
Using these formulations, we summarize the total computational overhead for each pruning method in Table 17 The total FLOPs of a method is the sum of the main computation FLOPs (Eq. 14 and the overhead FLOPs detailed here.
Table 17: Summary of Computational Overhead for Different Token Pruning Methods. Sp, denotes the number of vision tokens the input of pruning stage and S², denotes the number of remaining vision tokens after stage i. The initial number of vision tokens is 576.
Method
Pruning Stage
Metric Used & Token Count
Overhead FLOPS Formula
PSATV
Stage 1
Last token attention to vision tokens (S.,1 576)
FLOPs soare ce 576)
Pdrop &
Stage 1
Text token attention to vision tokens (S.. 576 + 8t, = 66)
FLOPSam (1,376)
LLM
Stage 2
Text token attention to vision tokens (S.. -6 + S., = 30)
FLO (1,66)
Stage 3
Text token attention vision tokens (S.,a 30 St,a 17)
FLOPs (1,
VisionZip
Stage 1
CLS token attention to vision tokens (S.. 576 - S 64)
FLOP% (1,376) FLO (64)
Stage 2
Cosine similarity among remaining vision tokens (S".. 64)
Nüwa
Stage I
CLS token attention vision tokens (S.,, 576 + S 112)
FLOPs 376)
Cosine similarity & 1.2- 1.2-norm among remaining tokens (8), 112)
+2 S (112) FLOPs_cosine( 112.oooinn(122NNeeee (tex)
Stage 2
Text-Vision Cosine similarity (S, 112)
D VISUALIZE RESULTS
In this section, we supplement with additional visualization results, including some case studies, VAE and OCC visualizations from visual encoders (CLIP and SIGLIP2) and more visual represen- tations of the conclusions in the paper.
D.1 LLAVA
Attention flow
Here, we present the complete attention flow, encompassing all token types.
25

---

Published as a conference paper at ICLR 2026
Pook
Groundinng Task
VQA Task
Information Flaw Type
Lab Los
Walando-5p Spa ky
Figure 7: Complete Attention Flow for VQA and VG Tasks LLAVA-1.5 7B.
Text-Vsion Alignment Figures and 9 illustrate the layer-by-layer alignment process of vision and text tokens within the LLM. Complementing this, we present the layer-wise similarity heatmap, which together demonstrate the alignment status of multimodal data across different stages.
Figure 8: Two-Dimensional Visualization of Tision-Text Similarity.
26

---

Published as a conference paper at ICLR 2026
Lever
VT
VTuyee
VTune
Vi
Lover
VT
Input Text: The on the left
Figure 9: Visualization results of Layer-wise Vision-Text Similarity Heatmap.
Pillar Token We present three types of heatmaps the Figure 10 The second row displays heatmap with token L2 norm values, with the top and bottom rows showing tokens that made positive contributions during decoding for the VQA task and grounding task, respectively.
Reasoning
L2-Nort
TopS Dvorlap ratestus
Grounding
Figure 10: Visualization of "register" token and making significant contributions to the prediction wgrihient-weighted attention values).
27

---

Published as a conference paper at ICLR 2026
Here, conduct further visual analysis the "register" token. As mentioned in Section 3.1.3 these high-norm tokens receive significant attention both VQA and grounding tasks. We calculate the overlap rate between the top-3 tokens with the highest L2 norm in the "register" token and the top-3 tokens with the highest weighted attention scores in the VQA and VG tasks, finding a high overlap rate of 72% in VQA and 66% in VG.
Pruning Results We present some cropped visualization results Figure serves the integrity of the global space.
11
Our approach pre-
Figure 11: Visualization results of Pruming results.
D.2 CLIP
We provide additional visualizations of attention maps and object-centric similarity maps during the CLIP processing (Figure 12
Layer
Laver
Layer
Layer 13
Layer
Layer 17
Laver
Map
Object Sim-Map
Object
Centric
Sim-Map
Figure 12: More visualization results processed by CLIP-VIT.
28

---

Published as a conference paper at ICLR 2026
D.3 SIGLIP
Since SIGLIP lacks a CLS token, the attention map visualization is based on the attention values received by each token. We provide some sample visualization results (Figure 14) and the metric VAE and OCC (Figure [3]. By examining visual results and metrics such as OOC, it can be observed that compared to CLIP, SIGLIP exhibits later phase of ine-grained feature extraction, and its overall trend is less pronounced than that of CLIP.
Figure 13: The VAE and OCC metric form SIGLIP2.
Layer 0
Laver
Layer
Layer
Layer
Layer
Layer 23
Ata Map
Object Centric
Sim-May ap
Alla
Map
Object
Centric
Sim-Map
Map
Object
Centric
Sim Sim-Map
Figure 14: More visualization results processed by SIGLIP2.
E
CASE STUDY
In this subsection, present the results of vision token pruning and localization under several figurations of the Nüwa framework. Specifically, we evaluate three primary settings: 1) token selec- tion without regional partitioning, 2) employment of Relative Position Mapping Extension (RPME).
29

---

Published as a conference paper at ICLR 2026
and 3) the Nüwa with full configuration. Furthermore, we analyze failure case wherein localization inaccuracies arise from the model's ' inherent misinterpretation the text and the designated task.
Nuwa
Niiw with rich spatial information
Nüwa-wlo Region
Locate the man on the right in this image and output the bounding box coordinates in JSON.
Niwa without Region selection.
Nüwa
Accurate target positioning.
Nüwa-wll Region
Skewed toward the upper left.
Locate the person the button left this image output bounding box coordinates JSON.
Figure 15: A comparison between the original Nüwa setting and the token selection setting without regional partitioning. Red and green bounding boxes represent the prediction and ground truth, respectively.
Regional partitioning provides a significant advantage As illustrated Figure 15 the configu- ration employing regional partitioning demonstrates superior preservation of spatial integrity during token pruning, enabling the observation of all four corners of the image. Conversely, the failure to retain sufficient spatial information appears to impair the model's perception of global positioning, resulting in predictions that are systematically biased towards the top-left corner. The underlying reasons for this phenomenon discussed in Section 2.3
30

---

Published as a conference paper at ICLR 2026
Nüwa-wlo Region
Nüwa
Region selection.
Nüwa RPME Nüwa with RPME instead of region selection.
Locule man on right image and output the bounding box coordinates ISON.
Nuwa-wid wio Region
Skewed toward the upper loft.
Nüwa RPME
Locate the person the button left in this image and output the bounding box coordinates in ISON.
The issuc has been somewhat alleviuted, but it still lacks precision.
Figure 16: comparison between the token selection setting without regional partitioning and the setting with RPME. Red and green bounding boxes represent the prediction and ground truth, spectively.
RPME can mitigate the loss of spatial information. As shown in Figure 16 the prediction results from the RPME-enabled setting are more accurate. Specifically, the localization no longer exhibits a consistent bias towards the top-left corner. However, while the overall position is correct, the shape (i.c., aspect ratio) of the predicted box is problematic. This may be attributed to the image 'distortion' introduced by the stretching operations in RPME, which could interfere with the model's understanding of the object's geometry.
Failure Case
Successful Cases
Locate the baseball cutcher
Locate the fruit in the upper right
Locate the sandwich
Locate the person in the ned shirt
Figure 17: Localization failure attributable to the model's comprehension. Red and green bounding boxes represent the prediction and ground truth, respectively.
31

---

Published as a conference paper at ICLR 2026
Localization Failures Attributable to Model comprehension. As shown in Figure another potential failure case for localization is presented, which arises from the VLM's incorrect under- standing of the target. In the successful case on the right, observe clear positive correlation between the model's predicted bounding box and the regions where it assigns high attention to the vision tokens. In the failure case the left, however, the model mistakenly identifies the person in the distance wearing a baseball glove as the target, assigns extremely high attention to that region, and consequently produces incorrect localization.
32


Reviews:
[{"review_id": "ZGouBMvny4", "reviewer": "Reviewer_rCym", "summary": "This paper tackles spatial degradation in vision token pruning for VLMs and proposes RPME to preserve spatial integrity during token reduction. The idea is clear and grounded in practical bottlenecks for high-resolution inference. Experiments across grounding and VQA benchmarks show meaningful gains, especially on spatial tasks, and the ablations strengthen the causal claims.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation grounded in real multimodal deployment constraints.\n\n2. Strong empirical gains on grounding datasets; convincing ablations.", "weaknesses": "1. The paper primarily evaluates on LLaVA-1.5-7B (and NeXT-7B), but does not include stronger or more modern VLM backbones such as Qwen-VL, InternVL. Since the field is rapidly standardizing around Qwen-VL and InternVL as competitive baselines, the absence of these comparisons leaves uncertainty about generalizability across architectures.\n\n2. Lack of comparison with stronger baselines, such as PyramidDrop and Vscan.\n\n[1] PyramidDrop: https://arxiv.org/abs/2410.17247\n[2] Vscan: https://arxiv.org/abs/2505.22654\n\n3. Limited discussion of failure modes or robustness to cluttered scenes and occlusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fMsaGozlys", "reviewer": "Reviewer_DEDu", "summary": "This paper presents a two-stage token pruning framework, Nuwa, where the first stage focus on reducing the number of visual tokens by aggregating them via a proximity matrix, and the second state further prune the tokens with text guidance. The proposed Nuwa method is able to reach SOTA performance on multiple VQA benchmarks as well as visual grounding tasks compared to prior methods.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a thorough and insightful analysis of the challenges, pitfalls, and properties of token pruning in VLMs.\n- The multiple metrics and visualizations in the analysis provides a solid support for the findings.\n- The proposed Nuwa is compared with multiple current arts and is able to provide a higher pruning quality that retain the VQA and grounding accuracy with aggressive pruning.", "weaknesses": "- It is not clear how those findings have the contributed to the design of the Nuwa algorithm.\n- In table 6, the results of FEATHER is only reported for the case of 192 tokens.\n- The visual pruning part remain largely depend on the [CLS] token, so it is hard to apply the proposed method on vision backbones without [CLS].\n- The text-guided token pruning is pretty strightforward and is not utilizing the prior insights.\n- It is a two stage design that requires explicit text-guidance.\n- The comparison against Visionzip is not very fair as Visionzip is a text-agnositc method which does not leverage text-guided pruning. It would be better to include the ablaiton study where the first-stage only Nuwa is compared with the Visionzip on several benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QO0NYgpSgS", "reviewer": "Reviewer_6Jqk", "summary": "This paper introduces a novel method for VLM token pruning. The key insight motivating the approach is the recognition that preserving spatial information remains challenging yet is crucial for maintaining performance on grounding tasks after token reduction. The proposed method successfully leverages this insight to achieve superior performance and efficiency across a series of diverse VLM tasks.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper offers a comprehensive examination of VLM token pruning, which yields valuable insights regarding the weaknesses of existing methods. \n* A novel and intuitive token pruning method is proposed and shown to be effective across a series of tasks.", "weaknesses": "* While the authors provide experimental results across diverse tasks, the selection of VLMs is limited. The generalizability and robustness of the proposed method should be confirmed by replicating the key results on a broader range of VLM architectures currently employed in the field.\n* Furthermore, several tables and figures are currently too small and visually dense, making them difficult to read and interpret. The authors are encouraged to enlarge these elements and ensure that all text and data points are clearly legible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tswZHy2L1Q", "reviewer": "Reviewer_MQfG", "summary": "The paper studies why vision token pruning in vision-language models hurts visual grounding much more than VQA, and proposes a new pruning framework called Nüwa to fix it. The authors analyze the VLM visual processing pipeline and argue that grounding depends on a global spatial reference frame built from positional embeddings and mid-layer vision tokens, while most pruning methods destroy this frame. They present a two-stage pruning method: (1) before the LLM, partition vision tokens into regions, pick salient tokens, and aggregate nearby tokens into them while keeping global spatial anchors intact; (2) inside the LLM mid-layers, prune again using text–vision similarity to keep only task-relevant visual tokens. They claim large speedups while keeping VQA performance and sharply improving grounding retention.", "questions": "1. How does this paper differ from the prior two-stage pruning approaches, and what are the novel contributions?\n2. The RefCOCO results reported for the baselines differ notably from those in other works. Could the authors clarify their experimental settings and explain the reason for this discrepancy?\n3. Has any failure analysis been conducted?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The analysis of attention/gradient flow, visual attention entropy, and object-centric cohesion across vision and LLM layers is thoughtful. It supports the claim that grounding relies on mid-stage spatial structure, not just final semantic alignment. This is valuable diagnostic science for the community.\n2. Insight about position embeddings. The taxonomy of PE handling and the position reconstruction experiment is convincing.\n3. Method makes sense relative to the diagnosis. Stage 1 is explicitly designed to preserve a dense global spatial scaffold. This is more principled than naive semantic clustering. Stage 2 then does task-aware pruning in LLM space, which matches the observed task divergence in mid layers. The story is coherent end to end.\n4. They compare to random and pooling. They acknowledge that pooling is surprisingly strong and then explain why (implicit topology preservation).", "weaknesses": "1. The idea of a two-stage token pruning pipeline is not new,  similar observations and motivations have appeared in prior and concurrent works [1-9]. The authors should more carefully compare against these approaches, and make explicit what is actually novel here.\n2. The RefCOCO results reported for the baselines differ notably from those in other works for example [2]. I may be misunderstanding the experimental settings, so the authors should clarify their setup and explain the discrepancy.\n3. No failure analysis, for example, whether the text-conditioned Stage 2 introduces text bias (the model may discard visual evidence that contradicts misleading text).\n\n[1] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction.\n[2] VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models.\n[3] GLDS: Global–Local Diversity Selection for Scalable Token Pruning in Vision–Language Models.\n[4] Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?\n[5] Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration.\n[6] Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference.\n[7] ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming.\n[8] Cross-modal Information Flow in Multimodal Large Language Models.\n[9] LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)