Each Complexity Deserves a Pruning Policy
Authors: Hanshi Wang
yuhao xu
Zekun Xu
Jin Gao
Yufan Liu
Weiming Hu
Ke Wang
Zhipeng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language Models
Pruning
Primary area: applications

Abstract: The established redundancy in visual tokens within large vision–language models (LVLMs) allows for pruning to effectively reduce their substantial computational demands. Empirical evidence from previous works indicates that visual tokens in later decoder stages receive less attention than shallow layers. Then, previous methods typically employ heuristics layer-specific pruning strategies where, although the number of tokens removed may differ across decoder layers, the overall pruning schedule is fixed and applied uniformly to all input samples and tasks, failing to align token elimination with the model’s holistic reasoning trajectory. Cognitive science indicates that human visual processing often begins with broad exploration to accumulate evidence before narrowing focus as the target becomes distinct. Our experiments reveal an analogous pattern in LVLMs. This observation strongly suggests that neither a fixed pruning schedule nor a heuristics layer-wise strategy can optimally accommodate the diverse complexities inherent in different inputs. To overcome this limitation, we introduce Complexity-Adaptive Pruning (AutoPrune), which is a training-free, plug-and-play framework that tailors pruning policies to varying sample and task complexities. Specifically, AutoPrune quantifies the mutual information between visual and textual tokens, and then projects this signal to a budget-constrained logistic retention curve. Each such logistic curve, defined by its unique shape, is shown to effectively correspond with the specific complexity of different tasks, and can easily guarantee adherence to a pre-defined computational constraints. We evaluate AutoPrune not only on standard vision-language tasks but also on Vision-Language-Action (VLA) models for autonomous driving. Notably, when applied to LLaVA-1.5-7B, our method prunes 89% of visual tokens and reduces inference FLOPs by 76.8%, but still retaining 96.7% of the original accuracy averaged over all tasks. This corresponds to a 9.1% improvement over the recent work PDrop (CVPR'2025), demonstrating the effectivenes.  Code is available at https://github.com/AutoLab-SAI-SJTU/AutoPrune.

Each Complexity Deserves a Pruning Policy
Hanshi Wang Yuhao Xu², Zekun Xu Jin Gao
Yufan Liu 1,2,5 Weiming Hu 1,2,5,6 Ke Wang', Zhipeng Zhang 3,4f
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA
School of Artificial Intelligence, University of Chinese Academy of Sciences
8 Lat School of Artificial Intelligence, Shanghai Jiao Tong University *Anyverse Intelligence
*Bejjing Key Laboratory of Super Intelligent Security of Multi-Modal Information 6 School of Information Science and Technology, ShanghaiTech University "KargoBot
(hanshi wang..cy... cv, zhipeng. zhang. (rann..vv)@ooo eo)toollokk. com; jin. ain.gao@nlpr. aa.ac.a.............. cn
Abstract
The established redundancy in visual tokens within large vision-language models (LVLMs) allows for pruning to effectively reduce their substantial computational demands. Empirical evidence from previous works indicates that visual tokens later decoder stages receive less attention than shallow layers. Then, previous meth- ods typically employ heuristics layer-specific pruning strategies where, although the number of tokens removed may differ across decoder layers, the overall pruning schedule fixed and applied uniformly to all input samples and tasks, failing align token elimination with the model's 's holistic reasoning trajectory. Cognitive science indicates that human visual processing often begins with broad exploration to accumulate evidence before narrowing focus as the target becomes distinct. Our experiments reveal an analogous pattern in LVL.Ms. This observation strongly sug- gests that neither fixed pruning schedule nor heuristics layer-wise strategy can optimally accommodate the diverse complexities inherent different inputs. To overcome this limitation, we introduce Complexit--Adaptin Pruning (AutoPrune), which is a training-fres plug-and-play framework that tailors pruning policies varying sample and task complexities. Specifically, AutoPrune quantifies the mutual information between visual and textual tokens, and then projects this signal to a budget- constraied logistic retention curve. Each such logistic curve, defined by its unique shape, is shown to effectively correspond with the specific complexity pre-defined computa-
of different tasks, and can easily guarantee adherence tional constraints. We evaluate AutoPrune not only on standard vision- languaee tasks but also Vision-Language-An Notably, when applied to LLaVA-1.55 VA- 1,5-BB, our method prunes 89% of visual tokens and reduces inference FLOPs by 76.8%, but still retaining 96. 6.7%% of the original accuracy averaged over all tasks. This corresponds to 9. 1% improvement over the recent work PDrop (CVPR 2025), demonstrating the effectivenes. Code is available at https:///githu com AutoLab SAI---SAAI-- SAI- AutU/uuu
(VLA) models for autonomous driving.
Introduction
Vision and Language Models (VLMs) have rapidly emerged the backbone of modern multimodal systems, powering tasks such as image captioning [12 3), visual question answering (VQA) 54.5)
and multimodal dialogue [6] 77) Recent extensions embodied intelligence like the wildly deployed
autonomous driving system, exemplified by Vision-Language Action frameworks [8] g 10, further
couple these perceptual capabilities with driving control, permitting end-to-end reasoning. However,
"This work was completed during Hanshi's remote internship at SJTU and co-mentored by Prof. Zhipeng CCorespponiigg author.
Zhang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tokenizing high-resolution images or video for LLMs yields excessively long visual sequences, which in turn create memory and latency bottlenecks and make efficient pruning essential for real-time use. Among the diverse methods aimed at boosting the efficiency of VLMss, training-free free token pruning
stands out as significant technique because of its simplicity [1 12 13 14
16
A review of related literature reveals a prevailing understanding that the informational contribution of visual tokens substantially diminishes during the later stages of the VLM decoder [18]. Existing training-free ffree token pruning methods reflecting this principle typically adhere to predetermined fixed pruning schedules 19 or alternatively they employ layer specific heuristics [2] 21 yet without explicit adherence to a global computational budget. However for reasoning intensive tasks which necessitate iterative inference and dynamic cross modal fusion. such fixed pruning policies lack adaptability and cannot meet the sample specific and task specific demands. Our experiments in Sec. 3.2 underscore this limitation showing that saliency patterns and consequently token importance vary significantly with the input image and the posed query. While certain layer wise heuristics attempt to prune tokens differently based on factors like decoder layer depth, their handcrafted nature presents challenges as these designs often fail to guarantee adherence to target token count FLOPs budget without extensive manual tuning nor do they provide robust evidence of generalizability across diverse scenarios. This context therefore naturally motivates a critical question that "Is it feasible to develop a pruning methodology that A dynamically adjusts to the varying complexities of individual samples and tasks while 2 readily adhering a predefined computational budget and concurrently upholding principles of simplicity and broad generalizability"
Since humans excel at complex visuolinguistic reasoning, we tend to find the answer from neuro- science first. The studies in which reveals that for clearly expressions (simple samples and tasks), ventral visual stream and temporal-language areas rapidly converge on the referent, yielding single sustained fixations 222 23 24]. In contrast, ambiguous or indirect descriptions (complex samples and
tasks) engage dorsolateral prefrontal and parietal networks, with the prefrontal cortex maintaining
competing hypotheses while the dorsal attention system drives iterative gaze shifts (25 26 Mirroring
this -expitiooeelliiitiionnooooo cycle, our analysis of VLMs in Fig. also proves that simple samples and tasks induce a rapid collapse of cross- cross-moda modal attention within early layers, whereas complex samples and tasks sustain diffuse attention and exhibit pronounced inter-layer saliency fluctuations. These observations demonstrate that fixed pruning schedules, whether aggressive or conservative, cannot satisfy the varied demands of reasoning.
Leveraging these insights, our work primarily proposes Complexity-Aappie Pruning (AutoPrune). framework that endows each input with an individualized pruning policy. To achieve this goal, we believe the core challenge lies in quantifying sample complexity in a manner and representing the latent thought process compatible with training-free deployment and adherence to a fixed computa- tional budget. Drawing on neuroscientific evidence that tightly coupled cross-modal signals shorten human reasoning paths, we measure the mutual information between early-layer visual and textual tokens to identify input complexity. A high value implics a simple sample with an easily localized answer, while low value flags complex sample requiring broader exploration. We map this scalar complexity estimate onto logistic retention curves that mimic the human explore-commit-stabiillie pattern observed in eye-tracking studies. Each curve represents distinct pruning policy, where the values at different points along the curve dictate the degree of token pruning at varying depths of the decoder. The curve's 's slope and inflection point are modulated linearly by the mutual information score, yielding aggressive front-loaded pruning for simple samples and conservative, late-onset pruning for complex ones. To guarantee the pre-defined cost, we analytically integrate each curve, rescaling it so that the area under the curve equals a user-specified token or FLOPs budget.
Due to its simple and and-play architecture, our AutoPrune can be scamlessly integrated into variety of VLM and VLA models, including LLaVA- (27). LLaVA-NeXT 28). and Senna [29] for autonomous driving. Experiments standard vision-language benchmarks, autonomous driving scene understanding and planning demonstrate that AutoPrune consistently outperforms existing vrining-free methods across broad range of pruning ratios. For instance, when applied to LLaVA- 1.5-7B, AutoPrune prunes 89% of visual tokens and reduces inference FLOPs by 76.8%, but still retaining 96.7% of the original accuracy averaged over all tasks. This corresponds a 9.1% improvement over the recent work PDrop (CVPR2025) [3]
Our contributions include: We present cognitive neuroscience aeuroscience-inspreed analysis that system- atically links sample and task complexity with token retention decay and inter-layer fluctuations
2

---

in cross-modal attention. V We propose AutoPrune, training- free complexity-adapiie pruning framework that computes mutual information from risual-texture attention and maps it to budget- constrained logistic retention schedule, assigning each sample and task customized pruning curve under any specified token or FLOPPs budget. We demonstrate the generality of our approach by integrating AutoPrune into multiple VLM and VLA models and benchmarking against diverse baselines. Extensive experiments show that our method consistently outperforms prior state-of-the-art approaches across various tasks and reduction ratios.
2 Related Work
Vision-Language Models (VLMs). VLMs have achieved significant progress in integrating visual and textual modalities, enabling sophisticated tasks such as image captioning 12 3. visual question answering (VQA) 55, and multimodal dialogue [ 7. Their broad world knowledge has spurred embodied applications and led to VLA models [8 22. which add action generation for control, with autonomous driving as a representative application. A typical design uses a visual encoder for features and an LLM for multimodal reasoning and output. This pairing grants visual perception but expands inputs into long token sequences. High resolution images [30 and video BI 32] amplify memory and latency. Consequently, optimizing the inference efficiency of these powerful models is a critical prerequisite for their practical deployment in diverse methods aimed at boosting the efficiency of VLMs, token pruning stands out significant technique, broadly divisible into maining-based and training-free paradigms.
real-world scenarios. Among the
Token Pruning. In pursuit of task-optimized efficiency, one prominent line of research focuses on training-base -bsed pruning methodologies. These approaches necessitate supplementary training or fine- tuning stages to instill task-specific pruning behaviors, potentially enhancing performance metries on target applications [1] 33 34 35). Training-fris pruning avoids retraining and can be applied directly to pretrained models [8]. Methods commonly grouped by pruning stage. Pre decoder pruning selects a subset visual tokens with unsupervised similarity or lightweight scores before the LLM, as in V [36] and FasterVLM [9.. Intra decoder pruning removes tokens during inference across LLM layers using preset layerwise schedules or attention statistics, as in PyramidDrop [13] and VL [14. Despite their appeal and diverse application points (pre- intra-decoder), critical unresolved shortcoming persists in existing training-fre free methods. Specifically, regardless of the pruning stage, the vast majority employ fixed pruning policies [1] [2]
[1] 16 [7]. The rigidity of such static approaches proves problematic. Our experimental observations within VLMs reveal that, even for the same input image, the pattern of token importance varies dynamically across decoder layers depending on the specific question posed. Consequently, a fixed pruning policy is inherently ill-suited to adapt to these variations that are contingent on both input query and processing depth within decoder, underscoring the need for more adaptive, context-aware pruning strategies.
[3] 14
3 Method
3.1
Preliminaries
We cast token pruming as a constrained optimisation problem whose decision variables specify 1) how many visual tokens survive in each transformer block, 2) how to select specific tokens for retention, and 3) how to revive discarded tokens. All three decisions are optimised jointly under a global constraint on total computation. Concretely, number of tokens preserved at each layer i, the token-sclection policy as governing token retention, and the token-revival policy as p dictating how discarded tokens are revived and remapped. For data distribution D and task loss E, the expected loss is defined as:
denote the token-allocation policy as & specifying the
C(S, x, p) = E(V,T.y)~ fyy, fo T: n. p)).
(1)
where V e N. xd and T E N xd are the image and text tokens, y is the ground- and fo is the vision language model. We aim to minimise L subject to global compute budget Cmaxi
L
min C(E, p) s.t. T.
Σ (((E)) &
Cmax:
(2)
where p) measures the computational cost incurred by pruning and potential revival at layer i.
3

---

Figure 1: Layer-wise Visual-Textual Interaction Patterns. By visualizing cross-modal attention at layers 2, 4, 8 and 16 of the VIM, observe that for tasks requiring only object identification, attention rapidly converges on the salient region and remains stable, whereas for reasoning-intensive tasks attention shifts progressively across layers.
We focus on optimizing the token-allocation policy E. which governs how many tokens are preserved at each layer. Prior approaches fall into two camps: 1) uniform pruning schedule applied identically across all tasks, which cannot adapt to varying visual-textual demands, and 2) per-layer schemes that tune pruning independently but lack mechanism to enforce global compute budget, often resulting in insufficient pruning and limited speedup. In contrast, our method dynamically allocates token budgets in global manner which rigorously satisfying the overall computation constraint, thereby unifying adaptability and acceleration.
3.2 Neuroscience Inspiration and Analysis
Neuroscientific research [2 23 24 25 26
shows that the neural resources engaged in visuolinguistic processing scale with task complexity. When the text unanbiguously specifies a target (simple lasks), object-selective regions in the ventral visual stream cooperate with temporal-languaa areas to form a rapid, stable representation, and attention remains anchored to the relevant image region. Indirect or ambiguous links (complex tasks) clicit additional activity in dorsolateral prefrontal and parietal control networks, the prefrontal cortex maintains competing interpretations while the dorsal attention system reallocates gaze among candidate referents. Eye-tracking corroborates this shift, revealing single, sustained fixations in simple conditions but iterative scans in complex ones, which provides evidence of top-down guidance by language riven inference.
Guided by these neuroscience cognitive insights, we conduct a fine- tne-grained analysis of coosl attention in vision-language models. As illustrated in Fig. []] we analyze the model's behavior on both simple and complex tasks and identify two key findings.
Tample Adaptive Token Number Decay, Fig. (left) demonstrates that in simple tasks and samples (here in this example, "task" indicates the asked question, "sample" denotes the input image), where the referring expression unambiguously identifies the target, cross-modal attention outside the relevant region collapses within the first few layers. At that stage, only tokens corresponding to the target remain active. In contrast, as shown in Fig. Tright). for complex tasks and samples that demand non- non-trivial visual inference, attention over image tokens decays gradually and remains widely dispersed in the initial layers, indicating the model's uncertainty about where pertinent evidence resides. Hence, simple tasks and samples permits aggressive pruning at shallow depths, whereas reasoning-intensive prompts benefit from postponing token removal until deeper layers. These findings motivate task-adaptive pruning policy that dynamically models global token trajectories, preserving a wide set of tokens when alignment is ambiguous and confidently discarding irrelevant tokens at shallow depths when alignment is clear. However, discussed before, existing methods cannot simultaneously capture both the gloabl token trajectories and the compute budget, limiting their ability to reconcile adaptivity with efficiency.
Inter-Layer Saliency Position Variation. simple tasks, cross-modal attention converges by an early layer and remains stable thereafter (Fig. [ left), indicating that further exploratory inference provides no additional benefit. In complex lasks, the saliency of individual visual tokens varies across layers. Specifically, as shown in Fig. rrghtt line 1), the model initially attend to a chalkboard, shift focus to surrounding regions in intermediate layers, and return to the true target by layer 16. These fluctuations reveal an intrinsic search mechanism that probes alternative regions under weak initial

---

cues and progressively refines attention as higher-order features emerge. In summary, for simple tasks, aggressive pruning can be applied once attention has converged. for complex tasks, maintaining a larger token set across layers is essential to support ongoing inference and resolve ambiguity.
These findings indicate that effective pruning must follow dynamically modeled, globally coherent trajectory. However, existing approaches either lack a mechanism to capture such dynamics or fail to achieve global pruning trajectory. Our framework employs complexity pruning that adapts to each sample, task, and layer saliency position variation to produce a dynamic, globally consistent trajectory within fixed computational budget.
3.3 Complextty-Adaptive Pruning
Then the next questions are: (I) how to formulate a reliable indicator for assessing the sample and task complexity? and (2) how to translate this indicator into a policy that not only formally simple but also allows for straightforward management of the overall computational budget? In pursuing an answer, we again drew inspiration from neuroscience.
Neuroscientific evidence indicates 37 that in semantically congruent audiovisual contexts informa- tion flow from early visual areas language integration regions is both stronger and more direct, reflecting enhanced bidirectional coupling. In incongruent contexts higher order regions such as the prefrontal cortex are recruited to resolve the mismatch, which attenuates direct exchange between lower level sensory areas. This contrast implies that the extent of information exchange in the initial layers can serve a proxy for complexity. To quantify this, for the first question above, we compute the mutual information between visual tokens and textual tokens. Specifically, high mutual information denotes a direct, "simple" task that allows aggressive pruning, whereas low mutual information signifies an indirect, "complex" task that demands conservative token retention.
show that, in the image-based guestion-answeringg tasks, the time course of human fixations is well described by a logistic (S-shaped) curve. Specifically, an initial epoch of broadly distributed gaze is followed by a steep rise in -igeetiiiatiion probability once task-relevant evidence exceeds a cognitive threshold, after which fixations stabilise on small region of the scene, This appears in purely visual settings and in cross-modal variants that combine visual scenes with spoken or multisensory cues, indicating
Extensive eye-tracking and electroencephalographh researches 25
38
39
40)
conprr--commt-stailiiiiii. pattern
pondality-generte principle of attention allocation. Inspired by this finding, for the second question above. we define logistic retention function that emulates a human- like, iterative inspection process, applying aggressive early pruning in simple tasks with strong alignment to isolate key tokens and reserve budget for deeper analysis, whereas in complex tasks we prune conservatively at first to avoid discarding critical information prematurely. Besides the neurological explanation, we select the logistic function due to its inherent simplicity and the facility with which its shape can be modulated by adjusting hyperparameters. These hyperparameters can be efficiently derived from the indicator we introduce. As illustrated in Fig. [ steeper slopes indicate lower mutual information and correspond to lower sample scores, thereby validating our method.
Subsequently, we elaborate on these two essential components.
Mutual Information for Cross-Modal Alignment. To translate the qualitative insights from our neuroscientific analysis into a quantitative signal that can steer pruning, we require a scalar measure of how tightly textual prompt constrains the visual scene. Mutual information naturally fulfils this role because it captures the reduction in visual uncertainty provided by the text and is directly computable from cross- cross-modal attention. Specifically, we quantify the mutual information 41 between the visual tokens V and textual tokens T by
P(u, 4g) p(u1) p(t))
I(V. T)
((n tj) log
where Ny and N; denote the numbers of visual and textual tokens, respectively. We estimate the joint and marginal probabilities by interpreting the transformer's softmax-normalized attention weights oji (from text token lj to visual token v;) as probabilities p(u; (j) under a uniform text prior, yielding
1 N.
p(G,tt) ti)
ajai p(er) = Σ p(n..))) p(t)) N.
(4)
5

---

This approach leverages the fact that attention's 's softmax outputs form a valid distribution 42 43 allowing direct computation of mutual information from the attention maps.
As shown in Fig. 2 a large value of T(V, T) indicates that the textual prompt sharply constrains the space of plausible visual interpretations, allowing the model to confidently localize the relevant region early and perform aggressive front-loaded pruning of non-essential visual tokens (orange curve). Conversely, a small mutual information means weak or indirect correspondence, so the network must preserve multiple visual hypotheses across layers, leading to gradual, unstable reduction of attention (blue curve) 44 45). Thus, mutual information provides principled scalar proxy for task specispecific text-vision alignment, guiding the pruning schedule introduced in this work.
Budget-Constrained Logistie Retention. For a question answer pair q, the policy is defined as,
Nint
fa(x) = 1 exp((ç(( zg)
where Minit is the initial token count, Kg controls the steepness of the retention decay, and z denotes the layer at which the retention rate falls to half of Ninit-
Practical deployment requires an fixed compute budget Cinax prior methods approximate this cost by reporting the average per-layer token number 13). Based on their method and to meet requirement of compute budget we integrate (xx over the depth domain [0, LJ.
(x(x)= = / fa(() dx = Ninit (x - xi)) In(1 + e
(6)
kn
and compute the area Ig FL( - (O). We then renormalise the curve by
Cmax/ In
fa(x)= 1 (Kg (x x1i))
(7)
so that Jo fq( dr = Cmax- This procedure preserves the shape of curve while guarantee- ing identical computational complexity across tasks. Since network layers are discrete, and token counts must be integer-valued. We there- fore evaluate (i) at each layer E {O..... L} to the nearest integer, and adjust a global scale factor s until E s (t) = Cmax with bi- nary search. The same procedure applies when the budget is expressed in FLOPs rather than token counts. Let c(x) denote the per-layer cost incurred by retaining fq(x)) tokens. Replacing token number by Jo" c(f((x)) dx and rescaling as above yields a schedule that respects any de- sired FLOPs target.
Bedd 6.3)
Midum a1
10
05
Retention Ratio
64
Layer Index
Logistic retention curves on the TextVQA dataset. Each curve corresponds to QA pair, and is parameterized by the mutual infor- mation between visual and textual tokens. Sam- ples/Tasks exhibiting lower mutual information show more conservative retention.
Figure 2:
Dynamic Logistic Pruning Policy. We treat the mutual information between visual and tex- tual tokens as a measure of alignment strength that distinguishes simple tasks from complex ones. When mutual information is high, indicat- ing strong correspondence between modalities, we configure the logistic retention function to decline rapidly in the early layers, enabling the model to prune surplus tokens in easy tasks (orange curve). Conversely, when mutual information is low, signifying weaker alignment, we maintain a prolonged platcau in the logistic curve and defer its sharp descent until later layers, thereby preserving higher token budget safeguard critical evidence in demanding tasks (blue curve).
We implement this effect by letting both the slope i and the inflection point mo depend linearly on the mutual information. Concretely, we set
hg ko 7 Lg (V, T),
2 = + 8 (V, T)T),
(8)

---

where 7. > 0 and To is the given params. Hence, a small (V, T) produces a larger slope k and higher zij. thereby retaining more tokens prior to the inflection point to avoid prematurely discarding critical information. In contrast, large 7 (V, T) reduces the number of tokens preserved before rij enabling the model repeatedly concentrate computations on the most salient features.
3.4 Theoretical Analysis of Computational Complexity
To evaluate the efficiency of our pruning algorithm, we derive its overall time complexity as (Nh N₁ Nr + Nv log(N.) L + LN, log(N.)) x N,..
where Mr. N₁. and N. denote the numbers of attention heads, textual tokens and visual tokens, respec- tively, and L is the number of layers. The first term N₁ Nr Nr corresponds estimation, the second term N, log( N,) I. reflects the generation of the logistic function and normal- ization, and the third term I N, log(N,) captures per-layer token sorting. None of these operations depends on feature dimension d (e.g. d = 4096), and under typical settings (N, = 32, N., Ny R several hundred, 32) the additional overhead is negligible compared with overall inference cost.
nutuallinformatioon
4 Experiments
We evaluate our framework across diverse suite of vision-language benchmarks, comparing against state-of-the-ar token pruning methods on a single NVIDIA Tesla A100 GPU, Tab. [ reports results on multi-modal tasks commonly employed in previous token pruning studies. Tab, assesses the generalizability of our work by applying our AutoPrune to other VLMs. Furthermore, we demonstrate the generalizability of AutoPrune embodied robots for autonomous driving (Tab. 3}. Notably, the core AutoPrune pipeline and even hyper-parameters parameters remains unaltered when applied to the embodied task, thereby clearly and fairly demonstrating the broad upplicability of our work.
4.1 Results with LLaVA
We evaluate our AutoPrune integrated into LLaVA-1.5-7B 27 on five standard vision- lisiun-language benchmarks, including MME [46), MMB [47). ScienceQA (SQA) 48), GQA [49], and TextVQA [50]. As shown in Tab. [ our AutoPrune consistently outperforms all competitors across the entire sparsity spectrum and exhibits graceful degradation the visual token budget diminishes. At aggressive pruning rate of 89% (retaining only 64 tokens), AutoPrune maintains 96.7% of the full-model accuracy, whereas the strongest baseline (PDrop [13 in (2255)025) achieves only 87.6% Under moderate pruning (78% removal, 128 tokens), AutoPrune preserves 98. 98.1% of original performance, compared to 95, 55,6% for PDrop and under 93% for other methods. At a pruning level of 66% removal (192 tokens), AutoPrune achieves virtually lossless performance by maintaining 99.0% accuracy while reducing FLOPs by over 57%. These results demonstrate that our complexity-adaptiv pruning schedule not only attains the highest absolute accuracy at all pruning levels but also minimizes performance degradation as the token budget decreases.
4.2 Results with LLAVA-NeXT
To validate the generality of our approach, we evaluate its performance on LLaVA-NeXT-7B 28 as detailed Tab, 2 utilizing three distinct token budgets (640, 320, and 160). For equitable comparison, all methods are benchmarked on datasets employed in prior work [19], encompassing VQA V2
[2]
GQA 49 TextVQQ (50, POPE [53], and MME [46. When retaining 320 tokens, our method maintains relative performance retention of 98.2%, outperforming all compared methods. Under the most stringent budget of 160 retained tokens, our approach preserves 94.9% of its original performance, exceeding the nearest competitor FasterVI.M (86.7%) by more than five percentage points. These results affirm the robustness of our pruning strategy across diverse token budgets. Consequently, our method proves efficacy in maintaining high performance with different VLMs.
4.3 Validating Generality for Autonomous Driving Scene Understanding and Planning
To assess the generalization abilities of our pruning strategy, we conduct comparative study on scene understanding and driving planning tasks. This evaluation utilized the Senna model [10] and its
7

---

Table Comparison of our methods with other training-free free token pruning methods. "Avg.token's refers to the average number of tokens that will be retained. Ratio represents the average percentage of performance maintained at the corresponding reduction ratio.
Method
Present at Avg. tokens MME MMB SQA GQA TextVQA
Ratio
FLOPs
LLLVA--L.--B
NeurlPS 24|
576
1862
64.7
69.5
61.9
58.2
100%
100%
ToMe 15
arXiv'22
192
1.563
60.5
65.2
54.3
52.1
89.9%
44.3%
FastV
ECCV*24
1612
61.2
67.3
52.7
52.5
90.6%
45.7%
Sparse
20
arXiv'
192
1721
62.5
69.1
57.6
56.3
95.5%
46.3%
PDrop 13
CVPR'25
192
1797
63.3
69.2
57.3
56.5
96.8%
43.9%
Ours
192
1832
64.9
69.6
60.4
57.7
99.0%
42.9%
ToMe
15
arXiv'22
128
1343
53.3
59.6
52.4
49.1
81.1%
35.1%
FastV
ECCV'24
128
1490
56.1
60.2
49.6
50.6
83.9%
36.8%
SparscVLM
20
arXiv'24
128
1 1696
60.0
67.1
56.0
54.9
93.0%
37.3%
PDrop
13
CVPR'25
128
1761
61.6
68.4
57.1
56.6
95.6%
35.1%
Ours
128
1785
64.3
69.7
59.9
57.4
98.1 %
33.7%
ToMe
15
arXiv'22 22
64
1138
43.7
50.0
48.6
45.3
70.5%
25.7%
FastV 18
ECCV*24
1256
48.0
51.1
46.1
47.8
73.7%
27.9%
Sparse
20
arXiv'24
1505
56.2
62.2
52.7
51.8
85.9%
28.2%
PDrop
13
CVPR'25
64
1561
58.8
69.0
47.5
50.6
87.6%
25.5%
Ours
64
1745
63.6
69.6
57.7
57.1
96.7 %
23.2%
Table 2: Comparison of different pruning methods on LLaVA-NeXT-7B. Performance data for
the compared methods are drawn from prior publications. For methods where results on LA-VA- NeXT were not provided in existing literature, we have reproduced their experiments and present comparative analysis against our approach in the supplementary materials.
Method
Present
Tokens
VQA
GQA
TextVQA
POPE
MME
Ratio
ILAVVA-NeX-77
24
2880
81.2
62.9
59.6
86.3
1513.8
100.0%
FastV 18
ECCV"24
78.9
58.4
1477.3
97.0%
Spanse VLM 20
arXiv'24
640
78.2
59.1
56.2
80.9
1456.3
94.9%
VisionZip 17
CVPR125
640
79.2
60.1
58.5
82.2
1468.4
96.7%
FasterVLM 51
arXiv'24
640
79.8
61.6
59.3
85.9
1480.7
98.6%
Ours
640
80.5
62.6
59.6
86.7
1515.7
99.7%
FastV 18
ECCV"24
320
71.9
55.9
55.7
71.7
1282.9
87.7%
Spanse
20
arXiv'24
320
71.4
56.5
52.4
73.5
1342.7
87.9%
VisionZip
17
CVPR'25
320
74.2
58.1
55,3
75.0
1348.8
90.5%
LasterVLM 51
arXiv'24
320
75.7
58.4
57.6
80.4
1370.1
93.3%
Ours
320
78.9
61.3
59.5
85.6
1471.6
98.2%
VastV 18
ECCV*24
61.8
49.8
$1.9
51.7
1079.5
74.7%
SparseVLM 20
arXiv'24
160
62.2
50.2
45.1
54.6
1167.1
74.9%
VisionZip 17
CVPR'25
160
67.3
$4.3
54.7
59.4
1239.7
82.3%
FasterVI.M 51
arXiv '24
160
70.6
54.7
56.0
72.9
1226.0
86.7%
Ours
160
76.4
59.4
57.2
81.4
1457.0
94.9%
associated customized nuScenes dataset. The official task adopts "Planning Accuracy" as the official evaluation metric. Tab. reports performance retention when applying different pruning techniques to the Senna VLA model [10]. Notably, our method is applied to autonomous driving without any hyper-parameter tuning. As detailed in Tab. 3 our method consistently surpass all competing methods across these diverse pruning ratios. For instance, at 25% token retention level, our approach achieved remarkable 111.23% relative accuracy, outperforming not only PyramidDrop but also the original unpruned model. These findings strongly suggest that our pruning strategy effectively preserves essential visual information even within challenging, large-scale real-world scenes. An intriguing observation emerged as our pruned model occasionally outperformed the full model, indicating the potential presence of detrimental noisy visual tokens in VL.As trained on sparse, large-scale datasets. We intend to investigate this compelling finding in future research.
98.89 89%

---

Table The performance retention ratio on the nuScenes scene understanding and planning tasks. Language-basee data and baseline model come from Senna [TO].
Retention
26/128 (20%)
32/128 (25%)
38/128 (30%)
45/128 (35%)
51/128 (40%)
FLM 51
55.05%
52.81%
47.19%
45.50%
49.43%
Sparse VLM
20
84.29%
93.83%
95.52%
97.76%
101.13%
ProamidDrop
13
94,94%
88.89%
96.07%
98.29%
100.55%
Ours
96.63%
111.23%
106.75%
105.06%
104.51 %
Table 4: Impact of different indicators for complexity and pruning schedules.
TextVQA
GQA
TextVQA
GQA
Metric
Curve
64
128
64
128
64
128
64
128
Static Logistic
55.1
55.9
55.6
57.6
Linear
54.1
55.5
52.7
54.9
Cosine Similarity
55.8
56.7
56.1
57.9
Tanh
56.6
56.9
55.2
57.5
Average Attention
56.2
56.9
56.7
58.4
Exponential
56.1
56.6
54.2
56.3
Mutual information
57.1
57.4
57.7
59.9
Logistic
57.1
57.4
57.7
59.9
(a) Different indicators for complexity
(b) Different pruning schedule curves
4.4 Ablation Studies
Different Visual-Textual Relationship Indicators. We evaluate three distinct indicators to quantify the strength of visual-textual correlation, including proposed mutual information (MI), the average cross-attention magnitude. and the cosine similarity between visual and text embeddings. The empirical results presented in Tab, 4a) reveal that mutual information approach, which draws inspiration from cognitive science, achieves better average accuracy. This outcome substantiates the superior effectiveness of mutual information in guiding token retention strategies.
To specifically isolate the impact of different scheduling functions, we incorporated four canonical curves namely linear, logistic (sigmoid), hyperbolic tangent (tanh), and exponential into our AutoPrune framework. We then evaluate their respective performances on the GQA [49] and TextVQA (50) datasets. The results, presented in Tab. b)) indicate that the logistic curve yields superior performance compared to the other scheduling functions. This finding further corroborates the previously discussed cognitive science principles, underscoring the efficacy of the logistic function in modeling attentional allocation.
Impact of Pruning Schedule Curve.
Due to space limitations, additional analysis including I) comparison with more methods, 2) on other benchmarks, and 3) adaptations with fash-attention, are presented in the supplementary materials.
5 Conclusion and Limitation
In conclusion, this paper address the computational burden of long visual sequences in VLMs by introducing complexity-Aaaptii Pruning (AutoPrune), a novel training-free framework. Inspired by cognitive neuroscience, AutoPrune quantifies sample and task complexity via mutual informa- tion between early visual and textual tokens, mapping this to individualized, budget-constrained logistic retention curves that dictate token pruning across decoder layers. Our extensive evaluations demonstrate that AutoPrune offers simple, generalizable, and highly effective solution for enabling efficient real-time multimodal reasoning and embodied intelligence. Our investigation reveals nuance in attention distribution, a limitation also observed in related studies. While token importance generally decreases with decoder depth, our findings (Fig. I show deeper layers can occasionally retain more critical tokens than shallower ones. Although our work advances depth-aware pruning, further refinement needed for strategies to dynamically match this variable importance distribution across network depth. We leave this in future research.

---

Acknowledgements
This work was supported in part by the Natural Science Foundation China (Grant No. 6203601 I, 62422317, U22B2056, 62503323), the Beijing Natural Science Foundation (Grant No. JQ22014, L223003, JQ24022). The work Yufan Liu and Weiming Hu was also supported in part by the Natural Science Foundation of China (Grant No. 62192782, 62503323, U2441241, 62372451, 62192785, 62372082), the CAAl-Ant Group Research Fund (Grant No. CAAI-MYJJ 2024-02), the Young Elite Scientists Sponsorship Program by CAST (Grant No. 2024QNRC001).
References
[1] Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1..50.00325,, 2015.
[2] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900 PMLR, 2022.
[3] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv: 14100, 2022.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages 2425-2433, 2015.
[5] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 16871-16894 PMLR, 2023.
[6] Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, et al. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXivv2208.031/1 2022.
[7] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4178, 2023.
[8] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Paul Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-acttee
model. arXiv preprint arXiv:2406.0922 2024.
[9] Daniel Black, Daniel Brown, et al. Rg: A Dision-language. control. arXiv preprint arXiv:2410.2416. 2024.
flow model for general robot
[10] Bo Jiang, Shaoyu Chen, Bencheng Liao. Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large Lasiun-aaguuge models and end-to-end autonomous driving. arXiv preprint 2:1iv:2410.22313, 2024.
[11] Yanwei Li al. Llama-vid: An image is worth tokens in large language models. arXiv preprint a.Xiv:231 17044, 2023.
[12] Jiang Wu et al. Catp: Cross-attention token pruning for accuracy-efficienc trade-off in multimodal transformers. arXiv preprint (rXiv:2404.08667 2024.
[13] Xiang Chen et Pyramiddrop: Accelerating your large lision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.172 2024.
[14] Yifan Wang et al. Zipvl: Efficient large vasion-language models with dynamic token sparsifica- tion and kv cache compression. arXiv preprint arXiv:2410.08588,, 2024.
10

---

[15] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09661, 2022.
[16] Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1383-1392, 2024.
[17] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv: 044677 2024.
[18] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference Computer Vision, pages 19-35. Springer, 2024.
[19] Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vim inference faster. arXiv preprint arXiv:2412.0188
2024.
[20] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsification for efficient vision- languale model inference. In International Conference on Machine Learning, 2025.
[21] Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipvl: Efficient large vision-language models with dynamic token sparsification and ky cache compression. arXiv preprint arXiv:2410.0855. 2024.
[22] Keith Rayner. Eye movements in reading and information processing: 20 years of research. Psychological bulletin, 124(3):372, 1998.
[23] John M Henderson and Andrew Hollingworth. High-level scene perception. Annual review of psychology, 50(1):243-271, 1999.
[24] Michael K Tanenhaus, Michael J Spivey-Knnwloon Kathleen M Eberhard, and Julie C Sedivy. Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217) 1995.
[25] Falk Huettig, Joost Rommers, and Antje S Meyer. Using the visual world paradigm study language processing: A review and critical evaluation. Acta psychologica, 137(2):111-171,
2011.
[26] Vighnesh Subramaniam, Colin Conwell, Christopher Wang, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu. Revealing vinion-language integration in the brain with multimodal networks. ArXiv, pages arXiv-2406, 2024.
[27] Haotian Liu, Pengfei Zhang, Yizhu Xu, Hang Zhang, Xin Li, Lidong Bing, et al. Visual instruction tuning. arXiv preprint orXiv:2304. 08485, 2023.
[28] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024.
[29] Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large vasion-language models and end-to-end autonomous driving. arXiv preprint arXiv:2410.22311 2024.
[30] Kazi Hasan Ion Arif, Jin Yoon, Dimitrios Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: iicention-guide token dropping for efficient inference of high-resolution vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pages 1773-1784 2025.
[31] Yunsheng Ma, Amr Abdelraouf, Rohit Gupta, Ziran Wang, and Kyungtae Han. Video token spar- sification for efficient multimodal lmm autonomous driving. arXiv preprint arXiv:2409. /822 2024.
11

---

[32] Zhuqiang Lu, Zhenfei Yin, Mengwei He, Zhihui Wang, Zicheng Liu, Zhiyong Wang, and Kun Hu. B-vilm: A vision large language model with balanced ppatio-temporal tokens. arXiv preprini arXiv:2412.09911 2024.
[33] Hanning Chen, Yang Ni, Wenjun Huang, Yezi Liu, SungHeon Jeong, Fei Wen, Nathaniel D Bastian, Hugo Latapie, and Mohsen Imani. Vitp: Vision- guided token pruning for task-triented segmentation. 2025 JEFE/CVE Winter Conference on Applications of Computer Vision (WACV), pages 9353-9363. IEEE, 2025.
[34] Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention. arXiv preprint 2xiv:2209.113022. 2022.
[35] Quan Tang, Bowen Zhang. Jiajun Liu, Fagui Liu, and Yifan Liu. Dynamic token pruning in plain vision transformers for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 777-786, 2023.
[36] Cheng Yang. Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, et al. Topv: Compatible token pruning with inference time optimization for fast and low- low-memory multimodal vision language model. arXiv preprint arXivv2503. 18278, 2025.
[37] Remi Gau, Pierre-Louis Bazin, Robert Trampel, Robert Turner, and Uta Noppeney. Resolving multisensory and attentional influences across cortical depth in sensory cortices. elife, 9re46856,
2020.
[38] Dale Barr. Analyzing "visual world' enrer'ckytn data using multilevel logistic regression. Journal of memory and language, 59(4))577 474, 2008.
[39] Gerry TM Altmann and Yuki Kamide. Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73(3):247-264, 1999.
[40] Barry E Stein and M Alex Meredith. The merging of the senses. MIT press, 1993.
[41] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Gianni Brauwers and Flavius Frasincar. A general survey on attention mechanisms in deep learning. IEEE Transactions on Knowledge and Data Engineering, 3544):329--3298, 2021.
[44] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15671-15680, 2022.
[45] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 180.00388, 2018
[46] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, 2024.
[47] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024.
[48] Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Sci- enceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):28--301, 2022.
12

---

[49] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[50] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8317-8326, 2019.
[51] Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, MinQi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning:
Make vlm inference faster. arXiv preprini urXiv:24/2.01811 2024.
[52] Ziheng Jia, Zicheng Zhang, Jiaying Qian, Haoning Wu, Wei Sun, Chunyi Li, Xiaohong Liu, Weisi Lin, Guangtao Zhai, and Xiongkuo Min. Vqa 2: Visual question answering for video quality assessment. arXiv preprint 21..59241..03595, 2024.
[53] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen, Evaluating object ballucination in large vision-language models. arXiv preprint arXiv:2305. 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification:
Code and raw results will be publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc//uuuii wwwpc//ggue////eeeeceooc CoodeSubeiiie ssi asionnoll onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The running results are consistent and there is no statistical error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information on the computer resources, please
see Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and we confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets used in this paper are credited and the license is respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Code and raw results will be publicly available upon acceptance and we will include details about training, license, limitations, etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research in this paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research in this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We have reviewed the LLM policy and confirm that the method development in this research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: bestps:/nneuripeee /ttprippp...c/oooofeeeee/e////2 for what should or should not be described.
20


Reviews:
[{"review_id": "Vnt2dstWMR", "reviewer": "Reviewer_EN6i", "summary": "The paper introduces AdaPrune, a training-free, plug-and-play framework for adaptive token pruning in Vision-Language Models. It measures cross-modal mutual information (MI) between early visual and textual tokens to gauge sample/task complexity, then maps MI to a budget-constrained logistic retention curve that dynamically allocates token budgets per decoder layer. Applied to LLaVA-1.5, LLaVA-NeXT, and a VLA model, AdaPrune prunes up to 89% of tokens while retaining >96% accuracy, outperforming prior methods.", "questions": "How exactly do the authors map the mutual information score to the logistic curve’s slope and midpoint? Did you apply any clamping or calibration on MI values?\n\nIt would be good to see AdaPrune on much larger VLMs or on streaming/video inputs. What challenges do the authors anticipate?\n\nHow sensitive is performance to noise in the MI estimate? Did you observe cases where mis-estimated MI led to under- or over-pruning?\n\nIt would be helpful to share more implementation details. For easier reimplementation, the authors can show more about how you compute MI from attention, your default hyperparameters, and how you pick which tokens to drop each layer?", "limitations": "The paper omits an explicit “Limitations” section but combing it with the conclusion section. I think it is okay to do so.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe manuscript is well written and easy to follow.\n\nAdaPrune uses a simple, training-free signal (mutual information between early visual and text tokens) to tailor pruning intensity for each input, avoiding manual schedule design or retraining.\n\nThe logistic retention curve formulation guarantees that any token or FLOPs budget is met exactly.\n\nThe authors conducted various experiments on LLaVA-1.5, LLaVA-NeXT, and a vision-language driving model.\n\n---\n\nWeaknesses:\n\nThe mapping from mutual-information scores to the curve’s slope and inflection point is described only at a high level, more detail on hyperparameter choices and token-selection criteria is needed.\n\nRelying solely on an early-layer MI estimate risks misjudging complexity for inputs where initial attention patterns are misleading, potentially pruning critical tokens too soon.\n\nEvaluation is limited to 7B-parameter models on static images. It is unclear how AdaPrune scales to much larger models, video inputs, or streaming scenarios. Table 4 in supp is helpful information, do you consider moving it to the main pages?", "was_revised": true, "final_justification": "I appreciate the authors' efforts spent on the rebuttal.\nMy main concerns are resolved, and I'm willing to keep a positive rating leaning to acceptance.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We appreciate your recognition of our clear writing, the simple training‑free MI signal that adapts pruning per input, the exact‑budget logistic retention curve, and the broad evaluations on LLaVA‑1.5/NeXT and a driving VLM.\n\n**Q1: More detail on** **hyperparameter** **choices and token-selection.**\n\n**A1:** Thank you for raising this important concern. Below we provide additional detail on our hyperparameter choices and token‑selection criteria:\n\n1. **Hyperparameter** **mapping**\n\n   The mapping from mutual‑information score $I_q(V, T)$ to the slope $k_q$ and the inflection point $x_0^q$ is defined by \n\n   $$k_q = k_0 - \\gamma\\,I_q(V, T),\\quad  x_0^q = x_0 + \\beta\\,I_q(V, T). $$\n\n   To demonstrate robustness, we fix these values across all experiments (See below for parameter selection). Specifically we set \n\n   $$k_0 = 0.4,\\quad \\gamma = 0.3,\\quad x_0 = 15,\\quad \\beta = 0.3. $$\n\n   We apply only one clamp, enforcing $k_q \\ge 0$ so that the selection process can only decrease the token number and never increase it. We do not apply any calibration to the mutual‑information values.\n\n2. **Token‑selection criteria**\n\n   Visual tokens are scored by cross‑modal attention between image and text features, and we select the top $k$ tokens for downstream processing, with $k$ governed by our dynamic retention schedule. Our study targets dynamic control of the pruning budget rather than the design of ranking heuristics. To keep the analysis simple and comparable, we therefore use raw cross‑modal attention weights as the ranking criterion in all experiments. Although more sophisticated scoring mechanisms described in prior work may yield further gains, such enhancements fall outside the scope of the our study and are reserved for future investigation.\n\n3. **Robustness across hyperparameters**\n\n   - **Parameter selection strategy. 1) Curve-based criteria.** As shown in Fig. 2, the retention curve must clearly distinguish easy from hard samples. We therefore choose parameters that provide sufficient dispersion, avoiding curves that concentrate around the midpoint or flatten at the extremes, so that the pruning schedule remains balanced. **2) Empirical** **validation** A small series of preliminary experiments yielded the following hyperparameters without extensive tuning:\n\n     $$k_0 = 0.4,\\quad \\gamma = 0.3,\\quad x_0 = 15,\\quad \\beta = 0.3. $$\n\n     Specifying each value to one decimal place demonstrates that no overfitting occurred on any particular dataset.\n\n   - **More** **experiments.** To validate the robustness of these settings, we ran additional experiments under alternative hyperparameter configurations. Although some variants produced slightly improved results (see the table below), we elected to retain the original fixed values in order to ensure fairness and consistency across all datasets.\n\n     | Configuration                                      | TextVQA(token=64)↑ | TextVQA(token=128)↑ | GQA(token=64)↑ | GQA(token=128)↑ |\n     | -------------------------------------------------- | ------------------ | ------------------- | -------------- | --------------- |\n     | $k_0=0.3, \\gamma=0.2, x_0=16, \\beta=0.5$           | 57.2               | 57.6                | 57.3           | 59.4            |\n     | $k_0=0.5, \\gamma=0.4, x_0=14, \\beta=0.2 $          | 56.9               | 57.3                | 57.6           | 59.6            |\n     | $k_0=0.4, \\gamma=0.3, x_0=15, \\beta=0.3 $(default) | 57.1               | 57.4                | 57.7           | 59.9            |\n\n**Q2: Early-layer MI estimates risks misjudging.**\n\n**A2:** Thank you for raising this important and insightful concern. \n\n1. **Optimal Layer Selection for MI Estimation.** Estimating mutual information too early can indeed introduce inaccuracies, which is why we perform MI estimation beginning at the second layer of the network. To address your question, we have conducted additional experiments measuring MI at various layers. The results, summarized in the table below, reveal two key trends. First, estimating MI too early (at layer 1) injects spurious signals into the pruning schedule, causing the model’s accuracy to suffer. Second, delaying MI estimation until very late layers front‑loads an excessive share of the fixed compute budget, which also diminishes overall performance. Estimating MI at layer 2 therefore achieves the optimal balance between reliable signal extraction and efficient budget utilization. Notably, in layers where MI is not estimated, we employ a linear pruning schedule in our experiments.\n\n   | **Layer** | TextVQA(token=64)↑ | TextVQA(token=128)↑ | GQA(token=64)↑ | GQA(token=128)↑ |\n   | --------- | ------------------ | ------------------- | -------------- | --------------- |\n   | **1**     | 56.4               | 56.8                | 56.5           | 59.1            |\n   | **2**     | 57.1               | 57.4                | 57.7           | 59.9            |\n   | **3**     | 56.9               | 57.2                | 57.4           | 59.5            |\n   | **4**     | 56.6               | 57.0                | 57.2           | 59.4            |\n\n2. **Cross‑Modal Hybrid Token Scoring Mechanism.** Within the first layer, the fusion of textual and visual modalities remains insufficient. As shown in Sec. D of the supplementary materials, we use Cross‑Modal Weighted Pruning. We compute each visual token’s importance score by combining its self‑attention weight with its textual attention weight. This hybrid scoring prevents discarding tokens that carry critical visual content before stronger cross‑modal interactions form.\n\n**Q3: How AdaPrune scales to much larger models, video inputs, or streaming scenarios and Tab. 4 in supp.**\n\n**A3:** We appreciate the reviewer’s suggestion for scalability and future extensions to video and streaming.\n\n1. **Scales to much larger models.** \n\n   AdaPrune is entirely model‑agnostic and requires no architectural modifications to operate on larger vision‑language models. In addition to the LLaVA‑13B results presented in Supplementary Table 4, we also evaluated AdaPrune on CogVLM-17B [1] by pruning 90 % of the tokens during inference. As shown in table below, AdaPrune consistently delivers significant inference‑cost reductions with negligible performance loss across both 7B, 13B and 17B variants, demonstrating robustness to model scale.\n\n   | Method               | Retaining Ratio | VQAv2↑   | VizWiz↑  | SQA↑     |\n   | -------------------- | --------------- | -------- | -------- | -------- |\n   | CogVLM-17B           | 100%            | 80.9     | 49.6     | 68.4     |\n   | FastV (ECCV 24)      | 10%             | 74.2     | 42.9     | 63.5     |\n   | FasterVLM（ICCV 25） | 10%             | 74.6     | 46.6     | 68.2     |\n   | **Ours**             | **10%**         | **75.7** | **48.3** | **68.4** |\n\n​\t[1] Cogvlm: Visual expert for pretrained language models, NeurIPS 2024\n\n2. **Video and streaming inputs.** Thank you for this suggestion. Although our method is specifically designed for image–text VLMs, we believe its extension to video and real‑time streaming is highly promising and could bring significant benefits. The key challenge in this setting is maintaining temporal consistency in token selection, because pruning each frame independently may lead to flickering or unstable focus regions. We anticipate that simple extensions, such as (a) smoothing attention weights over successive frames to carry forward high‑importance tokens and (b) dynamically allocating a per‑frame token budget based on motion or scene change estimates, will address these issues with minimal changes to our framework. We plan to explore these temporal adaptations in future work.\n\n**Q4: Assessing AdaPrune’s Robustness to MI Estimation Noise.**\n\n**A4:** Thanks to the reviewer for raising the need to evaluate how MI estimation noise affects AdaPrune.\n\nWe acknowledge that errors in the mutual information estimate can influence pruning decisions. When MI is overestimated, key tokens may be pruned too early. When MI is underestimated, the token set may be larger than necessary. Despite these effects, the impact on overall accuracy remains limited. Tab. 4(a) shows that a fixed logistic retention schedule ignoring per‑layer MI still achieves competitive performance.\n\nTo measure sensitivity, we injected random perturbations of ±10 %, ±20 % and ±50 % into the MI values before computing the retention schedule and reran our experiments. The results show that perturbations in the 10–20% range cause only minor degradation in accuracy and only the large 50% noise produces a more noticeable drop. These findings confirm that AdaPrune remains robust, only degrading when MI estimation noise is substantial.\n\n|          | TextVQA(token=64)↑ | TextVQA(token=128)↑ | GQA(token=64)↑ | GQA(token=128)↑ |\n| -------- | ------------------ | ------------------- | -------------- | --------------- |\n| ±10 %    | 56.9               | 57.2                | 57.4           | 59.5            |\n| ±20 %    | 56.2               | 56.9                | 57.1           | 59.1            |\n| ±50 %    | 54.6               | 55.1                | 55.7           | 57.4            |\n| Original | 57.1               | 57.4                | 57.7           | 59.9            |\n\n**Q5: More implementation details of compute MI.**\n\n**A5:** We compute mutual information exactly as in Section 3.3 of the paper. Specifically, we first extract the attention map between the visual and textual modalities and normalize its entries to obtain the joint distribution $p(v_i, t_j)$. Treating the marginal distributions as $p(v_i)=\\sum_j p(v_i,t_j)$ and $p(t_j)=\\sum_i p(v_i,t_j)$, we then compute\n\n$$I(V, T) = \\sum_{i=1}^{N_v}\\sum_{j=1}^{N_t}  p(v_i, t_j)\\,\\log\\frac{p(v_i, t_j)}{p(v_i)\\,p(t_j)}. $$\n\nThis directly measures the amount of information shared between visual and textual tokens based on their attention weights."}, {"review_id": "duyH3gwgZV", "reviewer": "Reviewer_u9ow", "summary": "This paper concentrates on the vLLM's inference acceleration. In particular, previous methods typically employ heuristics layer-specific pruning strategies, and the overall pruning schedule is fixed and applied uniformly to all input samples and tasks, regardless of the samples, tasks or layers. In this work, inspired by the cognitive science, the authors introduce Complexity-Adaptive Pruning (AdaPrune), which is a training-free, plug-and-play framework that tailors pruning policies to varying sample and task complexities. The method is based the mutual information between visual and text inputs, and quantifies the compexity of a sample. Then, the pruning policy is designed based on the mutual information. Experiments are done on LLaVA and VQA benchmarks.", "questions": "See above", "limitations": "See above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The strengths are as follows:  \n1.This work is well motivated. Recent training free acceleration methods mainly rely on hand-crafted metrics and the pruing policy is fixed regardless of the layers. This paper proposes a sample or task complexity adaptied pruing model.  \n2.The idea based on cognitive science and mutual information is interesting. Using mutual information between visual and text input as a proxy objective to measure the sample's comlexity and prune tokens is new to me. It sounds reasonable.   \n\nThe weakness, also my questions are as follows:  \n1.Why the Budget-Constrained Logistic Retention is designed as Eq.5. It is not easy to understand how this formulation is introduced and why it is good to achieve the target?  \n2.In Figure 2, Why simple samples mainly lie in retention rate between 0.2-0.4 in both low and high layers? For hard samples, it is easy to understand the retention rate is high in low layers, but how to understand the retention rate is low in high layers? Do we need more information for hard samples in both low and high layers?  \n3.The experiments could be improved.  \na).The test set of VQA task is very simple, only Yes or No, or answer selection. Are there any other more complex generation task for evaluation? Image caption or image generaion?  \nb).Some related works are not included and discussed. Recently, there are some other training-free pruning methods, such as VTW[1], Turbo[2] and Folder[3]. These methods should be compared and discussed.  \nc).In Table 3, the reported performance is  a relative improvement. What is the absolute performance of compared baseline model, it is worthwhile to take the absolute performance into cosideration to check the relative improvement.  \n\n[1]Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference AAAI'25  \n[2]Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Models, ECCV'24  \n[3]FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance,ICCV'25", "was_revised": true, "final_justification": "Thanks for the response, my concerns are solved, I have raised my score. Please add these clarification and experiments in the final version.", "rebuttal": "Thank you for your detailed review and thoughtful comments. Thank you for noting the strong motivation and the novelty of using visual–textual mutual information to adapt pruning to task complexity rather than relying on fixed, hand‑crafted policies.\n\n**Q1: Why the Budget-Constrained** **Logistic** **Retention is designed as Eq.5?**\n\n**A1:** We adopt a logistic retention function for three key reasons:\n\n1. **Neuro‑inspiration:** As discussed in lines 208–210 of the main text, human visual reasoning over time follows an S‑shaped information accumulation curve, which closely matches the profile of a logistic function.\n2. **Proven efficacy:** The logistic curve, often referred to as the logistic sigmoid, is extensively employed in contemporary LLM and VLM research. It underlies sigmoid-based language–image pre-training losses [1], serves as the gating function in mixture-of-experts routing [2], and plays a central role in various optimization algorithms [3].\n3. **Empirical validation:** In Tab.4 (b), we compare linear, tanh, exponential, and logistic mappings. The logistic formulation consistently achieves the best balance between adhering to the token budget and maintaining task performance, justifying its selection.\n\n[1] Sigmoid Loss for Language-Image Pre-Training, ICCV 2023\n\n[2] Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts, NeurIPS 2024\n\n[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023\n\n**Q2: The reason for retention rate for easy and hard samples.**\n\n**A2:** We appreciate the reviewer’s insight. The observed retention patterns arise for three reasons:\n\n1. **Fixed overall budget.** To satisfy our compute‑cost constraint, we enforce the same total token budget for both easy and hard samples. The curves therefore show the optimal pruning schedule under an identical compute budget.\n2. **Behavior on easy samples.** Easy samples allow the model to rapidly locate in the relevant regions, so we can apply aggressive pruning in the earliest layers. After pruning, the model repeatedly attends to the retained tokens to guarantee correctness. (Note that if the compute budget were unconstrained, one could introduce further deep pruning to reduce computation even more for these easy cases.)\n3. **Behavior on hard samples.** Difficult images require an initial exploratory phase in which the model must survey many tokens to locate the target. Hence, we delay pruning in early layers (high retention). \n\n**Q3: Improvements of experiments.**\n\n**A3:** We thank the reviewer for these valuable suggestions:\n\n1. **More complex Image caption task for evaluation.**\n\n   We extend our pruning method to the COCO Caption dataset using the LLaVA 1.5-7B model and evaluate performance with the CIDEr and SPICE metrics. The unpruned baseline achieves a CIDEr score of 1.048. When retaining only 75%, 50% and 11.1% of the tokens, our approach continues to outperform existing methods on both metrics. Notably under heavy pruning at 11.1%, the gains are especially pronounced compared to others. These results underscore the strong generalization ability of our method.\n\n   | Method               | 75%       | 50%       | 11.1%     |\n   | -------------------- | --------- | --------- | --------- |\n   | FasterVLM（ICCV 25） | 1.032     | 0.997     | 0.872     |\n   | PyDrop (CVPR 25)     | 0.999     | 0.981     | 0.739     |\n   | **Ours**             | **1.039** | **1.028** | **0.946** |\n\n2. **Inclusion of VTW, Turbo, and Folder.**\n\n   We appreciate the reviewer’s recommendation to examine VTW, Turbo and FOLDER. In Tab. 2 of the supplementary material, we include a comparison with VTW. We have conducted new experiments comparing AdaPrune with Turbo and FOLDER, and we will include a discussion of all three methods in our related‑work section. As shown in table below, AdaPrune achieves superior accuracy, again demonstrating its robustness and efficiency. All data in the table are taken directly from the original FOLDER publication.\n\n   | Method      | Retaining Ratio | MMBench-en↑ | MMMU↑    | MME↑     |\n   | ----------- | --------------- | ----------- | -------- | -------- |\n   | LLaVA1.5-7B | 100%            | 62.8        | 32.2     | 1339     |\n   | Turbo       | 34%             | 60.1        | 24.7     | 1302     |\n   | FOLDER      | 34%             | 61.4        | 31.3     | 1350     |\n   | **Ours**    | **34%**         | **64.3**    | **34.2** | **1433** |\n\n3. **Absolute performance of compared baseline model.**\n\n   We have updated Tab. 3 to include the absolute scores of the baseline model (100% token retention). For example, the baseline accuracy is 41.49%, and AdaPrune achieves 46.15% at only 25% token usage. Including these numbers helps clarify that our relative gains correspond to a small absolute drop in accuracy while significantly reducing computation.\n\n   | **Method**      | **26/128 (20%)** | **32/128 (25%)** | **38/128 (30%)** | **45/128 (35%)** | **51/128 (40%)** |\n   | --------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n   | **FasterVLM**   | 22.84            | 21.91            | 19.58            | 18.88            | 20.51            |\n   | **SparseVLM**   | 34.97            | 38.93            | 39.63            | 40.56            | 41.96            |\n   | **PyramidDrop** | 39.39            | 41.03            | 39.86            | 40.78            | 41.72            |\n   | **Ours**        | **40.09**        | **46.15**        | **44.29**        | **43.59**        | **43.36**        |"}, {"review_id": "cXNyuRcf0e", "reviewer": "Reviewer_qWyB", "summary": "This paper introduces AdaPrune, a training-free token pruning framework that, drawing inspiration from neuroscience, adaptively selects layer-wise token retention ratios subject to a global constraint on inference compute. The neuroscience background suggests that humans typically narrow their focus to constrained locations in visual scenes in early processing stages when the task is simple, while they conversely preserve several hypotheses for longer when the task is complex. This work shows an analogous pattern in Vision-Language Models (VLMs), where the attention maps may \"converge\" in early or late layers, depending on the textual query associated with it.  \n\nGiven this empirical evidence, AdaPrune maps a notion of query difficulty (computed via mutual information between visual and textual tokens) to a logistic curve that determines how many tokens to retain at each layer. Specifically, the logistic curve maps a layer index to a retention ratio, and the slope and inflection point of the curve are linear functions of the aforementioned mutual information. Such a curve is further normalised to ensure its integral (which reflects the total number of tokens processed end-to-end during inference) matches a given compute constraint.  \n\nSuccessful experiments are conducted on standard Vision-Language benchmarks with LLaVA-v1.5-7B and LLaVA-Next-7B, as well as with Senna, a Vision-Language-Action model employed for autonomous driving.", "questions": "For a productive rebuttal, I would probably start with the following questions (expanding on the weaknesses above).\n\n1. Is Mutual Information for a single visual token the criterion used to choose which tokens are retained after determining a retention ratio with the logistic curve? So, something like Eq. (3) of the paper, where the outer summation over visual tokens is unpacked into $N_v$ scalars determining the importance scores of each of the visual tokens? I have reviewed Section D of the supplementary materials, where an ad-hoc importance score is defined for the first layer. Line 40 therein states: *\"We assign each visual token an importance score based on the textual-visual attention weight.\"* It is, however, unclear to me if this holds for all layers or only for the first one. In general, could the authors please clarify this aspect about token selection?  \n\n2. Could the authors please clarify how the parameters of the logistic curve are selected, i.e., which data was used, how many data points, etc?\n\n3. In my understanding, the logistic curve computed from Mutual Information determines how tokens are progressively dropped in the stacked blocks of the LLM decoder, meeting the global compute constraint after normalisation, but at which layer is this Mutual Information computed? While reading, I assumed this computation happens immediately in the first block, and the derived scalar determines computation throughout the rest of the decoder, but could the authors please clarify this aspect as well?", "limitations": "The final section of the paper is titled \"Conclusions and Limitations\", but it looks more like a \"Conclusions\" Section rather than a \"Limitations\" one. However, I do not see important unaddressed considerations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**.  \n1. The paper is well-written and nicely organised;  \n2. The underlying idea is simple, which I believe is a plus, is carefully designed (I particularly liked the overall reframing of attention scores into MI) and has a sound motivation; \n3. Experiments are comprehensive, and extending towards VLAs with fixed hyperparameters shows the practical applicability of the proposed framework beyond academic benchmarks; \n\n\n**Weaknesses**.  \nI think the major weakness of this paper is that some passages of the methodological section are unclear or partially omitted. For example:  \n1. In my understanding, the adaptive logistic curve provides a per-layer token retention ratio based on the overall scalar-valued mutual information, but I see no detail about which tokens are retained and which are discarded;  \n2. Other than the estimated mutual information, the logistic curve depends on some parameters ($k_0$, $\\gamma$, $x_o$, $\\beta$) which should be defined a priori. It's probably amiss for this paper not to report how these are selected;", "was_revised": true, "final_justification": "My review initially expressed concerns about the lack of details for three aspects:\n- Scoring function used for token selection;\n- Choice of the layer for MI estimation;\n- Hyperparameter configurations;\n\nThe authors explained the details with care during the initial phases of the discussion, as well as stated they will clarify them in the revised manuscript. Therefore, I am increasing my score.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We’re grateful for your comments on the paper’s clarity and organization, the simple yet well‑motivated reframing of attention as MI, and the comprehensive experiments.\n\n**Q1: Details about which tokens are retained.**\n\n**A1:** We thank the reviewer for requesting more detail on token retention.\n\n- **First layer.** Within the initial processing layer, the fusion of textual and visual modalities remains insufficient. As shown in Sec. D of the supplementary materials, we use Cross‑Modal Weighted Pruning. We compute each visual token’s importance score by combining its self‑attention weight with its textual attention weight. This hybrid scoring prevents discarding tokens that carry critical visual content before stronger cross‑modal interactions form.\n- **Subsequent layers.** Visual tokens are scored by cross‑modal attention between image and text features, and we select the top $k$ tokens for downstream processing, with $k$ governed by our dynamic retention schedule. Our study targets dynamic control of the pruning budget rather than the design of ranking heuristics. To keep the analysis simple and comparable, we therefore use raw cross‑modal attention weights as the ranking criterion in all experiments. More elaborate scoring strategies reported in prior work could yield additional gains. This direction is orthogonal to our contribution and is left for future research.\n\n**Q2: Selection of** $(k_0, \\gamma, x_0, \\beta)$**.**\n\n**A3:** Thank you for raising this important concern. Below we provide additional detail on our hyperparameter choices and token‑selection criteria:\n\n1. **Hyperparameter** **mapping**\n\n   The mapping from mutual‑information score $I_q(V, T)$ to the slope $k_q$ and the inflection point $x_0^q$ is defined by \n\n   $$k_q = k_0 - \\gamma\\,I_q(V, T),\\quad  x_0^q = x_0 + \\beta\\,I_q(V, T). $$\n\n   To demonstrate robustness, we fix these values across all experiments (See below for parameter selection). Specifically we set \n\n   $$k_0 = 0.4,\\quad \\gamma = 0.3,\\quad x_0 = 15,\\quad \\beta = 0.3. $$\n\n   We apply only one clamp, enforcing $k_q \\ge 0$ so that the selection process can only decrease the token number and never increase it. We do not apply any calibration to the mutual‑information values.\n\n2. **Token‑selection criteria**\n\n   Visual tokens are scored by cross‑modal attention between image and text features, and we select the top $k$ tokens for downstream processing, with $k$ governed by our dynamic retention schedule. Our study targets dynamic control of the pruning budget rather than the design of ranking heuristics. To keep the analysis simple and comparable, we therefore use raw cross‑modal attention weights as the ranking criterion in all experiments. Although more sophisticated scoring mechanisms described in prior work may yield further gains, such enhancements fall outside the scope of the present study and are reserved for future investigation.\n\n3. **Robustness across hyperparameters**\n\n   - **Parameter selection strategy. 1) Curve-based criteria.** As shown in Fig. 2, the retention curve must clearly distinguish easy from hard samples. We therefore choose parameters that provide sufficient dispersion, avoiding curves that concentrate around the midpoint or flatten at the extremes, so that the pruning schedule remains balanced. **2) Empirical** **validation****.** A small series of preliminary experiments yielded the following hyperparameters without extensive tuning:\n\n     $$k_0 = 0.4,\\quad \\gamma = 0.3,\\quad x_0 = 15,\\quad \\beta = 0.3. $$\n\n     Specifying each value to one decimal place demonstrates that no overfitting occurred on any particular dataset.\n\n   - **Supplementary experiments.** To validate the robustness of these settings, we ran additional experiments under alternative hyperparameter configurations. Although some variants produced slightly improved results (see the table below), we elected to retain the original fixed values in order to ensure fairness and consistency across all datasets.\n\n     | Configuration                                      | TextVQA(token=64)↑ | TextVQA(token=128)↑ | GQA(token=64)↑ | GQA(token=128)↑ |\n     | -------------------------------------------------- | ------------------ | ------------------- | -------------- | --------------- |\n     | $k_0=0.3, \\gamma=0.2, x_0=16, \\beta=0.5$           | 57.2               | 57.6                | 57.3           | 59.4            |\n     | $k_0=0.5, \\gamma=0.4, x_0=14, \\beta=0.2 $          | 56.9               | 57.3                | 57.6           | 59.6            |\n     | $k_0=0.4, \\gamma=0.3, x_0=15, \\beta=0.3 $(default) | 57.1               | 57.4                | 57.7           | 59.9            |\n\n**Q3: Which layer is this Mutual Information computed?** \n\n**A3:**  We appreciate the reviewer's valuable suggestion.\n\n1. **Optimal Layer Selection for MI Estimation.** Thank you for raising this important and insightful concern. Estimating mutual information too early can indeed introduce inaccuracies, which is why we perform MI estimation beginning at the second layer of the network. To address your question, we have conducted additional experiments measuring MI at various layers. The results, summarized in the table below, reveal two key trends. First, estimating MI too early (at layer 1) injects spurious signals into the pruning schedule, causing the model’s accuracy to suffer. Second, delaying MI estimation until very late layers front‑loads an excessive share of the fixed compute budget, which also diminishes overall performance. Estimating MI at layer 2 therefore achieves the optimal balance between reliable signal extraction and efficient budget utilization. Notably, in layers where MI is not estimated, we employ a linear pruning schedule in our experiments.\n\n   | **Layer** | TextVQA(token=64)↑ | TextVQA(token=128)↑ | GQA(token=64)↑ | GQA(token=128)↑ |\n   | --------- | ------------------ | ------------------- | -------------- | --------------- |\n   | **1**     | 56.4               | 56.8                | 56.5           | 59.1            |\n   | **2**     | 57.1               | 57.4                | 57.7           | 59.9            |\n   | **3**     | 56.9               | 57.2                | 57.4           | 59.5            |\n   | **4**     | 56.6               | 57.0                | 57.2           | 59.4            |\n\n2. **Cross‑Modal Hybrid Token Scoring Mechanism.** Within the first layer, the fusion of textual and visual modalities remains insufficient. As shown in Sec. D of the supplementary materials, we use Cross‑Modal Weighted Pruning. We compute each visual token’s importance score by combining its self‑attention weight with its textual attention weight. This hybrid scoring prevents discarding tokens that carry critical visual content before stronger cross‑modal interactions form.\n\n**Q4: Limitations: Important unaddressed considerations.** \n\n**A4:** We agree that our “Conclusions and Limitations” section can be strengthened by more explicitly acknowledging key shortcomings. We have identified and will add the following limitations to the revised manuscript: \n\n1. **Lack of a token recall mechanism.** Although token importance generally decreases with decoder depth, our analysis in Fig. 1 reveals that deeper layers sometimes retain tokens that are more critical than those in shallower layers. This finding motivates the development of a mechanism to recover valuable tokens that were discarded in early layers. \n2. **Application to video-based VLMs.** Our pruning method is, in principle, directly transferable to video vision-language models, but we have not yet introduced any temporal modeling. Incorporating temporal information into our token pruning framework represents a meaningful direction for future research. \n\nWe will incorporate these points into the Limitations section in the revision."}, {"review_id": "TmmX60YJnn", "reviewer": "Reviewer_zvmf", "summary": "AdaPrune is a method for flexibly removing visual tokens in large vision–language models (LVLMs) based on input complexity. Unlike prior approaches that apply a fixed, layer-wise token removal schedule uniformly across all samples and tasks, AdaPrune measures the mutual information between visual and textual tokens and generates a logistic retention curve tailored to each sample’s complexity. This ensures that only the necessary tokens are retained under a given computational budget, drastically reducing computation.", "questions": "Refer to Weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n- Training-Free Framework: Offers a plug-and-play solution that requires no additional training.\n- Cognitively Inspired: Leverages visual–textual mutual information to quantify complexity, mimicking biological attention mechanisms.\n- Per-Sample Token Control: Employs individualized, budget-constrained logistic retention curves to dynamically prune tokens per decoder layer.\n\n---\n\n**Weaknesses**\n\n- Unclear writing: It makes the reader get easily confused with the terminology and the motivation of neuroscience. This writing has so many expert-level terminology for brain and redendunt detail explanation. These easily made me distract from the main point of what the authors want to say. Line 53-63 and Line 154-165 are most paragraphs that I can't understand much. Even, the rest of the writings is also somewhat hard to read smoothly. Unless really focusing on the writing, it is easy to miss some parts. I don't think the writing with hard and complex terminology is great paper at least for AI community due to rapid change and development. Please make the writing narrowed for the scope of vision language-perspective and pruning-perspective, although the authors want to say everything that you can come up with the expert concept.\n\n- Vague standard for simple and complex: As in figure 1, it represents layer-wise visual text interaction patterns where simeple task (object identification) and complex task (reasoning-intensive) feels some awkward. This is because according to question, I can recognize whether it is simple or comlpex task. However, in the perspective of feature visualization and saliency map, i don't think it converges on the salient region and remains stable, even when simple task is given. I think the problem setup should be more solid or more explored.\n\n- Applicability to more recent released models: Your models feel so out-dated because recently, there are so many VLMs such as Qwen2/2.5-VL or InternVL2.5/3-VL. Because none of these recent models, it may limits the evidecneof your solid experiments and applicability.\n\n- Lack of interesting phenomenon: Figure2 seems so interesting results but there are none of additional results for other benchmarks, because Figures2 is achieved on TextVQA only. I would highly recommend that kinda graphs for numerous benchmarks and have some analysis for them.\n\n- Challenging Evaluation Benchmarks: Your selected evaluation benchmarks are mainly from easy benchmarks including just selecting the multiplie choice or simple answer. However, there are so many challenging benchmarks including free-form answer (long context answer) like MM-Vet (or -v2) and really challening knowledge evlauation benchmark such as MMMU.\n\n- Overall, I think the motivation is somewhat lack and there are some redundent explanation beyond the main point that the authors should convey. This paper could be improved in terms of writing quality and clear standard to discriminate how we can acquire insight from salieny map because it is merely obvious phenomenon that many papers have reported. Furthermore, this paper must be more solid and having dense experimental resulsts for consistent phenomenon of numerous benchmarks and applicability to recent models.", "was_revised": true, "final_justification": "The authors addressed the concerns that initially I mentioned, so I've updated the score. Especially, I recommend the authors should handle the motivation part to be clarified and more provide examples of figures regarding dataset-wise retention ratio curves that is sure to be interesting.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We appreciate your emphasis on our training‑free plug‑and‑play design, cognitively inspired MI, and per‑sample budgeted logistic pruning.\n\n**Q1: The introduction of neuroscience.**\n\n**A1:** Thank you for the insightful comment. We will revise the manuscript to simplify our prose and reduce the use of specialized neuroscience terminology. We included those details because the human brain is intrinsically skilled at reasoning and modern vision–language models demonstrate similar inference behaviors. Many approaches draw inspiration from human reasoning processes and have achieved excellent results. For example, chain‑of‑thought prompting simulates step‑by‑step human reasoning.  Moreover, dual‑system, brain‑inspired approaches, known as fast and slow thinking, have been widely applied in autonomous driving and embodied intelligence domains [1–3]. Our approach aims to leverage these human visual reasoning strategies to enhance model performance. Without neuroscience‑inspired heuristics, designing an effective token pruning mechanism would be very challenging.\n\n[1] Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning\n\n[2] Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agents\n\n[3] Language‑Conditioned Robotic Manipulation with Fast and Slow Thinking\n\n**Q2: “Simple vs. Complex” in Fig. 1.**\n\n**A2:** Thank you for raising this point. We apologize for the confusion. Fig. 1 is intended to illustrate two observations: (1) a shared depth‑wise trend across tasks, and (2) different convergence dynamics between simple and complex questions.\n\n1. **Shared trend across tasks.** As depth increases, the visual modality receives progressively less attention, as shown in Fig. 1, by Layer 31 the model’s focus almost completely shifts toward language tokens. Since this depth‑wise reduction of visual attention has been widely discussed in prior work, we did not emphasize it in the main text; we will make this point explicit to avoid misinterpretation.\n2. **Different** **convergence** **dynamics by task difficulty.** For simple queries (Q1–Q3, object identification), attention shifts rapidly toward the target between layers 2 and 16, often by layer 4 or layer 8, and then remains stable. In contrast, for complex, reasoning-intensive queries (Q4–Q6), the model does not correctly localize the object by layer 4 and only begins to identify and stabilize the relevant region between layers 8 and 16. These divergent layer‑wise interaction patterns show that simple and complex questions follow distinct processing trajectories. This observation inspired our layer and depth aware pruning framework, which dynamically adjusts token retention to each task’s complexity and attention dynamics.\n3. **On saliency stability.** Your observation that saliency “does not converge on the most salient region” at large depth is consistent with (1) above: late layers are increasingly language‑dominated. Our analysis focuses on the emergence and stabilization of correct visual grounding in intermediate layers, where the simple vs. complex distinction is most diagnostic. \n\n**Q3: Applicability to more recently released models.**\n\n**A3:** We thank the reviewer for this valuable suggestion. As shown in Tab. 2 of the manuscript, we have already validated our approach on LLaVA‑NeXT, which was released contemporaneously with Qwen2. In response, we have extended our evaluation to include Qwen2‑VL. As shown below, our method continues to deliver consistent improvements over the prior state of the art on this model, demonstrating that it generalizes robustly across the latest vision–language models.\n\n| MMB               | Token   | MMB↑     | POPE↑    | TextVQA↑ |\n| ----------------- | ------- | -------- | -------- | -------- |\n| Qwen-2VL          | Dynamic | 80.5     | 86.4     | 84.3     |\n| SparseVLM (ICML 25) | 600     | 79.6     | 86.5     | 80.3     |\n| **Ours**          | **600** | **80.1** | **86.5** | **81.7** |\n| SparseVLM (ICML 25) | 400     | 79.0     | 85.8     | 77.1     |\n| **Ours**          | **400** | **79.4** | **86.1** | **78.6** |\n\n**Q4: Visualizations for numerous benchmarks.**\n\n**A4:** Because mutual‑information values inherently occupy a similar numerical range across datasets, the resulting curves exhibit only minor visual differences. Unfortunately, the NeurIPS rebuttal format does not permit the inclusion of new figures. We will, however, incorporate these additional plots and accompanying analysis for multiple benchmarks in the revision.\n\n**Q5: More challenging benchmarks.**\n\n**A5:** To address this, we have added experiments on the MMMU benchmark. As shown below, our approach continues to outperform the prior state of the art, confirming that it generalizes robustly even on more challenging, free‑form and knowledge‑heavy datasets. The full LLaVA-1.5-7B achieves an accuracy of 34.9 %. Even when retaining only 75 %, 50 % and 11.1 % of the tokens our method still surpasses the competing techniques. These findings demonstrate the strong generalizability of our approach.\n\n| Method               | 75%      | 50%      | 11.1%    |\n| -------------------- | -------- | -------- | -------- |\n| FasterVLM（ICCV 25） | 34.1     | 33.9     | 33.7     |\n| PyDrop (CVPR 25)     | 33.6     | 33.4     | 32.2     |\n| **Ours**             | **34.4** | **34.2** | **34.3** |\n\n**Q6: Motivation of AdaPrune.**\n\n**A6:** We explain our motivation from the following two aspects:\n\n- **Limitations of existing approaches.** Although previous pruning methods typically rely on the well‑known observation that visual attention mass decreases with layer depth, they apply the same pruning schedule to all tasks. This approach fails to adapt to each question’s unique visual–textual requirements and to the specific locations where visual evidence appears.\n- **Our perspective.** We complement attention‑mass views with spatial dynamics of attention: how and when the model localizes relevant regions as depth increases. Inspired by human problem solving (rapid fixation for simple cases vs. search for complex ones) and supported by our visualizations, we show that simple queries stabilize early, whereas reasoning‑intensive queries exhibit delayed or shifting localization. This depth‑wise localization dynamic directly motivates a task‑adaptive pruning policy rather than a one‑size‑fits‑all schedule.\n\n**Q7: Insight from** **the** **salient map.**\n\n**A7:** Prior work (e.g., FastV) predominantly reports that attention magnitude on visual tokens shrinks with depth. However, in VLM pruning the focus of attention and its evolution during reasoning have received little scrutiny. Our saliency maps make this spatiotemporal relocation explicit across different question types. We then model these regularities to guide pruning decisions: stable regions trigger more aggressive pruning elsewhere, whereas shifting patterns call for conservative pruning until convergence. In the revised manuscript, we will include a clear operational definition of our saliency metric and specify the criteria that render it actionable."}]

Author rebuttal:

Meta-review: The authors propose an adaptive, training-free pruning strategy for large vision language models named AdaPrune, that tailors pruning policies to varying sample and task complexities. In short, AdaPrune quantifies the mutual information between visual and textual tokens, and then projects this signal to a budget-constrained logistic retention curve, quantifying the specific complexity of different tasks. This informs per-task pruning. Experiments on LLaVA-1.5, LLaVA-NeXT, and a vision-language driving model demonstrate the pruning performance of the proposed algorithm.


During the rebuttal phase, the authors provided a significant amount of new results, and details on their approach. They  evaluated AdaPrune on a larger VLM (CogVLM-17B) and to  Qwen2‑VL, extended their analysis to the MMMU benchmark, compared to Turbo and FOLDER, explored prunning performance over the COCO Caption dataset, and provided evaluations on the sensitivity to hyperparameters and the quality of the MI estimation. They also clarified how the  mutual information score is mapped the logistic curve, the scoring function used for token selection, how MI is estimated/the choice of the layer for MI estimation, and how hyperparameters were set. All of these should be added to the paper.  

The same is true about the discussion on challenges when extending to streaming inputs, as well as the intuition provided w.r.t. certain design choices.
Decision: Accept (poster)