AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidance

## Abstract

Abstract Vision–language models (VLMs) have achieved impressive performance on multimodal reasoning tasks such as visual question answering (VQA), but their inference cost remains a significant challenge due to the large number of vision tokens processed during the prefill stage. Existing pruning methods often rely on directly using the attention patterns or static text prompt guidance, failing to exploit the dynamic internal signals generated during inference. To address these issues, we propose AdaptInfer, a plug-and-play framework for adaptive vision token pruning in VLMs. First, we introduce a fine-grained, dynamic text-guided pruning mechanism that reuses layer-wise text-to-text attention maps to construct soft priors over text-token importance, allowing more informed scoring of vision tokens at each stage. Second, we perform an offline analysis of cross-modal attention shifts and identify consistent inflection locations in inference, which inspire us to propose a more principled and efficient pruning schedule. Our method is lightweight and plug-and-play, also generalizable across multi-modal tasks. Experimental results have verified the effectiveness of the proposed method. For example, it reduces CUDA latency by 61.3% while maintaining an average accuracy of 92.9% on vanilla LLaVA-1.5-7B. Under the same token budget, AdaptInfer surpasses SOTA in accuracy.

## Introduction

In recent years, building on the success of LLMs (Bommasani et al. 2021; Touvron et al. 2023; Brown et al. 2020), vision–language models (VLMs) have emerged to tackle multimodal reasoning by combining visual encoders ((Liu et al. 2022; Dosovitskiy et al. 2021)), with LLMs’ text decoders (Du et al. 2022).
This integration enables impressive performance on tasks such as captioning (Lin et al. 2014), image retrieval (Faghri et al. 2018), and visual question answering (VQA) (Antol et al. 2015), but it also introduces a new computational challenge: the sheer number of vision tokens.

During the inference process of VLMs, the number of vision tokens is often much larger than that of textual tokens, sometimes by an order of magnitude or more.
For instance, an image of size $672\times 672$ processed by a visual encoder with a patch size of $14\times 14$ typically results in $2304$ vision tokens (Radford et al. 2021; Zhang et al. 2025), whereas the corresponding text prompt may contain fewer than 100 tokens (Hudson and Manning 2019; Fu et al. 2023). Also, much previous research suggests that the vision tokens are more redundant and semantically repetitive (Zhang et al. 2024; Tong et al. 2025).

As a result, explorations in VLM acceleration primarily focus on the efficient pruning, compression or sparsification of vision tokens. This paradigm aims to reduce computational overhead by retaining only the most valuable vision tokens. Among them, some works introduce sparsity strategies within the visual encoders to generate less but useful enough vision tokens (Li, Wang, and Jia 2024; Chen et al. 2025b), while others further prune tokens based on either self-attention or cross-attention patterns during the prefill stage (Lin et al. 2025; Xing et al. 2024; Chen et al. 2025a; Li et al. 2025).
Nevertheless, not all the attention logits should be involved in the vision token ranking for the dispersion of the full attention patterns (Zhang et al. 2024). Granting voting rights only to the most salient tokens sharpens guidance, enabling more aggressive yet accurate vision‑token pruning.

To address this, SparseVLM (Zhang et al. 2025) introduces the concept of text prompt–guided pruning, selecting the most salient text tokens offline before the prefill pass. While this approach acknowledges the importance of textual cues, it does not fully address the underlying challenge: the dynamic nature of text token importance during inference. In practice, the informativeness of text tokens evolves across layers as the model progressively refines its internal representations (Tenney, Das, and Pavlick 2019; Clark et al. 2019). Our observations in Figure [2](https://arxiv.org/html/2508.06084v1#Sx3.F2) also indicate that the most prominent text tokens vary significantly across layers, making any static selection inherently suboptimal.

Figure: Figure 1: Illustration of Dynamic Text‑Guided Attention in VLMs. A schematic illustration showing the set of key text tokens and the corresponding visual regions shift across transformer layers.
Refer to caption: x1.png

Therefore, to truly harness the benefits of text-guided sparsification, it is crucial to develop a pruning strategy matched with dynamic fluidity of information, reflecting the effective cross-modal interaction throughout the inference process. In this work, we propose to reconstruct the dynamic importance ranking of text tokens at each layer by utilizing the text-to-text (t2t) attention maps. These attention maps provide a natural, layer-specific prior distribution for text-token importance, which we then use to reweight text-to-vision (t2v) attention scores for vision token pruning. Importantly, since the t2t attention maps can be directly extracted from the model’s attention computations, our method does not introduce additional computational overhead.

Moreover, current methods determine pruning hyperparameters (e.g. the pruning locations) primarily through either empirical rule-of-thumb or extensive hyperparameter optimization experiments (Xing et al. 2024; Chen et al. 2025a; Zhang et al. 2024). However, we argue that relying on manual tuning or grid search not only imposes substantial offline computational overhead, but also leads to task- or dataset-specific heuristics.
In this work, we take the first step in providing a principled pruning schedule. We provide our insights based on systematically analyzing the distributional characteristics of attention shifts to vision tokens during VLM inference. Specifically, we identify consistent attention inflection points at layer 1, 10, and 20 in LLava-1.5-7B (Liu et al. 2023b), suggesting that aggressive pruning immediately after these layers is a more effective and computationally efficient strategy on LLava.

The solutions we propose effectively address the limitations of existing works in the field of vision token sparsification for VLM acceleration. Our main contribution involves:

- •
We propose AdaptInfer, an adaptive vision token sparsification framework in which VLM dynamically determines text token guidance during inference. AdaptInfer is a plug-and-play solution.
- •
We introduce a novel observation of the distributional characteristics of attention shifts, and gain insights in a more effective and reasonable pruning schedule.
- •
We implement and evaluate our proposed solution, AdaptInfer, across multiple benchmarks and different vision token budget settings. Within the same token budget, our AdaptInfer outperforms state-of-the-art (SOTA) methods on the metric of the accuracy.

## Related Work

In this section, we will briefly introduce the previous works that are correlated with ours.

### Vision-Language Models

Early multi-modal systems paired convolutional vision backbones with recurrent language decoders (Karpathy and Fei-Fei 2017; Vinyals et al. 2015). Modern VLMs instead follow the Transformer paradigm (Vaswani et al. 2017) by representing an image as a sequence of *visual tokens* that interact with textual tokens in a shared self-attention space (Liu et al. 2023b; Chen et al. 2024a, b).

BLIP-2 (Li et al. 2023) and MiniGPT-4 (Zhu et al. 2023) introduce lightweight linear adapters that project features from a frozen CLIP encoder into the hidden space of a large language model, enabling efficient training. These explorations bridge the gap between frozen encoders and LLMs. The LLaVA family (Liu et al. 2023a, b, 2024a) refines this recipe with stronger instruction tuning, while a few other efforts such as Flamingo (Alayrac et al. 2022), CogVLM (Wang et al. 2024) and GPT-4V (OpenAI 2023) scale the approach to billions of parameters.

Recent explorations also contribute to resolution and multi-modal extensions. Models leveraging hierarchical perception (e.g., LLaVA‑NeXT (Liu et al. 2024a)) and adaptive patching (e.g., Qwen‑VL (Bai et al. 2023)) permit high‑resolution inputs, while large-scale vision encoders are adopted to generate richer hidden states of the vision tokens (Chen et al. 2024a, b). However, these gains come at the cost of dramatically more vision tokens, which is a key bottleneck addressed by our work.

### Inference Acceleration of VLMs

Previous approaches mainly focus on vision token sparsification. This is because the number of vision tokens is often an order of magnitude (or more) larger than that of text tokens. In addition, visual embeddings are naturally much more sparse and repetitive than human-made texts (Marr 2010). In this field, there are two research directions including efficient vision encoders and vision token pruning in LLM networks.

For example, methods like LLaVA-PruMerge (Shang et al. 2025) and FlowCut (Tong et al. 2025) follow the first direction, which cuts the encoder outputs or uses a lightweight projector to reduce the number of vision tokens. Recoverable compression (Chen et al. 2025b) is also introduced to repair the information loss from token pruning within vision encoders. Solutions follow the second direction not only to drop the vision tokens (Chen et al. 2025a; Lin et al. 2025; Xing et al. 2024; Li et al. 2025; Luan et al. 2025; Ye et al. 2025) during prefill, but also merge them to compress the numbers of the vision tokens (Bolya et al. 2023) and recover them in certain inference stage (Wu, Chen, and Wang 2025). SparseVLM (Zhang et al. 2025) tries to go deeper in exploration of static text prompt guidance but ignoring the evolving inherent of token information.

## Method

Figure: Figure 2: MIoU of key text tokens in different layers. Most values are lower than 0.5, which indicates the text token importance varies significantly.
Refer to caption: figures/layer_heatmap.png

Figure: Figure 3: Layer-wise distribution of attention shifts. The shift points appears consistently on two datasets. Results are aggregated over 1,000 samples for each dataset.
Refer to caption: figures/mme2.png

In this section, we will introduce our observations and the methods we propose.

Figure: Figure 4: The architecture of AdaptInfer. Text token importance is computed and guides vision token selection at every pruning layer adaptively. The right panel illustrates the internal computation details of the Rank & Prune module. Step I: Compute the text token prior. Step II: Use the transposed prior to reweight the t2v attention. Step III: Rank the vision tokens and prune the least informative ones.
Refer to caption: x2.png

### Observations

In this subsection, we will present a few observations and preliminary experiments that have illuminated our insights.

#### Text Token Importance

Given an input question prompt, we first tokenize it as $Prompt=[t_{1},\ldots,t_{n}]$. We then define the importance of a text token $t_{i}$ at layer $\ell$ as the total attention weight it receives from all text tokens in the layer-$\ell$ text-to-text (t2t) attention map:

$$ $\mathrm{Imp}_{\ell}(t_{i})=\sum_{j=1}^{n}\mathbf{A}^{(\ell)}_{\text{t2t}}[j,i],$ (1) $$

where $\mathbf{A}^{(\ell)}_{\text{t2t}}[j,i]$ denotes the attention directed
from token $t_{j}$ to $t_{i}$. Intuitively, tokens with high
$\mathrm{Imp}_{\ell}$ are the current key text tokens of
the language stream and are therefore best suited to guide cross-modal
pruning decisions at that layer.

#### Dynamics of Text Token Importance

To demonstrate that the importance of text tokens evolves significantly across layers, we conduct a simple empirical study on LLava-1.5-7B (Liu et al. 2023b). LLava-1.5-7B contains a 32-layer LLaMa (Touvron et al. 2023) as its language model. We extract t2t attention maps from 1,000 samples in the TextVQA dataset. On average, each sample contains approximately 100 text tokens.
At each chosen layer, we select the top 20% text tokens as the key text tokens of that layer.
We then compute the mean Intersection over Union (mIoU) between the indexes of the selected top tokens across layers, as reported in Figure [2](https://arxiv.org/html/2508.06084v1#Sx3.F2). The value in the i-th row and j-th column represents the mIoU of the key text token indexes of the i-th and j-th layers. Note that this figure is symmetrical, and the miou values along the diagonal are always 1 because they are comparisons within the same layer.

The consistently low MIoU values between different layers indicate that the set of the key text tokens changes substantially during inference. For example, the 0.169 mIoU between layer 0 and 24 refers that only 16.9% of key text tokens are overlapped while all others are different. This highlights the inherent dynamics of text token importance, where the VLM attends to different parts of the input question at different stages of reasoning. Consequently, any static text-prompt–guided pruning approach is likely to fail in capturing this evolving semantic alignment. These findings support the need for an adaptive, layer-wise text-guidance mechanism that can track and respond to the evolving attention distribution online.

#### Cross-Attention Shifts of VLM

We argue that a principled approach to setting pruning hyperparameters is not only necessary but preferable to complex, trial-and-error–based tuning. To support this claim, we performed an analysis to investigate the locations of cross-attention shifts during inference.

Specifically, we calculate cumulative text-to-vision (t2v) attention scores for visual tokens at each layer using LLava-1.5-7B (Liu et al. 2023b). For each sample, we first select the top 10% of vision tokens, approximately 58 tokens per image, which receive the highest total attentions in the prefill stage. Similar to the text tokens, these vision tokens are assumed to be the most critical ones for the corresponding image-based VQA task.

To understand when these important tokens become semantically salient, we analyze their cumulative attention trajectories across transformer layers. We apply change-point detection (Truong, Oudre, and Vayatis 2020) on each curve to identify the layer where the model’s attention pattern changes significantly. The technical details of change-point detection will be presented in the appendix. Intuitively, an attention shift point may indicate that either (1) the token begins to receive significantly more attention, becoming critical, or (2) the token’s informative content has already been fully extracted, becoming redundant. In both cases, these shifts mark the layer-wise transitions in how the model utilizes visual information.

Figure [3](https://arxiv.org/html/2508.06084v1#Sx3.F3) shows the distribution of the attention shift locations aggregated from 1,000 samples each in the MME and TextVQA datasets. Despite data set differences, we observe a highly consistent trend: attention shifts cluster densely in layer 1 and round layers 10-20, while layers 2-8 and 20 + show a frequency of near-zero attention shifts.

These findings provide a data-driven basis for pruning schedule design, where pruning locations are informed by the model’s own attention behavior, rather than by empirical intuition alone.

### Adaptive Token Pruning

Based on the two key observations above, we propose AdaptInfer for VLM inference acceleration.

#### Dynamic Text Guidance Pruning

To guide visual token pruning in a more informed and adaptive way, we propose a dynamic text-guidance mechanism that leverages the model’s internal attention signals. Instead of statically selecting a fixed subset of text tokens before entering the language model (Zhang et al. 2025), we dynamically infer their relative importance during inference at each predefined pruning layer. We reuse the attention maps already computed by the model. The architecture of Our AdaptInfer is shown in Fig [4](https://arxiv.org/html/2508.06084v1#Sx3.F4), and our dynamic text guidance mechanism can be divided into three steps.

Firstly, at each pruning layer, we extract the text-to-text attention matrix $\mathbf{A}_{\text{t2t}}\in\mathbb{R}^{T\times T}$, where $T$ is the number of text tokens. To estimate the importance of each text token, we aggregate attention scores across the query dimension:

$$ $\mathbf{w}=\sum_{i=1}^{T}\mathbf{A}_{\text{t2t}}[i,:]\in\mathbb{R}^{T},$ (2) $$

where $\mathbf{w}$ serves as a soft prior distribution over the text tokens, indicating how much attention each token receives from the rest of the sequence.

Secondly, we then use this prior to reweight the t2v attention matrix $\mathbf{A}_{\text{t2v}}\in\mathbb{R}^{T\times V}$ at the same layer, where $V$ is the number of visual tokens remained. $\mathbf{A}_{\text{t2v}}$ denotes the cross-attention matrix where text tokens are used as queries, and vision tokens serve as keys and values. The importance score of each visual token is computed as:

$$ $\mathbf{s}=\mathbf{w}^{\top}\cdot\mathbf{A}_{\text{t2v}}\in\mathbb{R}^{V}.$ (3) $$

Here, $\mathbf{s}_{j}$ reflects the aggregated and weighted attention from all text tokens to visual token $j$.

Finally, based on these scores, we rank all visual tokens and retain the top-$k$ for the current layer:

$$ $\mathcal{I}_{k}=\text{TopK}(\mathbf{s},k).$ (4) $$

Importantly, all text tokens participate in visual token scoring, but contribute in proportion to their dynamically inferred importance. Moreover, because both $\mathbf{A}_{\text{t2t}}$ and $\mathbf{A}_{\text{t2v}}$ are natively computed in standard forward passes, our method introduces little additional computational overhead. Note that this solution follows a training-free paradigm and can be seamlessly integrated as a plugin into existing VLMs.

#### Analysis of Computational Complexity

Assuming $n=T+V$ denotes the current sequence length, $d$ denotes the VLM’s hidden state dimension, and $m$ denotes the hidden size of projection layer in the FFN network. For each transformer layer in the prefill stage, the FLOPs can be estimated by

$$ ${FLOPs^{\text{prefill}}}=4nd^{2}+2n^{2}d+3ndm.$ (5) $$

For each pruning layer, the additional FLOPs of are computed below:

$$ ${FLOPs^{\text{prune}}}=T^{2}+2TV.$ (6) $$

Since both attention matrices are already computed during the forward pass, this additional cost is minimal relative to the main transformer computations.

During the decode stage, the FLOPs of each layer can be estimated by

$$ ${FLOPs^{\text{decode}}}=4d^{2}+2nd+3dm.$ (7) $$

#### Layer-wise Pruning Schedule

Following the attention shift analysis described above, we design a pruning schedule that aligns with the attention dynamics observed in VLMs. Our goal is to perform aggressive pruning while maintaining the inference accuracy. To balance the trade-off, we suggest that pruning should avoid the regions with intense attention shifts. Because the probability of incorrect pruning is quite high, tokens that are not important in the past layers are likely to become important in the next layer. On the contrary, pruning just before or after the dense regions can reduce this risk to some extent.

Based on the consistent attention shift distributions across both MME and TextVQA datasets, we select layers 1, 10, and 20 as pruning locations in our framework. Pruning at layer 1 implies performing a forward pass through the first layer, and subsequently using its attention outputs to identify and remove redundant vision tokens based on their ranked importance. Since layer 1 exhibits the earliest dense attention shift, retaining the salient vision tokens at this stage allows them to effectively participate in subsequent information exchange. Layer 10 begins to mark a region of high attention volatility until layer 20. As a result, we preserve the vision tokens within this range, only compress the token numbers at the beginning and the end. At layer 20 and after, where shift activity is negligible, we suggest that the cross-modality information exchange is already sufficient, so we apply aggressive pruning at this layer by removing nearly all remaining visual tokens.

This schedule balances caution and efficiency. Compared to heuristic or uniform pruning schemes, our approach is both data-driven and architecture-aware, requiring no expensive hyperparameter tuning while generalizing well across tasks and datasets.

### Discussion

The chosen pruning hyperparameters come from empirical observations on LLaVA-1.5-7B, and thus are tailored specifically to VLMs built upon the LLaMA-7B backbone. Nevertheless, our proposed adaptive pruning schedule is generalizable for it can be easily transferred to other models with different parameter scales or architectures by performing a simple, offline attention shift analysis. The extra experiments will be presented in the technical appendix.

Moreover, one of the datasets used in our observational studies is MME (Fu et al. 2023), a comprehensive multimodal benchmark comprising two major categories and fourteen subcategories. The consistent statistical patterns indicate that the attention dynamics are largely stable and transferable across different types of multimodal tasks.

## Experiment

**Table 1: Comparison of methods under different pruning budgets on LLava-1.5-7B. We report results on five datasets, average retained tokens, accuracy, overall ratios and FLOPs. Results in bold present the highest accuracy under same pruning budgets.**
| Method | Avg. Tokens | MME | GQA | MMBench | SQA | TextVQA | Acc. Ratio (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Vanilla | 576 | 1864 | 61.9 | 64.6 | 69.5 | 58.3 | 100 |
| ToMe (ICLR23) | 128 | 1343 | 52.4 | 53.3 | 59.6 | 49.1 | 81.8 ($\downarrow$ 18.2) |
| FastV (ECCV24) | 128 | 1490 | 49.6 | 56.1 | 68.6 | 52.5 | 87.1 ($\downarrow$ 12.9) |
| PDrop (CVPR25) | 128 | 1761 | 57.1 | 61.6 | 68.4 | 56.6 | 95.5 ($\downarrow$ 4.5) |
| SparseVLM (ICML25) | 128 | 1746 | 58.4 | 64.5 | 68.6 | 56.7 | 96.8 ($\downarrow$ 3.2) |
| AdaptInfer (Ours) | 128 | 1796 | 58.5 | 64.0 | 70.0 | 57.0 | 97.7 ($\downarrow$ 2.3) |
| ToMe (ICLR23) | 64 | 1138 | 48.6 | 43.7 | 50.0 | 45.3 | 71.4 ($\downarrow$ 28.6) |
| FastV (ECCV24) | 64 | 1255 | 46.1 | 47.2 | 68.7 | 45.9 | 78.5 ($\downarrow$ 21.5) |
| PDrop (CVPR 25) | 64 | 1561 | 47.5 | 58.8 | 69.0 | 50.6 | 87.5 ($\downarrow$ 12.5) |
| SparseVLM (ICML 25) | 64 | 1589 | 53.8 | 60.1 | 69.8 | 53.4 | 91.4 ($\downarrow$ 8.6) |
| AdaptInfer (Ours) | 64 | 1652 | 53.5 | 61.3 | 70.0 | 54.7 | 92.9 ($\downarrow$ 7.1) |

### Experimental Settings

#### Datasets

For multimodal evaluation, we test our solutions on five widely used benchmarks, including MME (Fu et al. 2023), GQA (Hudson and Manning 2019), MMBench (Liu et al. 2024b), ScienceQA (SQA) (Lu et al. 2022) and TextVQA (Singh et al. 2019).

#### Baselines

We select four classic and latest VLM acceleration works in the field of vision token sparsification within the plug-and-play paradigm in LLM forward pass as baselines, including FastV (Chen et al. 2025a), ToMe (Bolya et al. 2023), Pyramid Drop (PDrop) (Xing et al. 2024) and SparseVLM (Zhang et al. 2025). Note that we only use the training-free version of PDrop to fit our requirements.

#### Implement Details

To ensure a fair and comprehensive comparison between baseline methods and AdaptInfer, we report the results in different average retained token budgets (e.g. 128, 64, 48 and 32). AdaptInfer utilizes the pruning parameters described above while other baselines keep their original settings for all core components. Our experiments are performed on two different types of VLMs, including LLava-1.5-7B (Liu et al. 2023b) and InternVL-chat-7B (Chen et al. 2024a). Our experiments are conducted on a server with 8 $\times$ Nvidia RTX 4090 24G with a cpu of Intel Xeon Platinum 8358P.

### Main Results

Figure: Figure 5: Performance trends on two datasets. AdaptInfer outperforms SparseVLM and PDrop across all retained token budgets.
Refer to caption: figures/mme_plot.png

**Table 2: Performance of AdaptInfer on InternVL. Our method still maintain high accuracy.**
| Tokens | MME | SQA | TextVQA | Ratio (%) |
| --- | --- | --- | --- | --- |
| Origin | 1849 | 69.1 | 56.9 | 100 |
| 128 | 1758 | 69.2 | 55.7 | 97.7 |
| 64 | 1528 | 69.0 | 53.0 | 91.9 |
| 32 | 1183 | 68.1 | 46.7 | 81.5 |

**Table 3: Latency test of AdaptInfer. While the FLOPs estimations are close, our method reduces more real CUDA latency.**
| Method | Tokens | Metrics | MME | GQA | MMBench | TextVQA | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Vanilla | 576 | FLOPs (T) | 4.268 | 4.250 | 4.623 | 4.611 | 4.438 |
|  |  | Latency (ms) | 82.0 | 76.5 | 91.3 | 91.2 | 85.3 |
| PDrop | 64 | FLOPs (T) | 0.975 | 0.958 | 1.316 | 1.305 | 1.138 ($\downarrow$ 74.3%) |
|  |  | Latency (ms) | 33.0 | 32.0 | 36.4 | 36.6 | 34.5 ($\downarrow$ 59.6%) |
| SparseVLM | 64 | FLOPs (T) | 0.974 | 0.958 | 1.316 | 1.305 | 1.138 ($\downarrow$ 74.3%) |
|  |  | Latency (ms) | 34.4 | 34.7 | 38.1 | 39.5 | 36.7 ($\downarrow$ 57.0%) |
| AdaptInfer (Ours) | 64 | FLOPs (T) | 0.975 | 0.959 | 1.317 | 1.305 | 1.139 ($\downarrow$ 74.3%) |
|  |  | Latency (ms) | 31.0 | 30.9 | 34.5 | 35.5 | 33.0 ($\downarrow$ 61.3%) |

In Table [1](https://arxiv.org/html/2508.06084v1#Sx4.T1), we report the performance of AdaptInfer on LLava-1.5-7B. We provide comparisons with other baselines under average retained vision token numbers of 128 and 64, with metrics of inference accuracy on each dataset. The accuracy has an upper bound of Vanilla LLava with all 576 tokens kept, so an overall accuracy ratio is supported by each baseline. As shown in this Table, AdaptInfer achieves the highest overall accuracy under both 128 and 64 vision token budgets. Under 128 token budgets, our average accuracy ratio reaches 97.7%, which is 0.9% higher than the second-best method, SparseVLM. While this may seem like a small margin, it is in fact approaching the upper bound of Vanilla LLava. In practice, retaining only 64 tokens on average reduces the prefill computational load by 88.9%, enabling significantly more efficient inference. Despite this aggressive pruning, our framework still preserves 92.9% of the original inference accuracy, and outperforms the second-best method, which scores 91.4%, with a clear improvement of 1.5%. The results highlight the robustness of our approach even under tight token constraints.

In Figure [5](https://arxiv.org/html/2508.06084v1#Sx4.F5), we present the performance trends of AdaptInfer with the latest baseline methods SparseVLM and PDrop. AdaptInfer outperforms others in all retained token budgets with a clear margin.

We further evaluate the proposed AdaptInfer on InternVL‑Chat‑7B (Chen et al. 2024a) in Table [2](https://arxiv.org/html/2508.06084v1#Sx4.T2). InternVL adopts a more powerful 6B parameter ViT encoder, which produces finer‑grained vision tokens with richer hidden states. Experiments confirm that our token‑sparsification strategy preserves the original semantic richness of these vision features with high inference accuracy. Under token budgets of 128 and 64, AdaptInfer maintains a high accuracy ration of 97.7% and 91.9% respectively. However, in the case of extreme pruning, where only 32 average vision tokens are retained, the accuracy has a significant drop, reaching only 81.5%. This is because under such extreme pruning conditions, a large number of informative tokens were inevitably removed, which affected the performance.

### Latency Test

In addition, we conduct a comparative analysis of CUDA latency time, and FLOPs on LLaVA-1.5-7B, in order to show the real acceleration performance. The results are shown in Table [3](https://arxiv.org/html/2508.06084v1#Sx4.T3). This experiment is carried out on a single NVIDIA RTX 4090 24G GPU. All results are the average values per sample.

We replicate the performance of PDrop (Xing et al. 2024) and SparseVLM (Zhang et al. 2025) under 64-token budget, and compare them with that of AdaptInfer. The comparisons are based on four commonly used benchmarks, including MME, GQA, MMBench and TextVQA. According to the table, while theoretical FLOPs estimations are close, our implementation on AdaptInfer reaches a lower average cuda latency of 33.0 ms per sample than 34.5 ms and 36.7 ms by PDrop and SparseVLM. After all, our method rarely introduces additional computational load, such as any extra attention computation or offline feature matching step.

**Table 4: Performance w/ and w/o dynamic text-guidance. Comparisons are between static (SparseVLM) and dynamic text‑guidance (Ours) methods under extreme pruning on LLava-1.5-7B.**
| Method | Tokens | MME | TVQA | Ratio (%) |
| --- | --- | --- | --- | --- |
| Vanilla | 576 | 1864 | 58.3 | 100 |
| SparseVLM | 48 | 1416 | 48.1 | 79.2 |
| AdaptInfer | 48 | 1490 | 53.1 | 85.5 |
| SparseVLM | 32 | 1098 | 47.7 | 70.4 |
| AdaptInfer | 32 | 1190 | 49.8 | 74.6 |

### Ablation Study

Firstly, in order to illustrate the necessity of dynamic text guidance, we provide an experiment to compare results with a static text guidance work SparseVLM in Table [4](https://arxiv.org/html/2508.06084v1#Sx4.T4). Also, we conduct this study as an exploration of extreme pruning. In the experiment, AdaptInfer consistently outperforms SparseVLM on MME and TextVQA (TVQA) under extreme low pruning budgets of only 48 and 32 vision tokens retained averagely. Specifically, AdaptInfer can keep a relatively high overall performance of 85.5% and 74.6% respectively. These results prove the effectiveness of our dynamic text guidance design.

Furthermore, we conduct an intuitive study to assess the effectiveness of our pruning hyperparameters, which are inspired by the observations. We compare our observation-driven pruning strategy with three baselines: uniform pruning, single-layer pruning, and random-layer pruning which averages results over five random configurations. The results shown in Table [5](https://arxiv.org/html/2508.06084v1#Sx4.T5) confirm that the chosen hyperparameters are the best suited for AdaptInfer on the LLaMa-7B backbone.

**Table 5: Performance of different pruning locations on LLava-1.5-7B with 128 tokens. The chosen hyperparameters, inspired by our observations, yield the highest scores.**
| Method | Pruning Loc. | MME | GQA | TextVQA |
| --- | --- | --- | --- | --- |
| Ours | 1,10,20 | 1796 | 58.5 | 57.0 |
| Uniform | 0,9,18,27 | 1792 | 57.6 | 56.2 |
| Uniform | 2,12,22 | 1764 | 57.5 | 56.7 |
| Single | 1 | 1647 | 53.7 | 56.3 |
| Random | - | 1693 | 56.4 | 56.0 |

## Conclusion

This paper proposes a novel plug-and-play solution AdaptInfer for VLM acceleration via dynamic text-guided pruning. We further provide an offline analysis of cross-attention shifts, which motivates a principled pruning schedule. Our VLM acceleration plugin introduces minimal additional computational overhead while maintaining high accuracy. Specially, our AdaptInfer achieves SOTA accuracy under all the token budgets chosen in the experiments. For instance, AdaptInfer reduces CUDA latency by 61.3% and retains an average accuracy of 92.9% on LLaVA-1.5-7B, with only 64 vision tokens preserved per layer on average.

## Appendix

## Appendix A Change-Point Detection

To identify shifts in cross-attention across transformer layers, we employ change-point detection (Truong, Oudre, and Vayatis 2020) on cumulative cross-attention curves. The intuition is that a significant change in attention behavior corresponds to a noticeable inflection in the cumulative attention distribution over layers.

Given a token’s cross-attention curve across $L$ layers, denoted as:

$$ $\mathbf{a}=[a_{1},a_{2},\dots,a_{L}],\quad\text{where }a_{\ell}\in\mathbb{R}$ (8) $$

we first compute the cumulative attention:

$$ $\mathbf{c}=\left[\sum_{i=1}^{1}a_{i},\sum_{i=1}^{2}a_{i},\dots,\sum_{i=1}^{L}a_{i}\right]$ (9) $$

This sequence $\mathbf{c}$ captures the aggregate build-up of attention, which we treat as a univariate time series. We then apply change-point detection to identify the most likely layer index where a shift in trend occurs.

We adopt the ruptures Python library(^1^11https://github.com/deepcharles/ruptures) for offline detection. Specifically, we use the Binary Segmentation (Binseg) algorithm with an $\ell_{2}$ cost model. This algorithm aims to find a segmentation point that minimizes the within-segment variance. Formally, given the cumulative curve $\mathbf{c}=[y_{1},y_{2},\dots,y_{L}]$, we solve:

$$ $\min_{b}\left(\sum_{t=1}^{b}(y_{t}-\mu_{1})^{2}+\sum_{t=b+1}^{L}(y_{t}-\mu_{2})^{2}\right),$ (10) $$

where $\mu_{1}$ and $\mu_{2}$ are the mean values of the first and second segments, respectively. This corresponds to the $\ell_{2}$ cost model used in ruptures, which measures the homogeneity of each segment by its squared deviation from the mean.

We restrict the number of change points to 1, making the detection both efficient and robust. The returned breakpoint $b\in\{1,\dots,L\}$ indicates the first significant shift in attention accumulation for the given token. This layer index is treated as the token’s cross-attention shift point, which guides our downstream pruning strategy.

## Appendix B Attention Shift Detection on LLava-1.5-13B

Figure: Figure A.6: Layer-wise distribution of attention shifts on LLava-1.5-13B.
Refer to caption: figures/13b_mme.png

Figure: Figure A.7: Visualization of AdaptInfer on different examples.
Refer to caption: x3.png

To validate the transferability of our proposed pruning schedule, we conduct attention shift detection on LLaVA-1.5-13B (Liu et al. 2023b), which is built upon a 40-layer LLaMA-2-13B backbone (Touvron et al. 2023). The resulting distributions on two benchmark datasets, MME and TextVQA, are presented in Figure [A.6](https://arxiv.org/html/2508.06084v1#A2.F6). We observe that the distribution of attention shift points still remains highly consistent on MME and TextVQA. The early-stage concentration of attention shifts consistently occurs at layer 1. Meanwhile, in later layers, dense change points are slightly delayed compared to those in LLaVA-1.5-7B, which typically occurs around layers 11 to 22. This suggests that for LLaVA-1.5-13B and other VLMs built on the LLaMA-2-13B backbone, an appropriate pruning schedule should select representative layers such as 1, 11, and 22.

Moreover, our attention shift detection pipeline is both simple and lightweight. For example, on the MME dataset, extracting 1000 attention maps and performing change-point detection takes only about 6 minutes in total. This confirms the efficiency and transferability of our pruning schedule across different model scales and architectures.

## Appendix C Performance of AdaptInfer on LLava-1.5-13B

Using the parameters above, we test our AdaptInfer on LLava-1.5-13Be to further validate the effectiveness of the proposed pruning schedule. The experiments are based on three widely used dataset MME (Fu et al. 2023), MMBench (MMB) (Liu et al. 2024b), and TextVQA (Singh et al. 2019) under 128 and 64 vision token budgets. The results are shown in Table [A.6](https://arxiv.org/html/2508.06084v1#A3.T6).

**Table A.6: Performance of AdaptInfer on LLava-1.5-13B. Our method still maintain high overall accuracy.**
| Tokens | MME | MMB | TextVQA | Ratio (%) |
| --- | --- | --- | --- | --- |
| 576 | 1826 | 61.2 | 68.5 | 100 |
| 128 | 1803 | 59.9 | 68.0 | 98.6 |
| 64 | 1759 | 57.9 | 65.7 | 95.6 |

Under the more demanding 13-billion-parameter setting of LLaVA-1.5, AdaptInfer continues to deliver high performance. As Table [A.6](https://arxiv.org/html/2508.06084v1#A3.T6) shows, pruning the vision token numbers from 576 to 128 on average per layer results in only a 1.4% drop in overall accuracy (from 100% to 98.6%). Even with an aggressive 64 vision token budget, AdaptInfer achieves an 8.9× reduction in vision-token count, the model still retains 95.6% of its full-token accuracy. These numbers demonstrate that our token-adaptive pruning solution scales gracefully with model size, confirming AdaptInfer’s robustness.

## Appendix D Visualization

To qualitatively assess the pruning effect of the proposed AdaptInfer, we conduct an additional visualization study. In line with the main paper, vision tokens are pruned after the 1st, 10th, and 20th transformer layers. The resulting figures depict the spatial distribution of the remaining tokens at each stage: transparent regions correspond to tokens that survive pruning, whereas semi-transparent patches indicate positions that have been discarded.

The experiment is carried out on the TextVQA (Singh et al. 2019) benchmark, from which we sample a representative set of examples for display. As the layers deepen, AdaptInfer rapidly suppresses tokens associated with semantically irrelevant background areas while consistently preserving those aligned with question-critical cues. The visualization is shown in Figure [A.7](https://arxiv.org/html/2508.06084v1#A2.F7).

## Appendix E Performance Analysis

Figure: Figure A.8: Examples where AdaptInfer outperforms SparseVLM. AdaptInfer performs better when text prompts are rich in descriptive modifiers.
Refer to caption: x4.png

We provide further analysis of the advantages of our dynamic text guidance method, AdaptInfer, compared with the static text guidance solution, SparseVLM (Zhang et al. 2025). We extract some VQA examples from the TextVQA dataset (Singh et al. 2019), and the answers of both methods are presented in Figure [A.8](https://arxiv.org/html/2508.06084v1#A5.F8).

Within each question, blue highlights denote the static key text tokens selected offline by SparseVLM, whereas red highlights trace the key text tokens adaptively chosen from various layers by AdaptInfer. Close inspection reveals a consistent pattern: SparseVLM almost exclusively locks onto the text tokens that bear the strongest direct semantic link to the visual scene (e.g., printer or players), but often discards adjectival or adverbial modifiers that disambiguate entities (e.g., pointing, two or behind). When the prompt becomes syntactically richer and contains multiple content nouns and stacked modifiers static text guidance leads to an under-specified textual context and, consequently, a notable drop in answer accuracy. Conversely, AdaptInfer updates its key text token set dynamically during inference, rarely missing any prominent guidance. This dynamical adjustment make AdaptInfer overcome such limitation.

## Appendix F Experiment Setup

In this appendix section, we will briefly introduce the details of our experiment setups.

### Hardware Environment

To facilitate reproducibility, we ran every experiment on a single workstation that hosts an Intel Xeon Platinum 8358P processor together with eight NVIDIA RTX 4090 GPUs, each furnished with 24 GB of VRAM. The system operates under Ubuntu Linux 24.04, and all code was compiled using GCC 13.2.0 with Binutils 2.42 as the linker.

### Software Configuration

To improve reproducibility, we provide the main software configuration employed in our experiments. All runs were performed with the package versions listed below. We introduce the core deep-learning frameworks first, follow with performance-oriented extensions, and conclude with supporting libraries.

- •
Python: 3.10.18
- •
PyTorch: 2.1.2
- •
Transformers: 4.37.0
- •
Accelerate: 0.21.0
- •
Flash-Attn: 2.3.3
- •
LLaVA: 1.7.0.dev0
- •
Tokenizers: 0.15.1
- •
TorchVision: 0.16.2
- •
ruptures: 1.1.9

### Implement Detail

In the paper, we evaluated three vision–language models, including LLaVA-1.5-7B, LLaVA-1.5-13B, and InternVL-Chat-7B (Chen et al. 2024a) in total. To stay within GPU-memory limits, we ran the LLava-7B variant on one GPU, the LLava-13B variant on three GPUs, and InternVL-Chat-7B on two GPUs (InternVL-Chat-7B contains a 6B-parameter vision encoder). All training and inference are performed in torch.float16 precision to further conserve memory usage. All the results of AdaptInfer reported in the context are the average of five runs.

### Randomness Report

Table 5 in the main paper contrasts our proposed layer-wise pruning schedule with three baselines: uniform‐layer pruning, single‐layer pruning, and random‐layer pruning.
For the random baseline, we executed five independent runs, each time drawing a different set of transformer layers to prune while keeping the token budget fixed.
The sampled pruning locations are {3}, {2,15}, {2,8,16}, {2,4,8,16}, {3,6,23}.

## References

- Alayrac et al. (2022)
Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millicah, K.; Reynolds, M.; Ring, R.; Rutherford, E.; Cabi, S.; Han, T.; Gong, Z.; Samangooei, S.; Monteiro, M.; Menick, J.; Borgeaud, S.; Brock, A.; Nematzadeh, A.; Sharifzadeh, S.; Binkowski, M.; Barreira, R.; Vinyals, O.; Zisserman, A.; and Simonyan, K. 2022.
Flamingo: a visual language model for few-shot learning.
In *Proceedings of the 36th International Conference on Neural Information Processing Systems*, NIPS ’22. Red Hook, NY, USA: Curran Associates Inc.
ISBN 9781713871088.
- Antol et al. (2015)
Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015.
VQA: Visual Question Answering.
In *2015 IEEE International Conference on Computer Vision (ICCV)*, 2425–2433.
- Bai et al. (2023)
Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023.
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.
*arXiv preprint arXiv:2308.12966*.
- Bolya et al. (2023)
Bolya, D.; Fu, C.-Y.; Dai, X.; Zhang, P.; Feichtenhofer, C.; and Hoffman, J. 2023.
Token Merging: Your ViT but Faster.
In *International Conference on Learning Representations*.
- Bommasani et al. (2021)
Bommasani, R.; Hudson, D. A.; Adeli, E.; Altman, R. B.; Arora, S.; von Arx, S.; Bernstein, M. S.; Bohg, J.; Bosselut, A.; Brunskill, E.; Brynjolfsson, E.; Buch, S.; Card, D.; Castellon, R.; Chatterji, N. S.; Chen, A. S.; Creel, K.; Davis, J. Q.; Demszky, D.; Donahue, C.; Doumbouya, M.; Durmus, E.; Ermon, S.; Etchemendy, J.; Ethayarajh, K.; Fei-Fei, L.; Finn, C.; Gale, T.; Gillespie, L. E.; Goel, K.; Goodman, N. D.; Grossman, S.; Guha, N.; Hashimoto, T.; Henderson, P.; Hewitt, J.; Ho, D. E.; Hong, J.; Hsu, K.; Huang, J.; Icard, T.; Jain, S.; Jurafsky, D.; Kalluri, P.; Karamcheti, S.; Keeling, G.; Khani, F.; Khattab, O.; Koh, P. W.; Krass, M. S.; Krishna, R.; Kuditipudi, R.; and et al. 2021.
On the Opportunities and Risks of Foundation Models.
*CoRR*, abs/2108.07258.
- Brown et al. (2020)
Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.
Language models are few-shot learners.
In *Proceedings of the 34th International Conference on Neural Information Processing Systems*, NIPS ’20. Red Hook, NY, USA: Curran Associates Inc.
ISBN 9781713829546.
- Chen et al. (2025a)
Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2025a.
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models.
In Leonardis, A.; Ricci, E.; Roth, S.; Russakovsky, O.; Sattler, T.; and Varol, G., eds., *Computer Vision – ECCV 2024*, 19–35. Cham: Springer Nature Switzerland.
ISBN 978-3-031-73004-7.
- Chen et al. (2025b)
Chen, Y.; Xu, J.; Zhang, X.-Y.; Liu, W.-Z.; Liu, Y.-Y.; and Liu, C.-L. 2025b.
Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 39(2): 2293–2301.
- Chen et al. (2024a)
Chen, Z.; Wang, W.; Tian, H.; Ye, S.; Gao, Z.; Cui, E.; Tong, W.; Hu, K.; Luo, J.; Ma, Z.; et al. 2024a.
How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.
*Science China Information Sciences*, 67(12): 220101.
- Chen et al. (2024b)
Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024b.
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 24185–24198.
- Clark et al. (2019)
Clark, K.; Khandelwal, U.; Levy, O.; and Manning, C. D. 2019.
What Does BERT Look At? An Analysis of BERT’s Attention.
*CoRR*, abs/1906.04341.
- Dosovitskiy et al. (2021)
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
In *International Conference on Learning Representations*.
- Du et al. (2022)
Du, Y.; Liu, Z.; Li, J.; and Zhao, W. X. 2022.
A Survey of Vision-Language Pre-Trained Models.
In Raedt, L. D., ed., *Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22*, 5436–5443. International Joint Conferences on Artificial Intelligence Organization.
Survey Track.
- Faghri et al. (2018)
Faghri, F.; Fleet, D. J.; Kiros, J. R.; and Fidler, S. 2018.
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives.
In *Proceedings of BMVC*.
- Fu et al. (2023)
Fu, C.; Chen, P.; Shen, Y.; Qin, Y.; Zhang, M.; Lin, X.; Yang, J.; Zheng, X.; Li, K.; Sun, X.; et al. 2023.
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models.
*arXiv preprint arXiv:2306.13394*.
- Hudson and Manning (2019)
Hudson, D. A.; and Manning, C. D. 2019.
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*.
- Karpathy and Fei-Fei (2017)
Karpathy, A.; and Fei-Fei, L. 2017.
Deep Visual-Semantic Alignments for Generating Image Descriptions.
*IEEE Trans. Pattern Anal. Mach. Intell.*, 39(4): 664–676.
- Li et al. (2023)
Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023.
BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.
In *Proceedings of the 40th International Conference on Machine Learning*, ICML’23. JMLR.org.
- Li et al. (2025)
Li, Y.; Jiang, H.; Zhang, C.; Wu, Q.; Luo, X.; Ahn, S.; Abdi, A. H.; Li, D.; Gao, J.; Yang, Y.; and Qiu, L. 2025.
MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention.
arXiv:2504.16083.
- Li, Wang, and Jia (2024)
Li, Y.; Wang, C.; and Jia, J. 2024.
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models.
In *Computer Vision – ECCV 2023*.
- Lin et al. (2014)
Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.
Microsoft COCO: Common Objects in Context.
In Fleet, D.; Pajdla, T.; Schiele, B.; and Tuytelaars, T., eds., *Computer Vision – ECCV 2014*, 740–755. Cham: Springer International Publishing.
ISBN 978-3-319-10602-1.
- Lin et al. (2025)
Lin, Z.; Lin, M.; Lin, L.; and Ji, R. 2025.
Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 39(5): 5334–5342.
- Liu et al. (2023a)
Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2023a.
Improved Baselines with Visual Instruction Tuning.
- Liu et al. (2024a)
Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024a.
LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.
- Liu et al. (2023b)
Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023b.
Visual Instruction Tuning.
- Liu et al. (2024b)
Liu, Y.; Duan, H.; Zhang, Y.; Li, B.; Zhang, S.; Zhao, W.; Yuan, Y.; Wang, J.; He, C.; Liu, Z.; et al. 2024b.
Mmbench: Is your multi-modal model an all-around player?
In *Computer Vision – ECCV 2023*, 216–233. Springer.
- Liu et al. (2022)
Liu, Z.; Mao, H.; Wu, C.-Y.; Feichtenhofer, C.; Darrell, T.; and Xie, S. 2022.
A ConvNet for the 2020s.
In *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 11966–11976.
- Lu et al. (2022)
Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.
In *The 36th Conference on Neural Information Processing Systems (NeurIPS)*.
- Luan et al. (2025)
Luan, B.; Zhou, W.; Feng, H.; Wang, Z.; Li, X.; and Li, H. 2025.
Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models.
arXiv:2503.08019.
- Marr (2010)
Marr, D. 2010.
*Vision: A Computational Investigation into the Human Representation and Processing of Visual Information*.
The MIT Press.
ISBN 9780262514620.
- OpenAI (2023)
OpenAI. 2023.
GPT-4V System Card.
Online.
Available at https://cdn.openai.com/papers/GPTV_System_Card.pdf.
- Radford et al. (2021)
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021.
Learning Transferable Visual Models From Natural Language Supervision.
In Meila, M.; and Zhang, T., eds., *Proceedings of the 38th International Conference on Machine Learning*, volume 139 of *Proceedings of Machine Learning Research*, 8748–8763. PMLR.
- Shang et al. (2025)
Shang, Y.; Cai, M.; Xu, B.; Lee, Y. J.; and Yan, Y. 2025.
LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models.
In *ICCV*.
- Singh et al. (2019)
Singh, A.; Natarjan, V.; Shah, M.; Jiang, Y.; Chen, X.; Parikh, D.; and Rohrbach, M. 2019.
Towards VQA Models That Can Read.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 8317–8326.
- Tenney, Das, and Pavlick (2019)
Tenney, I.; Das, D.; and Pavlick, E. 2019.
BERT Rediscovers the Classical NLP Pipeline.
In Korhonen, A.; Traum, D.; and Màrquez, L., eds., *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, 4593–4601. Florence, Italy: Association for Computational Linguistics.
- Tong et al. (2025)
Tong, J.; Jin, W.; Qin, P.; Li, A.; Zou, Y.; Li, Y.; Li, Y.; and Li, R. 2025.
FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models.
arXiv:2505.19536.
- Touvron et al. (2023)
Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023.
LLaMA: Open and Efficient Foundation Language Models.
arXiv:2302.13971.
- Truong, Oudre, and Vayatis (2020)
Truong, C.; Oudre, L.; and Vayatis, N. 2020.
Selective review of offline change point detection methods.
*Signal Processing*, 167: 107299.
- Vaswani et al. (2017)
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.
Attention is all you need.
In *Proceedings of the 31st International Conference on Neural Information Processing Systems*, NIPS’17, 6000–6010. Red Hook, NY, USA: Curran Associates Inc.
ISBN 9781510860964.
- Vinyals et al. (2015)
Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015.
Show and tell: A neural image caption generator.
In *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 3156–3164.
- Wang et al. (2024)
Wang, W.; Lv, Q.; Yu, W.; Hong, W.; Qi, J.; Wang, Y.; Ji, J.; Yang, Z.; Zhao, L.; Song, X.; Xu, J.; Xu, B.; Li, J.; Dong, Y.; Ding, M.; and Tang, J. 2024.
CogVLM: Visual Expert for Pretrained Language Models.
arXiv:2311.03079.
- Wu, Chen, and Wang (2025)
Wu, Z.; Chen, J.; and Wang, Y. 2025.
Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 39(8): 8550–8558.
- Xing et al. (2024)
Xing, L.; Huang, Q.; Dong, X.; Lu, J.; Zhang, P.; Zang, Y.; Cao, Y.; He, C.; Wang, J.; Wu, F.; and Lin, D. 2024.
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction.
*CoRR*, abs/2410.17247.
- Ye et al. (2025)
Ye, W.; Wu, Q.; Lin, W.; and Zhou, Y. 2025.
Fit and prune: Fast and training-free visual token pruning for multi-modal large language models.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 22128–22136.
- Zhang et al. (2024)
Zhang, Q.; Cheng, A.; Lu, M.; Zhuo, Z.; Wang, M.; Cao, J.; Guo, S.; She, Q.; and Zhang, S. 2024.
[CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster.
*arXiv preprint arXiv:2412.01818*.
- Zhang et al. (2025)
Zhang, Y.; Fan, C.-K.; Ma, J.; Zheng, W.; Huang, T.; Cheng, K.; Gudovskiy, D.; Okuno, T.; Nakata, Y.; Keutzer, K.; et al. 2025.
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference.
In *International Conference on Machine Learning*.
- Zhu et al. (2023)
Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023.
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.
arXiv:2304.10592.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: The questions are mainly covered by the weakness above.
soundness: 3
strengths: - The method is both training-free and, critically, reuses attention maps that are already computed during the forward pass. This means it introduces almost no additional overhead (Sec 3.2.2), making it an extremely practical solution for real-world inference acceleration.

- The paper shows clear and consistent performance gains over its closest SOTA-level competitor (SparseVLM) across multiple benchmarks and token budgets (Table 1, Fig 3). The latency test (Table 3) confirms that these theoretical gains translate to real-world speedups.

- The primary contribution—using dynamic text-to-text attention as a proxy for text-token importance—is a novel and elegant solution to the limitations of static guidance. It's theoretically well-motivated by the (correct) observation that information importance evolves by layer.
summary: The authors propose "AdaptInfer," a plug-and-play, training-free framework designed to address these two specific problems.

First, the core technical novelty is a dynamic text-guided pruning mechanism. Instead of using a static set of "key" text tokens, AdaptInfer re-computes text-token importance at each designated pruning layer. It cleverly reuses the existing text-to-text (t2t) attention maps (which are already computed) to create a "soft prior" distribution over the text tokens. This prior is then used to reweight the text-to-vision (t2v) attention, yielding a more informed and context-aware ranking of vision tokens for pruning. This is an elegant solution, as it introduces minimal computational overhead.

Second, to replace heuristic pruning schedules, the authors conduct an offline analysis to identify "attention inflection points." By applying change-point detection to the cumulative t2v attention trajectories, they discover consistent, data-driven locations in the architecture (e.g., layers 1, 10, and 20 for LLaVA-1.5-7B) where the model's utilization of visual information significantly shifts. These inflection points are then used as principled locations for applying the pruning.
weaknesses: - The specific schedule (layers 1, 10, 20) is architecture-dependent (LLaMA-7B). While the method for finding the schedule is general, it requires a new, non-trivial offline analysis (running 1000+ samples through the model and performing change-point detection) for every new backbone

- The main paper emphasizes the locations of pruning, but the amount to prune at each location is also a critical hyperparameter. Appendix G.3 mentions that these "pruning ratios" are also selected, following a method from prior work. The sensitivity of the model to these ratios, and how they interact with the pruning locations, is not fully explored, adding a layer of tuning that is under-discussed.

- In Sec 3.1.3, the authors posit that an attention shift can mean a token is either becoming critical or redundant. The logic in Sec 3.2.3 for building the schedule around these points ("pruning should avoid the regions with intense attention shifts... pruning just before or after the dense regions") feels slightly post-hoc and could be articulated more clearly. For instance, why is pruning at layer 1 (an inflection point) and at layer 10 (the start of a dense region) the optimal strategy?

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - The proposed dynamic cross-attention guided visual token pruning is enhanced by reusing text-token attantion to put a higher focus on important text tokens.
- The proposed AdaptInfer leverage the observation of attention distribution shift to guide choices of hyperparameters like the insertion location of the pruning layer.
- The proposed method provides notable savings in inference latency.
summary: This paper proposes an adaptive token pruning method for VLMs, of which the core design is a plug-and-play framework to reuse the text-to-text attention maps for a soft prior of text-token importance. In the experiments, The proposed method outperforms FastV, SparseVLM, and other baselines on the average accuracy of 5 VQA benchmarks, showing the effectiveness of the AdaptInfer module.
weaknesses: - The proposed method is only verified on 5 different VQA datasets, whereas the baseline methods are usually evaluated on much more diverse benchmarks. For example the PyramidDrop is evaluated on 16 different benchmarks.
- Some benchmarks are also evaluated on video VQA benchmarks, including PyramidDrop and SparseVLM. A more well-rounded comparison will be more convincing especially when the improvement over SparseVLM is not consistent.
- The newer SOTA benchmark VisionZip [1] is not included in the comparison.
[1] Yang, Senqiao, Yukang Chen, Zhuotao Tian, et al. n.d. VisionZip: Longer Is Better but Not Necessary in Vision Language Models.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: 1. **Reasoning VLMs.** Reasoning VLMs also face the problem of token redundancy. The author should discuss whether this method can be applied to Efficient Reasoning VLMs[1].

2. **Resize.** In [1], the researchers discussed that in most general scenarios, even simple resizing can achieve strong performance. How do the authors view this issue? I look forward to the author discussing this point.

[1] VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
soundness: 2
strengths: 1. **Reasonable Design.** Using important text tokens to mine important vision tokens is reasonable.
summary: This paper presents AdapterInfer, a plug-and-play training-free adaptive token pruning approach for VLMs. First, they use layer-wise text-to-text attention map to enhance the text-based vision-token pruning, use important text tokens to mine important vision tokens. Second, they identify consistent inflexion locations in inference via analysis of cross-modal attention in LLaVA, and improve the pruning schedule.
weaknesses: 1. **Weak Baseline.** This paper uses LLaVA-1.5 as a baseline, which is an open-source VLM released 2 years ago. It's too weak and completely outdated. For a training-free method, the evaluation should be conducted on more recent, competitive and widely used VLMs, such as Qwen2.5-VL.

2. **Lack Generalizability.** The consistent attention inflection point found in this paper is based on LLaVA-1.5. The author should check other base VLMs like Qwen2-VL and Qwen2.5-VL for a similar situation.

3. **Lack Comparison.** This paper should compare with newer baselines, like VisionZip[1], a widely used efficient VLM approach accepted in CVPR25.

4. **Cannot work with flash-attention.** AdapterInfer based on t2t and t2v attention score, which cannot be obtained when inference with flash-attention.

[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Based on the experiments in the author's methodology, I am curious whether this claim can be generalized to different tasks? For example, do classification tasks and QA tasks perform consistently across different layers? In addition, intuitively, classification tasks should be able to tolerate more aggressive pruning, so is it acceptable to prune layers where attention shift is obvious?
2. The method proposed by the author can prevent important information from being drop. Does this method ensure that the knowledge of the model remains consistent? As far as I know, token pruning methods can ensure that overall performance does not drop significantly. However, they may introduce knowledge boundary drift, meaning that answers that were previously correct may now be incorrect, and vice versa [1]. This possibility could have negative consequences in scenarios where critical questions must be answered correctly. If possible, could the authors conduct a simple analysis to see whether the proposed method leads to changes in individual case performance?

[1] Sun, Yizheng, et al. "Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study." arXiv preprint arXiv:2503.06794 (2025).
soundness: 3
strengths: The author uses experiments to heuristically verify which layers are more suitable for pruning and which are not. For example, the attention shift is more obvious in the 1st, 10th, and 20th layers.
summary: This paper proposes a training-free token pruning method. It directly leverages the attention computation mechanism of the base model for pruning. Furthermore, the paper provides insightful guidance on which layers are suitable for pruning and which are best left alone. The experiments in the paper are relatively comprehensive and provide strong support for the authors' claims.
weaknesses: 1. The claims in 3.1.3 are based on experiments, but the authors tested them on only one model. They provide a discussion of generalizability, but there are limitations on whether these claims can be generalized to different models.
2. Tables 1 and 2 are too far removed from the analysis of the results to be easily readable.


Decision: Reject