SparseVLM: Visual Token Sparsification for Efficient Vision Language Models Inference

## Abstract

Abstract In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens and require additional training data. Differently, we propose an efficient training-free token optimization mechanism dubbed SparseVLM without extra parameters or fine-tuning costs. Concretely, given that visual tokens complement text tokens in VLMs for linguistic reasoning, we select visual-relevant text tokens to rate the significance of vision tokens within the self-attention matrix extracted from the VLMs. Then we progressively prune irrelevant tokens. To maximize sparsity while retaining essential information, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that our SparseVLM improves the efficiency of various VLMs across a range of image and video understanding tasks. In particular, LLaVA equipped with SparseVLM reduces 61% ∼ similar-to \sim ∼ 67% FLOPs with a compression ratio of 78% while maintaining 93% of the accuracy.
Our code is available at https://github.com/Gumpest/SparseVLMs .

## 1 Introduction

Benefiting from tremendous advancements in large language models (LLMs) (Radford et al., ; Brown et al., ; Achiam et al., ; Touvron et al., ; Peng et al., ; Bi et al., ), the realm of vision-language models (VLMs) has undergone a revolutionary progress. To combine visual signals with textual semantics, the mainstream practice in VLMs (Team et al., ; Bai et al., ; Chen et al., ; Li et al., ; ) employs sequential visual representation, where images are extracted into vision tokens and sent into an LLM decoder. With modal alignment and instruction fine-tuning (Du et al., ; Liu et al., ; Zhu et al., ), recent VLMs successfully adapt LLMs to the vision domain and inherit their perception and reasoning abilities.

Despite the promising performance, further incorporation of visual tokens inevitably introduces a huge memory and computation overhead when compared to LLMs, particularly for high-resolution images (Li et al., ) and multi-frame videos (Lin et al., ). For instance, a $672\times 672$ image in LLaVA (Liu et al., ) yields $2304$ vision tokens that span over half of the context length. However, the information in images is typically more sparse than in natural languages (Marr, ), resulting in inefficiency in directly processing both modalities.
To address this, existing methods extract more compact image representations by modifying the image encoder or projector (Alayrac et al., ; Li et al., ; Dai et al., ; Cha et al., ). While some recent works further sparsify vision tokens during the decoding (Ye et al., ; Chen et al., ; Cai et al., ), they still ignore the guidance from the language tokens, which contradicts the multimodality paradigm. We argue that visual tokens should be sparsified adaptively based on the question prompt, as the model might focus on different parts (e.g., foreground or background) when dealing with various questions, as shown in Figure 1. Furthermore, current approaches generally train a network to prune redundant visual tokens and require additional training data (Li et al., ; Ye et al., ).

In this paper, we introduce a text-guided training-free framework dubbed SparseVLM for efficient vision language model inference.
We reuse the self-attention matrix of visual-text tokens directly from the decoder layers without extra training parameters for sparsification.
We ascertain that not all prompt tokens should be considered as some could be less relevant, which leads to inaccurate correlation results and downgrades the performance of sparse inference.
Specifically, our SparseVLM first identifies text tokens strongly correlated with visual signals via cross-attention.
Then, we measure the contribution of visual tokens to the selected visual-relevant text tokens (raters) and adaptively prune the insignificant vision tokens.
Instead of directly discarding the pruned tokens, we further recycle and cluster them to reconstruct more compact tokens to minimize information loss.
Due to the information density varying for different image inputs, we employ the rank of the attention matrix to indicate the redundancy level and set an adaptive sparsification ratio accordingly.

The proposed method is simple yet practical. It can act as a plug-and-play module to improve the efficiency of VLMs without additional fine-tuning.
Extensive experiments demonstrate that our SparseVLM effectively reduces the computation of various VLMs without sacrificing their performance in a wide range of image and video understanding tasks.
For example, LLaVA (Liu et al., ) equipped with SparseVLM achieves a $4.5\times$ compression rate while maintaining $93\%$ of its original performance. Alternatively, the latency (CUDA time) can decrease by $53.9\%$ with only a $13\%$ drop in accuracy.
To investigate the effectiveness of our method in the video tasks, we further apply SparseVLM to VideoLLaVA (Lin et al., ) to additionally compress temporal frames.
Without complex design changes, SparseVLM can sparsify video frames into an adaptive number of vision tokens and outperforms existing vision compression methods in video question-answering benchmarks. For instance, our method average exceeds FastV (Chen et al., ) by $34.4\%$.

In summary, the contributions of this paper are threefold:

- 1.
We introduce a novel sparsification framework dubbed SparseVLM for vision-language models. To the best of our knowledge, it is the first attempt to explore the potential of text-aware guidance for efficient inference of VLMs, where additional training is unnecessary.
- 2.
In the framework, we first assign visual-relevant text tokens as raters, to judge the importance of vision tokens. Additionally, the rank of the attention logits is employed to reflect the redundancy and adaptively prune VLMs. Finally, we recycle partial tokens from the pruned pools and reconstruct them to accommodate more information within fewer slots.
- 3.
We apply our SparseVLM framework to both image and video VLMs and conduct extensive experiments across various benchmarks. Our approach consistently outperforms the existing state-of-the-art method FastV by $7.7\%\sim 14.8\%$ on LLaVA, $10.2\%\sim 21.6\%$ on MiniGemini, and $34.4\%$ on VideoLLaVA.

Figure: Figure 1: Visualization of different visual token sparsification methods. Unlike previous methods with text-agnostic visual sparsification (c) e.g., recent FastV (Chen et al., ), our SparseVLM (b) is guided by question prompts to select relevant visual patches. Best viewed in color.
Refer to caption: x1.png

## 2 Related Work

Vision-Language Models. With the impressive success of large language models (LLMs) (Achiam et al., ; Touvron et al., ; Bai et al., ), recent works on generative vision-language models (VLMs) (Liu et al., ; Chen et al., ; Li et al., ) improve multimodal comprehension and generation by receiving a long visual token sequence. Moreover, processing higher-resolution images inevitably entails an exponential growth in the length of the visual sequence. For example, LLaVA encodes $336\times 336$ images into $576$ tokens (Liu et al., ) and processes images with a greater resolution of $672\times 672$ into $2304$ tokens (Liu et al., ). Along the same lines, mini-Gemini-HD (Li et al., ) converts images into $2880$ vision tokens based on the standard of high resolution $1536\times 1536$ and low resolution $672\times 672$. Comprehending videos or multiple images inherently necessitates increased token slots for visual signals. The VideoLLaVA (Lin et al., ) and VideoPoet (Kondratyuk et al., ) models allocate thousands of tokens to afford the representation of multiple frames. However, the large number of vision tokens leads to a huge bottleneck for computational infrastructure. Further research and development in sparsification technologies are urged to overcome these hurdles and fully unleash the potential of VLMs.

Visual Compression for VLMs. Compression of vision tokens is necessary because, on the one hand, their quantity is usually tens to hundreds of times that of language tokens. On the other hand, visual signals are inherently redundant compared to dense human-designed texts (Marr, ). Past efforts to address the above problem can be categorized into two directions. The first one centers on the vision tower or projection of visual modality and cuts vision tokens with external modules. For instance, LLaMA-VID (Li et al., ) exploits the use of Q-Former with context token while DeCo (Yao et al., ) employs an adaptive pooling to downsample the visual tokens at the patch level. The other type methods (Ye et al., ; Chen et al., ; Cai et al., ) go deeper into the text modality and sparsify vision tokens during LLM decoding, but they still neglect the guidance from the text tokens. In our paper, SparseVLM takes note of this and improves performance upon it.

Token Merging for VLMs.
Token merging has recently received much attention in the field of VLMs where its algorithms mainly fall into two directions. One focuses on the matching algorithm using the Bipartite Soft Matching (BSM) (Bolya et al., ). For example, ToMe (Bolya et al., ) prunes according to the relevance between visual tokens and text and merges both modalities through the BSM algorithm. Other methods rely on clustering methods for token merging. For instance, LLaVolta (Chen et al., )
proposes the visual context compressor (an average pooling) to merge the output tokens from the vision tower and progressively enhance the VLMs’ efficiency by training. Inspired by TCFormer (Zeng et al., ), we apply $k$-nearest neighbor density peak aggregation algorithm (Rodriguez, ) to the field of vision-language models. Unlike TCFormer, our method focuses on merging the dropped tokens and makes full use of their detailed information.

## 3 Proposed Approach: SparseVLM

In this section, we present our SparseVLM framework for efficient VLM inference.
We first review the attention mechanism in VLMs and then introduce the detailed strategies for our visual sparsification, including visual significance estimation, relevant text token selection, and sparsification level adaptation.
We further propose the token recycling method to reduce information loss and provide a theoretical analysis of computation savings.
The overall architecture is shown in Figure 2.

Figure: Figure 2: The architecture of SparseVLM. In stage (a), text raters are pre-selected before entering the sparsification LLM. In stage (b), adaptive sparsification is performed on LLM layers, involving computing redundancy and the recycling of reconstructed tokens. Best viewed in color.
Refer to caption: x2.png

### 3.1 Preliminary: Attention in VLM Decoders

The VLM decoders typically use the causal self-attention in the original transformer model (Vaswani et al., ) for token interactions.
Without loss of generality, we discuss the condition of single-head attention.
Formally, the self-attention matrix with logits ${\bm{A}}\in\mathbb{R}^{L\times L}$, where $L$ denotes the length of total tokens (e.g., system, image, and question prompt tokens), is computed by

$$ ${\bm{A}}=\text{Attention}({\bm{Q}},{\bm{K}})=\text{Softmax}\left(\frac{{\bm{Q} }{\bm{K}}^{T}}{\sqrt{D}}\right),$ (1) $$

where the scalar $D$ represents the matrix dimension, and the ${\bm{Q}}\in\mathbb{R}^{L\times D}$ and ${\bm{K}}\in\mathbb{R}^{L\times D}$ are the query and key matrices, respectively. The keys and queries in a self-attention layer are computed in parallel by using multi-layer perceptrons (MLPs) to transform the input hidden states ${\bm{H}}$ into a common space, where aligned interactions between different modalities occur.

### 3.2 Sparsification Guidance from Text to Visual Modality

Estimation of Visual Token Significance. For a multimodal vision-language model, we should consider its impact on other modalities when deleting a single modal information. In our case, we need to understand how relevant a visual token is to the textual tokens in order to determine whether it should be removed. Therefore, we naturally come up with reusing the self-attention logits in VLMs transformer layers as a reference, since they already contain language-to-vision query results.

In particular, we take the interaction between the query-dimensional part of logits of the textual modality and the key-dimensional part of the visual modality as the basis for sparsification priority matrix ${\bm{P}}\in\mathbb{R}^{L_{t}\times L_{v}}$, where $L_{t}$ and $L_{v}$ are the lengths of text and vision tokens, that is defined by

$$ ${\bm{P}}={\bm{A}}[i_{t},i_{v}],$ (2) $$

with

$$ $i_{t}\in\{x|{\bm{A}}[x,:]={\mathbb{L}}\},\ \ \ i_{v}\in\{y|{\bm{A}}[:,y]={ \mathbb{I}}\},$ (3) $$

where ${\mathbb{L}}$ and ${\mathbb{I}}$ denote the language instruction and image tokens set, respectively.

Next, we average scores of all instruction tokens to obtain the estimate $\bar{{\bm{P}}}_{j}$ for $j$th vision token as

$$ $\bar{{\bm{P}}}_{j}=\frac{1}{L_{t}}\sum\nolimits_{i=1}^{L_{t}}{\bm{P}}[i,j],\ \ j\in\{1,2,\ldots,L_{v}\},$ (4) $$

where we use $\bar{{\bm{P}}}_{j}$ as the significance indicator for sparsification and a larger value in $\bar{{\bm{P}}}_{j}$ means higher significance for the corresponding token.
Calculation of equation 4 costs $L_{t}\times L_{v}$ FLOPs, while the correlation matrix ${\bm{A}}$ and the indexing process is free, which benefits the inference efficiency.

Relevant Text Token Selection. It is not appropriate to use all text tokens as a reference for visual sparsification. Figure 3 shows four representative cases where we compute the correlation between the prompt and the image. Case $3$ highlights Tylenol, Advil, ibuprofen, while top, sticker, fridge in case $4$ are significant, where a large proportion of question tokens in light red include little visual relevance. Therefore, it is unreasonable to make insignificant text tokens to rate vision tokens, and we need to select relevant text tokens (i.e., “raters”) for guidance.

Specifically, for an input image ${\bm{x}}_{v}$, the vision embedding tokens ${\bm{H}}_{v}$ can be computed as

$$ ${\bm{H}}_{v}={\bm{W}}{\bm{Z}}_{v},$ (5) $$

where ${\bm{Z}}_{v}$ is the visual feature provided by visual encoder ${\bm{Z}}_{v}=g({\bm{x}}_{v})$, and ${\bm{W}}$ is the projection matrix to convert ${\bm{Z}}_{v}$ into vision embedding tokens ${\bm{H}}_{v}$.
For the language instruction ${\bm{x}}_{q}$, it is transformed into text embedding tokens ${\bm{H}}_{q}$ through the tokenizer. The above tokens both have the same dimensionality as the word embedding space. Then, we start to recognize which characters in the prompt are visually relevant and assign them the role of raters, which can be formulated as

$$ ${\bm{S}}=\{\ i\ |\ {\bm{R}}[i]\geq m,\ \ i\in\{1,2,\ldots,L_{t}\}\}.$ (6) $$

$$ ${\bm{R}}=\frac{1}{L_{v}}\sum_{j=1}^{L_{v}}\text{Softmax}({\bm{H}}_{v}{{\bm{H}} _{q}}^{T}),$ (7) $$

where $m=\text{Mean}({\bm{R}})$ and only candidates who exceed the $m$ will become raters. The strategy ${\bm{S}}$ defines the index of selected raters from the candidate list containing $L_{t}$ tokens with priority. The equation 7 costs $L_{t}\times L_{v}\times 2D$ FLOPs, only computed once before entering the decoder.

Figure: Figure 3: Sample prompts from four representative multimodal benchmarks. The darker the word, the greater its relationship to the image and the more valuable it is for reference. We see that some words are irrelevant to the vision domain (e.g., prepositions and pronouns) and should not be considered for visual sparsification. Best viewed in color.
Refer to caption: x3.png

Sparsification Level Adaptation.
Having obtained the token significance, we further propose a rank-based strategy to adaptively determine the level of vision sparsification at each decoder layer.
Considering that a full-rank matrix implies that all its rows or columns are linearly independent, we use the rank of ${\bm{P}}$ to demonstrate the redundancy of the visual tokens.
We argue that the difference between the dimension and rank of ${\bm{P}}$ reflects its redundancy and utilize a scaling factor $\lambda$ to determine the number of deletions as

$$ $N=\lambda\times(L_{v}-\text{Rank}({\bm{P}})).$ (8) $$

We then remove $N$ visual tokens with the smallest values in ${\bm{P}}$.
Notably, if the result of $N$ in a decoder layer is 0 0, we skip the layer and abandon sparsification.
This stage requires $L_{t}\times L_{v}\times\text{min}(L_{t},L_{v})$ FLOPs for rank computation per layer.

### 3.3 Visual Token Recycling

We progressively sparsify visual tokens in each layer in the decoder, which results in more discarded tokens at later stages.
Despite less significant, the pruned vision tokens with relatively large values in ${\bm{P}}$ still contain certain information.
To efficiently preserve more visual details with fewer tokens, we propose a token recycling strategy to aggregate and reconstruct tokens to be sparsified.

Token Aggregation.
We first recycle the pruned visual tokens $\bar{h}_{v}$ with the top-$\tau$ (%) highest values in ${\bm{P}}$ from the deleted pool.
Then, we group $\bar{h}_{v}$ tokens with $k$-nearest neighbor density peak aggregation algorithm (Rodriguez, ) for adaptive token aggregation.

In particular, we first compute the local density $\rho_{i}$ of the $i$th token of total $\tau\times N$ recycled tokens according to its $k$-nearest neighbors ${\mathcal{K}}(\bar{h}_{v}^{i})$ as

$$ $\rho_{i}=\exp\left(-\frac{1}{k}\sum\nolimits_{\bar{h}_{v}^{j}\in{\mathcal{K}}( \bar{h}_{v}^{i})}^{i,j}\left\|\bar{h}_{v}^{i}-\bar{h}_{v}^{j}\right\|_{2}^{2} \right).$ (9) $$

Then, we compute the minimum distance between the recycled token $\bar{h}_{v}^{i}$ and any other token with higher density (denoted as the distance indicator $\delta_{i}$) that is defined by

$$ $\delta_{i}=\left\{\begin{array}[]{ll}\min&\left\|\bar{h}_{v}^{i}-\bar{h}_{v}^{ j}\right\|_{2},\text{ if }\exists j\text{ s.t. }\rho_{j}>\rho_{i},\\ \max&\left\|\bar{h}_{v}^{i}-\bar{h}_{v}^{j}\right\|_{2},\text{ otherwise }. \end{array}\right.$ (10) $$

We use $\rho_{i}\times\delta_{i}$ to indicate the score of each token, where the tokens with higher scores are likely to be cluster centers.
Other tokens are then assigned to the nearest cluster center via cosine similarity.
The FLOPs cost in this stage is $L_{r}\times(3L_{r}-1)\times 2D+L_{r}$, where $L_{r}=\tau\times N$ is the length of recycled tokens, $C=\theta\times L_{r}$ is the number of cluster centers, and $\tau$ and $\theta$ are hyperparameters.

Token Reconstruction.
Having performed token aggregation, the recycled tokens with similar semantics are classified into the same group.
Then, the tokens ${\mathbb{T}}\in\mathbb{R}^{N_{k}\times D}$ in the $k$th group are reconstructed into a new compressed token ${\bm{T}}_{k}\in\mathbb{R}^{1\times D}$ via the element-wise sum operation as

$$ ${\bm{T}}_{k}=\sum\nolimits_{i=1}^{N_{k}}{\mathbb{T}}[i],\ \ k\in\{1,2,\ldots,C\},$ (11) $$

where $N_{k}$ is the token number of the $k$th group and the operation costs $D\times(L_{r}-C)$ FLOPs.

### 3.4 Theoretical Analysis of Computational Complexity

We consider the computation of multi-head attention and feed-forward network (FFN) modules in the FLOPs estimation.
Assuming $N$ is the number of pruned tokens, $D$ is the hidden state size, which is the same as the intermediate size in FFN, the FLOPs for one transformer layer can be reduced by $6(N-C)D^{2}+2(N-C)^{2}D$.
Besides, our framework introduces additional computational overhead for the sparsification step with the details provided in Appendix A.2.
Thus, we estimate the FLOP savings as the reduction part minus the additional sparsification overhead computed as

$$ $\begin{array}[]{ll}\underbrace{\scriptstyle{\textstyle\sum_{i}}6(N_{i}-C_{i})D ^{2}+2(N_{i}-C_{i})^{2}D}_{\text{reduction part}}-\underbrace{\scriptstyle 2L_ {t}L_{v}D-{\textstyle\sum_{i}}L_{t}^{i}L_{v}^{i}(1+\min(L_{t}^{i},L_{v}^{i}))- (6{L^{i}_{r}}^{2}+2L_{r}^{i})D-L_{r}^{i}}_{\text{overhead part}}\\ \approx-2L_{t}L_{v}D+\sum_{i}D(6DN_{i}(1-x)+N_{i}^{2}(2+2x^{2}-4x-6(\tau)^{2}) )-{L^{i}_{t}}^{2}L_{v}^{i}\\ \approx-2L_{t}L_{v}D+\sum_{i}DN_{i}(6D+2N_{i})-{L^{i}_{t}}^{2}L_{v}^{i},\ \ i \in\{1,2,\ldots,\Omega\},\end{array}$ (12) $$

where $\Omega$ is the number of total layers, and $x=\tau\times\theta$ is a very small decimal that can be ignored.

## 4 Experiments

**Table 1: Performance of SparseLLaVA under different vision token configurations. The vanilla number of vision tokens is $576$. The first line of each method is the raw accuracy of benchmarks, and the second line is the proportion relative to the upper limit. The last column is the average value.**
| Method | GQA | MMB | MME | POPE | SQA | VQA${}^{\text{V2}}$ | VQA${}^{\text{Text}}$ | ConB | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Upper Bound, 576 Tokens  (100%) |  |  |  |  |  |  |  |  |  |
| Vanilla | 61.9 | 64.7 | 1862 | 85.9 | 69.5 | 78.5 | 58.2 | 19.8 | 100% |
| 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |  |  |
| Retain 192 Tokens  $\mathbf{\mathcolor{ForestGreen}{(\downarrow 66.7\%)}}$ |  |  |  |  |  |  |  |  |  |
| ToMe (ICLR23) | 54.3 | 60.5 | 1563 | 72.4 | 65.2 | 68.0 | 52.1 | 17.4 | 88.4% |
| 87.7% | 93.5% | 83.9% | 84.3% | 93.8% | 86.6% | 89.5% | 87.9% |  |  |
| FastV (ECCV24) | 52.7 | 61.2 | 1612 | 64.8 | 67.3 | 67.1 | 52.5 | 18.0 | 88.1% |
| 85.1% | 94.6% | 86.6% | 75.4% | 96.8% | 85.5% | 90.2% | 90.9% |  |  |
| SparseVLM | 57.6 | 62.5 | 1721 | 83.6 | 69.1 | 75.6 | 56.1 | 18.8 | 95.8% |
| 93.1% | 96.6% | 92.4% | 97.3% | 99.4% | 96.3% | 96.4% | 94.9% | $\mathbf{\mathcolor{Forestred}{\uparrow(7.4\%)}}$ |  |
| Retain 128 Tokens  $\mathbf{\mathcolor{ForestGreen}{(\downarrow 77.8\%)}}$ |  |  |  |  |  |  |  |  |  |
| ToMe (ICLR23) | 52.4 | 53.3 | 1343 | 62.8 | 59.6 | 63.0 | 49.1 | 16.0 | 80.4% |
| 84.7% | 82.4% | 72.1% | 73.1% | 85.8% | 80.2% | 84.4% | 80.8% |  |  |
| FastV (ECCV24) | 49.6 | 56.1 | 1490 | 59.6 | 60.2 | 61.8 | 50.6 | 17.1 | 81.9% |
| 80.1% | 86.7% | 80.0% | 69.4% | 86.6% | 78.7% | 86.9% | 86.4% |  |  |
| SparseVLM | 56.0 | 60.0 | 1696 | 80.5 | 67.1 | 73.8 | 54.9 | 18.5 | 93.3% |
| 90.5% | 92.7% | 91.1% | 93.7% | 96.5% | 94.0% | 94.3% | 93.4% | $\mathbf{\mathcolor{Forestred}{\uparrow(11.4\%)}}$ |  |
| Retain 64 Tokens  $\mathbf{\mathcolor{ForestGreen}{(\downarrow 88.9\%)}}$ |  |  |  |  |  |  |  |  |  |
| ToMe (ICLR23) | 48.6 | 43.7 | 1138 | 52.5 | 50.0 | 57.1 | 45.3 | 14.0 | 70.2% |
| 78.5% | 67.5% | 61.1% | 61.1% | 71.9% | 72.7% | 77.8% | 70.7% |  |  |
| FastV (ECCV24) | 46.1 | 48.0 | 1256 | 48.0 | 51.1 | 55.0 | 47.8 | 15.6 | 72.1% |
| 74.5% | 74.2% | 67.5% | 55.9% | 73.5% | 70.1% | 82.1% | 78.8% |  |  |
| SparseVLM | 52.7 | 56.2 | 1505 | 75.1 | 62.2 | 68.2 | 51.8 | 17.7 | 86.9% |
| 85.1% | 86.9% | 80.8% | 87.4% | 89.4% | 86.9% | 89.0% | 89.4% | $\mathbf{\mathcolor{Forestred}{\uparrow(14.8\%)}}$ |  |

In this section, we validate our method within various VLM architectures on comprehensive multimodal benchmarks to assess its effectiveness including image and video understanding tasks.

### 4.1 Image Understanding Tasks

Datasets. For image-based multimodal evaluation, we conduct experiments on eight widely adopted benchmarks including GQA (Hudson & Manning, ), MMBench (MMB) (Liu et al., ), MME (Fu et al., ), POPE (Li et al., ), SQA (Lu et al., ), VQA${}^{\text{V2}}$ (VQA V2) (Goyal et al., ), and VQA${}^{\text{Text}}$ (TextVQA) (Singh et al., ). Furthermore, we check the consistency of SparseVLM on ConBench (Zhang et al., ). More details are included in the Appendix A.4.

Implementation Details. We verify the proposed SparseVLM on two popular VLM frameworks: LLaVA (Liu et al., ) and Mini-Gemini (Li et al., ). LLaVA-1.5 employs CLIP-pretrained ViT-L as the visual tower, while Mini-Gemini (MGM) further introduces a LAION-pretrained ConvNeXt-L (Liu et al., ) for high-resolution refinement. For LLaVA-1.5-7/13B and Mini-Gemini, we follow the same inference setting as the original paper as it is publicly available(^1^11[github.com/haotian-liu/LLaVA](github.com/haotian-liu/LLaVA) and [github.com/dvlab-research/MGM](github.com/dvlab-research/MGM).).

Main Results. In Table 1, we present the performance of SparseLLaVA (LLaVA equipped with SparseVLM) on image understanding benchmarks. To intuitively assess the performance, we provide the results by percentage format for comparative analysis, and the accuracy of the vanilla model with the 100% upper limit. We set $3$ vision token count configurations ($192$, $128$, and $64$) to check the advantages of SparseVLM. When pruning from $576$ to $192$ tokens, the SparseLLaVA only decreases the average accuracy by $4.2\%$ without additional training and exceeds ToMe (Bolya et al., ) $7.4\%$. Furthermore, when only $64$ tokens are left, our method outperforms FastV (Chen et al., ) by a significant margin of 14.8%, while ToMe performs worst due to its direct merging.

Figure 4 shows the performance of SparseMGM, and we visualize the results on POPE, TextVQA, and GQA. We find that our framework has an obvious advantage over FastV and ToMe. With the reduction of tokens, the gap between FastV and SparseVLM is increasing sharply. The reason is that, compared to FastV and ToMe, the text-aware strategy enables us to accurately locate visual tokens with more details, while the recycling of pruned tokens further reduces information loss.

Figure: Figure 4: Performance of MGM armed with SparseVLM on three multimodal benchmarks. The horizontal axis represents the remaining number of vision tokens, while the vertical axis means the accuracy after percentage normalization. FastV is included for comparison.
Refer to caption: x4.png

### 4.2 Video Understanding Tasks

**Table 2: The results of Video-LLaVA with SparseVLM on video question answering task. The original number of video tokens is $2048$, while our experiment collectively prunes it down to $135$ tokens. FastV is included for comparison. The GPT-3.5 turbo is adopted for assistive evaluation.**
| Method | TGIF | MSVD | MSRVTT | ActivityNet | Avg |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc | Score | Acc | Score | Acc | Score | Acc | Score | Acc | Score |  |
| Video-LLaVA | 47.1 | 3.35 | 69.8 | 3.92 | 56.7 | 3.48 | 43.1 | 3.35 | 100.0% | +0.00 |
| FastV (ECCV24) | 23.1 | 2.47 | 38.0 | 2.71 | 19.3 | 2.02 | 30.6 | 2.82 | 52.1% | -1.02 |
| 49.0% | -0.88 | 54.4% | -1.21 | 34.0% | -1.46 | 71.0% | -0.53 |  |  |  |
| SparseVLM | 44.7 | 3.29 | 68.2 | 3.90 | 31.0 | 2.68 | 42.6 | 3.32 | 86.5% | -0.17 |
| 94.9% | -0.06 | 97.7% | -0.02 | 54.7% | -0.80 | 98.8% | -0.03 | $\mathbf{\mathcolor{Forestred}{\uparrow(34.4\%)}}$ | $\mathbf{\mathcolor{Forestred}{\uparrow(0.85)}}$ |  |

Datasets. We test our method on four common video question answering benchmarks, TGIF-QA (Jang et al., ), MSVD-QA (Xu et al., ), MSRVTT-QA (Xu et al., ) and ActivityNet-QA (Yu et al., ), where video-question pairs are massively disproportional in length. We adopt the evaluation framework proposed by Video-ChatGPT (Maaz et al., ) that utilizes both accuracy and ChatGPT score as key performance metrics with details in the Appendix A.4.

Implementation Details.
We directly apply our SparseVLM for Video-LLaVA (Lin et al., ), which is a commonly used VLM framework for video question answering. Video-LLaVA is composed of several key components, including language bind encoder $f^{v}_{M}$(Zhu et al., ) for extracting features from raw visual inputs (e.g., images or videos), a language decoder model $f_{L}$ such as Vicuna (Touvron et al., ), a visual projection layer $f_{P}$, and a word embedding layer $f_{T}$. We adopt the same inference setup as the original Video-LLaVA code base(^2^22[github.com/PKU-YuanGroup/Video-LLaVA](github.com/PKU-YuanGroup/Video-LLaVA).), as it is publicly available.

Main Results.
In Table 2, we set the Video-LLaVA with $2048$ video tokens as our upper bound for an overall average accuracy of $100.0\%$ and a score of $+0.00$. To make a fair comparison, we both preserve $135$ vision tokens ($93.4\%$ pruning ratio) for FastV (Chen et al., ) and SparseVLM.
It is clear that our approach consistently outperforms FastV across all benchmarks, both in accuracy (Acc.) and GPT evaluation score. SparseVideoLLaVA achieves a total average accuracy of $86.5\%$, a significant 34.4% higher than $52.1\%$ of FastV. From the GPT score perspective, SparseVLM only loses $0.17$ points compared to $1.02$ points of FastV.
These improvements suggest that when handling video modality containing temporal features, SparseVLM continues to deliver strong performance, generating accurate responses to diverse questions while utilizing significantly fewer tokens. This achieves an effective trade-off between inference efficiency and model performance.

## 5 Analysis

### 5.1 Effects of Relevant Text Token Selection

Figure: Figure 5: The ablation study of text raters.
Refer to caption: x5.png

We propose a selection mechanism to localize visually irrelevant text tokens to limit their negative effects in rating the significance of vision tokens. Here we conduct experiments to analyze the effects of the mechanism in Figure 5. Under the same number of vision tokens ($142$), we have 3 settings (using all tokens, only text tokens, and only text raters we select) with LLaVA (Liu et al., ) to judge vision token candidates. In TextVQA (Singh et al., ), our mechanism improves the vanilla text-aware method (only text tokens) by $0.79\%$, which validates that our extra selection is effective. Besides, by building upon the text-aware manner, we further outperform the baseline (all tokens) by $\textbf{4.3}\%$ on POPE (Li et al., ). The huge margin means POPE sparsification is quite sensitive to question prompts, and text guidance is necessary. In summary, text rater selection is general and improves the performance across various scenarios.

### 5.2 Effects of Pruned Tokens Recycling

**Table 3: Ablation study on token reconstruction (TR). Experiments are conducted on TextVQA and POPE using LLaVA with various sparsification ratios that highlight our TR generality.**
| Benchmark | Tokens | Avg |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 64 | 96 | 128 | 192 |  |  |
| TextVQA | 49.6 | 52.9 | 54.2 | 55.7 | 53.1 |
| + TR | 51.6$(\mathbf{\mathcolor{Forestred}{\uparrow 2.0}})$ | 54.5$(\mathbf{\mathcolor{Forestred}{\uparrow 1.6}})$ | 55.0$(\mathbf{\mathcolor{Forestred}{\uparrow 0.8}})$ | 56.0$(\mathbf{\mathcolor{Forestred}{\uparrow 0.3}})$ | 54.3$(\mathbf{\mathcolor{Forestred}{\uparrow 1.2}})$ |
| POPE | 57.3 | 71.7 | 77.3 | 82.1 | 72.1 |
| + TR | 75.0$(\mathbf{\mathcolor{Forestred}{\uparrow 17.7}})$ | 78.2$(\mathbf{\mathcolor{Forestred}{\uparrow 6.5}})$ | 80.5$(\mathbf{\mathcolor{Forestred}{\uparrow 3.2}})$ | 83.6$(\mathbf{\mathcolor{Forestred}{\uparrow 1.5}})$ | 79.3$(\mathbf{\mathcolor{Forestred}{\uparrow 7.2}})$ |

To validate the effectiveness of our token recycling strategy, we perform ablation experiments on the LLaVA model (Liu et al., ). The results are presented in Table 3. Across multiple sparsity ratios ($64$, $96$, $128$, $192$), our algorithm achieves a significant average performance improvement of $\textbf{1.2}\%$ and $\textbf{7.2}\%$ on TextVQA (Singh et al., ) and POPE (Li et al., ), respectively. Notably, as the number of pruned vision tokens increases, the benefit brought by our recycling method increases. For instance, when pruning from $192$ to $64$ tokens, the pruned token recycling significantly boosts the accuracy from 1.5% to 17.7% on POPE. We argue that when the size of the deleted pool grows, the amount of lost information increases. Our method effectively recycles the lost information and compresses it into few slots using the proposed reconstruction mechanism.

### 5.3 Efficiency Analysis

**Table 4: Efficiency analysis of LLaVA with SparseVLM. The detailed metric includes storage (cache memory), latency (CUDA time), and computation (FLOPs). $\Delta$ denotes the reduction ratio.**
| Method | Token | Acc. | Storage | $\Delta$ | CUDA | $\Delta$ | FLOPs | $\Delta$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Memory (MB) | Time (ms) $\downarrow$ | (T) $\downarrow$ |  |  |  |  |  |  |
| Baseline | 576 | 100% | - | - | 419.9 | - | 9.6 | - |
| FastV | 192 | 88% | 100.8 | 66.7% | 290.8 | 30.7% | 2.3 | 76.0% |
| SparseVLM | 64 | 87% | 33.6 | 88.9% | 193.5 | 53.9% | 1.5 | 84.4% |

SparseVLM affords significant efficiency and storage gains for the inference process. We conduct a comparative analysis of storage memory, CUDA time, and FLOPs on LLaVA-7B, and compare our method with the baseline method and FastV (Chen et al., ).
As displayed in Table 4, we conduct an inference efficiency analysis on a single NVIDIA A100-80GB with identical lengths of text prompts and single-image inputs.
Compared to the baseline model, SparseVLM achieves a significant reduction of $53.9\%$ in CUDA time and $84.4\%$ in FLOPs while keeping $88\%$ accuracy. Despite SparseVLM has an additional overhead to calculate text raters and cluster-pruned vision tokens, it leads to fewer than FastV tokens with comparable accuracy. Additionally, SparseVLM demonstrates lower metrics in terms of CUDA latency time and FLOPs by $23.2\%$ and $8.4\%$, respectively.

### 5.4 Visualization

As shown in Figure 6, we visualize SparseVLM on various VQA questions. From left to right, we visualize the results after we apply token pruning to different layers. On the very right, the dialogue box contains the prompt with the highlighted in red the most relevant text, and below is the generated answer in green produced by the remaining pruned image tokens. As the number of layers increases, more tokens are pruned and the Region of Interest (ROI) is gradually refined. The model systematically reduces less relevant image information while retaining key tokens that are closely tied to the question. The visualization reveals that SparseVLM, although discarding some overall image details, effectively retains essential visual tokens. These preserved tokens encapsulate the features necessary for answering the question, focusing on more relevant visual regions through their interaction with the question. More visualization cases can be found in the Appendix A.6.

Figure: Figure 6: Visualization of SparseVLM on different VQA prompts. From left to right, the visual representation becomes increasingly sparse, leaving fewer vision tokens. Best viewed in color.
Refer to caption: x6.png

## 6 Conclusion

This paper introduced an efficient text-aware training-free token optimization mechanism called SparseVLM which significantly enhanced the efficiency of various VLMs in image and video understanding tasks. Unlike existing methods, SparseVLM was able to optimize VLMs without introducing extra parameters and fine-tuning costs.
We achieved a more compact visual representation by progressively pruning the less relevant vision tokens. In addition, we employed the matrix rank to adaptively determine sparsification ratios and recycled the pruned tokens via reconstruction to reduce the information loss.
Experiments demonstrated that SparseVLM increased the efficiency of various VLMs. Particularly, LLaVA equipped with SparseVLM achieved a reduction of 53.9% latency with a compression ratio of 88.9% while maintaining 87% accuracy. Moreover, our method exceeded FastV accuracy by 34.4% in video understanding tasks. Our SparseVLM can provide practical benefits for deploying off-the-shelf VLMs on edge devices and in the cloud.

## Appendix A Appendix

### A.1 The Redundancy in Vision Tokens

In text-unrelated tasks, such as classification or segmentation, it is common to use a downsampling strategy which reduces redundancy in visual modality and makes the model more efficient to train (Zhang et al., ). In Figure 7, which starts by comparing the original image with a downsampled version. The downsampled image reduces the number of tokens from 1166 to 576, achieving a 50% increase in efficiency. However, this process results in a 15% loss of information, as indicated by the decrease in entropy from 7.44 to 6.13. This trade-off is deemed acceptable for tasks unrelated to text such as classification or segmentation. For text-related tasks, such as visual question answering (VQA), there are two different modalities, text and vision. In this figure, the prompt is ”What is written on the top of the yellow sticker on the fridge?” The output generated is ”Warning”. Pay attention to the highlighted part in both text and image, the text with the highest information density is highlighted with color, accounting for 88% of the total text; the region of interest (related to the prompt) part in the image only rates 38% in the whole image, which demonstrates that the information in images is typically more sparse than in natural language. Therefore, we proposed the SparseVLM to prune redundancy in visual tokens progressively. With our method, visual redundancy is reduced while maintaining the essential information required for accurate task performance, effectively improving the model’s efficiency and effectiveness across different vision tasks.

Figure: Figure 7: Comparison of visual redundancy in different vision tasks.
Refer to caption: x7.png

### A.2 Computing Budget Detailed Estimation

Estimation of Visual Token Significance.
In this stage, only the equation 4 averaging process requires computation. Each vision token undergoes $L_{t}-1$ additions and one division. With $L_{v}$ vision tokens in total, the number of FLOPs for this stage is $(L_{t}-1+1)\times L_{v}=L_{t}\times L_{v}$.

Relevant Text Selection. In this process, given that official PyTorch implementation for Softmax and Averaging operations, the FLOPs for equation 7 can be approximately simplified to the matrix multiplication between $H_{v}$ and $H_{q}$. The result has a shape of $L_{v}\times L_{t}$, where each element undergoes $D$ multiplications and additions. Therefore, the FLOP count can be expressed as $L_{t}\times L_{v}\times 2D$.

Sparsification Level Adaptation.
The rank of a matrix is typically computed using singular value decomposition (SVD) (Stewart, ). With the selected appropriate threshold, the number of above the threshold singular values determines the rank of the matrix. The FLOPs involved in this process can be approximated as $L_{t}\times L_{v}\times\text{min}(L_{t},L_{v})$.

Token Aggregation.
At this stage, the first part is to perform a nearest neighbor search for each element in the matrix. With the $L_{r}\times D$ matrix, this task can be simplified to calculate the distances between $L_{r}$ elements, resulting in a total of $L_{r}\times(L_{r}-1)/2$ distance calculations. Each distance computation requires sequentially executing subtraction, squaring, addition, and square root operations on $D$ elements. Consequently, the number of FLOPs in the nearest neighbor search is $L_{r}\times(L_{r}-1)/2\times 4D=L_{r}\times(L_{r}-1)\times 2D$.

The second part is density calculation. Since the operations of averaging and applying the exponential function are implemented by the official PyTorch, this part can be simplified by the matrix squaring. Therefore, the FLOPs for this part are $L_{r}\times L_{r}\times 2D$.

The third part is distance indicator calculation. The computation can be approximately simplified to compute $\rho_{i}\times\delta_{i}$. Therefore, the FLOPs for this part can be approximated as $L_{r}\times L_{r}\times 2D$.

The last part is clustering. In this part, we need to select $C$ tokens with the highest scores from a total of $L_{r}$ tokens to serve as cluster centers, and the FLOPs can be approximated as $L$.

In summary, the total FLOPs for this stage are given by

$$ $\text{FLOPs}=\underbrace{L_{r}\times(L_{r}-1)\times 2D}_{\text{Nearest Neighbors Search}}+\underbrace{L_{r}\times L_{r}\times 2D}_{\text{Density Calculation}}+\underbrace{L_{r}\times L_{r}\times 2D}_{\text{Distance Indicator Calculation}}+\underbrace{L}_{\text{Select Cluster Center}}$ $$

$$ $=L_{r}\times(3L_{r}-1)\times 2D+L.$ $$

Token Reconstruction.
Token reconstruction involves performing a weighted sum for each group, excluding the cluster center. Thus, there are $L_{r}-C$ elements to sum where each one has $1\times D$ dimensions. Consequently, the number of FLOPs for this operation is $D\times(L_{r}-C)$.

### A.3 Efficiency Details

We present a comparative efficiency analysis of SparseVLM, the baseline, and FastV (Chen et al., ) during the inference phase in Table 4. In this section, we provide additional details on the CUDA time measurement during the inference phase. Following VoCo-LLaMA Ye et al. () setting, we primarily consider the following components that contribute to the reported CUDA time: image encoding time (if applicable), kv cache load time (if applicable), and transformers forward time. We exclude other computational times that are not dependent on the model itself and the caching strategy, such as model loading time, from the CUDA time measurement. Specifically, the attention operation is implemented by Sdpa Attention: [https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial](https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial).

### A.4 dataset

We conducted experiments on several widely used visual understanding benchmarks.

GQA. (Hudson & Manning, ) The GQA benchmark is composed of three parts: scene graphs, questions, and images. The image part contains images, as well as the spatial features of images and the features of all objects in images. The questions in GQA are designed to test the understanding of visual scenes and the ability to reason about different aspects of an image.

MMBench. (Liu et al., ) The MMBench benchmark comprehensively evaluates the model’s overall performance across multiple dimensions. It includes three levels of ability dimensions. The first level (L-1) consists of two main abilities, perception and reasoning. The second level (L-2) expands based on the first level, including six sub-abilities. The third level (L-3) further refines the second level, encompassing 20 specific ability dimensions. This hierarchical structure enables a granular and comprehensive evaluation of the model’s various capabilities.

MME. (Fu et al., ) The MME benchmark is also a comprehensive benchmark meticulously designed to thoroughly evaluate various aspects of a model’s performance. It consists of 14 subtasks that specifically aim to evaluate both the model’s perceptual and cognitive abilities. By utilizing manually constructed instruction-answer pairs and concise instruction design, it effectively mitigates issues such as data leakage and unfair evaluation of model performance.

POPE. (Li et al., ) The POPE benchmark is primarily used to evaluate the degree of Object Hallucination in models. It reformulates hallucination evaluation by requiring the model to answer a series of specific binary questions regarding the presence of objects in images. Accuracy, Recall, Precision, and F1 Score are effectively employed as reliable evaluation metrics to precisely measure the model’s hallucination level under three different sampling strategies.

ScienceQA. (Lu et al., ) The ScienceQA benchmark covers a rich diversity of domains, including natural science, language science, and social science. Within each subject, questions are categorized first by the topic, then by the category, and finally by the skill. This hierarchical categorization results in 26 topics, 127 categories, and 379 skills, providing a comprehensive and diverse range of scientific questions. It provides a comprehensive evaluation of a model’s capabilities in multimodal understanding, multi-step reasoning, and interpretability.

VQA-v2. (Goyal et al., ) The VQA-v2 benchmark evaluates the model’s visual perception capabilities through open-ended questions. It consists of 265,016 images, covering a wide variety of real-world scenes and objects, providing rich visual contexts for the questions. For each question, there are 10 ground truth answers provided by human annotators, which allows for a comprehensive evaluation of the performance of different models in answering the questions accurately.

TextVQA. (Singh et al., ) The TextVQA benchmark focuses on the comprehensive integration of diverse text information within images. It meticulously evaluates the model’s text understanding and reasoning abilities through a series of visual question-answering tasks with rich textual information. Models need to not only understand the visual content of the images but also be able to read and reason about the text within the images to answer the questions accurately.

ConBench. (Zhang et al., ) The ConBench benchmark predominantly focuses on the consistency of the model’s answers across a wide variety of different tasks and question types. It presents three core capabilities in a hierarchical manner, namely observation ability (sensation), complex reasoning (reasoning), and professional knowledge (knowledge). This hierarchical design aims to gradually challenge the performance of models on different tasks and provides fine-grained evaluation indicators, so as to evaluate the performance and consistency of the model.

TGIF-QA. (Jang et al., ) The TGIF-QA benchmark is an extension of the image question answering (ImageQA) task to the video domain, aiming to promote the development of video question answering techniques. It contains 165,000 question answer pairs in total and requires the model to comprehend the details of GIF videos. Specifically, it introduces three new tasks for VideoQA (repetition count, repeating action, and state transition), which require spatio-temporal reasoning from videos, and frame QA tasks that can be answered from one of the frames.

MSVD-QA. (Xu et al., ) The MSVD-QA benchmark is based on the existing Microsoft Research Video Description (MSVD) dataset and contains 1970 video clips and approximately 50.5K QA pairs. The questions and answers are diverse in nature, covering a wide range of topics and aspects related to the video content. Due to its relatively large data size and the diversity of questions, it is widely used for video question answering tasks and video caption tasks. The tasks formed in it are open-ended questions, consisting of five types of questions: what, who, how, when and where.

MSRVTT-QA. (Xu et al., ) The MSRVTT-QA benchmark consists of 10K video clips and 243k question answer pairs. One of the main challenges addressed by the MSRVTT-QA benchmark is the complexity of understanding and reasoning about video content. Videos contain both visual and temporal information, and models need to be able to effectively process and integrate these aspects to answer the questions accurately. The tasks formed in it also consist of five types of questions, similar to the MSVD-QA benchmark.

ActivityNet-QA (Yu et al., ) The ActivityNet-QA benchmark contains 58,000 human-annotated QA pairs on 5,800 videos derived from the ActivityNet dataset. The questions are designed to cover a range of types, including motion, spatial relationship, and temporal relationship, which challenge the model to understand and reason about the video content at different levels and evaluate the performance of VideoQA models in long-term spatio-temporal reasoning.

### A.5 Implementation Details.

All of our experiments are conducted on a single Nvidia A100-80G GPU. The implementation was carried out in Python 3.10, utilizing PyTorch 2.1.2, CUDA 11.8, and transformers 4.31.0. The inference follows the evaluation settings established by LLaVA(Liu et al., ).

### A.6 Visualization

Figure 8 contains more visualization examples of SparseVLM on various VQA prompts.

Figure: Figure 8: More visualization examples of SparseVLM on different VQA prompts.
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Suggestion: Instead of using the rank, an alternative approach (which is not prone to the numerical issues that I discussed above), is to prune based on the (relative) singular values of $\mathbf{P}$: 
    1) First compute the singular values of $\mathbf{P}$, assume that these are returned in decreasing order.
    2) Divide each value by the total sum of singular values (a.k.a. the "energy" of the matrix). Let's call this vector $E$, i.e. the relative energy of each singular value.
    3) Prune $N - k$ tokens, where $k$ is the smallest value such that $\sum_{i=1}^k E_i \geq \lambda$.
- Could you please add the memory used by the baseline in Table 4?
soundness: 2
strengths: - The proposed method is compared against two recent and popular baselines: ToMe and FastV.
- The paper is well structured and the method is well explained (modulo some typos / ambiguous equations, see weaknesses).
- The method does not require any type of fine-tuning, so it can be used on top of different VLMs, which broadens its potential adoption.
- For the same number of retained tokens, Table 1 and Table 2 show that the proposed method represents a huge accuracy improvement respect the baselines.
- The paper ablates the use of token reconstruction in Table 3, which shows that the proposed improvement significantly improves over the core SparseVLM method.
summary: The paper introduces SparseVLM, a method to accelerate vision-language models (VLMs) by prunning vision tokens incrementally over layers, based on its significance for (a subset) the text tokens. The set of (significant) visual tokens to keep is computed from the self-attention scores in each layer, and the set of relevant text tokens is computed just once, using the dot-product between the text and image tokens after being embedded to the same size. This tries to reduce the computational overhead of the method, achieving real wallclock time speed-ups, for different prunning levels. The authors also propose to aggregate and reconstruct some tokens, to prevent completely losing the information of the tokens that are decided to prune. 
The paper presents results in different image and video understanding benchmarks, and compares the proposed method against two recent baselines (ToME and FastV). The results show that the proposed method improves over these baselines across different prunning thresholds, and achieves significant memory, FLOP and runtime reduction with roughly the same accuracy, when compared to FastV.
weaknesses: - The results in Table 1 and Table 2 do not reflect the reduction in neither FLOP, nor wallclock runtime. Only Table 4 offers some results comparing the proposed method only against FastV. However it's not clear to which benchmark(s) the reported accuracy corresponds to. Also, the baseline storage memory is missing (although it can be calculated from the remaining values). I would suggest that the authors report a similar figure to Figure 4 with two plots showing the avg accuracy w.r.t. a) letency or total runtime and b) FLOPs. This would represent much better the cost-vs-quality tradeoffs, for SparseVLM applied on both LLaVA and MGM. This is crucial to increase the rating of the paper. If space is limited, I would suggest reporting values for individual datasets in the appendix, and report only the average in the main text (unless there are any major outliers).
- Table 2 does not even represent the speed-vs-accuracy trade-off, nor the "token budget"-vs-accuracy, since only a single token budget of 135 is represented. Also, this value is not the same used in any of the image results reported in Table 1. Which begs the question: why was this particular value used? Please, provide figures as described above.
- It's not 100% clear how $\mathbf{P}$ in section 3.2 is calculated. According to eq. (1) and (2), $\mathbf{P}$ is a subset of rows and columns of the attention matrix (after the softmax), but lines 183-184 refer the "logits" (i.e. $\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{D}}$). It's also not clear if the attention matrix is re-normalized after the selected visual tokens are dropped from the keys or not.
- Notation in eq. (7) is ambiguous. The index $j$ in the sum isn't used anywhere. Also, notice that the size of $\mathbf{H}_v \mathbf{H}_q^\top$ is $L_v \times L_q$, which is inconsistent with the sum over $j$, assuming $j$ denotes a column index, since $\mathbf{R}_i$ supposed to be the average over visual tokens for the $i$-th query token. This is a small mistake that can be fixed by using $\mathbf{H}_q \mathbf{H}_v^\top$, to match the dimension order of $\mathbf{P}$ is $L_v \times L_q$ (i.e. text $\times$ vision).
- The choice of the relevant text token select threshold $m = \text{Mean}(\mathbf{R})$ isn't justified. Why this threshold and not something else? E.g. the text tokens in the with the highest $R$ score such that the sum of 
- The number of vision tokens to prune is based on the rank of $\textbf{P}$, this can be problematic due to numerical precision. For instance, suppose that $n = L_t = L_v$, what happens if we get that half of the the singular values of P are $10^{-5}$ and the rest are $10^5$? The rank would be technically $n$, but is it really or do we get $10^{-5}$ rather than 0 due to numerical errors?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Due to the weakness of this paper, I tend to be borderline negative about this paper. See weakness section for details of my concerns and questions.



###################

Thanks for your reply. Most of my concerns are solved. I will raise my rating to 6. Hope to see the SparseVLM with Flash Attention to be released.
soundness: 2
strengths: 1. The proposed SparseLVM framework, including rank-based strategy and token recycling, is reasonable.

2. The paper is clear to read. It is easy for the audience to follow the sophisticated designs in SparseLVM.

3. Experiments are performed on both image and video benchmarks.
summary: This paper introduces SparseLVM, a training-free method to prune redundant visual tokens in LVLMs. SparseLVM leverages visual-relevant text tokens to rate the significance of vision tokens within the self-attention matrix, leading to the progressive pruning of irrelevant visual tokens. Specifically, SparseLVM proposes a rank-based strategy to adaptively determine the sparsification ratio for each layer and a token recycling method that compresses pruned tokens into center tokens. SparseLVM reduces the number of tokens with less performance drop than ToMe and FastV.
weaknesses: 1. The proposed SparseLVM is not practical for two reasons. 

- First, it is not compatible with FlashAttn, which is a standard solution for accelerating the calculation of self-attention. In SparseLVM, the attention matrix must be explicitly obtained to select redundant visual tokens in **each layer** of LVLMs. However, FlashAttn does not support attaining the explicit attention matrix. Without compatibility with FlashAttn, SparseLVM will be limited in its efficiency. The SparseLVM should be compared with the original LVLMs with FlashAttn. 
- Second, although the performance drop of SparseLVM is less than ToMe and FastV, it is still considerably large. More explanations and discussions are necessary.

2. Some important ablation studies are not shown.

- For verifying efficiency, the SparseLVM should be compared with the original LVLMs with FlashAttn. 
- For verifying effectiveness, the SparseLVM should report more results on high-resolution image understanding benchmarks, such as DocVQA, InfoVQA, AI2D, etc, as in leading LVLMs [1].


3. Some details of SparseLVM are not clearly introduced. 
- What is the value of m in equation (6), lambda in equation (8), and tau in equation (9)? How does SparseLVM determine them?
- After Visual Token Recycling, how does SparseLVM insert these recycled tokens into the preserved tokens? It seems that these recycled tokens have the risk of spatial relationship between different image tokens. 



[1] Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness for details.
soundness: 3
strengths: 1. The proposed Text-Guided Visual Token Pruning is novel, which  introduce text-aware guidance for visual token sparsification. The experiments showing this approach outperforms text-agnostic methods like FastV by 14.8% on LLaVA when retaining only 64 tokens, which validate the effectiveness of using text tokens as "raters" for visual importance.

2. The proposed method is training-free, which is easy to deploy. 

3. The paper introduces a rank-based strategy to adaptively determine the sparsification ratio for each layer, which saves the number of hyperparameters and reduces the engineering effort.

4. Instead of directly pruning tokens, the proposed method merges them into compact representations. Ablation studies show this recycling mechanism improves accuracy from 1.5% to 17.7% on POPE when pruning to 64 tokens, demonstrating significant information preservation.
summary: This paper introduces SparseVLM, a training-free token optimization that improves the efficiency of Vision-Language Models (VLMs) by reducing visual token. The method improve the efficency of VLM by three steps: 1) first identify text tokens strongly correlated with visual signals via cross-attention, and then 2) measure the contribution of visual tokens to the selected visual-relevant text tokens (raters), and finally 3) adaptively prune the insignificant vision token. Experiments show the LLaVA equipped with SparseVLM reduces 61%∼67%
FLOPs with a compression ratio of 78% while maintaining 93% of the accuracy. The proposed method consistently outperforms the
existing state-of-the-art method FastV by 7.7%∼14.8% on LLaVA, 10.2%∼21.6% on MiniGemini, and 34.4% on VideoLLaVA.
weaknesses: 1. The proposed method requires the attention scores to select the visual tokens to be pruned. This would not be compatible with FlashAttention, and may require significantly more memory and possibly extra latency. The authors are encourage to do comparison with baselines powered with FlashAttention and show the result. My concerns is that without using FlashAttention the proposed method could cost much more memory, and make it harder or infeasible to be deployed. Specifically, author should show the peak memory consumption and latency comparision between proposed method vs Baseline with FlashAttention.

2. The experimental evaluation lacks comparison with latest token reduction methods that outperform FastV. Notably absent are Token summarization[https://arxiv.org/abs/2410.14072 ], Progressive Token Pruning[https://arxiv.org/abs/2301.13741]- all of which have better performance comparing to FastV in different tasks. Including these state-of-the-art baselines is essential for a comprehensive evaluation.

3. The experimental focuses on a single VLM architecture: LLaVA, which limiting evidence of the method's broader applicability. Testing across other VLM architectures like Flamingo would better demonstrate generalizability, particularly with different visual and textual feature fusion mechanisms.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How to deal with RoPE for the sparsified visual tokens?
2. In Equation 7, why was it chosen to use the features from the visual encoder and text embeddings to select raters? Does this lead to the method performing poorly on problems that require logical reasoning, such as the performance on MMMU、Reasoning-related subset of MMBnech?
soundness: 2
strengths: 1. The paper is well-written, showcasing a clear and articulate presentation of ideas.
2. The paper is simple and easy to follow.
3. The training-free token optimization mechanism is more universal and can be better adapted to various VLM models compared to methods that require training.
summary: This paper proposes an efficient training-free token optimization mechanism dubbed SparseVLM without extra parameters or fine-tuning costs.

The contributions of this paper are summaried as follows:
1. The paper introduces a sparsification framework dubbed SparseVLM for vision-language models. 
2. The paper first assigns visual-relevant text tokens as raters, adaptively prunes VLMs with the rank of the attention logits, and recycles partial tokens.
3. Consistently outperforms the FastV.
weaknesses: 1. One motivation of the paper is that visual tokens should be sparsified adaptively based on the question prompt. This prompt-aware sparsification, while preserving the original model's performance as much as possible, causes the VLM to lose its ability for multi-turn conversations.
2. The method in the paper requires explicitly obtaining the attention map, but in many inference acceleration frameworks, the attention map is not accessible, such as in FlashAttention. In Table 4, is the baseline using the standard attention mechanism? If compared with FlashAttention, does it still have a speed advantage?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In Table 1, for the experimental results under the settings "Retain 192/128/64 Tokens," what exactly do these settings mean? For FastV, does this mean that only this number of image tokens is retained across all layers?
2. 5.1 section，"3 settings (using all tokens, only text tokens, and only text raters we select)"，explain the settings in detail.
3. Are you doing the pruning and recycling process in the prefilling stage? "we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer". If as said like this, do we prune and recycle at each layer in the prefilling stage to keep 192/128/64 tokens in experiment? Please give a clear explanation of your sparsification process, which is not stated in the paper.
soundness: 3
strengths: 1. The paper presents SparseVLM, a training-free mechanism designed to improve the efficiency of vision-language models (VLMs) by optimizing the handling of visual tokens.
2. The paper is well-written and clearly presents the proposed framework. The authors provide detailed descriptions of their methodology.
3.  Considering the recycling of deleted image tokens is an effective method to alleviate performance degradation.
summary: This paper introduces SparseVLM, an efficient, training-free optimization method for visual tokens in vision-language models (VLMs). Recognizing that visual tokens in VLMs often introduce high computational costs due to their low information density, SparseVLM selectively prunes redundant tokens without needing additional training data or parameters. By using the visual-relevant text tokens (from the self-attention matrix) to rate the importance of visual tokens, SparseVLM identifies and prunes unnecessary tokens progressively. A rank-based strategy is used to determine the pruning ratio per layer, while a token recycling method condenses pruned tokens into compact forms, maintaining essential visual information.
weaknesses: 1. The primary focus on efficiency must maintain performance; otherwise, efficiency becomes meaningless. In Table 1, even the setting with the least acceleration, "Retain 192 Tokens," exhibits substantial performance drops across multiple benchmarks. Specifically, GQA drops by 4.3%, POPE by 2.3%, VQAv2 by 2.9%, MMB by 2.2%, and TextVQA by 2.1%, which are unacceptable losses.
2. In Section 5.1, why was the unusual number of 142 image tokens chosen for the experiment? Additionally, if the goal is to demonstrate the effectiveness of the "text rater," it would be insufficient to test only one efficiency setting. A range of settings retaining different proportions of image tokens should be used to substantiate its effectiveness across varying conditions.
3. In the section "Sparsification Level Adaptation",  N is calculated to determine the number of tokens deleted in each layer for adaptive purposes. However, in the later experimental sections, the number of retained image tokens (e.g., 192) is specified directly. If the result of N in a decoder layer is 0, how can you specify retained image tokens to 192? Isn’t this contradictory? 
4. Rank(P) is a rather unusual way to compute visual redundancy. P represents a part of the attention map, but it is unclear why the linear correlation among attention vectors would relate to visual redundancy. Is there any supporting evidence for this, such as a reference to a paper?
5. Figure 1 shows that the patches selected by fastv are identical under different questions, which is unreasonable. Since fastv relies on the attention between text and image (this can be found in the source code), the selected patches should not be exactly the same. You may check for any errors in the process.
6. The paper mentions, "We reuse the self-attention matrix of visual-text tokens directly from the decoder layers without extra training parameters for sparsification." However, if the method requires outputting the self-attention matrix, it can not use FlashAttention, which would significantly impact inference speed.
7. In Table 1, it would be helpful to include efficiency evaluation like FLOPs and latency directly alongside performance scores on the benchmarks to facilitate comparison, the number of retained image tokens is not sufficient to evaluate efficiency.
8. One contribution claims, "it is the first attempt to explore the potential of text-aware guidance for efficient inference of VLMs." This is inaccurate, as the "fastv" approach also prunes image tokens based on text tokens’ attention to image tokens.
9. The description of the ToMe method in the Related Work section is inaccurate. "For example, ToMe (Bolya et al., 2022) prunes according to the relevance between visual tokens and text and merges both modalities through the BSM algorithm."
10. In the introduction, the calculation of the number of image tokens seems incorrect. The claim, "For instance, a 672 × 672 image in LLaVA (Liu et al., 2024) yields 2304 vision tokens that span over half of the context length," does not align with the correct calculation of 576 × 5 (four sub-images plus one resized original image). You can check it again, there might be an error somewhere.

Meta-review: The authors address a timely and relevant problem: The number of visual tokens to a VLM are often redundant and substantially increase the computational cost of the model. The authors propose to address this by proposing an iterative token sparification strategy that reduces the overall number of tokens in a VLM, by conditioning on the input question, inspired by previous works in the literature on token reduction such as ToMe and FastV. Moreover, the approach does not require any training, which means that it can be potentially applied to a wide range of VLMs.

Reviewers had multiple concerns in the rebuttal, some of which the authors addressed convincingly: For example, the proposed approach is still comparible with FlashAttention.

However, a remaining concern is that the proposed sparsisfication strategy still requires computation itself. Therefore, comparing to prior works based only on the number of tokens as in Table 1, is not sufficient. The authors should transparently compare the runtime and GFLOPs as well, as raised by some reviewers (It has also been widely reported [before](https://arxiv.org/abs/2110.12894) that reporting only specific efficiency indicators can be misleading). Although the authors provided these values in the rebuttal, the results are underwhelming for a range of token budgets. Overall this is concerning, as well as the fact that this, and subsequent replies to Reviewer-RuY4 are not included in the revised main paper. 

Reviewer-YHTW also mentioned three more recent papers which achieve better results than FastV which are not compared to. The AC and Senior-AC agree that Victor[1] and UPop[2] are not fair comparisons. But for PDrop, the authors have once again not compared the actual runtime and GFLOPs.

Therefore, after some deliberation, the AC and Senior-AC decided to reject the paper. Authors are encouraged to revise their paper according to the reviewers' feedback, in particular comparing fairly to prior works not only in the token budget, but the FLOPs and runtime as well, and to resubmit to another venue.

Decision: Reject