SparseVLM: Visual Token Sparsification for Efficient Vision Language Models Inference

Authors: Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Shanghang Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Sparsification, Vision Language Model, Efficiency

Abstract:
In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens and require additional training data. Differently, we propose an efficient training-free token optimization mechanism dubbed SparseVLM without extra parameters or fine-tuning costs. Concretely, given that visual tokens complement text tokens in VLMs for linguistic reasoning, we select visual-relevant text tokens to rate the significance of vision tokens within the self-attention matrix extracted from the VLMs. Then we progressively prune irrelevant tokens. To maximize sparsity while retaining essential information, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that our SparseVLM improves the efficiency of various VLMs across a range of image and video understanding tasks. In particular, LLaVA equipped with SparseVLM reduces 61\% $\sim$ 67\% FLOPs with a compression ratio of 78\% while maintaining 93\% of the accuracy.

Decision: reject
Reviewer scores: 6, 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730040200953

thread_type:
official_review

<thread_object>
creator: reviewer_rbrk

<content>
summary:
This paper introduces SparseVLM, an efficient, training-free optimization method for visual tokens in vision-language models (VLMs). Recognizing that visual tokens in VLMs often introduce high computational costs due to their low information density, SparseVLM selectively prunes redundant tokens without needing additional training data or parameters. By using the visual-relevant text tokens (from the self-attention matrix) to rate the importance of visual tokens, SparseVLM identifies and prunes unnecessary tokens progressively. A rank-based strategy is used to determine the pruning ratio per layer, while a token recycling method condenses pruned tokens into compact forms, maintaining essential visual information.

strengths:
1. The paper presents SparseVLM, a training-free mechanism designed to improve the efficiency of vision-language models (VLMs) by optimizing the handling of visual tokens.
2. The paper is well-written and clearly presents the proposed framework. The authors provide detailed descriptions of their methodology.
3.  Considering the recycling of deleted image tokens is an effective method to alleviate performance degradation.

weaknesses:
1. The primary focus on efficiency must maintain performance; otherwise, efficiency becomes meaningless. In Table 1, even the setting with the least acceleration, "Retain 192 Tokens," exhibits substantial performance drops across multiple benchmarks. Specifically, GQA drops by 4.3%, POPE by 2.3%, VQAv2 by 2.9%, MMB by 2.2%, and TextVQA by 2.1%, which are unacceptable losses.
2. In Section 5.1, why was the unusual number of 142 image tokens chosen for the experiment? Additionally, if the goal is to demonstrate the effectiveness of the "text rater," it would be insufficient to test only one efficiency setting. A range of settings retaining different proportions of image tokens should be used to substantiate its effectiveness across varying conditions.
3. In the section "Sparsification Level Adaptation",  N is calculated to determine the number of tokens deleted in each layer for adaptive purposes. However, in the later experimental sections, the number of retained image tokens (e.g., 192) is specified directly. If the result of N in a decoder layer is 0, how can you specify retained image tokens to 192? Isn’t this contradictory? 
4. Rank(P) is a rather unusual way to compute visual redundancy. P represents a part of the attention map, but it is unclear why the linear correlation among attention vectors would relate to visual redundancy. Is there any supporting evidence for this, such as a reference to a paper?
5. Figure 1 shows that the patches selected by fastv are identical under different questions, which is unreasonable. Since fastv relies on the attention between text and image (this can be found in the source code), the selected patches should not be exactly the same. You may check for any errors in the process.
6. The paper mentions, "We reuse the self-attention matrix of visual-text tokens directly from the decoder layers without extra training parameters for sparsification." However, if the method requires outputting the self-attention matrix, it can not use FlashAttention, which would significantly impact inference speed.
7. In Table 1, it would be helpful to include efficiency evaluation like FLOPs and latency directly alongside performance scores on the benchmarks to facilitate comparison, the number of retained image tokens is not sufficient to evaluate efficiency.
8. One contribution claims, "it is the first attempt to explore the potential of text-aware guidance for efficient inference of VLMs." This is inaccurate, as the "fastv" approach also prunes image tokens based on text tokens’ attention to image tokens.
9. The description of the ToMe method in the Related Work section is inaccurate. "For example, ToMe (Bolya et al., 2022) prunes according to the relevance between visual tokens and text and merges both modalities through the BSM algorithm."
10. In the introduction, the calculation of the number of image tokens seems incorrect. The claim, "For instance, a 672 × 672 image in LLaVA (Liu et al., 2024) yields 2304 vision tokens that span over half of the context length," does not align with the correct calculation of 576 × 5 (four sub-images plus one resized original image). You can check it again, there might be an error somewhere.

questions:
1. In Table 1, for the experimental results under the settings "Retain 192/128/64 Tokens," what exactly do these settings mean? For FastV, does this mean that only this number of image tokens is retained across all layers?
2. 5.1 section，"3 settings (using all tokens, only text tokens, and only text raters we select)"，explain the settings in detail.
3. Are you doing the pruning and recycling process in the prefilling stage? "we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer". If as said like this, do we prune and recycle at each layer in the prefilling stage to keep 192/128/64 tokens in experiment? Please give a clear explanation of your sparsification process, which is not stated in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730456662451

thread_type:
official_review

<thread_object>
creator: reviewer_7sbx

<content>
summary:
This paper proposes an efficient training-free token optimization mechanism dubbed SparseVLM without extra parameters or fine-tuning costs.

The contributions of this paper are summaried as follows:
1. The paper introduces a sparsification framework dubbed SparseVLM for vision-language models. 
2. The paper first assigns visual-relevant text tokens as raters, adaptively prunes VLMs with the rank of the attention logits, and recycles partial tokens.
3. Consistently outperforms the FastV.

strengths:
1. The paper is well-written, showcasing a clear and articulate presentation of ideas.
2. The paper is simple and easy to follow.
3. The training-free token optimization mechanism is more universal and can be better adapted to various VLM models compared to methods that require training.

weaknesses:
1. One motivation of the paper is that visual tokens should be sparsified adaptively based on the question prompt. This prompt-aware sparsification, while preserving the original model's performance as much as possible, causes the VLM to lose its ability for multi-turn conversations.
2. The method in the paper requires explicitly obtaining the attention map, but in many inference acceleration frameworks, the attention map is not accessible, such as in FlashAttention. In Table 4, is the baseline using the standard attention mechanism? If compared with FlashAttention, does it still have a speed advantage?

questions:
1. How to deal with RoPE for the sparsified visual tokens?
2. In Equation 7, why was it chosen to use the features from the visual encoder and text embeddings to select raters? Does this lead to the method performing poorly on problems that require logical reasoning, such as the performance on MMMU、Reasoning-related subset of MMBnech?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609024265

thread_type:
official_review

<thread_object>
creator: reviewer_yhtw

<content>
summary:
This paper introduces SparseVLM, a training-free token optimization that improves the efficiency of Vision-Language Models (VLMs) by reducing visual token. The method improve the efficency of VLM by three steps: 1) first identify text tokens strongly correlated with visual signals via cross-attention, and then 2) measure the contribution of visual tokens to the selected visual-relevant text tokens (raters), and finally 3) adaptively prune the insignificant vision token. Experiments show the LLaVA equipped with SparseVLM reduces 61%∼67%
FLOPs with a compression ratio of 78% while maintaining 93% of the accuracy. The proposed method consistently outperforms the
existing state-of-the-art method FastV by 7.7%∼14.8% on LLaVA, 10.2%∼21.6% on MiniGemini, and 34.4% on VideoLLaVA.

strengths:
1. The proposed Text-Guided Visual Token Pruning is novel, which  introduce text-aware guidance for visual token sparsification. The experiments showing this approach outperforms text-agnostic methods like FastV by 14.8% on LLaVA when retaining only 64 tokens, which validate the effectiveness of using text tokens as "raters" for visual importance.

2. The proposed method is training-free, which is easy to deploy. 

3. The paper introduces a rank-based strategy to adaptively determine the sparsification ratio for each layer, which saves the number of hyperparameters and reduces the engineering effort.

4. Instead of directly pruning tokens, the proposed method merges them into compact representations. Ablation studies show this recycling mechanism improves accuracy from 1.5% to 17.7% on POPE when pruning to 64 tokens, demonstrating significant information preservation.

weaknesses:
1. The proposed method requires the attention scores to select the visual tokens to be pruned. This would not be compatible with FlashAttention, and may require significantly more memory and possibly extra latency. The authors are encourage to do comparison with baselines powered with FlashAttention and show the result. My concerns is that without using FlashAttention the proposed method could cost much more memory, and make it harder or infeasible to be deployed. Specifically, author should show the peak memory consumption and latency comparision between proposed method vs Baseline with FlashAttention.

2. The experimental evaluation lacks comparison with latest token reduction methods that outperform FastV. Notably absent are Token summarization[https://arxiv.org/abs/2410.14072 ], Progressive Token Pruning[https://arxiv.org/abs/2301.13741]- all of which have better performance comparing to FastV in different tasks. Including these state-of-the-art baselines is essential for a comprehensive evaluation.

3. The experimental focuses on a single VLM architecture: LLaVA, which limiting evidence of the method's broader applicability. Testing across other VLM architectures like Flamingo would better demonstrate generalizability, particularly with different visual and textual feature fusion mechanisms.

questions:
See weakness for details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707495847

thread_type:
official_review

<thread_object>
creator: reviewer_hizv

<content>
summary:
This paper introduces SparseLVM, a training-free method to prune redundant visual tokens in LVLMs. SparseLVM leverages visual-relevant text tokens to rate the significance of vision tokens within the self-attention matrix, leading to the progressive pruning of irrelevant visual tokens. Specifically, SparseLVM proposes a rank-based strategy to adaptively determine the sparsification ratio for each layer and a token recycling method that compresses pruned tokens into center tokens. SparseLVM reduces the number of tokens with less performance drop than ToMe and FastV.

strengths:
1. The proposed SparseLVM framework, including rank-based strategy and token recycling, is reasonable.

2. The paper is clear to read. It is easy for the audience to follow the sophisticated designs in SparseLVM.

3. Experiments are performed on both image and video benchmarks.

weaknesses:
1. The proposed SparseLVM is not practical for two reasons. 

- First, it is not compatible with FlashAttn, which is a standard solution for accelerating the calculation of self-attention. In SparseLVM, the attention matrix must be explicitly obtained to select redundant visual tokens in **each layer** of LVLMs. However, FlashAttn does not support attaining the explicit attention matrix. Without compatibility with FlashAttn, SparseLVM will be limited in its efficiency. The SparseLVM should be compared with the original LVLMs with FlashAttn. 
- Second, although the performance drop of SparseLVM is less than ToMe and FastV, it is still considerably large. More explanations and discussions are necessary.

2. Some important ablation studies are not shown.

- For verifying efficiency, the SparseLVM should be compared with the original LVLMs with FlashAttn. 
- For verifying effectiveness, the SparseLVM should report more results on high-resolution image understanding benchmarks, such as DocVQA, InfoVQA, AI2D, etc, as in leading LVLMs [1].


3. Some details of SparseLVM are not clearly introduced. 
- What is the value of m in equation (6), lambda in equation (8), and tau in equation (9)? How does SparseLVM determine them?
- After Visual Token Recycling, how does SparseLVM insert these recycled tokens into the preserved tokens? It seems that these recycled tokens have the risk of spatial relationship between different image tokens. 



[1] Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution

questions:
Due to the weakness of this paper, I tend to be borderline negative about this paper. See weakness section for details of my concerns and questions.



###################

Thanks for your reply. Most of my concerns are solved. I will raise my rating to 6. Hope to see the SparseVLM with Flash Attention to be released.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730855020769

thread_type:
official_review

<thread_object>
creator: reviewer_ruy4

<content>
summary:
The paper introduces SparseVLM, a method to accelerate vision-language models (VLMs) by prunning vision tokens incrementally over layers, based on its significance for (a subset) the text tokens. The set of (significant) visual tokens to keep is computed from the self-attention scores in each layer, and the set of relevant text tokens is computed just once, using the dot-product between the text and image tokens after being embedded to the same size. This tries to reduce the computational overhead of the method, achieving real wallclock time speed-ups, for different prunning levels. The authors also propose to aggregate and reconstruct some tokens, to prevent completely losing the information of the tokens that are decided to prune. 
The paper presents results in different image and video understanding benchmarks, and compares the proposed method against two recent baselines (ToME and FastV). The results show that the proposed method improves over these baselines across different prunning thresholds, and achieves significant memory, FLOP and runtime reduction with roughly the same accuracy, when compared to FastV.

strengths:
- The proposed method is compared against two recent and popular baselines: ToMe and FastV.
- The paper is well structured and the method is well explained (modulo some typos / ambiguous equations, see weaknesses).
- The method does not require any type of fine-tuning, so it can be used on top of different VLMs, which broadens its potential adoption.
- For the same number of retained tokens, Table 1 and Table 2 show that the proposed method represents a huge accuracy improvement respect the baselines.
- The paper ablates the use of token reconstruction in Table 3, which shows that the proposed improvement significantly improves over the core SparseVLM method.

weaknesses:
- The results in Table 1 and Table 2 do not reflect the reduction in neither FLOP, nor wallclock runtime. Only Table 4 offers some results comparing the proposed method only against FastV. However it's not clear to which benchmark(s) the reported accuracy corresponds to. Also, the baseline storage memory is missing (although it can be calculated from the remaining values). I would suggest that the authors report a similar figure to Figure 4 with two plots showing the avg accuracy w.r.t. a) letency or total runtime and b) FLOPs. This would represent much better the cost-vs-quality tradeoffs, for SparseVLM applied on both LLaVA and MGM. This is crucial to increase the rating of the paper. If space is limited, I would suggest reporting values for individual datasets in the appendix, and report only the average in the main text (unless there are any major outliers).
- Table 2 does not even represent the speed-vs-accuracy trade-off, nor the "token budget"-vs-accuracy, since only a single token budget of 135 is represented. Also, this value is not the same used in any of the image results reported in Table 1. Which begs the question: why was this particular value used? Please, provide figures as described above.
- It's not 100% clear how $\mathbf{P}$ in section 3.2 is calculated. According to eq. (1) and (2), $\mathbf{P}$ is a subset of rows and columns of the attention matrix (after the softmax), but lines 183-184 refer the "logits" (i.e. $\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{D}}$). It's also not clear if the attention matrix is re-normalized after the selected visual tokens are dropped from the keys or not.
- Notation in eq. (7) is ambiguous. The index $j$ in the sum isn't used anywhere. Also, notice that the size of $\mathbf{H}_v \mathbf{H}_q^\top$ is $L_v \times L_q$, which is inconsistent with the sum over $j$, assuming $j$ denotes a column index, since $\mathbf{R}_i$ supposed to be the average over visual tokens for the $i$-th query token. This is a small mistake that can be fixed by using $\mathbf{H}_q \mathbf{H}_v^\top$, to match the dimension order of $\mathbf{P}$ is $L_v \times L_q$ (i.e. text $\times$ vision).
- The choice of the relevant text token select threshold $m = \text{Mean}(\mathbf{R})$ isn't justified. Why this threshold and not something else? E.g. the text tokens in the with the highest $R$ score such that the sum of 
- The number of vision tokens to prune is based on the rank of $\textbf{P}$, this can be problematic due to numerical precision. For instance, suppose that $n = L_t = L_v$, what happens if we get that half of the the singular values of P are $10^{-5}$ and the rest are $10^5$? The rank would be technically $n$, but is it really or do we get $10^{-5}$ rather than 0 due to numerical errors?

questions:
- Suggestion: Instead of using the rank, an alternative approach (which is not prone to the numerical issues that I discussed above), is to prune based on the (relative) singular values of $\mathbf{P}$: 
    1) First compute the singular values of $\mathbf{P}$, assume that these are returned in decreasing order.
    2) Divide each value by the total sum of singular values (a.k.a. the "energy" of the matrix). Let's call this vector $E$, i.e. the relative energy of each singular value.
    3) Prune $N - k$ tokens, where $k$ is the smallest value such that $\sum_{i=1}^k E_i \geq \lambda$.
- Could you please add the memory used by the baseline in Table 4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732352556266

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear reviewer RuY4, hiZv, YHTW, 7sbX, and Rbrk

We sincerely thank you for your valuable and constructive comments. We have provided detailed answers and revised the paper accordingly. Specifically, it includes the following aspects:

1. **Expanded Efficiency Experiments**: We have incorporated comprehensive experiments such as latency-vs.-accuracy and FLOPs-vs.-Accuracy analyses for image (e.g., LLaVA and MGM) and video (e.g., VideoLLaVA) understanding architectures across diverse benchmarks, with a detailed examination of memory costs.

2. **In-depth Specific Tasks Experiments**: We have augmented our study with additional experiments focusing on logical reasoning and higher-resolution tasks. Furthermore, we have implemented SparseVLM in the Qwen-VL architecture to validate its generality.

3. **Detailed Performance Discussion**: Our methodology shows state-of-the-art performance at efficiency levels comparable to other existing methods. The adaptability of our approach is highlighted through the proposed sparsification level adaptation, allowing users to easily configure the sparsification ratio for desired efficiency-performance trade-offs.

4. **Supplementary Ablation Experiments**: We have included further comparisons between rank and SVD, explored additional token settings for testing text raters, and conducted more experiments regarding text token selection thresholds.

5. **Enhanced Explanations**: We have provided elaborations on Equations (6), (7), (8), and (9), elucidated the strategy for position embedding, and demonstrated our advantages in multi-turn conversations.

As most reviewers were concerned about **the compatibility of SparseVLM with FlashAttention**, we have provided a general response: Our exploitation of the attention matrix is to compute the token sparsification, which is similar to the self-attention operation. Therefore, we can readily utilize FlashAttention to accomplish this by employing a special values matrix in self-attention. We have included the detailed implementation in the individual responses. As we only perform sparsification in a few layers in the LLM, the additional use of FlashAttention will not bring significant computation overhead. We have provided a more detailed experimental analysis in the revised version.

Additionally, we have revised the paper and highlighted the changes in blue fonts. Below is a summary of the updates:

1. Line 67-68, update the caption in Figure 1(c).
2. Line 123-124, update the number of image tokens.
3. Line 143-145, update the description of the ToMe method.
4. Line 215, update the equation (7).
5. Line 450, add the memory used by the baseline in Table 4.
6. In A.7, add more trade-off results in Figures 9, 10, and 11.

Thank you once again for your valuable feedback and suggestions, which have significantly improved the quality of our work. If you have any further questions or if anything remains unclear, please don’t hesitate to let us know. We would be more than happy to discuss your concerns in greater detail. If the response is satisfactory, we hope you will raise the score so that more people can see this paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors address a timely and relevant problem: The number of visual tokens to a VLM are often redundant and substantially increase the computational cost of the model. The authors propose to address this by proposing an iterative token sparification strategy that reduces the overall number of tokens in a VLM, by conditioning on the input question, inspired by previous works in the literature on token reduction such as ToMe and FastV. Moreover, the approach does not require any training, which means that it can be potentially applied to a wide range of VLMs.

Reviewers had multiple concerns in the rebuttal, some of which the authors addressed convincingly: For example, the proposed approach is still comparible with FlashAttention.

However, a remaining concern is that the proposed sparsisfication strategy still requires computation itself. Therefore, comparing to prior works based only on the number of tokens as in Table 1, is not sufficient. The authors should transparently compare the runtime and GFLOPs as well, as raised by some reviewers (It has also been widely reported [before](https://arxiv.org/abs/2110.12894) that reporting only specific efficiency indicators can be misleading). Although the authors provided these values in the rebuttal, the results are underwhelming for a range of token budgets. Overall this is concerning, as well as the fact that this, and subsequent replies to Reviewer-RuY4 are not included in the revised main paper. 

Reviewer-YHTW also mentioned three more recent papers which achieve better results than FastV which are not compared to. The AC and Senior-AC agree that Victor[1] and UPop[2] are not fair comparisons. But for PDrop, the authors have once again not compared the actual runtime and GFLOPs.

Therefore, after some deliberation, the AC and Senior-AC decided to reject the paper. Authors are encouraged to revise their paper according to the reviewers' feedback, in particular comparing fairly to prior works not only in the token budget, but the FLOPs and runtime as well, and to resubmit to another venue.