SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Authors: Yuan Zhang
Chun-Kai Fan
Junpeng Ma
Wenzhao Zheng
Tao Huang
Kuan Cheng
Denis A Gudovskiy
Tomoyuki Okuno
Yohei Nakata
Kurt Keutzer
Shanghang Zhang
Venue: ICML 2025 poster (2025)
Keywords: Sparsification
Vision Language Models
Efficiency
Primary area: deep_learning->algorithms
TL;DR: We propose an efficient text-aware training-free vision token optimization mechanism called SparseVLM.

Abstract: In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens and require additional training data. Differently, we propose an efficient training-free token optimization mechanism dubbed **SparseVLM** without extra parameters or fine-tuning costs. Concretely, given that visual tokens complement text tokens in VLMs for linguistic reasoning, we select visual-relevant text tokens to rate the significance of vision tokens within the self-attention matrix extracted from the VLMs. Then we progressively prune irrelevant tokens. To maximize sparsity while retaining essential information, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that our SparseVLM improves the efficiency of various VLMs across a range of image and video understanding tasks. In particular, when LLaVA is equipped with SparseVLM, it achieves a 54\% reduction in FLOPs, lowers CUDA time by 37\%, and maintains an accuracy rate of 97\%. Our code is available at https://github.com/Gumpest/SparseVLMs.

SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Yuan Zhang" 1 Chun-Kai Fan"
Junpeng Ma *2 Wenzhao Zheng Tao Huang*
Kuan Cheng'
Denis Gudovskiy² Tomoyuki Okuno Yohei Nakata Kurt 3
Shanghang Zhang'
Abstract
significant progress. To combine visual signals with textual semantics, the mainstream practice in VLMs (Team et al., 2023; Bai et al., 2023; Chen et al., 2024b; Li et al., 2024c; 2023a) employs sequential visual representation, where im- ages are extracted into visual tokens and sent into an LLM decoder. With modal alignment and instruction fine-tuning (Du et al., 2022; Liu al., 2024a; Zhu et al., 2024b), recent VLMs successfully adapt LLMs to the vision domain and inherit their perception and reasoning abilities.
In vision-language models (VLMs), visual tokens usually bear a significant amount of computa- tional overhead despite sparsity of information in them when compared to text tokens. To ad- dress this, most existing methods learn network to prune redundant visual tokens using certain training data. Differently, we propose a text- guided training-free token optimization mecha- nism dubbed Sparse VLM that eliminates the need of extra parameters or fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, select relevant text tokens to rate the significance of visual to- kens using self-attention matrices and, then, prune visual tokens using the proposed strategy to max- imize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside token recycling method that compresses pruned tokens into more compact representations. Experimental results show that Sparse VLM increases the efficiency of various VLMMs in a number of image and video under- standing tasks. Our code available at ht ttsp:: / //github... www/ (omon//ttmett....mm /Spppassee V......
Despite the promising performance, further incorporation of visual tokens inevitably introduces a huge memory and com- putational overhead when compared to LLMs, particularly for high-resolution images (Li et al., 2024c) and long videos (Lin et al., 2024). For instance, a 672 x 672 image in LLaVA (Liu et al., 2024b) yields 2304 visual tokens that span over half of the context length. However, the information in images is typically more sparse than in natural languages (Marr, 2010), resulting in inefficiency when naïvely pro- cessing both modalities. To address this, existing methods extract more compact image representations by modifying the image encoder projector (Alayrac al., 2022; Li et al., 2024b; Dai et al., 2023; Cha et al., 2024). While some recent works further sparsify visual tokens during the decod- ing (Ye et al., 2025; Chen et al., 2024a; Shang et al., 2024), they still ignore the guidance from the language tokens, which contradicts the multimodality paradigm. We argue that visual tokens should be sparsified adaptively based the question prompt, the model might focus dif- ferent parts (e.g., foreground or background) when dealing with various questions as shown in Figure 1. Furthermore, current approaches generally train a network to prune redun- dant visual tokens and require additional training data (Li et al., 2024b; Ye et al., 2025; Cai et al., 2025).
1. Introduction
Benefiting from advancements in large language models (LLMs) (Radford et al., 2019; Brown et al., 2020; Touvron et al., 2023; Peng et al., 2023; Zhang et al., 2024a), the realm of vision-language models (VLMs) has undergone
In this paper, we introduce text-guided training-free frame- work dubbed SparseVLM for efficient vision language model inference. We reuse the self-attention matrix of visual-text tokens directly from the decoder layers with- out extra training parameters for sparsification. We ascer- tain that not all prompt tokens should be considered as some could be less relevant, which leads inaccurate cor- relation results and downgrades the performance of sparse
Equaa contribution 'State Key Laboratory of Multimedia Infor- mation Processing, School of Computer Science, Peking Univer- sity FFudan University 'EECS, UC Berkeley 'Shanghai Jiao Tong University Panasonicc Holdings Corporation. Correspondence to: Wenzhao Zheng <wzzheng @ bed Shanghang Zhang <shanghang pku.edu.cn>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Q-I: is written on this blue sign?
Q-2: How many buses are there in the image?
Q:3: What color is this roof?
YOAM IT
REPERBO
CAR
(b) Ours. Text-guided Visual Sparsification
(a) Input Image
(c) Existing Methods. Text-agnostic Visual Sparsification
Figure 1. Comparison of visual token sparsification methods. Unlike previous methods with text-agnostic visual sparsification (c) e.g.. LacoLLaMA (Ye et al., 2025), our Sparse VLM (b) is guided by question prompts to select relevant visual patches from the image (a).
inference. Specifically, our Sparse VLM first identifies text tokens strongly correlated with visual signals via cross- attention. Then, we measure the contribution of visual to- kens to the selected visual-relevant text tokens (i.e., "raters") and adaptively prune the insignificant visual tokens. Instead of directly discarding the pruned tokens, we further recy- cle and cluster them to reconstruct more compact tokens to minimize the loss of information. Due the information density varying for different image inputs, employ the rank of the attention matrix to indicate the redundancy level and set an adaptive sparsification ratio accordingly.
We introduce novel sparsification framework dubbed SparseVLM. To the best of our knowledge, it is the first training-free approach that explores text-aware guidance for efficient VLM inference.
Particularly, we propose a strategy to select relevant text tokens as raters of visual tokens, a method to assess the significance of visual tokens followed by pruning of redundant visual tokens with recycling mechanism to minimize the loss of information.
When applied to a number of VLMs, Sparse con- sistently outperforms prior state-of-the -hettrr methods in various image and video understanding benchmarks.
The proposed method is simple yet practical. can act as a plug-and-plaa module to improve the efficiency of VLMs without additional fine-tuning. Extensive experi- ments demonstrate that our Sparse VLM effectively reduces computational overhead of various VLMs without sacrific- ing their performance in a wide range of image and video understanding tasks. For instance, VA (Liu et al., 2024b) when armed with Sparse achieves a 4.5x compression rate while maintaining 97% of its original performance. Al- ternatively, the CUDA latency can decrease by 37% with only 0.9% drop in accuracy. To investigate the effec- tiveness of our method in video tasks, we further apply VL. to Videol.LaVA (Lin et al., 2024) to compress frames with temporal dimension. Without complex design changes, VLM can sparsify video frames into an adaptive number of visual tokens and outperform existing methods in video question-answering benchmarks. Our ap- proach consistently outperforms prior state-of-the-art F method (Chen et al., 2024a) by 11.2 17.3% on LLa 9.2 20.4% on MiniGemini, and 14.7% on VideoL.LaVA when both have similar latencies.
2. Related Work
Vision-Language Models. Recent works on vision- language models (Liu et al., 2024a; Chen et al., 2024b; Li et al., 2024c) improve multimodal comprehension and gen- eration by processing longer visual token sequences, More- over, the usage of higher-resolution images inevitably entails exponential growth in the length of visual sequences. For example, LLaVA typically encodes 336 x 336 images into 576 tokens (Liu et al., 2024b) with up to 672 x 672 max- imum resolution using 2880 token sequences (Liu et al., 2024a). Similarly, mini-Gemini-Hll (Li al., 2024c) con- verts 1536 x 1536 high resolution and 672 672 low reso- lution images into 2880 visual tokens. Moreover, compre- hending videos multiple images leads increased token allocations for visual signals. For instance, the Lide (Lin et al., 2024) and VideoPoet (Kondratyuk et al., 2024) use thousands tokens to encode multiple image frames. However, large number of visual tokens results in a com- putational bottleneck. Further research on sparsification
Our main contributions are summarized as follows:
2

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
urged to further unleash VLM capabilities.
sparsification. Please refer to the Appendix B.
Visual Compression for VLMs. Compression of visual tokens is necessary because, on the one hand, their quantity is usually tens to hundreds of times that of language to- kens. On the other hand, visual signals are inherently more sparse in information when compared to texts that have been produced by humans (Marr, 2010). Past efforts to address the above problem can be categorized into two directions. The first one centers the compression of vision tower or an efficient projection of vision modality. For instance, LLaMA-VID VID (Li et al., 2024b) exploits the Q-Former with the context token while DeCo (Yao et al., 2024) employs an adaptive pooling to downsample the visual tokens at the patch level. Methods that belong to the second direction (Ye al., 2025; Chen al., 2024a; Wu al., 2024) go deeper into the text modality and sparsify visual tokens during the LLM decoding stage, but they still lack guidance from the text tokens. In this paper, SLMrseVLM takes note of this limitation and improves performance upon it.
3.2. Sparsification Guidance from Text to Vision
Estimation of Visual Token Significance. For a multi- modal model, we aim to estimate an impact of deleting a single token from one modality to other modalities. In the VLM case, need to quantify how relevant visual to- ken is to text tokens in order to determine whether it can be pruned. Therefore, we naturally reuse the self-attention logits from VLM's transformer layers as a reference since they already contain language-to-vision query results.
In partieular, we take the interaction between the query- dimensional part of the language modality and the key- dimensional part of the vision modality as the basis for sparsification priority matrix P E R where L, and Ly are the lengths of text and visual tokens, defined by
P= = A.j. and (i.5) E [I,
(2)
where L and I denote the language instruction and image token sets, respectively.
3. Method
In this section, we present our SparseVLM for efficient VLM inference. We first review the attention mechanism in VLMs and then introduce the detailed strategies for our visual sparsification including visual significance estima- tion, relevant text token selection, and sparsification level adaptation. We further propose token recycling to reduce information loss and provide a theoretical analysis of com- putation savings. The pipeline is shown in Figure 2.
Next, we obtain a vector p that estimates the significance of all visual tokens w.r.t. the text dimension as
i = Lpi P2, ........ pr.. = 1.
Σ f T 1=1 P.
(3)
where we use an indicator for sparsification and a larger value in p means higher significance of the corresponding visual token. Calculation of (3) costs L, x L, FLOPs only while the access to already computed A is considered as free, which is highlights low complexity of the Sparse V..
3.1. Preliminary: Attention in VLM Decoders
VLM decoders typically rely on the causal self-attention from the original transformer architecture (Vaswani al., 2017) for token interactions. Without loss of generality, we describe the single-head attention below. Formally, the self- attention matrix with logits A E RRxI, where L denotes the length of a sequence with all kinds of tokens e.g. text and visual, is computed by
Relevant Text Token Selection. It not appropriate to use all text tokens as reference for visual sparsification. Figure 3 shows four representative cases where we compute the correlation between the prompt and the image. Case 3 highlights Tylenol, Advil, ibuprofen, while st sticker, fridge in case 4 are significant, where a large proportion of question tokens in light red include little visual relevance. Therefore, it is unreasonable to make in- significant text tokens to rate visual tokens, and need to select relevant text tokens (i.e.. "raters") for guidance.
QKT VD
T
A= = Attention () = Softmax
(1)
where the scalar D represents the matrix dimension, and the Q E R4xD and K E LxD are the query and key matrices, respectively. The keys and queries in a self-attention layer are computed in parallel by using multi-layer perceptrons to transform the input hidden states H into common space, where aligned interactions between modalities occur.
Specifically, for an input image xe the vision embedding tokens H. can be computed as
H W2.
(4)
where Z, is the visual feature provided by visual encoder Z, g(x.). and W is the projection matrix to convert Z, into vision embedding tokens H., For the language instruction xG,, it is transformed into text embedding tokens H, through the tokenizer. The above tokens both have the same dimensionality as the word embedding space. Then,
Often, the matrix A cannot be directly accessed due to FlashAttention-tyye (Dao et al., 2022) optimizations. There- fore, we develop an approach to extract A while maintaining compatibility with the Attentinn when applying our
3

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Vision Token
Q: What is the brand of this camera? Answer the question using a single word or a phase.
Sparse Attention
Text
X
4
8
X
Deleted Pools
N
Query
Tokenize
2
3
6
Text Tokens
Attention Matrix
X
2
X
5
Recycle
N
4
1
8
2
2
Select Raters
Image
5
2
x
4
5
Rank(P)- [2]
Mean
Select
N D[5] Rank(P)[3]
Relation Level
Redundancy Level
4
3
2
5
Results
(a) Before LLM
(b) During LLM
Figure 2. The architecture of VLL.seVLM. In stage (a), text raters pre-selected before entering the sparsification LLM. In stage (b), adaptive sparsification is performed on LLM layers, involving computing redundancy and the recycling of reconstructed tokens.
we start to recognize which characters in the prompt are visually relevant and assign them the role of raters, which can be formulated as
information. To efficiently preserve more visual details with fewer tokens, we propose a token recycling strategy to aggregate and reconstruct tokens to be pruned.
(ilm m), i Le).
(5) Token Aggregation. We first recycle the pruned visual tokens h2 with the top-r (%) highest values in P from the deleted pool. Then, we group h, tokens with k-nearest neighbor density peak aggregation algorithm (Rodriguez, 2014) for adaptive token aggregation.
Tu L
r
=
Softmax (H,H,")
(6)
where m = mr) and only candidates that exceed the m threshold become raters. The strategy s contains the indices of selected raters from the candidate list L, tokens. The (6) costs L, L. x 2D FLOPs that only computed once before the decoder layer processing.
In particular, we first compute the local density P of the ith token of total T N recycled tokens according to its k-nearest neighbors KC(Rt))
P: = exp (
(8)
Sparsification Level Adaptation. Having obtained the to- ken significance, we further propose rank-based strategy to adaptively determine the level of vision sparsification at each decoder layer. Considering that a full-rank matrix im- plies that all its rows or columns are linearly independent, we use the rank of P demonstrate the redundancy of the visual tokens. We argue that the difference between the dimension and rank of P reflect its redundancy and utilize scaling factor \ to determine the number of deletions as
Then, we compute the minimum distance between the re- cycled token hú and any other token with higher density (denoted as the distance indicator 8,) that defined by
min
h h if s.t. Pj Pi. h. h
8,
=
max
otherwise
(9)
We use x 6; to indicate the score of each token, where the tokens with higher scores are likely to be cluster centers. Other tokens are then assigned to the nearest cluster center via cosine similarity. The FLOPs cost in this stage is L, x (3 - 1) x + I.,, where I,, = T x N is the length of recycled tokens, C centers, and and 0 are hyperparameters.
N A x (L, - rank rank(P)).
(7)
We then remove N visual tokens with the smallest values in P. Notably, if the result of N in a decoder layer is O, we skip the layer without sparsification. This stage requires L, x Ly x min(Le L.) FLOPs for rank computation.
L, is the number of cluster
Token Reconstruction. Having performed token aggrega- tion, the recycled tokens with similar semantics are classi- fied into the same group. Then, the tokens E R x in the kth group are reconstructed into a new compressed token TІ via the element- tiiee sum operation as
3.3. Visual Token Recycling
We progressively sparsify visual tokens in each layer in the decoder, which results in more discarded tokens at later stages. Despite being less significant, the pruned visual tokens with relatively large values in still contain certain
No. Tí k (1. C},
Tg
(10)

---

VparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Question Prompt (from MME)
Question Prompt (from VQAv2)
this a picture of California Historical Landmarks in San Mateo County, California? Answer the question using a single word phase.
2
Do the walls have the same pattem? Answer the question using single word or phase.
Question Prompt (from Vizwiz)
Question Prompt (from Textvqa)
3
Can you tell what type of pill these are. If its Tylenol, Advil, or ibuprofen, something like that. Answer the question using a single word or phase.
4
What is written the top of the yellow sticker on the fridge? Answer the question using a single word or phase.
The level of text related to the image.
Figure 3. Sample prompts from four representative multimodal benchmarks. The darker the word, the greater its relationship to the image and more valuable is for reference. We see that some words are irrelevant to the vision domain (e-g., prepositions and pronouns) and should not be considered for visual sparsification. It is best viewed in color.
where N& is the token number of the kth group and the operation costs D (I,, C)) FLOPs.
including GQA (Hudson & Manning, 2019), MMBench (MMB) (Liu al., 2024c), MME (Fu al., 2023), POPE (Li et al., 2023b), SQA (Lu al., 2022), SEED-Bench (SEED) (Li et al., 2024a), VQA Tex (TextVQA) (Singh et al., 2019), and MMVet (Yu et al., 2024).
3.4. Theoretical Analysis of Computational Complexity
We consider the computation of multi-heas attention and feed-forward network (FFN) modules in the FLOPs estima- tion. Assuming N is the number of pruned tokens, D is the hidden state size, which is the same as the intermediate size in FFN, the FLOPs for one Transformer layer can be reduced by 6(N D2D2 + 2(N - C) Besides, our partial step introduces minimal computation with the de- tails provided in Appendix C. Thus, we estimate the FLOPs savings as the reduction part minus the additional overhead:
Implementation Details. We verify Sparse on three VLM frameworks: LLaVA (Liu et al., 2024b), Mini-Gemini (MGM) (Li al., 2024c), and Qwen2-VL (Bai et al., 2023). LLa employs CLIP-pretrained ViT-L as the visual tower, MGM further introduces LLAIONN-retraiin ConvNeXt-L (Liu et al., 2022) for high-resolution refine- ment, while Qwen2-VL owns dynamic resolution encoder.
Main Results. In Table 1, we present the performance of SparselLLaA (LLavV equipped with SparseVLM) on im- age understanding benchmarks. To intuitively assess the performance, we provide the results by percentage format for comparative analysis, and the accuracy of the vanilla model with the 100% upper limit. We set 3 vision token count configurations (192, 128, and 64) to check the ad- vantages of Sparse comprehensively. When pruning from 576 to 192 tokens, the Sparsel only decreases the average accuracy by 0.9% without additional training and exceeds ToMe (Bolya et al., 2023) 10.2%. When only 64 tokens are kept, our method outperforms FastV (Chen et al., 2024a) by a significant margin of 17.3%, while ToMe performs worst due to its direct merging. Furthermore, we also compare the recent method PDrop (Xing et al., 2025) training-free free version, which has lower FLOPs computation. However, our method outperforms in accuracy and latency, which are the most crucial metrics for practical deployment.
reduction part
21, I. D- E, (1+mini (2 (61, +2
overhead part
2L,,,D4Y, D6DDN -o-------------0- -())))))))))))))))))))))))))-))) 1
~ AL,,, DN,(6))222))- L.
(11)
where: 21, 2) and the number of total layers,
and x= = x is a very small decimal that can be ignored.
4. Experiments
In this section, we validate our method within various vision- language architectures on comprehensive multimodal bench- marks, including image and video understanding tasks, to assess its generality, effectiveness, and efficiency.
Figure 4 visualizes the performance of SparseMGM on POPE, TextVQA, and GQA. We find that our framework has an obvious advantage over F and ToMe. With the
4.1. Image Understanding Tasks
Datasets. For mmage-baser multimodal evaluation, we con- duct experiments on eight widely adopted benchmarks,
5

---

VparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Table Performance of Sparsel.Lar LaVA under different vision token configurations. The vanilla number vision tokens is 576. The first line of each methad is the raw accuracy of benchmarks, and the second line is the proportion relative to the upper limit.
Method
GQA MMB MME POPE
SQA
SEED VQA lew
MMVet
Acc. (%)
FLOPs (T)
Latency (ms)
Upper Bound, 576 Tokens (100%)
61.9
64.6
1864
85.9
69.5
60.3
58.3
30.9
Vanilla
100
4.62
57.82
100%
100%
100%
100%
100%
100%
100%
100%
Retain
192 Tokens (1 66.7%)
54.3
60.5
1563
72.4
65,2
53.1
52.1
27.9
ToMe (1CLR23)
88.9 (4 11.1)
2.05
34.06
87.7%
93.5%
83.9%
84.3%
93.8%
88.1 1%
89.5%
90.3%
52.6
61.0
1605
64.8
69.1
52.1
52.5
26.7
FastV (BCOV24)
87.9 (4 12.1)
2.11
34.87
85.0%
94.4%
86.1%
75.4%
99.4%
86.4%
90.1
86.4%
57.1
1766
82.3
70.2
54.7
56.1
PDrop (OVER25)
95.9 (4 4.1)
2.03
36.74
92.2%
97.8%
94.7%
95.8%
101.0%
90.7%
96.2%
98.7%
59.5
64.1
1787
85.3
68.7
58.7
57.8
33.1
Sparse VLM
99.1 0.9)
2.14
36.50
96.1%
99.2%
95.9%
99.3%
98.8%
97.3%
99.1%
107.1%
Retain 128 Tokens (J 77 (7.
52.4
53.3
1343
62.8
59.6
50.9
49.1
27.2
ToMe (ICGR23)
81.9 (4 18.1)
30.00
84.7%
82.4%
72.1%
73.1%
85.8%
84.4%
84.4%
88.0%
FastV (ECCV24)
49.6
56.1
1490
53.4
68.6
48.1
50.5
26.3
82.4 (4 17.6)
1.70
30.70
80.1%
86.8%
79.9%
62.2%
98.7%
79.8%
86.6%
85.1%
56.0
1664
82.3
69.9
53.3
55.1
PDrop (CVER25)
94.3 (4 5.7)
1.62
37.77
90.5%
95.4%
89.3%
95.8%
100.6%
88.4%
94.5%
99.7%
58.4
64.5
1746
85.0
68.6
58.2
56.7
29.0
VLM
96.7 (4 3.3)
1.72
33.28
94.3%
99.8%
93.7%
99.0%
98.7%
96.5%
97.3%
93.9%
Retain 64 Tokens
(4 88.9%)
48.6
43.7
1138
52.5
50.0
44.0
45.3
24.1
ToMe (ICLR23)
71.1 (4 28.9)
1.19
26.52
78.5%
67.5%
61.1%
61.1%
71.9%
73.0%
77.8%
78.0%
46.1
47.2
1255
38.2
68.7
43.7
47.8
19.6
FastV (ECCV24)
72.0 (L 28.0)
1.29
27.30
74.5%
73.1%
67.3%
44.5%
98.8%
72.5%
82.0%
63.4%
41.9
1092
55.9
69.2
45.9
PDrop (CVER25)
73.4 (4 26.6)
1.18
43.41
67.7%
51.6%
58.6%
65.1%
99.6%
66.3%
78.7%
99.4%
53.8
60.1
1589
77.5
69.8
52.2
53.4
24.9
SLM
89.3 (L 10.7)
1.30
29.89
86.9%
93.0%
85.2%
90.2%
100.4%
86.6%
91.6%
80.6%
Table Performance SSparseeLL on Qwen2-VL.
We further investigate efficacy Qwen2-VL. In Table
2, when 54.5% of vision tokens are removed, Qwen2-VL maintains an accuracy of 98.0%. Furthermore, for every 100 tokens pruned, the accuracy only drops by approximately 0.8%. This validates the effectiveness of our method at high resolutions and its compatibility with variable resolutions.
Tokens
MMB
POPE
VQA Text
Avg.
Dynamic
80.5 (1323)
86.4 (1311)
84.3 (1326)
83.7
600
79.6
86.5
80.3
82.1
500
78.8
86.3
79.0
81.4
4.2. Video Understanding Tasks
400
79.0
85.8
77.1
80.7
Datasets. We test on four common video question answer- ing benchmarks, TGIF-QA (Jang et al., 2017), MSVD-QA (Xu al., 2017), MSRVTT-QA (Xu et al., 2017), and ActivityNet-QA (Yu et al., 2019). Specifically, following FastV's (Chen et al., 2024a) setup, we use the first 1000 samples per benchmark and score them using the Video-
reduction of tokens, the gap between FastV and Sparse is increasing sharply. The reason is that, compared to FastV and ToMe, the text-aware strategy enables us to accurately locate visual tokens with more details, while the recycling of pruned tokens further reduces information loss.
6

---

VparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
POPE
TextVQA
GQA
Baseline
Baseline
Baseline
80
60
60
Accuracy (%)
Accuracy (%)
Accuracy (%)
60
ToMe
50
ToMe
ToMe
40
FastV
FastV
50
FastV
Ours
Ours
Ours
45
20
Baseline
40
Baseline
Baseline
Tokens
500
400
300
200
100
500
400
300
200
100
500
400
300
200
100
Figure 4. Performance of MGM w/ Sparse on three multimodal benchmarks. The horizontal axis represents the remaining
number of vision tokens, while the vertical axis means the accuracy after percentage normalization.
Table The results of Video-LLaVA with Sparse on video question answering task. The original number of video tokens 2048, while our experiment collectively prunes it down to 194 tokens. FastV (Chen et al., 2024a) is included for comparison.
Text VQA (Acc.)
POPE (Acc.)
53.8
80.0
53.6
53.4
77.5
53.4
TGIF
MSVD
MIRVTTT
AetivityNee
Avg.
53.2
75.0
Mahhod
33,0
Scure
Acc.
Acc.
Score
Acc.
Score
Scure
53.0
725
18.9
2.54
72.0
3.95
57.1
3.45
43.6
3.81
52.8
700
Video Video-LLaVA LLaVA
47.9
3.44
52
1008
-0.00
1008
+0.00
100%
40.00
100%
40.00
52.6
67.5
10.2
2.29
58.3
3.62
52.3
3.42
41.3
3.76
52.4
664
RRNY (
3.27
65.0
54.0%
-0.34
$1.0%
-0.53
91.698
-0.03
94.7%
-1.12
52.
14.9
2,41
71.7
3.94
56.1
3.45
45.1
3.11
52.0
62.5
Ouis
3
All Tokens
All Text
Text Raters
All Tokens
All Text
Text Raters
78.8%
-0.13
99.65
-0.01
98.3%
-0.02
103.4%
-0.00
Figure 5. The ablation study of text raters on LLaVA 7B.
ChatGPT (Maaz et al., 2024) evaluation tool, acknowledg- ing the characteristic length imbalances in these datasets.
5. Analysis
5.1. Relevant Text Token Selection
Implementation Details. We directly apply our Sparse- VLM for Video-LLaVA (Lin et al., 2024), which is com- posed of several key components, including language bind encoder fïg (Zhu et al., 2024a) for extracting features from raw visual inputs (c.g., images videos), a language de- coder model ft such as Vicuna (Touvron et al., 2023), visual projection layer fP, and word embedding layer fr.
We propose selection mechanism to localize visually irrel- evant text tokens to limit their negative effects in rating the significance of vision tokens. Here we conduct experiments to analyze the effects of the mechanism in Figure 5. Under the same number of vision tokens (64), we have 3 settings (using all tokens, only text tokens, and only text raters select) with LLaVA (Liu et al., 2024a) to judge vision token candidates. In VexiVQA (Singh et al., 2019), by building upon the text-aware manner, our mechanism improves the baseline (all tokens) by 0.8%, which validates that our extra selection effective. Besides, we further outperform the vanilla text-aware method (only text tokens) by 2.7% on POPE (Li et al., 2023b). The huge margin means POPE sparsification is quite sensitive to question prompts, and text guidance is necessary. In summary, text rater selection is general and improves the performance across scenarios.
Main Results. In Table 3, we set the Video-LLaVA with 2048 video tokens as our upper bound for an overall average accuracy of 100.0% and a score of +0.00. To make a fair comparison, we both preserve 194 vision tokens (90.5% pruning ratio) for FastV (Chen et al., 2024a) and Sparse- VLM. It is clear that our approach consistently outperforms Fast'V across all benchmarks, both in accuracy (Acc.) and GPT evaluation score. Sparse VideoLLaV achieves a total average accuracy of 95.0%, a significant 14.7% % higher than 80.3% of FastV. (From the GPT score perspective, Spar- seVLM only loses 0.04 points compared to 0.17 points of FastV.) These improvements suggest that when handling video modality containing temporal features, Sparse VI.M continues to deliver strong performance, generating accurate responses to diverse questions while utilizing significantly fewer tokens. This achieves an effective trade-off between inference efficiency and model performance.
5.2. Recycling of Pruned Tokens
To validate the effectiveness of our token recycling strategy, we perform ablation experiments on the LLaVA model (Liu et al., 2024a). The results are presented in Table 4. Across multiple sparsity ratios (64, 96, 128, 192), our algorithm achieves a significant average performance improvement of
7

---

Sparse VLM: Visual Token Sparsification for Efficient Vision- Vision-Language Model Inference
DRUPALCON
What does the small white text spell?
Copenhagen.
What number on the player's jersey?
22.
Is this denny's?
Denay's
Yes.
Figure Visualization SL on different VQA prompts. From left to right, the visual representation becomes increasingly sparse, leaving fewer vision tokens. Best viewed in color.
67% cache memory compared to vanilla LLaVA (where 302.4MB is reduced to 100.8MB), while keeping 99.1%
Table 4. Ablation study on token reconstruction (TR). Experi-
ments are conducted GQA and POPE on LLaVA 7B.
accuracy. More efficiency visualization (e.g., efficiency on VidcoLLaVA) can be found the Appendix
Tokens
Benchmark
Avg.
64
96
128
192
5.4. Qualitative Visualization
GQA
52.2
55.2
58.1
59.4
56.2
As shown Figure 6, we visualize SparseVLM on various VQA questions. From left to right, we visualize the results after we apply token pruning different layers. As the number of layers increases, more tokens are pruned and the Region of Interest (ROI) is gradually refined. The model sys- tematically reduces less relevant image information while retaining key tokens closely tied to the question. The visual- ization reveals that SparseVLM, although discarding some overall image details, effectively retains essential visual tokens. These preserved tokens encapsulate the features necessary for answering the question, focusing more relevant visual regions through their interaction with the question. More cases are in the Appendix H.
TR
53.8
56.4
58.4
59.5
57.0
POPE
72.8
77.5
83.7
85.2
79.8
+ TR
77.5
81.9
85.0
85.3
82.4
1.2% and 7.2% on TextVQA (Singh al., 2019) and POPE (Li et al., 2023b), respectively. Notably, as the number of pruned vision tokens increases, the benefit brought by our recycling method increases. For instance, when pruning from 192 to 64 tokens, the pruned token recycling signifi- cantly boosts the accuracy from 1.5% to 17.7% on POPE. We argue that when the size of the deleted pool grows, the amount of lost information increases. Our method effec- tively recycles the lost information and compresses it into few slots using the proposed reconstruction mechanism.
6. Conclusion
This paper introduced text-aware training- -fee token opti- mization approach called Sparse VLM which significantly decreased the test-time computations of various VLMs. Un- like prior methods, Sparse VLM optimized VLMs without introducing extra parameters and fine-tuning costs. We achieved a more compact visual representation by employ- ing the rank of attention matrices to determine pruning ratios and by recycling the pruned tokens via the reconstruction mechanism reduce the information loss. Experiments demonstrated that egg. the LLaVA when equipped with Spar- VLM achieved 37. 700% reduction in latency with a compres- sion ratio of 77.8% while maintaining 97% of the original accuracy. Moreover, our method exceeded FastV accuracy by 14.7% in video understanding tasks. Our Sparse can provide practical benefits for deploying off-the-shelf VLMs on edge devices and in the cloud setting.
5.3. Computational Efficiency
SparseVLM affords significant efficiency and storage gains for the inference process. We conduct comparative analy- sis of CUDA time, and FLOPs on VA-VA- and compare our method with the baseline method and FastV (Chen et al., 2024a). As displayed Table 1, we conduct an inference efficiency analysis on single NVIDIA A100-80GB with identical lengths of text prompts and single-image inputs. Compared to the baseline model, Sparse VLM achieves significant reduction of 43.1% in CUDA time and 62.8% in FLOPs while keeping 96.7% accuracy. Despite SparseVLM has a minimal overhead to calculate text raters and cluster- pruned vision tokens, it leads to fewer than FastV tokens with comparable accuracy. Additionally, Sparse saves
8

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Acknowledgments
generic visual-linguistic tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, 2024b.
This work was supported by the National Science and Tech- nology Major Project (No. 2022ZD01 17800) and by the National Natural Science Foundation of China under Grant 62472008.
Dai, W. Li, J., Li, D., Tiong, A., Zhao, J., Wang, W. Li, B., Fung, P., and Hoi, S. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Sys- tems, 2023.
Impact Statement
Our SparseVLM provides practical advantages for deploy- ing off-the-shelf large vision-language models on edge de- vices and cloud platforms. While our work does not present any evident societal implications, believe it is unneces- sary to emphasize this aspect in the current context.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. FlashAt- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 2022.
Du, Z.. Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang. J. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics, 2022.
References
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 2022.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang. J., Zheng. X., Li, K., Sun, X., al. MME: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv:2306. 13394, 2023.
Bai, J., Bai, S.. Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-VL: VL: A fron- tier large vision-language model with versatile abilities. arXiv:2308. 22338.129660 12966, 2023.
Goyal, Y., Khot, T., Summers- Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C.,, and Hoffman, J. Token merging: Your vit but faster. In International Conference on Learning Representations, 2023.
Hudson, D. A. and Manning, C. D. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learn- ers. Advances in Neural Information Processing Systems, 2020.
Jang, Y., Song, Y., Yu, Y., Kim, Y., and Kim, G. Tgif- qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Cai, M., Yang. J.. Gao, J., and Lee, J. Matryoshka multi- modal models. In International Conference on Learning Representations, 2025.
Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birodkar, V., et al. Videopoet: A large language model for zero- shot video generation. In International Conference on Machine Learning, 2024.
Cha, J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality- enhanced projector for multimodal 11m. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024.
Chen, L.,, Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In Proceedings of the European Con- ference Computer Vision, 2024a.
Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., and Shan, Y. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2024a.
Li, J., Li, D.,, Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-imen pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023a.
Chen, Z., WU, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023b.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arXiv:2304.03277, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models unsupervised multitask learners. OpenAl blog. 2019.
Li, Y., Wang, C., and Jia, J. ILLMA-VID: An image is worth tokens in large language models. In Proceedings of the IEEE Conference Computer Viston and Pattern Recognition, 2024b.
Rodriguez, A. Clustering by fast search and find of density peaks. Science, 2014.
Shang, Y., Cai, M., Xu, B., Lee, Y. J., and Yan, Y. Llava- prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 15388, 2024.
Li, Y., Zhang, Y., Wang, C., Zhong, 7., Chen, Y., Chu, R., Liu, S., and Jia, Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403. 18814, 2024c.
Singh, A., Natarjan, V., Shah, M., Jiang. Y.. Chen, X., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Lin, B., Ye. Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-Ilava Learning united visual representation by alignment before projection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2024.
Stewart, G. W. On the early history of the singular value decomposition. SIAM review, 1993.
Liu, H., Li, C.,, Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning, In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, 2024a.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu,
J.. Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: family of highly capable multimodal models. 2. 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y.J. Visual instruction tun- ing. Advances in Neural Information Processing Systems, 2024b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv:2302. 13971, 2023.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z. et al. Mmbench: Is your multi-modal model an all-around player? In Proceedings of the European Conference on Computer Vision, 2024c.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you need. Advances in Neural Information Processing Systems, 2017.
Liu, Z.. Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Wu, S., Chen, J., Lin, K. Q., Wang, Q., Gao, Y., Xu, Q., Xu, T., Hu, Y., Chen, E., and Shou, M. Z. Videollm-mods Ef- ficient video-language streaming with mixture-of-depths vision computation. Advances in Neural Information Processing Systems, 2024.
Lu, P., Mishra, S., Xia, T., Qiu, L.. Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 2022.
Xing, I.,, Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., Cao, Y., He, C., Wang. J., Wu, F., et al. Pyramiddrop: Ac- celerating your large vision-language models via pyramid visual redundancy reduction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. Vidco- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2024.
Xu, D., Zhao, Z., Xiao, J., Wu, F,, Zhang, H., He, X., and Zhuang, Y. Video question answering via gradually refined attention over appearance and motion. In Proceed- ings of the ACM international conference on Multimedia, 2017.
Marr, D. Vision: computational investigation into the hu- man representation and processing of visual information. MIT press, 2010.
10

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Yao, L., Li, L.,, Ren, S., Wang. L., Liu, Y., Sun, X., and Hou, I.. DeCo: Decoupling token compression from mantic abstraction in multimodal large language models. arXiv:2405.20985, 2024.
Ye, X., Gan, Y., Huang, X., Ge, Y., Shan, Y., and Tang, Y. VoCo-LLaMA: Towards vision compression with large language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025.
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang. L. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International Con- ference Machine Learning, 2024.
Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., and Tao, D. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
Zhang, Y., Huang, T., Fan, C.-K., Dong, H., Li, J., Wang, J.. Cheng, K., Zhang, S., Guo, H., et al. Unveiling the tapestry of consistency in large vision-language mod- els. Advances in Neural Information Processing Systems, 2024a.
Zhang, Y., Huang. T., Liu, J., Jiang, T., Cheng, K., and Zhang, S. Freekd: Knowledge distillation via semantic frequency prompt. In Proceedings the IEEE Confer- ence Computer Vision and Patlern Recognition, 2024b.
Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., HongFa, W., Pang, Y., Jiang, W., Zhang, J., Li, Z.. al. Languagebind: Extending video-language pretraining n-modality by language-based semantic alignment. In International Conference on Learning Representations, 2024a.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations, 2024b.

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
Appendix
A. The Redundancy of Visual Tokens in VLMs
In non-textual tasks, such as classification or detection, downsampling is commonly used reduce visual redundancy and enhance model training efficiency (Zhang et al., 2024b). Figure 7 illustrates this process, showing the reduction of tokens from 1166 to 576 in downsampled image, resulting in 50% efficiency boost but 15% information loss (entropy decreased from 7.44 to 6.13). This trade-off is acceptable for such tasks. Conversely, for text-related tasks like visual question answering (VQA), which involve both text and vision modalities, a distinct approach is required. Highlighting the most information-dense text (88% of total text) alongside the region pertinent to the query in the image (38% of total image), we observe that image information is typically sparser than textual data. Hence, our Sparse VLM method incrementally prunes visual token redundancy, maintaining crucial information for task accuracy. This strategy enhances model efficiency.
Visual Redundancy Challenges in VLMS
Our Method
sampled Image
Stage
Pruning
Original Image
Q: What is written the top of the yellow Sticker on the iridge?
30%
Stage Pruning
Output: Warning.
Down sampled Image
written on the top
yellow sticker fridge
10%
Token Number: 1166 576
Stage Pruning
Lffiiiincy: -50%
Entropy: 7.44 to 6.13
Information: 15%
Trade-off Acceptable
38%
88%
2%
Text Unrelated Tasks (e.g, Classification)
Text Related Tasks (e-gv VQA)
Prune Redundancy Progressively
Figure 7. Analysis of visual redundancy in different vision tasks.
B. Compatibility with FlashAttention
To ensure compatibility between Sparse VLM and FlashAttention (Dao al., 2022) when extracting the matrix A or P. devise the dual-flash attention operation directly obtain the average attention scores relative the text raters. This operation is lightweight and enjoys the efficiency of FlashAttention. Specifically, the first forward pass operates identically to the original FlashAttention, generating the necessary hidden states. In the second forward pass, we introduce a specially designed V matrix. In this matrix, for the rows corresponding to the text raters we wish to analyze, we set the values to the reciprocal of the number of text raters. This configuration allows the inner product between the attention map and the V matrix to return the mean value of the attention scores for the selected text raters directly in FlashAttention. With the mean value, perform top-k selection to identify the visual tokens retain. Tokens that excluded during this process converted into masks, which are then applied to the hidden states produced by the first FlashAttention pass to complete the pruning operation. This method enables efficient integration of pruning with FlashAttention while preserving compatibility and computational efficiency. The specific principles and calculation of Sparse VLM FlashAttention are as follows:
1. Attention Score Calculation. For each block B, compute the scaled dot- dot-product attention scores as
QBK7
SB
dx
where Sp is the attention score matrix computed within the block.
2. Block-wise Softmax. To ensure numerical stability, the Softmax is computed using the log-sum-exp trick as
12

---

VparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
(a) Subtract the maximum value for numerical stability:
BB SB m-aaax SB. axis 1)
(b) Normalize:
exp(S
PB
Σ exp( S axis =
3. Special V Matrix. In order to return the mean value of the attention scores for the selected text raters directly with the FlashAttention, we need to design special V matrix.
1/n,
if (11, (2,,,2,..... i
Vij-
0:
otherwise.
Here, V is an x d matrix, n is the total number of rows in the matrix, is the row index, 1 n. = fi m), E (1,2,..., [ define the text raters which we selected in Section 3.2.
4. Incremental Accumulation. Rather than storing P explicitly, the result is directly accumulated into the output using:
OB PB VB
The final result is obtained by concatenating all blocks:
0 = C-Cocat Oz.....0p OB)
5. Streaming Softmax. When combining multiple blocks, an incremental softmax computation ensures that normalization is maintained across the entire sequence:
Softmax =xp((S) Σ SSp
This avoids global dependencies and enables efficient block- wise computation.
6. Top-k Selection for Visual Tokens. The top-k selection be expressed as:
O₂ {xg O, rank(x;, O.) k);
O. {リj E mean (O) visual tokens start < S visual tokens end).
where O = Concat(O, O2, O....., Og) is the output array of the second FlashAttention, O is the visual tokens part of O, rank(r;, O..) represents the position of x; in O, when sorted in descending order.
The corresponding indices of the top-k elements are:
I (i 2 E0).
7. Summary of SparseVLM with FlashAttention using Top-k Selection. The complete process of Sparse VLM FlashAttention can summarized as
I = {i x4 E {yg O, Jrank(y mean
mean(Concat (Concat
U Softmax
QBKI Vdn
max(S())
VB
[visual tokens start : visual tokens end)))}-
Here, each block B is processed independently, and the results are combined using incremental normalization.
13

---

VparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
C. Computing Budget Detailed Estimation
Estimation of Visual Token Significance. this stage, only the equation 3 averaging process requires computation. Each visual token undergoes L. additions and one division. With L, visual tokens in total, the number of FLOPs for this stage is (L, 1++11 L L, x Ly.
Relevant Text Selection. In this process, given that official Py Torch implementation for Softmax and Averaging operations, the FL.Pss for equation 6 can be approximately simplified to the matrix multiplication between H, and H The result has a shape of Ly x Lt. where each element undergoes D multiplications and additions. Therefore, the FLOP count can be expressed as L x Ly x 2D.
Sparsification Level Adaptation. The rank of a matrix typically computed using singular value decomposition (SVD) (Stewart, 1993). With the selected appropriate threshold, the number of above the threshold singular values determines the rank of the matrix. The FLOPs involved in this process can be approximated as L, L, x min(L,, L,).
Token Aggregation. At this stage, the first part is to perform nearest neighbor search for each element in the matrix. With the L, x D matrix, this task can be simplified to calculate the distances between L, elements, resulting in a total of Lr x (Ir - 1 distance calculations. Each distance computation requires sequentially executing subtraction, squaring, addition, and square root operations on D elements. Consequently, the number of FLOPs in the nearest neighbor search is L, x (L, 1 x 4D L, x (I, 1) 2D.
The second part is density calculation. Since the operations of averaging and applying the exponential function implemented by the official PyTorch, this part can be simplified by the matrix squaring. Therefore, the FLOPs for this part are L, x L, x 2D.
The third part is distance indicator calculation. The computation can be approximately simplified to compute Pa x o,. Therefore, the FLOPs for this part can be approximated as L, x Lp x 2D.
The last part is clustering. In this part, we need select C' tokens with the highest scores from total of L, tokens to serve cluster centers, and the FLOOSs be approximated as L..
In summary, the total FLOPs for this stage are given by
FLOPs = L, x (L, - 1) 2D
L, X X
L, L 2D
+
Nearest Neighborr Seurch
Density Calculation
Distance Indicator Calculation
Select Cluster Center
L, x (3L, 1) 2D L.
Token Reconstruction. Token reconstruction involves performing weighted sum for each group, excluding the cluster center. Thus, there are L, - C elements to sum where each one has 1 x D dimensions. Consequently, the number of FLOPs for this operation is D x (Lr - C.).
D. Dataset
We conducted experiments several widely used visual understanding benchmarks.
GQA. (Hudson & Manning, 2019) The GQA is composed of three parts: scene graphs, questions, and images. The image part contains images, well as the spatial features of images and the features of all objects in images. The questions in GQA are designed to test the understanding of visual scenes and the ability to reason about different aspects of an image.
MMBench. (Liu et al., 2024c) The MMBench benchmark comprehensively evaluates the model's overall performance across multiple dimensions. It includes three levels of ability dimensions. The first level (L-1) consists of two main abilities, perception and reasoning. The second level (L-2) expands based on the first level. including six sub-abilities. The third level (L-3) further refines the second level, encompassing 20 specific ability dimensions. This hierarchical structure enables granular and comprehensive evaluation of the model's various capabilities.
MME. (Fu et al., 2023) The MME benchmark is also a comprehensive benchmark meticulously designed to thoroughly evaluate various aspects of a model's performance. consists of 14 subtasks that specifically aim to evaluate both the model's perceptual and cognitive abilities. By utilizing manually constructed instruction-anse answer pairs and concise instruction design, effectively mitigates issues such data leakage and unfair evaluation of model performance.
14

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
POPE. (Li al., 2023b) The POPE benchmark is primarily used to evaluate the degree of Object Hallucination in models. It reformulates hallucination evaluation by requiring the model to answer series of specific binary questions regarding the presence of objects in images. Accuracy, Recall, Precision, and F1 Score are effectively employed as reliable evaluation metries to precisely measure the model's hallucination level under three different sampling strategies.
ScienceQA. (Lu et al., 2022) The ScienceQA benchmark covers a rich diversity of domains, including natural science, language science, and social science. Within each subject, questions are categorized first by the topic, then by the category, and finally by the skill. This hierarchical categorization results in 26 topics, 127 categories, and 379 skills, providing a comprehensive and diverse range of scientific questions. It provides a comprehensive evaluation of model's capabilities in multimodal understanding, multi-step reasoning, and interpretability.
VQA-v2. (Goyal et al., 2017) The VQA-v2 benchmark evaluates the model's visual perception capabilities through open- ended questions. It consists of 265,016 images, covering a wide variety of real-world scenes and objects, providing rich visual contexts for the questions. For each question, there are 10 ground truth answers provided by human annotators, which allows for a comprehensive evaluation of the performance of different models answering the questions accurately.
TQA (Singh et al., 2019) The TextVQA benchmark focuses on the comprehensive integration of diverse text information within images. It meticulously evaluates the model's 's text understanding and reasoning abilities through series of visual question-answering tasks with rich textual information. Models need to not only understand the visual content of the images but also be able to read and reason about the text within the images to answer the questions accurately.
MMVet. (Yu et al., 2024) The MMVet benchmark is designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision- Lasionn- capabilities. Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination.
TGIF-QA.
(Jang et al., 2017) The TGIF-QA benchmark is an extension of the image question answering (ImageQA) task the video domain, aiming to promote the development of video question answering techniques. It contains 165,000 question answer pairs in total and requires the model to comprehend the details of GIF videos. Specifically, it introduces three new tasks for VideoQA (repetition count, repeating action, and state transition), which require spatio-temporal reasoning from videos, and frame QA tasks that can be answered from one of the frames.
MSVD-QA. (Xu et al., 2017) The MSVD-QA benchmark is based on the existing Microsoft Research Video Description (MSVD) dataset and contains 1970 video clips and approximately 50.5K QA pairs. The questions and answers are diverse in nature, covering wide range of topics and aspects related to the video content. Due to its relatively large data size and the diversity of questions, it is widely used for video question answering tasks and video caption tasks. The tasks formed in it are open-ended questions, consisting of five types of questions: what, who, how, when, and where.
MSRVTT-QA, (Xu et al., 2017) The MSRVTT-QA benchmark consists of 10K video clips and 243k question answer pairs. One of the main challenges addressed by the MSRVTT-QA benchmark is the complexity of understanding and reasoning about video content. Videos contain both visual and temporal information, and models need to be able to effectively process and integrate these aspects answer the questions accurately. The tasks formed in it also consist of five types of questions, similar to the MSVD-QA benchmark.
ActivityNet-QA -AA (Yu et al., 2019) The ActivityNet-QA benchmark contains 58,000 human-annotated QA pairs on 5,800 videos derived from the ActivityNet dataset. The questions are designed to cover a range of types, including motion, spatial relationship, and temporal relationship, which challenge the model to understand and reason about the video content at different levels and evaluate the performance of VideoQA models in long-term spatio-temporal reasoning.
E. Implementation Details
All of our experiments are conducted on single NVIDIA A100-80G GPU. The implementation is carried out in Python 3. 10, utilizing PyTorch 2.1.2, CUDA 11.8, and transformers 4.31.0. The inference follows the evaluation settings established by LLaVA (Liu et al., 2024b). For setting as the original paper as it is publicly available" 2.3, For video understanding tasks, we adopt the same inference setup
Mini-Gemini (MGM), and Qwen-VI. we follow the same inference
"github. hat.ubb.com/hattia---lliilu
"github. Phtpps: thub. wttp://cccooweeee//eeeeee
15

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
the original Video-LLaVA code base", it is publicly available.
F. Efficiency Details
We present a comparative efficiency analysis of SparseVLM, the bascline, and F during the inference phase in Table I. In this section, we provide additional details on the CUDA time during the inference phase. Following VoCo-LLaMA (Ye et al., 2025), we primarily consider the following components that contribute to the reported CUDA time: image encoding time (if applicable). KV cache load time (if applicable), and transformers forward time. We exclude other computational times that not dependent on the model itself and the caching strategy, such as model loading time, from the CUDA time measurement. Specifically, the attention operation is implemented by FlashAttention (Dao et al., 2022).
G. More Detailed Efficiency Analysis
To better validate the efficiency of our method, we provide the siteney-vs.-accurge and PLOPs-vs.-Acurrae trade-offs for Sparse applied to LLaVA and MGM across three benchmarks: POPE, TextVQA, and MME, which are shown in Figure and Figure 9. Besides, we also analyze Video-LLaVA matched with Sparse VLM in Figure 10 TGIF and MSVD.
POPE
TextVQA
MME
60
1900
baseline
1875
58
85
1850
Accuracy (%)
Accuracy (%)
56
Scores
1825
1800
83
1775
82
random sparse SparseVLM
random sparse
1750
---- random sparse
81
52
SparseVLM
1725
SparseVLM
baseline
baseline
baseline
80
50
1700
350
300
250
120
110
100
Cuda Times(s)
Cuda Times(s)
Cuda Times(s)
POPE
TextVQA
MME
60
1900
86
baseline
58
1850
85
Accuracy (%)
Accuracy (%)
56
1800
84
Scores
83
1750
82
random sparse
52
random sparse
1700
random sparse
81
SparseVLM
SparseVLM
SparseVLM
baseline
baseline
baseline
50
1650
80
2
2
TFLOPs
TFLOPs
TFLOPS
Figure 8. Trade-offs for Sparse LLLVA: (a) Latency vs. Accuracy, and (b) FLOPs Accuracy. Both studies demonstrate comparisons among random sparse, our SparseVLM, and baseline models.
H. More Sparsification Visualization
Figure 11 showcases diverse array of visualization examples that demonstrate the application of Sparse across spectrum of visual question-answering (VQA) prompts. These visualizations offer deeper insight into how our Sparse processes and responds to different types of queries posed in a visual context.
16

---

SparseVIM: Visual Token Sparsification for Efficient Vision-Language Model Inference
POPE
TextVQA
MME
1900
86
baseline
1880
85
64
Accuracy (%)
Accuracy (%6
1860
84
Scores
1840
83
1820
82
random sparse SparseVLM
58
random sparse SparseVLM
random sparse SparseVLM
81
1800
baseline
baseline
baseline
1780
400
350
300
300
250
200
120
100
80
Cuda Times(s)
Cuda Timesís)
Cuda Times(s)
POPE
TextVQA
MME
1900
86
65.0
1875
84
62.5
1850
Accuracy (%)
Accuracy
82
60.0
Scores
1825
80
57.5
1800
55.0
1775
78
random sparse SparseVLM
random sparse SparseVLM
random sparse
52.5
1750
SparseVLM
76
baseline
baseline
baseline
50.0
1725
3
4
3
TFLOPs
TELOPs
TFLOPs
Figure 9. Trade-off's performance for SLa on MGM: (a) Latency vs. Accuracy, and (b) FLOPs vs. Accuracy. Both studies demonstrate comparisons among random sparse, our VLM,,eVLM, and baseline models.
TGIF
MSVD
20
baseline
65
15
Accuracy (86)
60
10
55
Fasty
FastV
SparseVLM Full Sparseifictian ficitian Query
SparseVLM Full Sparscificttion Query
SparseVLM
SparseVLM
baseline
baseline
40
225
175
150
CUDA Times (5)
CUDA Times
TGIF
MSVD
baseline
Accuracy (86)
Accuracy 186)
60
20
55
SparseVLM Full Sparselfictton Query
SparseVLM Full Sparseificition Query
SparseVLM
SparseVLM
baseline
basell
200
400
600
800
1000
200
400
600
800
1000
Tokens budget (per)
Takens budget (per)
Figure 10. Trade-off' for Sparse LM on Video-LLaVA: (a) Latency vs. Accuracy, and (b) Token budget vs. Accuracy, Both studies demonstrate comparisons among random sparse, our Sparse and baseline models.
17

---

Sparse Visual Token Sparsification for Efficient Vision-Language Model Inference
Is there backpack image?
Yes.
Are shorts large and blue?
No.
Are there both toothbrushes and mats in this picture?
No.
Do the balls to the left of the other ball look right?
Yes.
Does the sweater look open and blue?
No.
Is there cake in the image?
Yes.
What color are the pants?
Red.
What appliance the refrigerator larger than?
Stove.
What kind of animal is beautiful?
Zebra.
Who is wearing the dress?
Woman.
Figure IJ. More visualization examples of Sparse on different prompts. Best viewed in color.
18


Reviews:
[{"review_id": "kTkSFAZvnq", "reviewer": "Reviewer_PtfS", "summary": "The paper introduces SparseVLM, which is a training-free technique to optimize number of visual tokens in a Vision-Language Model. The method consists of two main components, i.e. to identify relevant text tokens to rate visual tokens and method to prune and recycle visual tokens based on its significance. Furthermore, the authors explain how the method can be integrated with FlashAttention for efficient inference and present latency benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the **reviewer PtfS** for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> **1. The analysis of the efficiency of input image resizing and vision token pruning in VLM, which supports dynamic image resolutions.**\n\nFirstly, while dynamic resolution capability in VLMs requires additional training compared to fixed-resolution ones, our method is entirely training-free. We still conducted comparative experiments on Qwen2-VL to evaluate our effectiveness. The table below shows that SparseVLM incurs a comparatively minimal performance degradation compared with the resizing manner. Secondly, our method is orthogonal to dynamic resolution and can further reduce the number of tokens on resized images while maintaining nearly lossless accuracy.\n\n| Method | MMB | POPE | TextVQA | Avg. (Loss)\n|---|---|---|---|---|\n|**Qwen2-VL** | 80.5 (1323) | 86.4 (1311) | 84.3 (1326) | 83.7 (0)\n|**Retain 600 tokens** \n|Resize | 78.1 | 85.3 | 78.0 | 80.5 (-3.2)\n|SparseVLM | 79.6 | 86.5 | 80.3 | **82.1 (-1.6)**\n|**Retain 500 tokens** \n|Resize | 78.1 | 85.3 | 75.3 | 79.6 (-4.1)\n|SparseVLM | 78.8 | 86.3 | 79.0 | **81.4 (-2.3)**\n|**Retain 400 tokens** \n|Resize | 77.0 | 85.1 | 73.4 | 78.5 (-5.2)\n|SparseVLM | 79.0 | 85.8 | 77.1 | **80.7 (-3.0)**\n\n---\n\n> **2. Extending SparseVLM to Hierarchical Vision Encoders VLM architecture.**\n\nSince FastVLM has not yet open-sourced its weights and model architecture, we conducted experiments on ConvLLaVA-7B (256-token setting). The table below shows that our results own near-lossless accuracy ($0.6$%) on ConvNeXt vision encoders under the 192 tokens setting. Besides, under extreme compression (64 tokens setting), the performance of PDrop drops by $19.0$%, while SparseVLM only drops by $8.8$%.\n\n| Method | MMB| POPE | SEED | TextVQA | MMVet | MMMU | Avg. (Loss)\n|---|---|---|---|---|---|---|---|\n|**ConvLLaVA-7B** | 68.8 | 87.6 | 69.3 | 62.5 | 44.4 | 35.1 | 61.3 (0)\n|**Retain 192 Tokens**\n|PDrop [*CVPR 2025*]| 67.3 | 84.0 | 62.5 | 60.0 | 42.9 | 33.7 | 58.4 (-2.9)\n|SparseVLM | 68.0 | 87.0 | 68.7 | 62.0 | 44.5 | 34.5 | **60.7 (-0.6)**\n|**Retain 128 Tokens**\n|PDrop [*CVPR 2025*]| 65.6 | 83.9 | 61.0 | 59.2 | 42.5 |  30.2 | 57.1 (-4.2)\n|SparseVLM | 67.0 | 86.3 | 66.2 | 60.8 | 43.2 | 32.7 | **59.3 (-2.0)**\n|**Retain 64 Tokens**\n|PDrop [*CVPR 2025*]| 35.8 | 57.0 | 46.0 | 49.4 | 42.0 | 24.8 | 42.3 (-19.0)\n|SparseVLM | 63.2 | 76.1 | 59.7 | 54.4 | 35.0 | 26.6 | **52.5 (-8.8)**\n\n---\n\n> **3. Extending SparseVLM to Multiple Vision Encoders VLM architecture.**\n\nOur existing experiments (Figure 4) cover MiniGemini, a VLM architecture employing multiple vision encoders (CLIP and ConvNeXt). Following the reviewer's advice, we further conducted experiments on Cambrian-1 13B. Shown in the table below, our method is well-suited for multiple vision encoders. Under the 192-token setting, our method achieves a $2.3$% lower performance drop than PDrop; meanwhile, under the 64-token setting, our approach maintains an accuracy loss below $8.0$%.\n\n|Method|GQA|MMB|SQA|SEED|TextVQA|MMMU| Avg. (Loss) \n|---|---|---|---|---|---|---|---|\n|**Cambrian-1-13B**|64.0| 75.5 | 79.3 | 74.4 | 72.8 | 40.0 | 67.7 (0) \n|**Retain 192 Tokens**\n|PDrop [*CVPR 2025*]| 59.5 | 72.5 | 75.3 |68.4 | 70.0 | 38.0 | 63.9 (-3.8) \n|SparseVLM|61.6|74.2| 79.0 | 72.0 | 71.4 | 38.8 | **66.2 (-1.5)** \n|**Retain 128 Tokens**\n|PDrop [*CVPR 2025*]|56.8|70.9|74.0| 65.5 | 68.8 | 35.4 | 61.9 (-5.8) \n|SparseVLM | 60.4|73.0|78.2|70.6|70.2|38.0| **65.1 (-2.6)**\n|**Retain 64 Tokens**\n|PDrop [*CVPR 2025*]| 46.2 | 61.6 | 71.7 | 56.4 | 58.0 | 26.1 | 53.3 (-14.4) \n|SparseVLM | 54.4|68.7|78.0|64.0| 62.2|31.2|**59.8  (-7.9)** \n\nIn summary, the experiments on MiniGemini in the paper, along with the above Cambrian-1 and ConvLLaVA experiments, demonstrate that ***our method can match well with various visual encoder architectures***. The reason is that ***our method is applied to the decoder of VLM, where the visual features are pre-aligned by the projection layers, ensuring the robustness of selecting appropriate text raters***. \n\n---\n\n> **4. The potential implications of SparseVLM in multi-turn conversations.**\n\nCurrently, most vision token pruning methods (*e.g.*, FastV [*ECCV 2024*]  and PDrop [*CVPR 2025*]) fail to effectively maintain compatibility with multi-turn conversations. Here, we propose a potentially viable approach, where we can integrate DyCoke-like [*CVPR 2025*] dynamic pruning: ***(1) KV Cache Preservation***: Pruned tokens' KV Cache is stored via dynamic pruning (DP-KV Cache). ***(2) On-demand Updates***: New prompts trigger DP-KV Cache retrieval for retained tokens and updates for pruned ones. This ensures multi-round consistency by maintaining historical context while adapting to evolving queries.\n\n---\n*We sincerely appreciate your valuable review and suggestions. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.*", "claims_and_evidence": "The authors claim it is the first training-free approach that explores text guidance to prune visual tokens. After reviewing relevant literature, it might be accurate to claim that the proposed method is the first training-free approach to rely on explicit text-guidance, as opposed to implicit guidance used in methods like FastV. Claims on efficiency is backed by actual latency estimates, which adds credibility to the proposed method.", "methods_and_evaluation_criteria": "The proposed method evaluates the popular LLaVA-1.5 model on typical VLM benchmarks like GQA, SQA, MMB, POPE, SEEDBench, etc., which is common for most works in this domain. They also, show performance of SparseVLM applied to Qwen2VL, a more recent VLM which is trained to support dynamic image resolution.", "theoretical_claims": "The theoretical claims seem accurate, I do not find any major issue with the proposed method and its complexity analysis.", "experimental_designs_or_analyses": "1. Recent VLMs, like Qwen2-VL support dynamic image resolutions, an accurate baseline for Table 2 would be to simply resize the input image to support the same visual tokens as listed in rows 2, 3 and 4. \n2. The method also focuses primarily on vision encoders that are ViT-based, for completeness, analysis of the method applied to visual tokens from hierarchical backbones like ConvNeXt and FastViTHD used in ConvLLaVA[1] and FastVLM[2] respectively would be good to have. \n3. The method does not discuss any cases when multiple vision encoders with different pre-trainings are used in a VLM, for example Cambrian-1[3].   \n\n\n[1] - ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models\n[2] - FastVLM: Efficient Vision Encoding for Vision Language Models\n[3] - Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs", "relation_to_broader_scientific_literature": "The introduced method is highly relevant to the broader community that optimizes VLM inference. More recent VLMs like Qwen2-VL or InternVL2 support resolutions as high as 4-7MP, which can lead to significant increase in visual tokens. Training-free techniques like SparseVLM (introduced in paper), enables cost-effective optimization of these models for inference.", "essential_references_not_discussed": "I believe the paper discusses most of the relevant works that are published and not concurrent works.", "other_strengths_and_weaknesses": "One potential weakness of the method could arise in multi-turn QA setup. From an initial assessment, it seems like the prefilled KV cache may not be the most accurate to answer a follow-up question that may pertain to a different region of the image. Token recycling may not be sufficient to alleviate this issue.", "other_comments_or_suggestions": "It would be good to discuss potential implications of the method in a multi-turn QA setup.", "questions_for_authors": "Based on the recent trends in the VLM literature, it would be good to have the following comparisons in the paper. \n1. SparseVLM applied to visual tokens from hierarchical backbones like ConvNeXt/FastViTHD used in ConvLLaVA[1]/FastVLM[2] respectively. \n2. SparseVLM applied to VLMs with multiple vision encoders with different pre-trainings, for example Cambrian-1[3].   \n3. For VLMs trained to support dynamic image resolutions like InternVL2, Qwen2-VL, comparisons with appropriate input sizes to match SparseVLM token lengths. \n\n[1] - ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models\n[2] - FastVLM: Efficient Vision Encoding for Vision Language Models\n[3] - Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rrUc7soiyg", "reviewer": "Reviewer_TyrH", "summary": "This paper presents a novel framework for sparsifying visual tokens to enhance the efficiency of Vision-Language Models (VLMs) in a training-free manner. It proposes a strategy to select relevant text tokens as evaluators of visual tokens, followed by pruning redundant visual tokens with a recycling mechanism to minimize the loss of information. LLaVA equipped with SparseVLM can achieve 54% reduction in FLOPs, 37% decrease in CUDA latency while maintaining 97% performance.\n\n## update after rebuttal\nThank the author for the rebuttal. I will keep my original rating, which is weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the **reviewer TyrH** for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> **1. The further explanation of our claim about \"first attempt to explore text-aware guidance\" in vision token sparsification.**\n\nOur method is the first to ***explicitly*** utilize the guidance of text for the sparsification of VLMs. The detailed discussions are as follows.\n\n(1) ***LLaVA-PruMerge only leverages the class tokens*** to predict the importance of spatial visual tokens. The basis for this derives from the statement on page 5 of their paper: \"*Note that we use the class attention value from the penultimate layer for this calculation.*\" Reference: https://arxiv.org/pdf/2403.15388.\n\n(2) ***FastV directly utilizes all the tokens***, including text tokens, vision tokens themselves, and system tokens to evaluate the significance of vision tokens. The evidence is derived from the following statement on page 8 of their paper: \"*We simply compute the average attention-score one token received from all other tokens as the criteria in our experiment.*\" Reference: https://arxiv.org/pdf/2403.06764.\n\n(3) In contrast, ***our approach builds solely on text tokens and further filters the visual-relevant text raters to improve performance***, which has a fundamental difference from LLaVA-PruMerge and FastV. Our effectiveness is also validated in Table 1, where our method shows superiority.\n\n---\n\n> **2. Further explanation on how the correlation among attention vectors relates to visual redundancy.**\n\nWe identified several relevant studies that also investigated the correlation between matrix rank, model training, and the informativeness of visual tokens. \n\nThe first study [1] reveals a positive correlation between the rank of attention matrices and the performance of Transformer models, suggesting that the rank of the attention matrix influences both the representational capacity and learning effectiveness of the attention mechanism. A higher rank generally indicates a greater ability to encode diverse and informative token interactions. On the other hand, lower ranks often indicate linearly dependent attention vectors, which correspond to overlapping or redundant visual information.\n\nMoreover, some methods leverage singular value decomposition (SVD) to prune the attention output matrix, illustrating that low-rank structures often correspond to redundant information. For example, in [2], the authors use SVD to quantify each token’s information contribution. Combined with attention scores, this allows them to prioritize and retain the most informative tokens. This approach closely aligns with our use of rank, as both rely on analyzing the redundancy of visual tokens in attention matrices.\n\nFor the mathematical definition of rank, the rank captures the number of linearly independent directions in the attention space—lower rank implies that multiple tokens share similar attention patterns, justifying the removal of redundant ones. So, based on these findings and the mathematical definition of rank, we use Rank(P) to quantify redundancy among visual tokens.\n\nWe also validate our approach experimentally by analyzing attention maps in various vision tasks. Results confirm that attention matrices often contain redundant structure, and pruning based on Rank(P) effectively reduces computational overhead with minimal impact on performance.\n\n[1] *Min, Zeping, and Zhong Li. \"On the Limitation and Redundancy of Transformers: A Rank Perspective.*\"\n\n[2] *Tan, Xudong, et al. \"TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models.*\"\n\n---\n\n*We sincerely appreciate your thorough review and valuable suggestions to improve our work. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.*", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper contributes to the area of efficient multi-modal large language models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow, allowing readers to readily grasp the inherent design principles behind SparseLVM.\n2. Experiments on both image and video setting prove the effectiveness of SparseVLM.\n3. The performance has been validated across three distinct MLLM architectures, demonstrating its strong generalization capability.\n\nWeakness:\n1. Lack of discussion with similar works like LLaVA-PruMerge[1] and FastV. As far as I'm concerned, they are also methods for text-aware guidance for visual token sparsification. You need to further support your claim 'first attempt'.\n2. Questionable method for calculating visual redundancy: The approach uses Rank(P) based on attention maps to determine visual token redundancy. However, the correlation between attention vectors and visual redundancy is unclear, with no supporting evidence or references provided to justify why this approach is valid.\n\n[1] Shang, Y., Cai, M., Xu, B., Lee, Y. J., & Yan, Y. (2024). Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kZCvALiSgX", "reviewer": "Reviewer_xeQB", "summary": "This paper presents SparseVLM, a text-guided, training-free token optimization mechanism that improves the efficiency of vision-language models (VLMs). SparseVLM selects relevant text tokens to evaluate the significance of visual tokens using self-attention matrices and then progressively prunes irrelevant tokens. To enhance sparsity, it employs a rank-based strategy and a token recycling method. Experimental results demonstrate that SparseVLM improves VLM efficiency across various image and video understanding tasks. For example, integrating SparseVLM with LLaVA reduces FLOPs by 54% and CUDA latency by 37%, while retaining 97% of the original accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n\n> **1. References on the computation of the rank of the matrix.**\n\nThe rank of a matrix is the maximum number of linearly independent rows, determined by singular value decomposition. We will add a reference to the corresponding supplementary part.\n\n---\n\n> **2. Explanation of the selection via the rank method and its threshold.**\n\nMathematically, matrix rank quantifies non-redundant information via its linearly independent vectors. The motivation of Rank(P) is to adaptively determine dynamic thresholds to identify redundant visual tokens per layer for sparsification, eliminating heuristic sparsity ratios requiring manual tuning in other methods. Experiments scale Rank(P) solely for fair comparisons under matching sparsity to demonstrate algorithm superiority. Practical deployment requires no scaling adjustments or tuning.\n\n---\n\n> **3. Ablation study on determining the number of deletions.**\n\nWe conducted comparative experiments with min-$k$ (top-$k$) selection strategy. Compared to the min-$k$ method, our rank method outperforms across all token settings. The results suggest that the rank-based method is more advantageous in information-constrained scenarios.\n\n|Method|GQA| MMB| SQA| SEED| TextVQA| Avg.|\n|---|---|---|---|---|---|---|\n|**Retain 192 Tokens**|\n|min-k | 57.2 | 62.7 | 70.1 | 56.6 | 56.7 |60.7\n|rank| 59.5 | 64.1 | 68.7 | 58.7| 57.8 |**61.8 (+0.9)**\n|**Retain 128 Tokens**|\n|min-k|55.2 |60.9 | 69.7| 54.0 |55.6| 59.1\n|rank|58.4 |64.5 | 68.6|58.2|56.7| **61.3 (+2.2)**\n|**Retain 64 Tokens**|\n|min-k | 48.5 | 52.5|70.1|45.4|50.1| 53.3\n|rank|53.8|60.1|69.8|52.2|53.4|**57.9 (+4.6)**\n\n---\n\n> **4. Adjustment of the position IDs.**\n\nToken pruning in SparseVLM creates layer-varying KV token counts, typically below the original sequence length. Using unpruned position IDs misaligns RoPE, as IDs mismatch the pruned KV cache. We solve this via dynamic position ID recalibration:  (1) Track retained tokens from prior steps via LLaVA’s KV cache. (2) Reassign position IDs to preserved/recycled tokens. (3) Apply RoPE with updated IDs. Our method updates positional encodings during token additions/removals, ensuring alignment with pruned sequences and spatial consistency across encoding schemes.\n\n---\n\n> **5. Performance of SparseVLM on more challenging benchmarks.**\n\n(1) **Image tasks** \n\nWe tested on MMBench (LR) and MMMU Pro. Our performance on MMMU Pro approaches the average accuracy reported in Table 1 across all token settings. Besides, SparseVLM progressive token sparsification even improves MMBench (LR) performance at 192 tokens by shifting focus from global to local.\n\n|Settings|MMMU Pro (Loss)|MMBench (LR) (Loss)\n|---|---|---| \n|LLaVA-7B|30.3 (0)|30.5 (0)|\n|**192 tokens**|30.0 (-0.3)|33.9 (+3.4)\n|**128 tokens**|29.4 (-0.9)|30.5 (0)\n|**64 tokens**|26.4 (-3.9)|26.7 (-3.8)\n\n(2) **Video tasks** \n\nWe tested our method on VideoMME and LVBench. Our method prunes $90$% of vision tokens, retaining only 227, while preserving around $99$% of the original accuracy. It outperforms FastV by $6.8$% on VideoMME and $2.2$% on LVBench. \n\n|Method|VideoMME (Loss)|LVBench (Loss)\n|---|---|---|\n|VideoLLaVA|39.9 (0)|26.3 (0)\n|FastV|32.9 (-7.0)|23.8 (-2.5)\n|SparseVLM|**39.7 (-0.2)**|**26.0 (-0.3)**\n  \n---\n\n> **6. Claim about \"first attempt to explore text-aware guidance\" in vision token sparsification.**\n\n(1) VideoLLM-MoD uses a trained linear projection to predict the importance of vision tokens. The basis is from page 5 of their paper: \"*The LayerExpert determines the importance score μ for a given vision token using a linear projection.*\" Reference: arXiv:2408.16730.\n\n(2) FreeVideoLLM uses the prompt feature for temporal and spatial samplings. The evidence is from page 5 of their paper: \"*We calculate the relation score (i.e., cosine similarity) between the frame features and the prompt feature.*\" Reference: arXiv:2410.10441. We consider it as concurrent work with ours since it is not accepted yet. Still, we will update it in our related work.\n\n(3) ***Our approach builds solely on text tokens and filters the visual-relevant text raters to improve performance, which has a fundamental difference***.\n\n---\n\n> **7. Failure cases for sparsification.**\n\n**\"n281241.jpg\" in GQA**. Q: \"*What is the picture hanging above?*\" The label is \"*Chair*,\" but the VLM predicts \"*Wall.*\" The question asks what is above the object, not where the picture is hung. Misguided attention to background tokens causes spatial misinterpretation, resulting in incorrect answers.\n\n**\"n494677.jpg\" in GQA**. Q: \"*Is the weather cloudy?*\" The label is \"*Yes,*\" but the VLM predicts \"*No.*\" Despite sparse vision tokens accurately capturing the sky, the model still errs, indicating reasoning limitations rather than sparsification flaws.\n\nFor visualizations, please refer to the anonymous link: https://imgur.com/a/Hq0qYYa.\n\n---\n\n*We appreciate your thorough suggestions. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.*", "claims_and_evidence": "The claims in this paper—\"visual tokens should be sparsified adaptively based on the question prompt\" and \"not all prompt tokens should be considered\"—are intuitively reasonable and supported by experimental results.", "methods_and_evaluation_criteria": "**Method**\n\n- L202-203: When first encountering the term \"rank,\" it is unclear how it is computed. Although the supplementary material provides details, please add a reference here for clarity.\n- Rank(P) Selection: Is Rank(P) too tricky? How is the \"appropriate threshold\" selected? Additionally, how can you ensure that this threshold remains valid across all LLM layers?\n- Deletions: Where is the ablation study on determining the number of deletions? Please compare this approach with a simpler min-k method.\n- RoPE Position IDs: What are the position IDs of RoPE for the recycled token and the maintained vision tokens?\n\n**Evaluation**\n\nOverall, the evaluation setup is well-designed. If possible, could you provide results on more challenging benchmarks, such as VideoMME, MMMU Pro, or LVBench?", "theoretical_claims": "I have checked Section 3.4, and it looks fine.", "experimental_designs_or_analyses": "The experimental results are indeed impressive, significantly surpassing previous methods. However, as mentioned earlier, please provide results on more challenging benchmarks if possible.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "This paper is not the first to propose that \"visual tokens should be sparsified adaptively based on the question prompt.\" Please include a discussion of related works, such as [a] VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation (NeurIPS 2024) and [b] Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs (arXiv 2410.10441).", "other_strengths_and_weaknesses": "Could you also provide some failure cases for Figure 6? This would help in understanding the scenarios where the method does not work.", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LvjtETb62", "reviewer": "Reviewer_bpD8", "summary": "SparseVLM introduces a text-guided visual token sparsification framework for efficient VLM inference without significant performance loss. The key idea is to use the textual input to identify which image regions (visual tokens) are most relevant and prune away the rest.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the **reviewer bpD8** for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> **1. The analysis of the generality of SparseVLM to other general multimodal tasks.**\n\nWe sincerely appreciate your valuable suggestion. To investigate how SparseVLM sparsifies visual tokens when dealing with questions similar to \"*Describe this photo in detail.*\", we further visualized the token sparsification results from different layers of SparseVLM when handling these questions. It was surprising that when confronted with such questions, SparseVLM first shifts the model's attention to the background information of the image. Subsequently, it sparsifies the tokens corresponding to this part and focuses on the information of the main objects in the image (*e.g.*, the people in the image). Finally, it pays attention to the detailed information of the objects (*e.g.*, the nose, mouth, and eyes of a person), enabling the model to describe the image smoothly and progressively. ***Therefore, our method follows a global-to-local understanding process, yet still effectively captures the key information in captioning tasks.*** For a visualization example, please refer to the anonymous link: https://imgur.com/a/lbp1qkI.\n\n---\n\n> **2. The performance of SparseVLM on other general multimodal tasks.**\n\n***(1) Qualitative analysis***\n\nHere we take the captioning task as an example and conduct a qualitative analysis. Specifically, we ask the model with the question_id of 1 in LLaVA-Bench (In-the-Wild), where the prompt is \"*Describe this photo in detail*\".\n\n**The GPT-4 answer is**: *This photo features an aerial view of Diamond Head, located in the Hawaiian Islands. The volcanic tuff cone appears prominent and impressive, with its ridges and slopes covered in lush green vegetation. The crater of Diamond Head is clearly visible, surrounded by the steep walls of the structure. In the background, you can see the vast expanse of the ocean with varying shades of blue, showcasing the pristine waters surrounding the island.*\n\n**The original LLaVA answer is**: *The image showcases a breathtaking view of a large, rocky island with a mountainous terrain. The island is surrounded by a beautiful blue ocean, and it appears to be a tropical paradise. The island is covered in lush greenery, and there are several houses scattered across the landscape. In the distance, a city can be seen, adding to the picturesque scene. The combination of the island's natural beauty and the city's presence creates a unique and captivating view.*\n\n**Our method (192 tokens) answer is**: *The image showcases a breathtaking aerial view of a large, rocky island with mountainous terrain. The island is surrounded by a beautiful blue ocean, and the shoreline is dotted with houses scattered across the landscape. The houses vary in size and are positioned at different distances from the shore, creating a lively and picturesque scene.*\n\n***From the answers, it can be seen that our method successfully preserves the general style, atmosphere, and scene composition of images.*** The observed minor omissions in specific details (such as '*lush greenery*') represent a conscious trade-off, as we intentionally maintain only $33.3$% (192/576) of visual tokens to optimize efficiency.\n\n***(2) Quantitative analysis***\n\nWe further performed experiments on LLaVA-Bench (In-the Wild) and LLaVA-Bench (COCO), and the results are shown in the table below. It can be observed that SparseVLM achieves significant performance in the task of general visual dialogue. When the number of tokens is sparsified to 192 tokens, it may even lead to a slight performance improvement, which demonstrates the effectiveness of our algorithm. ***The reason is that SparseVLM is capable of progressively sparsifying tokens, enabling the model's attention to gradually focus, transitioning from the image background to the objects within the image and then to the details of those objects.*** This allows the model to understand the images in a step-by-step manner, and achieve outstanding performance in tasks such as general visual dialogue.\n\n| Settings | In-the Wild (Loss) | COCO (Loss) | Avg. (Loss) |\n| ---- | ---- | ---- | ---- |\n| LLaVA-7B | 62.7 (0) | 72.3 (0) | 67.5 (0) |\n| **192 tokens** | 63.7 (+1.0) | 71.9 (-0.4) | 67.8 (+0.3) |\n| **128 tokens** | 61.7 (-1.0) | 70.4 (-1.9) | 66.1 (-1.4) |\n| **64 tokens** | 60.4 (-2.3) | 66.2 (-6.1) | 63.3 (-4.2) | \n\n---\n\n*We sincerely appreciate your thorough review and valuable suggestions to improve our work. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.*", "claims_and_evidence": "Based on the motivation, \"visual tokens should be sparsified adaptively based on the question prompt\", the author showed the performance of the SparseVLM with experiments. The motivation is clear, and experiments are well-done, although the overall paper is based on emprical results.", "methods_and_evaluation_criteria": "See above.", "theoretical_claims": "-", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "The paper proposed SparseVLM, an efficient inference method for VLM without additional finetuning.", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "The method’s applicability might be somewhat limited to scenarios where a guiding text is present. For task such as captioning without a specific query) SparseVLM in its current form might not know which tokens to prune because there’s no external query focusing it. \nThey don’t explicitly address how their method might extend to, for example, caption generation or more complex tasks. Since the proposed method aims for efficient VLM, it should show generability to other general VLM tasks, rather than just simple QA with classification.", "other_comments_or_suggestions": "-", "questions_for_authors": "- As noted, SparseVLM currently assumes a guiding text query. It would be interesting to see how it could be applied to tasks like image captioning or general visual dialogue, where the prompt might be generic (e.g., “Describe the image”).\nI'm curious about the experiment results of more general tasks, such as Tab.7 or Appendix Fig.1 / 2 from the original LLAVA paper [1], with  LLaVA-Bench (In-the Wild) and LLaVA-Bench (COCO) datasets.\n\n[1] https://arxiv.org/pdf/2304.08485", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wUWl8BJLtH", "reviewer": "Reviewer_SmbH", "summary": "SparseVLM is a training-free framework that optimizes VLMs by reducing computational load through selective pruning of visual tokens based on relevance to text tokens. Using self-attention matrices, it introduces a rank-based adaptive sparsification strategy and a token recycling mechanism to retain essential information. Compatible with models like LLaVA, Mini-Gemini, and Qwen2-VL, it reduces latency and costs while preserving accuracy.\n\n## update after rebuttal\nThank the author for the rebuttal. I will keep my original rating, which was already positive.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the **reviewer SmbH** for the effort in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> **1. The discussion examines how effectively SparseVLM performs on highly complex tasks.**\n\nThank you for your attention to our performance on more complicated tasks. Firstly, due to adaptive sparsification, our method can determine whether the vision tokens in the current layer are redundant for the prompt to avoid losing critical information. Besides, our token recycling mechanism can retain as much information as possible in a cost-effective manner. To validate the effectiveness, we perform experiments on MMMU Pro, MMBench (Attribute Reasoning, AR), and MMBench (Logical Reasoning, LR) with LLaVA-7B. Our performance on MMMU Pro consistently approaches the average accuracy reported in Table 1 across all token settings, demonstrating our method's robustness in highly complex tasks. Furthermore, even with a severe reduction in the number of tokens from 576 to 192, the MMBench (LR) performance surpasses that of the baseline. The reason is that SparseVLM progressively sparsifies tokens, enabling the model to gradually focus, transitioning from the image background to the objects within the image and then to the details of those objects, and achieve outstanding performance. ***In summary, our SparseVLM can well handle highly complex tasks while maintaining vital information.***\n\n| Settings         | MMMU Pro         | MMBench (AR)    | MMBench (LR)    | Avg.  (Loss) |\n| ------------- | ----------------| ---------------| --------------| --------------| \n| *Upper bound*  |  30.3       | 73.3        | 30.5         | 44.7 (0)    |\n| **192 tokens**   | 30.0     | 71.9  | 33.9  | **45.3 (+0.6)** |\n| **128 tokens**   | 29.4   | 70.4 | 30.5 | **43.4 (-1.3)**     |\n| **64 tokens**   | 26.4  | 67.2   | 26.7  | **40.1 (-4.6)**|\n\n---\n\n> **2. The analysis of the impact of token recycling in more complex, multi-step reasoning tasks.**\n\nWe appreciate the insightful suggestion. To further explore the effectiveness of our token recycling strategy in more complex and multi-step reasoning tasks, we conduct ablation experiments on MMBench (LR) and MMMU Pro. Across multiple sparsity ratios (64, 128, 192), our algorithm achieves a significant average performance improvement of $0.8$% and $1.0$% on MMBench (LR) and MMMU Pro, respectively. Notably, as the number of pruned vision tokens increases, the benefit brought by our recycling method increases. For instance, when pruning from 192 to 64 tokens, pruned token recycling significantly boosts accuracy from $0.2$% to $1.6$% on MMMU Pro. ***In short, compared to common tasks (e.g., GQA in Table 4), our token recycling mechanism proves its higher value in complex, multi-step reasoning tasks.***\n\n| Benchmark| 64 tokens| 128 tokens | 192 tokens | Avg. |\n| --- | :---: | :---: | :---: | --- |\n| **MMBench (LR)** | 25.2 | 30.0 |  33.6 | 29.6\n| **+ TR**| 26.7 | 30.5 | 33.9 | 30.4\n| **MMMU Pro** | 24.8 | 29.2 | 28.8 | 27.6\n| **+ TR**| 26.4 | 29.4 | 30.0 | 28.6\n\n---\n\n> **3. Further experiments on other transformer-based VLM architectures.**\n\nWe sincerely thank you for your advice. We have conducted experiments across various transformer-based architectures, including different vision encoders (e.g., CLIP in LLaVA and CLIP+ConvNeXt in Mini-Gemini) and different LLM decoders (e.g., LLaMA in LLaVA and Qwen in Qwen2-VL). To further validate compatibility, we also tested our approach on Cambrian-1 13B (576 tokens setting), which is another transformer-based VLM architecture. Shown in the table below, under the 192-token setting, our method achieves a $2.3$% lower performance drop than PDrop; meanwhile, under the 64-token setting, our approach maintains an accuracy loss below $8.0$%. ***Therefore, our method is fully compatible with other transformer-based architectures.***\n\n\n| Method | GQA | MMB| SQA | SEED | TextVQA  | MMMU | Avg. (Loss) |\n| --- | --- | --- | --- | --- | --- | --- |  --- |\n| **Cambrian-1-13B** | 64.0 | 75.5 | 79.3 | 74.4 | 72.8 | 40.0 | 67.7 (0)\n| **Retain 192 Tokens**|\n| PDrop [*CVPR 2025*]| 59.5 | 72.5 | 75.3 |68.4 | 70.0 | 38.0 | 63.9 (-3.8)\n| SparseVLM | 61.6 | 74.2 | 79.0 | 72.0 | 71.4 | 38.8 | **66.2 (-1.5)**\n| **Retain 128 Tokens**|\n| PDrop [*CVPR 2025*]| 56.8 | 70.9 | 74.0 | 65.5 | 68.8 | 35.4 | 61.9 (-5.8)\n| SparseVLM | 60.4 | 73.0 | 78.2 | 70.6 | 70.2 | 38.0 | **65.1 (-2.6)**\n| **Retain 64 Tokens**|\n| PDrop [*CVPR 2025*]| 46.2 | 61.6 | 71.7 | 56.4 | 58.0 | 26.1 | 53.3 (-14.4)\n| SparseVLM | 54.4 | 68.7 | 78.0 | 64.0 | 62.2 | 31.2 | **59.8  (-7.9)**\n\n---\n\n*We sincerely appreciate your thorough review and valuable suggestions to improve our work. If you find our responses satisfactory, we would be grateful for your reconsideration of the rating.*", "claims_and_evidence": "SparseVLM can reduce computational overhead in VLMs without sacrificing performance.\n- Experimental results demonstrate that SparseVLM can reduce FLOPs by up to 77.8%, decrease CUDA latency by up to 37%, and maintain 97% of the original accuracy in image tasks like VQA.\n\nSparseVLM improves the efficiency of video question answering task.\n- In video understanding tasks, SparseVLM outperforms FastV by 34.4% in accuracy and achieves a smaller decrease in performance (0.17 GPT score drop vs. 1.02 for FastV).\n\nSparseVLM provides a trade-off between efficiency and performance.\n- The method shows significant improvement in efficiency (e.g., 62.8% reduction in FLOPs) with minimal accuracy loss, validated through extensive experiments on multiple benchmarks.", "methods_and_evaluation_criteria": "- Methods: SparseVLM uses a two-step process for pruning and refining visual tokens. First, visual tokens are rated based on their correlation with relevant text tokens (text raters). Then, the tokens are sparsified using an adaptive rank-based strategy. For tokens that are pruned, a recycling mechanism is employed to aggregate and compress them into more compact forms. The method integrates well with Flash Attention for efficient computation.\n- Evaluation Criteria: The evaluation focuses on accuracy, FLOPs, and latency reduction across multiple vision-language benchmarks, such as GQA, VQAv2, and MSRVTT, as well as video benchmarks like TGIF-QA and MSVD-QA. Trade-offs between computational savings and performance are carefully analyzed.", "theoretical_claims": "- Visual tokens can be sparsified adaptively based on text tokens to optimize VLM performance: The authors argue that not all visual tokens are equally relevant to a given task and that adaptive sparsification based on textual context can improve both efficiency and accuracy. This claim is supported by the results showing the effectiveness of the rank-based strategy and text-aware pruning.\n- Token recycling helps preserve important visual information during sparsification: The authors suggest that recycling pruned tokens minimizes information loss, which is crucial for maintaining performance in downstream tasks. This is backed by experimental improvements in performance when token recycling is used.", "experimental_designs_or_analyses": "- Design: SparseVLM was tested on multiple VLMs (LLaVA, Mini-Gemini, Qwen2-VL) across image and video understanding tasks. Three token configurations (576, 192, 64 tokens) were used to assess the impact of sparsification. Comparative experiments with other methods (e.g., FastV, ToMe) were conducted to evaluate improvements in efficiency and accuracy.\n- Analysis: The primary analysis focuses on comparing the reduction in FLOPs and latency with minimal loss in accuracy. The method was also analyzed on its ability to handle video tasks, where temporal and spatial dependencies are critical.", "relation_to_broader_scientific_literature": "- SparseVLM is an important contribution to the ongoing research on making vision-language models more efficient. It addresses the challenge of visual token redundancy that is prevalent in high-resolution image and video tasks, complementing other recent works that focus on reducing the computational burden in VLMs.\n- The method is compared with other pruning and sparsification strategies (e.g., FastV, ToMe, PDrop) and outperforms them in both computational efficiency and accuracy.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses: The method might struggle with extremely complex tasks where minimal information loss is critical, as the adaptive sparsification might lead to dropping crucial visual tokens in certain edge cases.", "other_comments_or_suggestions": "It would be useful to see a deeper exploration of the impact of token recycling in more complex, multi-step reasoning tasks where preserving information is more crucial.", "questions_for_authors": "Can SparseVLM be generalized to work with other transformer-based architectures beyond the ones tested in the paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper is about a training-free token optimization for vision language models, which has been a popular area.  All reviewers agree the novelty and the effectiveness by the experiment results. More than 1 reviewer raise the issue of missing related works. Please include all of them in the revision.  


VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation, NeurIPS 2024

FastVLM: Efficient Vision Encoding for Vision Language Models, CVPR 2025

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs, ICLR 2025


Llava-prumerge: Adaptive token reduction for efficient large multimodal models, 2024

ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models, 2024 

Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs, 2024
Decision: Accept (poster)