On Exploring Visual Attention Shrinking for Accelerating VLMs for Video Understanding

Authors: Chang Liu, Jian Jia, Ye Ma, Quan Chen, Peng Jiang, Zhijie Deng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Visual Language model, Inference Acceleration, Visual Attention Shrinking

Abstract:
Vision-language models (VLMs) have shown promise in a variety of challenging video comprehension tasks. VLMs typically extract frames from the source video and take the corresponding encoded visual tokens as input. A rapid increase in the number of visual tokens, e.g., when handling lengthy videos, can swiftly lead to a long-context dilemma during the inference process of VLMs, posing an efficiency challenge for real-world applications. Given that significant redundant and task-irrelevant information may exist in the visual tokens across both spatial and temporal axes, we advocate removing less important visual tokens during the prefilling phase of the inference procedure to improve the computation and storage efficiency of VLMs. We first identify an interesting phenomenon termed as \emph{Visual Attention Shrinking (VAS)}, wherein certain visual tokens receive progressively diminishing attention during the processing stages of the model. This implies that the model itself knows what to care about and what to discard. With this understanding, we develop a robust algorithm to detect attention shrinking at each layer of the model using states from preceding layers. Based on the detection results, we perform token removal in both temporal and spatial axes. Our approach does not require parameterized modifications to the original VLM and is compatible with the prevalent KV cache strategy. Through extensive experiments across different VLMs, our approach witnesses an average speedup of $1.98\times$ in generating the first response token, utilizing only 47.2% of the visual tokens, without compromising the task performance. Additionally, when applied to the huge VILA1.5-40B, our method can achieve up to $4.16\times$ speedup compared to the vanilla model.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729544650151

thread_type:
official_review

<thread_object>
creator: reviewer_qub7

<content>
summary:
The large number of visual tokens creates significant inconvenience for both the training and inference of large multimodal models (LMMs). Therefore, reducing and removing unnecessary visual tokens is an important direction for development. This paper observes that certain visual tokens receive progressively less attention during the processing stages of the model. Based on this observation, token removal is performed along the temporal and spatial axes without requiring architectural modifications. This strategy can reduce the number of visual tokens by approximately 50%, with almost no performance degradation, while significantly speeding up inference.

strengths:
1. The efficiency and performance of the proposed method are competitive compared to ToMe and FastV.

2. This is an architecture-agnostic approach, meaning it can be widely applied to various LMMs.

3. Although the pipeline figure in the paper is somewhat confusing, the textual organization is well-structured and makes it understandable.

4. In addition to the basic experiments, the authors included some observations (such as Figure 2) and ablation studies (Tables 3 and 4) to make the work more comprehensive.

5. The motivation of the paper is reasonable, and I believe this is indeed an urgent issue that needs to be addressed in LMMs.

weaknesses:
1. The proposed VAS method does not actually show a sufficiently significant performance difference compared to ToMe [1] and FastV [2]. As shown in Figure 1, when VAS is applied to PLLaVA-7B, the performance on MVBench, VideoMME, and MLVU does not demonstrate a notable advantage (e.g., 46.6 vs 46.9, 48.5 vs 49.0). Furthermore, when VAS is applied to LongVA-7B, it consistently falls behind ToMe across all benchmarks.

2. There are many instances of "Failed" in Table 1, and the authors lack a reasonable explanation for this. The only mention of it is in Section 4.2.2, where it is briefly noted that "FastV’s aggressive reduction approach proves to be overly drastic, often resulting in model abnormal responses." However, this is merely a superficial description of the phenomenon.

3. The authors lack a direct comparison of the proposed VAS method with prior works, such as ToMe [1] and FastV [2], in terms of motivation and technique. Compared to FastV [2], both methods evaluate the importance of visual tokens based on attention weights, identifying tokens that can be discarded without sacrificing model performance. Both approaches use gradual token reduction strategies, ensuring that important tokens are preserved during the early stages of inference to prevent the loss of critical information.

4. The authors lack citations and discussions of other works focused on token merging, such as LlamaVid [3] and MovieChat [4], among others. Additionally, the authors' discussion of Chat-UniVi is inaccurate. The claims that it "attempted to adjust the architecture of the visual encoder" and "introduced additional training requirements" do not align with the original paper's descriptions.

5. Although the authors emphasize "lengthy videos" in the abstract, the experiments lack analysis and emphasis on VAS's performance and observations in long video sequences. For example, does attention dispersion become more severe with longer sequences? It would have been better if the authors had conducted more detailed observations on benchmarks like MovieChat-1K [4] or LongVideoBench [6].

[1] Bolya, Daniel, et al. "Token merging: Your vit but faster." arXiv preprint arXiv:2210.09461 (2022).

[2] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." arXiv preprint arXiv:2403.06764 (2024).

[3] Li, Yanwei, Chengyao Wang, and Jiaya Jia. "Llama-vid: An image is worth 2 tokens in large language models." European Conference on Computer Vision. Springer, Cham, 2025.

[4] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Jin, Peng, et al. "Chat-univi: Unified visual representation empowers large language models with image and video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[6] Wu, Haoning, et al. "Longvideobench: A benchmark for long-context interleaved video-language understanding." arXiv preprint arXiv:2407.15754 (2024).

questions:
Please revise the Weaknesses section point by point. This is a paper with great potential. If the authors can provide additional responses to certain issues, discuss related work more thoroughly, and include more experiments and observations, I would be very happy to raise my score. Additionally, please redraw Figure 1, as it took me twice the time to understand what VAS is actually doing!
</content>
</thread_object>

</thread>
<thread>
creation date:
1730111136735

thread_type:
official_review

<thread_object>
creator: reviewer_jjkn

<content>
summary:
This paper identifies an interesting phenomenon termed as Visual Attention Shrinking (VAS), wherein certain visual tokens receive progressively diminishing attention during the processing stages of the model, and develop a robust algorithm to detect attention shrinking at each layer of the model using states from preceding layers.

The contributions are summaried as follows:
1. observe an interesting phenomenon called Visual Attention Shrinking.
2.  develop an algorithm that utilizes the attention states from previous layers to robustly detect attention shrinking, and continuously removes visual tokens based on the detection results. 
3. extensive experiments on various VLMs, across multiple video benchmarks, validate the effectiveness of propsoed method.

strengths:
1. The phenomenon of Visual Attention Shrinking where some frames or positions consistently exhibit a downward trend in attention scores, is interesting.
2. In the current field of video VLM, a large number of visual tokens are commonly used to represent videos. It is very meaningful to explore token reduction in this field.
3. The experiments are comprehensive, including multiple Benchmarks and video VLM models, and the method shows a significant acceleration for the models.

weaknesses:
1. In certain settings, the proposed method experiences a higher drop in performance, such as with LongVA-7B and VILA1.5-40B.
2. Although the article focuses on accelerating the inference of video VLM models, it lacks substantial design specific to video.

questions:
1.  Is the shape of $T$  $m$ or $n$? If it is $m$, then in Line 8 of Algorithm 1, $T$ should be summed over dimension 1. It appears there is a conflict between Line 8 and Line 10 in Algorithm 1.
2. How about multi-turn conversations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627519658

thread_type:
official_review

<thread_object>
creator: reviewer_tqfb

<content>
summary:
This paper introduces a dynamic token removal method based on the phenomenon of Visual Attention Shrinking (VAS), which identifies less important visual tokens during inference. The algorithm operates without modifying the original VLM and is compatible with KV cache strategies.

strengths:
The paper introduces the phenomenon of Visual Attention Shrinking (VAS) and develops a dynamic token removal algorithm based on this phenomenon, providing a new approach to improve the inference efficiency of video VLMs.
The proposed algorithm does not require parameterized modifications to the original VLM and is compatible with the prevalent KV cache strategy, demonstrating good generality.
The proposed method can perform token removal in both temporal and spatial dimensions, enhancing inference speed.

weaknesses:
The paper primarily demonstrates the effectiveness of VAS through experimental results, but it lacks more intuitive visual analyses. It is recommended to use visualization tools (such as heatmaps, attention maps, etc.) to clearly illustrate the changes in attention distribution of the model before and after token removal, as well as the impact on the model's inference process.

The proposed method does not perform as well on certain specific tasks (such as Egoschema and MLVU) and fails to adequately explore the reasons for this.

questions:
The paper mentions that the token reduction algorithm performs token reduction at certain layers, but what is the basis for selecting which layers to apply token reduction?
Can the dynamic visual token reduction algorithm you proposed be extended to other types of multimodal data beyond video?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639760004

thread_type:
official_review

<thread_object>
creator: reviewer_6r1m

<content>
summary:
This paper proposes a method to improve the efficiency of vision-language models for video tasks by reducing redundant visual tokens during inference. The authors introduce an algorithm to detect and remove low-importance tokens along spatial and temporal dimensions. Their approach requires no parameter changes, is compatible with KV caching, and achieves substantial speedups, using only 47.2% of tokens without degrading task performance, especially benefiting large models like VILA1.5-40B.

strengths:
1. Tuning-free and plug-and-play: Being tuning-free and plug-and-play, The proposed method can be seamlessly integrated with existing VLMs without the need for extensive modifications or retraining, facilitating broader adoption.
2. Efficient Token Usage. Removing less important visual tokens during inference is an intuitive motivation.
3. Well-structured presentation. The presentation of this paper is clear and easy to understand.

weaknesses:
1. Lack sufficient experimental support. It would be beneficial to include evaluations on other challenging video benchmarks, such as long video datasets, to validate effectiveness and enable a more comprehensive comparison.
2. Limited novelty. The concept of removing redundant tokens based on attention scores is not entirely new and has been explored in other VLM and transformer-based model optimizations. This approach may not offer a substantial advancement beyond existing methods.

Minor issues: These are some typos in the paper, such as Line 464 and the captions of Table 1.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732301299590

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear reviewers,

We appreciate each reviewer for the time spent reviewing our paper. Thanks for your positive attitude towards our work. 

Thanks for your valuable questions and constructive suggestions, which have inspired us to address some of the previous shortcomings. We have responded to your concerns and uploaded a revised version. Here, we would like to briefly outline the changes made:

+ We correct the inaccurate description in the Introduction and redraw Figure 1, also adding relevant work.

+ In the Experiments section, we clarify the differences between the various methods and explain the reasons for the failures of some methods.

+ Additional experimental results addressing the reviewers' concerns are included in the Appendix.

+ Minor adjustments, such as correcting mistakes and avoiding ambiguous phrasing.

If you are willing, you can take a look at the revised version, where we highlight the main modifications in red in the main text. Of course, I believe and hope that our responses will address most of your concerns.

Thank you again for your feedback and valuable suggestions.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a novel method to reduce video tokens both spatially and temporally to improve the efficiency of VLMs for video understanding. The proposed method is based on an observation denoted as visual attention shrinking. Proposed method was tested on different VLMs across several benchmarks show its effectiveness.

As pointed out by several reviewers, the proposed method is tuning-free and plug-and-play and can efficiently reduce the number of video tokens both spatially and temporally, showing good potential. Reviewers also pointed out some concerns: 1. Limited novelty (reviewer 6r1M) and lack of special design for video (reviewer jJKn) . 2. Missing some implementation details (reviewer TQfB). Considering all discussions, the difference of the proposed method is not significant enough from existing ones and the missing of implementation details makes the proposed method ad-hoc. I recommend rejection.