LLM-VTP: LLM-Reasoned Visual Token Pruning for Efficient Multi-Modal Video Understanding

Authors: Xiaohu Huang, Hao Zhou, Kai Han
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Understanding, Token Pruning

Abstract:
In this paper, we introduce LLM-VTP, a visual token pruning method designed to enhance the efficiency of multi-modal video understanding. Large Language Models (LLMs) have shown promising performance in video tasks due to their extended capabilities in comprehending visual modalities. However, the substantial redundancy in video data presents significant computational challenges for LLMs. To address this, we propose a training-free approach that leverages the inherent reasoning abilities of LLMs to selectively prune visual features based on question tokens, thereby optimizing model efficiency. We validate our method across multiple-choice, open-ended, and text-generation benchmarks. Our results demonstrate that LLM-VTP can prune 80\%-90\% of tokens while maintaining competitive performance. This highlights its superior effectiveness and efficiency compared to existing pruning methods. The source code will be released to facilitate future research.

Decision: reject
Reviewer scores: 5, 8, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729968897613

thread_type:
official_review

<thread_object>
creator: reviewer_ikan

<content>
summary:
This paper proposes a video LLM that focuses on visual token pruning. While previous work has used training-based approaches to reduce redundancy in video data, the authors consider them suboptimal due to the lack of question-specific designs. They develop a training-free pruning method based on attention scores between questions and video tokens. In particular, only the top-ranked video tokens are retained and the KV cache is compressed accordingly, which significantly reduces the computational cost. The proposed method is validated with two video LLMs on multiple video understanding benchmarks, showing significant speedup by pruning 80-90% of the tokens with competitive performance.

strengths:
* The paper is generally well written and easy to follow.
* The authors' analysis of the current limitations of visual token pruning approaches is reasonable, and the proposed approach directly addresses these limitations.
* Extensive quantitative and qualitative results are provided to support the effectiveness of the proposed method.

weaknesses:
* The "LLM-reasoned" in the tile is an overclaim. While LLMs have demonstrated impressive reasoning capabilities, they are not explicitly utilized in the paper. There is also a lack of evidence or further analysis to prove that the LLM has implicitly reasoned about how to prune the visual tokens. The visualizations in Figure 1b and Figure 4 may be the authors' attempts to illustrate this, but they seem too vague for me to draw any conclusions.
* Limited technical contribution. Both token pruning and KV compression are widely used techniques in (multimodal) LLMs, and the proposed approach through attention score-based top-$\alpha\%$ selection does not seem to be a significant contribution. This strategy should be sensitive to the hyperparameter $\alpha$ and might fail on certain videos with more dense information.
* The comparative results with PruMerge and Look-M in Table 2 are mixed. The proposed method does not always achieve a better performance-efficiency tradeoff than the baselines, and there is no intuitive summary of the comparison. The authors have provided some explanation in lines 362-371, but they do not fully address these concerns.
* The retained ratio may not be not a good metric because it does not consider token merging as an effective strategy for reducing computational cost. For token merging, the authors only mentioned that "merging the less informative tokens actually negatively impacts performance" in line 485, which is not a very solid argument since the result is only obtained from a single comparison. It would be better if additional metrics such as FLOPS are included to provide a comprehensive analysis of computational efficiency.

questions:
* Could the authors provide visualizations of which tokens are pruned by their ranking-based selection strategy? Perhaps in the form of a side-by-side comparison of the original and pruned attention heatmaps. This would make the results more intuitive.
* Could the authors tune the hyperparameters for the PruMerge baseline and compare with it on the same retained ratio? I cannot tell if the performance disadvantage of the proposed method is due to a higher compression rate or some weakness of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730353989701

thread_type:
official_review

<thread_object>
creator: reviewer_27y2

<content>
summary:
This paper proposes LLM-VTP, a training-free token pruning method for multi-modal video understanding.
Specifically, authors uses the attention score between question token and visual tokens to select semantically related tokens. 
Authors integrate LLM-VTP with two Video LLMs, i.e., PLLaVA and Tarsier and conduct experiments on several benchmarks including MVBench, MSVD-QA, MSRVTT-QA, ActivityNet-QA, TGIF-QA and VideoChatGPT-Bench. 
Results show that LLM-VTP reduces sequence length of the transformer while maintaining a comparable performance.

strengths:
- The motivation is clear, i.e., using question related visual tokens to prune redundant tokens.
- Experiments are conducted on various benchmarks, demonstrating the effectiveness of LLM-VTP.
- The research topic, i.e., video LLM pruning, has real-world application potential which is valuable.
- On MVBench, the pruned video LLM model even show a better performance compared with the original model.

weaknesses:
-The proposed method uses an encoder to extract all video tokens, a process that is significantly more computationally expensive than reducing tokens in subsequent layers. Reducing tokens post-attention is largely ineffective, as the necessary feature information is already captured by other tokens.
- Is the ablation study of different alpha parameters conducted using PLLaVA? Does similar results appear for Tarsier?
- Does LLM-VTP work on other models like ST-LLM, VideoChat, etc. One concern is that for different models, authors select different hyperparameters for token selection ratio and model layer. This might restrict the generalization ability of LLM-VTP.
- Why Table 3 compares LLM-VTP on Tarsier with other methods on PLLaVA? Is it fair as the alpha and M are different in the two video LLMs.
- For the Figure 4 and Figure 1 (b), the highlighted activation area in different layers are quite similar, then what leads to the performance variance between pruning different layers?

questions:
see the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621036437

thread_type:
official_review

<thread_object>
creator: reviewer_sepg

<content>
summary:
This paper introduces LLM-VTP, which leverages the ability of LLMs to identify useful visual signals for pruning visual features. Experiments on two video LLMs and multiple video understanding tasks validate that this method can effectively reduce the number of tokens without compromising model performance.

strengths:
1. This paper is well written. 
2. The proposed method is training-free and small computational burden.

weaknesses:
1. What is the performance of this method in long videos or scenes with drastic changes in temporal?
2. The author should verify the validity of the method on more different types of backbone.
3. The comparative experimental Settings of some tables require further clarification. For example, in Table 1, under the Tarsier method, why is 20% Retained Ratio retained? If 10% is retained, what are the relevant experimental results? In Table 2, under PLLaVA and Tarsier, the Retained Ratio is 20%. So what does that look like for the 10 percent? In Table 3, the comparison methods are all based on PLLaVA. Why does the authors express their own method as Tarsier w/LLM-VPT (Ours)? So what do all the comparisons look like under the same method?
4. The author lacks in-depth analysis of many experimental phenomena. For example "However, the choice of which layer's attention weights to use differs across datasets."

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697886712

thread_type:
official_review

<thread_object>
creator: reviewer_afxj

<content>
summary:
LLM-VTP is a training-free visual token pruning method that enhances multi-modal video understanding efficiency by leveraging LLMs' reasoning abilities to selectively prune visual features based on questions, reducing 80%-90% of tokens while maintaining competitive performance.

strengths:
The paper introduces LLM-VTP, a simple yet effective, training-free framework that prunes video tokens by leveraging LLMs' ability to locate question-specific visual tokens from shallow layers, enhancing multi-modal video understanding efficiently without compromising performance, and demonstrating superior results across multiple benchmarks.

weaknesses:
Impaired Relative Position Understanding: The proposed method appears to diminish the model's ability to capture relative positional information. This is indicated by performance drops in the Object Spatial (OS) and Motion Direction (MD) metrics on the MV-Bench dataset for both PLLaVA with the proposed approach and Tarsier with LLM-VTP. Could the authors explain why the method lacks proficiency in calculating relative positions? If the method is effective in this aspect, please provide experimental evidence or references that support this capability.

Evaluation on Long-Form Video Understanding: The paper does not address how the proposed model performs on long-form video understanding tasks, which require the model to handle challenging temporal complexities. Evaluating the method on datasets containing very long-form videos, such as a subset of EgoSchema, and comparing it to the original models like Tarsier, would provide valuable insights into whether localization by attention and pruning is sufficient to induce long temporal relations. Such an evaluation would demonstrate the model's applicability and effectiveness in handling extended video sequences, highlighting its strengths or revealing areas for improvement.

questions:
I asked questions in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704440732

thread_type:
official_review

<thread_object>
creator: reviewer_g3xm

<content>
summary:
The paper proposed a new method to improve the inference efficiency of LLMs for long video understanding. The core idea of the paper is very straightforward. Due to the redundancy of the video data, the LLMs do not need to process every token from the video sequence. By pruning video tokens, the inference speed of the LLMs can be improved. The criteria for pruning used by the paper is the learned attention weights between the query questions and extracted video tokens. In combination with techniques such as KV cache, the paper is able to achieve 1.4x faster inference speed without sacrificing the accuracy. Experimental results are provided using two baseline LLMs (PLLaVA and Tarsier).

strengths:
The idea of the paper is very intuitive and straightforward, and can also generalize to different LLMs which make the paper more applicable in real-world applications.

The paper provided thorough experimental results and ablation study to justify the effectiveness of the proposed method.

The writing of the paper is clear and easy to follow.

weaknesses:
Based on the results from Figure 3, the choice of the target layer M seems to be very sensitive. It also seems to be data dependent as well. This makes it tricky to pick a propper M without extensive experiments, which defeats the whole purpose of a training-free approach.

The paper motivated the idea from the application of the long video understanding. However, the paper didn’t provide any analysis about how well the method works on long videos.

questions:
Please consider addressing my questions in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732275940434

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear reviewers and area chairs,

We sincerely thank all the reviewers and the area chair for their time and efforts in reviewing our paper. We are delighted that the reviewers find the paper to be well-written (Reviewers G3XM, sEPg, and iKaN), the method to be well-designed and intuitive (Reviewers G3XM, aFxJ, sEPg, and 27Y2), and the experiments to be comprehensive (Reviewers G3XM, aFxJ, 27Y2 and iKaN).

We have provided experiments, analyses, and clarifications to respond to each reviewer’s comments, respectively. We hope that our response has addressed all the concerns and we will be more than happy to address further concerns if there are any.

Below, we would like to highlight the major updates made to the paper during the discussion period.

* **Sec. 4.5**: Following the question raised by Reviewer G3XM, sEPg, and 27Y2, we have included additional experiments (Fig. 3 and Fig. 4) and analysis (Sec. 4.5) on studying the effects of the hyper-parameters.

* **Sec. 4.5**: Following the suggestion proposed by Reviewer iKaN, we have included the comparison of computational cost in Tab. 3, and corresponding analysis in Sec. 4.5.

* **App. A.1**: Following the question raised by Reviewer G3XM, aFxJ, and sEPg, we have included experiments on long-form video benchmarks in Tab. 7 and corresponding analysis in App. A.1.

* **App. A.1**: Following the question raised by Reviewer sEPg and 27Y2, we have included experiments with two additional video LLMs in Tab. 8, and corresponding analysis in App. A.1.

* **App. A.1**: Following the question raised by Reviewer iKaN, we have included a comparison of how well the token pruning methods can maintain baseline results in Tab. 9 and the corresponding analysis in App. A.1.

* **App. A.2**: Following the suggestion proposed by Reviewer iKaN, we have included a side-by-side token selection visualization in Fig. 6, and corresponding analysis in App. A.2.

* **App. A.2**: Following the suggestion proposed by Reviewer iKaN, we have included a comparison of attention map visualization in Fig. 7 and Fig. 8, and corresponding analysis in App. A.2.

Thanks,

Paper6361 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732474391293

thread_type:
official_comment

<thread_object>
creator: area_chair_y2kf

<content>
comment:
Dear Reviewers,

This is a friendly reminder that the discussion period will end on Nov 26th (Anywhere on Earth). If you have not already, please take a careful look at the other reviews and author responses, and comment on whether your original rating stands. Thank you.

Best, AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732810106305

thread_type:
official_comment

<thread_object>
creator: area_chair_y2kf

<content>
comment:
Dear reviewers,

This is a friendly reminder that the discussion period has been extended until December 2nd. If you haven’t yet, we kindly encourage you to review the authors' rebuttal and messages at your earliest convenience and confirm whether your comments have been adequately addressed.

We greatly appreciate your service to this process.

Best, AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
To reduce the computational overhead from video redundancy, this paper proposes LLM-VTP, which tries to leverage the reasoning abilities of LLMs to identify and prune visual features by focusing attention on video frames relevant to the question.  The initial scores were 5,5,5,5,6.  Strengths include intuitive idea, clear writing, thorough experiments, and promising results.  Weaknesses include sensitivity to the hyperparameter M, lacking long video analysis, impaired relative positional information, more in-depth analysis needed including different types of backbones, overclaiming statement on LLM reasoning, and limited technical contribution.  Several of these concerns, including long video analysis, more in-depth experiments, sensitivity to M, were adequately addressed, leading to two reviewers increasing their scores for a final score of 5,5,5,6,8.  However, remaining concerns included overclaimed contributions and the approach in terms of pruning design choices.  This is a borderline paper, with split reviews, and largely borderline ratings except for one accept (8) review.  The AC carefully considered the paper, reviews, rebuttal, discussion, and author messages, and unfortunately, feels that the weaknesses outweigh the strengths, and that the paper does not yet meet the bar for publication at ICLR.  The AC would like to encourage the authors to improve the paper and resubmit to a future venue.