PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

Authors: Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, Haibo Lu, Jiankun Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video LLM, Prompt-guided Pooling, PPLLaVA

Abstract:
The past year has witnessed the significant advancement of video-based large language models. However, the challenge of developing a unified model for both short and long video understanding remains unresolved. Most existing video LLMs cannot handle hour-long videos, while methods custom for long videos tend to be ineffective for shorter videos and images. In this paper, we identify the key issue as the redundant content in videos. To address this, we propose a novel pooling strategy that simultaneously achieves token compression and instruction-aware visual feature aggregation. Our model is termed Prompt-guided Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three core components: the CLIP-based visual-prompt alignment that extracts visual information relevant to the user's instructions, the prompt-guided pooling that compresses the visual sequence to arbitrary scales using convolution-style pooling, and the clip context extension designed for lengthy prompt common in visual dialogue. Moreover, our codebase also integrates the most advanced video Direct Preference Optimization (DPO) and visual interleave training. Extensive experiments have validated the performance of our model. With superior throughput, PPLLaVA achieves better results on image benchmarks as a video LLM, while achieving state-of-the-art performance across various video benchmarks, excelling in tasks ranging from caption generation to multiple-choice questions, and handling video lengths from seconds to hours. The codes are promised to be made public.

Decision: reject
Reviewer scores: 6, 6, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730656501463

thread_type:
official_review

<thread_object>
creator: reviewer_qkbp

<content>
summary:
The paper introduces a novel approach to video understanding by leveraging prompt guidance to enhance the performance of video language models (VLMs). The methodology focuses on reducing video redundancy and extracting key content to enhance the performance of VLMs. It uses CLIP-based visual-prompt alignment to extract relevant visual information and compresses visual sequences using convolution-style pooling. Direct Preference Optimization (DPO) was also used to improve its performance. In experiments, PPLLaVA demonstrated good results on both long and short video benchmarks, and achieves over an 80% compression rate.

strengths:
1. The paper introduces methods for video understanding by leveraging prompt guidance through the use of CLIP-based visual-prompt alignment and convolution-style pooling.
2. PPLLaVA shows versatility by performing well on both long and short video benchmarks, demonstrating robust performance for varied video sequence understanding. This adaptability is crucial for handling diverse video lengths and complexities, making it a robust solution for varied video sequence understanding5.
3. PPLLaVA uses Direct Preference Optimization (DPO) to reduce hallucinations in video-based dialogue and also applies CLIP Context Extension to expand text encoding capacity.

weaknesses:
1. PPLLaVA does not show leading results compared to some training-free compression methods like SLOWFAST-LLAVA[1], on benchmarks such as MSVD and MSRVTT.
2. The use of Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO) lacks innovation. Also, LLaVA-Next-Video also achieve great accuracy improvements using above methods, but this paper does not highlight any unique advantages of these methods within PPLLaVA. 

[1] SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models

questions:
1. The method seems similar to the involution kernel[1]. What are the  differences between the two? 
2. Have any ablation studies been conducted using regular 3D convolution for pooling instead of prompt-guided methods? 
3. In Table 5, why is the context length for average pooling set to 576 instead of 1024?
4. Can PPLLaVA  extend to multimodal **generative** models?

[1] Involution: Inverting the Inherence of Convolution for Visual Recognition
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703412997

thread_type:
official_review

<thread_object>
creator: reviewer_vfqm

<content>
summary:
This paper proposed PPLLAVA, a video-LLM based on a prompt-guide pooling strategy. The core idea is to conduct text query/prompt-dependent pooling for video features before putting them into LLM. The authors conduct extensive experiments to show the effectiveness of the proposed method. The proposed PPLLAVA outperforms other training-free/training-based video-LLM at a similar model scale.

strengths:
(1) this paper conducts extensive experiments to show the effectiveness of the proposed methods.

(2) pooling redundant tokens based on visual-text similarity is an efficient solution to capture query-related information in a video with large redundancy.

(3) The design of extending the CLIP content window is smart enough to adapt CLIP to a longer context with minimal modification.

weaknesses:
(1) one thing I'm a bit confused about is the video captioning setting when adapting this prompt-guided pooling strategy. As the caption prompts are not as diverse as that question answering, holistic captioning should be affected unless the user is querying a specific object-related caption. I would suggest some experiments on video caption benchmarks, e.g. activitynet caption or newer DREAM-1K, to further support the strong video understanding ability claim.

(2) it seems the paper missed some related baseline methods, see LongVA [1], and Kangaroo [2]. It should be helpful to include a more comprehensive analysis and comparison with those methods, even if some of them achieve better performance on some datasets. 

(3) My other big concern is the novelty side. While I understand this paper did a good job of validating their idea/design with extensive experiments and ablations, comparing some zero-shot baseline like PLLAVA/LLaVA-NeXT-Video, the big boost of the performance might come from fine-tuning on 1.3M multimodal data and the prompt-guided pooling seems a bit incremental from my personal perspective. So I am a bit worried that if we conduct the same level of training with a stronger LLM backbone that has a long content window, the model can learn this query-related attention itself, so the proposed core idea, prompt-guided pooling, seems sub-optimal. 

(4) A potential solution might be to fine-tune zero methods that conduct their token merging idea with the same data and show the performance difference.

[1] Long Context Transfer from Language to Vision
[2] Kangaroo: A powerful video-language model supporting long-context video input

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705321430

thread_type:
official_review

<thread_object>
creator: reviewer_hdf1

<content>
summary:
The paper proposes a novel method named PPLLAVA, which is based on Video LLM. my concerns are as below.
1. The authors claim that the method is for long video, however, I cannot see the comparison of performances when applying the method to long/short videos. Also, how to define "long video"? 1 hour long?
2. I think LLAVA is for images, instead of video. While if the authors want to apply LLAVA to videos, why not do like this: a. locate the related frames; b. VQA on frames. I cannot see the necessity of the three steps way.
3. The authors seems want to satisfy the user's needs/reply to user's questions. What if the questions are open-ended questions without an answer?
4. why we need "Prompt-Guided Pooling"? I cannot see the necessity of prompt here.

strengths:
The paper is well-written, with clear pipeline, framework, and experimental results.

weaknesses:
As can be found in the summary. I cannot see why the approach should be applied to videos instead of image-level applications.

questions:
Will the method also be applied to image-level applications? is video necessary?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706754320

thread_type:
official_review

<thread_object>
creator: reviewer_tf5y

<content>
summary:
This paper proposes a novel pooling method to enhance video large language models (LLMs) by enabling token compression and prompt-aware feature extraction. The approach can be summarized as follows:
- Prompt-Relevant Visual Feature Extraction: The method begins by identifying visual features relevant to the prompt through a fine-grained visual-prompt alignment.
- 3D Token Compression: Leveraging prompt-vision alignment, the authors employ a 3D convolutional kernel to compress tokens to a specified 3D size, adjusting the output stride accordingly.
- Asymmetric Positional Embedding: To further enhance the model's capabilities, the authors introduce asymmetric positional embedding extensions to expand the capacity of text encoding.

The proposed method achieves a significant compression rate and supports both short and long token inputs. In comparison to Q-Former, the authors argue that their approach offers greater flexibility and adaptability.

Extensive experiments across a variety of datasets demonstrate promising performance improvements with this method.

strengths:
- This paper begins with a clear analysis of video LLMs and highlights limitations in existing pooling techniques, effectively motivating the proposed approach.
- The proposed method, which comprises three key components, is technically sound and well-justified, addressing the identified challenges.
- The training strategy is particularly interesting, utilizing detailed video captions as proxies for video content and performing DPO with feedback from the language model serving as a reward signal.
- Experimental results demonstrate substantial performance improvements, supporting the effectiveness of the proposed approach.
- The paper includes detailed analysis, providing insights into the efficiency and performance gains of the model, with evidence suggesting that DPO plays a critical role in achieving the final results.
- The paper is well-written and easy to understand.

weaknesses:
- The diagrams could benefit from more detailed captions to enhance clarity. For example, Figure 2 requires additional context to quickly convey the relationship between [TOK] and the 3D blue rectangle, as this took some time to interpret.
- The fairness of the comparisons is questionable, as many previous works do not utilize DPO in their training. This difference may give the proposed method an advantage, making it challenging to assess its improvements solely based on the new pooling technique.

questions:
- Since the proposed method builds on the image-domain LLaVA, would it be feasible to adapt this approach to existing video LLMs directly? It would be interesting to understand any potential challenges or modifications required for integration with current video-specific models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730735883495

thread_type:
official_review

<thread_object>
creator: reviewer_1y1n

<content>
summary:
The paper proposes a new model that can handle short and long videos with state-of-the-art performance under comparable model sizes. By using DPO's fine-tuning strategy, the proposed PPLLaVA is able to outperform selected baseline models. This is achieved by incorporating three components: the CLIP-based visual-prompt alignment that extracts visual information relevant to the user’s instructions, the prompt-guided pooling that compresses the visual sequence to arbitrary scales using convolution-style pooling, and the clip context extension designed for lengthy prompt common in visual dialogue.

strengths:
1. The paper's motivation is interesting: applying pooling based on the text-frame similarity is reasonable. This is effective on removing redundant video frames conditioned on the user prompt input.
2. The author also proposes methods to handle long user inputs.
3. The final results are promising.

weaknesses:
1. How the capability of CLIP affects the model performance is not discussed. Since the method relay on CLIP to remove redundant video frames, its accuracy could be the bottleneck of the method. Will a stronger text-image matching encoder further improve the model's performance? I believe adding this type pf discussion to the paper will make it stronger.

questions:
Please refer to the weakness section: Will a stronger text-image matching encoder further improve the model's performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733140765767

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Author Response

comment:
We appreciate all the responsible reviewers and chairs, as well as the time they’ve dedicated to reviewing our work. 

It is a pity that Reviewer hDF1 and Reviewer Qkbp did not engage in any effective communication with us throughout the review process. To clarify any potential misunderstandings for others, we will provide a final response to the comments from the two reviewers.

Reviewer hDF1's review is the most absurd we have ever received. The majority of their concerns have already been emphasized multiple times in the paper, and the remaining issues show a lack of basic knowledge about video understanding. It seems that Reviewer hardly read our paper and lacks common sense in the field. Unsurprisingly,  Reviewer hDF1 also did not provide any feedback within 18 days.

Reviewer Qkbp's reason for rejecting our paper can be summarized in two points: 1) The performance on MSRVTT and MSVD is not better than SlowFast-LLaVA; 2) The use of DPO is not novel. For the first point, we have already responded that this is due to the unfair comparison regarding the GPT version. Additionally, we found that training-free methods (FreeVA, IGVLM, SlowFast LLaVA) generally perform well on these two datasets, often comparable to trained methods. Furthermore, recent well-known MLLMs such as LLaVA-Video, LLaVA-OneVision, and LongVU did not report results on these two datasets. Therefore, we believe that rejecting our paper based on results from two non-specialized benchmarks, especially under such unfair comparison conditions, is unjust. For the second point, we have never claimed that the use of DPO is the main contribution or innovation of our work. We are puzzled as to why more implementations and experiments are ultimately criticized for lacking novelty. 

We had hoped that our rebuttal would clear up Reviewer Qkbp's misunderstandings. However, the response, "Thanks for your reply, I'll maintain my score," which came after 14 days, was the shortest response we have encountered. And then, there was no further communication. 

We have remained patient and polite with these two reviewers. However, due to their passive behavior, we were unable to resolve any misunderstandings. Hence, it is greatly appreciated if our work can be fairly judged.

Finally, thank you again for your time and for being part of the review process.

With sincere regards,

Authors of Paper 1865
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes PPLLaVA, a video LLM utilizing prompt-guided pooling to efficiently process short and long videos. The method demonstrates strong performance across multiple video benchmarks. However, reviewers raised several concerns, including incremental  novelty, CLIP dependency, fairness of comparisons and writting issues. Some of these concerns are addressed and confirmed by reviewers, while some of them are not well addressed. 

While the contributions are promising, the paper would benefit from addressing these concerns, including a deeper exploration of motivation and novelty, and a more thorough comparison with related works and baselines. Given the average score of 4.80 and intense competition at ICLR, the AC recommends revising the paper and considering submission to a future conference.