Understanding Long Videos with Multimodal Language Models

Authors: Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, Michael S Ryoo
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: long-video, visual question answering, interpretability

Abstract:
Large Language Models (LLMs) have allowed recent LLM-based approaches to achieve excellent performance on long-video understanding benchmarks. We investigate how extensive world knowledge and strong reasoning skills of underlying LLMs influence this strong performance. Surprisingly, we discover that LLM-based approaches can yield surprisingly good accuracy on long-video tasks with limited video information, sometimes even with no video-specific information. Building on this, we explore injecting video-specific information into an LLM-based framework. We utilize off-the-shelf vision tools to extract three object-centric information modalities from videos, and then leverage natural language as a medium for fusing this information. Our resulting Multimodal Video Understanding (MVU) framework demonstrates state-of-the-art performance across multiple video understanding benchmarks. Strong performance also on robotics domain tasks establishes its strong generality. Code: github.com/kahnchana/mvu

Decision: accept (poster)
Reviewer scores: 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730150375665

thread_type:
official_review

<thread_object>
creator: reviewer_naus

<content>
summary:
This paper proposes to inject video-specific information into an LLM based framework for video understanding. Specifically, the authors utilize off-the-shelf vision tools to extract three object-centric information modalities from videos and then leverage natural language as a medium to fuse the information. I think the proposed method seems novel and interesting. But I think this paper leaks a comparison with recent papers and the performance is not good enough.

strengths:
The proposed method seems novel and I think that makes sense for video understanding. But I think the key frame selection plays the most important role for video understanding -- it seems the authors do not propose new method on this.

weaknesses:
1.Figure 1 caption: “(left-right)” -> “(left-bottom)”;
2.I think the authors used both \cite{} and \citep{} in their writing;
3.Missing comparison with recent methods, for examples the publish papers in 2024. I think that is necessary to better your paper;
4.The performance seems not so good, only a marginly superiority compared to the counterparts even without comparing the recent methods. I think the performance is not good;
5.How to select the frames for most relevant frames? Is there any novelty to do that -- I think this is the most important part for video understanding.

questions:
See the fifth point of weakneses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644740152

thread_type:
official_review

<thread_object>
creator: reviewer_a8nc

<content>
summary:
This paper proposes a training-free approach to understanding long-form videos by extracting explicit image-/object-level information. The extracted information is translated into natural language descriptions to make LLM ‘see’ the visual input. Experimental results on several videoQA benchmarks demonstrate the superiority.

strengths:
1. The proposed likelihood selection approach offers a good way to speed up the inference in autoregressive LLMs. 
2. Going beyond existing video datasets, this paper further evaluates the generalization ability on Open-X-Embodiedment.
3. The paper is well written and easy to follow.

weaknesses:
1.	It seems that using likelihood as a selection criterion still focuses on the exact match between the generated text and the answer candidates in a per-logit manner, without considering the semantic meaning. For example, ‘C is washing plates’ vs ‘C is cleaning dishes’. 
2.	Unlike prior approaches where all answer candidates are fed together to the language model, the proposed likelihood selection method organizes the Q-A pairs in a batch dimension. In this way, it seems that LLM fails to analyze the relationship between answer candidates, increasing the difficulty of QA.
3.	If the frames are uniformly sampled across the entire long video, how can you ensure the consistent occurrence of objects? In certain cases, the appeared objects in each frame are completely different. Another related question is whether using X_{OSL} and X_{OMT} extracted from densely sampled frames (i.e. with better object/trajectory consistency) would lead to performance gain. 
4.	The authors are encouraged to evaluate the model on more long-video QA benchmarks, especially those designed to mitigate the language bias of existing long-video QA benchmarks (e.g., EgoSchema). 
5.	In Tables 2 and 3, a series of recent state-of-the-art approaches are not compared. [1][2]

[1] Juhong Min et al. MoReVQA: Exploring Modular Reasoning Models for Video Question Answering, CVPR 2024.

[2] Xiaohan Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent, ECCV 2024.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730784488203

thread_type:
official_review

<thread_object>
creator: reviewer_v8po

<content>
summary:
This paper introduces two baselines and a novel approach called the Multimodal Video Understanding (MVU) framework for video understanding tasks. The baselines explore using either a single frame or no visual input at all. In contrast, MVU aggregates multimodal information relevant to the video to enhance understanding.

strengths:
+ It is interesting that this work introduces three key attributes essential for video understanding: Global Object Information, Object Spatial Location, and Object Motion Trajectory. These attributes contribute significantly to a more comprehensive analysis of video content.

weaknesses:
- Although this paper focuses on long-video understanding, it lacks specific design elements to address long-video scenarios. Challenges such as context length limiting input frames and modeling long-range temporal information are not directly addressed. The attributes introduced—Global Object Information (GOI), Object Spatial Location (OSL), and Object Motion Trajectory (MOT)—are not tailored to tackle these issues in long-video understanding.

- In Table 1, the comparisons may be weaker due to the differing training setups and base models used across experiments. As such, the superior performance of SF-LLM over the state-of-the-art does not necessarily imply that the number of frames is irrelevant for video understanding.

- Since the model is based on LLaVA-v1.5-13B, an important baseline is missing: the use of multiple frames as input for LLaVA-v1.5-13B.

- Certain details are lacking, such as the method for using LLaVA-v1.5-13B for frame sampling, as mentioned in Figure 3.

- Likelihood Selection is widely used in the MCQ benchmark as an additional track. For a fairer comparison, other methods should also incorporate this strategy for comparison results.

- The benchmark in this paper only includes mid-length videos, roughly under three minutes. To more competitively demonstrate MVU's capabilities in long-video understanding, it would be beneficial to evaluate on benchmarks like VideoMME and LongVideoBench.

questions:
As mentioned in the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732325103636

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal

comment:
We first thank all reviewers for the positive feedback: 
*Likelihood selection approach offers a good way to speed up the inference. Paper further evaluates the generalization ability* (**R-a8nc**). 
*Proposed method seems novel and I think that makes sense for video understanding* (**R-NaUs**).

Next we reiterate our key contributions as,
1. Highlighting performance of modality constrained baselines
2. Adapting likelihood selection to long video tasks
3. Video QnA framework MVU

We highlight how the proposed MVU is a framework that can be combined with existing approaches. Our main comparison was to LLoVi, where integrating MVU achieves both accuracy and inference speed improvements. These results cover multiple datasets including in the robotics domain establishing generality of MVU. 

In our rebuttal, we provide evaluations for MVU combined with a more recent approach (LVNet) that involves explicit frame selection and achieves much stronger results. The combined LVNet+MVU variant further improves performance. This highlights how MVU can be integrated with existing methods for additional improvement.  

We also highlight how MVU, especially with more dense sampling of frames, implicitly acts as an information bottleneck similar to frame selection works. 

We provide more details on these as we respond to each reviewer. We hope the rebuttal is able to better establish the contributions of our paper and how it can be useful to the computer vision community.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces the Multimodal Video Understanding (MVU) framework, which leverages object-centric information (Global Object Information, Object Spatial Location, and Object Motion Trajectory) to enhance video understanding by translating visual data into natural language for use in large language models (LLMs). While the novel approach demonstrates marginal improvements in mid-length video QA benchmarks, reviewers highlight several limitations. Key concerns include the lack of focus on long-video-specific challenges, such as modeling long-range temporal dependencies, and the absence of comparisons with recent state-of-the-art methods. Additionally, the methodology lacks innovation in critical areas, such as frame selection, and the performance gains are only marginal. The paper is generally well-written but would benefit from evaluations on more competitive long-video QA benchmarks and more substantial comparisons to recent advancements. Overall, the work is promising but falls short in execution and impact.