Visual Context Window Extension: A New Perspective for Long Video Understanding

Authors: Hongchen Wei, Zhenzhong Chen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Multimodal Models, Long Video Understanding, Visual Context Window

Abstract:
Large Multimodal Models (LMMs) have demonstrated impressive performance in short video understanding tasks but face great challenges when applied to long video understanding. In contrast, Large Language Models (LLMs) exhibit outstanding capabilities in modeling long texts. Existing work attempts to address this issue by introducing long video-text pairs during training. However, these approaches require substantial computational and data resources. In this paper, we tackle the challenge of long video understanding from the perspective of context windows, aiming to apply LMMs to long video tasks without retraining on long video datasets. We first conduct an in-depth analysis of why pretrained LMMs struggle to understand lengthy video content, identifying that discrepancies between visual and language modalities lead to different context windows for visual and language tokens, making it difficult to directly extend the visual tokens to match the language context window. Based on this, we propose to adapt LMMs for long video understanding tasks by extending the visual context window, eliminating the need for retraining on large-scale long video datasets. To further mitigate the significant memory consumption caused by long sequences, we introduce a progressive pooling inference strategy that selectively adjusts the spatial resolution of frame embeddings, reducing the number of visual tokens while retaining important spatial information. Across multiple long video understanding benchmarks, our method consistently improves the performance as the number of video frames increases. On the MLVU benchmark, our method outperforms GPT-4o, even though our model size is only 7B. Additionally, in the 256-frame setting, our method reduces memory usage by approximately 45% compared to the baseline, without introducing any performance loss.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729199399214

thread_type:
official_review

<thread_object>
creator: reviewer_ele1

<content>
summary:
This paper proposes visual context window extension of MLLMs for long video understanding, allowing them to process longer video sequences without retraining on long video datasets. Additionally, the paper uniformly pooling several frame tokens for memory concern. The method is evaluated across several benchmarks, showing competitive performance, surpassing GPT-4o on the MLVU benchmark.

strengths:
+ The method adapt YaRN, originally designed for extending language context windows, to handle visual context windows separately.
+ The performance is impressive on several long-video benchmarks such as VideoMME, MLVU and LongVideoBench
+ The finetuning process does not require long video datasets.

weaknesses:
+ Several theoretical insights are missing. Specifically, it is unclear why a uniformly sampled group is assumed to represent an event, and the rationale behind choosing to maintain the first frame at a high resolution is not explained. For further details, refer to questions 1, 2, 3.
+ The contribution is limited in terms of progressive pooling. Firstly, there is no progressive process. The method simply maintains the first frame at high resolution while spatially pooling the subsequent frames to a lower resolution. 
+ The proposed strategy seems hard to handle the scenarios when the input consists of interleaved visual and text embeddings.

questions:
1. In Figure 1(b), the diagram illustrates the modality gap between visual and language embeddings, which leads to the visual context window extending beyond its range, resulting in a decrease in performance. Could you clarify how extending the visual context helps to narrow this gap? 
2. How does the clustering visualization of visual and language embeddings change after implementing the proposed window extension strategy?
3. In line 100, the author states that they "believe the first frame preserves rich information" and in line 264, they "assume each uniform group represents an event." These statements do not seem technically sound, and the author does not provide substantial evidence to support them. Given that a video contains dynamic content, a uniform sampling strategy might not effectively separate distinct events. Could you explain why the first frame is considered to preserve rich information?
4. Does this visual/text context extension support the interleaving of visual and text features?
5. How does the LLaVA-OneVision model perform when utilizing the full context length of Qwen2, instead of uniformly sampling 32 frames on MLVU and LongVideoBench?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454615519

thread_type:
official_review

<thread_object>
creator: reviewer_rzut

<content>
summary:
The paper focuses on adapting LMMs for long videos understanding. The authors propose extending the visual context window to adapt LMMs for long videos. They introduce a pooling strategy to reduce memory usage by selectively lowering the resolution of frame embeddings. This approach improves performance on long video tasks, while cutting memory usage by 45% without losing accuracy.

strengths:
1.  The motivation is clear. Figure (a) effectively illustrates the challenge of long video understanding due to the limited visual context window.

2.  The proposed visual context window extension significantly improves baseline performance without additional tuning.

3.  The paper is well-written and easy to follow.

weaknesses:
Overall, the technical contribution appears limited, as it primarily involves modifications to YaRN.

Additionally, Table 3 shows that the proposed method performs worse on short videos.

questions:
The authors fixed the input frames to 256/512. Could this approach be extended to handle more frames?

What are the results of fine-tuning on 256 frames and then extending to a higher number of frames?

Could the authors provide results on MVbench? I'm interested in whether this visual context extension impacts the model's ability to understand short videos.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538269740

thread_type:
official_review

<thread_object>
creator: reviewer_yzzs

<content>
summary:
The paper introduces a novel approach called Visual Context Window Extension for enhancing long video understanding using Large Multimodal Models (LMMs). It addresses the challenge by extending the visual context window without the need for retraining on long video datasets, leveraging the differences between visual and language modalities. The paper also proposes a progressive pooling strategy to reduce memory consumption, leading to improved performance and efficiency in long video understanding tasks.

strengths:
This paper demonstrates a significant strength in addressing the limitations of Large Multimodal Models (LMMs) in long video understanding by proposing the Visual Context Window Extension, which leverages the inherent differences between visual and language modalities. 
Another strength is the introduction of a progressive pooling strategy that effectively reduces memory consumption by approximately 45% without sacrificing performance, making the approach more feasible for practical applications.
 Lastly, the paper's approach shows consistent improvements in performance as the number of video frames increases, outperforming other models on benchmarks like MLVU, which is a substantial contribution to the field of long video understanding.

weaknesses:
1. The progressive pooling strategy may also result in some loss of information.

2. The paper lacks a detailed overall framework diagram to showcase the entire process.

3. The paper may not be entirely fair in comparison with methods that use fewer frames, as a higher number of frames inherently contains more information.

questions:
1. The progressive pooling strategy may also result in some loss of information.

2. The paper lacks a detailed overall framework diagram to showcase the entire process.

3. The paper may not be entirely fair in comparison with methods that use fewer frames, as a higher number of frames inherently contains more information.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621562100

thread_type:
official_review

<thread_object>
creator: reviewer_lsst

<content>
summary:
This paper investigates the application of large language models (LLMs) for long video understanding. It first identifies that discrepancies between visual and language modalities lead to different context windows for visual and language tokens. Inspired by the YarN approach, the paper proposes an improvement to the visual context window for video understanding models using RoPE encoding. Additionally, to address the issue of high token count and the associated training and inference overhead for long videos, a progressive pooling strategy is introduced, reducing memory consumption by 45%. The proposed model achieves state-of-the-art results on benchmarks such as MLVU.

strengths:
1. Effective adaptation and optimization of YarN for long video understanding. The paper highlights the need for different context windows for vision and language, enhancing long video comprehension.

2. The progressive pooling strategy with dual compression ratios reduces training and inference costs while retaining dynamic information.

weaknesses:
1. More ablation study and analysis should be done regarding progressive pooling strategy, raising doubts about the role of high-compression frames (see questions 3 and 4).

2. If the answer to Question 1 is "the same context window is used for both language and video parts in video question answering tasks", this part seems more like parameter tuning specific to a particular model and training data setting. It has some contributions but may lack novelty.

questions:
1. Regarding the visual context window extension, some clarification is needed to prevent misunderstanding. In the context of video question-answering, which of the following interpretations is correct?
- Interpretation 1: The visual context window and language context window coexist. In other words, during video question-answering, RoPE encoding for the visual part uses the visual context window for interpolation, while the textual part uses the language context window.
- Interpretation 2: The same visual context window is used for interpolation for both the visual and textual parts during video question-answering.

If Interpretation 1 is correct, how is attention computed between the visual and textual parts given the difference in relative position encoding?

2. According to the citation of allava dataset, it seems to be an image dataset. How can it be used to fine-tune the video understanding model in the paper?

3. In the ablation study of progressive pooling strategy, why does (2,4),4 perform better in some cases compared to (2,2),0?

4. Why does (2,8),4 outperform (2,4),4? Moreover, given that 8 is a very high compression ratio, if the ViT patch size is 14×14, this would compress a 224×224 image frame down to a length of $224/14/8=2$ per side, equating to 4 tokens. Since inter-frame pooling is independent, there is a possibility that dynamic information is not captured.
Could it be that the highly compressed frames are actually not contributing? A simple experiment would be, comparing with (2,2), 0 with 64 sampled frames. 

5. Will the code and model checkpoints be open-sourced?

If all the questions above are solved, I would raise my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732810141893

thread_type:
official_comment

<thread_object>
creator: area_chair_5hed

<content>
comment:
Dear reviewers,

This is a friendly reminder that the discussion period has been extended until December 2nd. If you haven’t yet, we kindly encourage you to review the authors' rebuttal and messages at your earliest convenience and confirm whether your comments have been adequately addressed.

We greatly appreciate your service to this process.

Best, AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the challenge of long video understanding by redefining the context window of LMMs into visual and language components, enabling the use of a visual context window extension to apply LMMs trained on short videos to long video tasks without fine-tuning.  The final scores are 5, 5, 6, 6.  Strengths include good motivation, finetuning process not requiring long video datasets, and promising results.  Weaknesses include limited novelty, limited experiments, lacking theoretical insights, some issues with the approach, and weak results on short video datasets.  The rebuttal addressed several of these concerns, including evaluation on interleaving of visual and text features, the key benefit of the visual context length extension, theoretical insights, and good results with 256 frames, which led to a reviewer increasing their score.  However, some concerns, including limited novelty were not adequately addressed.  The AC carefully reviewed the paper, rebuttal, and messages between the authors and reviewers.  Unfortunately, the AC feels that the weaknesses outweigh the strengths, and that the paper is not ready for publication.  The AC encourages the authors to revise and improve the paper and resubmit to a future conference.