Visual Context Window Extension: A New Perspective for Long Video Understanding

## Abstract

Abstract Large Multimodal Models (LMMs) have demonstrated impressive performance in short video understanding tasks but face great challenges when applied to long video understanding.
In contrast, Large Language Models (LLMs) exhibit outstanding capabilities in modeling long texts.
Existing work attempts to address this issue by introducing long video-text pairs during training.
However, these approaches require substantial computational and data resources.
In this paper, we tackle the challenge of long video understanding from the perspective of context windows, aiming to apply LMMs to long video tasks without retraining on long video datasets.
We first conduct an in-depth analysis of why pretrained LMMs struggle to understand lengthy video content, identifying that discrepancies between visual and language modalities lead to different context windows for visual and language tokens , making it difficult to directly extend the visual tokens to match the language context window.
Based on this, we propose to adapt LMMs for long video understanding tasks by extending the visual context window, eliminating the need for retraining on large-scale long video datasets.
To further mitigate the significant memory consumption caused by long sequences, we introduce a progressive pooling inference strategy that selectively adjusts the spatial resolution of frame embeddings, reducing the number of visual tokens while retaining important spatial information.
Across multiple long video understanding benchmarks, our method consistently improves the performance as the number of video frames increases.
On the MLVU benchmark, our method outperforms GPT-4o, even though our model size is only 7B.
Additionally, in the 256-frame setting, our method reduces memory usage by approximately 45% compared to the baseline, without introducing any performance loss.
Project page: https://hcwei13.github.io/Visual-Context-Window-Extension/

## 1 Introduction

Large Multimodal Models (LMMs), built on pre-trained Large Language Models (LLMs) and trained on massive image-text pairs, have shown remarkable capabilities in image understanding (Li et al., ; Gao et al., ; Dai et al., ; Zhu et al., ; Ye et al., ; Li et al., ; Liu et al., ).
Recently, by segmenting high-resolution images into multiple sub-images for input, LMMs have not only improved in fine-grained image understanding but also demonstrated zero-shot video understanding capabilities (Liu et al., ; Yao et al., ; Li et al., ).
Despite these advancements, current LMMs are still limited to short video understanding tasks and face difficulties when applied to long videos due to the excessive sequence lengths involved.

Several approaches (Li et al., ; Jin et al., ; Song et al., ) have explored using visual resamplers to reduce the number of visual tokens, allowing the models to process more video frames.
However, this token reduction inevitably leads to a loss of critical information, negatively affecting performance.
Recent efforts (Xue et al., ; Liu et al., ) have tackled this issue by incorporating long video-text pair datasets during pre-training.
However, this approach faces significant challenges due to the high computational cost associated with the quadratic complexity of the attention mechanism (Vaswani et al., ) and the scarcity of high-quality long video-text data.

Figure: (a)
Refer to caption: x1.png

To alleviate the high computational costs and data collection challenges associated with long video understanding, we approach the problem from the perspective of the context window.
First, we observe that in recent open-source LMMs, language decoders generally support longer language modeling (Yao et al., ; Li et al., ).
For instance, the latest LMM, LLaVA-OneVision (Li et al., ), employs Qwen2 (Yang et al., ) as its language decoder. As illustrated in Figure 1a, the performance of LLaVA-OneVision in language understanding tasks improves consistently as the input sequence length increases (yellow curve). However, for visual understanding tasks, the performance initially improves but then declines as sequence length grows (blue curve).
Further visualization of the latent space inside the language decoder shows that visual and language embeddings form distinct clusters (Figure 1b), indicating significant modal differences in the latent space.
This explains the performance of LMMs on visual understanding tasks shown in Figure 1.
We believe that due to the differences between the visual and language modalities, LMMs pre-trained on short visual sequences cannot directly extrapolate visual tokens to the effective context window size of the language decoder.
Therefore, we redefine the context window in LMMs as two distinct windows: the visual context window, representing the maximum length of visual tokens during pre-training, and the language context window, referring to the maximum length of language tokens during pre-training.

Building on this observation, we propose to extend the commonly used language context window extension method, YaRN (Peng et al., ), to LMMs for long video understanding.
Specifically, we redefine the scaling factor of the base frequency in positional embeddings as the ratio of the visual context window to the target context window.
By modulating the rotational frequency of the positional embeddings, we expand the effective range of the visual context window, enabling LMMs to handle longer video sequences.

Additionally, to alleviate the rapid memory consumption caused by long sequences, we propose a progressive pooling strategy to handle video frame embeddings.
Specifically, considering the redundancy between consecutive frames in the same event—such as a static background—we uniformly sample the video frames into multiple groups. We assume that each group represents an event, and we control the group size through hyperparameters.
In each group, the first frame’s embedding retains a higher spatial resolution, while the subsequent frames are pooled with a larger stride to lower resolutions.
We believe the first frame preserves rich spatial, fine-grained information, while the remaining frames reduce intra-group redundancy.
This approach minimizes the loss of spatial information while reducing the number of visual tokens.

Across multiple long video understanding benchmarks, our method consistently improves performance as the number of video frames increases.
Notably, on the MLVU benchmark (Zhou et al., ), our method outperforms GPT-4o.
Most importantly, our approach does not require retraining, allowing it to benefit from continuous advancements in open-source LMMs.

In summary, our paper makes the following key contributions:

- •
We exploit the modality difference between visual and language tokens in the language decoder to redefine the effective context window in LMMs: the visual context window and the language context window.
- •
We propose a method to extend positional embeddings within the visual context window, enabling LMMs to handle long video tasks without the need for training on long video-text paired data.
- •
We introduce a progressive pooling strategy for visual frame embeddings, mitigating reducing memory consumption in long video sequences.

## 2 Background and Related Work

### 2.1 Rotary Position Embeddings

Rotary Position Embeddings (RoPE) (Su et al., ) introduce a rotation matrix to incorporate relative positional information into the self-attention mechanism, enhancing the model’s ability to capture positional relationships between words.

Given a sequence $\textbf{S}=\{\mathbf{w}_{i}\}_{i=1}^{N}$ with corresponding embeddings $\textbf{E}=\{\mathbf{x}_{i}\}_{i=1}^{N}$, the query and key vectors are computed as: $\mathbf{q}_{m}=f_{q}\left(\mathbf{x}_{m},m\right)$, $\mathbf{k}_{n}=f_{k}\left(\mathbf{x}_{n},n\right)$,
where $m$ and $n$ are positions in the sequence. The unnormalized attention scores are then calculated by dot-producting two vectors: $\mathbf{q}_{m}^{T}\mathbf{k}_{n}$.
To incorporate relative positional information, the query and key vectors are represented in complex form:

$$ $f_{q}\left(\mathbf{x}_{m},m\right)=e^{im\Theta}\left(\mathbf{W}_{q}\mathbf{x}_ {m}\right),\quad f_{k}\left(\mathbf{x}_{n},n\right)=e^{in\Theta}\left(\mathbf{ W}_{k}\mathbf{x}_{n}\right),$ (1) $$

where $\Theta=\operatorname{diag}\left(\theta_{j}=b^{-2j/d},j\in[1,2,\ldots,d/2]\right)$ and $b=10000$ is the diagonal matrix.

In real coordinates, RoPE can be expressed using the following function:

$$ $\displaystyle f_{q}\left(\mathbf{x}_{m},m\right)=\mathcal{R}_{m}\left(\mathbf{ W}_{q}\mathbf{x}_{m}\right)=$ (2) $\displaystyle\left(\begin{array}[]{ccccc}\cos m\theta_{1}&-\sin m\theta_{1}& \cdots&0&0\\ \sin m\theta_{1}&\cos m\theta_{1}&\cdots&0&0\\ 0&0&\cdots&0&0\\ 0&0&\cdots&0&0\\ 0&0&\cdots&\cos m\theta_{d/2}&-\sin m\theta_{d/2}\\ 0&0&\cdots&\sin m\theta_{d/2}&\cos m\theta_{d/2}\end{array}\right)\mathbf{W}_{ q}\mathbf{x}_{m}.$ $$

Therefore, when the word embedding $\mathbf{x}_{m}$ at position $m$ is multiplied by matrix $\mathcal{R}_{m}$, and the word embedding $\mathbf{x}_{n}$ at position $n$ is also multiplied by matrix $\mathcal{R}_{n}$, resulting in the transformed query and key vectors, the attention weights will inherently include the relative positional information.
We provide a more detailed derivation of RoPE in Appendix A.1.

### 2.2 Related work

Large Multimodal Models LMMs typically consist of a visual encoder, a pre-trained LLM, and a modality projection module that converts visual content into token sequences for the LLM.
Leveraging large amounts of high-quality image-text paired data, LMMs have shown strong capabilities in image understanding (Li et al., ; Gao et al., ; Dai et al., ; Zhu et al., ; Ye et al., ; Li et al., ; Liu et al., ; ; Yao et al., ; Li et al., ).
By sampling videos into multiple frames, LMMs can extend to video understanding tasks (Xu et al., ; Chen et al., ; Maaz et al., ; Liu et al., ; Li et al., ; ).
Examples include Video-ChatGPT (Maaz et al., ), VideoChat2 (Li et al., ), and PLLaVA (Xu et al., ), which enhance LMMs’ video understanding through high-quality data and fine-tuning methods.
However, these methods face challenges with long videos due to the large number of visual tokens generated per frame.

To address this, visual token compression methods have been proposed (Li et al., ; Jin et al., ; Song et al., ).
For instance, LLaMA-VID (Li et al., ) uses only two tokens per frame, and MovieChat (Song et al., ) introduces a memory mechanism to compress long video tokens into a fixed size.
These methods, however, often result in information loss.

In recent work, LongVILA (Xue et al., ) attempted to introduce long video-text pairs into the training of LMMs to expand the context window size.
LongVA (Zhang et al., ) expands the context window by continuously training LLMs on long texts, transferring its long text understanding capabilities to long video understanding.
However, they inevitably introduce high computational costs and data collection challenges.

Context Window Extension for LLMs The fixed context length during pre-training limits the inference performance of language models in scenarios involving long sequence inputs.
To address this issue, researchers have proposed a series of RoPE-based language positional embedding extension methods, such as Position Interpolation (PI) (Chen et al., ; kaiokendev, ), NTK Interpolation (bloc97, ), and YaRN (Peng et al., ).
Specifically, PI scales the positions of long texts that exceed the context window down to the original window size.
However, it compresses distances between nearby tokens, which can degrade performance.
NTK interpolation extends the context window by adjusting the rotational speed of RoPE through reducing the base frequency.
Building upon NTK interpolation, YaRN further distinguishes between high-frequency and low-frequency information to accommodate different RoPE embeddings.

Figure: Figure 2: Examples of RoPE embeddings under different context extension methods. Upper: RoPE directly extrapolated beyond the pre-training range. Middle: YaRN interpolating and extrapolating different RoPE dimensions beyond the pre-training range. Down: Our method further distinguishes between visual and language context windows in YaRN, allowing for different interpolation and extrapolation of RoPE dimensions.
Refer to caption: x3.png

## 3 Method

In this section, we first introduce the corresponding modifications of the language position embedding extension method to the visual context window.
We then further discussed another factor that limit long video understanding: memory constraints.

### 3.1 Visual Context Window Extension

In Section 2.1, we describe the commonly used position embedding method in LLMs and LMMs, RoPE (Rotary Position Embedding).
LLMs typically have a fixed context window size, and when the input sequence exceeds this limit, the model struggles to accurately understand positional information, leading to a decline in performance.
As shown in Figure 1, LMMs encounter similar issues when processing long video sequences.

To address this, we adapt the language position embedding extension method, YaRN (Peng et al., ), for the visual context window to better support long video understanding.
Figure 2 illustrates an example of our method.
In our approach, we define the training context length for visual data as $L_{\text{train}}^{v}$ (i.e., visual context window), and the extended context length as $L_{\text{test}}^{v}$.
Consequently, we define the scaling factor $s$ as follows:

$$ $s=\frac{L_{\text{test}}^{v}}{L_{\text{train}}^{v}}.$ (3) $$

Then, we selectively interpolate the hidden dimensions based on the wavelength $\lambda_{i}$ of the RoPE embeddings:

$$ $\lambda_{i}=\frac{2\pi}{\theta_{i}}=2\pi b^{\frac{2i}{d}}.$ (4) $$

Following this, we define $r_{i}=\frac{L_{\text{train}}^{v}}{\lambda_{i}}$ to determine which dimensions require interpolation.
Finally, following YaRN, combining the scaling factor $s$ with the wavelength $\lambda_{i}$, the base frequency is modified as follows:

$$ $\theta_{i}^{\text{new }}=\left[\gamma_{i}+\left(1-\gamma_{i}\right)\frac{1}{s} \right]\theta_{i},\quad\gamma_{i}=\begin{cases}1,&r_{i}>\beta\\ 0,&r_{i}<\alpha\\ \frac{r_{i}-\alpha}{\beta-\alpha},&\text{ otherwise, }\end{cases}$ (5) $$

where, $\alpha$ and $\beta$ are hyperparameters.
When $r_{i}<\alpha$, we apply linear interpolation proportionally based on $s$.
When $r_{i}>\beta$, no interpolation is applied.
For cases between $\alpha$ and $\beta$, we apply a linear interpolation transition.
We provide detailed derivations of context window extension method in Appendix A.2.
It is important to note that our modifications to YaRN are minimal (only redefining the scaling factor $s$), ensuring simplicity and compatibility with various acceleration techniques, such as flash-attention (Dao et al., ).

### 3.2 Progressive Pooling

Figure: Figure 3: Pipeline of progressive pooling strategy.
Refer to caption: x4.png

In this section, we discuss another factor that limits the performance of long video understanding: memory constraints.
Taking LLaVA-OneVision as an example, given a video $V$ uniformly sampled into $N$ video frames, the visual encoder and multimodal projection module process these frames to obtain the video sequence embeddings $F_{v}\in\mathbb{R}^{N\times 729\times d}$.
To reduce the number of visual tokens, LLaVA-OneVision performs bilinear pooling with a stride of 2 on each video frame embedding, which then serves as the input to the LLM decoder.
However, even after bilinear pooling, a video sequence of 256 frames generates 50,176 tokens.

Long sequences contribute to high memory consumption.
Inference in LMMs can be divided into two stages: prefill and decoding.
During the prefill stage, all visual tokens are projected into a high-dimensional space and stored as KVCache for efficient decoding later.
This incurs substantial memory costs.
Even with bilinear pooling, processing 256 frames generates 50,176 tokens, requiring approximately 73 GB of GPU memory.
This greatly limits the deployment of LMMs for long video understanding.

To alleviate excessive memory consumption, we propose a progressive pooling strategy.
As shown in Figure 3, we first uniformly divide the video sequence embeddings $F_{v}$ into multiple groups, with a division stride defined as $K$.
We assume that each group represents an event.
Considering the redundancy between consecutive frames in the same event, such as a static background, we retain only the first frame of each group at a higher spatial resolution.
The remaining frames within each group are stored at a lower spatial resolution using a larger pooling stride.
Specifically, the video sequence embeddings $F_{v}$ are divided into multiple groups, each containing $K$ frames, resulting in a total of $M=\frac{N}{K}$ groups.:

$$ $\{F_{v,i}\}_{i=1}^{N}\rightarrow\{\{F_{v,w,j}\}_{j=1}^{K}\}_{w=1}^{M}.$ (6) $$

In each group, the first frame $F_{v,w,1}$ is retained at high resolution:

$$ $F_{v,w,1}^{\text{high-res}}=\text{Pool}(F_{v,w,1},\text{stride}=s_{h}).$ (7) $$

The remaining frames are pooled at a lower resolution with a larger stride $s_{l}$ ($s_{h}\textless s_{l}$), resulting in:

$$ $\{F_{v,w,j}^{\text{low-res}}=\text{Pool}(F_{v,w,j},\text{stride}=s_{l})\}_{j=2 }^{K}.$ (8) $$

Finally, the processed frames are reassembled into a new video sequence embedding $F_{v}^{\text{new}}$:

$$ $F_{v}^{\text{new}}=\{\{F_{v,w,1}^{\text{high-res}},F_{v,w,2}^{\text{low-res}}, \ldots,F_{v,w,K}^{\text{low-res}}\}_{w=1}^{M}\},$ (9) $$

where $\text{Pool}(\cdot,\text{stride})$ represents the pooling operation with the specified stride.

The progressive pooling strategy significantly reduces the number of visual tokens while preserving the integrity of spatial information.

**Table 1: Performance evaluation on VideoMME (Fu et al., ) benchmark. * indicates the results of reproduction.**
| Methods | Frames | Short | Medium | Long | Overall |
| --- | --- | --- | --- | --- | --- |
| Qwen-VL-Chat-7B (Bai et al., ) | 4 | 46.9 | 38.7 | 37.8 | 41.1 |
| VideoLLaVA-7B (Lin et al., ) | 8 | 45.3 | 38.0 | 36.2 | 39.9 |
| VideoChat2-Mistral-7B (Li et al., ) | 16 | 48.3 | 37.0 | 33.2 | 39.5 |
| VideoLLaMA2-7B (Cheng et al., ) | 16 | 56.0 | 45.4 | 42.1 | 47.9 |
| LLaVA-NeXT-Qwen2-7B (Liu et al., ) | 32 | 58.0 | 47.0 | 43.4 | 49.5 |
| LLaVA-OneVision-7B* (Li et al., ) | 32 | 69.3 | 55.1 | 49.7 | 58.2 |
| Chat-UniVi-V1.5-7B (Jin et al., ) | 64 | 45.7 | 40.3 | 35.8 | 40.6 |
| ST-LLM-7B (Liu et al., ) | 64 | 45.7 | 36.8 | 31.3 | 37.9 |
| LongVA-7B (Zhang et al., ) | 128 | 61.1 | 50.4 | 46.2 | 52.6 |
| LongVILA-8B (Xue et al., ) | 256 | 61.8 | 49.7 | 39.7 | 50.5 |
| Ours | 256 | 72.7 | 58.2 | 52.9 | 61.3 |
| 512 | 71.9 | 58.7 | 51.3 | 60.6 |  |

## 4 Experiments

### 4.1 Experiment Setting

We evaluate the long video understanding capabilities of our method on three key benchmarks: VideoMME (Fu et al., ), MLVU (Zhou et al., ), and LongVideoBench (Wu et al., ).

VideoMME is a widely used benchmark for assessing the ability of LMMs to handle long videos in real-world scenarios.
It divides the test set into three subsets based on video length: short videos ($\textless$ 2 minutes), medium-length videos (4 to 15 minutes), and long videos (30 to 60 minutes), with durations ranging from 11 seconds to 1 hour.

MLVU offers a diverse collection of video lengths, types, and evaluation tasks.
It includes long video understanding tasks (TR: Topic Reasoning, AR: Anomaly Recognition), single-detail long video understanding tasks (NQA: Needle QA, ER: Ego Reasoning, PQA: Plot QA), and multi-detail long video understanding tasks (AO: Action Order, AC: Action Count).
The benchmark includes videos of various types, such as movies, surveillance footage, egocentric videos, cartoons, and game videos, with lengths ranging from 3 minutes to over 2 hours.

LongVideoBench focuses on long-span understanding, particularly on referential reasoning problems that depend on long-frame inputs and cannot be resolved using only a single or sparse frames.
It evaluates videos of varying lengths, including (8s, 15s], (15s, 60s], (180s, 600s], and (900s, 3600s].

### 4.2 Implementation Details

To validate the effectiveness of our approach, we use the latest LMM, LLaVA-OneVision 7B, as the backbone and baseline model. This model employs a classic multimodal encoder-decoder architecture, consisting of a visual encoder (SigLIP (Zhai et al., )), an LLM decoder (Qwen2), and a multimodal projection module (MLP).
For each video frame, the visual encoder and multimodal projection module encode the frame into video sequence embeddings $F_{v}\in\mathbb{R}^{N\times 729\times d}$. Through bilinear pooling with a stride of 2, this is reduced to $F_{v}\in\mathbb{R}^{N\times 196\times d}$.

Following the default settings in YaRN, we set the hyperparameters $\alpha$ and $\beta$ (in Section 3.1) to 1 and 32, respectively.
Previous research (Peng et al., ; Chen et al., ; Ding et al., ) has shown that fine-tuning after interpolation enhances a model’s ability to interpret scaled RoPE embeddings.
Therefore, we compare the results of both tuning-free and fine-tuned approaches.
Specifically, we randomly sample 10K instances from the allava instruction dataset (Chen et al., ) and fine-tune the LLaVA-OneVision language decoder using LoRA (Hu et al., ), setting $lora_r$ to 64 and $lora_\alpha$ to 16.
The learning rate is set to $1e-5$ with a batch size of 1.
In our experiments, unless otherwise stated, we use the tuning-free model to present our results.
The default parameters for the progressive pooling method are: division stride $K=4$, high-resolution pooling stride $s_{h}=2$, and low-resolution pooling stride $s_{l}=8$.

### 4.3 Quatitative Results

**Table 2: The overall performances on MLVU (Zhou et al., ). Two input strategies are used by the LMMs in evaluation: Uniform Sampling, which evenly samples $N$ frames from the video; Frame Rate Sampling (N fps), which samples $N$ frames per second. ${\dagger}$ denotes proprietary models.**
| Methods | Frames | Holistic | Single Detail | Multi Detail | M-Avg |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TR | AR | NQA | ER | PQA | AO | AC |  |  |  |
| GPT-4o^† (OpenAI, ) | 0.5 fps | 87.4 | 74.5 | 64.8 | 57.1 | 65.1 | 56.7 | 46.3 | 64.6 |
| LLaMA-VID-7B (Li et al., ) | 1 fps | 50.8 | 34.5 | 30.1 | 32.7 | 32.5 | 23.9 | 27.8 | 33.2 |
| LLaVA-1.6-7B (Liu et al., ) | 16 | 60.6 | 41.0 | 43.1 | 38.4 | 41.0 | 25.5 | 25.7 | 39.3 |
| InternVL-1.5-7B Chen et al. () | 16 | 78.8 | 67.0 | 52.7 | 43.5 | 54.4 | 32.8 | 23.8 | 50.4 |
| LLaVA-OneVision-7B* (Li et al., ) | 32 | 88.6 | 74.0 | 73.0 | 62.2 | 67.9 | 43.2 | 28.6 | 64.2 |
| TimeChat-7B (Ren et al., ) | 96 | 23.1 | 27.0 | 24.5 | 28.4 | 25.8 | 24.7 | 32.0 | 30.9 |
| LongVA-7B (Zhang et al., ) | 256 | 83.3 | 58.5 | 69.3 | 50.0 | 67.2 | 38.6 | 27.2 | 56.3 |
| MovieChat-7B (Song et al., ) | 2048 | 29.5 | 25.0 | 24.2 | 24.7 | 25.8 | 28.6 | 22.8 | 25.8 |
| Ours | 256 | 87.5 | 74.5 | 76.3 | 65.3 | 75.9 | 52.9 | 31.6 | 68.6 |
| 512 | 87.1 | 76.5 | 75.5 | 65.3 | 76.1 | 52.5 | 37.4 | 69.1 |  |

**Table 3: Performance evaluation on LongVideoBench (Wu et al., ) benchmark.**
| Methods | Frames | Duration Group (s) | Avg |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| (8, 15] | (15, 60] | (180, 600] | (900, 3600] |  |  |  |
| LLaVA-1.5-13B (Liu et al., ) | 8 | 49.0 | 51.1 | 41.8 | 39.6 | 43.4 |
| LLaVA-Next-Mistral-7B (Liu et al., ) | 8 | 53.4 | 57.2 | 46.9 | 42.1 | 49.1 |
| VideoLLaVA-7B (Lin et al., ) | 8 | 43.1 | 44.6 | 36.4 | 34.4 | 39.1 |
| VideoChat2-7B (Li et al., ) | 8 | 49.3 | 49.3 | 39.0 | 37.5 | 39.3 |
| LLaVA-Next-Video-34B (Liu et al., ) | 8 | 57.6 | 61.6 | 48.7 | 45.9 | 50.5 |
| PLLaVA-34B (Xu et al., ) | 8 | 60.1 | 66.8 | 50.8 | 49.1 | 53.2 |
| LLaVA-OneVision-7B* (Li et al., ) | 32 | 68.8 | 70.4 | 54.6 | 48.1 | 56.0 |
| LongVA-7B (Zhang et al., ) | 256 | 57.4 | 60.4 | 47.3 | 44.7 | 49.7 |
| Ours | 256 | 68.8 | 69.2 | 56.1 | 51.2 | 57.5 |
| 512 | 66.1 | 67.4 | 58.5 | 52.1 | 58.0 |  |

Results on VideoMME
Table 1 presents the results on the VideoMME benchmark.
Compared to the baseline model, LLaVA-OneVision, our method shows consistent improvements across all intervals for short, medium, and long videos.
Notably, for long videos, the accuracy improved by 3.2%.
In comparison to the latest long video understanding models, our approach continues to achieve optimal performance.
For instance, compared to LongVILA-8B, which was pre-trained on long video-text pairs, our method demonstrates an improvement of 10.8%.
Crucially, our method achieves these gains without requiring any pre-training or fine-tuning on long video-text pairs.

Results on MLVU and LongVideoBench
MLVU and LongVideoBench are two benchmarks specifically designed to evaluate long video understanding tasks.
Table 2 presents the results on MLVU, where our method significantly outperforms all comparison models, even surpassing GPT-4o.
Table 3 provides the results on LongVideoBench, where test samples are categorized into various duration intervals to highlight different models’ performance in long video comprehension.
Our method shows a slight performance drop in the intervals (8, 15] and (15, 60] when sampling 512 frames compared to the baseline LLaVA-OneVision.
This performance drop in shorter intervals can be attributed to the fact that dense frame sampling results in excessively long input sequences for shorter videos, which leads to attention distraction and degrades model performance.
Using different frame sampling strategies for videos of varying durations can alleviate this issue.

### 4.4 Ablation Studies

**Table 4: Performance evaluation of different context window extension methods on the VideoMME.**
|  | Frames | Short | Medium | Long | Overall |
| --- | --- | --- | --- | --- | --- |
| LLaVA-OneVision-7B | 256 | 64.9 | 53.3 | 50.4 | 56.2 |
| LLaVA-OneVision-7B + YaRN | 256 | 67.6 | 56.3 | 51.7 | 58.5 |
| Ours (Tuning-free) w/o progressive pooling | 256 | 71.6 | 59.1 | 52.2 | 61.0 |
| Ours (Fine-tuning) w/o progressive pooling | 256 | 71.9 | 60.2 | 53.2 | 61.8 |

**Table 5: Ablation studies on the VideoMME benchmark, where all videos are uniformly sampled to 256 frames. Specifically, $s_{h}$ represents the high-resolution pooling stride for the first frame of each group; $s_{l}$ indicates the low-resolution pooling stride for the remaining frames within each group; and $K$ denotes the grouping stride, which refers to the number of frames within each group.**
| $(s_{h},s_{l}),K$ | Memory (GB) | Short | Medium | Long | Overall |
| --- | --- | --- | --- | --- | --- |
| (2, 2), 0 | 73 | 71.6 | 59.1 | 52.2 | 61.0 |
| (4, 4), 0 | 37 | 70.8 | 59.0 | 51.2 | 60.3 |
| (8, 8), 0 | 29 | 68.1 | 56.2 | 49.7 | 58.0 |
| (2, 4), 4 | 45 | 72.4 | 58.3 | 51.3 | 60.7 |
| (2, 8), 4 | 40 | 72.7 | 58.2 | 52.9 | 61.3 |
| (2, 4), 8 | 41 | 70.1 | 57.6 | 50.8 | 59.5 |
| (2, 8), 8 | 35 | 69.7 | 56.4 | 51.4 | 59.2 |
| (2, 4), 16 | 40 | 68.6 | 57.4 | 51.4 | 59.1 |
| (2, 8), 16 | 31 | 70.3 | 56.3 | 50.7 | 59.1 |

To validate the effectiveness of the proposed module, we conducted experiments on VideoMME, focusing on visual context window extension and progressive pooling strategies.

Visual Context Window Extension
Table 4 presents the comparative results under the scenario of uniformly sampling 256 frames, including direct extrapolation, YaRN interpolation, and our method.
It is noteworthy that all results in the table did not utilize the progressive pooling strategy.
The results indicate that using YaRN interpolation improves model performance, confirming the effectiveness of positional interpolation.
Our method, which applies interpolation on the visual context window, achieves a significant performance enhancement compared to YaRN.
Additionally, we fine-tuned the model using 10K image-text pairs after interpolation, further improving model performance.
This aligns with the conclusions drawn from context window extension methods in LLMs.

Progressive Pooling
Table 5 presents the comparative results of different pooling strategies and progressive pooling parameters on VideoMME.
It is important to note that all experiments in the table utilized visual context window extension.
The upper half of the table displays the results of uniform pooling with pooling strides of 2 (the default pooling strategy of the baseline model), 4, and 8.
It is evident that as the pooling stride increases, memory consumption decreases gradually, but performance declines progressively.
The lower half of the table shows the results of our proposed progressive pooling strategy.
We conducted experiments with varying pooling strides and grouping strides, comparing performance under different parameters.
The results indicate that the optimal performance occurs at $s_{h}=2$, $s_{l}=8$, and $K=4$.
In this setting, compared to the baseline method (with a uniform pooling stride of 2), our approach reduces memory usage by approximately 45% while achieving superior performance.
This is because shorter sequence lengths mitigate the issue of attention distraction.
Additionally, we found that the pooling stride has a smaller impact on the model, while the grouping stride has a significant effect.
This may be due to larger grouping strides leading to greater intra-group scene variation, resulting in a loss of spatial information.

### 4.5 Visual Needle-In-A-Haystack

Figure: Figure 4: Visualization of the Needle in the Long Video Haystack Experiment, where green represents correct answers, while red indicates incorrect answers. Left: progressive pooling parameters are set to $s_{h}=2$, $s_{l}=8$, $K=4$. Right: progressive pooling parameters are set to $s_{h}=2$, $s_{l}=4$, $K=4$. Our method enables LMMs, pre-trained on short videos (32 frames), to be extended to 1024 frames without requiring fine-tuning.
Refer to caption: x5.png

As shown in Figure 4, we utilize V-NIAH (Zhang et al., ) to measure the model’s long-context capabilities.
Probes are inserted at different positions within the video, and a question-answering task is conducted; a response is considered correct only when it matches the answer (indicated in green), otherwise, it is deemed incorrect (indicated in red).
It is evident that our method demonstrates outstanding performance across different progressive pooling parameters, effectively extending the model’s visual context window to 1024 frames without requiring fine-tuning.

Figure: Figure 5: Qualitative results from different methods demonstrate that our approach exhibits accurate and detailed video captioning capabilities.
Refer to caption: x6.png

### 4.6 Qualitative Results

Figure 5 illustrates the qualitative results of our method in video captioning.
It is evident that LLaVA-OneVision-7B generates incorrect descriptions when the default input is set to 32 frames.
When directly extrapolated to 256 frames, the model appears to forget information from the middle section of the video, only describing the beginning and the end.
In contrast, our method generates accurate and detailed descriptions for the input video when 256 frames are provided.

### 4.7 Concluding Remarks

In this paper, we address the long video understanding issue from the perspective of context windows, effectively avoiding the resource consumption associated with training from scratch.
By redefining the effective context window of LMMs into visual and language context windows, we propose the visual context window extension.
This approach allows LMMs trained on short videos to be applied to long video understanding tasks without fine-tuning.
Additionally, we introduce a progressive pooling strategy to mitigate memory consumption issues caused by long sequences.
In a 256-frame setting, this strategy reduces memory usage by approximately 45% without introducing any performance loss.
We hope this work will advance research in long video understanding and provide insights for the design of future long video understanding models.

## Appendix A Appendix

### A.1 Rotary Position Embeddings

Rotational Position Embeddings (RoPE) (Su et al., ) introduce a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation, enabling the model to capture the relative positional relationships between words, thereby enhancing its performance in processing sequential data.

Given a sequence $\textbf{S}=\{\mathbf{w}_{i}\}_{i=1}^{N}$, where $N$ represents the sequence length and $w_{i}$ represents the $i$-th word.
Its corresponding word embeddings are $\textbf{E}=\{\mathbf{x}_{i}\}_{i=1}^{N}$, where $\mathbf{x}_{i}$ is the embedding of the $i$-th word.
Before calculating attention, it is necessary to incorporate positional information into the word embeddings and transform them into the query vectors and the key vectors.

$$ $\displaystyle\mathbf{q}_{m}=f_{q}\left(\mathbf{x}_{m},m\right)\in\mathbb{R}^{d },\quad\mathbf{k}_{n}=f_{k}\left(\mathbf{x}_{n},n\right)\in\mathbb{R}^{d},$ (10) $$

where $m$ and $n$ represent different positions, respectively.
Next, attention is computed using the query and key vectors.

$$ $\operatorname{softmax}\left(\frac{\mathbf{q}_{m}^{T}\mathbf{k}_{n}}{\sqrt{d}} \right),$ (11) $$

where $\mathbf{q}_{m},\mathbf{k}_{n}$ are considered as column vectors so that $\mathbf{q}_{m}^{T}\mathbf{k}_{n}$ is simply the Euclidean inner product.

To incorporate relative positional information, we express the inner product between the query and key vectors as a function, denoted as $g(\cdot)$.

$$ $\left\langle f_{q}\left(\mathbf{x}_{m},m\right),f_{k}\left(\mathbf{x}_{n},n \right)\right\rangle=g\left(\mathbf{x}_{m},\mathbf{x}_{n},m-n\right).$ (12) $$

For the function $g(\cdot)$, it is evident that the inner product encodes positional information only in a relative form (i.e., $m-n$).

The next goal is to find an appropriate function $g(\cdot)$ that conforms to the aforementioned relation.
Specifically, we first represent the query and key vectors in complex form.
The representations of the query and key vectors are as follows:

$$ $\displaystyle f_{q}\left(\mathbf{x}_{m},m\right)$ $\displaystyle=e^{im\Theta}\left(\mathbf{W}_{q}\mathbf{x}_{m}\right),\quad f_{k }\left(\mathbf{x}_{n},n\right)$ $\displaystyle=e^{in\Theta}\left(\mathbf{W}_{k}\mathbf{x}_{n}\right).$ (13) $$

For the sake of clarity and ease of understanding in the subsequent formulas, where $\mathrm{i}^{2}=-1$ is the imaginary unit and $\Theta=\operatorname{diag}\left(\theta_{j}=b^{-2j/d},j\in[1,2,\ldots,d/2]\right)$ is the diagonal matrix.
RoPE associates each (complex-valued) hidden neuron with a distinct frequency $\theta_{j}$.
The benefit of this approach is that the dot product between the query and key vectors depends only on the relative distance $m-n$.
This process is represented by the following formula:

$$ $\displaystyle\left\langle f_{q}\left(\mathbf{x}_{m},m\right),f_{k}\left( \mathbf{x}_{n},n\right)\right\rangle$ (14) $\displaystyle=$ $\displaystyle\left\langle e^{im\Theta}\left(\mathbf{W}_{q}\mathbf{x}_{m}\right ),e^{in\Theta}\left(\mathbf{W}_{k}\mathbf{x}_{n}\right)\right\rangle$ $\displaystyle=$ $\displaystyle\operatorname{Re}\left(e^{i\Theta(m-n)}\mathbf{x}_{m}^{*}\mathbf{ W}_{q}^{*}\mathbf{W}_{k}\mathbf{x}_{n}\right)$ $\displaystyle=$ $\displaystyle g\left(\mathbf{x}_{m},\mathbf{x}_{n},m-n\right),$ $$

where $\operatorname{Re}(\cdot)$ is the real part of a complex number and $(\cdot)^{*}$ represents the conjugate complex number of $(\cdot)$.

According to Euler’s formula,

$$ $e^{i(m-n)\Theta}=\cos((m-n)\Theta)+i\sin((m-n)\Theta).$ (15) $$

In real coordinates, RoPE can be expressed using the following function:

$$ $\displaystyle f_{q}\left(\mathbf{x}_{m},m\right)=\mathcal{R}_{m}\left(\mathbf{ W}_{q}\mathbf{x}_{m}\right)=$ (16) $\displaystyle\left(\begin{array}[]{ccccc}\cos m\theta_{1}&-\sin m\theta_{1}& \cdots&0&0\\ \sin m\theta_{1}&\cos m\theta_{1}&\cdots&0&0\\ 0&0&\cdots&0&0\\ 0&0&\cdots&0&0\\ 0&0&\cdots&\cos m\theta_{d/2}&-\sin m\theta_{d/2}\\ 0&0&\cdots&\sin m\theta_{d/2}&\cos m\theta_{d/2}\end{array}\right)\mathbf{W}_{ q}\mathbf{x}_{m}.$ $$

Therefore, when the word embedding $\mathbf{x}_{m}$ at position $m$ is multiplied by matrix $\mathcal{R}_{m}$, and the word embedding $\mathbf{x}_{n}$ at position $n$ is also multiplied by matrix $\mathcal{R}_{n}$, resulting in the transformed query and key vectors, the attention weights will inherently include the relative positional information.
This is because the following identity holds:

$$ $\displaystyle\left(\mathcal{R}_{m}\mathbf{W}_{q}\mathbf{x}_{m}\right)^{\top} \left(\mathcal{R}_{n}\mathbf{W}_{k}\mathbf{x}_{n}\right)$ (17) $\displaystyle=(\mathbf{W}_{q}\mathbf{x}_{m}){\mathcal{R}}_{m}^{\top}{\mathcal{ R}}_{n}(\mathbf{W}_{k}\mathbf{x}_{n})$ $\displaystyle=(\mathbf{W}_{q}\mathbf{x}_{m})^{\top}{\mathcal{R}}_{n-m}(\mathbf {W}_{k}\mathbf{x}_{n}).$ $$

### A.2 Visual Context Window Extension

In this section, we provide a more detailed derivation of the visual context window extension based on YaRN.

Unlike the context window extension methods used in LLMs, we first define the visual context window ($L^{v}_{\text{train}}$), and the extended context window ($L^{v}_{\text{test}}$), with the scale factor $s$ representing the ratio between the two:

$$ $s=\frac{L^{v}_{\text{test}}}{L^{v}_{\text{train}}}.$ (18) $$

Based on the derivation of RoPE, the inner product between the query and key vectors can be expressed in complex form as follows:

$$ $\displaystyle\left({\mathcal{R}}_{m}\boldsymbol{q}\right)^{\top}\left({ \mathcal{R}}_{n}\boldsymbol{k}\right)=\operatorname{Re}\left[\sum_{i=0}^{d/2-1 }\boldsymbol{q}_{[2i:2i+1]}\boldsymbol{k}_{[2i:2i+1]}^{*}e^{\mathrm{i}(m-n) \theta_{i}}\right]$ (19) $$

where $\boldsymbol{q}=\mathbf{W}_{q}\mathbf{x}_{m}$ and $\boldsymbol{k}=\mathbf{W}_{k}\mathbf{x}_{n}$.
According to Euler’s formula, $e^{\mathrm{i}(m-n)\theta_{i}}$ can be represented as a point on the unit circle, where $m-n$ controls the angle on the circle.
Therefore, we define $\lambda_{d}$ as the wavelength of the RoPE embedding in the $d$-th hidden dimension.

$$ $\lambda_{i}=\frac{2\pi}{\theta_{i}}=2\pi b^{\frac{2i}{d}}.$ (20) $$

The wavelength describes the token length required for the RoPE embedding to complete a full rotation $(2\pi)$ in dimension $d$.

Next, we define $r$, which represents the ratio between the original context size and the wavelength.

$$ $r(i)=\frac{L}{\lambda_{i}}.$ (21) $$

This ratio determines which positional dimensions require interpolation.
Following YaRN, we introduces two hyperparameters to control the boundaries of the interpolation strategy.

$$ $\theta_{i}^{\text{new }}=\left[\gamma_{i}+\left(1-\gamma_{i}\right)\frac{1}{s} \right]\theta_{i},\quad\gamma_{i}=\begin{cases}1,&r_{i}>\beta\\ 0,&r_{i}<\alpha\\ \frac{r_{i}-\alpha}{\beta-\alpha},&\text{ otherwise, }\end{cases}$ (22) $$

When $r_{i}<\alpha$, linear interpolation is applied proportionally based on $s$.
When $r_{i}>\beta$, no interpolation is applied.
Otherwise, a linear interpolation transition is applied between the above two cases.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Regarding the visual context window extension, some clarification is needed to prevent misunderstanding. In the context of video question-answering, which of the following interpretations is correct?
- Interpretation 1: The visual context window and language context window coexist. In other words, during video question-answering, RoPE encoding for the visual part uses the visual context window for interpolation, while the textual part uses the language context window.
- Interpretation 2: The same visual context window is used for interpolation for both the visual and textual parts during video question-answering.

If Interpretation 1 is correct, how is attention computed between the visual and textual parts given the difference in relative position encoding?

2. According to the citation of allava dataset, it seems to be an image dataset. How can it be used to fine-tune the video understanding model in the paper?

3. In the ablation study of progressive pooling strategy, why does (2,4),4 perform better in some cases compared to (2,2),0?

4. Why does (2,8),4 outperform (2,4),4? Moreover, given that 8 is a very high compression ratio, if the ViT patch size is 14×14, this would compress a 224×224 image frame down to a length of $224/14/8=2$ per side, equating to 4 tokens. Since inter-frame pooling is independent, there is a possibility that dynamic information is not captured.
Could it be that the highly compressed frames are actually not contributing? A simple experiment would be, comparing with (2,2), 0 with 64 sampled frames. 

5. Will the code and model checkpoints be open-sourced?

If all the questions above are solved, I would raise my score.
soundness: 2
strengths: 1. Effective adaptation and optimization of YarN for long video understanding. The paper highlights the need for different context windows for vision and language, enhancing long video comprehension.

2. The progressive pooling strategy with dual compression ratios reduces training and inference costs while retaining dynamic information.
summary: This paper investigates the application of large language models (LLMs) for long video understanding. It first identifies that discrepancies between visual and language modalities lead to different context windows for visual and language tokens. Inspired by the YarN approach, the paper proposes an improvement to the visual context window for video understanding models using RoPE encoding. Additionally, to address the issue of high token count and the associated training and inference overhead for long videos, a progressive pooling strategy is introduced, reducing memory consumption by 45%. The proposed model achieves state-of-the-art results on benchmarks such as MLVU.
weaknesses: 1. More ablation study and analysis should be done regarding progressive pooling strategy, raising doubts about the role of high-compression frames (see questions 3 and 4).

2. If the answer to Question 1 is "the same context window is used for both language and video parts in video question answering tasks", this part seems more like parameter tuning specific to a particular model and training data setting. It has some contributions but may lack novelty.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The progressive pooling strategy may also result in some loss of information.

2. The paper lacks a detailed overall framework diagram to showcase the entire process.

3. The paper may not be entirely fair in comparison with methods that use fewer frames, as a higher number of frames inherently contains more information.
soundness: 3
strengths: This paper demonstrates a significant strength in addressing the limitations of Large Multimodal Models (LMMs) in long video understanding by proposing the Visual Context Window Extension, which leverages the inherent differences between visual and language modalities. 
Another strength is the introduction of a progressive pooling strategy that effectively reduces memory consumption by approximately 45% without sacrificing performance, making the approach more feasible for practical applications.
 Lastly, the paper's approach shows consistent improvements in performance as the number of video frames increases, outperforming other models on benchmarks like MLVU, which is a substantial contribution to the field of long video understanding.
summary: The paper introduces a novel approach called Visual Context Window Extension for enhancing long video understanding using Large Multimodal Models (LMMs). It addresses the challenge by extending the visual context window without the need for retraining on long video datasets, leveraging the differences between visual and language modalities. The paper also proposes a progressive pooling strategy to reduce memory consumption, leading to improved performance and efficiency in long video understanding tasks.
weaknesses: 1. The progressive pooling strategy may also result in some loss of information.

2. The paper lacks a detailed overall framework diagram to showcase the entire process.

3. The paper may not be entirely fair in comparison with methods that use fewer frames, as a higher number of frames inherently contains more information.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: The authors fixed the input frames to 256/512. Could this approach be extended to handle more frames?

What are the results of fine-tuning on 256 frames and then extending to a higher number of frames?

Could the authors provide results on MVbench? I'm interested in whether this visual context extension impacts the model's ability to understand short videos.
soundness: 3
strengths: 1.  The motivation is clear. Figure (a) effectively illustrates the challenge of long video understanding due to the limited visual context window.

2.  The proposed visual context window extension significantly improves baseline performance without additional tuning.

3.  The paper is well-written and easy to follow.
summary: The paper focuses on adapting LMMs for long videos understanding. The authors propose extending the visual context window to adapt LMMs for long videos. They introduce a pooling strategy to reduce memory usage by selectively lowering the resolution of frame embeddings. This approach improves performance on long video tasks, while cutting memory usage by 45% without losing accuracy.
weaknesses: Overall, the technical contribution appears limited, as it primarily involves modifications to YaRN.

Additionally, Table 3 shows that the proposed method performs worse on short videos.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In Figure 1(b), the diagram illustrates the modality gap between visual and language embeddings, which leads to the visual context window extending beyond its range, resulting in a decrease in performance. Could you clarify how extending the visual context helps to narrow this gap? 
2. How does the clustering visualization of visual and language embeddings change after implementing the proposed window extension strategy?
3. In line 100, the author states that they "believe the first frame preserves rich information" and in line 264, they "assume each uniform group represents an event." These statements do not seem technically sound, and the author does not provide substantial evidence to support them. Given that a video contains dynamic content, a uniform sampling strategy might not effectively separate distinct events. Could you explain why the first frame is considered to preserve rich information?
4. Does this visual/text context extension support the interleaving of visual and text features?
5. How does the LLaVA-OneVision model perform when utilizing the full context length of Qwen2, instead of uniformly sampling 32 frames on MLVU and LongVideoBench?
soundness: 3
strengths: + The method adapt YaRN, originally designed for extending language context windows, to handle visual context windows separately.
+ The performance is impressive on several long-video benchmarks such as VideoMME, MLVU and LongVideoBench
+ The finetuning process does not require long video datasets.
summary: This paper proposes visual context window extension of MLLMs for long video understanding, allowing them to process longer video sequences without retraining on long video datasets. Additionally, the paper uniformly pooling several frame tokens for memory concern. The method is evaluated across several benchmarks, showing competitive performance, surpassing GPT-4o on the MLVU benchmark.
weaknesses: + Several theoretical insights are missing. Specifically, it is unclear why a uniformly sampled group is assumed to represent an event, and the rationale behind choosing to maintain the first frame at a high resolution is not explained. For further details, refer to questions 1, 2, 3.
+ The contribution is limited in terms of progressive pooling. Firstly, there is no progressive process. The method simply maintains the first frame at high resolution while spatially pooling the subsequent frames to a lower resolution. 
+ The proposed strategy seems hard to handle the scenarios when the input consists of interleaved visual and text embeddings.

Meta-review: This paper tackles the challenge of long video understanding by redefining the context window of LMMs into visual and language components, enabling the use of a visual context window extension to apply LMMs trained on short videos to long video tasks without fine-tuning.  The final scores are 5, 5, 6, 6.  Strengths include good motivation, finetuning process not requiring long video datasets, and promising results.  Weaknesses include limited novelty, limited experiments, lacking theoretical insights, some issues with the approach, and weak results on short video datasets.  The rebuttal addressed several of these concerns, including evaluation on interleaving of visual and text features, the key benefit of the visual context length extension, theoretical insights, and good results with 256 frames, which led to a reviewer increasing their score.  However, some concerns, including limited novelty were not adequately addressed.  The AC carefully reviewed the paper, rebuttal, and messages between the authors and reviewers.  Unfortunately, the AC feels that the weaknesses outweigh the strengths, and that the paper is not ready for publication.  The AC encourages the authors to revise and improve the paper and resubmit to a future conference.

Decision: Reject