LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

## Abstract

Abstract Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM’s context size. To address this limitation, we propose LongVU , a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.

## 1 Introduction

Large Language Models (LLMs) (Brown, ; Ouyang et al., ; OpenAI, ; Achiam et al., ; Chiang et al., ; Touvron et al., ; Jiang et al., ) manifest universal capabilities that are instrumental in our progress towards general intelligence. Through the integration of modality alignment and visual instruction tuning, Multimodal Large Language Models (MLLMs) (Alayrac et al., ; Li et al., ; Zhu et al., ; Liu et al., ; Ye et al., ; Bai et al., ; Chen et al., ; Dong et al., ) have demonstrated exceptional competencies in tasks such as captioning and visual question-answering. Recent literatures have initiated explorations of extending MLLMs for the comprehension of video content (Li et al., ; Zhang et al., ; Maaz et al., ; Lin et al., ; Wang et al., ; Liu et al., ). Despite exhibiting potentials across specific benchmarks, effectively processing and understanding of exceedingly lengthy videos remains a significant challenge.

Figure: Figure 1: Effectiveness of our LongVU over commonly-used uniform sampling and dense sampling. Uniform sampling overlooks critical frames due to its sparse nature. Dense sampling may surpass the maximum context length, leading to truncation of tokens from targeted frames. In contrast, our method can adaptively conduct spatiotemporal compression, accommodating long video sequences while preserving more visual details.
Refer to caption: x1.png

One primary reason is that it is impractical to process all the information for hour-long videos, given that advanced MLLMs represent a single image using hundreds of tokens. For instance, 576 $\sim$ 2,880 tokens per image are used in LLaVA-1.6 (Liu et al., ) and 7,290 tokens are used in LLaVA-OneVision (Li et al., ). However, a commonly used and computationally manageable context length for multimodal training is 8k, which limits processing 125 frames (2-minutes video) even at 64 tokens per frame, while an hour-long video could require over 200k tokens. Consequently, in video scenarios with an extra temporal dimension, it is intractable for training due to the demand of excessive GPU memory. Various studies have attempted to establish a balance between the number of tokens and the frequency of frame sampling. Most of these studies (Li et al., ; Cheng et al., ; Zhang et al., ; Chen et al., ) opt for a uniform sampling of a fixed number of video frames as the input. However, these methods naively overlook non-uniform content, e.g., static vs dynamic scenes within the video, as shown in Figure 1. Other approaches (Li et al., , ; Jin et al., ) employ intensive resampling modules that significantly decrease the quantity of visual tokens, leading to a considerable loss of essential visual information.

In this paper, we propose LongVU that aims to preserve as much frame information as possible while accommodating lengthy videos without exceeding the context length of commonly used LLMs. Video by its nature contains significant temporal redundancy. MovieChat (Song et al., ) employs a similarity-based frame-level feature selection using visual representation from CLIP (Radford et al., ). While we argue that DINOv2 (Oquab et al., ), through self-supervised training with a feature similarity objective on vision-centric tasks, captures subtle frame differences and low-level visual features more effectively than vision-language contrastive methods (Radford et al., ; Zhai et al., ), as shown in Figure 6. Hence, (1) we apply a temporal reduction strategy on the frame sequence by leveraging similarity from DINOv2 (Oquab et al., ) features to remove redundant video frames. In addition, (2) we jointly capture the detailed spatial semantic and long-range temporal context by performing selective feature reduction via cross-modal query, where we preserve full tokens for frames that are relevant to the given text query, while applying spatial pooling to reduce the remaining frames to a low-resolution token representation. (3) A spatial token reduction mechanism based on temporal dependencies is applied for excessively long videos. As a result, our model is capable of processing 1fps sampled video input with high performance, which can adaptively reduce the number of tokens per frame to 2 on average to accommodate an hour-long video for MLLM within 8k context length.

To evaluate our method, we conduct extensive experiments across various video understanding benchmarks, including EgoSchema (Mangalam et al., ), MVBench (Li et al., ), VideoMME (Fu et al., ), and MLVU (Zhou et al., ). Our LongVU significantly outperformes several recent open-source video LLM models, such as VideoChat2 (Li et al., ), LongVA (Zhang et al., ), and LLaVA-OneVision (Li et al., ), by a large margin. For example, our LongVU outperforms a strong open-source baseline, LLaVA-OneVision (Li et al., ) by approximately $\sim$5% in average accuracy. We also observed that our light-weight LongVU, basing Llama3.2-3B (Llama, ) as the language backbone, significantly improves over previous state-of-the-art small video-LLMs, e.g., Phi-3.5-vision-instruct-4B (Abdin et al., ), by 3.4% on VideoMME Long subset. Our LongVU established new state-of-the-art results on video understanding benchmarks among video-language models. We believe that our proposed approach marks a meaningful progression towards long video understanding MLLMs.

## 2 Related Work

### 2.1 Vision Language Models

Early visual language models (VLMs) such as CLIP (Radford et al., ), is trained with a contrastive loss to project both vision and language embeddings to a shared representation space. SigLIP (Zhai et al., ) takes a sigmoid loss instead, allowing further scaling up training batch size with better performance.

The development of LLMs has significantly advanced VLMs. Kosmos-1 (Huang et al., ; Peng et al., ) introduces an end-to-end framework that integrates visual inputs with LLM in a cohesive training regime. Flamingo (Alayrac et al., ) and BLIP-2 (Li et al., ) merge visual and linguistic features through cross-attention and a Q-Former module, respectively. MiniGPT-4 (Zhu et al., ) and LLaVA (Liu et al., ) simplify the integration by projecting visual features directly into the LLM embedding space using a MLP.

Later studies (Chen et al., ; Peng et al., ; Wang et al., ; Chen et al., ) have expanded LMM applications to broader multi-modal tasks, enhancing spatial perception through visual grounding. Recent efforts (Liu et al., ; Dong et al., ) aim to create general models that unify diverse tasks, employing sophisticated optimization techniques, high-quality multi-task datasets, and complex training strategies to boost performance across extensive vision-language tasks. Cambrian (Tong et al., ) combines features from multiple vision encoders with Spatial Vision Aggregator (SVA) for a more capable MLLM. By exploring different vision encoders, Cambrian (Tong et al., ) finds that SigLIP (Zhai et al., ) is a strong language-supervised model and DINOv2 (Oquab et al., ) performs well on vision-centric tasks.

### 2.2 Video Large Language Models

Recent advancements in MLLMs have broadened their application to video understanding tasks. Video LMMs process videos by extracting and encoding frames, then rearranging these as final video features. Several works (Li et al., , ; Cheng et al., ), use the Q-Former module from BLIP-2 to merge visual and text features, while others (Lin et al., ; Luo et al., ; Ataallah et al., ) concatenate frame features directly.

When processing lengthy videos, the constraint on context length inevitably causes a trade-off between the number of tokens per frame and the number of frames to input. Most existing works (Li et al., ; Ataallah et al., ; Cheng et al., ; Zhang et al., ; Li et al., ) address this challenge by uniformly sampling frames from the video, which, however, results in a significant loss of visual details within the video. Video-ChatGPT (Maaz et al., ) employs pooling modules to reduce data dimensions, enhancing processing efficiency. Other works try to preserve the maximum number of frames in video content. LLaMA-VID (Li et al., ) employs an additional text decoder to embed the text query for cross-attention between frame features and compress the context token to one token per frame, while MovieChat (Song et al., ) and TimeChat (Ren et al., ) develop memory modules and timestamp-aware encoders to capture detailed video content. Golfish (Ataallah et al., ) segments long videos into shorter clips, processes each segment independently, and retrieves the most relevant segment in response to user queries. Our work focuses on maximizing the preservation of frames in video content (1fps) within given context length by proposing spatiotemporal compression of video tokens.

### 2.3 Video Token Compression

Recent methods has explored dynamic image tokens (Ma et al., ; Xu et al., ; Bolya et al., ) or video tokens (Lee et al., ; Ren et al., ; Choi et al., ) within the Transformer (Vaswani, ) framework. Chat-UniVi (Jin et al., ) extends the dynamic tokens for visual features in MLLMs by merging K-nearest neighbor tokens across frame features of the video input. SlowFast-LLaVA (Xu et al., ) uniformly samples 8 frames for high-resolution tokens, while performing spatial pooling to decrease the number of tokens in frames sampled at a higher frame rate. In our work, we propose a spatiotemporal adaptive token reduction strategy that leverages both cross-modal query and inter-frame dependencies. This approach effectively mitigates temporal redundancy in video content, thereby enabling the accommodation of long videos within given context length.

## 3 Method

Figure: Figure 2: Architecture of LongVU. Given a densely sampled video frames, we first utilize DINOv2 (Oquab et al., ) prior to remove redundant frames, and fuse the remaining frame features from both SigLIP (Zhai et al., ) and DINOv2 (Oquab et al., ), described in Section 3.1. Then we selectively reduce visual tokens via cross-modal query, detailed in Section 3.2. Finally, as demonstrated in Section 3.3, we conduct spatial token compression based on temporal dependencies to further meet the context length of LLMs.
Refer to caption: x2.png

We propose spatiotemporal adaptive compression in three steps to effectively process long video, as shown in Figure 2. Initially, we implement a temporal reduction strategy on the frame sequence by leveraging the prior knowledge from DINOv2 (Oquab et al., ) (Section 3.1). Then, we selectively preserve full tokens for key frames via cross-modal query, while applying spatial pooling to reduce the remaining frames into low-resolution token representations (Section 3.2). Furthermore, we implement a spatial token reduction mechanism based on inter-frame temporal dependencies (Section 3.3).

### 3.1 Frame Feature Extractor and Temporal Reduction

DINOv2 (Oquab et al., ), through its self-supervised (SSL) training with a feature similarity objective on vision-centric tasks, can effectively capture subtle frame differences and low-level visual features. In contrast, CLIP-based (Zhai et al., ; Radford et al., ) models are trained with vision-language contrastive loss in the semantic space, excelling at language alignment while sacrificing low-level features as shown in Figure 6. Moreover, Cambrian (Tong et al., ) discovered that combining features from both SigLIP (Zhai et al., ) and DINOv2 (Oquab et al., ) leads to a significant performance boost in vision-centric tasks. Therefore, we pioneer to leverage both SSL-based model DINOv2 (Oquab et al., ) with vision-language contrastive-based model SigLIP (Zhai et al., ) as frame feature extractors for MLLM in video understanding task.

Note that processing the entire long video can be computationally expensive. Given a 1fps-sampled video with $N$ frames, denoted as $I=\{I^{1},...,I^{N}\}$, we first use DINOv2 (Oquab et al., ) to extract features from each frame, leading to a set of DINO features $\{V_{\text{dino}}^{1},\ldots,V_{\text{dino}}^{N}\}$. We then calculate the average similarity $\text{sim}^{i}=\frac{1}{J-1}\sum_{j=1,j\neq i}^{J}\text{sim}(V_{\text{dino}}^{
i},V_{\text{dino}}^{j})$ within each non-overlapping window with $J=8$ frames and reduce frames that exhibit high similarity with other frames. This step significantly reduces video redundancy by temporally compressing the original $N$ frames to $T$ frames, which reduces approximately half of the video frames, as detailed in Section 4.6.

We then extract features of the remaining $T$ frames using SigLIP (Zhai et al., ) vision encoder, resulting in $T$ features $\{V_{sig}^{1},...,V_{sig}^{T}\}$. Subsequently, following Cambrian (Tong et al., ), we combine these two types of visual features via Spatial Vision Aggregator (SVA) (Tong et al., ) that employs learnable queries to spatially aggregate visual features from multiple vision encoders. We denote the fused frames features as $V=\{V^{1},...,V^{T}\}$.

### 3.2 Selective Feature Reduction via Cross-modal Query

After temporal reduction, we obtain a set of fused frame features from both vision encoders, $V=\{V^{1},...,V^{T}\}\in\mathbb{R}^{T\times(H_{h}\times W_{h})\times D_{v}}$, where $H_{h}\times W_{h}$ denotes the spatial dimension of the frame features, and $D_{v}$ indicates the channel dimension of the frame feature after SVA. If the concatenated frame features exceed the given context length, i.e., $T\times H_{h}\times W_{h}\geq L_{max}$, we develop a selective compression strategy for certain frames, in order to capture both the detailed spatial semantic and long-range temporal context.

To achieve this, we propose using text query to help reduce spatial tokens of certain frames from $H_{h}\times W_{h}$ to $H_{l}\times W_{l}$. Given the LLM embedding of the text query $Q\in\mathbb{R}^{L_{q}\times D_{q}}$, where $L_{q}$ is the length of text query and $D_{q}$ is the dimensionality of LLM’s embedding space, we strategically choose $N_{h}$ frames to preserve their original token resolution, while the remaining undergoes a process of spatial pooling to achieve a reduced resolution. The selection mechanism is based on the cross-modal attention scores between each frame feature and the text query. The number of frames to keep original resolution can be formulated as,

$$ $\mathbf{Top}_{N_{h}}\left(\frac{1}{H_{h}W_{h}L_{q}}\sum_{h,w,l}\mathcal{F}(V)Q ^{T}\right),\quad N_{h}=\max\left(0,\frac{L_{\text{max}}-L_{q}-TH_{l}W_{l}}{H_ {h}W_{h}-H_{l}W_{l}}\right),$ (1) $$

where $L_{max}$ is the given context length, $\mathcal{F}(\cdot)$ denotes a multi-layer perceptron (MLP)-based multimodal adapter designed to align visual features with the input space of the LLM. Note that we omit the system prompt in the instruction template for Equation 1 simplification. If $N_{h}=0$, indicating that no frames are selected for retention at their original resolution, we will skip the computation of attention scores and will directly perform spatial pooling across all the frames to the lower resolution.

### 3.3 Spatial Token Compression

As previously discussed, there are cases where the concatenated visual features with low resolution tokens still exceeds the given context length, i.e., $T\times H_{l}\times W_{l}\geq L_{max}$. Under these circumstances, further token compression is necessary. We partition the sequence of frame features into non-overlapping segments with a sliding window of size $K<T$, within which we conduct spatial token compression (STC). The first frame in each window retains its full token resolution. We then compute the cosine similarity between the first frame and subsequent frames within the window, conducting an element-wise comparison of spatial tokens between the first frame and its successors. Spatial tokens that exhibit a cosine similarity $\text{sim}(\cdot,\cdot)$ greater than the threshold $\theta$ with the corresponding tokens of the first frame at the same spatial location will be pruned, which can be formulated as,

$$ $v_{i}^{*}\leftarrow\begin{cases}v_{i}(h,w)&\text{sim}(v_{1}(h,w),v_{i}(h,w)) \leq\theta\\ \emptyset&\text{otherwise}\end{cases},\quad\forall h\in[1,H_{l}],w\in[1,W_{l}] ,i\in[2,K]$ (2) $$

Given that videos often contain significant pixel-level redundancy, particularly in static background, this method allows spatial tokens reduction via temporal dependencies.
We chose the first frame in each sliding window for comparison, assuming DINOv2 (Oquab et al., ) has effectively reduced video redundancy across frames, making each frame less similar. We also tested alternative strategies, like using the middle frame or adaptively selecting based on frame changes (Section 4.5), but these provided similar performance and compression rates. Therefore, we chose the first-frame strategy in each sliding window for its simplicity and effectiveness.

## 4 Experiments

### 4.1 Datasets

We adopt two stages of training in our experiments: image-language pre-training and video-language finetuning. For the image-language pre-training stage, previous methods (Chen et al., ; Peng et al., ; Wang et al., ; Chen et al., ; Liu et al., ; Dong et al., ) usually use two steps for alignment and finetuning. For simplicity, we combine these two steps in one stage using Single-Image data from LLaVA-OneVision (Li et al., ). For video-language finetuning, we utilize a large-scale video-text pairs sourced from several publicly accessible databases. The video training data contains a subset of VideoChat2-IT (Li et al., ), which includes TextVR (Wu et al., ), Youcook2 (Zhou et al., ), Kinetics-710 (Kay et al., ), NExTQA (Xiao et al., ), CLEVRER (Yi et al., ), EgoQA (Fan, ), TGIF (Li et al., ), WebVidQA (Yang et al., ), ShareGPT4Video (Chen et al., ), and MovieChat (Song et al., ) as the long video complementary. All the training datasets are listed in Table 6.

### 4.2 Benchmarks and metrics

We evaluate our model on EgoSchema (Mangalam et al., ), MVBench (Li et al., ), VideoMME (Fu et al., ) and MLVU (Zhou et al., ). VideoMME (Fu et al., ) (1 min $\sim$ 1 hour) and MLVU (Zhou et al., ) (3 mins $\sim$ 2 hours) are long video benchmarks for assessing long video understanding ability. For VideoMME (Fu et al., ), videos are officially split based on duration, which contains a subset of long videos ranging from 30 minutes to 1 hour. We perform standardized evaluations using greedy decoding (num_beams=1) and benchmark our results against other open-source and proprietary models.

Table: Table 1: Results on comprehensive video understanding benchmarks

### 4.3 Implementation Details

We use SigLIP (Zhai et al., ) (so400m-patch14-384) and DINOv2 (Oquab et al., ) as the vision encoder while choose Qwen2-7B (Qwen, ) and Llama3.2-3B (Llama, ) as our language foundation model. We only compute cross-entropy loss for autoregressive text generation. We use AdamW (Loshchilov, ) optimizer with a cosine schedule for all the trainings. In the image-language pre-training stage, we train the model for one epoch with global batch size of 128. The learning rate is set to 1e-5, and the warmup rate is 0.03. The number of tokens per image are set to 576. For the video-language finetuning stage, we train the model for one epoch with global batch size of 64. The learning rate is set to 1e-5, and the warmup rate is 0.03. The maximum number of tokens per frame are set to 144 ($H_{h}=W_{h}=12$), while each might be reduced by our proposed adaptive compression approach ($\leq 64,\,H_{l}=W_{l}=8$). The STC reduction threshold $\theta=0.8$ and the sliding window size $K=8$. Our model is trained on 64 NVIDIA H100 GPUs.

### 4.4 Video Understanding

Quantitative Results. Table 1 presents our experimental results on multiple video understanding benchmarks. Our results compares favorably to all the baselines across various video understanding benchmarks. For example, on VideoMME (Fu et al., ), our LongVU outperforms VideoChat2 (Li et al., ), LLaVA-OneVision (Li et al., ) by 6.0% and 2.4% respectively. Notably, on VideoMME Long subset (Fu et al., ), our model surpasses LLaVA-OneVision (Li et al., ) by 12.8%. These results indicate the strong video understanding capabilities of our model. Note that our model achieves significant improved performance with a much smaller training dataset, comparing to LLaVA-OneVision (Li et al., ) trained on OneVision-1.6M (multi-image, video) that has not yet been made publicly available(^1^11LLaVA-OneVision (Li et al., ) only release single-image set at the time of current submission. [https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data/discussions/6](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data/discussions/6)). With the same video training dataset from VideoChat2-IT (Li et al., ), our LongVU shows much higher performance than VideoChat2 (Li et al., ), $\sim$10% accuracy improvement in average.
Interestingly, we also find that our model can even beat proprietary model GPT4-o (OpenAI, ) on MVBench (Li et al., ) with densely sampled video input and reduce the accuracy gap comparing to proprietary models on other video benchmarks.

We also scale our LongVU with a lightweight LLM, Llama3.2-3B (Llama, ), to further demonstrate the strong video understanding capabilities. We observe the consistent improvement of our light-weight LongVU over baselines in Table 2. Our method outperforms Phi-3.5-vision-instruct (Abdin et al., ) on VideoMME (Long) by margin of 3.4% accuracy. This set of experiments validate the effectiveness of our method even scaling to a smaller size.

Table: Table 2: Results of small-size video language models across video understanding benchmarks.

Qualitative Results. We now provide the qualitative results in Figure 3. Specifically, we demonstrate various video understanding abilities in the examples, such as accurately recognizing the orientation of moving objects in Figure 3(a), providing detailed video descriptions in Figure 3(b), identifying inserted needle frames and conducting action counting in Figure 3(c), and responding precisely to questions about specific frames in an hour-long video in Figure 3(d). These results demonstrate that our model has competing video-language understanding capabilities.

Figure: Figure 3: Examples for various video understanding capabilities of LongVU model. We showcase that our LongVU is able to complete different types of video understanding tasks.
Refer to caption: x3.png

### 4.5 Ablation Studies

Effects of the number of tokens per frame. We ablate the number of tokens in our uniform-sampling baselines. There is a trade-off between the number of tokens per frame and the sampling frequency of frames. Table 3 shows the experimental results when using different number of tokens with different sampling. When applying uniforming sampling, 144 tokens per frame shows better performance than 64 tokens in an 8k context length on VideoMME (Fu et al., ) and MLVU Zhou et al. () while worse on EgoSchema Mangalam et al. (). With 144 tokens per frame, it preserves more visual details, but restricts the total number of frames, i.e., less than 60 frames within 8k context length. This demonstrate that adaptive tokens are needed for better performance across different video benchmarks.

DINOv2 vs SigLIP. Our results in Table 3 verify that DINOv2 (Oquab et al., ) features are more effective than SigLIP (Zhai et al., ) features. As expected, we also find that using DINO-based features for temporal frame reduction outperforms uniform sampling. Therefore, DINOv2 (Oquab et al., ) is an useful vision-centric feature extractor to help perform temporal reduction.

Query guided selection. We apply text-guided frame selection after temporal reduction, where relevant frames are maintained at full token capacity (144 tokens), while others are reduced to 64 tokens. This helps preserve essential visual features and accommodates more long-range context within the context length. In Table 3, we observe the improvement with query guided frame selection across all benchmarks. Moreover, in Table 4, the results of each subtask in MLVU (Zhou et al., ) show significant performance improvements when using cross-modal queries, particularly for frame-retrieval tasks such as counting and needle detection.

Spatial token compression. We further apply spatial token compression after query guided selection. We find that spatial token compression (STC) not only enhances performance within 8k context length, but also achieve results comparable or slightly better than 16k context length in Table 3. We also note some improvements for most subtasks in MLVU (Zhou et al., ).

Table: Table 3: Ablation studies of number of tokens per frame, different context lengths, and our spatiotemporal compression components.

Table: Table 4: Ablation study on each subtask in MLVU (Zhou et al., ).

Different strategies for spatial token compression. We now ablate different strategies of our spatial token compression mechanism. This analysis explores different strategies for determining anchor frames: the first/middle one in each sliding window, or the frame that exhibits significant changes compared to its adjacent frames. In Table 5, our results indicate that taking the first frame in each sliding window gives a slightly better performance with similar reduction rates across all strategies.

Table: Table 5: Different strategies for spatial token compression on VideoMME (Fu et al., ).

### 4.6 Spatiotemporal Compression Analysis

Compression analysis. We sampled hundreds of videos to demonstrate the distribution of frame/token reduction rate. Figure 4 (a) presents the number of frames before and after temporal reduction based on the similarity of DINOv2 features across frames. We find that $\sim$45.9% of the frames are maintained after temporal reduction on average. Figure 4 (b) shows the number of tokens before and after spatial token compression (Section 3.3). We observe that $\sim$40.4% tokens are reduced on average. These results demonstrate the effective video token compression with temporal and spatial token reduction.

Figure: Figure 4: We randomly sample hundreds of videos to demonstrate the frames/tokens level reduction rate. (a) The number of frames before/after temporal reduction based on DINOv2 features (Section 3.1). (b) The number of tokens before/after spatial token compression (Section 3.3).
Refer to caption: x4.png

Figure: Figure 5: Needle-in-a-Haystack results. Our adaptive token compression scheme improves the score for locating the needle frame within an hour-long video from 0.80 to 0.88 on average.
Refer to caption: x5.png

Long context analysis. Recently, the Needle-in-a-Haystack task (Hsieh et al., ; Kamradt., ) has been used to assess the ability of Large Language Models (LLMs) to retrieve long context information. We follow (Zhang et al., ) to conduct a video needle-in-a-haystack experiment to demonstrate the effectiveness of our compression strategy on identifying the needle frame within an hour-long video.

To facilitate this evaluation, we randomly select an one-hour-long test video from MLVU (Zhou et al., ). We then insert each image from a set of VQA problems as a needle frame into this long video for creating a challenging search task. We sample the video at 1 FPS and control the frame length ranging from 200 to 3.6k frames. We also vary the needle frame insertion depth from 0% to 100% of the total input frames. We conduct experiments with 8k context length and compare our adaptive token compression to the one without applying query-guided selection (w/o Query) and spatial token compression (w/o STC) after temporal reduction. Figure  5 demonstrates that our adaptive compression mechanism could accurately resolve the needle VQA problem of 1k frames within 8k context length and improve score with more frames. This demonstrates the advantage of our method for long context video understanding.

## 5 Conclusion

We introduced LongVU, a MLLM that can address the significant challenge of long video understanding within a commonly used context length. To achieve this, we proposed a spatiotemporal adaptive compression scheme of LongVU for helping reduce video tokens without losing much visual details of long videos by leveraging cross-modal query and inter-frame similarities. Experiments on various video understanding benchmarks consistently validate the advantages of our model. We also demonstrate that our method helps build a quality light-weight video language understanding model based on Llama3.2-3B, which suggests that LongVU has many potential applications in the vision-language community.

## 6 Training Datasets

For the image-language training stage, previous methods (Chen et al., ; Peng et al., ; Wang et al., ; Chen et al., ; Liu et al., ; Dong et al., ) usually use two stages of alignment and finetuning. For simplicity, we combine and alignment in one stage using single image version of LLaVA-OneVision (Li et al., ) data. For video-language training, we utilize a large-scale video-text pairs sourced from several publicly accessible databases. The video training data is a subset of VideoChat2-IT (Li et al., ), which includes TextVR (Wu et al., ), Youcook2 (Zhou et al., ), Kinetics-710 (Kay et al., ), NExTQA (Xiao et al., ), CLEVRER (Yi et al., ), EgoQA (Fan, ), TGIF (Li et al., ), WebVidQA (Yang et al., ), ShareGPT4Video (Chen et al., ), in addition to above, we use MovieChat (Song et al., ) as long video complementary. All the training data is demonstrated in Table 6.

**Table 6: Training data statistics.**
| Modality | Task | # Samples | Dataset |
| --- | --- | --- | --- |
| Image-Text | Single-Image | 3.2M | LLaVA-OneVision |
|  | Captioning | 43K | TextVR, MovieChat, YouCook2 |
|  | Classification | 1K | Kinetics-710 |
|  | VQA | 424K | NExTQA, CLEVRER, EgoQA,<br>TGIF, WebVidQA, DiDeMo |
| Video-Text | Instruction | 85K | ShareGPT4Video |

**Table 7: Comparison with other video LMMs on VideoMME (Fu et al., ) benchmark.**
| Model | Size | Frames | Short | Medium | Long | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Video-LLaVA (Lin et al., ) | 7B | 8 | 46.1 | 40.7 | 38.1 | 41.6 |
| ShareGPT4Video (Chen et al., ) | 8B | 16 | 53.6 | 39.3 | 37.9 | 43.6 |
| Chat-Univi-v1.5 (Jin et al., ) | 7B | 64 | 51.2 | 44.6 | 41.8 | 45.9 |
| VideoLLaMA2 (Cheng et al., ) | 7B | 16 | 59.4 | 47.6 | 43.8 | 50.3 |
| VideoChat2 (Li et al., ) | 7B | 16 | 52.8 | 39.4 | 39.2 | 43.8 |
| LongVA (Zhang et al., ) | 7B | 128 | 61.6 | 50.4 | 47.6 | 54.3 |
| LLaVA-OneVision (Li et al., ) | 7B | 32 | 69.1 | 53.3 | 46.7 | 58.2 |
| LongVU (Ours) | 7B | 1fps | 64.7 | 58.2 | 59.5 | 60.9 |

## 7 Frame-level Position Encoding

To alleviate potential confusion arising from frame-by-frame feature concatenation, we incorporate a frame-level position encoding to enforce the temporal boundaries across frames and capture inter-dependencies within each frame. Given that we temporally reduce several frames, a straightforward concatenation of all frames renders the model unaware of the relative timestep across frames. Furthermore, our dynamic token sampling strategy does not delineate clear boundaries between each frame. To address this, we incorporate frame-level positional embeddings (FPE) that correspond to the absolute timestep of each frame, utilizing a shared sinusoidal position encoding (Vaswani, ) for frames at time $t$, shown in Equation 3.

$$ $PE(t,2i)=sin(t/10000^{2i/d}),PE(t,2i+1)=cos(t/10000^{2i/d})$ (3) $$

The ablation shows in Table 8 and Table 9 that adding the FPE does not affect much to the overall performance across several benchmarks. Therefore, we decide not to include it in our default setting.

Table: Table 8: Ablation study on with or without FPE.

Table: Table 9: Strategy ablations on each subtask in MLVU (Zhou et al., ).

## 8 DINOv2 v.s. SigLIP

DINOv2 (Oquab et al., ), through self-supervised training with a feature similarity objective on visually-centric tasks, captures subtle frame differences and low-level visual features more effectively than vision-language contrastive methods (Radford et al., ; Zhai et al., ), as shown in Figure 6.

Figure: Figure 6: Similarity comparison between SigLIP (Zhai et al., ) and DINOv2 (Oquab et al., ) features. The similarity is calculated between the first frame and the remainings. DINO concentrating on vision centric task effectively capture subtle frame differences compared with SigLIP (Zhai et al., ) which is aligned on semantic space.
Refer to caption: x6.png

## 9 Needle-In-A-Video-Haystack

We conducted experiments using an 8k context length to evaluate our default setting, which incorporates our adaptive compression, against configurations without spatial token compression (w/o STC) and without querying guided reduction (w/o Query), as depicted in Figure 7. By integrating a cross-modal query to selectively retain full tokens of frames relevant to the text query, the model significantly enhances its ability to accurately identify key frames when the total number of video frames is fewer than 1.4k. Moreover, our adaptive token compression mechanism further boosts VQA accuracy with increased frames.

Figure: Figure 7: Needle-In-A-Video-Haystack results. Our spatiotemporal adaptive token compression scheme improves the score for locating the needle frame.
Refer to caption: x7.png

**Table 10: We mainly focus on video understanding task and use video-only data for video SFT stage. We observe a decrease in performance on image understanding after video SFT stage.**
| Model | SQA-IMG | MMVP | POPE | RealWorldQA |
| --- | --- | --- | --- | --- |
| Before video SFT | 95.44 | 51.33 | 86.65 | 61.06 |
| After video SFT | 83.94 | 32.00 | 81.23 | 47.65 |

## 10 Limitation

Our research is primarily concentrated on video understanding tasks, for which we employ video-only data during the video supervised fine-tuning (SFT) stage. As evidenced in Table 10, there is a decrease observed in the model’s image understanding capabilities after video SFT. A potential remedy could involve integrating a mix of image, multi-image, and video data during training. However, due to constraints in GPU resources, we leave it as a future work with larger datasets for stronger unified image and video models.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see weaknesses.
soundness: 3
strengths: 1. The concept of spatiotemporal adaptive compression to address the challenge of long video understanding is timely, given the increasing interest in multimodal learning and the limitations of current MLLMs.
2. The paper provides extensive experimental validation, showing significant improvements over existing methods on multiple video understanding benchmarks, which is a strong empirical demonstration of the proposed method's effectiveness.
3. The use of DINOv2 features for temporal reduction and the cross-modal query mechanism for selective frame feature reduction are well-motivated and technically sound.
summary: The paper presents LongVU, a spatiotemporal adaptive compression mechanism designed to handle long video sequences within the limited context length constraints of current multimodal large language models (MLLMs). The authors propose leveraging cross-modal query and inter-frame dependencies to reduce temporal and spatial redundancy in videos effectively. The method demonstrates promising results across various video understanding benchmarks, particularly for hour-long videos.
weaknesses: 1. While the paper discusses the effectiveness of LongVU, a deeper analysis into the computational complexity and scalability, especially regarding the trade-offs with comprasion ratio and context length, could be beneficial.
2. The paper focuses on video-language understanding. It would be useful to understand how well these techniques generalize to other modalities (e.g. image).
3. More ablation studies on the components of the spatiotemporal compression strategy could provide further insights into the contribution of each component to the overall performance.
4. The query-based comparison seems limited to single-turn dialogue scenarios. The authors should consider how LongVU would handle multi-turn interactions, where the context and relevance of frames may change dynamically across turns. Addressing this could significantly expand the applicability of LongVU.
5. The paper missed some relevant works, such as "LGDN: Language-guided denoising network for video-language modeling" (NeurIPS 2022).

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Is the sliding window size \( K = 8 \) an empirical value in spatial token compression (STC)?
2. Are there any analyses of failed examples?
soundness: 3
strengths: 1. This paper is competitive in long video understanding performance; for instance, its performance on VideoMME long videos surpasses that of LLaVA-OneVision by 12.8 points.
2. In smaller model sizes (LLaMA3.2-3B), LongVU also outperforms existing methods.
3. This method essentially proposes a token compression strategy that helps reduce token redundancy in long videos. It is a general approach that can be applied across different multi-modal large language models (MLLMs).
summary: This paper aims to address the challenge of token compression in long video understanding by proposing different token sampling strategies in the spatiotemporal dimension: 1) reducing temporal redundancy by leveraging inter-frame feature similarity (DINOv2 and SigLIP); 2) reducing feature redundancy by utilizing cross-modal similarity; and 3) reducing redundant tokens through element-wise cosine similarity within spatial windows. 
Through these effective and intuitive strategies, this research is capable of processing 1-hour-long videos within a common 8k context length. Additionally, this paper demonstrates superior numerical performance compared to other long video understanding methods such as VideoChat2 and LLaVA-OneVision.
weaknesses: 1. This paper is engineering-oriented and lacks some innovation. For instance, the combination of SigLIP and DINOv2 is based on the approach used in Cambrian. Additionally, the ability of DINOv2 features to more intuitively reflect low-level visual characteristics is evident, and this has also been compared with CLIP-based methods in the original paper.
2. In the comparative experiments, I believe that including the inference time for processing videos of the same length would contribute to a fairer comparison. This is because the token compression strategy employed in this paper inevitably introduces some preprocessing steps, which in turn adds more inference time.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The authors need to clarify certain ambiguous aspects of their method; please refer to the Weaknesses section for details. 

2. The authors should analyze the time and computational efficiency of the proposed compression method. For instance, what proportion of inference time is consumed by calculating DINO similarity between frames and spatial token similarity? Does this lead to significant time overhead during inference?
soundness: 4
strengths: 1. This paper introduces LongVU, which employs two well-designed strategies (Section 3.1, Section 3.3) to reduce redundant temporal frames and spatial tokens, while also proposing the selective retention of high-resolution frames based on queries (Section 3.2). Compared to earlier methods based on frame and token compression [1,2], LongVU demonstrates significant advancements in long video understanding tasks.

2. The authors have conducted comprehensive evaluations, including on the latest long video understanding benchmarks such as VideoMME and MLVU. LongVU achieves state-of-the-art performance in long video understanding tasks compared to recent advanced video MLLMs.

3. The spatial-temporal compression mechanism proposed by the authors has been proven effective for long video tasks, as demonstrated by the ablation study on MLVU in Table 4 and the Needle-in-a-Haystack task experiment in Figure 5.

[1] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Li, Yanwei, Chengyao Wang, and Jiaya Jia. "Llama-vid: An image is worth 2 tokens in large language models." European Conference on Computer Vision. Springer, Cham, 2025.
summary: This paper introduces LongVU, a novel approach designed to achieve effective long video understanding within the limited context of large language models (LLMs) by compressing the spatio-temporal tokens of videos. The authors provide a detailed discussion on effectively removing redundant temporal frames and spatial tokens. LongVU demonstrates state-of-the-art performance across multiple video understanding benchmarks. The extensive ablation studies convincingly illustrate LongVU's effectiveness in understanding long videos.
weaknesses: 1. There are some confusing aspects in the methodology section that need clarification:
   - (1) In the Temporal Reduction section (Section 3.1), the authors propose sampling \( T \) frames from \( N \) frames based on DINO similarity. According to the description, \( T \) seems to vary depending on the video. Is there a specific DINO threshold used here? If so, what is the threshold?
   - (2) The proposed method lacks a mechanism to ensure that the tokens fed into the LLM backbone remain within the context length. The authors do not explain how tokens exceeding the LLM context are handled. As shown in Figure 4(b), after token compression (Section 3.2), many video tokens exceed 8K, with some reaching over 15K. Given that the default context length setting is 8K, how are the excess tokens managed?
   - (3) Is the cross-attention module in Section 3.2 trained end-to-end during the video SFT stage?

2. The LongVU 7B model uses Qwen2-7B as the LLM backbone, which has a context length of 32K. Table 3 indicates that their 16K baseline (line 443; Uniform-16K) has a clear advantage over their 8K baseline (line 447; Uniform-8K). Why did the authors ultimately choose 8K for the model setting? Would combining DINO+Query+STC with a 16K or even 32K context length yield better results on hour-level long video tasks? Alternatively, are the authors emphasizing the superiority of DINO+Query+STC under limited context and computational constraints? If so, they should compare the additional computational overhead introduced by DINO+Query+STC.

3. There are some typos in the paper. For example, on line 46, "per framE" has an uppercase "E." In Table 1, some data is misaligned; for instance, the MLVU result for GPT-4o is 64.6, not 66.2.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Does the proposed compression approach hurt performance on short videos?

Table 3 lacks results on MVbench, a relatively shorter benchmark. Additionally, Table 7 indicates that the model performs less effectively on short videos. Are there specific limitations causing the proposed method to underperform on short videos? If so, it would be helpful to highlight these limitations.
soundness: 3
strengths: 1. The motivation for reducing the number of visual tokens is clear.

2. The evaluation is thorough, with ablation studies clearly demonstrating the effects of the proposed approaches.

3. The paper is well-written and easy to follow.
summary: This paper focuses on the challenge of limited context length in understanding long videos. The authors propose LongVU, which reduces the number of visual tokens across temporal and spatial dimensions. This approach enables the model to handle a large number of frames effectively, achieving strong performance on benchmarks such as VideoMME and MLVU.
weaknesses: Table 7 indicates that the model performs less effectively on short videos; it would be beneficial to include some analysis of this phenomenon.

This paper is quite comprehensive, and in my view, there are no other significant weaknesses.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 3
strengths: - The paper is generally well-written and easy to understand.
- The performance is encouraging.
summary: The paper proposes a video compression strategy, including: removing redundant frames, selective frame feature reduction, and spatial token reduction. This strategy effectively reduces the number of visual tokens in MLLM and significantly improves MLLM performance on long videos.
weaknesses: - There are many details in the pipeline that are not explained clearly.

1. Line 52: The statement "the resampling module causes important visual information loss" needs further explanation. The resampling module is widely used in MLLMs, such as in the latest **qwen2-vl** and **MiniCPMV-2.6**.

2. Equation 1 lacks a detailed explanation, and there are errors in the symbol definitions. For example, in line 242, why is it not $L_{max}$?

3. The setup in Section 3.1 lacks necessary details, such as why $ J $ is set to 8.  

4. In line 215, the statement "about half of the video frames were reduced" depends on the hyperparameter settings, but the paper does not explain how the hyperparameters were chosen.

5. Figure 4(a) shows the comparison of the number of frames before and after compression. However, this result depends on the choice of threshold, which is not explained in the paper.

6. In Table 4, the **STC** strategy is not always effective. What is the reason for this?

7. Line 207: The authors claim, "we pioneer to leverage both SSL-based model DINOv2 with vision-language contrastive-based model SigLIP as frame feature extractors for MLLM in video understanding tasks." In fact, several similar works already exist [1-2].

[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

- Compared to previous MLLMs, the three proposed modules in the paper require additional computational processes, such as inter-frame similarity calculation and frame-question similarity calculation. These dense computations will undoubtedly significantly increase inference time. However, the paper lacks a discussion on inference time.

- The proposed three compression strategies seem to be transferable to existing MLLMs in a training-free manner.

Meta-review: This paper proposes LongVU, a spatiotemporal adaptive compression mechanism for long video-language understanding. It aims to address the challenge of limited context length in current multimodal large language models (MLLMs) by reducing video tokens while preserving visual details. Experiments are conducted on multiple benchmarks, including VideoMME and MLVU.

The main strengths are: 1) well-motivated and technically sound model design (Spatiotemporal compression reduces tokens and retains key details) and 2) good performance.

The main weaknesses are: 1) limited novelty (the combination of SigLIP and DINOv2 is based on the approach used in Cambrian, and several similar works already exist [1-2]) and 2) missing technique details and explanation of experimental settings.  
[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs  
[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Since the technique novelty of this paper is limited (recognized by the reviewers oDkQ, and kRvH), the AC does not recommend acceptance at this conference. The authors are encouraged to address these concerns for future submissions.

Decision: Reject