LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

Authors: Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, Vikas Chandra
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Long Video Understanding;Video-Language;Spatiotemporal

Abstract:
Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by the limited context length. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism to reduce the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within limited context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance. Our code will be made publicly available.

Decision: reject
Reviewer scores: 6, 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729866577949

thread_type:
official_review

<thread_object>
creator: reviewer_krvh

<content>
summary:
The paper proposes a video compression strategy, including: removing redundant frames, selective frame feature reduction, and spatial token reduction. This strategy effectively reduces the number of visual tokens in MLLM and significantly improves MLLM performance on long videos.

strengths:
- The paper is generally well-written and easy to understand.
- The performance is encouraging.

weaknesses:
- There are many details in the pipeline that are not explained clearly.

1. Line 52: The statement "the resampling module causes important visual information loss" needs further explanation. The resampling module is widely used in MLLMs, such as in the latest **qwen2-vl** and **MiniCPMV-2.6**.

2. Equation 1 lacks a detailed explanation, and there are errors in the symbol definitions. For example, in line 242, why is it not $L_{max}$?

3. The setup in Section 3.1 lacks necessary details, such as why $ J $ is set to 8.  

4. In line 215, the statement "about half of the video frames were reduced" depends on the hyperparameter settings, but the paper does not explain how the hyperparameters were chosen.

5. Figure 4(a) shows the comparison of the number of frames before and after compression. However, this result depends on the choice of threshold, which is not explained in the paper.

6. In Table 4, the **STC** strategy is not always effective. What is the reason for this?

7. Line 207: The authors claim, "we pioneer to leverage both SSL-based model DINOv2 with vision-language contrastive-based model SigLIP as frame feature extractors for MLLM in video understanding tasks." In fact, several similar works already exist [1-2].

[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

- Compared to previous MLLMs, the three proposed modules in the paper require additional computational processes, such as inter-frame similarity calculation and frame-question similarity calculation. These dense computations will undoubtedly significantly increase inference time. However, the paper lacks a discussion on inference time.

- The proposed three compression strategies seem to be transferable to existing MLLMs in a training-free manner.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730459758132

thread_type:
official_review

<thread_object>
creator: reviewer_rw6v

<content>
summary:
This paper focuses on the challenge of limited context length in understanding long videos. The authors propose LongVU, which reduces the number of visual tokens across temporal and spatial dimensions. This approach enables the model to handle a large number of frames effectively, achieving strong performance on benchmarks such as VideoMME and MLVU.

strengths:
1. The motivation for reducing the number of visual tokens is clear.

2. The evaluation is thorough, with ablation studies clearly demonstrating the effects of the proposed approaches.

3. The paper is well-written and easy to follow.

weaknesses:
Table 7 indicates that the model performs less effectively on short videos; it would be beneficial to include some analysis of this phenomenon.

This paper is quite comprehensive, and in my view, there are no other significant weaknesses.

questions:
Does the proposed compression approach hurt performance on short videos?

Table 3 lacks results on MVbench, a relatively shorter benchmark. Additionally, Table 7 indicates that the model performs less effectively on short videos. Are there specific limitations causing the proposed method to underperform on short videos? If so, it would be helpful to highlight these limitations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730481159578

thread_type:
official_review

<thread_object>
creator: reviewer_jb4v

<content>
summary:
This paper introduces LongVU, a novel approach designed to achieve effective long video understanding within the limited context of large language models (LLMs) by compressing the spatio-temporal tokens of videos. The authors provide a detailed discussion on effectively removing redundant temporal frames and spatial tokens. LongVU demonstrates state-of-the-art performance across multiple video understanding benchmarks. The extensive ablation studies convincingly illustrate LongVU's effectiveness in understanding long videos.

strengths:
1. This paper introduces LongVU, which employs two well-designed strategies (Section 3.1, Section 3.3) to reduce redundant temporal frames and spatial tokens, while also proposing the selective retention of high-resolution frames based on queries (Section 3.2). Compared to earlier methods based on frame and token compression [1,2], LongVU demonstrates significant advancements in long video understanding tasks.

2. The authors have conducted comprehensive evaluations, including on the latest long video understanding benchmarks such as VideoMME and MLVU. LongVU achieves state-of-the-art performance in long video understanding tasks compared to recent advanced video MLLMs.

3. The spatial-temporal compression mechanism proposed by the authors has been proven effective for long video tasks, as demonstrated by the ablation study on MLVU in Table 4 and the Needle-in-a-Haystack task experiment in Figure 5.

[1] Song, Enxin, et al. "Moviechat: From dense token to sparse memory for long video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Li, Yanwei, Chengyao Wang, and Jiaya Jia. "Llama-vid: An image is worth 2 tokens in large language models." European Conference on Computer Vision. Springer, Cham, 2025.

weaknesses:
1. There are some confusing aspects in the methodology section that need clarification:
   - (1) In the Temporal Reduction section (Section 3.1), the authors propose sampling \( T \) frames from \( N \) frames based on DINO similarity. According to the description, \( T \) seems to vary depending on the video. Is there a specific DINO threshold used here? If so, what is the threshold?
   - (2) The proposed method lacks a mechanism to ensure that the tokens fed into the LLM backbone remain within the context length. The authors do not explain how tokens exceeding the LLM context are handled. As shown in Figure 4(b), after token compression (Section 3.2), many video tokens exceed 8K, with some reaching over 15K. Given that the default context length setting is 8K, how are the excess tokens managed?
   - (3) Is the cross-attention module in Section 3.2 trained end-to-end during the video SFT stage?

2. The LongVU 7B model uses Qwen2-7B as the LLM backbone, which has a context length of 32K. Table 3 indicates that their 16K baseline (line 443; Uniform-16K) has a clear advantage over their 8K baseline (line 447; Uniform-8K). Why did the authors ultimately choose 8K for the model setting? Would combining DINO+Query+STC with a 16K or even 32K context length yield better results on hour-level long video tasks? Alternatively, are the authors emphasizing the superiority of DINO+Query+STC under limited context and computational constraints? If so, they should compare the additional computational overhead introduced by DINO+Query+STC.

3. There are some typos in the paper. For example, on line 46, "per framE" has an uppercase "E." In Table 1, some data is misaligned; for instance, the MLVU result for GPT-4o is 64.6, not 66.2.

questions:
1. The authors need to clarify certain ambiguous aspects of their method; please refer to the Weaknesses section for details. 

2. The authors should analyze the time and computational efficiency of the proposed compression method. For instance, what proportion of inference time is consumed by calculating DINO similarity between frames and spatial token similarity? Does this lead to significant time overhead during inference?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730401747

thread_type:
official_review

<thread_object>
creator: reviewer_odkq

<content>
summary:
This paper aims to address the challenge of token compression in long video understanding by proposing different token sampling strategies in the spatiotemporal dimension: 1) reducing temporal redundancy by leveraging inter-frame feature similarity (DINOv2 and SigLIP); 2) reducing feature redundancy by utilizing cross-modal similarity; and 3) reducing redundant tokens through element-wise cosine similarity within spatial windows. 
Through these effective and intuitive strategies, this research is capable of processing 1-hour-long videos within a common 8k context length. Additionally, this paper demonstrates superior numerical performance compared to other long video understanding methods such as VideoChat2 and LLaVA-OneVision.

strengths:
1. This paper is competitive in long video understanding performance; for instance, its performance on VideoMME long videos surpasses that of LLaVA-OneVision by 12.8 points.
2. In smaller model sizes (LLaMA3.2-3B), LongVU also outperforms existing methods.
3. This method essentially proposes a token compression strategy that helps reduce token redundancy in long videos. It is a general approach that can be applied across different multi-modal large language models (MLLMs).

weaknesses:
1. This paper is engineering-oriented and lacks some innovation. For instance, the combination of SigLIP and DINOv2 is based on the approach used in Cambrian. Additionally, the ability of DINOv2 features to more intuitively reflect low-level visual characteristics is evident, and this has also been compared with CLIP-based methods in the original paper.
2. In the comparative experiments, I believe that including the inference time for processing videos of the same length would contribute to a fairer comparison. This is because the token compression strategy employed in this paper inevitably introduces some preprocessing steps, which in turn adds more inference time.

questions:
1. Is the sliding window size \( K = 8 \) an empirical value in spatial token compression (STC)?
2. Are there any analyses of failed examples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732220740

thread_type:
official_review

<thread_object>
creator: reviewer_lghk

<content>
summary:
The paper presents LongVU, a spatiotemporal adaptive compression mechanism designed to handle long video sequences within the limited context length constraints of current multimodal large language models (MLLMs). The authors propose leveraging cross-modal query and inter-frame dependencies to reduce temporal and spatial redundancy in videos effectively. The method demonstrates promising results across various video understanding benchmarks, particularly for hour-long videos.

strengths:
1. The concept of spatiotemporal adaptive compression to address the challenge of long video understanding is timely, given the increasing interest in multimodal learning and the limitations of current MLLMs.
2. The paper provides extensive experimental validation, showing significant improvements over existing methods on multiple video understanding benchmarks, which is a strong empirical demonstration of the proposed method's effectiveness.
3. The use of DINOv2 features for temporal reduction and the cross-modal query mechanism for selective frame feature reduction are well-motivated and technically sound.

weaknesses:
1. While the paper discusses the effectiveness of LongVU, a deeper analysis into the computational complexity and scalability, especially regarding the trade-offs with comprasion ratio and context length, could be beneficial.
2. The paper focuses on video-language understanding. It would be useful to understand how well these techniques generalize to other modalities (e.g. image).
3. More ablation studies on the components of the spatiotemporal compression strategy could provide further insights into the contribution of each component to the overall performance.
4. The query-based comparison seems limited to single-turn dialogue scenarios. The authors should consider how LongVU would handle multi-turn interactions, where the context and relevance of frames may change dynamically across turns. Addressing this could significantly expand the applicability of LongVU.
5. The paper missed some relevant works, such as "LGDN: Language-guided denoising network for video-language modeling" (NeurIPS 2022).

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732307858733

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their insightful feedback. We are encouraged by their recognition of the following: 
+ Our proposed spatiotemporal adaptive compression demonstrates **effectiveness** in understanding long videos (@LGhK, @jb4v), and also scales well in **smaller models** (@oDkQ).
+ The **motivation is clear**: to tackle the challenge of long video understanding in current MLLMs (@LGhK, @Rw6v).
+ The paper is **well-written** (@Rw6v, @kRvH), **technically sound** (@LGhK), and supported by **comprehensive experiments** across extensive evaluations (@jb4v, @Rw6v).

We address common questions raised by the reviewers here.

 **Model inference time comparison**

To evaluate the computational overhead introduced by our proposed spatiotemporal compression approach, we compare it with various baselines using input videos of the same length (20 minutes) sampled at 1 FPS. The experiments were conducted on an A100 GPU with 80 GB memory. LLaMA-VID encounters a CUDA out-of-memory (OOM) issue when processing 20-minute videos as input. As the table shows below, our method demonstrates faster performance compared to the token compression approach of Chat-UniVi, which relies on a KNN-based strategy to merge similar tokens. Furthermore, it is more efficient than the resampler-based method VideoChat2, which compresses video inputs using learnable queries in Q-Former. When compared to methods without compression, such as LLaVA-OneVision, our approach is slightly slower, requiring 1.27x the processing time, however, our model improves the performance by 5.2% on average across all the benchmarks shown in Table 1 in the main paper.

| Models | LLaMA-VID | Chat-UniVi | VideoLLaMA2 | VideoChat2 | LLaVA-OneVision | Ours |
| -------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
| Inference time (sec) | OOM | 49.06 | 58.62 | 45.22 | 25.84 | 32.96 |

**Inference time of each compression component**

We begin by using the DINOv2 vision encoder to extract features from all frames and then reduce redundant frames based on DINO feature similarity. After this reduction, the remaining frames are processed using SigLIP. We use input videos of 20 minutes sampled at 1 FPS and the experiments are conducted on an A100 GPU with 80 GB memory. As shown in the table below, the primary computation lies in frame feature extraction, which, in real-world applications, can be preprocessed offline. Notably, our proposed compression components contribute only a small portion to the overall inference overhead.
| Component | Extract all frame features | **DINO similarity** | Extract remaining feature for SigLIP | **Query** | **STC** | Total |
| -------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
| Inference time (sec) | 22.2 | 1.05 | 4.32 | 0.27 | 0.18 | 32.96 |

One significant advantage of our method is that the DINO-based frame reduction step substantially decreases the computation required for the remaining frames in subsequent steps. Therefore, while most baseline models encounter OOM issues when processing hour-long videos at a 1 FPS sampling rate, our model can process hour-long videos with an inference time of 83.41 seconds.

We respond to specific queries in each reviewer's section. We will incorporate all constructive feedback into our revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes LongVU, a spatiotemporal adaptive compression mechanism for long video-language understanding. It aims to address the challenge of limited context length in current multimodal large language models (MLLMs) by reducing video tokens while preserving visual details. Experiments are conducted on multiple benchmarks, including VideoMME and MLVU.

The main strengths are: 1) well-motivated and technically sound model design (Spatiotemporal compression reduces tokens and retains key details) and 2) good performance.

The main weaknesses are: 1) limited novelty (the combination of SigLIP and DINOv2 is based on the approach used in Cambrian, and several similar works already exist [1-2]) and 2) missing technique details and explanation of experimental settings.  
[1] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs  
[2] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Since the technique novelty of this paper is limited (recognized by the reviewers oDkQ, and kRvH), the AC does not recommend acceptance at this conference. The authors are encouraged to address these concerns for future submissions.