LongVILA: Scaling Long-Context Visual Language Models for Long Videos

Authors: Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Yihui He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, Song Han
Primary area: generative models
Keywords: Large language models, Long context, Multi-modality, Video understanding

Abstract:
Long-context capability is critical for multi-modal foundation models, especially for long video understanding. We introduce LongVILA, a full-stack solution for long-context visual-language models by co-designing the algorithm and system. For model training, we upgrade existing VLMs to support long video understanding by incorporating two additional stages, i.e., long context extension and long video supervised fine-tuning. However, training on long video is computationally and memory intensive. We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing. LongVILA efficiently extends the number of video frames of VILA from 8 to 2048, achieving 99.8% accuracy in 6,000-frame (more than 1 million tokens) video needle-in-a-haystack. LongVILA-7B demonstrates strong accuracy on 9 popular video benchmarks, e.g., 65.1% VideoMME with subtitle. Besides, MM-SP is 2.1x - 5.7x faster than ring style sequence parallelism and 1.1x - 1.4x faster than Megatron with a hybrid context and tensor parallelism. Moreover, it seamlessly integrates with Hugging Face Transformers.

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730643168888

thread_type:
official_review

<thread_object>
creator: reviewer_mtsu

<content>
summary:
This paper proposes LongVILA, a solution for long-context visual-language models. The authors incorporate two additional stages, i.e., long context extension and long video supervised fine-tuning. A long-context multi-modal sequence parallelism system is proposed to efficiently parallelize long video training and inference.

strengths:
1. This paper implements a five-stage training curriculum: multi-modal alignment, large-scale pre-training, short supervised fine-tuning, context extension for LLMs, and long supervised fine-tuning.
2. An efficient and user-friendly framework (multi-modal sequence parallelism) is proposed to support training and inferencing
memory-intensive long-context VLMs.
3. The presentations are good.

weaknesses:
1. I suggest the authors to involve more baselines in Table 3. For example, PLLaVA, Flash-VStream, Video-LLaMA-2.
2. This paper has not reported results on some (long) video QA benchmarks, for example, MoVQA, ActivityNet-QA, Ego-Schema, MV-Bench. I suggest the authors to include them.
3. Is there any model complexity analysis? For example, #Params, GFLOPs.


[1] PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning

 [2] VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs 

[3] Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643400929

thread_type:
official_review

<thread_object>
creator: reviewer_7lyg

<content>
summary:
The paper introduces LongVILA, a comprehensive framework aimed at enhancing long-context video understanding capabilities in multi-modal foundation models. To optimize model training for long videos, LongVILA introduces two pivotal stages: long context extension and long video supervised fine-tuning. Considering the computational and memory challenges associated with training on long videos, the authors present the long-context Multi-Modal Sequence Parallelism (MM-SP) system. This system significantly improves training efficiency, enabling the processing of context lengths up to 2 million on 256 GPUs without relying on gradient checkpointing. The results demonstrate a remarkable increase in performance metrics, with the long video captioning score rising from 2.00 to 3.26 and achieving 99.5% accuracy in video needle-in-a-haystack tasks. Additionally, LongVILA shows strong performance on the VideoMME benchmark, with scores of 57.5% and 60.6% without and with subtitles, respectively. Overall, LongVILA presents a significant advancement in the realm of long-context visual-language models, providing both theoretical contributions and practical solutions to the challenges posed by long video understanding.

strengths:
* The paper is well-written and easy to understand. The figure illustration and captions are informative.
* The authors provide a full-stack design for long-context Vision-Language Models, including both training curriculum and system implementation. These contributions are significant in the multimodal foundation model community.
* The proposed model, LongVILA, presents strong performance on VideoMME and other long video understanding tasks.
* The Multi-Modal Sequence Parallelism design can greatly reduce the memory cost in both training and inference.

weaknesses:
* The authors do not seem to provide performance metrics on general video understanding benchmarks, such as the Perception Test [A] or EgoSchema [B], after improving long video understanding capabilities. It is worth noting whether the performance on general benchmarks is affected after enhancing long-context capabilities.
* The proposed Multi-Modal Sequence Parallelism design is interesting and effective. However, it is not clear whether the source code will be released, which will be beneficial to the community.

---

[A]Patraucean, V., Smaira, L., Gupta, A., Recasens, A., Markeeva, L., Banarse, D., ... & Carreira, J. (2024). Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36.

[B]Mangalam, K., Akshulakov, R., & Malik, J. (2024). Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36.

questions:
Refer to weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702280700

thread_type:
official_review

<thread_object>
creator: reviewer_ddgn

<content>
summary:
The paper proposes a novel long-context Multi-Modal Sequence Parallelism (MM-SP) system specifically designed to enhance the performance of Vision-Language Models (VLMs) when processing extended contexts, increasing the window size from 8 to 1024 frames. The authors demonstrate how their approach can effectively manage and leverage longer input sequences, aiming to address limitations in current VLMs regarding video processing capabilities.

strengths:
- The introduction of MM-SP is a significant contribution to the field, as it creatively combines techniques for managing long-context sequences in a multi-modal setting. This approach addresses a gap in existing models that struggle with extensive video inputs.

weaknesses:
- The evaluation primarily utilizes a single benchmark (VideoMME), which may not sufficiently demonstrate the capabilities of the proposed system in handling long videos. It would be beneficial to incorporate additional benchmarks focused on long video analysis, such as those introduced in recent works (e.g., arXiv:2406.09367 and arXiv:2406.14129) to validate the robustness of the findings.
- While the authors mention the ability to process 1024 frames, the results are only provided for 256 frames (Figure 2(a)). It would strengthen the paper to include performance metrics for 512 and 1024 frames across the discussed benchmarks to fully substantiate the claims made.

questions:
- If the authors were to mix the stage 3 with stage 5 at the end, which is stage 1-2-4-(3&5), what anticipated effects on performance or learning dynamics might arise? Would the integration lead to any synergies or drawbacks?
- How would the performance change if context extension were prioritized by using a long-window LLM first, followed by stages 4-1-2-3-5?
- How would the performance change if you conbine these two stratgies, which is 4-1-2-(3&5)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732144052370

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate all reviewers for their detailed and constructive feedback. We have revised the paper accordingly. We upload the revised paper, with major updates highlighted in blue.

1. **Expanded Evaluation**: We have included **8 additional video benchmarks** for evaluation: ActivityNet-QA [1], EgoSchema [2], EventBench [3], LongVideoBench [4], PerceptionTest [5], MVBench [6], NExT-QA [7], and VNBench [8]. Alongside VideoMME [9], we now compare LongVILA against state-of-the-art methods on a total of **9 benchmarks**, demonstrating consistently strong performance. The results are shown in **Table 3 in the revision**.
2. **Needle in the Long Video Haystack Experiment**: We report stronger results, with the LongVILA model trained on 2048 frames achieving 99.8% accuracy on **6,000 frames (exceeding 1 million tokens)**. The results are illustrated in **Figure 2 in the revision**.
3. **VideoMME Update**: We have updated our results on **VideoMME** by adding the LongVILA-1.5B model and results for 256 frames. LongVILA-7B achieves **60.1% / 65.1%** accuracy for the settings without and with subtitles, respectively, as shown in **Table 4 in the revision**.
4. **Model Complexity Analysis**: We present a detailed analysis of **model complexity** across various factors including model size, components, number of frames, context length, latency, and FLOPs in **Table 10 in the revision**.
5. **Additional Baselines**: We compare our model against more baselines, including proprietary models (GPT-4V [10], GPT-4o [11], Gemini-1.5-Pro [12]) and open models (Flash-VStream [13], VideoLLaMA2.1 [14], PLLaVA [15], LLaVA One-Vision [16]), as presented in **Table 3 in the revision**.
6. **Additional Ablations**: We conducted further ablations on training schedule settings, as presented in **Table 1 in the revision**.

The following sections provide detailed responses to all reviewer comments.

[1] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, Dacheng Tao: ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering. AAAI 2019

[2] Karttikeya Mangalam, Raiymbek Akshulakov, Jitendra Malik: EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. NeurIPS 2023

[3] Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen: Towards Event-oriented Long Video Understanding. 2024

[4] Haoning Wu, Dongxu Li, Bei Chen, Junnan Li: LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. 2024

[5] Viorica Patraucean, et. al: Perception Test: A Diagnostic Benchmark for Multimodal Video Models. NeurIPS 2023

[6] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, Yu Qiao: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. CVPR 2024

[7] Junbin Xiao, Xindi Shang, Angela Yao, Tat-Seng Chua: NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions. CVPR 2021

[8] Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, Jing Liu: Needle In A Video Haystack: A Scalable Synthetic Framework for Benchmarking Video MLLMs. 2024

[9] Chaoyou Fu, e t. al.: Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. 2024

[10] OpenAI. Gpt-4v. 2023

[11] OpenAI. Hello gpt-4o. 2024

[12] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.

[13] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, Xiaojie Jin: Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams. 2024

[14] Zesen Cheng, et. al.: VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. 2024

[15] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See-Kiong Ng, Jiashi Feng: PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning. 2024

[16] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li: LLaVA-OneVision: Easy Visual Task Transfer. 2024
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel long-context Multi-Modal Sequence Parallelism system to enhance Vision-Language Models in processing extended contexts, increasing the VILA model's capability from 8 to 1024 frames. Extensive experiments conducted after the rebuttal stage demonstrate the effectiveness of their approach in managing and leveraging longer input sequences.

This paper received consistent positive scores, and the ACs recommend it for acceptance as a spotlight paper.