LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Authors: name: Fuzhao Xue, name: Yukang Chen, name: Dacheng Li, name: Qinghao Hu, name: Ligeng Zhu, name: Xiuyu Li, name: Yunhao Fang, name: Haotian Tang, name: Shang Yang, name: Zhijian Liu, name: Ethan He, name: Hongxu Yin, name: Pavlo Molchanov, name: Jan Kautz, name: Linxi Fan, name: Yuke Zhu, name: Yao Lu, name: Song Han

Long-context capability is critical for multi-modal foundation models. We
introduce LongVILA, a full-stack solution for long-context vision-language
models, including system, model training, and dataset development. On the
system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP)
system that enables long-context training and inference, enabling 2M context
length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster
than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in
text-only settings. Moreover, it seamlessly integrates with Hugging Face
Transformers. For model training, we propose a five-stage pipeline comprising
alignment, pre-training, context extension, and long-short joint supervised
fine-tuning. Regarding datasets, we meticulously construct large-scale visual
language pre-training datasets and long video instruction-following datasets to
support our multi-stage training process. The full-stack solution extends the
feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and
improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5%
accuracy in 1400-frames video (274k context length) needle in a haystack.
LongVILA-8B also demonstrates a consistent improvement in performance on long
videos within the VideoMME benchmark as the video frames increase.