LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Authors: name: Fuzhao Xue; name: Yukang Chen; name: Dacheng Li; name: Qinghao Hu; name: Ligeng Zhu; name: Xiuyu Li; name: Yunhao Fang; name: Haotian Tang; name: Shang Yang; name: Zhijian Liu; name: Ethan He; name: Hongxu Yin; name: Pavlo Molchanov; name: Jan Kautz; name: Linxi Fan; name: Yuke Zhu; name: Yao Lu; name: Song Han
Published: 2024-08-19 17:48:08

Abstract: Long-context capability is critical for multi-modal foundation models. We
introduce LongVILA, a full-stack solution for long-context vision-language
models, including system, model training, and dataset development. On the
system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP)
system that enables long-context training and inference, enabling 2M context
length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster
than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in
text-only settings. Moreover, it seamlessly integrates with Hugging Face
Transformers. For model training, we propose a five-stage pipeline comprising
alignment, pre-training, context extension, and long-short joint supervised
fine-tuning. Regarding datasets, we meticulously construct large-scale visual
language pre-training datasets and long video instruction-following datasets to
support our multi-stage training process. The full-stack solution extends the
feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and
improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5%
accuracy in 1400-frames video (274k context length) needle in a haystack.
LongVILA-8B also demonstrates a consistent improvement in performance on long
videos within the VideoMME benchmark as the video frames increase.