VideoLLM-online: Online Video Large Language Model for Streaming Video
Authors: name: Joya Chen; name: Zhaoyang Lv; name: Shiwei Wu; name: Kevin Qinghong Lin; name: Chenan Song; name: Difei Gao; name: Jia-Wei Liu; name: Ziteng Gao; name: Dongxing Mao; name: Mike Zheng Shou
Published: 2024-06-17 17:55:32

Abstract: Recent Large Language Models have been enhanced with vision capabilities,
enabling them to comprehend images, videos, and interleaved vision-language
content. However, the learning methods of these large multimodal models
typically treat videos as predetermined clips, making them less effective and
efficient at handling streaming video inputs. In this paper, we propose a novel
Learning-In-Video-Stream (LIVE) framework, which enables temporally aligned,
long-context, and real-time conversation within a continuous video stream. Our
LIVE framework comprises comprehensive approaches to achieve video streaming
dialogue, encompassing: (1) a training objective designed to perform language
modeling for continuous streaming inputs, (2) a data generation scheme that
converts offline temporal annotations into a streaming dialogue format, and (3)
an optimized inference pipeline to speed up the model responses in real-world
video streams. With our LIVE framework, we built VideoLLM-online model upon
Llama-2/Llama-3 and demonstrate its significant advantages in processing
streaming videos. For instance, on average, our model can support streaming
dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU. Moreover, it
also showcases state-of-the-art performance on public offline video benchmarks,
such as recognition, captioning, and forecasting. The code, model, data, and
demo have been made available at https://showlab.github.io/videollm-online.