Flash-VStream: Memory-Based Real-Time Understanding for Long Video
  Streams
Authors: name: Haoji Zhang; name: Yiqin Wang; name: Yansong Tang; name: Yong Liu; name: Jiashi Feng; name: Jifeng Dai; name: Xiaojie Jin
Published: 2024-06-12 11:07:55

Abstract: Benefiting from the advancements in large language models and cross-modal
alignment, existing multi-modal video understanding methods have achieved
prominent performance in offline scenario. However, online video streams, as
one of the most common media forms in the real world, have seldom received
attention. Compared to offline videos, the 'dynamic' nature of online video
streams poses challenges for the direct application of existing models and
introduces new problems, such as the storage of extremely long-term
information, interaction between continuous visual content and 'asynchronous'
user questions. Therefore, in this paper we present Flash-VStream, a
video-language model that simulates the memory mechanism of human. Our model is
able to process extremely long video streams in real-time and respond to user
queries simultaneously. Compared to existing models, Flash-VStream achieves
significant reductions in inference latency and VRAM consumption, which is
intimately related to performing understanding of online streaming video. In
addition, given that existing video understanding benchmarks predominantly
concentrate on offline scenario, we propose VStream-QA, a novel question
answering benchmark specifically designed for online video streaming
understanding. Comparisons with popular existing methods on the proposed
benchmark demonstrate the superiority of our method for such challenging
setting. To verify the generalizability of our approach, we further evaluate it
on existing video understanding benchmarks and achieves state-of-the-art
performance in offline scenarios as well. All code, models, and datasets are
available at the https://invinciblewyq.github.io/vstream-page/