InternVideo2: Scaling Video Foundation Models for Multimodal Video
  Understanding
Authors: name: Yi Wang; name: Kunchang Li; name: Xinhao Li; name: Jiashuo Yu; name: Yinan He; name: Guo Chen; name: Baoqi Pei; name: Rongkun Zheng; name: Jilan Xu; name: Zun Wang; name: Yansong Shi; name: Tianxiang Jiang; name: Songze Li; name: Hongjie Zhang; name: Yifei Huang; name: Yu Qiao; name: Yali Wang; name: Limin Wang
Published: 2024-03-22 17:57:42

Abstract: We introduce InternVideo2, a new video foundation model (ViFM) that achieves
the state-of-the-art performance in action recognition, video-text tasks, and
video-centric dialogue. Our approach employs a progressive training paradigm
that unifies the different self- or weakly-supervised learning frameworks of
masked video token reconstruction, cross-modal contrastive learning, and next
token prediction. Different training stages would guide our model to capture
different levels of structure and semantic information through different
pretext tasks. At the data level, we prioritize the spatiotemporal consistency
by semantically segmenting videos and generating video-audio-speech captions.
This improves the alignment between video and text. We scale both data and
model size for our InternVideo2. Through extensive experiments, we validate our
designs and demonstrate the state-of-the-art performance on over 60 video and
audio tasks. Notably, our model outperforms others on various video-related
captioning, dialogue, and long video understanding benchmarks, highlighting its
ability to reason and comprehend long temporal contexts. Code and models are
available at https://github.com/OpenGVLab/InternVideo2/.