VideoGLUE: Video General Understanding Evaluation of Foundation Models
Authors: name: Liangzhe Yuan, name: Nitesh Bharadwaj Gundavarapu, name: Long Zhao, name: Hao Zhou, name: Yin Cui, name: Lu Jiang, name: Xuan Yang, name: Menglin Jia, name: Tobias Weyand, name: Luke Friedman, name: Mikhail Sirotenko, name: Huisheng Wang, name: Florian Schroff, name: Hartwig Adam, name: Ming-Hsuan Yang, name: Ting Liu, name: Boqing Gong

We evaluate existing foundation models video understanding capabilities using
a carefully designed experiment protocol consisting of three hallmark tasks
(action recognition, temporal localization, and spatiotemporal localization),
eight datasets well received by the community, and four adaptation methods
tailoring a foundation model (FM) for a downstream task. Moreover, we propose a
scalar VideoGLUE score (VGS) to measure an FMs efficacy and efficiency when
adapting to general video understanding tasks. Our main findings are as
follows. First, task-specialized models significantly outperform the six FMs
studied in this work, in sharp contrast to what FMs have achieved in natural
language and image understanding. Second,video-native FMs, whose pretraining
data contains the video modality, are generally better than image-native FMs in
classifying motion-rich videos, localizing actions in time, and understanding a
video of more than one action. Third, the video-native FMs can perform well on
video tasks under light adaptations to downstream tasks(e.g., freezing the FM
backbones), while image-native FMs win in full end-to-end finetuning. The first
two observations reveal the need and tremendous opportunities to conduct
research on video-focused FMs, and the last confirms that both tasks and
adaptation methods matter when it comes to the evaluation of FMs.