MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal
  LLMs
Authors: name: Tianhao Peng, name: Haochen Wang, name: Yuanxing Zhang, name: Zekun Wang, name: Zili Wang, name: Ge Zhang, name: Jian Yang, name: Shihao Li, name: Yanghai Wang, name: Xintao Wang, name: Houyi Li, name: Wei Ji, name: Pengfei Wan, name: Wenhao Huang, name: Zhaoxiang Zhang, name: Jiaheng Liu
Organization: NJU-LINK Lab

The advent of Multimodal Large Language Models (MLLMs) has expanded AI
capabilities to visual modalities, yet existing evaluation benchmarks remain
limited to single-video understanding, overlooking the critical need for
multi-video understanding in real-world scenarios (e.g., sports analytics and
autonomous driving). To address this significant gap, we introduce MVU-Eval,
the first comprehensive benchmark for evaluating Multi-Video Understanding for
MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies
through 1,824 meticulously curated question-answer pairs spanning 4,959 videos
from diverse domains, addressing both fundamental perception tasks and
high-order reasoning tasks. These capabilities are rigorously aligned with
real-world applications such as multi-sensor synthesis in autonomous systems
and cross-angle sports analytics. Through extensive evaluation of
state-of-the-art open-source and closed-source models, we reveal significant
performance discrepancies and limitations in current MLLMs' ability to perform
understanding across multiple videos. The benchmark will be made publicly
available to foster future research.