MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal
  LLMs
Authors: name: Tianhao Peng; name: Haochen Wang; name: Yuanxing Zhang; name: Zekun Wang; name: Zili Wang; name: Ge Zhang; name: Jian Yang; name: Shihao Li; name: Yanghai Wang; name: Xintao Wang; name: Houyi Li; name: Wei Ji; name: Pengfei Wan; name: Wenhao Huang; name: Zhaoxiang Zhang; name: Jiaheng Liu
Published: 2025-11-10 16:02:33

Abstract: The advent of Multimodal Large Language Models (MLLMs) has expanded AI
capabilities to visual modalities, yet existing evaluation benchmarks remain
limited to single-video understanding, overlooking the critical need for
multi-video understanding in real-world scenarios (e.g., sports analytics and
autonomous driving). To address this significant gap, we introduce MVU-Eval,
the first comprehensive benchmark for evaluating Multi-Video Understanding for
MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies
through 1,824 meticulously curated question-answer pairs spanning 4,959 videos
from diverse domains, addressing both fundamental perception tasks and
high-order reasoning tasks. These capabilities are rigorously aligned with
real-world applications such as multi-sensor synthesis in autonomous systems
and cross-angle sports analytics. Through extensive evaluation of
state-of-the-art open-source and closed-source models, we reveal significant
performance discrepancies and limitations in current MLLMs' ability to perform
understanding across multiple videos. The benchmark will be made publicly
available to foster future research.