OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni
  MLLMs
Authors: name: Caorui Li, name: Yu Chen, name: Yiyan Ji, name: Jin Xu, name: Zhenyu Cui, name: Shihao Li, name: Yuanxing Zhang, name: Jiafu Tang, name: Zhenghao Song, name: Dingling Zhang, name: Ying He, name: Haoxiang Liu, name: Yuxuan Wang, name: Qiufeng Wang, name: Zhenhe Wu, name: Jiehui Luo, name: Zhiyu Pan, name: Weihao Xie, name: Chenchen Zhang, name: Zhaohui Wang, name: Jiayi Tian, name: Yanghai Wang, name: Zhe Cao, name: Minxin Dai, name: Ke Wang, name: Runzhe Wen, name: Yinghao Ma, name: Yaning Pan, name: Sungkyun Chang, name: Termeh Taheri, name: Haiwen Xia, name: Christos Plachouras, name: Emmanouil Benetos, name: Yizhi Li, name: Ge Zhang, name: Jian Yang, name: Tianhao Peng, name: Zili Wang, name: Minghao Liu, name: Junran Peng, name: Zhaoxiang Zhang, name: Jiaheng Liu
Organization: NJU-LINK Lab

Recent advances in multimodal large language models (MLLMs) have demonstrated
substantial potential in video understanding. However, existing benchmarks fail
to comprehensively evaluate synergistic reasoning capabilities across audio and
visual modalities, often neglecting either one of the modalities or integrating
them in a logically inconsistent manner. To bridge this gap, we introduce
OmniVideoBench, a large-scale and rigorously designed benchmark dedicated to
assessing synergistic audio-visual understanding, with a strong emphasis on
modality complementarity and logical consistency. Specifically, OmniVideoBench
comprises 1000 high-quality question-answer(QA) pairs, each annotated with
step-by-step reasoning traces, derived from 628 diverse videos ranging from
several seconds to 30 minutes, and manually verified to guarantee complete
correctness and uniqueness. Moreover, OmniVideoBench encompasses 13 carefully
designed question types, covering temporal reasoning, spatial localization,
counting, causal inference, summarization, and beyond, thereby capturing the
essential challenges of video understanding. Evaluation of multiple MLLMs on
OmniVideoBench reveals a pronounced gap between model performance and human
reasoning, with open-source models lagging significantly behind their
closed-source counterparts, underscoring the inherent difficulty of genuine
audio-visual reasoning. We will release OmniVideoBench to foster the
development of MLLMs with stronger and more generalizable reasoning
capabilities.