VideoVista-CulturalLingo: 360^circ Horizons-Bridging Cultures,
  Languages, and Domains in Video Comprehension
Authors: name: Xinyu Chen, name: Yunxin Li, name: Haoyuan Shi, name: Baotian Hu, name: Wenhan Luo, name: Yaowei Wang, name: Min Zhang

Assessing the video comprehension capabilities of multimodal AI systems can
effectively measure their understanding and reasoning abilities. Most video
evaluation benchmarks are limited to a single language, typically English, and
predominantly feature videos rooted in Western cultural contexts. In this
paper, we present VideoVista-CulturalLingo, the first video evaluation
benchmark designed to bridge cultural, linguistic, and domain divide in video
comprehension. Our work differs from existing benchmarks in the following ways:
1) Cultural diversity, incorporating cultures from China, North America, and
Europe; 2) Multi-linguistics, with questions presented in Chinese and
English-two of the most widely spoken languages; and 3) Broad domain, featuring
videos sourced from hundreds of human-created domains. VideoVista-CulturalLingo
contains 1,389 videos and 3,134 QA pairs, and we have evaluated 24 recent
open-source or proprietary video large models. From the experiment results, we
observe that: 1) Existing models perform worse on Chinese-centric questions
than Western-centric ones, particularly those related to Chinese history; 2)
Current open-source models still exhibit limitations in temporal understanding,
especially in the Event Localization task, achieving a maximum score of only
45.2%; 3) Mainstream models demonstrate strong performance in general
scientific questions, while open-source models demonstrate weak performance in
mathematics.