VideoVista2: 360° Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension
Authors: Xinyu Chen, yunxin li, Haoyuan Shi, Baotian Hu, Wenhan Luo, Yaowei Wang, Min Zhang
Accepted at: acl25

VideoVista-CulturalLingo: 360° Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension

Xinyu Chen1, Yunxin Li1, Haoyuan Shi1, Baotian Hu1*, Wenhan Luo2, Yaowei Wang1, Min Zhang1,
1Harbin Institute of Technology, Shenzhen, China,
2Division of AMC and Department of ECE, HKUST,
{chenxinyuhitsz, liyunxin987}@163.com
{hubaotian, zhangmin2021}@hit.edu.cn

Abstract

Assessing the video comprehension capabilities of multimodal AI systems can effectively measure their understanding and reasoning abilities. Most video evaluation benchmarks are limited to a single language, typically English, and predominantly feature videos rooted in Western cultural contexts. In this paper, we present VideoVista-CulturalLingo, the first video evaluation benchmark designed to bridge cultural, linguistic, and domain divide in video comprehension. Our work differs from existing benchmarks in the following ways: 1) Cultural diversity, incorporating cultures from China, North America, and Europe; 2) Multilingualism, with questions presented in Chinese and English—two of the most widely spoken languages; and 3) Broad domain, featuring videos sourced from hundreds of human-created domains. VideoVista-CulturalLingo contains 1,389 videos and 3,134 QA pairs, and we have evaluated 24 recent open-source or proprietary video large models. From the experiment results, we observe that: 1) Existing models perform worse on Chinese-centric questions than Western-centric ones, particularly those related to Chinese history; 2) Current open-source models still exhibit limitations in temporal understanding, especially in the Event Localization task, achieving a maximum score of only 45.2%; 3) Mainstream models demonstrate strong performance in general scientific questions, while open-source models demonstrate weak performance in mathematics.1

1 Introduction

Large Multimodal Models (LMMs) built upon Large Language Models (LLMs) have demonstrated unprecedented capabilities across various domains, including text, image, video, and audio over several years. Particularly in the past year, there has been a surge in the development of LMMs capable of processing video inputs. The dramatic expansion in the length of video frame sequences—from just a few frames to several hundred—demonstrates significant progress in video understanding capabilities. Meanwhile, video evaluation benchmarks have also emerged, evolving from early-stage basic video question answering tasks (Yu et al., 2019; Xu et al., 2017) to general video evaluation benchmarks (Fu et al., 2024; Zhou et al., 2024; Wang et al., 2024b). However, existing video evaluation benchmarks predominantly select videos from sources such as YouTube, Shutterstock, or established video datasets like Ego4D (Grauman et al., 2022) and Movie101 (Yue et al., 2023). These datasets are primarily Western-centric, with a limited representation of Chinese-centric videos as shown in Figure 1. In addition, current video evaluation benchmarks tend to focus on specific events within the videos, neglecting the cultural context and connotations embedded in the content while overlooking the scientific principles and in-

![An example of Traditional Chinese Culture from VideoVista-CulturalLingo. The ground truth is highlighted in yellow.](page_1012_370_388_246.png)

Figure 1: An example of Traditional Chinese Culture from VideoVista-CulturalLingo. The ground truth is highlighted in yellow.

1Evaluation Codes and Data are available at https://videovista-culturallingo.github.io/
<table>
  <tr>
    <th>Category</th>
    <th>Size</th>
  </tr>
  <tr>
    <td>Task Classes</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Subtask Classes</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Video Sources</td>
    <td>1,389</td>
  </tr>
  <tr>
    <td>Video Clips</td>
    <td>2,052</td>
  </tr>
  <tr>
    <td>Max Duration</td>
    <td>1,877.7</td>
  </tr>
  <tr>
    <td>Average Duration</td>
    <td>267.5</td>
  </tr>
  <tr>
    <td>YouTube Video Domains</td>
    <td>30</td>
  </tr>
  <tr>
    <td>RedNote Video Domains</td>
    <td>104</td>
  </tr>
  <tr>
    <td>Bilibili Video Domains</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Chinese Question Number</td>
    <td>1,446</td>
  </tr>
  <tr>
    <td>English Question Number</td>
    <td>1,668</td>
  </tr>
  <tr>
    <td>Chinese Culture QA Number</td>
    <td>231</td>
  </tr>
  <tr>
    <td>American Culture QA Number</td>
    <td>200</td>
  </tr>
  <tr>
    <td>European Culture QA Number</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Average Question Length</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Average Option Length</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Average Choice Number</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Total Samples</td>
    <td>3,134</td>
  </tr>
  <tr>
    <td>Total Questions</td>
    <td>3,134</td>
  </tr>
</table>

Figure 2: (Left) Comprehensive statistics from different perspectives. The durations reported are based on the statistics from the 2,052 video clips. The question and answer length is count in tokens; (Right) Videos in VideoVista-CulturalLingo is sourced hundreds of domains from 3 popular video websites across the world. In the video sourced from Xiaohongshu (RedNote), we only present 42 of the all domains.

formation that the videos are intended to convey.

To advance the development of LMMs, we introduce VideoVista-CulturalLingo, the first video evaluation benchmark designed to bridge cultures, languages, and domains in video comprehension. In Figure 2, we present detailed statistics on the questions and videos in VideoVista-CulturalLingo. It comprises 3,134 questions organized into 14 tasks, spanning 2,052 video clips of varying lengths and reflecting both Western and Chinese cultures. English-language videos are sourced from YouTube, while Chinese videos are collected from Xiaohongshu and BiliBili. These videos cover hundreds of distinct domains, ranging from everyday life topics—such as news reports, travel recommendations, sports events, and vlogs—to scientific topics, including calculus, deep learning, organic chemistry, and quantum mechanics.

To efficiently annotate such a large-scale video dataset, we employ a hybrid annotation framework that combines the strengths of both (M)LLMs and human efforts. This framework leverages the powerful capabilities of existing large models, such as Qwen2-VL (Wang et al., 2024a) and DeepSeek-R1 (DeepSeek-AI et al., 2025), to generate an initial pool of question-options-answer (QA) pairs. Human annotators then select the high-quality questions from generated QA pairs and further refine them to enhance clarity and quality.

We have evaluated 24 state-of-the-art (SOTA) LMMs, including proprietary LMMs such as GPT-4o, Gemini-2.0-Flash, as well as open-source video LMMs like Qwen2.5-VL (Team, 2025) and VideoLLaMA3 (Zhang et al., 2025), and image LMMs such as Molmo (Deitke et al., 2024) and DeepSeek2-VL (Wu et al., 2024). Experimental results show that Gemini-2.0-Flash demonstrates the strongest performance among all models, achieving an accuracy score of \( \mathbf{76.3\%} \). Among open-source video LMMs, Qwen2.5-VL-72B achieves the highest score of 61.3%, with a large performance gap compared to Gemini-2.0-Flash in video location tasks. Interestingly, Qwen2.5-VL performs best on cultural understanding, yet still achieves only 65.8% in Chinese cultural understanding. In summary, the main contributions are as follows:

• We present the first video evaluation benchmark that covers diverse domains, languages, and cultures in video comprehension.
• We introduce an autonomic video annotation framework, harnessing the strengths of (M)LLMs (including Qwen2-VL and DeepSeek-R1) and visual recognition tools (including SAM2) to improve the efficiency of video annotation.
• We conduct extensive experiments and in-depth
analysis with VideoVista-CulturalLingo, revealing the limitations of existing LMMs in videos with different cultural or linguistic contexts.

2 Related Work

Development of Video LMMs. Unified modalities encodings have become the mainstream approach adopted by LMMs (Li et al., 2025c) over the past year. LongVA (Zhang et al., 2024a) utilizes a unified encoding method, Uni-Res, which allows models trained solely on image datasets to demonstrate strong potential in video evaluation tasks. Qwen2-VL (Wang et al., 2024a) and Qwen2.5VL (Team, 2025) introduce the M-RoPE positional encoding, incorporating temporal, height, and width components, enabling unified positional modeling across text, image, and video modalities. LLaVA-Video (Zhang et al., 2024b) draws inspiration from the SlowFast approach, encoding video frames at varying granularities into visual sequences of different lengths, effectively addressing the issue of excessively long sequences during video encoding. Current LMMs (Chen et al., 2024c; Yao et al., 2024; Li et al., 2024a, 2025b, 2024c) are capable of unified encoding for image and video modalities, leveraging rich image modality data to enhance visual capabilities and demonstrate strong performance in video evaluation tasks.

Progress of Video Benchmark. Video evaluation benchmarks have also made significant progress. Previously, evaluation datasets (Yu et al., 2019; Xu et al., 2017) typically involved posing broad questions and having the model generate a one or a few-word answer, which was then assessed for accuracy and scored by LLMs (Maaz et al., 2024). The videos used in these datasets were often limited to just a few dozen seconds or minutes in length. Recent video benchmarks (Li et al., 2024b) have seen considerable improvements, both in the variety of evaluation tasks and the duration of the videos. Video-MME (Fu et al., 2024) has extended the evaluation video length to an hour, while also introducing twelve distinct evaluation tasks, including Temporal Reasoning and Information Synopsis. MLVU (Zhou et al., 2024) includes videos of varying lengths, ranging from 3 minutes to 2 hours, covering nine different evaluation tasks, such as Needle Question-Answering. The process of video benchmarks (Fang et al., 2024; Wang et al., 2024b; Liu et al., 2024a) have undoubtedly provided a significant boost to the development of LMMs.

3 VideoVista-CulturalLingo

3.1 Video Collecting and Preprocessing

The videos in our study can be divided into two categories: non-scientific and scientific videos. Non-scientific English videos are randomly crawled from YouTube, while their Chinese counterparts are collected from Xiaohongshu to ensure diversity within the dataset. The domains of these videos come from the original categories on the video platforms. For scientific videos, we first identified four major disciplines: mathematics, physics, chemistry, and computer science. Within each discipline, we further defined four representative sub-disciplines, such as linear algebra in mathematics and quantum mechanics in physics. Domains of these videos are derived from search keywords. These sub-disciplines guide the collection of English scientific videos via the YouTube Data API. For Chinese scientific videos, human annotators manually collected videos from BiliBili.

All videos undergo audio extraction via FFmpeg, followed by transcription using Whisper-Large-v3 with sentence-level timestamp alignment. An audio quality assessment pipeline is implemented using Qwen2.5-32B (Yang et al., 2024), evaluating three dimensions: logical coherence, continuity, and information density. Videos are subsequently classified as either audio-rich (high-quality speech) or audio-noisy (including silent videos). For audio-rich videos, the Qwen2.5-72B model segments transcriptions into contextually coherent paragraphs, which are synchronized with visual content through Whisper’s sentence-level alignment to generate short video clips. Audio-noisy videos are processed using the semantics-aware video splitting algorithm from Panda-70M (Chen et al., 2024b), which utilizes visual features to partition videos into semantically consistent segments. This process is illustrated in Figure 3 (a).

To address the challenges of Chinese homophone ambiguity in transcriptions, we develop a context-aware refinement module using Qwen2.5-72B. This module performs three key operations: (1) disambiguation of homophones through semantic analysis, (2) correction of domain-specific terminology, and (3) fluency enhancement, while strictly preserving original semantic content.

3.2 Automatic QA Annotation

The annotation framework comprises four distinct tasks: Event, Culture, Object, and Science. Our
Figure 3: The three-stage annotation process of VideoVista-CulturalLingo.

<table>
  <tr>
    <th>Benchmarks</th>
    <th>#Videos</th>
    <th>#Clips</th>
    <th>Len.(s)</th>
    <th>#QA Pairs</th>
    <th>Anno.</th>
    <th>M.L.</th>
    <th>M.C.</th>
    <th>M.D.</th>
    <th>Open.</th>
  </tr>
  <tr>
    <td>MSRVTT-QA (Xu et al., 2017)</td>
    <td>2,990</td>
    <td>2,990</td>
    <td>15.2</td>
    <td>72,821</td>
    <td>A</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MSVD-QA (Xu et al., 2017)</td>
    <td>504</td>
    <td>504</td>
    <td>9.8</td>
    <td>13,157</td>
    <td>A</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>TGIF-QA (Li et al., 2016)</td>
    <td>9,575</td>
    <td>9,575</td>
    <td>3.0</td>
    <td>8,546</td>
    <td>A&M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>ActivityNet (Yu et al., 2019)</td>
    <td>800</td>
    <td>800</td>
    <td>11.4</td>
    <td>8,000</td>
    <td>M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>TVQA (Lei et al., 2018)</td>
    <td>2,179</td>
    <td>15,253</td>
    <td>11.2</td>
    <td>15,253</td>
    <td>M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>NEXT-QA (Xiao et al., 2021)</td>
    <td>1,000</td>
    <td>1,000</td>
    <td>39.5</td>
    <td>8,564</td>
    <td>A</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MV-Bench (Li et al., 2023)</td>
    <td>3,641</td>
    <td>3,641</td>
    <td>16.0</td>
    <td>4,000</td>
    <td>A</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Ego-Schema (Mangalam et al., 2024)</td>
    <td>5,063</td>
    <td>5,063</td>
    <td>180.0</td>
    <td>5,063</td>
    <td>A&M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>TempCompass (Liu et al., 2024a)</td>
    <td>410</td>
    <td>500</td>
    <td>11.4</td>
    <td>7,540</td>
    <td>A&M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Video-MME (Fu et al., 2024)</td>
    <td>900</td>
    <td>900</td>
    <td>1024.0</td>
    <td>2,700</td>
    <td>M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Video-MLU (Zhou et al., 2024b)</td>
    <td>894</td>
    <td>3,402</td>
    <td>12.0</td>
    <td>24,906</td>
    <td>A</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MLU (Zhou et al., 2024a)</td>
    <td>1,323</td>
    <td>1,233</td>
    <td>720</td>
    <td>2,593</td>
    <td>A&M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>LV-Bench (Wang et al., 2024b)</td>
    <td>500</td>
    <td>500</td>
    <td>4,101.0</td>
    <td>1,549</td>
    <td>M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>MMBench-Video (Fang et al., 2024)</td>
    <td>600</td>
    <td>600</td>
    <td>165.4</td>
    <td>1,998</td>
    <td>M</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <th>VideoVista-CulturalLingo</th>
    <th>1,389</th>
    <th>2,052</th>
    <th>267.5</th>
    <th>3,134</th>
    <th>A&M</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
  </tr>
</table>

Table 1: The comparison of various benchmarks involves several key aspects: total number of videos (#Videos), number of clips (#Clips), average video duration (Len.), number of QA pairs (#QA Pairs), annotation method (Anno., where M/A indicates manual/automatic annotation), whether the videos span multiple language (M.L.), whether the videos span multiple culture background (M.C.), whether the videos span multiple duration levels (M.D.), and if the videos are sourced from diverse open domains (Open.)

pipeline employs Qwen2-VL-72B as the primary annotator, Qwen2.5-72B for text-only annotation tasks, and paraphrase-multilingual-MiniLM-L12-v2 for embedding generation. For non-scientific tasks, DeepSeek-V3 (DeepSeek-AI et al., 2024) is employed as the question generator, while DeepSeek-R1 (DeepSeek-AI et al., 2025) is used for generating scientific questions. During the annotation process, while generating questions, four options and the correct answer are also created. The process of automatic QA annotation is illustrated in Figure 3 (b). The details and prompt for annotation is provided in Appendix D.

Event. We input the segmented video clips and refined audio transcriptions into the event annotator to label the events occurring in each video segment. For the i-th segment, the model receives historical event annotations from the previous \( i-1 \) segments to maintain temporal consistency. Each annotated segment follows the structure (event, audio, start, end), where start and end denote the timestamps marking the beginning
and conclusion of the current video segment within the full video. The aggregated event sequence is then fed into the question generator, which generates questions of the corresponding task, along with four options for each question and correct answer. Specifically, for event prediction questions, the model is instructed to select the segment that is most logically related to the preceding context as the predicted content. During this process, each task is associated with a specific prompt.

Object. We feed videos into the object classifier to filter those videos that meet three criteria: real-world content, richness in objects, and motion in objects. The filtered videos are then processed by the object extractor to identify three to five primary objects followed by frame-wise presence detection via InternVL2-8B at 1fps sampling. The detected objects are processed through a pipeline combining Grounding-DINO (Liu et al., 2023a) for bounding box prediction and SAM2 (Ravi et al., 2024) for image segmentation. The resulting information is then fed into the object description annotator to generate object-level descriptions that capture both the temporal and spatial aspects of each object. Finally, the object-level descriptions, along with the aggregated event sequence, are input into the question generator to generate the questions.

Culture. We input videos and audio transcriptions into the cultural classifier to evaluate their relationship to Chinese, American, and European cultures individually. Culturally relevant videos are then processed by the cultural concept extractor to identify the two most prominent cultural concepts. These cultural concepts are subsequently encoded into embeddings, which are used to retrieve the entries from pre-encoded Wikipedia data. Using these entries, along with a local backup of Wikipedia, we can retrieve Wikipedia articles corresponding to the identified cultural concepts. By combining this external knowledge with the aggregated event sequence, we input the data into the specific question generator to generate the questions.

Science. The video is input into the science classifier to evaluate its quality based on scientific thematic relevance and knowledge density. After filtering, the aggregated event sequence of the video is fed into the question generator, DeepSeek-R1, to generate questions. In our initial experiments, we observed two recurring issues: generated questions either relied excessively on domain knowledge—detached from the video itself and thus answerable without viewing—or exhibited distractor choices that were either too divergent or too similar, producing items that were trivial or ambiguous. To resolve these shortcomings, we impose deterministic, rule-based constraints that (i) require every question to depend on video context for its solution and (ii) ensure a balanced, pedagogically meaningful set of answer options. Specifically, each question presents four choices: Correct Option, Video Comprehension Error, Domain Knowledge Error, and Dual Error. This structured design rigorously evaluates a model’s ability to integrate visual comprehension with scientific reasoning.

3.3 Human Check and Revision
All candidate questions are first filtered linguistically using Qwen2.5-7B with the CircularEval strategy (Liu et al., 2023b) to remove any video-agnostic items. We then establish a Gradio-based annotation platform that includes three assessment dimensions: correctness, type relevance, and video relevance. The correctness score ranges from 0 to 1, assessing whether the model-generated answer is correct; the type relevance score ranges from 0 to 2, evaluating the degree of relevance between the question and task type; and the video relevance score ranges from 0 to 2, determining the degree of relevance between the question and the video content, ensuring that questions are not unrelated to the video frames. Questions achieving maximum scores (score=5) across all dimensions are selected. For borderline cases (score=4), we utilize differentiated handling: first, for the question with wrong answer (correctness=0), we manually correct the answers; second, for the question with suboptimal type or video relevance, we manually refine the questions, options, and answers based on the original questions. We have illustrated this process in the Figure 3 (c). Specifically for cultural questions, two annotators—one of whom is a native speaker of the relevant culture—individually assess each question to ensure cross-validation. Overall, this hybrid automatic/manual pipeline eliminates approximately 60 % of low-quality questions.

3.4 Statistic and Analysis
As shown in Figure 2, VideoVista-CulturalLingo consists of 2,052 video clips or full videos derived from 1,389 original videos, with an average duration of 267.5 seconds. Additionally, VideoVista-CulturalLingo contains 1,446 questions in Chinese
<table>
  <tr>
    <th>Model</th>
    <th>LLM</th>
    <th>Frames</th>
    <th>Overall</th>
    <th>Event</th>
    <th>Object</th>
    <th>Culture</th>
    <th>Science</th>
  </tr>
  <tr>
    <th colspan="8">Open-source Video LMMs</th>
  </tr>
  <tr>
    <td>ShareGPT4Video (Chen et al., 2024a)</td>
    <td>Vicuna-7B-v1.5</td>
    <td>16f</td>
    <td>25.6</td>
    <td>23.2</td>
    <td>18.9</td>
    <td>31.4</td>
    <td>34.1</td>
  </tr>
  <tr>
    <td>VideoChat2-Mistral (KunChang et al., 2023)</td>
    <td>Mistral-7B-Instruct-v0.2</td>
    <td>16f</td>
    <td>29.6</td>
    <td>27.5</td>
    <td>25.9</td>
    <td>34.7</td>
    <td>33.1</td>
  </tr>
  <tr>
    <td>Video-LLaYA (Lin et al., 2023a)</td>
    <td>Vicuna-7B-v1.5</td>
    <td>8f</td>
    <td>38.2</td>
    <td>42.2</td>
    <td>34.4</td>
    <td>34.5</td>
    <td>41.1</td>
  </tr>
  <tr>
    <td>VideoLLaMA2 (Cheng et al., 2024)</td>
    <td>Mistral-7B-Instruct-v0.2</td>
    <td>32f</td>
    <td>31.4</td>
    <td>33.6</td>
    <td>23.3</td>
    <td>34.9</td>
    <td>36.6</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision (Li et al., 2024a)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>32f</td>
    <td>41.8</td>
    <td>43.9</td>
    <td>33.8</td>
    <td>38.8</td>
    <td>53.5</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6 (Yao et al., 2024)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>1fps(64)</td>
    <td>42.9</td>
    <td>44.1</td>
    <td>24.1</td>
    <td>49.4</td>
    <td>62.9</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3 (Ye et al., 2024)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>1fps(128)</td>
    <td>49.9</td>
    <td>54.4</td>
    <td>41.9</td>
    <td>45.0</td>
    <td>60.1</td>
  </tr>
  <tr>
    <td>Oryx-1.5 (Liu et al., 2024b)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>128f</td>
    <td>41.4</td>
    <td>43.8</td>
    <td>32.2</td>
    <td>37.6</td>
    <td>55.8</td>
  </tr>
  <tr>
    <td>LLaVA-Video (Zhang et al., 2024b)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>1fps(64)</td>
    <td>51.0</td>
    <td>57.9</td>
    <td>39.1</td>
    <td>48.8</td>
    <td>60.3</td>
  </tr>
  <tr>
    <td>Qwen2-VL (Wang et al., 2024a)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>1fps(300)</td>
    <td>49.7</td>
    <td>50.1</td>
    <td>33.8</td>
    <td>54.8</td>
    <td>68.0</td>
  </tr>
  <tr>
    <td>InternVL2.5 (Chen et al., 2024c)</td>
    <td>Internlm2.5-7b-Chat</td>
    <td>64f</td>
    <td>52.0</td>
    <td>56.5</td>
    <td>35.5</td>
    <td>56.1</td>
    <td>65.7</td>
  </tr>
  <tr>
    <td>MiniCPM-o 2.6 (Yao et al., 2024)</td>
    <td>Qwen2.5-7B-Instruct</td>
    <td>1fps(64)</td>
    <td>49.0</td>
    <td>52.9</td>
    <td>28.5</td>
    <td>55.9</td>
    <td>67.1</td>
  </tr>
  <tr>
    <td>TPO (Li et al., 2025a)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>1fps(96)</td>
    <td>50.6</td>
    <td>57.2</td>
    <td>37.8</td>
    <td>49.6</td>
    <td>60.4</td>
  </tr>
  <tr>
    <td>InternVideo2.5 (Wang et al., 2025)</td>
    <td>Internlm2.5-7b-Chat</td>
    <td>1fps(512)</td>
    <td>52.0</td>
    <td>52.5</td>
    <td>38.1</td>
    <td>58.2</td>
    <td>65.9</td>
  </tr>
  <tr>
    <td>VideoLLaMA3 (Zhang et al., 2025)</td>
    <td>Qwen2.5-7B-Instruct</td>
    <td>1fps(180)</td>
    <td>60.7</td>
    <td>58.0</td>
    <td><b>66.4</b></td>
    <td>53.1</td>
    <td>64.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B (Team, 2025)</td>
    <td>Qwen2.5-7B-Instruct</td>
    <td>1fps(300)</td>
    <td>54.3</td>
    <td>56.7</td>
    <td>38.9</td>
    <td>55.2</td>
    <td>73.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-72B (Team, 2025)</td>
    <td>Qwen2.5-7B-Instruct</td>
    <td>1fps(300)</td>
    <td><b>61.3</b></td>
    <td><b>61.0</b></td>
    <td>40.5</td>
    <td><b>71.2</b></td>
    <td><b>83.3</b></td>
  </tr>
  <tr>
    <th colspan="8">Open-source Image LMMs</th>
  </tr>
  <tr>
    <td>VILA1.5-13B (Lin et al., 2023b)</td>
    <td>Vicuna-13B-v1.5</td>
    <td>1f</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>29.2</td>
    <td>33.9</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>VILA1.5-13B (Lin et al., 2023b)</td>
    <td>Vicuna-13B-v1.5</td>
    <td>8f</td>
    <td>36.9</td>
    <td>38.2</td>
    <td>31.3</td>
    <td>38.2</td>
    <td>41.9</td>
  </tr>
  <tr>
    <td>Molmo-7B-D (Deitke et al., 2024)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>1f</td>
    <td>38.3</td>
    <td>44.5</td>
    <td>25.3</td>
    <td>39.8</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>Molmo-7B-D (Deitke et al., 2024)</td>
    <td>Qwen2-7B-Instruct</td>
    <td>8f</td>
    <td>40.3</td>
    <td>44.3</td>
    <td>30.1</td>
    <td>41.8</td>
    <td>48.0</td>
  </tr>
  <tr>
    <td>DeepSeek2-VL (Wu et al., 2024)</td>
    <td>DeepSeekMoE-27B</td>
    <td>1f</td>
    <td>40.9</td>
    <td>44.3</td>
    <td><b>32.2</b></td>
    <td>39.3</td>
    <td>50.5</td>
  </tr>
  <tr>
    <td>DeepSeek2-VL (Wu et al., 2024)</td>
    <td>DeepSeekMoE-27B</td>
    <td>8f</td>
    <td><b>42.6</b></td>
    <td><b>47.0</b></td>
    <td>27.2</td>
    <td><b>44.4</b></td>
    <td><b>57.5</b></td>
  </tr>
  <tr>
    <th colspan="8">Proprietary LMMs</th>
  </tr>
  <tr>
    <td>GPT-4o-2024-11-20</td>
    <td>GPT-4o</td>
    <td>1fps(128)</td>
    <td>56.7</td>
    <td>53.4</td>
    <td>38.2</td>
    <td><b>68.0</b></td>
    <td>78.3</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Flash</td>
    <td>Gemini-1.5-Flash</td>
    <td>1fps</td>
    <td>69.4</td>
    <td>70.0</td>
    <td>65.8</td>
    <td>59.0</td>
    <td>84.7</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash-Lite</td>
    <td>Gemini-2.0-Flash-Lite</td>
    <td>1fps</td>
    <td>70.7</td>
    <td>63.1</td>
    <td>71.6</td>
    <td>63.1</td>
    <td>82.1</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>Gemini-2.0-Flash</td>
    <td>1fps</td>
    <td><b>76.3</b></td>
    <td><b>74.0</b></td>
    <td><b>77.1</b></td>
    <td><b>68.0</b></td>
    <td><b>87.4</b></td>
  </tr>
</table>

Table 2: Evaluation results on VideoVista-CulturalLingo benchmark. The large language model used by LMMs (LLM), frames sample strategy (Frames), overall evaluation scores (Overall), evaluation scores in Event Task(Event), evaluation scores in Object Task (Object), evaluation scores in Culture Task (Culture), evaluation scores in Science Task (Science). -[Nf] indicates this LMM task N frames uniformly sampled from a video as input. -[Nfps(M)] indicates this LMM uses N frames per second uniformly sampled from a video as input, with a max frames number M. We have highlighted the highest results in each tasks using bold. Meanwhile, the highest results within the 7B/8B open-source Video LMMs are highlighted with an underline.

and 1,668 questions in English, with a comparable number of questions in both languages. In Table 1, we compare the key characteristics of our benchmark with others. Notably, VideoVista-CulturalLingo includes the largest collection of raw videos, totaling 1,389, among benchmarks that have videos multiple duration levels. These 1,389 original videos encompass a diverse range of languages and cultural backgrounds, a feature that sets our benchmark apart from previous ones. Additionally, we employed a annotation approach that combines (M)LLM preliminary annotation with human verification and question revision.

4 Experiment

4.1 Baselines

We conducted evaluations on 17 open-source video LMMs, 3 image LMMs, and 4 proprietary LMMs, including the recently released Gemini-2.0-Flash, Qwen2.5-VL (Team, 2025), VideoLLaMA3 (Zhang et al., 2025), DeepSeek2-VL (Wu et al., 2024), among others. The detailed experiment settings are shown in Appendix B.

4.2 Main Results

As shown in Table 2, Qwen2.5-VL-72B exhibits the best performance among all open-source video LMMs, achieving an overall score of 61.3%. Additionally, VideoLLaMA3 demonstrates the best performance among all 7B/8B models, with an overall score of 60.7%. This is primarily due to VideoLLaMA3’s exceptional capabilities in fine-grained object tasks, making it the only open-source LMM that can compete with proprietary LMMs in this task. In the event task, VideoLLaMA3 also outperforms all other 7B models. Among the open-source image LMMs, DeepSeek2-VL achieved the highest score of 42.6% under 8-frame uniform sam-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Event</th>
    <th colspan="3">Object</th>
    <th colspan="3">Culture</th>
    <th colspan="4">Science</th>
  </tr>
  <tr>
    <th>ED</th>
    <th>EP</th>
    <th>ES</th>
    <th>EL</th>
    <th>OTL</th>
    <th>OTS</th>
    <th>OSL</th>
    <th>CC</th>
    <th>AC</th>
    <th>EC</th>
    <th>SS</th>
    <th>COM</th>
    <th>AP</th>
    <th>SP</th>
  </tr>
  <tr>
    <td>MiniCPM-0.2.6</td>
    <td><b>83.6</b></td>
    <td>55.0</td>
    <td>53.1</td>
    <td>35.2</td>
    <td>20.1</td>
    <td>52.4</td>
    <td>35.7</td>
    <td>48.9</td>
    <td>56.3</td>
    <td>63.7</td>
    <td>72.1</td>
    <td>61.3</td>
    <td>69.5</td>
    <td>52.7</td>
  </tr>
  <tr>
    <td>InternVideo2.5</td>
    <td>80.5</td>
    <td>52.7</td>
    <td>60.3</td>
    <td>33.0</td>
    <td>37.1</td>
    <td>61.2</td>
    <td>31.8</td>
    <td>53.7</td>
    <td>56.3</td>
    <td>65.2</td>
    <td>72.1</td>
    <td>61.3</td>
    <td>64.0</td>
    <td>54.8</td>
  </tr>
  <tr>
    <td>VideoLLaMA3</td>
    <td>77.9</td>
    <td>57.4</td>
    <td>61.7</td>
    <td>45.2</td>
    <td><b>72.1</b></td>
    <td>64.1</td>
    <td><b>56.6</b></td>
    <td>45.5</td>
    <td>55.8</td>
    <td>59.2</td>
    <td>70.2</td>
    <td>54.7</td>
    <td>64.0</td>
    <td>55.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-72B</td>
    <td>79.2</td>
    <td><b>60.5</b></td>
    <td><b>78.9</b></td>
    <td>42.1</td>
    <td>31.5</td>
    <td>67.0</td>
    <td>49.7</td>
    <td><b>65.8</b></td>
    <td><b>67.8</b></td>
    <td><b>80.6</b></td>
    <td><b>86.4</b></td>
    <td><b>85.3</b></td>
    <td><b>79.3</b></td>
    <td><b>79.6</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>86.3</td>
    <td>47.3</td>
    <td>70.3</td>
    <td>28.6</td>
    <td>29.4</td>
    <td>61.2</td>
    <td>46.5</td>
    <td>57.1</td>
    <td>71.9</td>
    <td>76.6</td>
    <td>81.6</td>
    <td>77.3</td>
    <td>80.5</td>
    <td>65.6</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>92.9</td>
    <td>51.9</td>
    <td>73.7</td>
    <td>70.7</td>
    <td>87.2</td>
    <td>74.8</td>
    <td>59.1</td>
    <td>62.3</td>
    <td>64.8</td>
    <td>77.6</td>
    <td>88.2</td>
    <td>87.8</td>
    <td>81.7</td>
    <td>90.7</td>
  </tr>
</table>

Table 3: Detailed Evaluation results on VideoVista-CulturalLingo benchmark. We only showcase 6 mainstream LMMs. Abbreviations used in the table: Event Description (ED), Event Prediction (EP), Event Sequence (ES), Event Localization (EL), Object Temporal Localization (OTL), Object Temporal Sequence (OTS), Object Spatial Localization (OSL), Chinese Culture (CC), American Culture (AC), European Culture (EC), Summarization & Synthesis (SS), Comparison & Contrast (COM), Application & Procedure (AP), Scientific Principle (SP). The full evaluation results are provided in the Appendix C.5, and an introduction to tasks is presented in Appendix E.

![Bar charts showing evaluation results by culture, language, and duration](page_101_1012_1242_246.png)

(a) Culture-based Evaluation Results. (b) Language-based Evaluation Results. (c) Duration-based Evaluation Results.

Figure 4: The LMMs performance divided by Culture, Language and Duration. The Duration in (c): <2 minutes (Short), 2–10 minutes (Medium), >10 minutes (Long).

pling, demonstrating its superior generalization capacity on sequential image data. However, this still shows a gap compared to the leading open-source video LMMs, indicating that questions in VideoVista-CulturalLingo generally require longer video durations to answer. Among proprietary LMMs, Gemini-2.0-Flash clearly outperforms all others, surpassing the strongest open-source video LMM, Qwen2.5-VL-72B, by 15.0%. The largest performance gap between these two models is observed in fine-grained object understanding tasks.

4.3 Detailed Analysis

We present the detailed evaluation results of 6 mainstream models across 14 sub-tasks in Table 3.

Event. The Event task consists of four sub-tasks: Event Description, Event Prediction, Event Sequence, and Event Localization, all of which require the model to have a coarse-grained understanding of video content. Current open-source video LMMs exhibit performance comparable to that of proprietary LMMs on the first three sub-tasks, but there remains a gap in the Event Localization task when compared to Gemini-2.0-Flash, with a performance difference of up to 25.5%.

Object. The Object task consists of three sub-tasks: Object Temporal Localization, Object Temporal Sequence, and Object Spatial Localization, which assess the LMMs’ ability to perceive the spatial-temporal aspects of fine-grained objects in videos. Video-LLaMA3 and Gemini-2.0-Flash demonstrate strong temporal localization capabilities in the Object Temporal Localization task, achieving scores more than 30% higher than those of other LMMs. Additionally, both LMMs exhibit commendable spatial understanding in the Object Spatial Localization task.

Culture. The Culture task consists of three sub-tasks: Chinese Culture, American Culture, and European Culture, primarily evaluating the model’s understanding and generalization abilities across different regional cultures. As shown in Figure 4a, compared to the more prevalent Western cultures in the training data, current LMMs exhibit relatively weaker recognition of Chinese Culture.

Science. The Science task consists of four sub-tasks: Summarization & Synthesis, Comparison & Contrast, Application & Procedure, and Scientific Principle. The first three sub-tasks involve
Figure 5: The LMMs performance divided by domains from 3 video sources: Gemini-2.0-Flash, GPT-4o, Qwen2.5-VL-72B, VideoLLaMA3, InternVideo2.5, MiniCPM-o 2.6. In Figures 5a and Figures 5b, we present only the 18 domains with the highest number of videos. In Figure 5c, we exclude domains containing fewer than 10 videos. The domains in these figures are represented by abbreviations, as described in Appendix A.2.

Figure 6: Two cases from VideoVista-CulturalLingo. The ground truth is highlighted in yellow. The correct answers of LMMs are highlighted in green, and the incorrect answers are highlighted in red.

course-oriented educational videos, while the last one focuses on experimental videos. This task primarily evaluates the model’s ability to summarize, comprehend, and apply scientific knowledge from videos. The difficulty level covers general knowledge areas rather than in-depth specialized topics. The questions are relatively simple and can be answered with one or two-hop reasoning, so most models perform well in these tasks. We observe that existing open-source LMMs perform comparably to proprietary LMMs across most disciplines. However, there remains a noticeable gap in performance within math. The detailed comparison is presented in the Appendix C.1.

4.4 Ablation Study

Language. In Figure 4b, we present the performance differences of 6 mainstream LMMs on Chinese and English. The results in the figure are based on 7 subtasks from the culture and science tasks, as these subtasks contain more domain-specific terms, providing a more accurate assessment of an LMM’s capabilities in each respective language. The experiments reveal a noticeable performance gap between the majority of mainstream LMMs when evaluated on Chinese versus English.

Duration. In Figure 4c, we compare the performance of 6 mainstream LMMs across 4 subtasks of event task from videos of varying lengths. The experimental results indicate that as the video duration increases, the performance of model tends to decrease, including Gemini-2.0-Flash.

Domain. In Figure 5, we illustrate the performance of LMMs across different video domains on various video websites. It can be observed that Gemini-2.0-Flash demonstrates strong performance across all domains of videos.

4.5 Case Study

Figure 6 presents two cultural examples alongside evaluation results: the top case illustrates a Chinese cuisine scenario, and the bottom case illustrates a European cuisine scenario. In the Chinese example, the majority of LMMs erroneously choose “C. Stir-fried Yellow Beef,” a hallmark Hunan dish. This mistake likely arises from conflation between Jiangxi and Hunan cuisines—both characterized by liberal use of chili peppers—and from the greater domestic and international visi-
bility of Hunan cooking. Such errors reveal that Video-LMMs tend to default to dominant cultural representations, overlooking more localized culinary nuances. By contrast, in the European example all LMMs correctly select option D, indicating robust performance on Western culinary content. Together, these cases exemplify a systematic bias: Video-LMMs achieve higher accuracy on Western cultural contexts but underperform on non-Western ones, such as videos rooted in Chinese culture.

5 Conclusion

In this paper, we introduce the benchmark VideoVista-CulturalLingo, the first video evaluation benchmark that spans multiple languages, cultures, and domains. VideoVista-CulturalLingo includes comprehensive evaluation metrics, ranging from coarse-grained event understanding to fine-grained object recognition, and from exploring the cultural context of videos to uncovering their scientific implications, enabling a comprehensive assessment of current LMMs’ capabilities on video tasks. Through our extensive experiments, we highlight weaknesses in the spatial-temporal localization abilities of existing open-source video LMMs and their limitations in recognizing Chinese culture. We hope that VideoVista-CulturalLingo will inspire the development and advancement of video LMMs.

6 Acknowledge

We thank editor and reviewers for their efforts to help improve the quality of our paper. This work was jointly supported by grants: Natural Science Foundation of China (No. 62422603), Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515110078), Shenzhen Science and Technology Program (No. ZDSYS20230626091203008). This work was also supported in part by HKUST-MetaX Joint Laboratory for Advanced AI Computing (Grant No. METAX24EG01-D)

Limitations

The proposed benchmark has several limitations: 1) The scientific questions in the benchmark lack domain-specific depth, which prevents them from effectively showcasing the model’s performance in specialized scientific fields. In future versions, we plan to incorporate more human expert annotators to enhance the professionalism and complexity of the scientific questions. 2) Due to limitations in the linguistic proficiency and backgrounds of the annotators, the benchmark questions are restricted to two major languages, Chinese and English. This excludes other widely spoken languages such as Spanish, Portuguese, German, and Japanese.

References

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. 2024a. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325.

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. 2024b. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jingguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024c. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-lmms. arXiv preprint arXiv:2406.07476.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengdu Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing
Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaokang Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yu-jia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zhihu Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incen-tivizing reasoning capability in llms via reinforcement learning. Preprint. arXiv:2501.12948.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Le'an Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qiaohu Zha, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaobang Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yan-hong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yuja He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zhihu Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. 2024. Deepseek-v3 technical report. Preprint. arXiv:2412.19437.

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Fayyen Bastani, Eli Vanderbilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tamay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Witlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146.

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406.14515.

Chaoyou Fu, Yuhai Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renru Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragnemeni, Qichen Fu, Abhram Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Wesley Khoo, Jáchym Kolář, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu,
Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz, Meray Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995–19012.

Li KunChang, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355.

Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. Tvqa: Localized, compositional video question answering. In EMNLP.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2023. Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.17005.

Rui Li, Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Yeung-Levy. 2025a. Temporal preference optimization for long-form video understanding. Preprint, arXiv:2501.13919.

Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetrelaut, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. 2016. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650.

Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. 2024b. Videovista: A versatile benchmark for video understanding and reasoning. Preprint, arXiv:2406.11303.

Yunxin Li, Baotian Hu, Xinyu Chen, Lin Ma, Yong Xu, and Min Zhang. 2024c. Lmeye: An interactive perception network for large language models. IEEE Transactions on Multimedia, 26:10952–10964.

Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. 2025b. Uni-moe: Scaling unified multimodal llms with mixture of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–15.

Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinpeng Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, and Min Zhang. 2025c. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.04921.

Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023a. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122.

Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. 2023b. Vila: On pre-training for visual language models. Preprint, arXiv:2312.07533.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023a. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023b. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281.

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishou Chen, Xu Sun, and Lu Hou. 2024a. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476.

Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. 2024b. Oryx mllm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961.

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024).

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2024. Egoschema: A diagnostic benchmark for very long-form video language understanding. In NeurIPS.

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädele, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. 2024. Sam 2: Segment anything in images and videos. Preprint, arXiv:2408.00714.

Qwen Team. 2025. Qwen2.5-v1.
Peng Wang, Shuai Bai, Siman Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2024b. Lyvbench: An extreme long video understanding benchmark. Preprint, arXiv:2406.08035.

Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. 2025. Internvideo2.5: Empowering video mlms with long and rich context modeling. arXiv preprint arXiv:2501.12386.

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. 2024. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. Preprint, arXiv:2412.10302.

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777–9786.

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tanchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024a. Minicpm-v: A gpt-4v level mlm on your phone. arXiv preprint arXiv:2408.01800.

Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. Preprint, arXiv:2408.04840.

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127–9134.

Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, and Qin Jin. 2023. Movie101: A new movie understanding benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4669–4684.

Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenci Zhang, Fan Wang, Lidong Bing, and Deli Zhao. 2025. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106.

Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024a. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852.

Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024b. Video instruction tuning with synthetic data. Preprint, arXiv:2410.02713.

Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264.

A Additional Dataset Statistics

A.1 Further Statistics

In Figure 7a, we present the statistics for all task categories in VideoVista-CulturalLingo. In VideoVista-CulturalLingo, the number of English questions is slightly higher than that of Chinese questions, with an additional 222 English questions. The task type with the fewest questions in the dataset is "Comparison & Contrast", with a total of only 75 questions, while the task type with the most questions is "Object Temporal Localization," with a total of 537 questions. Figure 7b (b) shows the temporal distribution of video clips. Due to the fine-grained object recognition task, the selected videos are often short segments of longer videos, resulting in a larger proportion of videos that are
(a) The statistics of 14 subtasks divided by languages.

(b) The statistics of duration of videos in VideoVista-CulturalLingo.

Figure 7: (a) shows the quantity statistics for the 14 task categories under both Chinese and English languages. (b) presents the duration statistics of all video clips in VideoVista-CulturalLingo, measured in minutes.

under one minute in length in the dataset. However, VideoVista-CulturalLingo still contains 315 videos longer than 10 minutes, with these long videos primarily concentrated in the Event and Science task categories.

A.2 Abbreviations of Domains
We provided the abbreviations of domains in Figure 5 in Table 4.

B Detailed Experiment Setting

B.1 Open-source Video LMMs
We evaluated the newly released Qwen2.5-VL (Team, 2025), VideoLLaMA3 (Zhang et al., 2025), InternVideo2.5 (Wang et al., 2025), and TPO (Li et al., 2025a) from 2025. Additionally, we evaluated several popular video-capable LMMs introduced in the past two years, including InternVL2.5 (Chen et al., 2024c), LLava-Video (Zhang et al., 2024b), mPLUG-Owl3 (Ye et al., 2024), and others.

In evaluating open-source video LMMs, we use the default hyperparameters specified in their respective open-source implementations for inference. The temperature is generally set to 0 or 0.2, num_beamsis set to 1, do_sampleis set to False, and top_pis set to 1.0. The frame sampling methods for different video models are provided in the Table 2. Specifically, for the Qwen2.5-VL and Qwen2-VL models, we set the maximum resolution per frame to 224x224 to avoid excessively long sequence lengths.

B.2 Open-source Image LMMs
We also evaluated three open-source image LMMs on our benchmarks, including VILA 1.5 (Lin et al., 2023b), DeepSeek2-VL (Wu et al., 2024), and Molmo (Deitke et al., 2024). For open-source image LMMs, we employed two video input methods: uniform sampling of 1 frame and uniform sampling of 8 frames.

In evaluating these open-source image LMMs, we also adopted the hyperparameter settings provided in the implementations for inference. Regardless of whether single-frame or eight-frame input is used for evaluation, all images are presented at their original resolution without compression. Specifically, due to an error in the official code of the Molmo model when inputting eight images simultaneously, we concatenated the eight images horizontally into a single image and noted this in the prompt. An example of this image is Figure 8.

B.3 Proprietary LMMs
For proprietary LMMs, we evaluated the newly released Gemini 2.0-Flash and Gemini 2.0-Flash-Lite in February, which are currently the workhorse models of the Google Gemini series. Additionally, we conducted evaluations on other prominent proprietary LMMs, including GPT-4o and Gemini 1.5-Flash.

In evaluating proprietary LMMs, we optimize API resource usage and accelerate the evaluation
Figure 8: An example of eight images combined in a horizontal layout.

<table>
  <tr>
    <th>Full Name</th>
    <th>Abbreviation</th>
  </tr>
  <tr>
    <th colspan="2">YouTube Domains</th>
  </tr>
  <tr>
    <td>News & Politics</td>
    <td>NP</td>
  </tr>
  <tr>
    <td>Sports</td>
    <td>Spt</td>
  </tr>
  <tr>
    <td>Entertainment</td>
    <td>Ent</td>
  </tr>
  <tr>
    <td>Howto & Style</td>
    <td>HS</td>
  </tr>
  <tr>
    <td>People & Blogs</td>
    <td>PB</td>
  </tr>
  <tr>
    <td>Autos & Vehicles</td>
    <td>AV</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>Edu</td>
  </tr>
  <tr>
    <td>Travel & Events</td>
    <td>TE</td>
  </tr>
  <tr>
    <td>Film & Animation</td>
    <td>FA</td>
  </tr>
  <tr>
    <td>Comedy</td>
    <td>Com</td>
  </tr>
  <tr>
    <td>Chemical Experiments</td>
    <td>CE</td>
  </tr>
  <tr>
    <td>Science & Technology</td>
    <td>ST</td>
  </tr>
  <tr>
    <td>Artificial Intelligence</td>
    <td>AI</td>
  </tr>
  <tr>
    <td>Physics Experiment</td>
    <td>PE</td>
  </tr>
  <tr>
    <td>Pets & Animals</td>
    <td>PA</td>
  </tr>
  <tr>
    <td>Quantum Mechanics</td>
    <td>QM</td>
  </tr>
  <tr>
    <td>Calculus</td>
    <td>Cal</td>
  </tr>
  <tr>
    <td>Linear Algebra</td>
    <td>LA</td>
  </tr>
  <tr>
    <th colspan="2">Xiaohongshu Domains</th>
  </tr>
  <tr>
    <td>Travel Scenery</td>
    <td>TS</td>
  </tr>
  <tr>
    <td>Cooking Process</td>
    <td>CP</td>
  </tr>
  <tr>
    <td>Cooking Tutorial</td>
    <td>CT</td>
  </tr>
  <tr>
    <td>Entrepreneurship</td>
    <td>Ent</td>
  </tr>
  <tr>
    <td>TV Series Commentary</td>
    <td>TSC</td>
  </tr>
  <tr>
    <td>Tourist Attractions</td>
    <td>TA</td>
  </tr>
  <tr>
    <td>Food Review</td>
    <td>FR</td>
  </tr>
  <tr>
    <td>Food Exploration</td>
    <td>FE</td>
  </tr>
  <tr>
    <td>Food Curiosities</td>
    <td>FC</td>
  </tr>
  <tr>
    <td>Astronomy Knowledge</td>
    <td>AK</td>
  </tr>
  <tr>
    <td>Art Explanation</td>
    <td>AE</td>
  </tr>
  <tr>
    <td>Historical Gossip</td>
    <td>HG</td>
  </tr>
  <tr>
    <td>Product Information</td>
    <td>PI</td>
  </tr>
  <tr>
    <td>Travel Vlog</td>
    <td>TV</td>
  </tr>
  <tr>
    <td>Fashion Trends</td>
    <td>FT</td>
  </tr>
  <tr>
    <td>Travel Guide</td>
    <td>TG</td>
  </tr>
  <tr>
    <td>Civil Service Exam Preparation</td>
    <td>CSEP</td>
  </tr>
  <tr>
    <td>Relationship Issues</td>
    <td>RI</td>
  </tr>
  <tr>
    <th colspan="2">BiliBili Domains</th>
  </tr>
  <tr>
    <td>Organic Chemistry</td>
    <td>OC</td>
  </tr>
  <tr>
    <td>Advanced Mathematics</td>
    <td>AM</td>
  </tr>
  <tr>
    <td>High School Experiments</td>
    <td>HSE</td>
  </tr>
  <tr>
    <td>Mid School Experiments</td>
    <td>MSE</td>
  </tr>
  <tr>
    <td>University Physics</td>
    <td>UP</td>
  </tr>
  <tr>
    <td>Machine Learning</td>
    <td>ML</td>
  </tr>
  <tr>
    <td>Deep Learning</td>
    <td>DL</td>
  </tr>
  <tr>
    <td>Quantum Mechanics</td>
    <td>QM</td>
  </tr>
</table>

Table 4: Abbreviations of domains from different video websites in Figure 5. The Chinese domains have been translated into English using GPT-4o.

process by input multiple questions for each video. Thanks to the powerful instruction-following capability of Proprietary LMMs, they are able to return a dictionary in the format of {"question id": "prediction"} accurately. Although this may introduce some evaluation bias, Proprietary LMMs still demonstrated exceptional performance on our benchmark. Additionally, when evaluating the GPT-4 model, we compressed all video frames to a resolution of 512x512 for input.

C Further Experiments

C.1 Model Performance in Science

For the third finding discussed in the abstract, we present detailed experimental results in Figure 9. We present a performance comparison between the four best-performing open-source video LMMs and the strongest proprietary model, Gemini-2.0-Flash. As shown in Figure 9a, the primary performance gap between open-source Video LMMs and proprietary LMMs in scientific tasks is observed in the Mathematics disciplines. Specifically, for Physics, Chemistry, and Computer Science questions, the top-performing open-source Video LMM, Qwen2.5-VL-72B, exhibits a performance gap of less than 5% compared to Gemini-2.0-Flash. However, for Math questions, the gap between the two models increases to nearly 10%.

In Figure 9b, we further compare the performance differences of various models across specific math sub-disciplines. It is evident that, regardless of whether the questions are in Chinese or English, existing open-source video LMMs still exhibit a performance gap when compared to the proprietary LMM Gemini-2.0-Flash. The largest gaps are observed in the Calculus (English) and Statistics and Probability (English) categories, where the leading open-source video LMMs show a performance difference exceeding 10% compared to Gemini-2.0-Flash.
(a) Science-based Evaluation Results.

(b) Math-based Evaluation Results.

Figure 9: The Evaluation results in 4 disciplines and 4 math sub-disciplines. The experimental results in the figure represent the average values of the four scientific sub-tasks. In (a), we have list the four disciplines covered by the scientific videos in VideoVista-CulturalLingo: Math, Physics, Chemistry, and Computer Science; In (b), we have listed four math sub-disciplines with a larger number of questions: Calculus (English), Linear Algebra (English), Statistics and Probability (English), and Calculus (Chinese)/Advanced Mathematics.

Figure 10: The Evaluation results divided by frames upper bound of Qwen2.5-VL-7B. We conducted experiments with four sampling methods at frame upper bound of 64, 128, 256, and 300 frames.

C.2 Impact of Frame Sampling

We conduct an experiment to evaluate the frame sampling upper bound for event task questions using the Qwen2.5-VL-7B model, and the results are shown in the Figure 10. It can be observed that as the frame sampling upper upper bound increases, the overall evaluation performance of the model gradually improves. However, there is no significant leap in performance, which could be due to the fact that our final frame sampling upper limit of 300 is still not high enough.

Figure 11: The Evaluation results divided by whether input audio transcript into Qwen2.5-VL-7B. The audio transcript is extracted using Whisper-Large-V3.

C.3 Impact of Audio Information

We also conduct experiments using the Qwen2.5-VL-7B model to investigate the impact of adding audio transcripts in VideoVista-CulturalLingo, with the experimental results are shown in the Figure 11. The input audio transcript is the unrefined version extracted from Whisper-Large-V3. It can be observed that incorporating additional informa-
tion from the audio modality, the model’s performance improves in the tasks of Event, Culture, and Science. In the Science task, the improvement in model performance is most significant. This is likely because the audio in the science videos we selected is generally clear and explicit, covering the experimental and course-related information. However, in the Event and Culture tasks, the inclusion of audio transcripts only resulted in a small improvement. We encourage LMMs to process both audio and video frames simultaneously, and therefore, we did not include the audio information in our model evaluation.

C.4 Impact of Temporal Relation

We conduct experiments to investigate the temporal relationship between the event of interest posed in the question and the corresponding video, with the experimental results are shown in the Figure 12. In the Event Localization task, we selected 75 video pairs, where each pair contained videos of similar durations, and the events in the questions occurred in the early (front half) and late (back half) parts of the video. The evaluation results of five major models on early vs. late questions are summarized in the table below. As shown, except for the powerful Gemini-2.0-Flash model, the accuracy for early questions is significantly higher than for late questions across the remaining Video-LMMs. This suggests that most existing Video-LMMs have a stronger understanding of events occurring early in the video but tend to struggle with those that happen later. We guess that the phenomenon is caused by the bias in the model’s training data or the model’s ability to handle long-contexts.

Figure 12: The Evaluation results of the Temporal Relationship Between the Event of Interest and the Corresponding Video

![Bar chart showing the evaluation results of the Temporal Relationship Between the Event of Interest and the Corresponding Video](page_68_68_627_312.png)

Figure 13: Prompt for Video Processing.

You are Qwen, created by Alibaba Cloud.
# Input Information
The input will consist of a noisy audio transcript extracted by the whisper model. The transcript may be in English or Chinese and may contain errors, inconsistencies, or homophonic ambiguities.
# Task Instruction
Your goal is to polish the input transcript to enhance its fluency and coherence without altering the original semantics.
The final output should be consistent with the original language of the input audio transcript.
When the input is in Chinese, replace homophones with different meanings based on the context. For example, in a video discussing Tang Dynasty culture, replace “chuan” with “zhuang” when the context is about weaving historical knowledge and contextual information.
# Ensure the following conditions are met:
1. Enhance the fluency and readability of the transcript while preserving the original content.
2. Maintain the integrity and context of the original transcript throughout the polishing process.
3. Ensure the final polished transcript is free of grammatical errors and is logically coherent.
4. When the input is in Chinese, replace homophones with their correct counterparts based on contextual understanding. Do not alter names or terms that are already correct.
# Output Format
The output should be structured in JSON format as follows:
{"polished transcript": "Polished transcript text"}

Figure 14: Prompt for Audio Refine.

You are Qwen, created by Alibaba Cloud.
# Input Information
The input will consist of a noisy audio transcript extracted by the whisper model. The transcript may be in English or Chinese and may contain errors, inconsistencies, or homophonic ambiguities.
# Task Instruction
Your goal is to divide the input paragraph into smaller, logically coherent paragraphs. The paragraphs should retain the exact same content as the input, with only the addition of paragraph breaks.
Do not alter, omit, or add any words or punctuation. Maintain the original context and collocation of words.
# Ensure the following conditions are met:
1. No new sentences or ideas are added during the division.
2. Splits occur at meaningful transitions or logical separations within the text.
3. The original integrity and context of the text are preserved.
4. The resulting paragraphs should be of roughly equal length and uniformly, and no single paragraph should be excessively long or excessively short.
5. If the text contains multiple languages, splits should occur between the main language and other languages. For example, if the text is primarily in Chinese with some Russian, do not split within the Russian sections.
# Output Format
The output should be structured in JSON format as follows:
{"result": ["Paragraph 1", "Paragraph 2", ...]}

C.5 Detailed Experiment Results

In Table 5, we provide a detailed presentation of the performance of all evaluated models across 14 sub-tasks. In Tables 6 and 7, we present the detailed evaluation results used to plot Figures 4b and 4c. These evaluation results effectively demonstrate the models’ performance across different languages and video durations.

D Detailed Annotations Pipeline

D.1 Prompt for Video Preprocessing

We introduce the prompt to determine whether the audio of video is noisy above Figure 13 and the prompt to split the video based on audio in below of Figure 13. Both two prompt are input to Qwen2.5-72B language model during the video preprocessing stage.

In the Figure 14, we present the prompt used
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Event</th>
    <th colspan="3">Object</th>
    <th colspan="3">Culture</th>
    <th colspan="3">Science</th>
  </tr>
  <tr>
    <th>ED</th><th>EP</th><th>ES</th><th>EL</th>
    <th>OTL</th><th>OTS</th><th>OSL</th>
    <th>CC</th><th>AC</th><th>EC</th>
    <th>SS</th><th>COM</th><th>AP</th><th>SP</th>
  </tr>
  <tr>
    <th colspan="15">Open-source Video LMMs</th>
  </tr>
  <tr>
    <td>ShareGPT4-Video</td><td>29.2</td><td>20.2</td><td>17.7</td><td>23.7</td><td>10.4</td><td>27.2</td><td>30.8</td><td>19.0</td><td>34.7</td><td>42.3</td><td>32.4</td><td>48.0</td><td>32.9</td><td>30.1</td>
  </tr>
  <tr>
    <td>VideoChat2-Mistral</td><td>38.5</td><td>28.7</td><td>31.1</td><td>19.3</td><td>25.1</td><td>26.2</td><td>27.6</td><td>25.1</td><td>40.2</td><td>40.3</td><td>36.4</td><td>44.0</td><td>23.8</td><td>31.2</td>
  </tr>
  <tr>
    <td>VideoLLaMA</td><td>51.3</td><td>46.5</td><td>31.1</td><td>41.6</td><td>32.2</td><td>24.3</td><td>42.3</td><td>27.7</td><td>35.2</td><td>41.8</td><td>42.6</td><td>38.7</td><td>40.9</td><td>38.7</td>
  </tr>
  <tr>
    <td>VideoLLaMA2</td><td>36.3</td><td>28.7</td><td>41.6</td><td>29.6</td><td>17.9</td><td>15.5</td><td>36.4</td><td>25.1</td><td>38.7</td><td>42.3</td><td>36.4</td><td>42.7</td><td>35.4</td><td>34.4</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision</td><td>47.8</td><td>34.9</td><td>44.0</td><td>44.5</td><td>30.7</td><td>35.0</td><td>39.2</td><td>36.4</td><td>41.7</td><td>38.8</td><td>55.1</td><td>44.0</td><td>57.9</td><td>48.4</td>
  </tr>
  <tr>
    <td>MiniCPM-V 2.6</td><td>74.3</td><td>38.0</td><td>41.1</td><td>30.8</td><td>18.8</td><td>35.0</td><td>30.1</td><td>44.6</td><td>48.7</td><td>55.7</td><td>70.6</td><td>53.3</td><td>60.4</td><td>52.7</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3</td><td>66.4</td><td>56.6</td><td>52.2</td><td>48.2</td><td>35.3</td><td>61.2</td><td>41.7</td><td>37.7</td><td>45.7</td><td>52.7</td><td>62.1</td><td>58.7</td><td>60.4</td><td>54.8</td>
  </tr>
  <tr>
    <td>Oryx-1.5</td><td>54.4</td><td>40.3</td><td>45.9</td><td>37.7</td><td>33.1</td><td>24.3</td><td>33.2</td><td>35.5</td><td>39.2</td><td>38.3</td><td>58.5</td><td>46.7</td><td>57.9</td><td>51.6</td>
  </tr>
  <tr>
    <td>LLaVA-Video</td><td>75.7</td><td>57.4</td><td>48.3</td><td>53.1</td><td>33.7</td><td>67.0</td><td>39.2</td><td>41.6</td><td>51.3</td><td>54.7</td><td>63.6</td><td>53.3</td><td>61.0</td><td>52.7</td>
  </tr>
  <tr>
    <td>Qwen2-VL</td><td>72.6</td><td>51.2</td><td>56.9</td><td>33.3</td><td>30.0</td><td>47.6</td><td>36.0</td><td>48.5</td><td>54.8</td><td>62.2</td><td>72.1</td><td>60.0</td><td>66.5</td><td>65.6</td>
  </tr>
  <tr>
    <td>InternVL2.5</td><td>81.4</td><td>57.4</td><td>59.3</td><td>41.1</td><td>35.9</td><td>47.8</td><td>30.4</td><td>55.4</td><td>47.7</td><td>65.2</td><td>69.8</td><td>56.0</td><td>65.2</td><td>62.4</td>
  </tr>
  <tr>
    <td>MiniCPM-o 2.6</td><td>83.6</td><td>55.0</td><td>53.1</td><td>35.2</td><td>20.1</td><td>52.4</td><td>35.7</td><td>48.9</td><td>56.3</td><td>63.7</td><td>72.1</td><td>61.3</td><td>69.5</td><td>52.7</td>
  </tr>
  <tr>
    <td>TPO</td><td>75.2</td><td>56.6</td><td>49.8</td><td>48.2</td><td>31.2</td><td>67.0</td><td>38.8</td><td>43.7</td><td>50.8</td><td>55.2</td><td>63.2</td><td>50.7</td><td>62.8</td><td>55.9</td>
  </tr>
  <tr>
    <td>InternVideo2.5</td><td>80.5</td><td>52.7</td><td>60.3</td><td>33.0</td><td>37.1</td><td>61.2</td><td>31.8</td><td>53.7</td><td>56.3</td><td>65.2</td><td>72.1</td><td>61.3</td><td>64.0</td><td>54.8</td>
  </tr>
  <tr>
    <td>VideoLLaMA3</td><td>77.9</td><td>57.4</td><td>61.7</td><td>45.2</td><td>72.1</td><td>64.1</td><td>56.6</td><td>45.5</td><td>55.8</td><td>59.2</td><td>70.2</td><td>54.7</td><td>64.0</td><td>55.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td><td>75.2</td><td>51.2</td><td>72.7</td><td>40.1</td><td>39.3</td><td>56.3</td><td>31.8</td><td>51.9</td><td>50.8</td><td>63.2</td><td>80.5</td><td>65.3</td><td>72.6</td><td>60.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-72B</td><td>79.2</td><td>60.5</td><td>78.9</td><td>42.1</td><td>31.5</td><td>67.0</td><td>49.7</td><td>65.8</td><td>67.8</td><td>80.6</td><td>86.4</td><td>85.3</td><td>79.3</td><td>79.6</td>
  </tr>
  <tr>
    <th colspan="15">Open-source Image LMMs</th>
  </tr>
  <tr>
    <td>VILA1.5-13B[1f]</td><td>33.3</td><td>33.3</td><td>29.2</td><td>33.9</td><td>26.8</td><td>26.2</td><td>34.6</td><td>31.6</td><td>30.7</td><td>39.8</td><td>36.8</td><td>46.7</td><td>39.6</td><td>39.8</td>
  </tr>
  <tr>
    <td>VILA1.5-13B[8f]</td><td>36.9</td><td>38.2</td><td>31.3</td><td>38.2</td><td>23.1</td><td>35.9</td><td>45.1</td><td>23.4</td><td>42.2</td><td>51.2</td><td>43.4</td><td>41.3</td><td>40.9</td><td>39.8</td>
  </tr>
  <tr>
    <td>Molmo-7B-D[1f]</td><td>38.3</td><td>44.5</td><td>25.3</td><td>39.8</td><td>26.6</td><td>34.0</td><td>19.6</td><td>39.0</td><td>40.7</td><td>39.8</td><td>46.3</td><td>41.3</td><td>50.0</td><td>45.2</td>
  </tr>
  <tr>
    <td>Molmo-7B-D[8f]</td><td>40.3</td><td>44.3</td><td>30.1</td><td>41.8</td><td>29.6</td><td>45.6</td><td>25.5</td><td>37.7</td><td>44.2</td><td>44.3</td><td>50.0</td><td>42.7</td><td>49.4</td><td>44.1</td>
  </tr>
  <tr>
    <td>DeepSeek2-VL[1f]</td><td>40.9</td><td>44.3</td><td>32.2</td><td>39.3</td><td>32.4</td><td>33.0</td><td>31.5</td><td>37.7</td><td>38.2</td><td>42.3</td><td>52.2</td><td>44.0</td><td>49.4</td><td>52.7</td>
  </tr>
  <tr>
    <td>DeepSeek2-VL[8f]</td><td>42.6</td><td>47.0</td><td>27.2</td><td>44.4</td><td>25.0</td><td>33.0</td><td>29.4</td><td>37.2</td><td>40.7</td><td>56.2</td><td>62.9</td><td>50.7</td><td>53.0</td><td>54.8</td>
  </tr>
  <tr>
    <th colspan="15">Proprietary LMMs</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>86.3</td><td>47.3</td><td>70.3</td><td>28.6</td><td>29.4</td><td>61.2</td><td>46.5</td><td>57.1</td><td>71.9</td><td>76.6</td><td>81.6</td><td>77.3</td><td>80.5</td><td>65.6</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Flash</td><td>92.5</td><td>42.6</td><td>63.6</td><td>69.4</td><td>87.3</td><td>69.9</td><td>23.7</td><td>49.4</td><td>61.3</td><td>67.7</td><td>87.9</td><td>87.7</td><td>82.9</td><td>77.4</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash-Lite</td><td>87.2</td><td>44.1</td><td>68.4</td><td>63.8</td><td>87.5</td><td>63.1</td><td>44.8</td><td>58.4</td><td>61.3</td><td>70.1</td><td>83.1</td><td>81.3</td><td>80.5</td><td>82.8</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td><td>92.9</td><td>51.9</td><td>73.7</td><td>70.7</td><td>87.2</td><td>74.8</td><td>59.1</td><td>62.3</td><td>64.8</td><td>77.6</td><td>88.2</td><td>87.8</td><td>81.7</td><td>90.7</td>
  </tr>
</table>

Table 5: Detailed Evaluation results on VideoVista-CulturalLingo benchmark. Abbreviations used in the table: Event Description (ED), Event Prediction (EP), Event Sequence (ES), Event Localization (EL), Object Temporal Localization (OTL), Object Temporal Sequence (OTS), Object Spatial Localization (OSL), Chinese Culture (CC), American Culture (AC), European Culture (EC), Summarization & Synthesis (SS), Comparison & Contrast (COM), Application & Procedure (AP), Scientific Principle (SP).

<table>
  <tr>
    <th>Model</th>
    <th>Chinese</th>
    <th>English</th>
  </tr>
  <tr>
    <td>MiniCPM-o 2.6</td><td>58.77</td><td>63.49</td>
  </tr>
  <tr>
    <td>InternVideo2.5</td><td>60.04</td><td>63.49</td>
  </tr>
  <tr>
    <td>VideoLLaMA3</td><td>52.26</td><td>63.78</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-72B</td><td>75.59</td><td>78.30</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>68.35</td><td>76.83</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td><td>76.49</td><td>78.30</td>
  </tr>
</table>

Table 6: Model Performance by Video Language.

<table>
  <tr>
    <th>Model</th>
    <th>Short</th>
    <th>Medium</th>
    <th>Long</th>
  </tr>
  <tr>
    <td>MiniCPM-o 2.6</td><td>54.46</td><td>52.91</td><td>44.30</td>
  </tr>
  <tr>
    <td>InternVideo2.5</td><td>53.12</td><td>52.69</td><td>48.10</td>
  </tr>
  <tr>
    <td>VideoLLaMA3</td><td>61.16</td><td>56.05</td><td>50.63</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-72B</td><td>62.72</td><td>59.64</td><td>59.49</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>54.91</td><td>52.69</td><td>49.37</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td><td>75.89</td><td>74.22</td><td>62.03</td>
  </tr>
</table>

Table 7: Model Performance by Video Duration. The Duration: <2 minutes (Short), 2-10 minutes (Medium), >10 minutes (Long).

to refine the audio transcripts recognized by WhisperX, primarily aimed at eliminating homophones in Chinese, reducing ambiguity, and enhancing fluency. This process is also carried out using the Qwen2.5-72B language model.

D.2 Prompt for QA Annotation

In Figure 15, we present the system prompt used in our automatic QA annotation process for labeling video events. This system prompt is input into the Qwen2-VL-72B model, along with the corresponding video frames, audio information, and prior events, to annotate the events.

In Figure 16, we present the specific prompt used to generate Event Description questions in the automatic QA annotation process. During the
# Input Information:
The input consists of:
- A video clip (a segment from a complete video).
- The corresponding audio transcript.
- All events from previous video clips in the sequence to provide comprehensive context.
- The audio transcript of the current video clip.

Your objective is to analyze both the video and the audio transcript to identify and summarize the main event depicted in the video.
The summary should be written in natural language and try to capture the main event.
Ensure the following conditions are met:
1. "Summary Accuracy": The summary accurately reflect the event depicted in the video and the information provided in the audio transcript without omitting or adding any information.
2. "Integration of Audio and Visual Data": Effectively integrates details from both the video and the audio transcript to create a comprehensive summary. Ensure that key points from the audio are included, especially if they provide specific information not easily discernible from the video.
3. "Primary Event Focus": If multiple significant events occur, prioritize summarizing the main event unless otherwise specified.
4. "Primary Event Focus": If multiple significant events occur, prioritize summarizing the main event unless otherwise specified.
5. "Event Sequence": The summary should be written in an objective and neutral tone, avoiding personal opinions or subjective interpretations.
6. "Conflict Resolution": In cases where there is conflicting information between the video and the transcript, prioritize information depicted in the video. However, ensure the summary is still accurate and relevant to the main event.
7. "Objective Tone": The summary should be written in an objective and neutral tone, avoiding personal opinions or subjective interpretations.
8. "Handle Uncertainty": If certain aspects of the event are unclear or missing from the video or transcript, acknowledge the uncertainty without making assumptions.
9. "Consistency": The summary should be consistent with the previously provided events when previous events is provided. This includes new actions, changes in setting, introduction of new participants, or any other alterations in the event sequence.
10. "Visual Content": For videos that are narration-based with minimal visual changes, focus on summarizing the events from the AIR audio transcript.
11. "Narration-Based Videos": For videos that are narration-based with minimal visual changes, use the AIR audio transcript as supplementary information, prioritizing the visual content in your summary.
   *A sample*: "Recognized event"
The final output should be a JSON object with the following structure:
{"event": "recognized event"}

Figure 15: Prompt for Event Annotation.

# Input Information:
The input consists of a sequence of video clips divided based on semantic content. For the i-th clip, the following information is provided:
- "Segment": [start_time, end_time] (in seconds)
- "Audio Transcript": a_s
- "Video Transcript": v_s
- "Event List": e_s (list of events in the i-th clip)

Your objective is to analyze the provided video clip annotations and generate three questions focused on specific details of events within the video. Each question must include four options: one correct answer and three incorrect options.
Ensure the following conditions are met:
1. "Question Selection":
   - Each question should primarily start with "What" or "Where" and inquire about specific details of an event in the video.
   - "Positioning": Use the phrase "early part of the video", "middle section of the video", or "towards the end of the video" to specify when the event takes place. Additionally, include the phrase "*in the video*" to provide clear context.
   - The incorrect options should reference plausible but incorrect details related to the event to ensure plausibility.
2. "Event Selection":
   - The event selected should be clear and specific events within the video.
   - Avoid selecting very short or brief events for questioning to ensure that the questions are meaningful and relevant.
3. "Question Generation":
   - Each question must have four options: one correct answer and three incorrect answers.
   - Length: The correct answer and each distractor option should be approximately equal in length.
   - "Avoid Ambiguous or Overlapping Options":
     - Each option should be distinct and mutually exclusive; no two options should be correct or partially correct.
     - The correct options (distractors) should be "plausible" and "relevant" to the event but "distinct" from the correct answer.
     - "Plausible": The distractors should be plausible but not correct. They should address a different plausible but incorrect aspect related to the event to prevent overlap.
     - "Relevant": Maintain a "consistent level of detail and complexity" across all options to prevent giving away the correct answer.
     - "Diversity": Distractors should vary in nature (e.g., actions, reasons, consequences) to cover a broader range of incorrect possibilities without overlapping.
4. "Number of Questions":
   - Generate exactly three questions as specified.
   - Output: JSON array of questions
The final output should be structured as follows:
{"questions": [{"question": "Question", "options": ["optionA", "optionB", "optionC", "optionD"], "answer": "correct option"}, ...]}

Figure 16: Prompt for Event Description Questions, Options and Answer Generation.

generation of Event questions, only the aggregated event sequence is input, without any additional information. The model used in this process is the DeepSeek-V3 language model.

In Figure 17, we present the specific prompt used to generate Chinese Culture questions in the automatic QA annotation process. Unlike the Event Description task, in addition to inputting the aggregated event sequence, we also provide pre-retrieved cultural background information from Wikipedia using embeddings model, requiring the model to generate questions that necessitate both video content and cultural background knowledge to answer. The model used in this process is the DeepSeek-V3 language model.

In Figure 18, we present the specific prompt used to generate Scientific Principle questions in the automatic QA annotation process. In contrast to the question generation above, where the options are more flexible, we strictly impose requirements on the model when generating options at this stage. This approach increases the complexity of the questions and prevents the possibility of answering the questions without reference to the video content. The model used in this process is the DeepSeek-R1 language model.

D.3 Webpage for Human Scoring

We built an annotation interface using Gradio, as shown in the Figure 19. Each annotator only needs to enter their name in the top left corner, watch the video, review question, options, and check whether the answers align. Then, they can select the appropriate score in the bottom right corner. For complex cultural questions, we provide the corresponding Wikipedia entry name within the Entry, enabling annotators to efficiently look up answers to questions they may not be familiar with. This benchmark
You are an AI assistant tasked with generating detail-oriented questions based on segmented video content.

# Input Information
1. The input consists of a sequence of video clips divided based on semantic content. For the "i-th" clip, the following information is provided:
   * "event": "e_i"
   * "duration": "d_i"
   * "begin_time": "t_i" (start time of the i-th clip in the original video)
   * "end_time": "t_{i+1}" (end time of the i-th clip in the original video)
2. The annotation information for all "n" clips is provided in the following format:
   [
     {"event": "e_1", "audio": "a_1", "begin": b_1, "end": e_1},
     ...
     {"event": "e_n", "audio": "a_n", "begin": b_n, "end": e_n}
   ]
3. Additionally, relevant external knowledge from Wikipedia related to the video content will be provided. This knowledge will begin with a wikipedia entry.
4. Please ensure that the questions are directly related to the video content; please assess based on the video and audio content.

# Task Instruction
You are asked to analyze the provided video clip annotations and generate two questions focused on specific details of events within the video. Each question must include four options: one correct answer and three incorrect answers.
Ensure that the following conditions are met:
1. *Language Requirement*:
   - All questions and answers must be in (region).
2. *Question Focus*:
   - Ensure that each question requires understanding both the video content and the external knowledge provided.
   - Questions should recognize cultural phenomena depicted in the video and utilize external knowledge to extend the inquiry.
   - Questions should indirectly refer to cultural phenomena depicted in the video without explicitly naming them.
   - Cultural phenomena should be referenced in the video and not be inferred or indirectly.
3. *Options*:
   - Each question must have four options: one correct answer and three incorrect answers.
   - Length of the options should be relatively consistent to avoid giving away the correct answer based on length.
   - Answers should be distinct and not overlap.
   - Ensure that all four options are "mutually exclusive"; no two options should be correct or partially correct.
   - The distractors should be plausible but not align with the context of the (region) culture.
   - Avoid extreme or outlandish options that do not align with the context of the (region) culture.
   - Randomly select distractors to make sure the correct answer is not obvious.
4. *Number of Questions*:
   - Ensure that the number of questions as specified.
5. *Output Format*:
   - The final output should be structured as follows:
   [
     {
       "question": "question",
       "options": ["option1", "option2", "option3", "option4"],
       "answer": "correct option"
     },
     ...
   ]

Figure 17: Prompt for Chinese Culture Quetions, Options and Answer Generation.

You are an AI assistant tasked with generating science reasoning questions focused on experimental phenomena or procedural principles from segmented video content in chemistry or physics.

# Input Information
1. The input consists of a sequence of video clips divided based on semantic content. For the "i-th" clip, the following information is provided:
   * "event": "e_i"
   * "duration": "d_i"
   * "begin_time": "t_i" (start timestamp)
   * "end_time": "t_{i+1}" (end timestamp)
2. Annotations for all "n" clips are formatted as:
   [
     {"event": "e_1", "audio": "a_1", "begin": b_1, "end": e_1},
     ...
     {"event": "e_n", "audio": "a_n", "begin": b_n, "end": e_n}
   ]
3. Task Instruction
   Analyze the video annotations and generate two questions that ask about the scientific principles behind experimental phenomena or procedural steps.
   All questions must belong to the unified category: Science Principle.
   Conditions:
   1. *Questions*:
      - Focus on explaining why an observed phenomenon occurs or why a specific experimental action is required.
      - Include contextual anchors like "in the video" or "according to the video."
      - Include contextual anchors like "in the video" or "according to the video."
   2. *Distractor Types*:
      - "Video Comprehension Error": Misapplies the principle demonstrated in the video (e.g., cites temperature changes when the video emphasizes concentration changes).
      - "Domain Knowledge Error": Misapplies the principle demonstrated in the video (e.g., omits a critical factor like surface area in a reaction rate explanation).
      - "Dual Error": Combines an incorrect scientific principle and irrelevant experimental factors (e.g., attributes a color change in magnetism instead of chemical equilibrium).
      - "Uniqueness": Ensure the correct option is unambiguous.
      - Correct Option: Must be relevant to the same technical domain (e.g., chemical kinetics for reaction rate questions).
      - Minimal Distortion: Domain Knowledge Errors should alter only one critical step/factor.
   3. *Output Format*:
      - Return exactly 2 questions from different categories
      - Prioritize question categories most relevant to the video's technical depth

# Output Format
The final output should be structured as follows:
[
   {
     "question": "Explanation of the question generation process",
     "options": ["Correct Option", "Video Comprehension Error", "Domain Knowledge Error", "Dual Error"],
     "answer": "Correct Option", "category": "category",
     "question": "question",
     "options": ["Correct Option", "Video Comprehension Error", "Domain Knowledge Error", "Dual Error"],
     "answer": "Correct Option", "category": "category"
   }
]

Figure 18: Prompt for Scientific Principle Questions, Options and Answer Generation.

includes a total of ten annotators, each with at least an undergraduate degree and proficiency in both Chinese and English.

D.4 Annotation Model

We organize our annotation models into three complementary categories, chosen for their balance of accuracy, speed, and cost:

First group: Small tool models. This includes WhisperX for audio extraction, MiniLM for embedding extraction, and Grounding-DINO, SAM2 for bounding box generation. We chose these models based on their open-source nature, accuracy, and inference speed. The tasks assigned to these models are relatively simple, often yielding high accuracy, making speed our primary evaluation criterion. For bounding box extraction, we also tested models like Florence2 and Grounding-DINO 1.5, but since the accuracy differences were minimal, we opted for the lighter, faster Grounding-DINO.

Second group: Multimodal large models for video annotation. For video content annotation, we referenced video evaluation benchmarks such as Video-MME and MVBench. Among the open-source models, Qwen2-VL-72B demonstrated the strongest performance, so we selected it for video annotation. We also tested InternVL2-76B, but found that its limited frame sequence length hindered its ability to capture full video information.

Third group: Powerful Large Language Models for question generation. In this category, we primarily used DeepSeek-V3 and DeepSeek-R1. For task categories like Event, Object, and Culture, we compared DeepSeek-V3 and GPT-4o models, judging the quality of generated questions through manual evaluation. While no significant quality
difference was found, DeepSeek-V3 proved to be more cost-effective. For science-related questions, both models performed poorly, as the generated questions could be answered without watching the video, and the answer options were often too similar or ambiguous. To address this, we applied stricter constraints to ensure the questions required watching the video, and that the options were meaningful. The long-reasoning model DeepSeek-R1 effectively applied these rules, generating questions that were more appropriate. Besides, the chain-of-thoughts generated during the process also helped human annotators make quicker judgments about the appropriateness of the questions.

Specifically, during the data annotation process, the Whisper, SAM2, Qwen series models, and InternVL series models were deployed for inference on local GPU servers. The DeepSeek-V3 and DeepSeek-R1 models is utilized the API services provided by the official 2. The specific Whisper model used in the experiment is WhisperX3, based on Whisper-large-V3. When obtaining Chinese transcripts, a special initial prompt "以下是中文普通话句子。" was set to ensure that the model could correctly add punctuation. The pipeline used for annotating objects, which involves Grounding Dino and SAM2, is derived from Grounded-SAM-24.

D.5 External Resources

The three websites to collect videos: YouTube5, Xiaohongshu(RedNote)6 and BiliBili7.

The multilingual Wikipedia used in the automatic QA annotation pipeline was downloaded from Wikimedia Downloads8, and the extraction and processing were performed using regular expression rules9. The tool used to collect videos from BiliBili is Downkyi10.

E Case Data

In Figures 20-33, we present a specific case for each proposed task type. Each case includes sampled frames from the video, along with the corresponding questions and options. The ground truth is highlighted in yellow.

2https://platform.deepseek.com/usage
3https://github.com/m-bain/whisperX
4https://github.com/IDEA-Research/Grounded-SAM-2
5http://www.youtube.com
6https://www.xiaohongshu.com
7https://www.bilibili.com
8https://dumps.wikimedia.org/backup-index-bydb.html
9https://spaces.ac.cn/archives/4176
10https://github.com/leiurayer/downkyi
Event Description. The Event Description task primarily focuses on explaining how a specific event in the video occurred, typically beginning with questions such as 'What' or 'How'.

Event Prediction. The Event Prediction task primarily involves predicting the event most likely to occur after the input video ends. In this task, the selected video is typically a segment from a full video, such as a clip spanning from 0 to 45 seconds of the full video

Event Sequence. The Event Sequence task primarily asks about the order in which multiple events occur in the input video, requiring the model to select the most accurate sequence of events from the options provided.

Event Localization. The Event Sequence task primarily focuses on determining the order in which multiple events occur in the input video, requiring the model to select the most accurate sequence of events from the available options.

Object Temporal Localization. The Object Temporal Localization task primarily requires identifying the timestamp of the first appearance of a specific object in the video. The selected object typically occupies a significant portion of the frame to ensure it is easily noticeable, avoiding objects that may be difficult for humans to detect.

Object Temporal Sequence. The Object Temporal Sequence task primarily focuses on determining the order in which multiple distinct objects appear in the video.

Object Spatial Localization. The Object Spatial Localization task primarily requires identifying the spatial bounding boxes of a specific object in the video at a particular time, typically when the object first appears. The answer is provided in a normalized format, represented as bounding boxes in the xyxy format.

Chinese Culture. The Chinese Culture task primarily focuses on the Chinese cultural background presented in the video, covering areas such as traditional culture, culinary traditions, ancient history, and more.

American Culture. The American Culture task primarily focuses on the American cultural background presented in the video, emphasizing areas such as political culture, superhero culture, pop culture, holiday traditions, and more.

European Culture. The European Culture task primarily focuses on the European cultural background presented in the video, emphasizing areas such as cultural differences between European countries, football culture, culinary traditions, classical culture, and more.

Summarization & Synthesis. The Summarization & Synthesis task primarily requires the model to summarize and synthesize the key points presented in educational or popular science videos, assessing the model’s ability to consolidate the essential concepts conveyed in the video.

Comparison & Contrast. "The Comparison & Contrast task primarily requires the model to compare the specific method described in the educational or popular science video with other similar methods, emphasizing the differences or distinctions between them. This task assesses the model’s ability to comprehend the key concepts presented in the video."

Application & Procedure. The Application & Procedure task primarily requires the model to determine the operational procedure or application method of a specific concept described in the educational or popular science video. This task assesses the model’s understanding of the key concepts presented in the video."

Scientific Principle The Scientific Principle task requires the model to comprehend the scientific principles underlying the experimental procedures or phenomena presented in the video.
Category: Event-Event Description
Question: 在视频的早期部分，女士如何将第一张纸张进行折叠？
(In the early part of the video, how does the lady fold the first sheet of paper?)
A. 横向四等分并折叠，然后竖向四等分并折叠。
(Fold horizontally into four equal parts, then fold vertically into four equal parts.)
B. 横向三等分并折叠，然后竖向四等分并折叠。
(Fold horizontally into three equal parts, then fold vertically into four equal parts.)
C. 横向四等分并折叠，然后竖向三等分并折叠。
(Fold horizontally into four equal parts, then fold vertically into three equal parts.)
D. 横向五等分并折叠，然后竖向二等分并折叠。
(Fold horizontally into five equal parts, then fold vertically into two equal parts.)

![A sequence of images showing a woman folding a sheet of paper in different ways](page_324_186_1097_246.png)

Figure 20: An Example of Event Description from VideoVista-CulturalLingo.

Category: Event-Event Prediction
Question: 在介绍完鸡爪后，女子接下来会做什么？
(After introducing the chicken feet, what will the woman do next?)
A. 她会尝试茅台制作成的果冻。
(She will try jelly made from Maotai.)
B. 她会尝试红烧肉作为晚餐。
(She will try braised pork as dinner.)
C. 她会尝试五花肉软糖。
(She will try pork belly gummies.)
D. 她会结束视频并道别。
(She will end the video and say goodbye.)

![A sequence of images showing a woman eating various snacks](page_324_1097_1097_246.png)

Figure 21: An Example of Event Prediction from VideoVista-CulturalLingo.
Category: Event-Event Sequence
Question: Which of the following event sequences is correct?
A. A news anchor reports on a historic cathedral fire in Paris, discusses a school cafeteria renovation, covers a landslide in China, and concludes with a weather update.
B. A news anchor discusses a school cafeteria renovation, reports on a historic cathedral fire in Paris, covers airline price increases, and concludes with a weather update.
C. A news anchor covers airline price increases, reports on a historic cathedral fire in Paris, discusses a school cafeteria renovation, and concludes with a weather update.
D. A news anchor discusses a school cafeteria renovation, covers airline price increases, reports on a landslide in China, and concludes with a weather update.

![An Example of Event Sequence from VideoVista-CulturalLingo.](page_324_186_1097_104.png)

Figure 22: An Example of Event Sequence from VideoVista-CulturalLingo.

Category: Event-Event Localization
Question: When does the video approximately start tasting the pizza?
A. 0:06:35    B. 0:05:35    C. 0:04:35    D. 0:07:35

![An Example of Event Localization from VideoVista-CulturalLingo.](page_324_574_1097_104.png)

Figure 23: An Example of Event Localization from VideoVista-CulturalLingo.

Category: Object-Object Temporal Localization
Question: At what time does the statue of Dr. Julius Kugy first appear in the video?
A. 0:00:16    B. 0:00:20    C. 0:00:12    D. 0:00:24

![An Example of Object Temporal Localization from VideoVista-CulturalLingo.](page_324_1062_1097_104.png)

Figure 24: An Example of Object Temporal Localization from VideoVista-CulturalLingo.
Category: Object-Object Temporal Sequence
Question: 以下哪个是视频中物体出现的正确顺序?
(In the video, which of the following is the correct order in which the objects appear?)
A. 蕃茄、生肉沫、白糖、酱油。
(Tomato, raw minced meat, white sugar, soy sauce.)
B. 蕃茄、酱油、白糖、生肉沫。
(Tomato, soy sauce, white sugar, raw minced meat.)
C. 蕃茄、白糖、酱油、生肉沫。
(Tomato, white sugar, soy sauce, raw minced meat.)
D. 生油、白糖、生肉沫、蕃茄。
(Soy sauce, white sugar, raw minced meat, tomato.)

![An Example of Object Temporal Sequence from VideoVista-CulturalLingo.](page_324_186_1097_180.png)

Figure 25: An Example of Object Temporal Sequence from VideoVista-CulturalLingo.

Category: Object-Object Spatial Localization
Question: Which of the following bounding boxes most accurately depicts the position of the silver trophy with red, white, and blue ribbons at 4 seconds? The bounding box in the answer is in 'xyxy' format and has been resized to the range [0, 1].
A. [0.14, 0.07, 0.30, 0.96]
B. [0.24, 0.14, 0.54, 0.92]
C. [0.63, 0.06, 0.79, 0.61]
D. [0.53, 0.03, 0.70, 0.95]

![An Example of Object Spatial Localization from VideoVista-CulturalLingo.](page_324_670_1097_180.png)

Figure 26: An Example of Object Spatial Localization from VideoVista-CulturalLingo.

Category: Culture-Chinese Culture
Question: 视频中提到的这道菜的主要的流行地点是?
(What are the main popular locations for the dish mentioned in the video?)
A. 山西、山东(Shanxi, Shandong)
B. 北京、河北(Beijing, Hebei)
C. 江苏、上海(Jiangsu, Shanghai)
D. 陕西、山西(Shaanxi, Shanxi)

![An Example of Chinese Culture from VideoVista-CulturalLingo.](page_324_1154_1097_180.png)

Figure 27: An Example of Chinese Culture from VideoVista-CulturalLingo.
Category: Culture-American Culture
Question: Who is the director of the movie mentioned in the video?
A. David Leitch    B. James Gunn    C. Tim Miller    D. Matthew Vaughn

![An Example of American Culture from VideoVista-CulturalLingo.](page_246_180_1057_93.png)

Figure 28: An Example of American Culture from VideoVista-CulturalLingo.

Category: Culture-European Culture
Question: In which country is the beverage mentioned in the video primarily associated with anti-social behavior?
A. England    B. Scotland    C. Ireland    D. Wales

![An Example of European Culture from VideoVista-CulturalLingo.](page_246_420_1057_93.png)

Figure 29: An Example of European Culture from VideoVista-CulturalLingo.

Category: Science-Summarization & Synthesis
Question: According to the video, what key quantum physics concepts are essential for understanding the quantum world?
A. Quantum tunneling, Heisenberg uncertainty principle, superposition, and the role of particles in multiverses
B. Superposition, quantum entanglement, photoelectric effect, and theories of wormholes and time travel
C. Wave-particle duality, quantum field theories, applications in cybersecurity, and the limits of relativity
D. Photoelectric effect, superposition, quantum tunneling, and applications in advanced computing and energy

![An Example of Summarization & Synthesis from VideoVista-CulturalLingo.](page_246_760_1057_93.png)

Figure 30: An Example of Summarization & Synthesis from VideoVista-CulturalLingo.
Category: Science-Comparison & Contrast
Question: How does the first activation function discussed in the video differ from modern alternatives like ReLU?

A. The video’s function utilizes a hyperbolic tangent structure for interpretability, while ReLU simplifies gradient flow in deep layers.
B. The video’s function compresses outputs to [0,1] for interpretability, while ReLU simplifies training via piecewise linearity.
C. The video’s function facilitates identity mapping for dynamic routing, while ReLU utilizes a slope-modifying kernel.
D. The video’s function uses piecewise linearity for speed, while ReLU compresses outputs to [0,1]

Figure 31: An Example of Comparison & Contrast from VideoVista-CulturalLingo.

Category: Science-Application & Procedure
Question: 视频中所讲述的最后一种搜索算法的正确实施步骤是以下哪一项？
(Which of the following is the correct implementation step for the last search algorithm described in the video?)
A. 维护k个候选序列 → 遍历所有可能扩展 → 选择当前时刻最优的k个 → 应用动态规划算法。
(Maintain k candidate sequences → Traverse all possible expansions → Choose the k most optimal at the current moment → Apply dynamic programming)
B. 使用贪婪搜索 → 遍历所有候选序列 → 选择概率最高的k个 → 调整序列长度权重。
(Use greedy search → Traverse all candidate sequences → Choose the k with the highest probability → Adjust the sequence length weight)
C. 维护所有可能的序列 → 遍历k个候选序列 → 选择概率最大的序列。
(Maintain all possible sequences → Traverse k candidate sequences → Choose the sequence with the highest probability)
D. 维护k个候选序列 → 遍历所有可能扩展 → 选择概率乘积最高的k个 → 平衡长短句子权重。
(Maintain k candidate sequences → Traverse all possible expansions → Choose the k with the highest product of probabilities → Balance the weight of long and short sentences)

Figure 32: An Example of Application & Procedure from VideoVista-CulturalLingo.
Category: Science-Scientific Principle
Question: 为什么视频中浸过某溶液的木条在火焰中不会燃烧？
(Why does the wooden strip soaked in a certain solution not burn in the flame in the video?)
A. 溶液中的成分与木条中的纤维发生反应，降低其燃性。
(The components in the solution react with the fibers in the wood strip, reducing its flammability)
B. 溶液分解时释放的抑制气体降低了氧气浓度，防止燃烧。
(The gases released during the decomposition of the solution lower the oxygen concentration, preventing combustion)
C. 高温下溶液成分形成隔绝氧气的保护层，阻止燃烧。
(The components of the solution form a protective layer that isolates oxygen at high temperatures, preventing combustion)
D. 溶液的高粘度使氧气难以接触木条表面，阻止燃烧。
(The high viscosity of the solution makes it difficult for oxygen to contact the surface of the wood strip, preventing combustion)

![A sequence of video frames showing a scientific principle demonstration](page_324_671_800_120.png)

Figure 33: An Example of Scientific Principle from VideoVista-CulturalLingo.

Reviewer paper summaries: This paper focuses on the development of a novel video understanding benchmark VideoVista2, and LMMs evaluation under this comprehensive benchmark. It constructs a three-stage annotation pipeline for automatic video data annotation, employing specialized models for different tasks such as Event, Culture, Object, and Science. The benchmark is remarkable for spanning multiple cultures (Chinese, American, European), languages (Chinese, English), and a wide range of video types. Through comprehensive evaluation of open-source video large-language models (LMMs), including conducting ablation experiments on video duration and domain, the paper reveals performance gaps and limitations of these models.

The paper introduces VideoVista2, a benchmark designed for evaluating multimodal AI systems across diverse cultural, linguistic, and domain contexts in video comprehension. The dataset includes 1389 videos from Chinese, North American, and European cultures, with 3134 QA tasks available in both Chinese and English. The authors employ a hybrid annotation process combining AI-generated questions with human refinement and provide comprehensive evaluation across numerous state-of-the-art models, revealing limitations in cultural understanding and temporal reasoning.

This paper presents VideoVista2, a novel benchmark for evaluating video large language models (LLMs) across multiple dimensions including cultures (Chinese, North American, European), languages (Chinese and English), and domains (videos from hundreds of human-created domains). The benchmark contains 1,389 videos, 2,052 video clips, and 3,134 QA pairs organized into 14 subtasks across 4 main categories: Event, Object, Culture, and Science. The authors propose a hybrid annotation approach combining LLMs and human effort, and evaluate 24 state-of-the-art models (both open-source and proprietary) on this benchmark. Key findings include performance gaps in Chinese cultural understanding, temporal reasoning, and certain scientific disciplines.
Summary of strengths: 1. **Automatic Annotation Pipeline**：The paper presents a well-designed three-stage automatic annotation pipeline. Specialized models are used for different types of tasks and different annotation steps, which makes the entire annotation process more professional and refined, and improves the accuracy and pertinence of the annotation. Automating the annotation, not only saves a vast amount of time and human effort but also ensures a certain level of consistency in the annotation results, which is crucial for large-scale video data synthesis.

2. **Comprehensive Video Understanding Benchmark**: The paper's introduction of a benchmark covering multiple cultures (Chinese, American, and European), languages (Chinese and English), and diverse video types (from general to scientific) is a major strength. This benchmark enables testing models on various cultural content, highlighting the importance of a model's ability to handle different cultural ways of presenting video information. 

3. **Thorough Evaluation of Multi-Modal Models**: The evaluation of image/video LMMs in this paper is comprehensive. Detailed analysis exhibits model performance gaps in diverse tasks (e.g. Event localization). Also, ablation experiments related to video duration and video domain. These comprehensive experiments deepen understanding of models' capabilities and limitations, which is crucial for LMMs improvement.

The paper introduces a well-constructed, culturally and linguistically diverse benchmark. Comprehensive evaluation involving 24 state-of-the-art multimodal models provides meaningful insights into model performance across different tasks and domains. Extensive and thorough ablation analyses clearly show the impact of culture, language, and video duration on model performance.

1. The paper addresses a clear gap in existing benchmarks by incorporating multiple cultures, languages, and domains, moving beyond the Western-centric nature of most current datasets.
2. The diverse task categories provide a comprehensive evaluation of video understanding capabilities, from coarse-grained event understanding to fine-grained object recognition.
3. The benchmark creation process is well-documented, with a thoughtfully designed pipeline combining automated and human annotation.
Summary of weaknesses: 1. **Unclear data collection and annotation in some aspects**：

1.1. The video collection process remains unclear, and there is no explicit indication of how the video domains are determined. It appears that there exists an overlap among them. For instance, the “travel & events” category on YouTube and the “travel vlogs” on Xiaohongshu seem to have overlapping content.

1.2. The scope of the cultural domain and video collection is rather narrow, being limited to China, the United States, and Europe only, without encompassing niche cultures such as Mexican, African, and Middle Eastern cultures. (It is necessary to inquire why more videos related to these niche cultures were not considered for inclusion.)

1.3. The selection of models in the annotation process, such as the use of DeepSeek - R1 for scientific questions, demands a more detailed elaboration. (It is essential to explain the reasons for choosing these models. Were there any ablation experiments conducted?)

1.4. The necessity of the strict rules for the “science” task and the specific details of these rules are not explained, which gives rise to confusion. 

2. **Examples Inappropriate**:

2.1. The upper example in Figure 6 does not seem very fitting. One can only tell that it is a variation of "Journey to the West", but it is not clear why it is regarded as "Journey to the North". The connection with culture in this case does not seem to be very strong. 

3. **Lack of further comparison with other video benchmarks**:

3.1. VideoVista2 has questions related to multiple languages and multiple cultures. Will it demonstrate different trends in the performance of models compared to other benchmarks?

1. The paper leverages mostly AI-generations (audio transcription, event annotation, question generation, and verification) in combination with human annotation to ensure high-quality data and result in 60% filtered QA. Would this limit the generalization as the questions are proposed by MLLMs?

2. Though results in Figure 4c presents model performance v.s. video length, there is no investigation into the temporal relation between the event of interest in the question and the video. For example, given two videos of the same length, the model could possibly have a different performance if the relevant event in the question appears early/late in the video.

1. The paper lacks sufficient details on how the cultural aspects were balanced and validated. For example, it's unclear how potential cultural biases were addressed during annotation.
2. There is limited analysis of failure cases and specific examples showing where and why models struggle with cross-cultural content.
Comments, suggestions and typos: 1. Answer the questions listed in the Summary Of Weaknesses

2. Add additional experiments comparison with other video benchmarks (such as Video-MME, MMBench-Video). VideoVista2 has questions related to multiple languages and multiple cultures. Will it demonstrate different trends in the performance of models compared to other benchmarks?

See Weakness.

- The paper would benefit from a more detailed discussion of how VideoVista2 could guide future model development, beyond identifying current limitations.
- For future work, expanding the linguistic coverage beyond Chinese and English would enhance the benchmark's value.
- Minor formatting issues in some tables and figures could be improved for readability.
Overall assessment scores: 4

3

3