MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos
Authors: Kejian Zhu
Zhuoran Jin
Hongbang Yuan
Jiachun Li
Shangqing Tu
Pengfei Cao
Yubo Chen
Kang Liu
Jun Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models
Multimodal Reasoning
Video Benchmark
Primary area: datasets and benchmarks
TL;DR: This paper introduces a video benchmark that requires multimodal deep reasoning, where question demand in-depth analysis across long-range, multi-frame video segments.

Abstract: The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus on understanding tasks, which only require models to match frames mentioned in the question (hereafter referred to as "question frame'') and perceive a few adjacent frames. To address this gap, we propose **MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos**. The benchmark is characterized by the following features. **(1) Long-range, multi-frame reasoning**: Models are required to infer and analyze evidence frames that may be far from the question frame. **(2) Beyond perception**: Questions cannot be answered through direct perception alone but require reasoning over hidden information. **(3) Reliability**: All tasks are manually annotated, referencing extensive real-world user understanding to align with common perceptions. **(4) Confusability**: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the best-performing model, Gemini-2.5-pro, achieves only 64.3% accuracy. Additionally, current reasoning enhancement strategies (Chain-of-Thought and scaling test-time compute) bring limited gains. Error analysis indicates that the CoT demanded for multi-modal reasoning differs from it in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.

Published as a conference paper at ICLR 2026
MMR V: What's Left Unsaid? A BENCHMARK FOR MULTIMODAL DEEP REASONING IN VIDEOS
Kejian Zhu Zhuoran Jin Hongbang Yuan Jiachun Li Shangqing Tur Pengfei Cao Yubo Chen Kang Liu Jun Zhao
'The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China SSchool of Artificial Intelligence, University of Chinese Academy of Sciences TTiinguua University zhukeji jian2025@ia...mmmmmmaa ac en (zhuoran. jin, hongbang, yuan @nlpr ia. ac. cn (pengfei. cao, yubo. chen, kliu, jzhao) cnlpr.ia.ae
ABSTRACT
The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus un- derstanding tasks, which only require models to match frames mentioned in the question (hereafter referred to "question frame") and perceive few adjacent frames. To address this gap, we propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos. The benchmark is characterized by the following fea- tures. (1) Long-range, multi-frame reasoning: Models are required to infer and analyze evidence frames that may be far from the question frame. (2) Beyond per- ception: Questions cannot be answered through direct perception alone but require reasoning over hidden information. (3) Reliability: All tasks are manually anno- tated, referencing extensive real-world user understanding to align with common perceptions. (4) Confusability: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the persforrrrn model, Gemini-2.5 Additionally, current reasoning enhancement strategies scaling test-time compute) bring limited gains. Error analysis indicates that the CoT demanded for multi-modal reasoning differs from in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.
achieves only 64.3% accuracy.
Thhhuuhh and
Benchmark Project
http://nggiiggecccoootcsss//aaar co//antseet//oookr//a//MMMe VBench https//mmmmr-..iithub..
INTRODUCTION
Recent models like OpenAl's ol (Jaech et al., 2024) and Deepscek-RI (Guo et al., 2025) have significantly improved text reasoning ability through reinforcement learning. This has sparked growing interest in multimodal reasoning (Wang et al., 2025). Models like o3 (OpenAI. 2025b) and GPT-5 (OpenAI, 2025a) have achieved impressive results on image reasoning tasks through tool use, integrating visual information into the reasoning process to enable deep reflection and evidence mining. However, most of these studies focus on images, with limited exploration of more challenging video reasoning tasks. Video involves sequential and richer multimodal information, requiring models to reason and mine evidence over real-world applications such as embodied intelligence and intelligent security monitoring (Hou et al., 2008; Yang et al., 2024b), it naturally raises important question: can current MLLMs perform deep multimodal reasoning and "think with videos" " like o3 on image tasks?
multi-frane. Since this capability is essential for
CCoreeponning authors.

---

Published as a conference paper at ICLR 2026
Explicit Reasoning
Question From
Euidenee Froee
Endesse Foumet
RVTDENCE
EVTDENCO
Qi How did the men the beginning of the video manage To hold the fiame from the lighter hi hanc?
Aighter
(4) He used holegrophic projector sinulite fiome.
acttegushing apray appliad
K hiddes lighter hand, a sleght t hend mrc
MMR-V
damaged
auddenly get fixed
Implicit Reasoning
RVIDE SIDEESCE
(i] Bezman the bey hold the gir't hand, grined good hes, mi! result, the unbrelle was restored
The buy's room aumtber 13, symbollies bad lacl. syndiolios lnch.
people
daring part, making
unbrell getting
aaredyrance changes strength and direct
contributed to the unbrels being repoined.
Evidence Frana
Dvidencs Prons
Endence Frome
Question Frone
Understanding
EVIDENCE
Previous Task
Qi Why did the and girl bump into each other when they first met?
The fatling from tie consine bim speed up.
Video
[e]] boy
forward
and bumping the
ipportunity wandial to know the get
Figure 1: Examples showing the MMR-V tasks and the difference from previous video benchmarks.
However, existing video benchmarks primarily focus on perception and understanding tasks (Zhou al., 2024; Fu et al., 2024). These tasks often only require locating frames mentioned in the question and understanding adjacent frames. For example, at the bottom of Figure 1, noticing the boy being hit by the metal frame is enough to understand why he ran into the girl. Such tasks fall short in evaluating multimodal reasoning abilities. We summarize their limitations as follows: (1) Limited frame context: Even for long videos, existing tasks often rely just few adjacent frames, failing exploit the long range sequential structure of the video. (2) Lack of reasoning: Many questions can be answered through direct perception. (3) Unrealistic task: Simple perception and adjacent-franm understanding tasks do not meet the real-world demands for Al system strong capabilities.
To address these shortcomings, Reasoning in Videos. We present two examples to illustrate the key differences with previous video understanding benchmarks in Figure/ MMR-V V offers the following features: (1) Long-range, multi-frame reasoning: tasks involve multimodal reasoning over non-adjacent video frames to locate and analyze multiple evidences; (2) Beyond perception: questions cannot answered by direct perception of question frame directly, requiring reasoning and the extraction of implications; (3) Reliability: All tasks are annotated manually, and potential subjective bias is reduced by cross- referencing the most popular video comments. (4) Confusability: We employ carefully designed annotation strategies to craft model-aligned distractor options, thereby ensuring confusability.
propose MMR-V Bench: A Benchmark for Multi-modal Deep
Inspired by cognitive and psychological theories (Evans, 1984; Sun, 2006; Polanyi. 2012). such as Kahneman's Dual Process Theory (Kahneman, 2011), we categorize the tasks MMR-V into implicit reasoning and explicit reasoning. The key distinction lies in whether the question requires
2

---

Published as a conference paper at ICLR 2026
reasoning beyond surface-level information infer underlying implications. Explicit reasoning is defined as questions that can be solved using perceivable information from the video. For example, the task shown in Figure 1 requires noticing the two lighters hidden in the hand. Implicit reasoning requires extracting and interpreting the underlying subtext behind visual information. For example, in the implicit reasoning girl's room number 7 symbolizes good luck. This is more of an assessment of EQ, testing whether the model can use its deep understanding of the world knowledge make implicit and subconscious reasoning paths like humans.
shown in Figure it requires inferring the underlying implication that the
MMR-V comprises 317 videos and 1257 tasks. The videos span six major categories, with lengths ranging from 7 to 3771 seconds, with an average of 277 seconds. Tasks are further divided into 10 categories and subcategories. Each task is in multiple-choice format with approximately ten options on average. Tasks typically require reasoning over average 12 video frames, covering about 60% of video duration, All questions and correct answers are human-annotated and reviewed. Distractors are generated using carefully designed annotation strategy (Details in Section 3.2).
We evaluated 11 proprietary models and 10 open-source models on MMR-V. The results reveal that even the best-performing model, Gemini-2.5 achieved only 64.3% accuracy, highlighting the significant challenge MMR-V V poses to current multimodal large language models. Our key findings follows. (1) Multimodal reasoning challenge: Our findings in Section 4.2 show that reasoning enhancement strategies (e.g., CoT and scaling test-time compute) yield limited improvements, indicating that MMR-V presents a greater challenge to current multimodal reasoning models. Further error analysis in Section 4.5 shows that the CoT demanded in multimodal reasoning differs from those in textual reasoning. Current models tend to rely on textual reasoning based on visual information from the question frame and few adjacent frames, lacking the multimodal reasoning needed to locate and analyze evidence from long-range frames. This limitation hinders the overall reasoning performance. (2) More modality will benefit: We found that for models that support all modalities, adding additional audio modalities will improve the performance (Accuracy improved by 1.4%, 1.0%, and 1. ..0 for Gemini 2.0-Flash, Gemini L0-Flash-Thiinknn
and Phi-4-Multimoodal-nssttrn respectively). (3) Human-model gap: In human experiments, we found that although models exhibit human-level performance on text reasoning tasks, there is still a significant gap between model and human on multimodal, especially video, reasoning tasks. We hope MMR-V will inspire further research into enhancing multimodal reasoning capabilities in AI systems.
2
TASK OVERVIEW
The tasks in MMR-V require deeper multimodal reasoning. Unlike previous tasks such as math and puzzle problems (Lu et al., 2023; Wang et al., 2024a; Zhang et al., 2024), we argue that the scope of multimodal reasoning should be more broadly defined. Previous work focuses more on text-oriented reasoning based on perceived visual information. In contrast, our task requires integrating the various forms of visual evidences, such as artistic style, lighting, and depth, into the reasoning process. Even more challenging, it involves reasoning over long-range, multi-frame visual evidence. Videos have a temporal dimension, which puts a greater challenge on the ability to find clues in different frames through multimodal reasoning.
2.1 DEFINITION FOR IMPLICIT AND EXPLICIT REASONING.
We categorize reasoning tasks in MMR-V into Implicit Reasoning and Explicit Reasoning, inspired by Kahneman's 's Dual Process Theory (Kahneman, 2011) and other cognitive theories (Evans, 1984; Sun, 2006; Polanyi, 2012). The most obvious difference is whether or not one needs to understand the subtext beneath the surface information. Secondly, implicit reasoning for human is often achieved by experience based world knowledge, thus consuming little attention resources. Tasks are further divided into 10 categories and 33 subcategories. Figure 2 shows six categories (top row: implicit; bottom: explicit). Further explanations and examples can be found in Appendix E.
Implicit Reasoning focuses on incorporating hidden meanings behind visual information into reasoning. In these tasks, surface-level visual cues often conceal deeper layers of meaning, such as metaphor. Besides, for human, "(implicit) operates automatically and quickly, with little or no effort and sense of voluntary control." Dual Process Theory.
3

---

Published as a conference paper at ICLR 2026
Implicit Reasoning
I. Metaphor Understanding
II. Theme Understanding
III. Emotion Recognition
Question: does brown
Qucetion:
does
Question: excntuallyy lozt
symboltte?
video impty?
his job, happy end?
Opetinus:
Options.
Optisne:
family's
(o] implin that sceryone should invest all theer
(A] was relleved beow stresffu
werk.
searching
stock mar ket get rish quickiy
(8)
losing job significnt
[o] People facs great pressure buying homes, as housing
downtum his
smmbolass the futher family, The difffionts.
prices increase while they saving money
[C] was happir because he pumsre passion
(6) shows at the socill lscue is that renting house
(0) felt nestral utree antiopatted the yrb low. was happy and felt unprecedented semse of rallet.
been followinn family for generations.
always wastc money, and buying house is the only way
to have a stable lit,
(P) Color of the house can determine ne person sochi status.
(0) was haspy as he finaly took long vacation.
Video Type: Animation
Video Type: Animation
Video Type: Film
Explicit Reasoning
IV. Causal Reasoning
V. Sequential Structure Reasoning
VI. Counterintuitive Reasoning
Quastion: What might the girl writing trriing card for?
Question: the viceo played reverse? Why?
Questinn: How does
make the pan disapear??
Options
Optiam:
[Ai talentine's Day ce celerrationn
Optlane
(A) pains the per and makes thouugh it has varihedd
kill's sirthdan.
the video not the card toneer collupaing played forward.
(8) camara out to give
the pen disapporiig
IC gradurtion celeborion.
[0] Applogy for mittae.
(4] drops the pen into lap while gesturing forward
played mesemse the letter from wall. align with
tell from the man
hands.
A shared achievement celebration.
(0 the sllpeed the pan
bleck pocket
him.
Il vistt her serioucly boyfteen and with him
i| played reverne; the man's forward motion when the frut buoket a filldd.
In pen hollow lapuss into hand.
[4 fnnwwel for move near city.
(4) uses thumb secrotly pon.
Video Type: Film
Video Type: Life
Video Type: Life
Figure 2:
Overview of six tasks in MMR-V V Bench.
Explicit Reasoning evaluates whether a model can perform reasoning based on multimodal details explicitly presented across long-range, multi-frame of video. However, solving these tasks demands fine-grained perception and rigorous logical reasoning. "(explicit) allocates attention to the effortful mental activities that demand it, including complex compulations." -uul Process Theory.
2.2 IMPLICIT REASONING TASKS
Metaphor Understanding (MU): MU tasks evaluate the ability to reason about metaphors for entities or environment. For example, the case in Figure interprets the metaphor the brown coat.
Theme Understanding (TU): TU assesses the ability to infer the main idea and attitude of the author through the full video. For example, the case in Figure II asks what social issue the video reveals.
Emotion Recognition (ER): ER tasks evaluate the ability to analyze character emotional states, as well as higher-level emotions such as the author's attitude and the audience's 's emotional response. For example, the case in Figure 2 III involves inferring whether the character feels happy at the end.
Comment Matching (CM): CM task is to predict the most fitting audience comments for video based on criteria. For example, selecting which comment would be the most humorous after watching the video. Detailed example can be found in Appendix E.
Implicit Symbol (IS): IS task is to interpret implicit symbols in the video, such as cultural elements. For example, inferring the ethnicity of the filming location. Details can be found in Appendix E.
2.3 EXPLICIT REASONING TASKS
Causal Reasoning (CAR): CAR assesses the ability to reason about causal relationships the video. For example, in Figure IV, it involves inferring the reason why the girl is making card.
Sequential Structure Reasoning (SSR): SSR tasks assess reasoning about temporal structure in video editing and storytelling. In the example from Figure 2 V. the task is to infer if the video is reversed. However, the creator of this video explains the video is played normally.
Counterintuitive Reasoning (CIR): CIR tasks evaluate the ability to analyze information that contradicts common sense, requiring detailed cross-frame analysis. In the example from Figure 2 VI, the task is to reason the principle behind the counterintuitive magic trick.
Codal Transfer Reasoning (CTR): To reason and match information out of the video that shares similar meaning. For example, find the quote with same theme of the video.
4

---

Published as a conference paper at ICLR 2026
Video Type and Intent (VTI): VTI tasks test the ability to infer key meta-level information such as the genre and communicative intent of the video from a global perspective. For example, the case in Appendix E.2 infers the release time by reasoning the video is set during COVID-19.
MMR-V BENCH
To ensure that MMR-V effectively evaluates multimodal reasoning abilities, we follow three princi- ples during construction: P1. Multi-frame: Questions require reference long-range, multi-frame information, prompting the model to reason across multiple visual cues. P2. Deep reasoning: An- swers should not be directly perceivable from the video; instead, they should demand understanding of the subtext multimodal reasoning, reflecting a deep comprehension of the content. P3. Realistic: Tasks should align with real-world question-anse -areiirnn needs, ensuring answers are consistent with common user understanding and free from individual cognitive biases prejudices.
3.1 VIDEO COLLECTION
We manually curated diverse original videos from Youtube with the following checklist: (1) Avoid- ance of linear, descriptive content: We excluded videos with straightforward structures, such as daily recordings or sports broadcasts, in order ensure that the tasks require deep reasoning over multi-frames (For Principle P1). (2) Creative and thematically rich videos: We selected videos that are intentional designed and edited by creators, often conveying well-crafted themes. This ensures that the questions require interpretation beyond surface-level visual content (For Principle P2). (3) Alignment with real-world: Highly Popular Videos were preferred, which are indicated by active comment sections and audience engagement. This helps avoid biases introduced by niche content and ensures alignment with general user cognition (For Principle P3). (4) Diverse coverage: To further promote generalizability, we ensured broad coverage across video types, topics, and durations, allow- ing MMR-V V to reflect the diversity of real-world video content (For Principle P3). As a result, our final benchmark comprises 317 videos spanning six major categories: Animation, Film, Philosophy, TV, Life, and Art. The specific categories are shown in the Appendix D. Furthermore, for problems where audio might be helpful, we ensure that the videos include audio.
3.2 DATA ANNOTATION & QUALITY ASSURANCE
All tasks in MMR-V V Bench are designed in a multiple-chotee format. There is one correct option and several wrong options. We make sure there are carefully crafted distractors among the wrong options. To ensure quality and plausibility of these distractors, we designed three distinet distractors annotation strategies. (1) Str. I: We prompt strong model GPT-4o (Hurst et al., 2024) to directly answer the manually annotated question. If the model generates an incorrect answer (as verified by human annotators), that answer is retained high-quality distractor. If correct, combine Humann-wiitee distractors with incorrect options generated by GPT-4o as distractors. (2) Str. Given the question and correct answer annotated manually, GPT-4o is prompted to generate distractors. (3) Str. 3: Human annotators construct distractors manually.
We conducted a test using 100 questions, using three strategies to form three test-set with 100 multiple-choice tasks. As shown in Table 1, dis- tractors generated by strategy are more confus- ing, significantly increasing the difficulty and qual- ity of our tasks. It is worth noting that in the above test process, when GPT-4o directly answered 100 tasks, the accuracy rate verified by humans was only 17%. This reflects the limitations of the current model in multimodal reasoning capabilities.
Table Performance on 100 questions anno- tated with different strategies (str.).
Models
Str.
Str. 2
Str.
GPT-4o
59%
70%
62%
Qwen-VL-7B
37%
51%
42%
To ensure high quality, we also developed an checklist based on the construction principles and invited human annotators to verify the accuracy and difficulty of the tasks using this checklist. We invited five annotators with at least bachelor's is degree participate in the annotation and review process. The checklist MMR-V is shown in the Appendix C. The overall annotation process and the annotation platform be found in Figure 7 and Figure in the Appendix C.
5

---

Published as a conference paper at ICLR 2026
3.3 DATA STATISTICS
Table Dataset Statistic of MMR V
MMR-V comprises a total of 317 videos spanning a wide range of content types, and includes 1,257 multiple-choice reasoning tasks. Each question is annotated with 7 to 11 candidate answers, with only one correct answer guaran- teed. As illustrated in Figure 9a, the videos are categorized into six major domains, each encompassing fin-ggriine subcategories to ensure diversity in content, style, and semantics. The reasoning tasks our benchmark are orga- nized across three levels of granularity, reflecting different dimensions of reasoning complexity and modality. The distribution of task types across these levels is shown in Figure 9b. More information shown in Table 2. Table 5 shows statistic comparison with previous benchmarks.
Dataset
Statistic
Task
Question Count
1257
Average Option Count
10
Average Question Words
14
Average Option Words
10
Video
Video Count
317
Minimum Length
7
Maximum Length (s)
3771
Average Length (s)
277
EXPERIMENTS
4.1 SETTINGS
We conducted extensive evaluations on 11 proprietary and 10 open-source models as detailed in the Appendix F. Our main experiments were conducted under two settings: zero-shot and zero-shot + CoT (Wei et al., 2022), in order to examine whether reasoning enhances performance. For further analysis, we introduced the following categories comparative models: (1) Models with different scales. (2) "Thinking" model and its base version. (e.g., Cemmni-FFaash and Gemini-Flash- -ee---Faahhhhii (lash-Thinking).
Multimodal Inputs: For models supporting full-modal inputs (c.g., Gemini-2, 2.0-flash), we further compare their performance with and without audio input to evaluate its influence reasoning.
Frame Selection: Since some models only support multiple images or short video clips, we standard- ized the number of input frames. Details of frame sampling are provided in Appendix F.
Human Experiment: To provide a meaningful upper bound for MMR-V and to examine the human- model gap, invited participants with at least bachelor degree to conduct human experiment. We sampled 100 tasks GPT-4o answered incorrectly and 100 tasks answered correctly for experiment.
4.2 MAIN RESULTS
Results Table highlight the challenge of MMR-V. The highest score was achieved by Gemini-22 pro with I fps video sampling, reaching only 64.3%. Under the setting of fixed imput frame number, GPT-5 obtained the best score of 60.9%. Among open-source models, Gemma-3-27bb performs the best, demonstrating relatively strong performance. However, there remains a gap compared to proprietary models. The performance on videos of different lengths is shown the Appendix G.I.
Current reasoning enhancements have limitations MMR-V. Results in Table 3 show that current reasoning enhancement strategies, which are relatively effective in textual domains, such as CoT prompt reasoning and scaling test-time compute (i.e., "Thinking" models), offer only limited gains on MMR-V. V. CoT brings only a 0.88% average gain, and "Thinking" model improves just 2.4%. This indicates that MMR-V presents significant challenge to the multimodal reasoning capabilities of existing models. Analysis of sampled model responses shows that visual analysis accounts for only about 10% of the CoT's This reveals that reasoning process of current model mostly text-based (reasoning on questions and options), relying on visual perception of question frame, instead of integrating visual reasoning and evidence mining into CoTs. Several examples are provided in Appendix J, and further analysis in Section 4.5 supports similar findings.
Model performance on MMR-V Bench shows a clear scaling law effect. Smaller models under the same architecture perform poorly on tasks that require complex reasoning. For instance, larger models like Qwen2.5-VL V-BB22 (39.1%) and GPT-4o (52.8%) outperform their smaller versions Qwen2.5-VL VL- 7B 13) and GPT-4o-mini (34.8%), showing relative gains of 9% and 1 18%, respectively.
Model performance across different tasks on MMR-V Bench.

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation results (%) on MMR-V. Results under CoT prompting are highlighted in gray. The random accuracy on MMR-V Bench is approximately 10%. Bold and underlined values indicate the best performance among proprietary and open-source models, respectively.
Tasks
Video Categories
Model
Overall
Implicit
Explicit
Art
Life
TV
Film
Ani.
Phi.
Open-source models
LLaVA-Videe
18.4
17.6
19.1
18.1
15.4
16.3
14.4
11.2
13.2
17.4
21.4
12.8
NVILA-8B- Video
25.5
25.3
26.2
24.2
23.9
25.9
17.3
21.3
23.5
21.6
38.0
21.8
Phi-4-mullimodaal-msttuuu
26.7
27.6
29.4
31.2
19.4
18.1
19.4
19.2
25.9
26.4
33.9
24.4
CogvIm2-video-llaman
25.6
26.1
25.4
26.2
26.1
25.7
15.5
18.3
24.7
19.1
43.2
20.8
Qwen2.5-VL-7B
30.1
32.4
33.7
36.2
20.8
22.5
20.9
18.1
29.6
21.2
48.4
19.8
Intern--BB
33.6
32.9
35.5
33.4
28.6
31.4
23.0
22.6
31.7
24.3
52,9
23.2
Gemma-3-12b-it
34.0
34.2
37.8
37.6
24.0
25.4
19.4
24.9
25.9
31.3
51.9
24.4
IntermVL2.5-38B
39.9
43.8
43.7
29.9
29.4
28.8
30.4
37.2
57.4
29.1
Qwen2.5-VL-72B
39.1
40.4
41.3
42.8
33.4
34.3
28.9
28.2
29.1
36.5
55.6
37.2
Gemma-3-27b-i
42.0
41.1
46.5
44.7
30.3
32.0
31.7
32.2
35.5
41.3
56.1
33.7
Proprietary modely
GPT-4o-mini-20244- 20220-0---2-221-111
34.8
35.2
38.0
38.6
26.3
26.3
29.5
25.4
29.6
33.0
48.7
Gemini-2.0-Flaat (16 frames)
42.6
44.3
44.3
45.9
38.3
40.0
30.9
32.2
40.7
40.6
58.5
24.4
1anul-335-Somme-2222444
43.3
44,2
45.0
46.1
38.9
39.1
33.8
31.1
41.3
41.3
55.8
4.4
Semmnn-2.0--Fash-ttinnee
45.0
43.5
46.6
46.0
40.6
37.1
34.5
31.6
38.6
48.3
60.1
25.6
GPT-441-2025-04-11
46.6
48.9
49.1
51.7
40.3
41.7
43.2
35.6
43.9
46.5
57.1
34.9
Gemini-2.0-Flash (512 frames)
48.0
49.9
50.5
52.6
41.6
42.9
36.7
36.7
39.7
46.2
66.7
31.4
Gemini-2.5-Flash
$1.2
50.5
52.9
52.3
46.9
45.7
45.3
39.5
50.3
47.9
65.6
34.9
44-mini-2025-04-16
52.5
52.1
54.6
54.5
47.1
46.0
48.2
40.1
54.0
51.7
65.3
27.9
(PTT-4--0224-11
52.8
55.0
54.7
48.1
51.4
42.5
50.0
67.7
38.4
03-2025-04 06-166
59.1
58.3
60.1
57.8
57.6
59.7
51.8
45.2
55.6
58.3
74.3
43.0
GPT-5-2025-00 25-20025-08-07 -07
60.6
60.9
61.0
60.6
59.7
61.4
56.8
41.8
56.6
60.4
76.5
45.4
Gemini-2.5-pro (Ifps)
64.3
63.2
65.9
64.4
60.2
60.3
57.6
58.1
65.6
62.5
73.0
54.6
Baseline
Best Performance of Models Human
64.3
65.9
60.3
57.6
58.1
65.6
62.5
74.3
54.6
86.0
80.6
91.2
57.7
92.3
90.6
92.3
90.7
70.0
Firstly, the models performed better on implicit tasks than on explicit tasks (with an average gain of +7.9%). Through analysis of tasks and model responses, we found that in implicit tasks, video cre- ators often embed implicit meanings throughout the entire video, resulting in abundant visual cues that can support reasoning. This reduces the requirements for multi-modal reasoning and clue localization. In contrast, explicit tasks demand finer-grained reason- ing and the ability to identify specific evidence. For example, in the implicit task at the bottom of Figure many frames provide clues suggesting that the girl symbolizes good luck (e.g., room number, flowers, lighting, weather, etc.). In contrast, the explicit task at the top contains only few key frames where the hidden lighter in magician's 's hand be seen.
GPT-4o
Gemini-2.0-EF 2..s--aah--hhiiiiii
Claude-3-5 SommSeet
Qwen2.5.VP VL
Gemini 2.5.
o4-mini
CAR
IS
SSR
CM
66
40
CIR
CTR
Secondly, the models performed particularly poorly on Counterintuitive Reasoning (CIR), Se- quential Structure Reasoning (SSR), and Comment Matching (CM) tasks. For CIR and SSR, poor performance mainly stems from the limited ability of models to perform multi-frame reasoning. These two tasks require the model to reason on long-range videos, rather than relying on internal knowledge. However, instead of analyzing to locate evidences in other frames, models often rely on surface-level visual perception of the question frame, followed by textual reasoning over question and options. For CM, the results highlight a significant gap between model and human capabilities in implicit reasoning. While humans can infer underlying information such as humor and emotion with minimal cognitive effort (Krishna et al., 2022), current models consistently fail to capture such subtleties.
Figure 3: Performance on different tasks.
7

---

Published as a conference paper at ICLR 2026
Table 4: The impact of adding audio modality on the performance (accuracy %) on different tasks.
Tasks
Categories
Overall
Imp.
Exp.
Life
TV
Film
Gemini-2.0
42.6
44.3
38.3
32.2
40.7
40.6
taudão
44.07%
46.2
38.3-0.0
31.070.1
31.6
42.3
41.070.4
61.1 11.1
29.1 M4
Gemini- 2.0 2.0-tiiki..tiiiingg thinking
45.0
46.6
40.6
34,5
31.6
38.6
48,3
60.1
25.6
taudio
46.071.0
48.4
39.7-9.9
31.712.8
33,9
44.4
42.745.6
62,4
32.677.0
32.6
Phi---mmttimddal--nnuuutt
26.7
29.4
19.4
19.2
25.9
26,4
33,9
24.4
+audio
17.7
31.371.9 11.
18.1-1.8
15.149.0
19.770.5
21.54144
27.8 87..
37.3
37.4
26, 722.3
Human Performance. Humans achieved an average score of 86%, highlighting a significant human- model gap. Although studies suggest models reached human-level performance on text tasks (Guo et al., 2025; OpenAl, 2023), they still lag on multimodal reasoning. Humans can identify clues in videos easily, while models tend to focus on question frames rather than exploring other evidence frames. Specially, unlike models, humans perform slightly worse implicit tasks, which is mainly due to the challenges posed by highly abstract implicit understanding art and philosophy.
4.3 INFLUENCE OF FRAMES COUNT
Accuracy vs. Frame Count
For Semmini-2.0-Flas which supports long video in- puts, we evaluated performance changes as the number of frames increases. As shown in Figure 4, accuracy improves with more frames, but the rate improve- ment gradually slows. After sampling and observing the CoTs, it is found that the initial gains come from the addition of evidence frames, while the slowdown is mainly due to limited multi-frame reasoning ability of implicit tasks continues to
50
48
46
Overall
Accurccy
Implicit Reasoning
44
Explicit Reasoning
42
the model. Performance improve in later stages, as visual cues for such tasks are often dispersed throughout the video (as discussed in Section 4.2); more frames tend to provide more clues. In contrast, explicit clues are fewer and more localized.
38
32
64
96
128
512
Frame Count
Figure 4: Accuracy with the increase of input frame counts.
4.4 INFLUENCE OF AUDIO INPUT
For models supporting full-modal input, we compared performance before and after incorporating audio. As shown in Table overall performance improved with the addition of audio. Specifically, Gemini 2.0-Flash. Gemini 2.0-Flash- Thinkinn and Phi-4-multimodal iallimoodallinstucct instruct improved by 1.4%, 1.0%, and 1.0%. respectively. This suggests that advancing research on fully multimodal models is promising direction. A case study illustrating how audio aids reasoning is provided in Appendix G.2.
Lock Visual Reasoning
Implicit Misinterpnetation
4.5 ERROR ANALYSIS
Knowledge Insufficiency
Reasoning Error
Output Formatting Issue
Hallucination
We sampled 100 incorrect responses from GPT-4o for ror analysis. The errors can be categorized as follows: (1) Lack of Visual Reasoning: the model often failed to locate the correct evidence frames and lack of long-range, multi-frame visual reasoning. (2) Implicit Misinterpre- tation: revealing significant understanding gap between the model and human cognition. (3) Knowledge Insuf- ficiency: the model lacks some intrinsic knowledge (4) Reasoning Error: during the multi-step deduction pro- cess. (5) Hallucination: the model introduced fake or unsupported information. (6) Output Formatting Issue: model refusals formatting errors prevent answer traction. Among error cases, Lack of Visual Reasoning accounts for the largest proportion. This indicates that
2%
13%
47%
13%
26%
Figure 5: Error analysis of GPT-4o.
8

---

Published as a conference paper at ICLR 2026
Phi-4- Mole--MMliiooil-nnneee
GPT-4-mini
GPT-5
Gemini-2 S-pco
100
Other Frime
Rosceeing Ratio (16)
Question Framx
Vileo Analysis
60-
Text Analysis
40
20
30
70 10
60 Ousput Progress (9%)
100
40 50 Output Progress (5%)
90
20 30 70 Ourput Progress (%)
Ourput Progress (%)
Figure 6: CoT content across different stages. The y-axis indicates the ratio of the 500 sampled CoTs that include analysis of these four types of content at each stage.
current models still lack genuine multimodal reasoning capabilitiis. They tend to rely on rext-based reasoning after briefly perceiving frames adjacent to the question, rather than engaging in long-range, multi-frame video reasoning. Most existing reasoning models remain inadequate in integrating multimodal information into the reasoning process and performing thorough analysis. In contrast, models like GPT-5, o3 exhibits a better reasoning paradigm, as shown in Figure 13 for comparison. Details of the error category annotation pipeline are provided in Appendix C.3.
We further analyzed model CoTs to examine performance differences by categorizing each reasoning step as either video or text analysis (e.g., options), with video analysis further divided into question- frame and wther-frame analysis (details in Appendix H). We sampled 500 CoTs from models, split each into 10 equal-length segments, and used GPT-4.1 I to label each segment. As shown in Figure 6, where models further to the right perform better on MMR-V, V,, models with better performance on MMR-V show more video analysis, especially Semini--25-prra stand out with strong analysis of non-question frames, highlighting the value of enhanced multi-frame visual reasoning in video reasoning tasks.
other frames (red line). Notably, GPT-5 and
RELATED WORK
Video Understanding Benchmark. Existing video benchmarks mainly evaluate models' perception and understanding of video, such as action recognition (Khattak et al., 2024: Mangalam et al., 2023; Patraucean al., 2023; Xiao al., 2021) and description (Xu et al., 2017: 2016). Recent works like Video-MME (Fu et al., 2024), MVBench (Li et al., 2024b), TempCompass (Liu et al., 2024a) and MMBench- Video (Fang et al., 2024) extend video understanding to diverse video types, enabling more comprehensive assessments. Benchmarks such as L.Bench (Wang et al., 2024b), Long VideeBeenen (Wu et al., 2024), and CG-Bench (Chen et al., 2024) further introduce long video QA. However, these largely test whether models can extract relevant information from long videos, with evaluation remaining perception-oriented MMR-V V instead assesses whether models can perform multi-frame, long-span span, multimodal autonomous reasoning from questions.
Multimodal Reasoning, Recent advancements has greatly enhanced LLM reasoning (Guo et al., 2025; Jaech al., 2024; Team et al., 2025; Zhao al., 2024), yet the evaluations still focus text-based reasoning (Hendrycks et al., 2021; Bai et al., 2024; Cobbe et al., 2021; Rein et al., 2024; Wang et al., 2024c). MLLMs still lack thorough assessment in this area (Qin et al., 2023; Wang et al., 2023). Current multimodal reasoning benchmarks mainly involve mathematical or coding tasks in image form (Wang al., 2024a; Shi al., 2024; Ying et al., 2024), which primarily test visual recognition followed by text reasoning, True multimodal reasoning requires richer cross-modal cues. To address this gap. MMR-V V is designed to evaluate video-based reasoning tasks.
Video Reasoning, Recently, several benchmarks have emerged to access reasoning in videos. VideoE- spresso (Han et al., 2025) utilizes a fully automated pipeline to generate scalable CoT reasoning data in videos. Video-MMMU (Hu et al., 2025) and MMVU (Zhao et al., 2025) focus on evaluat- ing specialized knowledge reasoning within multi-disciplinary videos (c.g., physics, economics). VRBench (Yu et al., 2025) targets narrative reasoning in long narrative videos and designs human- model collaborative stepwise annotation pipeline. VideoReasonBendd (Liu et al., 2025) assesses vision-centric reasoning using synthetic videos generated by engines. MMR-V V distinguishes itself by ensuring all QA pairs are purely manually annotated, evaluating general multimodal reasoning across diverse real-world open-domain videos.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
This paper introduces MMR-V: V: A Benchmark for Multimodal Deep Reasoning in Videos. All tasks are human-annotated. MMR-V poses significant challenge for current models, with the best performance still trailing human accuracy by 21.7%. This highlights a human-model gap in interpreting and reasoning about video information. Notably, we observe that models with higher accuracy on MMR-V V tend to perform more extensive and in-depth analysis of videos, suggesting that incorporating multi-frame reasoning into CoT or leveraging tool use may promising directions for advancing video reasoning. We hope MMR-V V will serve as reliable evaluation benchmark for the development of MI.LMs and offer valuable insights into advancing multimodal reasoning research.
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China (No. U24A20335, No. 62406321). This work is also supported by Beijing Natural Science Foundation (L243006).
ETHICS STATEMENT
All experimental procedures involving human participants were conducted in accordance with the relevant ethical guidelines. The MMR-V data was gathered and annotated by compensated internal annotators, from whom all necessary informed consent was obtained. The dataset was carefully reviewed for safety and potential biases. We have prepared the dataset for public release under the CC-BY 4.0 license.
REPRODUCIBILITY STATEMENT
We have taken several steps to enhance the reproducibility of our research. The code for the experiments and analyses reported in the main text is provided in the supplementary materials. The experimental setup of the main experiments and the selection of video frames detailed in Section 4.1 and Appendix F. Further details regarding dataset construction and preprocessing are described in Appendix C and Appendix E. Finally, our dataset has been prepared for open release to support future research. The JSON file containing the questions, answers, and related information for MMR-V can be found in the supplementary materials.
REFERENCES
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao. Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang. Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song. Tetyana Sych, Praneetha Vaddamanu, Shuohang Wang, Yiming Wang, Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang. Ziyi Yang, Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. CoRR, abs/2503.01143, 2025. doi: 10.48550/AXXW2250001142
/ttt:p/////oooooo///.445555///// URL https: /ol. org / 48550/ www.:./aavvvvvv.m 2503.....550.... 2503. 01743.
Anthropic. Anthropic: Introducing claude 3.5 sonnet, 2024. URL https //tp::/wwwwww/// www.rro.............. 5-sonet.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao. Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412. /5204 2024.
10

---

Published as a conference paper at ICLR 2026
Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg-bench: Clue slue-groudddd question answering benchmark for long video understanding. arXiv preprint arXiv:2412.12075, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv preprint, abs/21 141641 2021.
Jonathan St BT Evans. Heuristic and analytic processes in reasoning. British Journal of Psychology, 75(4):451-44) 468, 1984.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 37:89098 89124, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi- ilms video analysis. arXiv preprint arXiv:240S.21075. 2024.
Google DeepMind. 2025.
Gemini 2.5:
Our most intelligent ai model, March wwwch//ooo//ooggleeeeeeeeeee /gogggeeeeeeeedddddd
URL bttps::/blss
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
Songhao Han, Wei Huang, Hairong Shi, Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale thain-of-thought dataset for fine-grained video reasoning via core frame selection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26181-26191, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ArXiv:2103. 03874,, 2021.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
Jun Hou, Chengdong Wu, Zhongjia Yuan, Jiyuan Tan, Qiaogiao Wang, and Yun Zhou. Research of intelligent home security surveillance system based on zigbee. 2008 International Symposium on Intelligent Information Technology Application Workshops, pp. 554-557. IEEE, 2008
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint XrXiv:2501. /38.66.6.136..... 2025.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint 2.Xi::24/2.16720,, 2024.
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Tean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Róbert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil
11

---

Published as a conference paper at ICLR 2026
Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky. Jiaming Luo. Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Chuee Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Bimezak-Plucinskk Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, and Ivan Nardini. Gemma 3 technical report. CoRR, abs/2503.199886 2025. doi: 10.48500ARXIV:2503.U URL https: /tt.//oiiiia// wwwpoooo......ooo 48544/5554000 arxiv 2501119786.
Muhammad Uzair Khattak, Muhammad Ferjad Nacem, Jameel Hassan, Muzammal Naseer, Federico Tombari, Fahad Shahbaz Khan, and Salman Khan. How good is my video Imm? complex video reasoning and robustness evaluation suite for video-Imms. arXiv preprint arXiv:2405.0300 2024.
Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit lyyer. RankGen: Improving text generation with large ranking models. In Proceedings of EMNLP, pp. 199-232, 2022. URL https: //ac///a //acannhoology.. org/ 2022. cannp-main...
George Lakoff and Mark Johnson. Metaphors we live by. University of Chicago press, 2008.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.055 033326, 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195 22206, 2024b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXi::2311.10122. 2023.
Yuanxin Liu, Shicheng Li. Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video lims really understand videos? arXiv preprint 2rXiv:2403.004770 2024a.
Yuanxin Liu, Kun Ouyang, Haoning Wu, Liu, Lin Sui, Xinhao Li, Yan Zhong, Charles, Xinyu Zhou, and Xu Sun. Videoreasonbench:: Can milms perform vision-centric complex video reasoning? arXiv preprint arXiv:2505.23354 2025.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04465 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Hei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212-46244 2023.
OpenAl. Gpt-4 technical report. arXiv preprint arxiv:2003.08774, 2023. URL https://aaax axg/pdf/2303. 08774. pdf.
OpenAl. Openai: Hello got- 2024a.
URL https://openai..om//t
hel 1o-gpt-40/-
12

---

Published as a conference paper at ICLR 2026
OpenAI. Gpt-4o mini: advancing cost-efficient intelligence, 2024b. URL https://openai.
com/inde//ppt-400 htw.::/:waoo....a.aooooooooaaaaa
OpenAl. Gpt-5 is here, 2025a. URL httph:://///c. htttttss://peaaa..coo cttp://taa..cggp/t/.....os/
OpenAl. Openai: Introducing openai o3 and o4-mini, 2025b. URL httttt://///pnaa/i index/introducing
aad-o4-mini.
OpenAI. Introducing gpt-4.1 I in the
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36:
42748 42761, 2023.
Michael Polanyi. Personal knowledge. Routledge, 2012.
Haotong Qin, Ge-Peng Salman Khan, Deng-Ping Fan, Fahad Shahbaz Khan, and Luc Van Gool. How good is google bard's visual understanding? an empirical study on open challenges, 2023.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.05530, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proot q&a benchmark. In First Conference on Language Modeling, 2024.
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng. Lidong Bing, and Roy Ka-Wei Lee. Math-llava Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294 2024.
Ron Sun. The clarion cognitive architecture: Extending cognitive modeling to social simulation. Cognition and mulli-agent interaction, pp. 79-99, 2006.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.125999 2025.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024a.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: extreme long video understanding benchmark. arXiv preprint arXiv:2406.0800 08840035088, 2024b.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: comprehensive survey. Machine Intelligence Research, 20(4):447-455 2023.
Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal thain-of-thouu reasoning: A comprehensive survey. arXiv preprint arXiv:2503.12605 2025.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of thothought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 22-837, 2022.
13

---

Published as a conference paper at ICLR 2026
Haoning Wu, Dongxu Li, Bei Chen, and Jumnan Li. Longvideobench: A benchmark for long-context interleaved language understanding. Advances in Neural Information Processing Systems, 37:28828 2885-,887772 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786, 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan Hc, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui. Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024a. doi: 10.45550ARRXI22412.55111 URL https: /ww.i http///ii.oogo/////// 48850//r48555/xx1111. 2412. 15115.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171 2024b.
Kaining Ying, Fanqing Meng, Jin Wang. Zhiqian Li, Han Lin, Yue Yang, Hao Zhang. Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large lision- models towards multitask agi. arXiv preprint arXiv:2404. /6006, 2024.
Jiashuo Yu, Yue Wu, Meng Chu, Zhifei Ren, Zizheng Huang, Pei Chu, Ruijie Zhang, Yinan He, Qirui Li, Songze Li, et al. Vrbench: A benchmark for multi-step reasoning in long narrative videos. 2025.
arXiv preprint arXiv:2506.
Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks. arXiv preprint arXiv:2410. 12381, 2024.
Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi-discipline video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8475, 2025.
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411. 14405, 2024.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint ArXiv:2406.04266 2024.
Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu. Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 25Xi:2550441479 2025.
A LLM USAGE STATEMENT
In this paper, LLMs used to check grammar and polish text, and we carefully verified that their use did not alter the original meaning. In addition, AI assistants were employed to improve the presentation of some tables and figures, but we ensured the experimental results remained unchanged.
14

---

Published as a conference paper at ICLR 2026
1. Video Collection
Checklist
2.Data Annotation
(1) High recogrition
Youlube
YES
intergrstation?
Manually written QAs
Reject
stro srogghffonesrd
(2) Nos-
Audios
Why damaged di?? wind suddenty get
Video Pool
3. Quality Assurance
result, unboellls restiondd
Checklist
(1) lorg-spon, reesoning requined?
Generate False Options
desper understanding required
Watch the video and give a correct answer. Then generate misleading wrong options.
seteensecooeeee obtoied directly through
Manually Quality Check
perception
(3) the QAA corsistent with red sorrld and common
understanding!
Correct answer of GPT-46 44 and Manual some meaning?
Combination
Why did the brokin umbrella in the boy's hand, darmgged by the wind, get foxed while the sir?
Questioin Cornect Option False Options
Conect unswrr of GP 14 miseening optine
NO
Becase the held the hand, good lock result. umorddl restared.
Minually writien
(i) The dle was rever sed aring waiing
misleoding options
YFS
was garting
comtmbuud
Figure 7: MMR-V Construction Pipeline.
B LIMITATIONS
Despite our efforts to improve our work, several limitations remain. (1) Scaling MMR-V is challeng- ing due the high cost of manual annotation and verification, as all tasks and correct answers curated and reviewed by human annotators. (2) Although we strive to cover a wide range of video and task types, certain real-world categories (such mystery, puzzle-solving, and gaming) underrepresented. (3) The majority of videos in MMR-V are in English, with only a small proportion in other languages such as Chinese, French, Thai, and German, which constrains its multilingual applicability. We will further study and try to solve this issue in the future.
still
C MMR-V CONSTRUCTION
C.I CHECKLIST
According to the MMR-V construction principles introduced in the main text Section 3 we wrote the following annotation checklist:
(1) You are expected to watch the entire video before formulating any questions or answers.
(2) Each question must require long-distance, multi-frame reasoning and cannot be answered through direct perception (ensuring compliance with Principles and 2).
encour- aged to refer to the official interpretation of the original video author and user consensus (highly praised comments in the comment section) when writing or verifying the correct answer. This helps mitigate annotator bias and ensures that the reasoning task reflects the understanding of wider audience.
(3) To ensure consistency with real-world user perception (Principle 3), annotators
C.2 CONSTRUCTION PIPELINE
In this section, we present the construction process of MMR-V V Bench macro sense. The whole process is divided into three stages: video collection, data annotation, and quality assurance, For video collection, we designed a checklist to ensure the quality and diversity videos in the Bench. "High recognition interpretation?" ensures that the questions raised and the annotated answers based
15

---

Published as a conference paper at ICLR 2026
MMR-V Annotation Platform
Arcs
Bcd Canaf Doo Otfid Interted
Manaal Pariah ludlung Edbesun Oiher
Sove don 2014
ciicled
Figure 8: Annotation Platform of MMR-V,
on the video have references that are consistent with public cognition (official interpretations highly praised comments) to alleviate the subjective bias of the annotator. "Is it Non-straightforwward"" ensures that the video is not straightforward narrative, which is conducive to increasing the reasoning difficulty of the question. For data annotation, as described in section 3.2 of the main text, we use gpt-4o to assist in annotation with interference options. Let the model generate the correct answer based on the question, and manually review to ensure that the correct answer generated by the model different from the manual annotation. they different, the answer generated by gpt-4o used as the interference item, otherwise the interference item is manually written. For quality assurance, we designed checklist for human reviewers check the correctness and difficulty of the tasks. The annotation platform is shown in Figure 8.
C.3 PIPELINE OF ERROR ANALYSIS
To ensure the reliability of the error analysis presented in Section 4.5, we implemented annotation pipeline and quality control strategy. Firstly, we categorized model failures into six distinct types based initial analysis of the false answers. To standardize the criteria, selected representative anchor examples for each category to distinguish boundary cases. Some examples are presented in Appendix Next, mitigate author bias, invited non-author evaluators review the definitions and anchor examples, ensuring they align with general human understanding before formal annotation. Finally, the primary authors annotated the sampled responses based on the established guidelines. Any disagreements were discussed and resolved to ensure 100% consensus of authors on the final error distribution.
D DIVERSITY OF MMR-V
In this section, we show the diversity of MMR-V Bench, including video diversity and task diversity, For video, we show the six categories of videos in MMR-V V Figure 9a, including Life, Animation, Film, Art, TV, and Philosophy. At the same time, for each category, subcategories to better understand the classification video categories. Secondly, in section 3.3 we show the diversity of video length, ranging from 7 seconds to 3771 seconds. For tasks, divide them into two parts, ten categories and 33 subcategories, three levels. The division of the first and second levels, well the proportion of different types of tasks, be seen in Figure 9b.
divide it into several
16

---

Published as a conference paper at ICLR 2026
Muge
TV Show
Vedeo Type
and lnteit
crast moudh Tharatee
Public Sunsive
Understann perestaaaiin
Mtr Metophor
mestoyaphy
Commeriia
Martening
Social tnnves
Art
AL
Poleeoppyy
Structure Resioring
Sequential
Daily Theree
Animattonn
Aydoso
Psychulogy
Reasoring
Salt sof-itffeuzion
Understanding
Theme
Implicit Rewsoning
Cousal Peasoning
Hinorr
Fiii
Ljfe
perertttiee intro
Comvety
Cat Estirg
imptece Symbial
howe
Shout free
Jame Voleo
Franel
Emodion Recogmaion milon
Claricc
(a) Video categories.
(b) Proportion of different tasks.
Figure 9: (a) Video categories in MMR-V Bench. (b) Proportion of different tasks in MMR-V Bench.
Table 5: Comparison with existing video benchmarks. #Dur: Average video duration in seconds.
QA Anno.: Annotation method (A: Automatic, M: Manual, A&M: Both).
Benchmark
#Videos
#Dur.())
#QA Pairs
Data Source
Task Categories
Multilingual
QA Anno.
MVBench
3,641
16.0
4,000
Open-Domane
20
EgoSchemaa
5,063
180.0
5,063
Egocentric
X
X
A
TempCompass
410
11.4
7,540
Open-Domain
11
A&M
Video-MME
1017.9
2,700
Open-Domain
M
LongVideoBench
3,763
473.0
6,678
Open-Doman
17
X
M
LVBench
103
4101.0
1,549
Open-Domain
26
M
CGBench
1,219
1624.4
12,129
Open-Domain
Video-MMMU
300
506.2
900
Open-Domain
X
M
MMVU
1,529
51.4
3,000
Multi-Disc
M
VRBench
960
5796.0
8,243
Narrative
A&M
MMR-V (Ours)
317
277.0
1,257
Open-Domain
33
M
E TASK DETAILS
The tasks in MMR-V can be divided into three levels. Level 1: Implicit Reasoning & Explicit Reasoning, Level 2: Contains ten task classes, Level 3: Contains 33 task subclasses. Next, we will introduce these tasks with some task examples'.
E. IMPLICIT REASONING TASKS
I. METAPHOR UNDERSTANDING (MU)
For the definition of subclasses of the metaphor understanding task, mainly refer to the book Metaphors We Live (Lakoff & Johnson, 2008) By by George Lakoff and Mark Johnson, which introduces metaphor-related concepts in detail.
'The videas linked here are the original source versions. Note that the actual videos in MMR-V - were processed trim frames containing shortcuts the answers.
17

---

Published as a conference paper at ICLR 2026
Table 6:
Three-level classification of tasks in MMR-V.
Ability Type L1
Ability Type
Ability Type L3
Structural Metaphor, Orientational Metaphor,
Metaphor Understanding (MU)
Ontological Metaphor, Creative Metaphor
Philosophical Concepts, Social Issues, Personal Rellection,
Implicit Reasoning
Theme Understanding (TU)
Everyday Topics, Vidco Naming
Explicit Emotion, Implicit Emotion,
Emotion Recognition (ER)
Meta-emotion, Audience Emotion
Comment Matching (CM)
Humorous, Thought-prn provoking. Trending
Implicit Symbol (IS)
Cultural Symbols, Art Symbols, Other Symbols
Causal Reasoning (CAR)
Forwurd Reasoning. Backward Reasoning
Narrative Structure, Core Connecting Elements,
Sequential Structure Reasoning (SSR)
Explicit Reasoning
Inference on Editing Techniques, Hallucination
Magic Deconstruction Special Effects Editing,
Countcrintuitive Reasoning (CIR)
Artistic Techniques, Humor and Exaggeration
Cruss-modal Transfer Reasoning (CTR) Video Type and Intent (VTT)
Video-to- Video-to-Audio, Video-to- Video
Video Type, Video Intent
1.1. STRUCTURAL METAPHOR
Task Description: There are structural similarities between the subject and object. For example, time can be compared to flowing water, both of which have the structure of flow and passing away.
Example Question:
Question: What does the brown coat in the video symbolize?
Options:
(A) It is said to represent the family's 's long lost fortune that they still searching for.
(B) The brown coat symbolizes the lost hope of the family because was worn during a difficult time.
(C) It refers to coat that has been washed and taken out to dry, likely worn by the father.",
(D) It symbolizes the father in a family, who protects his family in times of difficulty.
(E) It represents the fear of the outside world.
(F) The unfulfilled dreams of the children in the family as they always saw as a sign of something unattainable.
(G) The brown coat in the video represents a raincoat, used to protect the clothes inside from getting wet.
(H) The bad luck that has been following the family for generations.
CosrectAnswer: (D)
Video: father minute emotional award winning video url
1.2. ORIENTATIONAL METAPHOR
Task Description: There are similarities in direction or composition between the subject and the metaphor, for example, walking up a staircase compared to ambition. Example Question:
Why does the dance, which is filled with artistry and beauty throughout, end with descent?
Question:
Options:
(A) There is connection between the fall and the creation at some point.
(B) It represents a dive to explore new depths, both literal and metaphorical.
(C) indicates the dancer's 's exhaustion, capturing a moment of fatigue.
(D) It reflects the calmness of the ocean, evoking a sense of tranquility.
(E) highlights the theme of rebirth, symbolizing renewal and transformation.
(F) It represents the beauty of underwater life, showcasing its unique allure.
18

---

Published as a conference paper at ICLR 2026
(G) symbolizes being weighed down by emotions, expressing inner turmoil.
(H) It symbolizes the end of dream, marking moment of conclusion.
(I) shows the dancer's connection to water, emphasizing fluidity and grace.
(J) It symbolizes return nature and surrender to life's forces, embracing the natural flow.
(K) It signifies the end the dance's energy, indicating point of culmination.
CorreetAnswer: (A)
Video: Falling Underwater dance video_ut
1.3. ONTOLOGICAL METAPIIOR
Task Description: This metaphor involves viewing an abstract concept as concrete entity. Usually, the core concept of the entire video is turned into a concrete entity to tell the story. Example Question:
Question:
The scene around 1:00 metaphorically represents what aspect of communities?
Options:
(A) Communities can build their resilience to setbacks by working together and adapting to new
challenges.
(B) Promoting individual success in competitive environments.
(C) Building resilience through community partnerships.
(D) Overcoming challenges for community progress.
(E) Celebrating the individual achievements of community members.
Developing sustainable practices for environmental harmony.
(G) Decision-making processes of a community.
(H) The interconnectedness of global communities.
(I) Isolation of communities for self-sufficiency.
(J) The role of external aid in community development.
(K) Highlighting the diversity of cultures within a community.
Cnswe: (A)
Video: Resilience: Anticipate, organise, adapt video_uri
1.4. CREATIVE METAPIIOR
Task Description: This metaphor is usually carefully designed by the author for a specific video and needs to be understood in the context of the video.
Example Question:
Question: What is the pink fairy ball in the film?
Options:
(A) It's toy the boy picked up on the street, having no special connection his condition.
(B) They are the microorganisms in this world, living in every corner.
(C) The pink fairy ball represents the boy's childhood dream of becoming a fairy.
(D) It's a hallucination caused by lack of sleep, not related to antidepressants at all.
(E) They are the boy's toys, which he bought help treat his depression.
(F) It is the effect of the antidepressants the boy is taking, which helps him see many things with
vitality and positive effects.
(G) It's an advertisement prop for new product in the background of the scene.
(H) The ball is a sign of the boy's 's wish to escape from his daily work routine.
(1) The pink fairy ball is symbol of the city's upcoming festival decorations
CorrectAnswer: (F)
Video: Soft Rain Animated Short Film (2023) video url
19

---

Published as a conference paper at ICLR 2026
II. THEME UNDERSTANDING (TU)
I1.1, PHILOSOPHICAL CONCEPTS
Task Description:
The themes of the videos are usually about concepts and principles related to philosophy and psychology. Example Question:
Question: What the overall message that the animation aims to convey?
Options:
(A) It suggests happiness comes solely from financial achievements.
(B) The animation emphasizes the need avoid all responsibilities.
(C) The animation aims to illustrate the ways to relieve stress.
(D) It illustrates the mechanical process of water flow.
(E) The animation encourages saving water to prevent wastage.
The animation conveys the importance of managing stress through self-care practices.
(G) The animation highlights achieving success through hard work.
(H) The animation suggests that ignoring stress leads to happiness.
(I) The video underlines the significance of collective teamwork.
(J) depicts progress and growth through constant work.
CorrectAnswed (C)
Video: The Stress Bucket video_ur
11.2. SOCIAL ISSUES
Task Description: The theme of the video usually to reflect some problems existing in today's society and express a strong appeal of the author.
Example Question:
Question: What social reality does this video satirize?
Options:
(A) The rise of environmental awareness in urban settings.
(B) The video represents the bystander effect in society.
(C) The economic disparities in urban vs. rural areas.
(D) The challenges of modern relationship dynamics.
(E) The impact of fashion trends on daily life.
(F) The inereasing complexity of urban development planning.
(G) The need for infrastructure improvement and road safety.
(H) The influence of social media public behavior.
(I) The rapid pace of technological advancement in transportation.
(J) The shift in societal values towards individualism.
CoswertAnswer: (B) Video: Stone Minute Short Film Hot Shot video url
II.3, PERSONAL REFLECTION
Task Description: The author hopes that the video will inspire people reflect and resonate with things in their lives.
Example Question:
Question: What is the core concept that the film aims to convey?
Options:
(A) Romantic relationships adolescence.
(B) The importance of education institutions.
(C) Overcoming supernatural challenges.
(D) The dynamics of family disagreements.
(E) Exploration of technological advancement.
(F) Not to judge others too quickly.
20

---

Published as a conference paper at ICLR 2026
(G) Journey of a superhero in saving the city.
(H) Inter-species relations Earth.
(I) Power struggles in political leadership.
(J) Historical recount of famous personality.
Coswe::Answer: (F)
Video: Award Winning SHORT FILMS Don't Judge BATTI Hindi Heart Touching Short
Movies Content Ka Keeda video_url
11.4. EVERYDAY TOPICS
Task Description: The themes expressed in the videos are usually the sublimation the insights and themes in daily life, such as praising maternal love, friendship, etc. Example Question:
Question: What is implied by the contrast between the scenes around 0:47 and 1:11? Options:
(A) The contrast shows that the mother is indecisive and can't 't make up her mind in a crisis.
(B) demonstrates the father's sense responsibility and bravery, praising paternal love.
(C) The contrast between the beginning and the end conveys
of tragedy, criticizing the
destruction of the ecological environment by humans.
(D) It shows that the father wants abandon the child when facing danger.
(E) It shows the bravery of the bird in the background, facing authority head-on, and praises
courage.
(F) The mother still protects her child at all costs even in the face of danger, which praises maternal love.
(G) It implies that the father is doing it for self preservation rather than out of love for the child.
(H) shows that even when there are many birds, they do not appear very united, and the face of danger, they become disorganized mess.
CorrectAnswer: (F)
Video: Mother 1 minute Sad Emotional Award Winning Iranian Short Film Animation Animated -video_ur _url
I1.5. VIDEO NAMING
Task Description: Come up with suitable title for this video or the core content of the video (dance, etc.). This tests the model's control over the overall content and whether it can get the subtleties of the title like humans.
Example Question:
Question: "Please come up with a suitable name for this dance.".
Options:
(A) The Dance of the Butterfly.".
(B) The Rhythm of the Phoenix.",
(C) The Grace of the Swan.",
(D) The Spirit of the Dragon.".
(E) The Charm of the Peony.",
The Step of the Tiger.".
(G) The Soul of "eacock"
(H) The Beat of the Forest.".
(I) The Leap of the Deer.".
(J) The Spin of the Star.".
(K) The Waltz of the Moon."
CorrectAnswer: (G)
Video: Yang Liping The Soul of Peacock Peacock Dance Traditional Dance video_url
21

---

Published as a conference paper at ICLR 2026
III. EMOTION RECOGNITION (ER)
III. 1. EXPLICIT EMOTION
Task Description:
Analyze the emotions characters in the video. Explicit emotions can usually be directly understood through facial expressions, body movements, etc.
Example Question:
Question: Summarize the boy's emotional changes between 6:00 and 7:00.
options:
(A) Anger - Fear- Surprise and happiness
(B) Sadness Excitement Helplessness
(C) Disappointment Let down Sorrow
(D) Loneliness- - Isolation -Solitude
(E) Sadness Grief Mourning
(F) Sadness Shock Surprise and happiness
(G) Disappointment Astonishment Stupefaction
(H) Loneliness Isolation Sorrow
(I) Disappointment Excitement Helplessness
correctAnswer: (F)
Video: CGI Animated Short Film: "Crunch" by Gof Animation CGMeetup video_url
I11.2. IMPLICIT EMOTION
Task Description:
Analyze the emotions of characters in the video. Implicit emotions usually need be analyzed indirectly through the environment, style, etc.
Example Question:
Question: What kind of emotional atmosphere does the stage lighting create?
options:
(A) Solemn and sorrowful atmosphere.
(B) Neutral and unemotional atmosphere.
(C) Intense and dramatic atmosphere.
(D) Joyful and festive atmosphere.
(E) Sadness and loss.
Confident and empowering atmosphere.
(G) Chaotic and confusing atmosphere.
(H) Calm and serene atmosphere.
(1) Playful and whimsical atmosphere.
(J) Romantic and loving atmosphere.
CorrectAnswer. (E)
Video: Stages of Grief- AVANTGARDE SHOW 2023 video url
111.3. META-EMOTION
Task Description: This part refers to the high-level emotions in the video, such as the emotions expressed by the author through video, and emotions expressed by the entire video. Example Question:
Question: Summarize the meaning of this short film in one word.
Options: [
(A) Creation
(B) Transformation
(C) Stress
(D) Mutation
(E) Metamorphosis
(F) Growth
(G) Rebirth"
22

---

Published as a conference paper at ICLR 2026
(H) Destruction
(I) Erosion
(J) Development
(K) Isolation
(L) Conversion
CorrectAnswer: (C)
Stress Shortfilm video_url
111.4. AUDIENCE EMOTION
Task Description: Analyze the emotions that viewers are most likely to feel after watching the video.
This is more advanced and relatively easy for humans to sense. Including the perception of humor. Example Question:
Question: What are the reasons for the high number of views on this video?
Options:
(A) The video features a well-known celebrity who has large fan base, drawing a lot of attention.
(B) The dance style is extremely unique and has never been seen before, sparking curiosity.
(C) People are under lot of stress and need videos that can help them unwind.
(D) The background music is a popular hit song that many people recognize and enjoy.
(E) The video released during major holiday season when people more likely to watch videos.
The choreography incredibly complex and impressive, showcasing the dancers' skills.
(G) The video has a strong and inspiring message that resonates with a wide audience.
(H) The video was featured on popular TV show or news segment, driving more views.
(I) The video was shared by a large number of dance schools and communities, spreading its
reach.
(J) The video was part of viral challenge that encouraged people share it.
(K) The video has high-quality production values that make stand out from other content. CorrectAnswer: "(C)
Satisfying and Relaxing Kinetic Sand ASMR shorts video url
IV. COMMENT MATCHING (CM)
IV.1. HUMOROUS
Task Description:
The video will spark laughter because of certain comments, making the audience feel funny, testing whether the model can match it correctly.
Example Question:
Question: Based on this video, which of the following comments likely to make people laugh?
Options:
(A) Did he just audition for a water ballet?
(B) How many fish does take catch a man?
(C) Is there a Walmart beneath the river?
(D) The fish are holding a grudge, watch out!
(E) Now that's what call splash of creativity.
(F) came for the fishing tips and stayed for the synchronized swimming.
(G) That water has more personality than my neighbor!
(H) I'm starting to think he's part fish.
(1) I think the fish caught him instead.
(J) That's definitely a land fish champion.
(K) That fish will never trust humans again.
CosrectAnswer: "(C)", He DI Lao video url
23

---

Published as a conference paper at ICLR 2026
IV.2. THOUGHT-PROVOKIIE
Task Description: Some comments under the video will enhance people's thinking and test whether the model can accurately understand.
Example Question:
Which of the following statements can better explain the social reality expressed in this animation?
Question:
Options:
(A) The animation showcases an idealized view of advancement within corporate ladder.
(B) The depiction highlights the dehumanization and mechanization of individuals in powerful social system.
(C) It portrays the joy of discovering one's 's true passions through societal pressures.
(D) The scenes show a man achieving happiness through daily routine.
(E) It represents personal ambition and the drive for success in individual careers.
(F) The animation indicates the triumph of an individual's s spirit the face of adversity.
(G) It reflects the disintegration of traditional family roles.
(H) The animation shows the importance of family support in work-life balance.
(I) It emphasizes the challenge of maintaining personal identity in urban settings.
(J) We are all working for others without realizing it due to our own needs.
(K) The animation illustrates the struggle with contemporary health issues.
CorrectAnswer. (J)
EL EMPLEO video url
IV.3. TRENDING
Task Description: is relatively difficult test whether the model can accurately infer and analyze the most popular comments under the video.
Example Question:
Question: Which of the following comments best summarizes the content conveyed by this film?
Options:
(A) Material possessions define one's s value.
(B) Selfless lead rewards that surpass material wealth.
(C) Loneliness is a desirable state.
(D) Personal gains are the ultimate goal of helping others.
(E) Isolation is the path to personal growth.
(F) True happiness found through wealth accumulation.
(G) Success comes from competitive behavior.
(H) Sharing leads to financial prosperity.
(1) He receives what money can't buy.
(J) Adversity breeds stronger individuals.
CorrectAnswer: (I)
Unsung Hero video_um __rl
V. IMPLICIT SYMBOL (IS)
V.I. CULTURAL SYMBOLS
Task Description: Test whether the model can infer and analyze the cultural characteristics hidden under the surface visual elements of the video (such as nationality, festivals, customs, religion, etc.). Example Question:
Question: The plaque inscribed with "Dominating Three Continents" that appears in the video most likely to be found in the architecture of which of the following religions?
Options:
(A) Taoism
24

---

Published as a conference paper at ICLR 2026
(B) Shinto
(C) Sikhism
(D) Judaism
(E) Islam
(F) Christianity
(G) Buddhism
(H) Hinduism
(I) Jainism
(J) Zoroastrianism
Cosre:tAnswer: (G)
[4K] Hangzhou 2024 in the misty rain West Lake, Lingyin Temple, Night walking in
Hefang Street video_vr url
V.2. ART SYMBOLS
Task Description:
Test whether the model can infer and analyze the art-related characteristics hidden under the surface visual elements of the video (such as dance style, artistic skills, imitation, etc.). Example Question:
Question: What is the shadow that appears in our view at 1:40 imitating?
Options:
(A) The shadow is imitating pole dancer.
(B) The shadow is imitating a person washing a dog.
(C) The shadow is imitating person brushing their hair.
(D) The shadow is imitating someone playing a violin.
people engaged in conversation. (F) The shadow is imitating someone painting a wall.
(E) The shadow is imitating
(G) The shadow is imitating person feeding horse.
(H) The shadow is imitating a person washing their car.
(I) The shadow is imitating a dog barking at person.
(J) The shadow is imitating someone performing magic trick.
(K) The shadow is imitating person holding an umbrella.
(L) The shadow is imitating someone walking large dog.
CosrectAnswer: (A)
LEAKED! Hilarious Shadow Puppets AGT 2023 Early Release video url
V.3. OTHER SYMBOLS
Task Description: Test whether the model can infer and analyze other special symbols (such as commercial advertisements, etc.) hidden under the surface visual elements the video. Example Question:
Question: "What do you think the chimpanzee that appears multiple times in the film symbol-
izes?".
Options:
(A) The chimpanzee symbolizes chaos and disruption in everyday life. (B) The chimpanzee symbolizes a childhood fear. (C) The chimpanzee symbolizes technology invading personal space. (D) The chimpanzee symbolizes the unpredictability of fate. (E) The chimpanzee symbolizes glue company. (F) The chimpanzee symbolizes lost opportunities. (G) The chimpanzee symbolizes an obsession with social status. (H) The chimpanzee symbolizes environmental degradation. (I) The chimpanzee symbolizes the desire for freedom. (J) The chimpanzee symbolizes misunderstanding between people. (K) The chimpanzee symbolizes reliability and trust in friendships.
CorrectAnswer: (E)
All Gorilla glue ads video_urf url
25

---

Published as a conference paper at ICLR 2026
E.2 EXPLICIT REASONING TASKS
1. CAUSAL REASONING (CAR)
I.1. FORWARD REASONING
Task Description: Forward reasoning can also be understood as the prediction of future events, including prediction of outcomes, prediction of content that has not yet appeared, etc. Example Question:
Question: What is the speculated ending of the film? Options:
(A) The movie concludes with unexpected twist where the flowers reveal hidden secret.
(B) The ending is cliffhanger, leaving the audience uncertain about the characters' fate.
(C) Her boyfriend passed away due to illness, leaving the girl devastated with grief.
(D) The film wraps up with joyous family reunion.
(E) The film ends with a dramatic breakup as one character leaves with heavy heart.
(F) The movie concludes with comedic mishap involving the flowers.
(G) The ending shows tragic farewell as one character moves a new city.
(H) The film ends with the revelation of long-lost sibling.
(I) The story concludes with the characters embarking on spontaneous road trip.
(J) The film ends melancholic note, reflecting on lost opportunities.
(K) The video closes with a heartwarming reconciliation between the main characters after exchanging heartfelt notes and gestures.
porrectAnswer (C)
For Milo AWARD WINNING Minute Short film (2020) video_url _ur
1.2. BACKWARD REASONING
Task Description: Backward reasoning means finding the cause from the effect and inferring the reason why event occurred.
Example Question:
Question: Why was the elderly black man warned by security the beginning of the film? Options:
(A) Mobile phones are not allowed for recording during magic shows.
(B) He was trying to sell unauthorized merchandise.
(C) He was recognized as a local celebrity causing disruptions.
(D) He was accused of stealing a bicycle.
(E) He was creating loud music disturbing the peace.
(F) He was believed to have lost his entrance ticket.
(G) He was inadvertently blocking the pathway.
(H) He was associated with another person causing trouble nearby.
(I) He was engaged in card tricks that security found suspicious.
(J) He was loitering without a purpose.
CorrectAnswer (A)
Now You See Me Official Opening Scene (2013) Mark Ruffalo, Morgan Freeman Movie HD video url
II. SEQUENTIAL. STRUCTURE REASONING (SSR)
I. NARRATIVE STRUCTURE
Task Description: Reasoning and analyzing the narrative order of the entire video, including the editing order, such as sequential, flashback, and interpolation. Example Question:
26

---

Published as a conference paper at ICLR 2026
Question: What kind narrative sequence does the film employ?
Options:
(A) non-linear flashback sequence, where events are shown out of chronological order, often
revealing backstory
(B) parallel overlapping sequences, showing multiple storylines happening simultancously with
some overlap
(C) cyclical narrative structure, repeating events or themes in circular pattern
(D) linear narrative sequence, following straightforward progression from beginning to end
(E) random jumps in the timeline, moving unpredictably between different points in time
(F) interwoven thematic structure, weaving together different themes and ideas throughout the story
(G) reverse chronological order, starting with the end and moving backwards time
(H) fragmented narrative, presenting the story in disjointed broken segments
(I) begins with a flashback and then proceeds chronological order
(J) episodic progression, advancing the story through series of distinct episodes or chapters
(K) multi-perspective narrative, telling the story from multiple characters' points of view
CorrectAnswer: (I) Identity SHORT FILM (Award Winning Inspirational Short) video_url
11.2. CORE CONNECTING ELEMENTS
Task Description: Videos with this type of question usually have a key connecting element that runs through the entire video. It is carefully designed by the producer and tests the model's inductive reasoning of the visual information of the entire video. Example Question:
Question: What the recurring element in the video, summarized in one word?
Options:
(A) Pareidolia
(B) Smile
(C) Alarm
(D) Work
(E) Mirror
(F) Mundane
(G) Routine
(H) Suit
(I) Coffee
(J) Sleep
(K) Bedroom
(L.) Portrait
Answe::Answer: (B)
PAREIDOLIA Minute Short Film Award Winning video url
IL.3. INFERENCE ON EDITING TECHNIQUES
Task Description: These tasks evaluate the models' deep analysis and multimodal reasoning about video editing strategies. Example Question:
"Please guess how many videos needed record the moment the man punched the punctured water ball at the beginning of the video?".
Question:
Options:
(A) At least two separate takes would be needed.
(B) At least one single take needed.
(C) Three separate takes are needed.
(D) Four separate takes are needed.
(E) Each scene can be captured in single continuous take.
27

---

Published as a conference paper at ICLR 2026
(F) Five separate takes are needed.
(G) Six separate takes needed.
(H) Eight separate takes are needed.
(I) Ten separate takes required.
(J) Twenty separate takes are necessary.
(K) At least separate takes are needed.
PorrectAnswer (C)
Playing With Time- video url
Note: The reasoning and analysis process of this question can refer to this disassembly video.
11.4. HALLUCINATION
Task Description:
Evaluate whether the model perceives various types of hallucinations when perceiving video content.
Example Question:
Question: How many dancers are there the video?
Options:
(A)
(B)
(C) 2
(D) 3
(E) 4
5
(G) 6
7
(I) 8
(J)
(K) options before are all false
CorrectAnswer: (B) Rat dance with falling body parts video_ur url
III. COUNTERINTUITIVE REASONING (CIR)
III.1 MAGIC DECONSTRUCTION OR SPECIAL EFFECTS EDITING
Task Description:
This type of video usually creates impossible magical effects, but magic tricks, and some are editing and special effects, which require deeply reasoning. Example Question:
Starting at 4:35, how did the man achieve this magical effect in the magic trick? Options: (A) Sleight of hand technique with hidden ring, using dexterity to conceal and reveal
Question:
the ring.
(B) Utilizing mirror to confuse the audience, creating optical illusions through reflection.
(C) A distraction technique with smoke bomb, diverting attention with sudden burst of smoke.
(D) A special ring that retracts into a fake thumb, using a concealed mechanism to make the ring disappear.
(E) Using a magnet hidden in the sleeve, manipulating objects with magnetic force.
A camera trick with video editing, altering footage create the illusion of magic.
(G) Sleight of hand technique with hidden string, using concealed thread to control objects.
(H) The bottle inside the paper bag had already been altered leave only the outer plastic skin.
(I) Employing twin assistant to swap the ring, using look-alike to deceive the audience.
()) The use of invisible thread, employing nearly undetectable line to move objects.
(K) A sound cue to mislead the audience's attention, using noise to distract from the real action.
CosrectAnswer: (H)
Level to 100 Magic Tricks Anyone Can Do video_uri
28

---

Published as a conference paper at ICLR 2026
II1.2. ARTISTIC TECHNIQUES
Task Description:
This type of video usually creates impossible scenes, but it is usually artistic expression deliherately designed by the author.
Example Question:
Question: Why is the shadow on the boy's s face illuminated by sunlight at 1:06?
Options: (A) Because the boy moves a position where a strong light source directly above him, not related to the girl.
(B) It's just a coincidence that the angle of the sun changes suddenly at that moment, and has nothing do with any special meaning.
(C) The sunlight illuminates the shadow because the cameraman adjusts the lighting equipment create better visual effect.
(D) The girl's appearance brings good luck, and the sunlight representing good fortune clears away the gloom of bad luck in his world.
(E) This is because the boy has walked into a neighborhood with better weather and climate.
(F) The sunlight lights up the shadow because there hidden light emitting device in the scene that is turned on at 1:06.
(G) It's result of the special lens filter used during filming, which makes the shadow on the boy's s face appear to be lit by sunlight.
(H) Because the boy didn't get hurt after falling and his mood improved, the sunlight is used to represent his improved mood.
CoswectAnswer: (D)
CGI Animated Short Film HD "Jinxy Jenkins & Lucky Lou" by Mike Bidinger & Michelle
Kwon CGMeetup video url
III.3. HUMOR AND EXAGGERATION
Task Description: A common technique in humorous videos is to exaggerated expressions that seem unreasonable, but there are some clues to understand the meaning. This type of question tests the model's ability reason about exaggerations and unusual techniques. Example Question:
Question: Why does the first half of the scene look sunny but also show rain?
Options:
(A) It is sunshower, when rain falls while the sun is shining.
(B) The character is dreaming of being both wet and warm.
(C) There are rainclouds directly above while sunlight comes from the side.
(D) It is snow instead of rain, reflecting the sunlight.
(E) The effect is caused by morning fog and light refraction.
(F) It's visual illusion caused by mist.
(G) The character moved to a different location quickly.
(H) A rainbow is forming which intensifies the sunlight.
(I) Dew drops from trees reflect sunlight.
(J) There are two unrelated weather animations merged together.
(K) The man wet the bed, which caused the presence of water in his dream.
CorrectAnswer: (K)
It now makes sense video_ur
IV. CROSS-MODAL. TRANSFER REASONING (CTR)
Evaluate the ability transfer reasoning from video to text, audio, video image (for example, video-to-text: the theme of video may have the same meaning as a famous quote) Task Description: Evaluate the ability transfer reasoning from video to text (for example, the theme of video may have the same meaning as famous quote) Example Question:
29

---

Published as a conference paper at ICLR 2026
Question:
Which the following proverbs best explains the theme of this short film?
Options:
(A) When one door closes, another opens.
(B) Opportunity knocks only once.
(C) Time heals all wounds.
(D) The early bird catches the worm.
(E) Never judge book by its cover.
All that glitters
gold.
(G) The grass is always greener on the other side.
(H) Actions speak louder than words.
(I) A stitch in time saves nine.
(J) Beauty in the eye of the beholder.
(K) Absence makes the heart grow fonder.
(L) penny saved is penny earned.
CoswectAnswer: (E)
Video: Award Winning SHORT FILMS Don't Judge BATTI Hindi Heart Touching Short Movies Content Ka Keeda video_ur!
V. VIDEO TYPE AND INTENT (VTI)
V.I. VIDEO TYPE
Task Description: Evaluate the model's ability analyze video types, such as commercials, science fiction films, comedies, etc.
Example Question:
Question: What type of video is this most likely to be?
Options:
(A) A documentary about airplane technology
(B) Advertisement for an ice-cream
(C) A drama set on an airplane
(D) A comedy film featuring an airline
(E) in-flight safety demonstration video
(F) A travel vlog featuring aerial views
(G) A science fiction movie on spaceship
(H) This is advertisement.
(I) video tour of an airplane factory
(J) A virtual reality experience of flying
(K) A news segment on turbulence incidents
Csswer (H)
Leo Messi vs Kobe Bryant- Legends on Board Turkish Airlines video_ url
V.2. VIDEO INTENT
Task Deseription:
Reasoning and analyzing the purpose and production intention video (e.g.
what kind of product performance is promoted in commercial advertisement, etc.)
Example Question:
Question: Which year do you think this video most likely released?
Options:
(A) 2018
(B) 2017
(C) 2016
(D) 2015
(E) 2023
(F) 2019
(G) 2023
(H) 2020
30

---

Published as a conference paper at ICLR 2026
(1) 2014
(J) 2013
CosrectAnswer: (H)
Lockdown One Minute Short Film Challenge Film Riot video url
F EVALUATION DETAILS
F.1 BASELINES
The baselines include closed-source models: (1) GPT series: GPT-4o (OpenAI, 2024a), GPT- 4o-mini (OpenAI, 2024b), GPT-4.1 (OpenAl, 2025), and GPT-5 (OpenAl, 2025a); (2) Gemini (Reid et al., 2024), and Gemini-2.5 (Anthropic, 2024); (4) o3, 04- mini (OpenAI,, 2025b); open- models: (1) Qwen series: Qwen2.5-V2 (7B/72B-Instrr)
series: Gemini-2.0-flash.
(flash/pro) (Google DeepMind, 2025); (3) -aainde----SSome-22222444
(Yang et al., 2024a); (2) Gemma series: Gemma-3 (12B/27B) (Kamath et al., 2025); (3) Intern series: Intern3-VL. (8B/38B) (Zhu et al., 2025); (4) LLLava series: LLLava-Oneviiion778 (Li et al., 2024a), Video-LLava-7B (Lin et al., 2023): (5) Phi-4-multimdaal I (Abouelenin et al., 2025); (6) Other video models: Dogvlm2-videe-llamaa
(Hong et al., 2024), NVILA-8B-VII (Liu al., 2024b). All local experiments are conducted on 4xA 100 80GB GPUs.
F.2 FRAME SELECTION
We followed the official configurations models that support multi-image input, as well as settings in previous works (Fu et al., 2024; Wang et al., 2024b), define the number of input frames for each model. Specifically, we fixed the number of frames per model and sampled them evenly across the video duration. We sampled 8 frames for LLaVA-One Vision, Video-LLaVA, and NVILA-8B- Video, thii4-multimodal-innttte
16 frames were sampled for Qwen2.55VL-777
twwen.5--mmii-i 21.222000FF---hhiiie
CogVLM2- CogVLM2-VVdee-LLLMM Video-LLaMA3-CCie 32 frames were sampled for Qwen2.5-VL-722h Intern InternVL-38B. Gemma-3-it-277B, GPT-4o-Mini, 04-mini, o3, GPT-5, Exceptionally, since 1omiini2.00-Faaht supports long video and multimodal context inputs, we sampled one frame per second across each video, with a maximum cap of 512 frames to ensure API stability. For Gemini-2.5-pre we also tested the official setting of Ifps sampling frequency. Additionally, to enable a fair comparison with Gemini-2 2.0--2200Flh mash-Thinking, we also tested version of Gemini-2.0-Flasn with 16 frames.
Intern VL-8B, Gemma-3-it 3-it--1-BB--BB- and
and Claude-3.5-Somnet; 300 frames for GPT-4o.
G FURTHER RESULTS
G.1 IMPACT OF VIDEO LENGTH ON MODEL PERFORMANCE
To provide a more comprehensive analysis, we evaluated model performance on videos of varying lengths. Following the categorization standards used prior work such as Video-MME (Fu et al., 2024) and based on the video length distribution of MMR-V, we divided the dataset into three groups: Short (< 2 minutes), Medium (2-6 minutes), and Long (> minutes). These categories comprise 329, 593, and 335 tasks, respectively. The detailed evaluation results presented in Table We observe general trend where the performance of most models decreases as the video length increases. However, it is worth noting that this performance gap becomes less pronounced for models equipped with long-context capabilities. For instance, Gemini-2.0 2-0ssh (processing 512 frames) and Gemini-2.5-PPo (using fps sampling) demonstrate remarkable stability across all duration categories.
G.2 BENEFIT OF AUDIO INPUT
In this section, we illustrate through several examples how incorporating audio as an additional modality provides extra reasoning cues that enhance video reasoning. In the following examples, green text highlights the model's analysis audio cues, which contributes to reaching the final answer.
31

---

Published as a conference paper at ICLR 2026
Table 7: Evaluation results (%) across different video length. Bold and underlined values indicate the best performance among proprietary and open-source models, respectively.
Video Length
Model
Overall
Short Video
Medium Video
Long Video
Open-source models
LLaVA-Vides
18.4
20.5
16.7
10.4
NVILA-&B-Vider
25.5
27.1
25.9
22.6
Phi-4-mullimoddal-nsttruuu
26.7
30.1
24.5
27.2
CogvIm2-video-llama3
25.6
27.4
25,9
22.6
Qwen2.5-VL-7B
30.1
34.7
28.0
29.3
InternV1.3-8B
33.6
34.9
33.0
32.8
Gemma-3-12b-it
34.0
35.7
33.5
31.1
InternLL2.5-388
39.9
43.9
39.1
38.2
Qwen2.5-VLL-72B
39.1
43.8
40.4
34.5
Gemma-3-27b-it
42.0
43.8
42.0
40.3
Proprietary models
GPT-4o-mmii-22044-07-11
34.8
39.2
34.1
31.6
Gemini-2.0-Flash (16 frames)
42.6
46.2
42.3
39.7
Chudee3---Sonmee222241022
43.3
47.7
43.5
38.5
Gemmin-2..-Flaas---hiiinnn
45.0
48.9
45.0
40.9
GPT-4.1-2025004-14
46.6
49.9
46.2
44.2
Gemini-2.0-Flash (512 frames)
48.8
48.9
48.6
49.0
Gemini-2.5-Flash
51.2
54.1
51.9
47.2
04-mimi-20255004116
52.5
57.1
53.0
47.2
GPT-44-2024-11-20
52.8
57.4
53.1
48.1
03-2025-04-16
59.1
62.9
59.7
54.3
GPT-5-2025-08-07
60.9
62.3
61.6
56.2
Gemini-2.5-pro (1fps)
64.3
64.7
64.1
63.9
In the first example (Figure 10), audio reveals that the library is actually noisy. When the protagonist puts on headphones, the video turns silent. Without audio, the model might mistakenly assume the library is quiet based solely visuals. The original video link is: Spec commercial SONY.
In the second example (Figure 11) the audio can better understand that 'parachute' and 'pair of shoes' homophones. The original video link is: Hasta la vista.
Moreover, audio can supply cues from other perspectives. For instance, background music can convey the overall mood of video, while magic performance videos often include voice narration.
32

---

Published as a conference paper at ICLR 2026
Benefit of Additional Audio Input
Question:
Which of the following statements can better summarize the core message of this video?
Options:
The video emphasizes importance of athletic competition.
portrays medical laboratory's dedication to scientific experiments.
(C) The primary fecus of the video is the pursuit of knowledge quiet library setting.
(D) It depicts cafe's lively social interactions its core message.
(E) The main theme the artistic creation studio space.
(F) The main theme is the chaos and bustle of city?
(G) The video is centered around Teamwork in a corporate office.
(H) It captures the essence of outdoor adventure and exploration.
I highlights museum' exploration and discovery process.
Find your focus.
Gemini-2.0 www...0--aa--hh----aaaa The user wants to identify the care message the video. need to analyze the video content and compare the given options. The video takes place entirely in a library. People are shown studying. There are several instances where the librarian shushes people, sounds of distraction are highlighted (like chewing gum or flipping pages loudly), which ane then silenced, The last frame has text, but it's not visible the provided images. Let's evaluate the ...,n((C) The primary focus of the video is the pursuit of knowledge quiet library setting.' This fits the visuals of the library, but the sound the first half of the video directly captures the recurring theme of individuals trying to concentrate amidst distractions in study environment. cues (shushing, noise being amplified then silenced, people trying to study) strongly support this. Vn'nFinal Answer: [[J]]
quiet Incorrect. ... (J) 'Find your focus.' This option
the video, students studying doing activities in the library. The whole library environment looks very quiet. Some people are looking through materials, some are looking for books they need, a librarian is using gestures to maintain quiet environment. Some people are wearing headphones and concentrating on studying. Let's analyze the options: ...(C)' The primary focus of the video is the pursuit of knowledge quiet library setting. This aligns well with the visual of library, people studying, and the emphasis on quiet. This correct. ... \nChoice (C) more appropriate because the scene is quiet library and the protagonist seems to be studying. \nAnFinal Answer: [[C]]]**\n) VnAn
Gemini-2.0- www...-0FFah-wihhhrrre
Figure 10: Benefit of Additional Audio Input. The green text highlights the analysis of audio cues.
33

---

Published as a conference paper at ICLR 2026
Benefît of Additional Audio Input
Hasta la
Hasta la vistal
sueft you said shoes
Witero
parachuts?
Question: What the primary humorous techniiqe used in this videa?
Options:
(A) Physical comedy due to action.
(B) Slepstick humar involving objects.
( Wordplay through
(D) The humor arises from the conflict and contradictions,
(E) Satire on travel and safety.
(F) Dark humor on life's choices.
(6) Misleading humor through onimation style.
(H) Parody action film scenes.
(I) Surreal humor through character design.
(J) Hyperbolic humor adventure.
(K) The humor derived from exaggerated expressions.
Gemini-2.0 www.-0-aaaah--hhhhhh. audio: The video features two characters jumping from plane. One character asks, 'Where's the parachute?' and the other replies, 'I thought you said 'pair of shoes'." This exchange is classic example of pun, where a word or phrase (parachute) deliberately misinterpreted as onother word phrase (pair of shoes) that sounds similar but has different meaning, leading to a humorous and unexpected situation. This reliance on the double meaning words the similarity in sound is the essence of wordplay.\nvn[[C]
Gemini-2 2.0-- ---llsh-withuut- audio: The humor in the video primarily relies on the dialogue and the misunderstanding between the two characters, leading to humorous situation. The joke at the end with the building is also reliant on this misunderstanding, Thhhhfeee,,,, the primary humorous technique used this video the conflict and contradictions in the dialogue. An\n[[[]]]
Figure 11: Benefit of Additional Audio Input. The green text highlights the analysis of audio cues.
34

---

Published as a conference paper at ICLR 2026
H DETAILS OF COT ANALYSIS EXPERIMENT
This section presents the CoT analysis experiments discussed Section 4.5. We use a representative model CoT illustrate 4 categories of analysis in Figure 12. Specifically, Text Analysis refers to the examination of textual information such as the question and options; Video Analysis focuses on the content of the video; Question Frame targets the specific frame referenced in the question for instance, the frame where the magician controls two flames; and Other Frame pertains to frames outside the scope of the question. In Figure 12, yellow, red, and blue represent text, question frame, and other frame analysis respectively. Red, blue, and green all represent video analysis.
Table 8: CoT analysis prompt.
Prompt for CoT Annotation
You will be given model's textual reply to video-hased question along with the video frames. Your task is determine four boolean labels for each chunk of the reply:
1. 'other frame desc': Does this chunk describe visual information from frames other than question frame?
2. "question frame desc': Does this chunk correctly describe visual information from question frame specified
in the question?
3. "video analysis": Does this chunk perform analysis the video content?
4. 'text analysis': Does this chunk perform analysis of the text (e.g.. question text, options) rather than visual content
'The question frame refers to the specific frame(s) referenced by the question prompt.
Other-frame descriptions are visual details not present in the question frame but from other frames.
Video analysis includes describing trends, motions, or visual inference beyond plain description.
Text analysis includes reasoming over question text, options, external text context. Respond strictly in JSON:
"other frame dese": true or false, "question frame desc": true or false,
"video analysis": true or false.
"text analysis": true or false
Question: (question)
Reply Chunk: [chunk]
Whole CoT Reply: (CoT)
35

---

Published as a conference paper at ICLR 2026
CoT Analysis Example
Question Frome
Other Frome
Other Frome
Question: How does man the beginning video pick up and casually
control the flame on the lighter?
Options:
(A) He used a holographic projector to simulate the flame.
(B) He used special flome-retardont chemical on his hand to create the illusion.
(C) He possessed innate immunity to fire.
(D) He practiced yaga meditation withstand any flame heat.
(E) quick extinguishing spray was applied that halted the flame.
(F) He surrounded the flome with an invisible film.
(G) He mostered the art of fire manipulation.
(H) The flame was made of non-flammable gas.
applied hidden cooling technology under his sleeve,
flome was actually an LED light.
hidden lighter in his hand, sleight of hand trick.
My thinking process arrive at the answer is follows:An\vn "*aannn""""ddiaiia **Understand the Question:**
The question is specifically asking about "how" the man at the beginning of the video picks up and *cosually controls* the flame on lighter. The key words are \"how\" and V l'casually controls, X" implying method or technique rather than supernatural ability or complex technology. AnAn2. ** Analyze the Vi noticed that the \u201cfloatingg Au201d flame always stayed just above his palm in a way that perfectly hid the source, and that when he distracted us he slipped a small butone lighter into position. In the close)u2010us reveal later, you can how the second lighter remains concealed until he produces the flame on his fingers. It\u2019s all done by classic palm concealment and sleight of hand with hidden lighter. \n\n2. ** Analyyze the Options:** I reviewed each option to see which one is most plausible and aligns with the context of a magic trick video. I considered the practicality and likelihood of each explanation. AnAn **(A) Holographic Projector, (J) LED Light, (H) Non-flammable gas:** These options suggest the flame isn't 't real. While possible for illusions, they are less likely for *casual* control and might be visually detectable as fake in a video focusing on close-up magic.An\n
**(C) Innate immunity
to fire, (D) Yoga meditation, (6) Mastered fire manipulation:** These options fantastical mystical and don't t explain a practical method. They are less likely to be the intended explanation in video about magic "tricks*, AnAn **Prioritize Simplicity and Plausibility:
the context of magic tricks, simpler explanations are often better. Option (K) sleiggh of hand - is the simplest and most direct explanation that fits the description of \'casual control. It's well- known method for creating illusions with small objects like lighters. AnAn5. **Eiminaae Less Likely Options:** eliminated the options that were:\n
Too complex technologically advonced (A, I)An Unrealistic or fantastical (C, D, G, H).An Less direct or slightly less probable for V'casual controlX" in close-up magic context companed to sleight of hand (B, F, AnAn6. **Select the Best Fit:** Option (K) V"A hidden lighter in his hand, a sleight of hand trick\" emerges the most likely and best-fitting answer becouse it is:An
common and effective Consistent with the ideo \'casual control. \'nn * Plausible the context of magic trick video. Andeo.nnAnNnTT NhhnnnTeer,eore, based on the question, the options, and the likely context of a magic trick video, option (K) is the most accurate onswer. Final Answer:** [[K]]
magic technique, An
simple and direct explanation. An
Figure 12:
CoT example of experiments in Section 4.5. Yellow, red, and blue represent text, question frame, and other frame analysis respectively. Red, blue, and green all represent video analysis.
36

---

Published as a conference paper at ICLR 2026
I EVALUATION PROMPT
We evaluated two settings the main experiment, zero shot and zero shot + CoT. The prompts used follows.
Table 9: Evaluation prompt for the Zero-Shot Setting.
Prompt for Zero-Shot Setting
[[INSTRUCTION]]
Please select the best answer to following multiple-choice question based on the video.
Only one option is the most accurate answer in relation to the question and the video. What is the correct answer to this question
[Question]
Options:
[Options]
[[END OF INSTRUCTION]]
[[OUTPUT FORMAT]]
Format your answer follows:
Please directly output the answer letter without thinking and explanation.
the correct option letters (A, B, C, D...) ) for the multiple-choice question X, give the final correct option
number in the following format: "[[X]]"
[[END OF OUTPUT FORMAT]]
Table 10: Evaluation prompt for the Zero-Shot + CoT Setting.
Prompt for Zero-Shot + CoT Setting
[[INSTRUCTION]]
Please select the best answer to following multiplc-choiee question based on the vidco.
Only one option is the most accurate answer in relation to the question and the vidco.
What is the correct answer to this Question:
Question]
Options:
{Options]
Let's think step by step.
[[END OF INSTRUCTION]]
[[OUTPUT FORMAT]]
Format your answer follows:
Your thinking process.
the correct option letters (A, B, C, D...) for the multiple-choice question X, give the final correct option
number in the following format: "[[X]]"
[[END OF OUTPUT FORMAT]]
J CASE STUDY
In this section, present reasoning processes and results from selected models the MMR-V V benchmark. Through these case studies, we aim to better illustrate the current shortcomings of models in multimodal reasoning tasks and provide insights that may inspire future research and advancements in this area.
Firstly, there is comparison between good CoT and poor CoT in 13. Yellow highlights indicate text-based analysis, while green highlights denote video-based analysis. As shown, the upper two CoT's engage in deep multimodal reasoning grounded in the video, ultimately reaching the correct answer. In contrast, the bottom CoT relies mainly on textual reasoning with only shallow video perception, leading to an incorrect outcome.
37

---

Published as a conference paper at ICLR 2026
Comparative Case Study of MCoTs
Question: What does object being chased by the people refer
Options:
(A) Difficulties in life.
(9) His fully automatic house that can walk autonomously.
(C) The house that this person is saving money to buy.
The old house that the family decided to renovate.
(E) The villa the man bought a vacation home.
The monsion that the celebrity owns in the city center.
The cottage that the couple wonts rent for their honeymoon.
(H) The enchanted house.
The animation depicts people in relentless pursuit of walking house- house-quite literally chasing after "home." Throughout the film, the house slips away through and snow, over seas across canyons, never within reach. In final scene, it stands on the opposite side of a vast divide, facing crowd of people separated from a cluster of buildings. The imagery serves metaphor for the unending struggle to afford and finally secure home. The pursued house symbolizes the one people dream of saving enough to buy.Wn'r Vn'n [[C]]
GPT-5- CoT:
The animated short shows a small house with legs and arms sprinting away, while more and more people join in the chase, Nothing in the story suggests that onyone already owns the house, or that it is just futuristic "automatic" home. Instead, comes across something everyone desperately wants but connot yet have, The running house clearly represents the dream owning home that ordinary people are always pursuing-thor place they hope to buy someday. In this sense, the object being chased is best understood as the house that person, and by extension everyone saving money to purchase. AnAn [[C]]
03-CoT:
Thinking Process:** In'nl, **Identify the *uestion:** The question asks about the meaning of \'the object being chased by the Prople\".Annne W*tth the video and observe the object:** The video clearly depicts house with legs walking and being chased by people. An\n3, **Analyze the options and compare them to the video:*A\n x **(A) Difficulties in life:** While metophorical interpretations are possible, the video literally shows house being chased. This option too abstract and doesn't 't directly relate to the visual content. An **(B) His fully automatic house that can walk autonomously:** This option directly describes what is seen in the video \u2013 house that walk its This strong contender. An * **() The that this person saving money to buy:** There no visual or norrative element in the short video clip about saving money or buying a house. This option not supported by the video.An ..... **(F) The monsion that the celebrity owns in the city center:** No celebrity, mansion, city center is depicted. This option is irrelevant An **(G) The cottage that the couple wonts rent for their honeymoon:** There is no mention of couple, honeymoon, cottage rental the video clip. This option irrelevant.An An **(H) The enchanted house:** While the walking house could be considered Veenchanted, \" option (B) more precise and descriptive of the "mechanism" of the house's 's unusual behavior (autonomously walking). A"Enchanted'" is more general and less accurate description compared to \'fully automatic house that con walk autonomously.. A \nn4,, **Select the best option:** Option (B) is the most direct, accurate, and visually supported answer based on the video. literally describes the object being chased, Final Atswer=*"[Vn[[[3]
Gemini-2 2, O-Flas----aa-- D--Faah-TTinnin---CCa- Thinking-CoT:
A comparison of CoT's from two models the same task. Yellow and green indicate text and video analysis, respectively. As shown, GPT-5 and o3's reasoning paradigm demonstrates a deeper analysis of the video content.
Figure 13:
38

---

Published as a conference paper at ICLR 2026
Lack of Visual Reasoning
Question: Why did the broken umbrella in the bay's hand, damaged by the wind, suddenly get fixed while the boy and the girl were the air?
Options:
(A) Because the boy initially used too much force, causing the umbrella to be blown apart, and it was repaired after he
adjusted his effort.
(B) Because the boy held the girl's hund, gained good luck, and as result, the umbrella was restored.
(C) Because the wind in the air happened to die down, the umbrella returned to normal.
Because girl held the boy's hand, making him extremely happy, which tumn allowed the umbrella to
repaired smoothly.
It's visual illusion
video, and the umbrella wasn't actually blown apart first place.
(F) The umbrella designed to be self repairing model, and it activated repair function automatically.
(G) was pre arrunged mugic trick, and the umbrella was never truly damaged.
(H) The video was reversed during that pari, making look like the broken unbrella was getting fixed.
(l) Acrodynamics, changes in wind strength and direction contributed to the umbrella being repaired.
(J) was a computer - generated special effect udded during post production to create magical look."
Video Type: Animation
Task Type Explicit Reasoning.
Task Type Counterintuitive Reasoning
Task Type L3: Artistic Techniques
[[Thinking Paocess][aTThe user asking for the best answer to the question ('hhy did the broken umbrella in the boy's hand, damaged by the wind, suddenly get fixed while the boy and the girl were in the air? based on the provided video and options. innAfier watching the video, it's clear that'al. The umbrella initially broken boy girl are into the by planos attached balloons/n3.
Gemini-2.0e 2.a-lllhh---
by the wind the ground.1n2. air, broken umbrella appears
observed in the vidco and consider the logical explanation within the context of the animation'ln'n" .... **( Because boy held restored. ** This whimsical, magical explanation. While the video has fantastical clement, !ggoed luck!" causing physicul repair is not the most direct or likely intended answerinin .I...)) Aerodynamics, changes in wind strength and direction contributed to the umbrella being repaired. This is more detailed and scientifically grounded explanation. As they ascend, changes in wind strength, direction, and the acrodynamic forces acting on umbrella could allow reshape itself. This very plausible: inin" While the video *is" computer-gemerated animation, option describes "how* shown, not *why" happens within video's
nnwnNww let's evaluate each option against
the girl's hand, gained good luck, and as result, the umbrella
narrative. comprehensive includes "scrodynamics"" in addition to changes wind, which more complete physics-based explanation relevant unbrella in motion in the air. Option (C) simplified version of (1)annnOOtion (I) is the most scientifically plausible and best explanation from the given options for why the umbrolla appears to get fixed within the context of the unimation !!
explain in-universe reason
repair Comparing options (C) (I), option (I)
Human Error Annotation: The model only analyzes visual information in the first small orange of the analysis, and only analyzes the frame corresponding to the question. After that, most of the CoT is doing text analysis (comparing the pros and cons of each option)
Figure 14: Error Case: Lack of Visual Reasoning.
39

---

Published as a conference paper at ICLR 2026
Lack of Visual Reasoning
Question: How did the man manage to insert the top curd into the deck and then reveal the card again, only to find it back in its original position?
Options:
"(A) The card deck was equipped with hidden compartment."
(B) He had two identical cards, and he switched them."
(C) The card dock was magnetic, allowing precise control.".
(D) The cards were fitted with special udhesive." :
(E) The man used camera tricks manipulate what was shown.".
(F) He used invisible threads to control card's position.".
(G) The cards had sensors that automatically rearranged them.".
(H) inserted the card in fake cut,
deck.",
The used sleight hand secretly return the card the top.".
The deck rigged with springs pop the card back up.".
When showed the cards. he actually turned over two cards."
Video Type: TV
Task Type L1: Explicit Reasoning.
Task Type L2: Counterintuitive Reasoning
Task Type 1.3: Magic Deconstruction
Gemmini2200-llshh Thinsiiiihiiig My thinking process to urrive at the correct answer, option (O), as follows/n'nl. Understandd the Question:** The question is asking for the *method* used by the man to make the top card reappear on top of the deck after being inserted into the deck. This is classic magic trick scenario. Watch the Video Carefully=** A*| watched the video multiple times, paying close attention the man's hund movements, especially when he ("inserts(" the card and then reveals it again.inin3. "nnalyzee the Actions the Video:**in He shows into deckin
"appears* to insert Clubs, 'nin4 **Eyaluate Each Option Against Video and Common Magic Principles: + went through each option, considering it's plausible based what
top card (2 Clubs).nn
of
immediately reveals
top again,
general knowledge of magic tricks. * (A) Hidden Compartment: Possible theory, but decks with hidden compartments less simple card tricks. It's also visually indicated the video. Less likely for this type quick trick. 'n ..... ***(1) Sleight of Hand:** This is the most probable and general explanation for card magic. Sleight of hand encompasees various techniques to manipulate cards secretly. This option directly addresses the core of card magic and fits the observed actions. "This is the most likely answer. inn +*) Rigged Springs:** Mechanically complex and unnecessary, Springs would be hard to conceal and control for this specific effect. Unlikely.'n **(KK) Turned Over Two Cards: Doesn't explain how the card returns the top after insertion. Also, visually, it looks like he's only tuming over one card. Unlikely and inrelevant to the question.'n'as. **Select Best
for
After evaluating all options, option (I) 'hee man used sleight top'" stands out
hand to secretly return the card core principle card magic and directly addresses the question of how the card returned the top. Option (B) is "type* sleight of hand, but broader and more encompassing general explanation. Option (H) also related sleight of hand option unswer/in'n[[]]
likely explanation. encompusses most generul, accurate,
direct question. In'n.. **Final Answer:* Therefore, option the best
model only analyzes the end vidco, magician explains that he tumned cards beginning and showed sccond card. The model did not observe the evidence frame of the decryption part through visual reasoning.
frames adjacent question
video.
Figure 15: Error Case: Lack of Visual Reasoning.
40

---

Published as a conference paper at ICLR 2026
Implicit Misinterpretation
FATHER
Question: What the main message this video convey?
Options:
(A) The video is mainly about showing how easy
solve family problemss without much effort.
(B) praises the mother's strength and kindness, and how teaches her children to interact with others.
clothes do get wet. (D) aims illustrate that family members should always rely external help when trouble.
(C) praises the sacrifice the brown cost, ensuring that
The video tries convey that fumily is not important compared personal
( video praises the father's
responsibility and commitment.
(Gy The main point that fathers always make mistakes and children have fix them.
(H) It aims to show that futhers should focus more on their careers than on family.
Video Type: Animation
Task Type Implicit Reasoning.
Task Type Theme Understanding
Task Type Everyday Toples
GPT-4o: [Analyze the best option for question] The video showcases brown coal protecting other clothes from rain on clothesline. The actions and symbolism suggest the brown coat's sacrifice, highlighting the theme of protection or
sacrifice for others, akin familial role, ininjustiiiicattin for your final choice based on the thinking
process.Jull hhrrough the coat's protective measures, video illustrates the theme sacrifice by brown cost to keep clothes dry. aligns closely option C, where the sacrifice the brown coat ensures that the other clothes get Give the final correct option number following format: V[[C]]
Human Error Annotation: The model only analyzes the actions in the frames adjacent to the question frame in the video. fact, end the video, the magicion explains that turned over two curds at the beginning and showed the second the evidence frame decryption part through visual reasoning.
card. The model
Figure 16: Error Case: Implicit Misinterpretation.
41

---

Published as a conference paper at ICLR 2026
Reasoning Error
Question: Please guess how many videns needed record the moment the punched the punctured water
bull the beginning of the video?
Options:
(A) lenst separate takes would needed.
(B) At least one single take is needed.
(C) Three separute takes are needed.
(1)) Four separate takes are needed.
(B) Each scene can be captured in single continuous take.
(F) Five separate takes are needed.
(G) Six separute takes are needed.
(H) Fight separate takes are needed.
Ten separate takes are required.
(J) Twenty sepurate takes are necessary.
(K)
separate takes needed.
Video Type: TV
Task Type L1: Explicit Reasoning.
Task Type L2: Counterintuitive Reasoning
Task Type L3: Magic Deconstruction
Gemini-2.0 Flasai-- ---saa-Thinkinng Thinking: [[Thinnkin Process] In question usks guess the number of videos needed record the Flash-
water balloon punch scene the beginning of the video. This question about the "process* of video making, specifically related to achieving good take. tuke.'n'n ''nnased on common video production knowledge and the nature of capturing dynamic event like water halloon bursting in slow motion, it highly unlikely that such a scene would he captured perfectly in single take. There are many variables such as timing of the punch, the balloon's 's burst pattern, camera focus, and framing /n inConsidrinng the options:'n" **Options suggesting single take (B) and (E)** are less likely.
Capturing a dynamic and potentially messy action like a water balloon burst perfectly on first try is improbable,
espocially for visually appealing slow-motion shot lm **Options suggesting multiple takes (A). (C). (D), (F), (G), (H), (I), (T), (K)** are more reasonable. Filmmaking often involves multiple takes to ensure the best possible shot Option (A) least two separate takes would needed."
most conservative option indicating very reasonable minimum guess.nn **ptionns with higher numbers
more than one
take probable.
(C), (D), (F), (G),
(H) (I), (J), (K)** are also possible, but they more specific guesses. Without explicit information from the video. these harder justify the *best* guess compared to the minimul l"at least two!" in **Option (K) 'tt least separate takes are needed." *** and **)] 'Twenty separate takes necessary. *** are likely overestimations for relatively simple shot like punching water balloon. While possible, they are less probable the *best* information. in inveniven iiven that we are asked to *guess* and choose the "best* answer, the most reasonable and safe guess, based on general video production practices and the nature of the seene, is that **at least two takes would be needed** This accounts for the likelihood of needing to adjust something or re-perform the action for a better result.n'nTThheeeee Therefore, Option (A) is the most accurate and reasonahle answer from provided options.nnnll [norreCrreee Option]]]n[[AM
without more
Human Error Annotation: model accurately found frame corresoondigg to the problem perceived evidence required reasoning,
Visual Reasoning. However,
made mistakes
orange part during reasoning.
Figure 17: Error Case: Reasoning Error. The reasoning and analysis process of this example can refer to the disassembly video.
42

---

Published as a conference paper at ICLR 2026
Correct Case
mubber
Question:
The entire video is viewed through black circle. What is this?
Options:
(A) It's circular hole in piece of paper the viewer is looking through.
(B) It's decorative frume added for uesthetic purposes.
(C) It is the scope of gun.
(D) It is the view through a telescope.
(F: It is a special pair observation glasses used to observe the behavior wild animals.
(F) is an astronomical telescope.
(G) It the lens telephoto DSLR cameru.
(H) the through kaleidoscope.
the lens of vintage
It's magic hoop that gives special vision.
Video Type: Animation
Task Type LI: Explicit Reasoning.
Task Type L2: Causal Reasoning
Task Type L3": Backward Reasoning
claude-3-S-sonnet: Let me think about this step by step'annn) Looking at the video, I notice the view is consistently shown through circular frame with crosshairs in it.laa22) The crosshairs (reticle) are a key identifying inginin feature they have the characteristic ("+\" shape with circle-inin)) Throughout the video, the view is used to target/aim at birds in the trees.ln'n4) The combination of the circular view, crosshairs, and targeting function strongly indicates this is scope used aiming/nnas Among the options, mentions l"the scope of a gun!" this matches perfectly all visual evidence. (n'66) While other options like telescopes cameras might have include targeting crosshairs like here.a''?) The clear targeting hunting context and reticle design confirms this specifically gun scope rather than any other optical device. [n'n[[C]]
views, they typically don't
Human Analysis: relatively good answer. model's thinking process before answering question mainly centered around the visual information of the vidco, rather than the entire CoT being spent analyzing the textual reasoning of the options.
Figure 18: Correct Case.
43


Reviews:
[{"review_id": "90T3ZiI5bx", "reviewer": "Reviewer_Xppq", "summary": "This paper introduces a novel video reasoning benchmark, MMR-V, to evaluate multimodal deep reasoning in videos. Unlike existing video understanding benchmarks, MMR-V emphasizes multi-frame reasoning, implicit and explicit reasoning. Extensive experiments show the limitation of current MLLMs in handling complex reasoning tasks.", "questions": "1. In L452, it is not very clear what GPT-4.1 is labeling, can you explain in detail?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The benchmark is carefully curated with human annotation, with evidenced distractors generated by GPT.\n\n2. The authors evaluated a wide range of models (including GPT-5, Gemini, Claude, and open-source alternatives) and provide detailed error analysis, scaling trends, and modality impact (e.g. audio).", "weaknesses": "1. The distinction between “implicit” and “explicit” is not always clear-cut in practice. For example, in detective films, it might require both implicit and explicit clues to determine the criminals. \n\n2. Comparsion with existing video reasoning benchmarks (e.g. VRBench) could be further discussed.\n\n[1] VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos. ICCV 25.\n\n3. To further understand MLLMs reasoning ability, it is encouraged to include the comparison based on either different video lengths (e.g. <60s, 60s-300s, etc), or different clue lengths (e.g. 1-frame, 5-10 frames).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QJetIlkQMe", "reviewer": "Reviewer_9zhX", "summary": "In this work, the authors propose MMR-V, a benchmark for multimodal reasoning in videos. MMR-V consists of 317 videos and 1,257 tasks, for  long-range, multi-frame reasoning in the real world. The authors also investigate the current MLLMs models on this benchmark.", "questions": "Please see the weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "*Clarity\n\nThe paper is well-written with good structure. Hence, the clarity is basically good.\n\n*Significance\n\nThis paper focuses on evaluating video reasoning capacity of MLLMs, which is an important and practical problem for video understanding. Hence, the significance is basically OK for video research community.", "weaknesses": "* Reference\n\n1) The recent work [VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos, ICCV 2025] proposes the similar topic for video understanding. Please clarify the key difference.\n2) Small suggestion: It would be clearer to include a table to show key statistics difference between this bench and the existing ones such as Video-MME,  LongVideoBench, LVBench, Video-MMMU, MMVU, etc.\n\n* Method Insight\n\n1) It woule be more interesting to investigate or indicate how to design MLLMs to tackle the tasks in this benchmark.\n2) Actually, there exists some agentic mechanism for long video reasoning such as VideoAgent, VideoTree, etc. These methods are suitable for reasoning via tool usage with multi-round clue discovery. Hence, it would be great to include these works for evaluating the benchmark. \n\n* Small Size\n\nThe authors collected only 317 original videos. The small number of videos would restrict the generalization of this benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1N5xICu3UI", "reviewer": "Reviewer_qv7x", "summary": "This paper introduces MMR-V, a new benchmark designed to evaluate the deep multimodal reasoning capabilities of large language models in videos. MMR-V requires models to locate and analyze evidence across multiple, long-range frames to answer questions that involve hidden information. Through extensive testing on 21 different models, the study reveals that even the best-performing model, Gemini-2.5-pro, only achieves 64.3% accuracy, indicating a significant gap between current AI capabilities and complex video reasoning.", "questions": "1.\tWhy do the authors only select the multiple-choice question as the evaluation protocol? Are open-ended questions essential for the reasoning tasks?\n2.\tFor the annotation process, are there any time interval limitations for the questions? The checklist requires long distance, how long is it, and how to guarantee it? \n3.\tFor the implicit reasoning question type, how can to ensure that all annotators can obtain all world knowledge to understand the metaphors in the videos? \n4.\tIs there the possibility that the discrepancy in the annotators’ cultural background leads to different reasoning results?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tIt is an interesting and novel step to divide the reasoning tasks to be implicit and explicit. Evaluating the model’s capability to combine the precepted information with previously learned world knowledge to understand the metaphors is important. \n2.\tThe entire benchmark is human-labeled, and several approaches are conducted from the video collection to data annotation to guarantee the overall quality to make the benchmark more reliable.\n3.\tExperiments are comprehensive, the authors evaluate enough frontier open-source and proprietary models, and compare with human performance. The authors also conducted a series of ablation studies regarding the input frames and modalities.", "weaknesses": "1.\tThe questions designed for reasoning are not as deep as the authors claim. First, all question-answer pairs are single-step and do not require reasoning chains. This makes the proposed benchmark less challenging compared with some widely adopted text or image reasoning benchmarks, which include some mathematical or scientific problems that require multi-step reasoning. Besides, for many reasoning question types, such as personal reflection, video naming, and meta-emotion, it is more likely to be visual perception tasks. What reasoning capabilities are needed for these tasks? Readers may think the authors overclaim some perception problems as “deep reasoning” tasks.\n2.\tThe authors have detailed the generation process of the wrong answers, yet the description of the human annotation process is vague and unclear. Are the annotators specifically trained for this labeling task? Is there any human review process for the human annotations? How to guarantee the overall quality of the human labeling process? More clarifications are needed.\n3.\tThe literature review only includes some video perception benchmarks, but more recent video reasoning benchmarks are not involved and compared, such as VideoEspresso, VRBench, VideoReasonBench, CG-Bench, and MMVU. I believe some discussions are needed to highlight the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WOeWeIcu7", "reviewer": "Reviewer_NiAg", "summary": "This paper introduces MMR-V, a new benchmark designed to evaluate the deep multimodal reasoning capabilities of large language models in the context of videos. The authors argue that existing video benchmarks primarily test perceptual understanding over short temporal windows, failing to assess more complex reasoning. MMR-V addresses this gap by featuring tasks that require long-range, multi-frame reasoning to infer information that is not directly visible. A key contribution is the categorization of tasks into \"implicit reasoning\" (understanding subtext, metaphors, themes) and \"explicit reasoning\" (logical deduction from perceivable evidence). The benchmark consists of 317 videos and 1,257 manually annotated multiple-choice questions with carefully designed distractors to challenge current models. The experimental results, covering a wide range of state-of-the-art models, demonstrate that even top-performing models like Gemini-2.5-pro struggle significantly, revealing a substantial gap between model and human performance and highlighting the limitations of current reasoning-enhancement techniques like CoT.", "questions": "1、The paper acknowledges that video categories like mystery and gaming are underrepresented. Could you elaborate on why these categories were excluded (e.g., difficulty in annotating reasoning tasks, lack of popular videos) and whether there are plans to expand MMR-V to include them in future updates?\n2、For implicit reasoning tasks, the paper attributes models’ better performance to \"abundant visual cues dispersed throughout the video.\" Have you explored whether models’ pre-trained world knowledge (e.g., understanding that \"room number 7 symbolizes good luck\") also contributes to this performance gap?\n3、The human experiment uses 200 tasks (100 correct/100 incorrect for GPT-4o). Were these tasks balanced across implicit/explicit reasoning and video categories? If not, could this imbalance affect the measured human-model gap?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1、The paper introduces a benchmark that addresses a critical gap in video understanding evaluation. It introduces a novel task structure focused on deep, multi-frame reasoning. This requires models to synthesize information across long temporal spans and infer hidden context, a much-needed step up in complexity that better reflects real-world reasoning challenges.\n2、The quality and rigor of the benchmark construction are a major strength. The reliance on manual annotation, guided by real-world user consensus, ensures high-quality, nuanced data that is difficult to create automatically. Furthermore, the innovative strategy of using model-generated errors to create plausible and challenging distractors is a key feature that effectively tests the robustness of models and mitigates the risk of them relying on superficial cues.\n3、The paper provides a clear, comprehensive, and valuable evaluation of a wide range of state-of-the-art models. This extensive testing not only establishes a strong and reliable baseline for future research but also offers insightful analysis into the current limitations of MLLMs in video reasoning. The results clearly highlight the significant performance gap between models and humans, underscoring the benchmark's importance and difficulty.", "weaknesses": "(1) The benchmark’s 317 videos primarily cover six categories but underrepresent real-world genres like mystery, puzzle-solving, or gaming (acknowledged in Section B as a limitation). Additionally, most videos are in English, with only a small proportion in other languages, restricting MMR-V’s applicability to multilingual video reasoning tasks.\n(2) While the paper notes that models perform better on implicit tasks (+7.9% average gain) due to dispersed visual cues, it does not explore why explicit tasks are harder. For example, whether it is due to fewer evidence frames or stricter logical requirements. This limits insights into how to improve model performance on specific reasoning types.\n(3) While the paper tests frame count impact (Figure 4) for Gemini-2.0-Flash, it does not report efficiency metrics (e.g., inference time, memory usage) for models processing long videos. This is relevant for real-world deployment, as long-video reasoning often requires balancing accuracy and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)