VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?
Authors: Yuanxin Liu
Kun Ouyang
Haoning Wu
Yi Liu
Lin Sui
Xinhao Li
Yan Zhong
Y.Charles
Xinyu Zhou
Xu Sun
Venue: ICLR 2026 Poster (2026)
Keywords: video reasoning
multimodal large language models
Primary area: datasets and benchmarks
TL;DR: We introduce a benchmark to evaluate vision-centric complex video reasoning

Abstract: Recent studies have shown that long chain-of-thought (CoT) reasoning can significantly enhance the performance of large language models (LLMs) on complex tasks. However, this benefit is yet to be demonstrated in the domain of video understanding, since most existing benchmarks lack the reasoning depth required to demonstrate the advantages of extended CoT chains. While recent efforts have proposed benchmarks aimed at video reasoning, the tasks are often knowledge-driven and do not rely heavily on visual content. To bridge this gap, we introduce **VideoReasonBench**, a benchmark designed to evaluate **vision-centric, complex video reasoning**. To ensure visual richness and high reasoning complexity, each video in VideoReasonBench depicts a sequence of fine-grained operations on a latent state that is only visible in part of the video. The questions evaluate three escalating levels of video reasoning skills: recalling observed visual information, inferring the content of latent states, and predicting information beyond the video. Under such task setting, models have to precisely recall multiple operations in the video, and perform step-by-step reasoning to get correct final answers for these questions. Using VideoReasonBench, we comprehensively evaluate 18 state-of-the-art multimodal LLMs (MLLMs), finding that most perform poorly on complex video reasoning—e.g., GPT-4o achieves only 6.9\% accuracy—while the thinking-enhanced Gemini-2.5-Pro significantly outperforms others with 56.0% accuracy.  Our investigations on "test-time scaling" further reveal that extended thinking budget, while offering none or minimal benefits on existing video benchmarks, is essential for improving the performance on VideoReasonBench.

Published as a conference paper at ICLR 2026
RR VIDEOREASONBENCH CAN MLLMS PERFORM VISION-CENTRIC COMPLEX VIDEO REASONING?
Yuanxin Liu Kun Ouyang' Haoning 2u
Yi Liu'
Lin Sui?
Xinhao Li
Yan Zhong
Y. Charles
Xinyu Zhou
Xu Sun
State Key Laboratory of Multimedia Information Processing,
School of Computer Science, Peking University
2 Moonshot AI Nanjing University
School of Mathematical Sciences, Peking University wubacningenoonshot.
1 iuyuanxin@stuu edu.edu. en
ABSTRACT
Recent studies have shown that long chain-of-thought (CoT) reasoning sig- nificantly enhance the performance of large language models (LLMs) on com- plex tasks. However, this benefit is yet be demonstrated in the domain of video understanding, since most existing benchmarks lack the reasoning depth required to demonstrate the advantages of extended CoT chains. While recent efforts have proposed benchmarks aimed at video reasoning, the tasks are of- ten knowledge-drive driven and do not rely heavily on visual content. To bridge this gap, we introduce uate vision-centric, complex video reasoning. To ensure visual richness and high reasoning complexity, each video in VIDEOREASONBENCH depicts a se- quence of fine-grained operations on latent state that only visible in part of the video. The questions evaluate three escalating levels of video reasoning skills: recalling observed visual information, inferring the content of latent states, and predicting information beyond the video. Under such task setting, models have to precisely recall multiple operations in the video, and perform step-by- step reasoning get correct final answers for these questions. Using VIDE- OREASONBENCH, we comprehensively evaluate 18 state-of-the-art multimodal LLMs (MLLMs), finding that most perform poorly on complex video reason- ing ing.e.g.. GPT-4o achieves only 6.9% accuracy whilee the bhinking-enhancee Ge5ini-2.5-Pro significantly outperforms others with 56.0% accuracy. Our in- vestigations on "test-time scaling" further reveal that extended thinking budget, while offering none or minimal benefits on existing video benchmarks, is essen- tial for improving the performance on VIDEOREASONBENCIL Project Page: wo/video https::/11yx9: .itiub..oovvieeee... reason bench/
benchmark designed to eval-
INTRODUCTION
Recent advances in long chain-of-thought (CoT) reasoning (DeepSeek-AI et al. 2025 Jaech al., Team et al.) 2025a) have remarkably enhanced the problem-solving capabilitie lities of large lan- guage models (LLMs). By scaling up the test-time compute with extended CoT reasoning chains, substantial performance gains have been observed in complex tasks such mathematics (MAA)
2024
2024 Lightman et al. 2024| Lu et al. 2024). coding (Jain et al. 2024 Jimenez et al.) 2024). and scientific reasoning (Rein al.) (023). However, the benefits of long CoT reasoning have not been fully demonstrated in the domain of video understanding. This gap is largely due to limitations in
existing benchmarks (Mangalam al. 2023 Fu al. 20244 Liu et al. 2024 Li et al., 2024b; Wul jet al. 2024 Zhou et 2024 Li et al. 2024c Shangguan et al. 2025), which often lack the rea- soning depth necessary showcase the advantages of extended CoT chains. As shown Figure [
"Project Lead
1 "Correspnndigg Author(s)

---

Published as a conference paper at ICLR 2026
Video-MME (General VideoQA)
TempCompass (Temporal Understanding))
sepuaace eneces Invelutig Reary
Thinking 3,619 takens of
No Thinking of
Thinking tokens
No Thinking
B. To map Serbian anshetzalt
taker
choudig ord his car key then Balling back irside the car
stoudig cee tie car key then solling back insare the car
MMVU Sehowledge-Intennee VideoQA)
Thinking 1,
444 tokens
Thinking /
willuss of the gas is eccressing
pressore of the increases 1s the veluse recresses
process
Adtabbttc compression Adtabotic
201 cokera
... tenperature the gas
increasing a.e pressure
takena
Isnbaric coppesssoon Isothereul exiausion
c..
tiral recerr ts Wihooeditss.
VideoReasonBench
If the emangasent of sumbers on the soord currently in the sume state ar it sas at the end of the video, and the follseing noves are executesl: "tefteord, opward, dieseary, leftsord, righbeard', sbet will arroogemnt nabbrrs beord?
Thinking 14,241 tokens
No Thinking king X
anditel state the bourd 02:20 2 2, 1e.21e 3. 14.31: 6, 14,40s
1,323 Loloot
expty
throughest
340 TORONS
1b,20 (b,2) (b, 12, 10,41s
(9,212
the video Lastata,220 (au21s (6,311 14,41:
16,332 1, (2.311 6, 16,311 T, 16
18:30 3. (622): 4, 16.511 by 10,4
Fiest tn a.. by13 13, (3,23s By 16.001 6. 16,411
Figure 1: Examples from VIDEoREASON and three existing VideoQA benchmarks. Re- sponses are generated by Gemini-2.5-Flash in both "Thinking" and "No Thinking" modes. The text highlighted in green/red indicate correct/incorrect responses. While questions from existing benchmarks can be answered correctly without "Thinking" using only few tokens, VIDEORE- ASONBENCI requires "Thinking" for accurate reasoning and consumes substantially more tokens (See Figure for quantitative results). also demands finer-grained visual perception during rea- soning.
the advanced multimodal LLM (MLLM) Gemini-2.5-Flas F.s-Faahh can correctly answer the questions from two popular benchmarks, Video-MME (Fu et al., (024) and TempCompass (Liu et al. 2024) using only few response tokens without activating the thinking mode.
To address this gap, several benchmarks have been proposed recently to better emphasize CoT rea- soning in video understanding. Video-MMMU (Hu al., 2025) and MMVU (Zhao et al. 2025) in- tegrate video understanding with domain-specific knowledge, thus introducing a need for reasoning. However, the required reasoning process is primarily noowledge-drivend lacking strong reliance the visual content. Two concurrent studies, VCR-Bench (Qi et al.] (2025) and MINERVA (Nagrani al., 2025), evaluate the correctness of video reasoning process in addition the final answer. Nonetheless, the videos and questions in these benchmarks often resemble those in general video understanding benchmarks, emphasizing short-horizon skills such as temporal grounding, action counting, and temporal order comprehension, while fall short in demanding deeper video reasoning.
Motivated by these limitations, this work introduces the capabilities of MLLMs in performing vision-centric, complex video reasoning. We define three levels of video reasoning, each requiring progressively more sophisticated reasoning: The first level is to precisely recall the sequential visual observations from the video. The second level is to infer latent information that not directly observable from the video. The third level to predict new information beyond the video. For instance, shown in Figure I video from VIDEOREASON- BENCH presents a "sliding number puzzle", which numbered tiles are initially visible but become masked sliding movements occur. To accurate answer the question, model must first recall the initial tile arrangement and all subsequent movements (Level I), then infer the final arrangement of tiles (Level 2), and finally apply this inferred information to prediet future tile positions (Level 3).
evaluate the
VIDEOREASON BENCH is constructed based on the aforementioned core ideas. Each video illus- trates sequence of operations (e.g., sliding movements) performed to latent state (e.g., tile
2

---

Published as a conference paper at ICLR 2026
rangement). The richness of visual information can be flexibly controlled by adjusting the size of the latent state and the number of operations. In addition to the "sliding number puzzle", , our bench- mark includes six types of video demonstrations spanning various scenes, featuring both synthetic and real-world videos. To evaluate reasoning across all three levels, we design six corresponding reasoning skills, with two for each level (see Figure
Based on VIDEOREASONBEENH, we comprehensively evaluate 18 state-of-the-art MLLMs. Our results reveal that most MLLMs struggle with vision-centric complex video reasoning, achieving ac- curacies below 10%. In contrast, the thinking-augmented Gemini-2.5-Pro significantly outperforms all other models, reaching an accuracy of 56%. Further analysis shows that while extended chain- of-thought (CoT) reasoning offers minimal performance improvements on existing benchmarks, it crucial to 'IDEOREASONBENCE leads to substantially larger drop in performance compared to other
Additionally, we observe that removing visual information
from FASON benchmarks, highlighting its strong reliance on visual content.
The main contributions of this work are summarized as follows:
1. We introduce the OODERREASONBENCE for evaluating vision-centre complex video rea- soning. poses necessity for models to correctly perceive multiple actions in a sequential order and perform step-by step-by-step reasoning to finally answer the questions, therefore by- principle featuring higher demand for reasoning depth and stronger reliance on the visual content.
2. We reveal the concerning deficiency of most SOTA MLLMs in our benchmark: Several latest thinking models, such 04-mini and Seed1.5-VL, only gets around 10% accuracy; non-thinking SOTA MLLMs (e.g. GPT-4o and Qwen2.5VL-72B) scores lower than 10%; all efficient MLLMS 10B) cannot reach even 2%
3. Our experimental investigation confirms that the accuracy of Semmin-2255-Flas drastically degrade while dropping 50% input video or disabling thinking-mode, while existing video benchmarks do not show similar properties. This result underscores the value of VIDEO- REASONBENCH as a paragon evaluate vision-centric complex video reasoning abilities.
2 VIDEOREASONBENCH
2.1 TASK DEFINITION
Existing research lacks clear and established definition of what is vision-centric complex video reasoning. To address this gap. we propose systematic framework that formally defines the task, incorporating both video content design and different reasoning question skills.
2.1.1 VIDEOS
We conceptualize videos as a sequence of state transitions, represented as (Se, oe, ) where an operation o transforms state S, into S++1- In framework, the full sequence of operations is visually observable, while the states are only partially visible-either at the beginning or at the end of the video. Thus, the visible components of video are either: {S), 01, -.--1 or {o), -T, }r.. This design enforces visual complexity via the dense sequence of operations and fosters reasoning complexity by requiring inference about latent states.
we design six categories of video demonstrations based on this principle. Number: The latent state is an N x N board with numbered tiles and one empty tile. Operations consist of sliding numbered tile into the empty space. Circle: The latent state is an N x N grid containing black and white pieces. red circle moves across the grid, flipping the color of the pieces it passes over, as well as their neighbors. Cup: The latent state is an N x N board with squares that may be empty or contain a coin, all covered by cups. Operations involve swapping the positions of two cups, altering the contents beneath. File: The latent state consists of N file paths. Operations include creating, deleting, copying, and moving files within/between these paths. Card: The latent state comprises N piles of cards. Operations involve adding card to the top of pile or removing card from the bottom. Chip: The latent state consists of N cups, each containing a number of chips. Operations involve adding removing chip from a cup.
As illustrated in Figure
3

---

Published as a conference paper at ICLR 2026
File
Card
Chip
Video Demonstrations
Show State
Show State
Show State
Shaw Operations Number
Show Operation
Show Operations
Circle
Cup
Show State
Show
Show Operations
Show Operations
Show Operations
State
Recall Order
Recall Count
Level Recall
Skills & Questions
ing rany
perforred video?
Infer State
Compare State
Level
What is the arrangesent of numbers on the board at the
Compare the number arrangenents on the board at the start video, what are the squares where boards?
and
Predict State
Predict Operation
Level Predict
If the arrangerent of numbers on the board is currently an the same state as it was at the end of the idee, and the following moves are upward board?
If the arrangerent of numbers on the board is currently ly in the same state as it was at the and of the video, what sequence noves (left, right, up, doan) should arrangesents
"upaard, arrangenent nurbers
4"3
Illustration of vision-centric complex video reasoning. Upper: In each video, the latent state is revealed either at the begin or the end, and sequence of observable operations is applied this state. There are six categories of videos, each featuring a different type of demonstration. Lower: The questions assess video reasoning across three levels, with two skills for each level.
Figure 2:
2.1.2 QUESTIONS
As Figure 2 shows, our framework evaluates video reasoning skills across three progressive levels.
Level 1 focuses fine-grained visual perception, comprising two sub-tasks: Recall Or- der, which requires recalling the exact sequence of operations, and Recall Count, which involves counting the frequency of specitic operations.
Level 2 assesses reasoning about latent states based the observed operations. This in- cludes Infer State, where the task is to infer the content of latent state at a certain moment, and Compare State, which requires comparing the latent state between two moments.
Level 3 advances counterfactual reasoning, requiring prediction based inferred infor- mation. It includes Predict State, where the goal is to predict future state after a sequence of operations, and Predict Operation, which involves identifying the operations needed to reach given target state.
Video Distribution
Question Distribution
Table 1: statistics.
VIDEOREASONBENCC
Order
Predit Operation
S003%9
600L7701
240117%
24012790
Statisties
Value
God
locall Court 200
PINKEE STATE 38317TA
Question Word Count (avg/mux)
81.5/262
Prompt Count (avg/max)
198.8/420
Duration (Seconds, avp/max)
54.3/154.8
Ap
Zil
Infer State
Compary Gtate
Videos by Operation
5
120
120
Videos by State Size
(3 2.& (4 x1xxX
120
120
Figure 4: VIDEORRASONBENCH video and question distri- butions.
4

---

Published as a conference paper at ICLR 2026
Video
Configuration
State Stxed N Operotior Count::
Video Engine
State Transitions
QA Pairs
Sy 0g .... 0r-1 ST
Question Engine
(Q2. Azh. (Qa.
Vi VigoReasonn
Engine state_transit(
Engine ques__gen()
video_syn() screen_shott) manual_shott)
ans_genn
Ieput: N.
Input:
Toputt
Output:
matplstlib
skil skill, denonstrattion
[S os.... 9r-1,
[S,.0 ST)
Output:
Output:
Figure 3: Overview of our data construction framework. The video engine generates state transitions from given configuration, producing videos via Matplotlib, command- screenshots, or real- world manual recordings. The question engine then generates questions and derives answers based on the state transitions, following the rules of each demonstration.
2.2 DATA CONSTRUCTION
We construct our dataset based on the previously described task definitions. Table I and Figure 4 summarizes the dataset statistics. In total, ViDeoReASONBENCE consists of 1,440 questions and 240 videos, with an equal number of questions per skill and an equal number of videos per demon- stration. The number of operations depicted each video ranges from to For demonstrations involving File, Card, and Chip, the latent states consist of one or two sets (e.g., one or two piles of cards), with each category containing 120 videos. For the Number, Circle, and Cup demonstrations, the latent states represented as boards ranging from 3 x 3 to 4 x 4. also with 120 videos per category. To collect the dataset at scale, we design semi-auter sumimautomatic data construction framework comprising two main components: video engine and question engine, demonstrated in Figure
3
2.2.1 VIDEO ENGINE
Given the state size and the number of operations T, the video engine begins by randomly initial- izing state, such integer matrix NNNN XN for the Number demonstration. It then generates sequence of operations and corresponding state transitions, represented as (Se, 0t, S++1 cording to predefined rules specific to each demonstration type. These transitions form a "script" that guides the video construction process. For demonstrations such as Number, Circle, and Cup, videos are generated programmatically using the Python Matplotlib library[] In the File demon- stration, we simulate file operations within command-ling interface and capture screenshots. For real-world demonstrations like Card and Chip, we record actual videos manually.
2.2.2 QUESTION ENGINE
For each combination of video demonstration and reasoning skill, we define specific question templates. For instance, in the Number demonstration with the Infer State skill, the correspond- ing template is "What is the arrangement of numbers (limestamp) of the video?", where timeslamp E {slarl, end] based on whether the la- tent state is revealed at the start end of video. To support accurate understanding of the task, each prompt also includes detailed description of the video demonstration, clarifying the state transition rules. Additionally, we append answer prompt-"PP "oommpttrPPPoiiiii rovide summary of the final answer after FFnnll Answer after the question to help extract the final answer from the model response. The complete prompts and comprehensive list of templates is provided in Ap- pendix B.I Using these templates and the associated state transition data, we automatically generate answers hand-crafted rules, enabling efficient and accurate dataset construction.
the board at the
hutps//mapllotib.org

---

Published as a conference paper at ICLR 2026
2.3 EVALUATION SCHEME
Except for the Predict Operation category, all questions in VIDEOREASONBENCH are paired with ground-truth answers. For these questions, we evaluate model responses by inputting the question, ground-truth answer, and model-generated answer into text-only L.M,, which assesses the cor- rectness of the response. In the Predict Operation category, however, ground-truth answers are not provided, as multiple valid sequences of operations can achieve the given target state. Instead, we extract operations from the model response using the text-only LLM, simulate the corresponding state transitions using the same functions employed by the video generation engine, and then verify whether the resulting state matches the target state. Detailed evaluation and operation extraction prompts are provided in Appendix B.2
EXPERIMENTS
3.1 EXPERIMENTAL SETUPS
Evaluated Models. Based on VIDEOREASONBEENCH conduct comprehensive evaluation of a wide range of MLLMs: Proprictary models include the advanced GPT-4o (2024-11-20) (Ope-
nAl 2024) and Semini-2.0-Flaash (Pichai al.
2024). along with the latest thinking- augmented MLLMs: 04-mini (OpenAT] 2025). Seed 1.5-VL Guo et al. (024). and Gemini 2.5 (Flash and Pro-
0506) (Google & DeepMind (025). Open-source models include mPLUG-OW13 (Ye et al. 2024).
MiniCPM-V 2.6 (Yao et al. 2024). MiniCPM-o 2.6 Yao et al. 2024), Kimi-VI VL-A- (Team et al.
2025b), LLa Vision (7B and 72B) (Li al. 2024a), LLaVA- Videoo (7B and 72B) (Zhang
et al. 2024). VL3 (8B and 78B) (Zhu et al. 2025) ) and Qwen2.5-VL (7B and 72B) (Bai et al. 2025 These models represent the current SOTA in video understanding tasks. We also introduce an additional setting for SeedI.5-VI, Seemini-2.0-Flastt and Gemini-2.5-Flash, which converts key video information into textual descriptions (see Appendix C.2 for details). This baseline, denoted as "vid2txt", allows to independently analyze visual perception and reasoning abilities.
Implementation Details. For proprietary models, we use official APIs to obtain model responses. For open-source models, we perform local inference using publicly available checkpoints. For eval- uation, we adopt Qwen2.5-72B (Yang et al. 2024) as the judge model. To support future research, we modify the the widely used VLMEvalKit framework (Duan et al. 2024) to support evaluation on VIDEOREASON BERRCASONNENCH, BENCH. Additional details regarding generation configurations and video process- ing are provided Appendix C.I
To evaluate human performance on VIDEOREASON BENCH, we randomly sample 240 examples from the full set of 1,440, selecting 40 examples per reasoning skill. Three of the authors indepen- dently annotate the data. Each annotator is presented with the same video and question pairs shown to the models, and provides answers in free-text format.
3.2
MAIN RESULTS
Table presents the evaluation results of various MLLMs and a human baseline on the proposed VIDEOREASONBENCH, from which derive the following findings:
Current MILMs struggle with vision-centric complex video reasoning. Open-source Efficient Models (< 10B parameters) perform near random. with accuracies below 2%. Larger "Flagship Models" (72B+ active parameters), as well as GPT-4o, also also fail to make substantial progress, with performance still below 10%. Even the most recent thinking-enhanced models, such as 04- mini and SeL, show only modest improvements, scoring 10.7% and 11.5% respectively. In contrast, the Gemini 2.5 series demonstrates markedly stronger performance, with Gemini-2.5- Pro-0506 achieving 56% accuracy. Since Gemini-2.5 models are not not specifically optimized for our task, this result highlights a notable gap between frontier proprietary systems and curret open-source MLLMs. Nonetheless, substantial disparity remains relative human performance, which reaches 73.8%. These findings suggest that even the most advanced MLLMs still fall short of human-level capability in the complex video reasoning tasks posed by VIDEOR FASONE BAAEAAANNCHH....C.. BENCH.

---

Published as a conference paper at ICLR 2026
Table 2: VIDEOREASONBENCA evaluation results across three levels of reasoning skills. "The human baseline was assessed on a subset of 240 examples (40 per skill), with an average response time of 223.2 seconds per example. "vid2txt" indicates replacing the video with textual context that summarizes key information of the video content.
Model
Act. Params
Think
Level
Level 2
Level 3
Overall
Recall Order
Recall
Infer State
Compare
Predict State
Predict Operation
Count
State
Human"
223.2s
87.5
90.0
80.0
75.0
67.5
42,5
73.8
Open-source Models
Effiiient Models
mPLUG-OW13
7B
0.0
0.0
0.0
MiniCPM-V 2.6
8B
X
2.1
0.4
0.4
0.0
1.2
0.4
0.8
MiniCPM-o 2.6
8B
X
1.2
0.4
0.4
0.8
1.2
0.4
0.8
LLaVA-OneVisien
0.0
0.8
LLaVA-Video
7B
0.0
0.0
0.0
InterV113
88
X
0.4
018
0.0
0.4
1.7
0.0
0.6
Owen2.5-VL
0.8
0.8
1.3
Kimi VL-A3B
3.3
0.0
1.4
Flagship Models
LLaVA-OneVision
72B
X
0.0
0.0
0.0
0.0
0.8
0.0
0.1
LLaVA-Video
72B
0.0
0.0
InterVLL3
78B
11.2
14.6
5.8
Qwen2,5-VL
72B
12.5
17.1
4.2
4.2
2.9
2.1
7.2
Proprictary Models
GPT-4o
14.2
15.8
4.2
6.2
0.8
0.0
6.9
04-mini
20.4
4.6
10.7
Seed1.5-VL
20B
24.2
27.1
2.1
11.5
Gemini-2.0Flass
18.3
22.5
6.7
6.7
5.0
3.3
10.4
Gemini-2.5-Flash
34.2
19.6
7.1
18.8
Gcmini-2.5-Flashh
44.6
41.7
9.6
27.4
Gemini-2.5PPro-0566
69.2
70,4
63.3
56.7
42.1
34,6
56,0
Qwen2,5-VL. (vddd)))
7B
32.1
30.8
7.9
13.3
1,7
1.7
14.5
Qwen2.5-VL (vid2t2t
72B
62.5
50.0
7.5
7.5
1.2
5.0
22.3
Gemini-2.0-Fassh (vid2tx1)
52.5
Seed1.5-VL (vid2txt)
20B
83.3
87.9
74.2
54.2
45,4
69.4
Gemini-225-Flas (vid2rxt)
86.7
82,5
84.2
75.8
56.2
47.9
72.2
Why do most MILMs fail? The poor performance of most MLLMs can be attributed two main limitations: (1) Insufficient fine-grained temporal perception. Current models struggle to capture dense sequential operations in videos, evidenced by their inability to surpass 30% accuracy even on basic "Level I" tasks. Strikingly, when videos replaced with textual summaries (vid2txt), performance improves substantially-for example, Seedl.5-V rises from 11.5% to 69.4%, and Gemini-2.5-Flash increases from 27.4% to 72.2%. This stark contrast highlights fine- grained video temporal perception as a major bottleneck for current MLLMs (Liu et al.) 2024: Li et al. 2024c] Shangguan et al. 2025 Li et al. 2025). (2) Limited capacity for multi-hop, in-depth reasoning. Thinking-enhanced MLLMs consistently outperform the none-thinking ones. For instance, Gemini- 2.5-Flash improves from 18.8% to 27.4% when enabling the thinking mode. Under the "vid2txt" setting, Seed-1.5-VL and Gemini-2.5-Flash with reasoning outperform the non-thinking Gemini- 2.0-Flash by nearly 30% accuracy and the Qwen2.5-VL models by nearly 50%. This highlights the importance of explicit reasoning mechanisms and extended CoT chains in tackling the complex video reasoning problems posed by ViDeoREASONBENCH ing in different video understanding benchmarks is provided in ş 3.3.1 Additionally, 3.5 presents case studies of model errors, offering a more intuitive illustration.
deeper analysis of the role of think-
ViDeOREASON poses substantial challenges-even for humans. Human annotators also face significant challenges, as answering single question requires recognizing multiple distinct operations (up to 14) within video and accurately inferring the corresponding latent state transi- tions. This process is cognitively demanding and time-intensive, taking annotators an average of 223.2 seconds per question. Furthermore. single misinterpretation could result in incorrect fi- nal answer, which helps explain the relative low human accuracy accuracy-especially on the tasks requiring Level 3 skills.

---

Published as a conference paper at ICLR 2026
Thinking Bikens
Response Takens
Accuracy %
Thinking
Resparee
Accuracy %
9760
20
7200
+2.4%
Generated Takens
7000
7025.6
5200
6000
5280.77
Accuracy %
Accuracy
5000
Cenerate
++0.3%
+1.9%
3200
3071/
2333
2200
1537
8
2000
1860.3
1000
1000
118
TempCompans
$299
Benchmark and Thinking Budget
$.99
HAVU
2092
0
Thinking Budget
1024
2008
8192
(a) Existing Benchmarks
(b) VIDEOREASONBENCH
Figure 5: Performance of Gemini-2 Somnini-2.5-PPe 2.5-Flash with varying thinking budgets on five benchmarks. The "Generated Tokens" is the sum of "Thinking Tokens" and "Response Tokens".
Table 3: Performance of Gemini-2.5-Flaan with different visual inputs on five benchmarks.
Benchmark
Full Video
Cut 50%
Single Frame
Text-only
TempCompass (MCQ) (Liu et al.
2024
79.6
73.5 (L 7.6%)
59.0 25.9%)
40.2 (1 49.5%)
Video-MME (w/o subs) Fu
2024
71.5
64.1 (L 10.3%)
51.3 () 28.3%)
45.6 36.2%)
MMVU (Zhao et al. 2025
70.3
69.1 1.7%)
60.6 13.8%)
44.8 (1 36.3%)
Video-MMIMU (Hu ct al. 2025
68.7
64.3 (C 6.4%)
56.3 (L 18.0%)
49.7 () 27.7%)
VIDEOREASONBENCH (ours)
27.4
12.2 (4 55.5%)
0.5 (L 98.2%)
1.0 (} 96.4%)
Reasoning difficulty increases from Level to Level 3. Performance consistently declines from Level Level 3 reasoning skill for both MLLMs and humans. This trend strongly aligns with the intended design of the benchmark, where higher level reasoning skills are built upon lower level skills, Such design ensures an increased difficult across the three levels.
3.3 ANALYSIS
3.3.1 EFFECT OF THINKING
The benefits of extended CoT reasoning remain underexplored in the domain of video understanding. To address this gap. we systematically investigate how varying the length of reasoning affects per- formance on VIDEOREASONBENNE focus on different abilities: TempCompass (multi-choice), Video-MME (w/o subwords), MMVU, Video-MMMU, We leverage the Gemini-2.5 model, which enables explicit control over the number of reasoning tokens through "Thinking Budget" parameter
and four representative video understanding benchmarks that
As shown Figure [ with the increase in thinking budget, VIDEOREASONBENCH demonstrates notable accuracy improvement mivrenenttrining by roughly 9%. In contrast, existing benchmarks exhibit minimal gains, all under 2.5%, when the same increase in thinking budget applied. This sug- gests that thinking contributes more to the performance of VIDEOREASON than existing benchmarks.
Additionally, the number of response tokens varies notably across benchmarks when thinking budget set to zero: For TempCompass and Video-MME, which primarily test basic temporal and general video understanding, responses Video-MMU, which demand Knowledge-intensee reasoning, show substantially higher response to- ken counts, averaging 183 and 1,537 tokens respectively. Notably, ViDeoReAsoN even longer responses, averaging 1, 1,860.1 tokens, when deprived of explicit "thinking" resources. This pattern highlights the challenging nature of VIDEOREASONBENCI..
srncisc-reuurring only tens of tokens. Conversely, MMVU and
produces
This parameter affects the number of thinking tokens but does allow for precise token-level control.
8

---

Published as a conference paper at ICLR 2026
Table 4: counts.
Results across different state sizes and operation
Table 5: Results across different state reveal timing.
Model
State Size
Operation Count
Model
Begin
End
& (3x3)
& (4x4)
5-9
10-14
Seed1 1.LVL.
12.1
10.8
11.9
11.0
15.1
7.8
Gemini-2.0 2.0-Flas
13.3
7.5
Gemini-2.0-Flaat
11.8
9.0
12.9
7.9
Gemini-2.5-HHe F.5--lshhh
35.3
19.6
Gemini-2 Fla-Flash
30.4
24.4
30.1
24.7
Gemini-2.5-PP
66.9
45.1
Gemini-2..5 2.5-Pro
59.9
52.2
58.2
53,9
3.3.2 EFFECT OF VISION RELIANCE
VIDEOREASON is designed to evaluate video reasoning that demands fine-grained visual perception. To assess its reliance on visual information and compare with existing benchmarks, we evaluate the performance of Gemini-2.5-Flaas under four different visual input conditions: the full video, a version that randomly cuts 50% of the video, single center frame, and text-only input (with no visual content).
The results are shown in Table For MMVU and Video-MMMU, which also involve reasoning, removing half of the video frames results in less than 7% relative performance drop. In con- trast, performance on VIDEOREASON the visual input is further reduced to a single frame, VIDEOREASON shows dramatic per- formance decline of 98.2%, whereas the largest drop observed in the existing benchmarks is only 28.3%. These results suggest that VIDEOREASONBENCE demands a much higher degree of vision reliance than current video understanding benchmarks.
decreases by 55% under the same condition. When
3.3.3 EFFECT OF VIDEO COMPLEXITY
As introduced in $ 2.2 our videos vary in operation count and state size, both which influence the richness of visual information. As decreases as operation count and state size increase. These findings indicate that video reasoning complexity can be effectively controlled by adjusting these two parameters, enabling flexible scaling of the benchmark's s difficulty in future evaluations. We also include an analysis on further extending operation count and state size for Number and Cup videos in Appendix A.3
in Table 4 the performance of MILMs generally
3.4 EFFECT OF STATE REVEAL TIMING
In VIDFORFASONBENCH, the latent state revealed either at the beginning or at the end of each video. compares model performance under these two different reveal timings. As shown, all four MLLMs exhibit lower accuracy when the latent state is revealed at the end. This performance drop occurs because, in this setting. the models must infer the initial state by reasoning backward through the sequence of state transitions. This reverse inference is inherently more challenging than following the transitions in their natural, forward order.
3.5
CASE STUDY
To better understand why most MLLMs struggle with complex video reasoning and why Gemini- 2.5-Pro excels, we conduct case study analyzing model failure cases (Table The example ques- tion requires inferring the number arrangement at the end of the video. However, Gemini-2.00-lass simply outputs the visible arrangement the beginning, showing a lack of reasoning. Qwen2.55 VL-72B attempts multi-hop reasoning track state transitions but misidentifies intermediate square shifts, leading to an incorrect final answer. These cases illustrate two representative failure modes of current MLLMs in VIDEOREASON tion. In contrast, Gemini-2 2.5-Pro successfully tracks state transitions through temporally grounded multi-hop reasoning, arriving at the correct final arrangement.
lack of reasoning and incorrect visual percep-

---

Published as a conference paper at ICLR 2026
Video
Question: Assuming emply square represented by '0', what arrangement of numbers on the hoard the end of the video? Provide the coordinates of each square along with the corresponding number (e.g., (a.1): (a.2): (b.1): 1. (b,2): 2).
Gemini2.0-Flatt
Response: Okay. have watched video tracked the movements
empty square.
Here's the final orrangement of the numbers the board:
(a.1): 2 (a.2): (a.3): 7 (b, (b,2): (b,3): 0 [c 6 (c,2); (c.3):
Analysis: Lack of reasoning, directly responding with the initial arrangement.
Qwen2.5-VL VLL-2BB
Response: To determine the final arringement of the numbers the hoard, let's follow the sequence of moves shown in the video:
The initial arrangement
-(a.1): 2. (a.2): I. (a.3): 7
-(h. 4, (b,2): 3, (h,3):
-(c. (c.2): (c.3);
The empty space is now (a.3). The number at (c.3) moves up to (a.3).
(aa.1): 2. (a.2): 1. (a.3): 8
(bb. 4, (b,2): 3. (b.3): 7
(c.I)) (c.2): 5. (c.3):
Final Answer: (a.1): 2, (a.2): 4, (a.3): 8, (b, (b,2): 1, (b,3): 7, (c, 5, (c,2); (c.3): Analysis: Incorrect visual perception step
Gemini2.5-Pro
Response: Okay, let's track the the puzzle.
**Initial State (00:00):**
(a,1): (a.2): (a.3):
(b,1): 4, (b,2): 3, (b,3); 0 (Empty)
(c.1): (c.2): 5, (c.3):
**Move 5 (00:12 00:14);* The tile at (e.3) (which is 8) moves left into the empty space (c.2).
(a.1): (a.2): 3. (a.3):
(b.1): 4. (b,2): S. (b.3)::
(c.1): (c.2): (c.3): (Empty)
Final Answer: The final arrangement numbers on the board, with representing empty square,
(a. 2, (a.2): 3, (a.3): (b,1): (b.2): 5, (b.3):
(c.1): (c.2): 8, (c.3):
Analysis: Temporalt-grrumded multi-hop reasoning with correct visual perception.
Table 6: Case study of responses by different MLLMs.
CONCLUSIONS
This paper presents VIDEOREASONBENCH to evaluate vision-centric, complex video reasoning abilities of MLLMs. Our results reveal that most SOTA MLLMs struggle with this task. achiev- ing very low accuracies, whereas the hüiking--nnhneee others. A detailed analysis highlights two primary bottlenecks: the lack of fine- mne-grained temporal perception and limited multi-hop, in-depth reasoning skills. We further show that extended chain- of-thought reasoning provides little benefit on existing video understanding benchmarks but proves essential for improving performance on moving visual information from VIDEOREASON ENCH leads to a substantially larger drop in per- formance compared to other benchmarks, underscoring its strong reliance visual content. Overall, VIDEOREASON BENCH provides a challenging and timely testbed to advance research in complex video reasoning.
Gemini-2.5-Pro significantly outperforms
Additionally, observe that
10

---

Published as a conference paper at ICLR 2026
5 ACKNOWLEDGEMENTS
We gratefully thank the anonymous reviewers for their insightful comments that substantially im- proved this work. This research supported in part by Moonshot AI and National Natural Science Foundation of China (No. 92470205).
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar X11::2502.13922 2025.
DeepSeek-AI. Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and Li. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. CoRR, abs/2501. 12948, 2025.
Haodong Duan, Jumming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. VImevalkit: An open-source toolkit for evaluat- ing large
models. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 11198-1120 1201, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. CoRR, abs/2405.210055, 2024.
Google and DeepMind. Gemini 2.5: Our most intelligent ai model, 2025.
Dong Guo. Faming Wu, Feida Zhu, Fuxing Leng. Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, Jingji Chen, Jingjia Huang. Kang Lei, Liping Yuan, Lishu Luo, Pengfei Liu, Qinghao Ye, Rui Qian, Shen Yan, Shixiong Zhao, Shuai Peng, Shuangye Li, Si- hang Yuan, Sijin Wu, Tianheng Cheng, Weiwei Liu, Wenqian Wang, Xianhan Zeng, Xiao Liu, Xiaobo Qin, Xiaohan Ding, Xiaojun Xiao, Xiaoying Zhang, Xuanwei Zhang, Xuehan Xiong, Yanghua Peng, Yangrui Chen, Yanwei Li, Yanxu Hu, Yi Lin, Yiyuan Hu, Yiyuan Zhang, Youbin Wu, Yu Li, Yudong Liu, Yue Ling, Yujia Qin, Zanbo Wang, Zhiwu He, Aoxue Zhang, Bairen Yi, Bencheng Liao, Can Huang, Can Zhang, Chaorui Deng, Chaoyi Deng, Cheng Lin, Cheng Yuan, Chenggang Li. Chenhui Gou, Chenwei Lou, Chengzhi Wei, Chundian Liu, Chunyuan Li, Deyao Zhu, Donghong Zhong, Feng Li, Feng Zhang. Gang Wu, Guodong Li, Guohong Xiao, Haibin Lin, Haihua Yang, Haoming Wang, Heng Ji, Hongxiang Hao, Hui Shen, Huixia Li, Jiahao Li, Jialong Wu, Jianhua Zhu, Jianpeng Jiao, Jiashi Feng, Jiaze Chen, Jianhui Duan, Jihao Liu, Jin Zeng. Jingqun Tang, Jingyu Sun, Joya Chen, Jun Long, Junda Feng. Junfeng Zhan, Junjie Fang, Junting Lu, Kai Hua, Kai Liu, Kai Shen, Kaiyuan Zhang, Ke Shen. Ke Wang, Keyu Pan, Kun Zhang, Kunchang Li. Lanxin Li, Lei Li, Lei Shi, Li Han, Liang Xiang. Liangqiang Chen, Lin Chen, Lin Li, Lin Yan, Liying Chi, Longxiang Liu, Mengfei Du, Mingxuan Wang. Ningxin Pan, Peibin Chen, Pengfei Chen, Pengfei Wu, Qingging Yuan, Qingyao Shuai, Qiuyan Tao, Renjie
11

---

Published as a conference paper at ICLR 2026
Zheng, Renrui Zhang, Ru Zhang, Rui Wang, Rui Yang. Rui Zhao, Shaogiang Xu, Shihao Liang, Shipeng Yan, Shu Zhong, Shuaishuai Cao, Shuangzhi Wu, Shufan Liu, Shuhan Chang, Songhua Cai, Tenglong Ao. Tianhao Yang, Tingting Zhang, Wanjun Zhong, Wei Jia, Wei Weng, Weihao Yu, Wenhao Huang, Wenjia Zhu, Wenli Yang, Wenzhi Wang. Xiang Long, XiangRui Yin, Xiao Li, Xiaolei Zhu, Xiaoying Jia, Xijin Zhang. Xin Liu, Xinchen Zhang. Xinyu Yang. Xiongeai Luo, Xiuli Chen, Xuantong Zhong, Xuefeng Xiao, Xujing Li, Yan Wu, Yawei Wen, Yifan Du Yihao Zhang, Yining Ye, Yonghui Wu, Yu Liu, Yu Yue, Yufeng Zhou, Yufeng Yuan, Yuhang Xu, Yuhong Yang, Yun Zhang, Yunhao Fang. Yuntao Li, Yurui Ren, Yuwen Xiong, Zehua Hong, Zehua Wang, Zewei Sun, Zeyu Wang, Zhao Cai. Zhaoyue Zha, Zhecheng An, Zhehui Zhao, Zhengzhuo Xu, Zhipeng Chen, Zhiyong Wu, Zhuofan Zheng, Zihao Wang. Zilong Huang, Ziyu Zhu, and Zuquan Song. Seed1.5e 1.5vl technical report. CoRR, abs/2505.07062, 2024.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. CoRR, abs/2501. 13826, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie. Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kap- pler, Daniel Levy. Daniel Selsam, David Dohan, David Farhi, David Mely. David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman. Hyung Won Chung, Ian Kivlichan, Ian O' O'Connlll, lan Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. Openai ol system card. CoRR, abs/2412. 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. CoRR, abs/2403.07974, 2024.
Carlos E. Jimenez, John Yang, Alexander Neetti,,hungg Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In ICLR. OpenReview.net, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv preprint, fbs/2408.033266 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mybench: A comprchensive multi-modal video under- standing benchmark. In CVPR, pp. 22195-22206. IEFE, 2024b.
Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text to video. In ICLR 2025. OpenReview.net, 2025.
Shicheng Li, Lei Li, Yi Liu, Shuhuai Ren, Yuanxin Liu, Rundong Gao, Xu Sun, and Lu Hou. VITATECS: diagnostic dataset for temporal concept understanding of video-language models. In ECCV (70), volume 15128 of Lecture Notes in Computer Science, pp. 331-348 Springer, 2024c.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR. OpenReview.net, 2024.
12

---

Published as a conference paper at ICLR 2026
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video lims really understand videos? In ACL (Findings), pp- 8731-8772. Association for Computational Linguisties, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng. Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR. OpenReview.net, 2024.
MAA. American invitational mathematics examination aime. In American Invitational Mathemat- ics Examination AIME 2024, February 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. In NeurlPS, 2023.
Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl Vondrick, Mikhail Sirotenko, Cordelia Schmid, and Tobias Weyand. Minerva: Evaluating complex video reasoning. arXiv preprint arXiv:2505.0068/ 2025.
OpenAL. Gpt-4o system card, 2024.
OpenAl. Introducing openai o3 and o4-mini, 2025.
Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. Introducing gemini 2.0: new ai model for the agentic era, 2024.
Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao. Ver-bench: A comprehensive evaluation framework for video thouugt reasoning. arXiv preprint arXiv:2504.07956, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proot q&a benchmark. CoRR, abs/2311 1.2022, 2023.
Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Ar- man Cohan. TOMATO: assessing visual temporal reasoning capabilities in multimodal founda- tion models. In ICLR. OpenReview.net, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuming Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang. Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li. Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5; Scaling reinforcement learning with 1lms. CoRR, htss/2501.1299) 2025a.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu. Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang. Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang. Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang. Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Meng- fan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng. Qizheng Gu, Runjie Zhou. Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song, Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang li,, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao,
13

---

Published as a conference paper at ICLR 2026
Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yulun Du, Yuxin Wu, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao, and Ziwei Chen. Kimi- VL technical report, 2025b.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In NeurIPS, 2024.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.151 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan 2025.
Qiu. Qwen3 technical report. arXiv preprint ar arXiv:2505.09335
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A GPT-4V level MILM on your phone. CoRR, abs/2408.01 000.0000......... 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models. CORR, abs/2408. 048400 2024.
Yuanhan Zhang, Jinming WU, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. CoRR, abs/2410.02713, 2024.
Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li. Junyang Song. Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan. MMVU: measuring expert-level multi misti--iseipline video understanding. CoRR, abs/2501. 2025.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding. CORR, abs/2406.04264 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui. Xuchui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang. Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang. Conghui Hc, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. In- ternv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
A MORE EXPERIMENTS
A.1 ANALYSIS OF LLM JUDGEMENT
To assess the robustness of using an LLM-based judge for evaluation, we conduct two complemen- tary analyses: (1) We paraphrase the MLLMs' answers and compare the evaluation results that of the original answers. (2) We replace our default judge model (Qwen2.5-72B (Yang et al. 2024)) with an alternative judge (Qwen3-235B-A225 (Yang et al., 2025)) and examine the consistency between their assessments. The results are presented in Table and Table g As we see, the performance of MLLMs remain highly consistent across both analyses, with average accuracy dif- ferences staying within 1%. This finding indicates that the evaluation is stable with respect both answer reformulation and judge model choice.
A.2 CORRELATION BETWEEN VIDEOREASON AND LMARENA PERFORMANCE
Since the videos in VIDEOREASONBENGE are designed as relatively controlled and "clean" scenar- ios, it is important understand whether performance on such scenarios generalizes to real-world visual reasoning. To examine this, we compare the performance of several frontier MLLMs on VIDEOREASON BENCH with their rankings on the LMArena vision leaderboard") where models are evaluated based on human preference votes over responses to user-submitted, real-world queries. As shown in Table 9 the results display a clear and consistent correlation. This finding suggests that strong performance TIDEOREASONBENC is reliable indicator of model's I broader and more generalized visual reasoning capabilities.
A.3 FURTHER EXTENSION OF STATE SIZE AND OPERATION COUNT
To further investigate the impact of scaling video content complexity, we extend both the state size and the operation count in the Number and Cup demonstrations using our automatic video and QA generation engine. As shown in Table TO and Table I1 model performance decreases either dimension grows. Notably, however, Demini-2.5-PP 2.5-Flash and Seed1.5- 1.5VL,, suggesting that it is more resilient to increases in video complexity. We view state size and operation count expansion as effective and convenient mechanisms for further increasing benchmark difficulty in future iterations.
exhibits a smaller relative drop than Gemini-
B MORE DETAILS OF VIDEOREASONBENCH
B. QUESTION TEMPLATES
Table 12 13 44
15 16
present the question templates for the six types of video demonstrations,
respectively. As provides a clear and precise description of the video demonstration along with the state transition rules, thereby eliminating ambiguity in task interpretation. (2) The Question states the query complete sentence and specifies the expected answer format. (3) The Answer Prompt instructs the models to summarize the final answer after the identifier phrase Final Answer:, which facili- tates easy extraction of the answer from the model's response.
see, the full prompt consists of three components: (1) The Task Instruction
B.2 EVALUATION PROMPTS
The evaluation prompt is illustrated in Table 18 and the prompts for extracting operations from the model response are shown Table 19
15

---

Published as a conference paper at ICLR 2026
Table Evaluation results with different answer formats.
Model
Answer
Order
State
Compare
Predict
Avg
Gemini 2.5-Flash
original
44,6
41.7
27.9
27.1
13.8
31,0
2.5-aal-.5--lashh
paraphrased
45.4
44.2
28.8
27.1
13.8
31.8
Gemini 2.5 Flash wio think
original
22.5
34.2
19.6
20.4
8.8
21.1
Deminn--2.5-Flashe wlo think
paraphrased
22.3
34.5
20.8
19.5
10.2
21.2
Seed1,5-VL.
original
24.2
27.1
3.8
7.9
3.8
13.3
Seed1.5-VL
paraphrased
24.2
26.7
3.8
3.8
13.3
Table 8: Evaluation results with different judge models.
Judge Model
Compare
Predict
Avg
Gemini-2.5-FFah
Qwea2.5-72B
44.6
41.7
27.9
27.1
13.8
31.0
Gomini-2.5--lass
Owan3-238B-A228
27.1
26.2
30.8
Gemini-2.5-Plan wlo think
Qwen2.5-728
22.5
34.2
19.6
20.4
8.8
21.1
Gemini- 2.5. Flash think
Qwen3-235B.A22
232B-A22B
22.5
32.9
19.6
20.4
21.0
Seed1.5-VV S-LL
Qwen2.5-72B
24.2
27.1
3.8
13.3
Seed1,5-VI.
Bwen3-2388-A2223
23.8
25.8
2.1
6.7
3.8
12.4
C MORE DETAILS OF EXPERIMENTAL. SETUPS
C.I INFERENCE CONFIGURATIONS
Table 23 summarizes the key inference settings for the MLLMs, including model version, input frames, generation temperature, and max tokens. Proprietary models are queried via official APIs at fps with a maximum frame limit. Open-source models run on internal 80 GB GPU clusters, with videos sampled at 32 frames for efficient models and 64 for flagship models.
C.2 VID2TXT BASELINE
Based on the ground-truth state transition sequences, we construct textual summary of key video information. The specific examples for the six types of video demonstrations illustrated in Table
20
22
C.3 HUMAN ANNOTATION
Figure illustrates the annotation interface used to establish the human performance baseline. An- notators are shown the video along with the complete question prompt prompt-excluding the Answer Prompt-identica what the evaluated models receive.
D LLM USAGE
We employ large language models (e.g., ChatGPT assist text polishing and grammar checking.
E LIMITATIONS AND FUTURE WORK
We acknowledge two primary limitations of the current work. First, the videos in VIDEOREASON- BENCH are relatively "clean"", and do not account for the broader perceptual challenges present in open-world scenarios, such as motion blur, occlusion, and other sources of visual ambiguity. Second, this work focuses on establishing rigorous evaluation benchmark and does not propose concrete methods for improving MLLM performance on the targeted reasoning tasks. Looking ahead, we see two promising research directions: (1) incorporating open-woee world videos to further test the robustness of video reasoning under greater perceptual stress and (2) constructing scalable video reasoning data construction pipeline and devising training objectives to enhance MI.LM's video reasoning capabil- ity.
https://chat.ppt.com
16

---

Published as a conference paper at ICLR 2026
Table 9: Performance VIDEOREASONBENCH and LMArena Vision Leaderboard (Nov 23, 2025).
Model
BIDEOREASONBENNH LMArena
Gemini-2.5-Pro
56.0
1249
Gemini-2..5FFaas
27.4
1214
04-mini
10.7
1202
Gemini-2.0-FFi
10.4
1169
GPT-4o
6.9
1119
Table 10: Results of varying state sizes for Number and Cup.
Model
3x3
4x4
5x5
6x6
Seed1.5- VL.
5.4
6.7
5.8
3.3
Gemini-2.5-Flast
25.8
19.2
7.9
6.7
Gemini-2.5-Pro
70.4
61.7
59.6
56.7
Table 11: Results of varying operation counts for Number and Cиp. The values in brackets denotes average video durations.
Model
5-9 (20.0s)
10-14 (32.4s)
15-19 (44.7s)
20-24 (57.1s)
25-29 (69.4s)
Secd
10.4
1.7
0.4
0.4
0.0
Gemini-2.5-Flast
25.8
19.2
9.6
5.8
5.0
Gemini-2.5-PP
68.8
63.3
61.2
55,4
51.7
Nextaing Sample Usi
The prrtant prrtarts di dlding partle on aad board The board fllld with numberrd aquares with ove emetr sorre vide log rale shoving the slerinn onoon ouegeee namber on rerembem atsal masked
cvertty pozza
1
2
Wher au tivits en tike sqaares being soveed for mnann square, proviee Miloennsssaatcc
2
1
7
onter trbti tranemoedd lnc, ie,
Theowediens menoo lenet 16.23.1.......
b
4
3
Anmunaation
6
5
Figure 6: Screenshot of the human annotation interface.
17

---

Published as a conference paper at ICLR 2026
Video
Full Prompt
(task instruction] {question} (answer prompt)
Task Instruction (Show at Begin): The video presents sliding puzzle on (N N} board. The board is filled with numbered squares, with one empty space. The vidco begins by showing the starting arrangement of the numbers the board. Then, the numbers are visually masked with blue overlay, and the puzzle undergoes series of movements-only the squares adjacent to the empty space can be shifted into Please carefully watch the video and answer the following question:
Task Instruction (Show at End): The video presents a sliding puzzle on a (N N board. The board is filled with numbered squares, with one empty space. Initially, all numbered squares visually masked with blue overlay. Then, the puzzle undergoes series of movements-only the squares adjacent to the emply space can shifted into The video ends by showing the final arrangement of the numbers on the board. Please carefully watch the video and answer the following question:
Answer Prompt: Provide summary of the final answer after "Final Answer:'
Questions
Recall Order: What are the {start_id) to {start_id] blue squares heing moved? For each moved blue square, provide the following details: The order in which it moved (e.g.. 1st, 2nd, 3rd, etc). The coordinates before was moved (e.g.. (a.1), (c,2), etc). The direction was moved (e.g., left, right, up, down)
Recall Count: How many times was the '(move)' move performed in the video? For each occurrence, provide the coordinate of the square (e.g., (a, (c.2)...) before the move.
Assuming the empty square is represented by '0', what is the arrangement numbers on the board at the (timestamp) of the video? Provide the coordinates of each square along with the corresponding number (e.g., (a, 3, (a,2): O, (b, I): 1, (b,2): 2).
Infer State:
Compare State: Assuming the empty square is represented by '0', compare the number arrangements the board at the start and end of the video. What the squares where the numbers differ between the boards? Provide their coordinates along with the corresponding number at the (timestomp) of the video (e.g.. (a,1): 3, (b,1): 1).
Predict State: If the arrangement of numbers the board currently in the state as it was at the (timestamp) of the video, and the following moves are exccuted: "[moves}', what will be the arrangement of numbers on the board? Assume that the empty square represented '0'. Provide the coordinates of each square along with the corresponding number (c.g., (a, I): (a,2): O, (b, ): (b,2): 2).
Predict Operations: If the arrangement of numbers the board is currently in the same state as was at the [timestamp) of the video, what sequence of moves (left, right, up. down) should be executed to achieve the desired number arrangement: 'number arrangementt Assume that the empty square is represented by '0', Note that moves cannot push any square beyond the board boundary.
Table 12: Prompt and question templates for the Number video demonstration.
18

---

Published as a conference paper at ICLR 2026
Video
Full Prompt jaakkinsttucttinn [question] (answer-prompt)
Task Instruction (Show at Begin): The video presents 'Tricky Cup' puzzle {N x N} board. The board filled with blue cups, each hiding either a yellow coin or nothing underneath. At the start, all cups are briefly lifted to reveal what's beneath them. Then, the cups begin series of moves-eac each move swaps the positions two cups, along with their hidden contents. Please carefully watch the video and answer the following question:
Task Instruction (Show at End); The video presents 'Tricky Cup' puzzle {N N} board. The board filled with blue cups, each hiding either a yellow coin or nothing underneath. Initially, the contents under the cups completely hidden. Then, the cups begin a series of moves-each move swaps the positions of two cups, along with their hidden contents. Toward the end, all cups are briefly lifted to reveal what's beneath them. Please carefully watch the video and answer the following question:
Answer Prompt: Provide a summary final answer after 'Final Answer:"
Questions
Recall Order: Assume that each time two cups swap their positions, it counts one move. What are the {start_id} to {start_id] moves shown in the video? For each move, provide the move number and the coordinates of the two cups that swapped positions. Format your response like this: 1st: (al, b2), 2nd: (c2, bl), 3rd: (a3, cl).
How many times were the cups in the row "(row_ide jdz)'' involved the swaps? For each instance, provide the coordinate(s) of the cup(s) before the swap occurred. Format your response like this: 1st: al, 2nd: a3, 3rd: (al,a2) (Use a single coordinate for individual cups, tuple for multiple cups involved in the same swap.)
Recall Count:
What are the positions of all the coins at the (limestamp) of the video? Provide the coordinates of each coin (e.g., a2, b1, c3).
Infer State:
Compare State: Compare the distribution of contents beneath the cups the start and end of the video. What are the positions where the contents beneath the cups differ between the two boards? Provide their coordinates along with the corresponding content the (timestamp) of the video. Format your response like this: al: empty, b3: coin.
Predict State: If the distribution of coins on the board is currently in the same state as it was at the (timestamp) of the video. and the following cup swaps are executed in order: "[moves}", what will be the new distribution of the coins? Provide the coordinates of the coins (e.g., al, b2).
Predict Operations: the distribution coins on the board currently the same state it was at the {timestamp) of the video, what sequence of cup swaps should be executed achieve the desired distribution of coins: 'board"? Format your response list of coordinate pairs, such as: (al, b2), (c3, bl). Each pair represents single swap between two cups.
Table 13: Prompt and question templates for the Cup video demonstration.
19

---

Published as a conference paper at ICLR 2026
Video
Full Prompt (task_instin jisskinstruction [question]
(answer premmtt
Task Instruction (Show at Begin): The video presents a (N N) grid. At the beginning the video, all positions on the grid are filled with either black or white piece. Then, these pieces are visually hidden but still remain in their original positions. A red circle then moves across the grid. Each time the red circle passes by position the grid (excluding the starting position), the color of the piece that position #and* the colors of its immediate orthogonal neighbors (up. down, left, and right) flipped: black becomes white, and white becomes black. Note that diagonal neighbors are *not* affected. Neighbors are only considered if they exist within the grid's houndaries. Please carefully watch the video and answer the following question:
Task Instruction (Show at End): The video presents a {N x N} grid. All positions on the grid are filled with either black white piece. These pieces are visually hidden at the beginning the video. red circle then moves across the grid. Each time the circle passes by position on the grid (excluding the starting position), the color the picce that position *and* the colors of immediate orthogonal neighbors (up. down, left, and right) flipped: black hecomes white, and white becomes black. Note that diagonal neighbors are "not" affected. Neighbors are only considered if they exist within the grid's boundaries. The video ends by showing the final arrangement of black and white pieces on the grid. Please carefully watch the video and answer the following question:
Answer Prompt: Provide a summary of the final answer after 'Final Answer:"
Questions
Assume that each time the red circle moves from one grid intersection What are the directions (left, right, up, down) of the (start_id} to (stort_id) moves made hy the red circle in the video? List them in order.
Recall Order:
adjacent (horizontally or vertically), it counts
Recall Count: Assume that each time the red circle moves from one grid intersection to an adjacent one (horizontally or vertically), counts as one move. Given the movement direction '(move): J", how many times does the red circle perform this move? For each oc- currence, provide the coordinate of the position before the move (e.g., (a, I), (c,2), etc).
Infer State: What is the arrangement of black and white pieces on the grid at the (timestamp) the video? Provide each piece's 's coordinates and color using format: (column, row): color (e.g.. (a. black, (c,2): white).
Compare State: Assume that each time the red circle moves from grid intersection to adjacent (horizontally vertically), it counts ment of black and white pieces the grid at the start and end of the video. What are the coordinates where the piece color differ between the two grids? Provide these coordinates along with the corresponding piece color at the [limestomp] of the video, using the format: (column, row): color (c.g.. (a.1): black, (c.2): whitc).
Compare the arrange-
Predict State: Assume that each time the red circle moves from one grid intersection to an adjacent one (horizontally or vertically), counts as one move. the arrangement black and white pieces and the position the red circle on the grid is currently the same state as it was the (timestanp) of video, '[moves]', what will be the arrangement of black and white pieces the grid? Provide each piece's coordinates and color using the format: (column. row): color (e.g.. (a.1): black; (c.2): white).
the following moves are executed:
Predict Operations: Assume that each time the red circle moves from one grid inter- section to an adjacent one (horizontally or vertically), it counts as one move. The red cir- cle cannot move beyond the grid boundary. If the arrangement of black and white pieces and the position of the red circle on the grid is currently in the same state as it was at the [timestomp) of the vidco, what sequence moves (left, right, up, down) should executed by the red circle to achieve the desired arrangement of black and white pieces: "[board) ?? List them order.
Table 14: Prompt and question templates for the Circle video demonstration.
20

---

Published as a conference paper at ICLR 2026
Video
Full Prompt
(task instruction]
[question]
(answer prompt)
Task Instruction:
The video demonstrates a series of file manipulation commands executed in the Linux command line. To ensure accurate understanding. note these assumptions: * 'touch' commands: All files created by 'touch' do not exist target dirce try prior to the command's exccution.
* 'rm -r' commands: All files deleted by 'rm -r' do exist in the target directory prior
the command's execution.
"cp" and 'mv' commands: All souree files used by "ep" and 'mv' do exist in the source dircctory prior to the command's ' exccution.
* The destination path for 'cp' and 'mv" commands does not contain the target files prior to the command.
Please carefully watch the vidco and answer the following question:
Answer Prompt: Provide a summary of the final answer after "Final Answer:"
Questions
Recall Order: What are the (start.id) to (start.id) (cmd_type) (ecittyyeccooooo commands shown in the video? Provide the order of each command (e.g., 1st, 2nd, 3rd, etc) along with the command content.
Recall Count: How many different {file files were involved in the fomddtypeeecee commands throughout the vidco? Provide the file count along with the specific file names (c.g.. 'ixt" files: a.tx,, b.txt).
Infer State: At the (timestamp) of the video, how many (file_ files remain in Ypoth_name) Provide the file count along with the specific file names (c.g., 'txxt' files: a.txt, b.tx)).
Compare State
What files were in 'path0/' at the start of the video, but were not there at the end of the video?
-What files were 'path0/' at the end of the video, but were not the video?
start After the command "(ema)" executed, what files were in "(poth namel) but were not [path_name?)]
Predict State: If the paths currently contain exactly the same files as they did at the (timestamp) of video, and we run command '(amd)', which (file type) files would be in [path_name]"?
Predict Operations: If the paths currently contain exactly the same files as they did the [timestamp) of the video, to ensure that "(path name]' contains exactly the following files: "[files)'. what sequence of commands should be executed? Rules: You may only use the commands 'touch' and 'rm -ff. 2. You may use most two commands. 3. Files specified in 'touch' must not appear in 'rm -rf" command, and vice versa (i.e., no overlap). Response Format: If multiple commands used, separate them with & For example, 'touch path0/(a.txe b.xt) rm -If (c.hh///ccpyyddjsoo.
Table 15: Prompt and question templates for the File video demonstration.
21

---

Published as a conference paper at ICLR 2026
Video
Full Prompt
(task instruction)
[question] (answer-prompp
Task Instruction (Show at Begin): The video showcases a sequence of operations involving
piles of cards. begins by displaying the initial arrangement of cards in each pile from top to bottom. The cards are then turned face down, after which a series of actions is carried out. Note that there are only two types of actions: adding one card to the top of the pile or removing one card from the bottom of the pile, Please carefully watch the video and answer the following question
Task Instruction (Show at End): The video showcases sequence of operations involving one or more piles of cards. Throughout the video, before the final reveal of each pile, only adding one card the top of the pile removing card from
types of actions the bottom of the pile. Then, the video ends by displaying the final arrangement of cards in each pile from top to bottom. Please carefully watch the video and answer the following question:
Answer Prompt: Provide a summary of the final answer after 'Final Answer:'
Questions
Recall Order: What are the {start_id) {start_id} cards being (action_type) any pile throughout the video? For each card, provide the following details: 1. The order (e.g.. 1st or 2nd) 2. The suit and value (e.g.. of Hearts) 3. The pile involved (e.g.. pileO, pilel) Format your response like this: 1st: 6 of Hearts, pileO 2nd: Jack of Spades, pilel.
Recall Count: How many cards were {action type] "(pile name)' throughout the video? For each card, provide its suit and value (e.g., of Hearts) Format your response
like this: 2 cards: 6 of Hearts, King of Clubs.
Infer State: At the (timestamp) of the video, what cards are in "(pile name}?? List them in order from top bottom, including both the value and suit of each card. Format your response like this:
Hearts, King of Clubs, 3 of Spades.
Compare State: What cards were in "oillename]+ at the (timestamp) of the video, but were not there at the [timestamp2] of the video? For each card, provide its suit and value. Format your response like this: 6 of Hearts, King of Clubs, of Spades.
Predict State: If the piles currently contain exactly the same cards as they did at the (timestamp) of the video, and now perform these actions in order: "(actions): What cards would be in {pile name)-? List them in order from top to bottom, including both the value and suit of each card. Format your response like this: 6 of Hearts, King of Clubs, 3 of Spades.
cards they did the (timestamp) of the video, to ensure that "(pile_ "all_nnmme" contains exactly the following cards from top bottom: "fcards)* what sequence of actions should be performed? Rules: Each action must either add a card to pile or remove a card from a pile. 2. You may only add cards the top of pile remove cards from the bottom of pile. Response Format: List the actions sequence, specifying the action, card, and pile. Separate each action with comma. For example, 'add 6 of Hearts to pileO, remove King of Clubs from pile0*
Predict Operations: If the piles currently contain exactly the
Table 16: Prompt and question templates for the Card video demonstration.
22

---

Published as a conference paper at ICLR 2026
Video
Start Revaalng Chips
End Rewaling Chips
Full Prompt
(task instruction) [question] (answer prompp]
Task Instruction (Show Begin): The video showcases a sequence operations involving one or more cup(s) and chips. It begins by showing the initial chips contained each cup. Then, a series of actions are carried out. Note that there are only two types of actions: adding one chip to cup or removing one chip from cup. Please carefully watch the video and answer the following question:
The video showcases a sequence of operations involving one or more cup(s) and chips. Throughout the video, before the final reveal of chips con- tained in each cup, only two types of actions occur: adding one chip to cup or removing one chip from cup. Then, the video ends by displaying the final chips contained in each cup. Please carefully watch the video and answer the following question:
Task Instruction (Show at End):
Answer Prompt: Provide a summary the final answer after "Final Answer:'
Questions
Recall Order: Disregarding the process of revealing all chips in the cup(s), what are the (start.dd id) {start_id) chips being {action type] any cup throughout the video? For each chip, provide the following details: 1. The order (e.g., 1st or 2nd) 2. The value (e.g.. 20) The cup involved (e.g., cup0, cupl) Format your response like this: 1st: 100, cupŌ 2nd: 20, cup1.
Recall Count: Disregarding the process of revealing all chips in the cup(s), how many chips were faction type) '(cup name)' throughout the video? For each chip, provide its value (order does not matter). Format your response like this: 4 chips: 20, 5, 100, 100.
Infer State: At the (timestamp) of the video, how many chips were in '(cup name)"? For each chip, provide value (order does not matter). Format your response like this: chips: 20, 5, 100, 100.
Compare State: At which point in the video the total value of chips higher, at [timestampl] } (timestamp2]? Also, what is the difference in value be- tween the two times? Format your response like this: "time with higheh.ghhrrlllll value". "differ- ence_in_td valuue" (e.g.. start, 115).
"
Predict State: If the cups currently contain exactly the same chips as they did the (timestamp) of the video, and now we perform these actions in order: "factions}: How many chips would in '(cup.name)-" For each chip, provide its value (order does not matter). Format your response like this: 4 chips: 20, 5, 100, 100.
Predict Operations: If the cups currently contain exactly the same chips as they did the (timestamp) of the video, to ensure that '(cup name) contains exactly the following chips: '[chips)' (order does not matter), what sequence of actions should be performed?
Rules: 1. Each action must either add a chip to a cup or remove chip from a cup. 2. Available chips for addition are: 10, 20, 50, 100. You may only remove a chip if already present in the cup. Response Format: List the actions in sequence, specifying the action, chip, and cup. Separate each action with comma. For example, 'add 20 cup0, remove 50 cupO
Table 17: Prompt and question templates for the Chip video demonstration.
23

---

Published as a conference paper at ICLR 2026
You will be given question, a model response and ground-truth answer. Your task determine whether the model response is correct based the ground-truth answer. The model response should contain all information in the ground-truth answer.
Question: {question)
Model Response: {response) Ground-Tree Truth Answer: {ground
Directly output "Correct" or "Incorrect":
Table 18: Prompt used for LLM-based evaluation.
Number & Circle: You will be given model-generated response describing sequence of movements. Your task to extract the movements the order they appear and return them list (e.g., I'left', 'up', 'down', 'right']). J) Model Response: (response) Extracted Movements:
Cup: You will be given model-generated response describing sequence of cup swaps.
Each swap is represented as pair of coordinates for example, (al, b2) b2)-indicating the positions being swapped.
Your task: Extract all coordinate pairs from the response in the exact order they appear, and return them list of tuples.
Format your answer like this: [('al,, 'b2"), ('cl", 'b1"), ('a3', 'b2")]
Model Response: (response)
Extracted Swaps:
File: You will be given a model-generated response regarding file operation command in Linux system.
Your task: Identify and extract only the actual command from the model response, removing any irrelevant descriptive text.
Model Response: (response)
Extracted Command:
Card: You will be given model- generated response describing sequence of operations performed to cards. Each operation either adds or removes a card from pileO or pilel. Your task:
Extract all valid operations and return them as a list of strings.
Each operation must involve either adding removing card or from pileO pilel. If no valid operations are found, return an empty list ([I)-
Format your answer like this: ['add 6 of Hearts to pileO", 'remove King of Clubs from pileO"]
Model Response: (response)
Extracted Operations:
Chip: You will be given model-generatee response describing sequence of operations involving chips and cups. Each operation either adds or removes a chip from cupO or cupl. Your task:
Extract all valid operations and return them as a list of strings.
Each operation must involve either adding or removing chip to or from cup0 cupl.
If no valid operations are found, return an empty list ([)).
Format your answer like this: a'add 20 to cup0', 'remove 50 cupO"]
Model Response: {response]
Extracted Operations:
Table 19: Prompts used for operation extraction.
24

---

Published as a conference paper at ICLR 2026
Video (Number)
Converted Text:
State I:
(a,1); 2, (a.2): 1, (a.3):
(b,1): 4, (b,2): 3, (b,3): 0 (empty)
(c,1): 6, (c.2): 5, (c.3):
State 2:
(a,1): 7, (a,2): 2, (a,3): (empty)
(b,1): 2, (b,2): ?, (b,3): ?
(c,1): 2, (c,2): 2, (c,3):
State 3:
(a,1): 2, (a,2): (empty), (a.3): (b,1): 2, (b,2): ?, (b.3): ?
(c.1): 2, (c,2): 2, (c.3): ?
State 4:
(a,1): 2, (a,2): 2, (a.3): ?
(b,1): 2, (b,2): 0 (empty), (b.3): ?
(c,1): 7, (c,2): ?, (c.3):
State 5:
(a,1): 2, (a,2): 2, (a,3):
(b,1): 2, (b,2): ?, (b.3): ?
(c,1): 2, (c,2): 0 (empty), (c.3):
State 6:
(a,1): 2, (a,2): 2, (a.3):
(b,1): 2, (b,2): ?, (b.3): ?
(c.1): 2, (c,2): 2, (c.3): (empty)
Video (Cup)
Converted Text:
Board state the start of the video: (a,1): coin, (a,2): empty, (a.3): empty (b, I): coin, (b,2): empty, (b,3): empty
(c,1): empty, (c,2): coin, (c.3): coin
Swaps:
Swap (b,2) (a,1)
Swap 2: (a.2) to (c,2)
Swap 3: (b,2)
(c.1)
Swap 4: (b,1) to (c,3)
Swap 5: (a.2) (c.3)
Table 20: Illustration of how to convert key video information into text ("vid2txt") for Number and
Cup videos.
25

---

Published as a conference paper at ICLR 2026
Video (Circle)
Converted Text:
State 1:
(a,0): white, (a, I): black, (a,2): white (circle)
(b,0): white, (b,1): white, (b,2): black
(c,0): white, (c,1): white, (c,2): white
State 2:
(a,0): ?, (a,1): (circle), (a.2):
(b,0): 2, (b,1): ?, (b,2): ?
(c,0): ?, (c,1): 2, (c.2):
State 3:
(a,0): 7, (a,1): 2, (a,2): ?
(b,0): 2, (b,1): ? (circle), (b.2):
(c,0): ?, (c,1): 2, ?
State 4:
(a,0): 2, (a,1): 2, (a.2):
(b,0): 2, (b,1): ?, (b,2): ?
(c,0): ?, (c,1): (circle), (c,2):
State 5:
(a,0): 7, (a,1): 2, (a.2): ?
(b,0): 2, (b,1): ?, (b,2): ?
(c,0): ?, (c,1): 2, (c,2): (circle)
State 6:
(a,0): ?, (a,1): 7, (a,2):
(b,0): ?, (b,1): ?, (b,2); ?
(c,0): ?, (c,1); (circle), (c,2):
Video (File)
Converted Text:
>>>touch oatpl//lwaaauuuaoaoooooooossssse
>>>>s patho
pathO: v.py m.txt g.txt h.esv n.esv d.csv o.ix x.Ixt y.xt p.py b.esv v.esv
>>>m -rf patt///ggxxtppphcxxxn..))
>>> >ouch patho//.sssugbonnppppxxxxxxxxxsp
>>>m -rf patho//1.px.o.xxt]
>>>tuuc http://w....aa...oooooo.pppppppp
>>>touch patto///cccv.jjoonttxt...oo
Table 21: Illustration of how convert key video information into text ("vid2txt") for Circle and
File videos.
26

---

Published as a conference paper at ICLR 2026
Video (Card)
Converted Text:
Reveal the cards at the start of the video:
pileO (top->bottom): King of Diamonds, of Diamonds, of Diamonds, 8 of Clubs, 7 of Spades
Actions:
Action I: add Queen of Clubs to pileO
Action 2: add Ace of Hearts to pileO
Action 3: add Jack of Diamonds to pileO
Action 4: add Jack of Clubs to pile0
Action 5: remove 7 of Spades from pileO
Video (Chip)
Revoaling
Converted Text:
Reveal the chips at the start of the video: cup0: 20, 10, 100
Actions:
Action 1: add 100 to cup()
Action 2: remove 100 from cupO
Action 3: add 10 to cup0
Action 4: remove 10 from cupO
Action 5: add 5 to cup0
Table 22: Illustration of how to convert key video information into text ("vid2txt"") for Card and Chip videos.
27

---

Published as a conference paper at ICLR 2026
Table 23: Inference configurations for the evaluated MLLMs. "Ifps/N" indicates that videos with duration < N seconds are processed ffs,, while for videos longer than N seconds, N frames are uniformly sampled. "The temperature set to 0.0 by default and increased to 1.0 if the response exceeds the "Max New Tokens" limit-this adjustment is applied in token count experiments to prevent excessively long responses with repeated tokens.
Input Frames
Max New Tokens
Model
Version
Temperature
Proprietary Models
GPT-4o
gpt-4p-2222--11-2
lfps/50
1.0
8,192
o4-mini
o2-min1-2025-04-22
Ifps/50
1.0
8.192
Socd1.5-VL.
Ifps/128
0.0
16,384
Gemini-2.0-Flash
gemini-2.0-fLash
lips
0.0
4,096
Gemini-2.5-Flaa
lfps
0.0/1.0'
65.536
Gemini-2.5-Pro
líps
0.0/1.0*
65,536
Open-source Models
Efficient Models
mPLUG-Ow13
ABLIG-OW13-1B-2200/2
32
1.0
1.024
MiniCPM-V
MiniCEM-V-2.6
32
MinCPM--o24 2.6
MiniCTM-O-2.6
32
1.0
1,024
LLaVA-OOeeiisoon-7
32
0.0
2,048
LLaVA-Vides-7B
Llava-video7T--xeenl
32
0.0
2,048
InternV1.3-8B
InternVL.3-B
32
4,096
Qwen2.5-VL--B
jwen2-5-Vl-78-Inttrm
32
0.01
4,096
Kimi-VL-A3B
Kimi-WWL-A3B-Tnsttrre
32
1.0
16,384
Flagship Models
LiVV--nneissoon722
64
0.0
2,048
LLaVA-VVdee-T Video-72B
LLaVA--V:dee-72B-Qeen
64
0.0
2,048
InternV1. 7187788
InternVi.3-788
64
0.0
4,096
Qwen2.5 VL--72B
Qwen2 www.5---7D--nss-uue
64
0.01
4,096
28


Reviews:
[{"review_id": "2No7sxOjG0", "reviewer": "Reviewer_mxvW", "summary": "The paper introduces a vision-centric benchmark for complex video reasoning. By revealing latent states only at the beginning or end of the video and presenting a sequence of visible actions, the benchmark systematically evaluates three progressive levels of reasoning: (Level 1) exact recall of the action sequence and counts; (Level 2) inference of invisible states from observed actions and comparison of differences; (Level 3) counterfactual prediction of future states or required actions. The benchmark spans six scenario types (numeric sliders, board flipping, cup–lid swapping, file operations, card stacks, chip–cup), comprising 240 videos and 1440 QA pairs, with evaluation via an LLM-based textual judge and executable simulation.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work effectively identifies a key deficiency in current video benchmarks: weak alignment with reasoning and low demand for exploiting video information. For instance, Table 2 shows that using raw video input underperforms a video-to-text pipeline, and Table 3 indicates that prior benchmarks can be partially solved with text-only inputs.\n2. As a benchmark for assessing MLLM reasoning, it demonstrates a positive correlation between accuracy and scaled reasoning length, which aligns with expectations for reasoning-focused evaluation.", "weaknesses": "1. The task format is relatively narrow, covering only six highly customized scenarios. If models are exposed to similar data during training, they may quickly overfit or “hack” the benchmark.\n2. The substantial gap between open-source and closed-source models in Table 2 warrants analysis. Is this driven by differences in training data coverage or by capability gaps? For example, open-source models may have been trained on fewer or less diverse data and thus not encountered similarly customized scenario tasks, leading to lower performance.\n3. In Fig. 5(b), the largest thinking budget (8192) yields the shortest response length. Please explain this phenomenon, e.g., by providing illustrative cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fGL0WdEkcd", "reviewer": "Reviewer_tQvt", "summary": "The paper presents VideoReasonBench, a vision-focused benchmark that emphasizes step-wise reasoning over long sequences of visually observable operations on partially observable latent states such as sliding-tile board states. It defines three increasing skill levels: recall for fine-grained operation perception, infer for latent state inference, and predict for counterfactual planning. It includes six demonstration types: Number, Circle, Cup, File, Card, and Chip. The dataset contains 240 videos and 1,440 questions, with structured prompts and automatic answer generation based on state-transition scripts. Evaluation uses a text-only LLM judge; for Predict Operation, it simulates model-proposed operations to verify target states. Among 18 MLLMs, most models perform poorly with less than 10 percent accuracy overall. Gemini-2.5-Pro in thinking mode reaches 56 percent, and the benchmark shows strong sensitivity to thinking budget and heavy reliance on vision compared to earlier video benchmarks.", "questions": "How consistent is the LLM-as-judge across paraphrased or differently formatted answers?\n\nCan you analyze the potential bias introduced by the six procedural video types and clarify whether these discrete settings generalize to naturalistic visual reasoning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "The benchmark covers both synthetic and real-world capture paths, including Matplotlib programmatic renderings, terminal screenshots, and manual videos, with balanced distributions across skills and demos and controlled settings for state size and operation length to adjust difficulty.", "weaknesses": "W1: For five of six skills, correctness is determined by a text-only LLM judge given GT and the model output. Even with careful prompts, LLM graders can show bias or instability and may prefer format-matched answers over truly correct content. The paper does not report inter-judge agreement, self-consistency, or adversarial sensitivity such as paraphrase, verbosity, or distractors. This is quite concerning.\nFor Predict Operation, answers are extracted using an LLM, then simulated. This introduces another potential failure point in extraction, but there is no error analysis distinguishing extraction failures from planning failures.\nvid2txt is generated from the ground-truth state transitions. It works well for isolating perception versus reasoning, but it effectively provides a clean event log the models do not need to perceive. It is possible that the performance gap partly reflects a best-case text abstraction that few real video systems could realistically obtain.\n\n\nW2: The authors stated that the videos and questions in these benchmarks often resemble those in earlier benchmarks and fall short in demanding deeper video reasoning. It is unclear how this dataset helps address that issue. It is also unclear in what way the previous ones resemble.\n\n\n\nW3: According to the paper, all videos are synthetic or semi-synthetic demonstrations built from controlled state-transition scripts rather than natural videos.\nIt seems the dataset favors discrete and symbolic reasoning. All operations appear atomic and reversible, meaning the benchmark strongly favors models that can internally represent counting, position updates, or ordering. This raises the concern that it may overfit to symbolic reasoning rather than perceptual or intuitive physics reasoning.\nIt may also underestimate challenges such as motion blur, occlusion, object continuity, or ambiguous visual events found in open-world settings. But this is a minor concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e6QbfQHOgD", "reviewer": "Reviewer_qkUM", "summary": "This paper introduces a diagnostic benchmark for video reasoning that treats each clip as a sequence of visible operations acting on a partially hidden state. Tasks span perception, latent-state inference, and counterfactual prediction across both synthetic and real scenes with programmatically generated ground truth. Evaluations show current multimodal models struggle, especially when fine-grained temporal perception and long-horizon reasoning are required; converting videos to textual state–operation traces and allowing longer chains of thought markedly narrows the gap. The paper highlights perception bottlenecks, the value of explicit reasoning, and the need for methods that integrate stronger visual tracking, external memory, and executable reasoning.", "questions": "Please refer to the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark formalizes video as “latent state + a sequence of visible operations,” decomposed into three reasoning levels with six concrete skills, which makes failure modes inspectable and comparable.\n\n2. The study evaluates a wide span of open-source MLLMs", "weaknesses": "1. Since most tasks are scored with a judge LLM, would you consider reporting a brief robustness study (e.g., prompt ablations, temperature sweeps, and an alternative judge model) and an agreement statistic?\n\n2. To make cross-model comparisons easier, could you add some description about budget setting, such as fps/frame count and max tokens?\n\n3. The results indicate accuracy drops as state size and operation count increase, and when the final state is only revealed at the end. What may happen if we extend scenarios to minutes-scale durations or larger state spaces?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KEVzaBVgru", "reviewer": "Reviewer_4Dkk", "summary": "This paper proposed a video question answering benchmark focusing on the visual reasoning ability of the MLLMs. In specifice, the videos are evenly created for six tasks and the question requires the compherehensive understanding to answer. Experiemnts shows that existing MLLMs struggles in answering the questions, and enableing thinking ability significantly boosts the models' performance.", "questions": "● The comparison of vid2txt setting in table 2 seems suggensting that failure of current MLLMs are mainly caused by the percetion failure rather than reasoning failure as the performance gap between vid2txt setting and traditional VidQA setting is significantly larger than that between models w./w.o. thinking. Therefore, the main challenge of this benchmark is perception rather than reasoning. How do author explain this performance gap and its contradictory with the main claim?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "● The paper is well motivated. Evaluating the visual reasoning ability of MLLMs can be crutial. \n● The proposed benchmark is of reasonable size and distribution.\n● Experiments verified the proposed benchmark is challanging for existing MLLMs, and performance gain of CoT supports the claim that the questions require visual reasoning to solve.", "weaknesses": "● The proopsed data construction pipeline is highly specialized for the proposed types of questions, and can hardly generalize to boarder question types.\n● In table 2, the experiment shows that vid2txt setting significantly outperforms traditional VidQA setting (over 48% on gemini-2.5-flash). This somewhat violates the claim that the benchmark is challanging the visual centric reasoning ability of MLLMs as this implies that the perceptual ability seems playing a more important role  than reasoning for the low performance of current MLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)