VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Video
Authors: Hanoona Abdul Rasheed
Abdelrahman M Shaker
Anqi Tang
Muhammad Maaz
Ming-Hsuan Yang
Salman Khan
Fahad Shahbaz Khan
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Reasoning
Video Question Answering
Mathematical Understanding
Temporal Reasoning
Visual Grounding
Primary area: datasets and benchmarks
TL;DR: Mathematical Reasoning using Video MLLM Benchmark

Abstract: Mathematical reasoning in real-world video presents a fundamentally different challenge than static images or text. It requires interpreting fine-grained visual information, accurately reading handwritten or digital text, and integrating spoken cues, often dispersed non-linearly over time. In such multimodal contexts, success hinges not just on perception, but on selectively identifying and integrating the right details from a rich and noisy stream of content. To this end, we introduce VideoMathQA, a benchmark designed to evaluate whether models can perform such temporally extended cross-modal reasoning on videos. The benchmark spans 10 diverse mathematical domains, covering videos from 10 seconds to over 1 hour. We employ graduate-level experts to ensure high quality, for over 920 man-hours of annotation. To reflect real-world scenarios, questions are designed around three core reasoning challenges: direct problem solving, conceptual transfer, which requires applying learned methods to new problems; and deep instructional comprehension, involving multi-step reasoning over extended explanations and partially worked-out solutions. Each question includes multi-step reasoning annotations, enabling fine-grained diagnosis of model capabilities. Through this benchmark, we establish an evaluation framework for models that must reason, rather than merely perceive, jointly ground concepts across visual, audio, and textual modalities, across temporally extended mathematical problem settings.

Published as a conference paper ICLR 2026
VIDEOMATHQA: BENCHMARKING MATHEMATICAL REA- SONING VIA MULTIMODAL UNDERSTANDING IN VIDEOS
Hanoona Rasheed Abdelrahman Shaker',Ann Tang', Muhammad Maaz!
Ming- Yang?.; Salman Khan Fahad Shahbaz Khan 1,5
MBZUAI University of California Merced &Google Research
Australian National University $ Liinköping University
https:/ http::://bbzui-oryy/ github. 1c wwt.iiiiiiiiiaaattoooooo
ABSTRACT
Mathematical reasoning in real-world video presents fundamentally different challenge than static images or text. It requires interpreting fine-grained visual information, accu- rately reading handwritten or digital text, and integrating spoken cues, often dispersed non-linearly over time. In such multimodal contexts, success hinges not just on perception, but on selectively identifying and integrating the right details from a rich and noisy stream of content. To this end, we introduce Matho. benchmark designed to evaluate whether models can perform such temporally extended moos--modal reasoning The benchmark spans 10 diverse mathematical domains, covering videos from 10 seconds to over 1 hour. We employ graduate-level experts to ensure high quality, for over 920 man-hours of annotation. To reflect real-world scenarios, questions are designed around three reasoning challenges: direct problem solving, conceptual transfer, which requires applying learned methods to new problems; and deep instructional comprehension, involv- ing multi-step reasoning over extended explanations and partially worked-out solutions. Each question includes multi-step reasoning annotations, enabling fine-graines diagnosis of model capabilities. Through this benchmark, we establish an evaluation framework for models that must reason, rather than merely perceive, jointly ground concepts across visual, audio, and textual modalities, across temporally extended mathematical problem settings.
videos.
1 INTRODUCTION
Unlike mathematical reasoning benchmarks for static-images, videos present unique challenges. In educa- tional and instructional videos, key information is conveyed through evolving diagrams, handwritten or digital notations, and spoken explanations that unfold non-linearly across time. reasoning model must therefore sift through high-resolution frames, align visual representations with subtitles or voice-over, and integrate disparate cues into a coherent problem-solving pipeline. This perell-in-a-multimonn aaultimmddl-haaystakk problem requires not only accurate perception (e.g., frame-aware OCR of equations) but also precision in symbolic manipulation and multi-step inference, where missing a single visual cue can lead to incorrect conclusions.
Existing benchmarks for mathematical reasoning (e.g., MathQA (Amini et al. 2019). ChartQA (Masry et al. 2022), and Math Vista (Lu et al.) (024)) have driven substantial progress in image-based and text-based settings. However, they typically evaluate models static diagrams, printed formulas, or single-turn queries and lack support for temporally extended contexts, dynamically evolving visual content, and do not offer true integration of vision, audio, text and background subject knowledge. Some recent efforts explore video question answering in general domains (Fu et al.] [0255] Mangalam et al. 2024 Li et af.) 2024d), but they do not target the precise, multi modal, multi-step inference demanded by mathematical problem solving.

---

Published as a conference paper at ICLR 2026
Problem Focused
Concept Transfer
Deep Comprehension
minutes)
(18 minitee)
(R2 minuter)
Question: Usiing data from the video on cost undorwest all pescedures,
Question: Deternirw the numbrr at possible trisngias example.
Question: tases on lecturo, can you complete
wouls be the total cost, and how each sore would that be compared to having the same procedures in Spain? sefect The closest patn tn lish, the onders awerage
treatworts
[centact: Deecrstrated netrod approach nust appllied to a stellar but neuly posed prohles. Locating the correct rigure is vitall to answering the questiio correctly.] ]
Requires understanding the context, Lcuntify what is being askes, interpret the partially sorved out solution the video, and coap' ete soluttor..
[Contest::
Spatn??
Category: Chart Reatiigg cpttinss 4) 778e5, 60229 B) ag000, 32000 () 77003, 50020 0) 76465, 51893
Category: counting Opttines
Casagory: Aritheetic and Calculus opcions:
F) 92400, 67728
A) 29 5) 28
()
14
03 30
27
A) 38/10) 88/3 e) 308/3
D) 160/3 F) 230/3
Steps:
3) Early video, rocus on healthcare spensing
The viceo shove that to count trianeles a box,
1) The Tecturer sets up as exsette probles to
wultiply largest
calculate the double incegral
pahl petvate funding 10-10-11 J. Toward
nutter 10--1--0::00-- tn explates that
wty ractingllar
the raddle, the exphisis shiftts to the cost of Lndividial teeateents, which is kay to arowering this quectioo [01:29]-
that joaned boves create extra triangles at the
te defieed [1,2] 1.4). This
juictions, which mist be included in the tosal (17:16-08:14).
curessee f[aty over x 2
11-31-19-21)--
23 Video shows series of bar charts conparing everage cast treatrants us other counteles (11225-01:32)--
23 The problio is to tand the neber or triangtes in
[e1 (2))
Fipire composed of euttiple square bowes
with respert to (teesing fieed) the
lecturer oxplairs for this restamauiar
intsgration
33 n.s. treatment cosrs
fotlinu: ingieplacty
Fach squrre haw festui dua ty contatin
with respect to which conresgonds to dividing the denali into forizoncal 'slitters' [10
$31,,00 (et: 28), hip Replacoment $35, 109 (Co pills, 20ng) $250 (R1:12]-
triangles, determined by writing numbers
(63:30],
miliislying
21:17)-
for three vercically conrected bowes, each
The order of integration is matter of choace and doss not offect the result.
triangles, totaling traangles.
5) rowarment rosts anaas tegtiop asty
treateents: $32,000 $20,180 $15, 200 $285 577,965,
5) when sooes joined, as adeitionna
the iotning 10:16]- with thrse boves, there two such
constant. The integral wtth respect from 1 to is cosputed (20125-00) se].
integrace
$7,000 (01:26]. Hip Replacenent to, see [01:24],
joining areas, contetharing stcttions triingles.
section $2, (01)20), Destostin 400
integral
respect
22).
6) This, the total from the three boves and their conrections is 24 (fron intividual toxes) (tree conrections
torturting from
7/%
6) Tocal healtheare cost in Spain for the stme treatments: 57, 900000- $0,500 22, 400 $20
this rasust ruttspized by yf, giving 7/3 yt. now integrate with respect to y from
28 triangles.
23 Adtitionally, whan thrse bowes are connected, large trtangle is forned arross three boces, trtangle (11:22].
7) The integrat of yd wet respect to io yly.
7) Oiffarence in cost between U.S. spain: gra, .00.
tvaluoting froe 1 to gives 51/4 1/4 - 28.
costrituting
muttiplytin
gives 548/3, Thus,
83 Fanms amser:: $77, 861 in the U.S., and the cost
Therwtore, the tanal count is (trem previous steps) (Targe tedangle) oriongles.
value of the doubte integrat al () 168/3.
68829
Aravers 4)
108/3
Figure 1: Examples from the VideoMathQA illustrating the three reasoning types: Problem Focused, Concept Transfer, and Deep Comprehension. Each sample includes carefully selected video, question, five multiple-choice options, expert-annotated reasoning steps with temporal localization, and the final answer.
2019b) or narrowly scoped tasks Moreover, existing video benchmarks often rely on synthetic (Yi et al.,
(Shangguan et al., (2024) without detailed reasoning annotations, making it diffiultt to diagnose whether a model's 's solution stems from genuine logical inference or simple pattern matching.
To bridge these gaps, we introduce VideoMathQA, comprehensive benchmark designed explicitly to evaluate deep mathematical reasoning in videos. It comprises manually annotated real-world video-questir pairs drawn from various educational resources. VideoMatthQA tests three core reasoning scenarios: a) Direct Problem Solving, where answers are grounded entirely in the presented content; b) Conceptual Transfer, requiring agents to apply learned methods to novel problems; and c) Deep Instructional Comprehension, involving multi-step reasoning over extended, partially worked-out explanations. Each instance includes high-resolution frames, aligned subtitles, and audio narration, and is annotated with fine- tineggrined reasoning traces that capture every inference step to enable assessment of both intermediate steps and final outcomes. Videos span ten mathematical domains (e.g., geometry, calculus, statistics, graph theory, chart) and range from 10s clips to hour-long lectures, ensuring both short-term perception and long-range dependency evaluation (see Fig. D for illustrative examples of the reasoning scenarios). Our key contributions are:
A multimodal video reasoning benchmark that demands precise integration of vision (high-resolution), text, audio and subject knowledge in solving complex math problems.
2

---

Published as a conference paper ICLR 2026
Artchmetic
0-20s
20s
Ziin
Zmiis 77nin
Pmin- the
Graph Theory 48
11%
20%
18%
200
1.5%
166
Short
Medium
Long
(b) Distribution of video length duration
Counting
36
Statietice & Prubuiditt
YouTube Dota API
Osestion aresented
Selving engagreeen with videe Milli-stpp and muttimodol reseownng required
Supplly EoT reasoningg required to antive at prouune Each quertion is aedgped ressoning
only with dtate haaees or andio
Rih Tenperal Progresion
Goometric 12%
waaoning type
155
Stage # Video Selection
Stage # QA Annotation
Stage # Step-wist Reasoning
deoottA
Topology
Goometric Area
t
Quality Assurance
Geonnsgic Angle V8
Quality assurance follows each annotation stage for high quality. Annotations are performed by expert science graduates.
Okgussa
(a) Math Concept-wise Distribution and Model Performance
(c) Annotation Pipeline used to create VideoMathQA Benchmark
Figure 2: a) Question distrihution and model accuracy across ten mathematical concepts in VideoMathQA, showing con- sistently low performance and highlighting current models' limitations in video-based mathematical reasoning. b) Video duration distribution, ranging from short 10s clips long 1hr videos. Three-stage annotation in Video conducted by expert science graduates with detailed step-by-step reasoning and strict quality assessment.
Our benchmark offers three real-world task categories: problem solving, concept transfer, and deep comprehension, reflecting instructional scenarios in videos. It spans ten mathematical domains and environments (whiteboard scribbles, digital slides, animated charts), drawn from diverse sources.
Fine-grained reasoning annotations and metrics for measuring both intermediate inference fidelity and final answer correctness, with explicit mechanisms to detect confabulations.
A comprehensive evaluation across 30 models reveals that success relies not just on perception, but on sustained attention to subtle cues dispersed across time and modalities.
2 VIDEOMATHQA
Despite significant progress in image-based mathematical reasoning benchmarks, existing datasets fail to capture the unique challenges posed by video-based math problems. In designing VideoMathQA, we asked: What core challenges must a benchmark capture to reflect real-world multimodal mathematical reasoning and understanding? Our answers shaped the core principles of the benchmark. We identified three critical challenges: the need to interpret dynamic visual representations, such as diagrams constructed or modified over time, often involving handwritten or digital content requiring robust, frame-aware OCR; the need for temporal reasoning over long, non-linear contexts, where concepts unfold gradually or are revisited across time; and the need for joint grounding across visuals, text, and audio, all three modalities often contribute distinct pieces of essential information. These interconnected challenges define the complexity of video-based mathematical tasks and directly inform the design of our benchmark.
Vatho consists of 420 carefully curated video-question pairs drawn from diverse mathematical instructional content, including structured problem walkthroughs, concept demonstrations with follow-up questions, full-length whiteboard or digital lectures, and animated documentaries involving chart-based reasoning. Each question includes multi-step reasoning steps, with a total 2, 945 expert-annotated steps across the dataset. These reasoning steps annotated with timestamps, grounding major observations to their precise location in the video and enabling direct evaluation of whether models can align reasoning traces with temporal evidence. Each question is characterized along four dimensions: (i) Mathematical concept, covering 10 domains such as geometry, arithmetic and calculus, statistics and probability, counting, graph theory, puzzles, topology, and chart reading, (See Fig. 21); (ii) Reasoning type, categorized problem
3

---

Published as a conference paper at ICLR 2026
focused, concept transfer, or deep comprehension: (iii) Video duration, ranging from 10 seconds to over an hour and grouped as short, medium, or long, supporting evaluation of both short-term and long-range reasoning (See Fig. [)); and (iv) Difficulty level, categorized as easy, medium, or hard (See Fig. I
The annotation process consists of three stages: video selection, question question-answer answer annotation, and step-wise wise reasoning (see Fig. ) requiring considerable expert effort. On average, 30 mins to identify a suitable video, 40 mins to craft a high-quality question-answer pair, and 1 hour to compose detailed step-by-step reasoning, resulting approximately 2 to 2.5 hours per sample. Across the dataset, this amounts to roughly 115 person-days of annotation by science graduates. Different annotators handled each stage of sample to ensure independent verification, with each stage revising the quality of the previous one. This systematic revision process ensures the benchmark's 's overall quality and consistency.
2.1 VIDEO SELECTION
We curate video sources that capture core challenges, including dynamic visual representations, long-range temporal dependencies, and the need for joint grounding across visual, textual, and audio modalities. We do this by selecting videos where the associated questions cannot be reliably answered using static frames or audio transcripts alone, focusing on content with rich temporal progression such as step-by-step diagram construction and incremental equation derivation. Static slide presentations and videos with minimal visual transitions are excluded. All videos are sourced via the YouTube API, manually reviewed, and trimmed to retain only question-relevant segments. Selection is also guided by conceptual difficulty, ensuring coverage from high-school to advanced university-level and Olympiad-style problems.
For chart-based questions, we include animated charts from documentary and news sources, spanning bar plots (vertical and horizontal), pie charts, line graphs, histograms, and stacked bar charts. We prioritize videos with multiple charts whose interpretations are temporally and conceptually linked. For the remaining nine mathematical categories, we collect a diverse mix of lecture videos, screen recordings, and digital whiteboard tutorials that reflect instructional formats commonly used in educational content.
2.2
QUESTION-ANSWER ANNOTATION
Each video in MAio paired with a carefully constructed multiple-choice question evaluate a model's ability to reason over visual and temporal mathematical content. Questions are written ensuring that solving them requires meaningful engagement with the video, rather than relying on surface-level cues from transcripts or isolated frames. This stage also serves as quality assurance for video selection, with about 70% of videos retained, and the rest discarded. We categorize questions into three reasoning types: (i) Problem Focused, where the question is explicitly stated and can be solved through direct observation and reasoning; (ii) Concept Transfer, where a demonstrated method or solution approach must be applied to a similar but new problem; and (iii) Deep Instructional Comprehension, which involves following long instructional content, such as a lecture or tutorial, to understand the context, identify what is being asked, interpret the partially worked-out solution, and complete the solution. Fig. T iillssrates example of each reasoning type.
2.3 STEP-WISE REASONING AND QUALITY ASSESSMENT
Following question-answer annotation, each sample undergoes a second stage of annotation, where a separate annotator writes a step-by-step explanation detailing the reasoning to arrive at the final answer. Each question is assigned four to ten steps, each step reflecting meaningful semantic progression, capturing a distinct, essential part of the overall solution. This results in a total of 2,945 high-quality expert-annotated reasoning cfain-of-thought responses are compared against these steps, enabling step-level evaluation of how far a model progresses toward the correct answer. This stage also serves as quality assurance for the question-ansee answer pairs, allowing the annotator to verify them, correct any errors improve clarity.
steps.
4

---

Published as a conference paper ICLR 2026
Approximately 30% of the questions were refined during this stage. Additionally, each question is tagged with a set of likely error categories to support structured error analysis during the step evaluation. Finally, we conduct a review of the step annotations, during which 788 out of the 2, 945 steps were revised.
2.4 COMPARISON WITH EXISTING VIDEO BENCHMARKS
While recent video-language benchmarks offer broad coverage across domains, they primarily focus on perceptual or narrative understanding, often lacking the depth, structure, and multi-modal precision required for mathematical reasoning. In contrast, Matho targets the
caceiiii---mmlimmdal--eeeeee challenge by demanding tightly aligned interpretation across high-resolution visuals, spoken explanations, and textual content, characteristics underrepresented in prior works. Unlike benchmarks that emphasize short- form clips or general comprehension, our dataset centers on domain-specinf annotated tfain-of-tthuuht traces, enabling fine- grained evaluation of both intermediate and final responses. We present a comparison of VideoMathQA with recent benchmarks in terms of domain, duration, STEM focus, and step-wise CoT annotation in the Appendix Tab. 3 By focusing on mathematical problem-solving in long instructional videos, it reveals challenges still underexplored in current video-language benchmarks.
iee-inteessive reasoning with
3 EXPERIMENTS
We evaluate a diverse set of models on VideoMathQA, including 5 proprietary multimodal models and 25 open-source models selected for their strong capabilities in video reasoning. We specifically focus on otate-of-the-arr video models. spanning 4 model size categories: 5B, 9B, 40B, and 80B parameters. We cover Qwen2.5-VL. (Bai et al. 2025). Intern VL2.5 (Chen et al.) (024). Intern (Zhu et al.) 2025). PLM- LLaMA (Cho et al. 2025), Oryx-1.5 (Liu al. 2024), VAVVV- (Li et al. 2024b), Long VA-DPO (Zhang et al. 2024a), Intern Video2.5 (Wang et al.) 2025), and Aria (Li et al. 2024c). In addition, we include recent reasoning models that supporting chain-of th-ouughht prompting (VideoChat-Rl (Li et al.) 2025), Video- RI (Feng et al. 2025 ). To contextualize the performance of video-specialized models, we include 2 baselines, one vision-blind LLM (Qwen-2.5), and one state-of of-the-art image model (Qwen2.5-V2) VLL evaluated on single key frame. We also include a human evaluation to ground the model results against meaningful reference.
3.1 EVALUATION STRATEGIES
Since the benchmark emphasizes the importance of multimodal grounding and temporal reasoning across modalities, ensure fairness by adapting input configurations to align with the strengths and operational capabilities of each model. Specifically, each model samples frames according to its optimal setting (32 for LLaVA-One Vision, up 768 for Qwen2.5-VL, and full-video access for Gemini). Consequently, subtitles are aligned with the sampled frames, providing richer audio context to models capable of handling longer temporal sequences. Our setup rewards models that can leverage extended temporal context and effectively integrate multimodal information. Please refer to Appendix B for additional implementation details and Appendix |F for the prompts used in CoT, postprocessing, sise evaluation and error analysis.
Inference Protocol:
We implement evaluation strategies to comprehensively assess model performance: (i) Multiple-Choice Evaluation (MCQ): Each question provides five options (one correct and four distractors). This direct evaluation format offers clear reproducibility without reliance on LLM-based scoring. However, it can inflate weak model scores, especially in smaller models (s 9B). (ii) Multi-Binary Evaluation (MBin); To better distinguish performance in such cases, construct binary answer against each distractor independently. A model must select the correct option across all pairs to be marked correct, significantly reducing randomness and more accurately revealing true model capabilities, particularly critical when evaluating across a wide range of model scales. (ii)) Chain-of- Chain-of-Thought (CoT) vs. Direct Answering: In direct evaluation, strict instructionfoolowiinn is crucial since models are instructed to
variants by pairing the correct
5

---

Published as a conference paper ICLR 2026
MCQ
MBii
Muthematic Concepts
Duration
Models
Size
V
+Sub
+Sub
GAng
GAre
GL.en
Chart
Stat
Arth
Topo
Grph
Cnig
Pzle
Short
Med
Long
Random
17.4
17.4
7.9
7.9
8.7
7.0
7.8
10.7
8.7
3.9
6.7
11.1
2.6
11.1
9.0
7.8
6.9
Human
80.7
80.7
91.3
83.1
80.4
81.3
87.0
80.8
60.0
88.9
84.2
70.4
80.3
82.1
79.6
Proprictary Models
Claude-3.7 3.7.omee
26.2
27.1
8.6
9.5
17.4
9.9
5.9
8.0
17.4
11.5
13.3
5.6
5.3
9.3
8.2
11.0
9.1
GPT-40
20.2
24.5
12.6
13.6
13.0
12.7
15.7
12.0
4.4
17.3
20.0
5.6
7.9
20.4
14.2
15.6
10.6
Gemini-1.5 1.5-Flash
20.5
23.1
12.6
17.6
26.1
15.5
19.6
9.3
17.4
23.1
22.2
15.8
24.
17.9
22.1
12.1
Gemini Flash
28.6
31.7
14.1
20.5
30.4
23.9
27.5
13.3
8.7
19.2
13.3
16.7
7.9
33.3
25.4
24.0
11.4
Open-source Models (<. GB)
Qwen2.5.V1. VI.
3B
26.9
27.6
19.3
19.6
26.1
23.9
23.5
21.3
34.8
17.3
26.7
11.1
15.8
20.4
25.4
23.4
15.9
InterVVL2.5
2B
24.3
20.7
14.5
21.7
27.5
4.4
15.4
20.0
17.9
16.9
PLM-LLaMA
3B
22.9
22.1
13.6
15.0
17.4
16.9
25.5
8.0
26.1
9.6
20.0
11.1
13.2
13.0
16.4
18.8
9.1
InteraVL3
2B
22.4
23.3
18.8
16.4
21.7
16.9
17.7
17.3
30.4
15.4
20.0
22.2
13.2
5.6
18.7
14.9
15.9
Open-source Models (<,
9B)
M.M-LLaMA
8B
22.1
23.1
16.7
14.5
13.0
11.3
17.7
13.3
17.4
17.3
20.0
11.1
10.5
16.7
16.4
14.9
12.1
0ryx-1.5
7B
22.6
22.6
16.9
17.4
13.0
23.9
23,5
9.3
21.7
23.1
20.0
5.6
18.4
11.1
20.2
20.8
10.6
LLaVA-OV OV
7B
20.7
21.2
14.8
15.5
8.7
15.5
17.7
16.0
30.4
17.3
13.3
5.6
15.8
11.1
16.4
18.8
10.6
LongVA VA-DO
7B
21.4
21.7
16.2
14.1
17.7
30.4
9.6
14.9
Video-R1
7B
21.4
17.4
16.0
16.2
8.7
22.5
25,5
16.0
26.1
13.5
6.7
5.6
13.2
9.3
16.4
16.9
15.2
InterVV1.2.5
8B
24.3
24.8
18.6
18.6
26.1
19.7
17.7
17.3
21.7
19.2
26.7
11.1
10.5
20.4
17.9
22.7
14.4
LLaVA- Video
7B
26,9
26.4
19.3
31.4
17.4
15.4
23.9
20.8
Videm2iddo2.55
8B
25.2
28.6
19.1
19.1
34,8
22.5
15.7
14.7
21.7
19.2
20.0
27.8
10.5
18.5
18.7
22.1
15.9
QWLn
7B
26.7
27.9
19.8
19.1
8.7
25.4
25.5
18.7
13.0
23.1
13.3
5.6
15.8
16.7
22.4
19.5
15.2
InLemVL33
8B
29.1
27.9
20.7
27.5
13.0
28.9
25.4
VideoChat-R1
7B
27.6
29.1
21.2
21.2
8.7
22.5
31.4
21.3
17.4
30.8
6.7
11.1
15.8
18.5
26.9
20.1
16.7
Open-source Models (<t 408B
Aria
34B
23.8
26.4
17.4
19.1
8.7
25.4
19.6
22.7
17.4
19.2
20.0
11.1
21.1
11.1
21.6
16.9
18.9
Oryx-1.5
32B
30.5
33.1
22.9
24.1
30.4
39.4
31.4
17.4
21.2
15.8
33.3
27.6
29.9
13.6
Qwen2.5.V1.
32B
32.4
32.6
25.7
24.8
43.55
31.0
25.5
14.7
26.1
26.9
6.7
27.8
10.5
33.3
28.4
30.5
14.4
InterVVL.2.5
38B
31.0
33.6
24.1
26.0
43.5
38.0
39.2
8.0
13.0
32.7
6.7
11.1
18.4
29.6
34.3
31.8
10.6
IntemVL3
38B
31.7
35.7
29.5
34,8
37.3
17.4
25.0
13.3
26.3
35.8
38.3
12.9
Open-source Models (<: 80B)
LLaVA-Video
72B
28.3
30.0
20.2
24.3
8.7
32.4
25.5
20.0
13.0
36.5
13.3
22.2
21.1
24.1
27.6
27.3
17.4
LLaVA-OV
72B
25.5
28.3
21.0
24.8
31.0
23.5
12.0
21.7
38.5
20.0
27.8
30.6
28.6
In.2m
78B
33.3
31.7
28.3
27.9
39.1
36.6
31.4
18.7
26.1
32.7
26.7
27.8
13.2
27.8
33.6
35.1
13.6
Qwea2,5-VL
72B
36.9
37.6
26.0
27.9
26.1
36.6
31.4
17.3
30.4
38.5
20.0
16.7
18.4
29.6
34,3
29.2
19.7
InterVL33
78B
33.3
31.7
28.3
27.9
39.1
36.6
31.4
18.7
26.1
32.7
26.7
27.8
13.2
27.8
33.6
35.
13.6
Table 1: Performance VideoMathQA using direct answer prompting in both MCQ and MBin settings. We report results with video-only (V) and videotsubtitle (+Sub) inputs. MBin performance further broken down across ten mathematical concepts and three video duration categories. The mathematic concepts are geometric angle (GAng), geometric area (GAre), geometric length (GLen), chart reading (Chart), statistics and probability (Stat), arithmetic and calculus (Arth), topology (Topo), graph theory (Grph), counting (Cntg) and puzzles (Pzle).
respond with only the correct option, allowing immediate, format-based extraction without post-processing. In contrast, CoT evaluations encourage models articulate detailed reasoning steps prior answering, reflecting human-like wobleem-oolving approaches, and providing leniency regarding response formatting. Here, we use a lightweight oftte-of-theann open-source LLM (Qwen-3-4B (Yang al., 2025a) in non- non-thinking mode to extract the final answer. This setup allows us to analyze whether reasoning enhances performance. (iv) Step- wise Reasoning Evaluation: For CoT responses, we further evaluate the quality of reasoning by comparing model-generated rationales with annotated solution steps. We use the Qwen-3-4B model in its thinking mode for this task, leveraging its strong mathematical reasoning ability to ensure fair and accurate evaluation. The model assigns each response score between 0 and 10, along with rationale. We further the rationales to enable eτror analysis against predefined eиror categories, providing deep. actionable insights that highlights model limitations and reasoning gaps. We validate the reliability of Qwen-3-4B the judge model against human scoring and confirm that the rubric accounts for alternative but logically valid reasoning paths; both analyses support the robustness of our evaluation protocol (see Appendix C).
6

---

Published as a conference paper ICLR 2026
MCQ
MBin
Mathematic Concepts
Duration
CoTT
Models
Size
V
+Sub
V
+Sub
GAng
GAre
GLen
Chart
Stut
Arth
Tupo
Grph
Cnig
Pale
Short
Med
Long
Eval
Random
17.4
17.4
7.9
7.9
8.7
7.0
7.8
10.7
8.7
3.9
6.7
11.1
2.6
11.1
9.0
7.8
6.9
Human
80.7
80.7
91.3
83.1
80.4
81.3
87.0
80.8
60.0
88.9
84.2
70.4
80.3
82.1
79.6
Open-source (<. 9B)
Vidco-RI
7B
23.8
27.6
18.1
20.0
13.0
26.8
23.5
9.3
34.6
16.7
21.6
26.0
LLade--Vdee
7B
26.4
23.6
20.0
16.0
4.4
15.5
23.5
16.0
21.7
7.7
26.7
0.0
21.1
18.5
16.4
16.9
14.4
2.7
Qwen2.5-VV.
7B
25.2
29.5
17.6
18.3
13.0
15.5
11.8
20.0
36.5
16.7
16.4
20.1
VL3
8B
28.8
26.9
17.9
20.0
17.4
22.5
27.5
13.3
4,4
17.3
13.3
16.7
7.9
24.1
19.4
23.4
9.9
3.4
Intern Video22
8B
24.3
27.6
18.3
19.8
26.1
25.4
13.7
14.7
26.1
21.2
13.3
27.8
18.4
18.5
17.9
25.3
15.2
3.0
VideoChar-RR R1
7B
22.4
28.3
21.4
19.8
13.0
29.6
15.7
10.7
17.4
26,9
6.7
22.2
18.4
24.1
20.9
25.3
12.1
3.6
Open-source (< (0B)
Oryx-1.5
32B
33.6
21.7
25.2
34,8
35.2
13.3
21.2
22.2
30.6
29.2
IntemVLL
38B
30.0
11.4
21.7
25.0
43,5
31.0
25.5
16.0
17.4
32.7
20.0
11.1
13.2
31..5
26.9
31.2
15.9
4.1
Q-
32B
31.4
36.9
22.6
27.1
47.8
29.6
33.3
16.0
26.1
32.7
13.3
16.7
23.7
29.6
29.1
32.5
18.9
4.9
Open-source (< 80B)
LLaVA- Video
72B
23.6
29.3
14.8
18.6
8.7
22.5
17.7
14.7
8.7
21.2
26.7
11.1
26.3
20.4
17.2
21.4
16.7
3.1
LLaVA-OV
72B
23.3
26.9
14.3
18.1
8.7
14.1
19.6
13.3
21.7
26.9
22.2
10.5
25.9
15.7
23.4
14.4
3.2
Owen2.5.VV.
72B
37.4
36.9
24.5
28.6
30.4
31.0
31.4
24.0
21.7
50.0
13.3
22.2
15.8
25.9
27.6
34.4
22.7
5.0
IntemVLL3
78B
37.1
25.2
27.9
39.1
39.4
33.3
13.3
26.1
23.1
22.2
10.5
40.7
28.4
36.4
Proprietary Models
Claude 3.7-sonnet
24.8
29.5
12.1
19.3
34,8
29.6
19.6
4.0
26.1
13.5
20.0
16.7
21.1
22.2
23.1
26.0
7.6
4.2
GPT-4a
27.1
34.3
18.6
22.9
26.1
22.5
17.7
17.3
30.4
32.7
20.0
33.3
13.2
25.9
19.4
29.9
18.2
4.9
2..FFai-
35.2
38.8
19.5
24,8
34.8
21.1
27.5
18.7
21.7
28.9
13.3
33.5
18.4
33.3
27.6
27.9
18.3
4.7
GPT-o4-mini
44.8
43.5
40.0
72.2
prompting in MCQ and Multi- Balti-Binary (MBin) settings. We report results with video-only (V) and vidco+subtitle (+Sub) inputs to highlight the effect of subtitles multimodal grounding. MBin performance is further broken down across ten mathematical concepts and three video duration categories. The table covers open-source models across multiple size tiers.
Table 2: Performance on VideoMathQA using
Human Evaluation: We perform comprehensive human evaluation using a pool of annotators. Each annotator is assigned a roughly equal subset of questions and is instructed to watch the corresponding video and solve the given question, with maximum of 20 minutes allowed per question. We allow flexibility in how they watch the video, they may choose to increase playback speed or skip segments at their choice.
3.2 QUANTITATIVE ANALYSIS
Tab. [ [presents the direct answer evaluation, and Tab. faa...ssba shows the thain-ooftthuggee (CoT) evaluation. Both tables cover the MCQ and MBin evaluation, with and without subtitles, providing a comprehensive view additionally reports CoT step evaluation scores based on alignment with annotated reasoning traces. Proprietary models tend to benefit significantly from CoT prompting, whereas open-source models show mixed gains. In step evaluation, GPT-o4-nn mini achieves the highest score of 6.9, with Qwen2.5-VL-77 L- leading among open models with a score of 5.0.
of model capabilities. Tab. 2
How does model size influence performance? Across both MCQ and MBin settings, we observe that model performance improves with scale, both with CoT prompting (Tab. 2 and direct answering (Tab. []. For instance, in the CoT MBin with subtitles, InternVL-3 improves steadily with size: 20.0% (8B) to 25.0% (38B) and 27.9% (72B). Similar trends hold for other models (LLaVA-Video, Qwen2.5-Vi VLL), indicating that larger models are better at retaining temporal context, focusing on key visual details, and grounding information across modalities, all crucial for video-based mathematical reasoning. However, scale alone does not determine performance. Smaller, newer models often outperform older, larger ones. For example, Intern VL-3-38B surpasses multiple 72B models LLLaVA-Video-721 LLaVA-One Vision- 72B) in both CoT and direct answers. Newer models benefit from stronger architectures, improved visual understanding, and better reasoning, enabling them to outperform larger, previously SoTA models.
Vision, Vineeniiioooon,
7

---

Published as a conference paper ICLR 2026
Slaort Mediuun
Witth instiiin
France 1M
20:
Azancy (1g
L0aVA-07-26 26-26
Iotera Volenz Valead-al
Visto Viteot Rt
Graald Flad
Jusen V24-201)
Oroad Vi-1i
Inken Lto Mo---0
EdaVa Video 7
Inteal (ele
Organ Ganini Fidd Lo--t))
Guen 1---22-
Lubern VL-1-78
Sbeet
Nedtom
Long
Vidoo Lenghh
(a) Math wine Distribution and Model Performance
Effect of subtitles in CoT evaluation
(c) Effect of frame sampling
Figure 3: The figure shows VideoMathQA performance a) Across video duration categories under CoT MBin +Sub; b) Impact of subtitles under the CoT MBin; and e) Effect of varying the number of input frames under CoT MCQ. Overall, models perform best on medium-length videos, and accuracy improves with the inclusion of subtitles and more frames.
How do proprietary models compare to open-source models? We evaluate 5 proprietary models with CoT prompting and find that Semini--20--Flaste and GPT-o4-mini deliver the best performance. GPT-o4-mini achieves the highest overall accuracy, with 44.8% in CoT MBin with subtitles. It performs particularly well in complex reasoning categories such as arithmetic-calculus (63.5%), statistics (65.2%), and geometric area (49.3%), with performance significantly higher than the average of other proprictary and open-source models. These results suggest that its strong performance stems from better ability to integrate understanding across vision, audio, text, and background subject knowledge, enabling more coherent mathematical reasoning. While proprietary models continue to lead, our results show that the gap between proprietary and open- source models is narrowing. Optimized open-source models such as Qwen2.5-VV VL- and Intern VL-3-78B outperform several proprietary counterparts, including Claude-3.7-Sonne
Ceminni-2.0-Flass and GPT-4o.
How do subtitles influence model performance? Subtitles consistently enhance model performance across both CoT evaluation (Tab. [ and direct answering (Tab. [ especially for larger open-source and proprictary models. However, the impact of subtitles is not uniform: smaller models (<5B and <9B) often show minimal or inconsistent gains. In contrast, ceasoning-capabll models like GPT-o4-mint improve from 42.1% (video-only) to 44.8% (video+sub), and Qwen2.5-VL. improves from 24.5% to 28.6% in the CoT MBin. As shown in Fig. B3,, models with stronger reasoning capabilities benefit more from subtitles, while smaller models struggle to extract meaningful gains from them. These improvements reflect the ability to integrate fine-grained audio cues with visual frames, 'needle-in---muull wwwai-oaalllalllaaaaaaaaaaaaa -aaiiiiddallh-shckkkkk challenge, where critical information is distributed across modalities, and stronger reasoning models are better equipped to ground these disparate cues into coherent solutions, while others may overlook small but essential verbal cues.
How does video length and frame sampling affeet performance? We evaluate model performance across short (<30s), medium 2min), and long (2min-1hr) videos and observe two distinct trends (see Fig. B)) First, while most models perform reasonably well in short videos, accuracy generally improves in medium- length videos and declines for longer durations. These trends align with the three reasoning challenges in the benchmark. Short videos often correspond to problem focused questions, requiring the model to derive a solution, where success hinges on general mathematical competence and the ability to extract key visual or verbal cues. Medium-lenghe videos commonly involve concept transfer questions, where the model is first shown a solution or method, then asked to adapt it to related problem, favoring models that can effectively comprehend instructions. In contrast, long videos correspond to deep instructional comprehension questions, which require following extended, often non-lineat instructional sequences to interpret the context. Here, the information load is higher and the essential clues for solving the problem scattered over modalities
8

---

Published as a conference paper ICLR 2026
Un
STER Tixk
Earge
GP
x0
211
Aocency (5) 180 SUMite
304
Choelety
Yeur 180
lime
BateneV 78D
Quan 2-72
queed 8-20
Qeeed 8-70
Qeved 2u 7001
Vision-blind vs Image-only Video-MLLM
(b) Impact of Question Difficulty
(c) Chain-of Chain-of-thoughts error analysis
Figure 4: The figure shows a) Comparison among vision-blind, image-only, and video models, highlighting the need for video-level understanding to perform well VideoMathhA. b) Distribution of questions in Video Q Aacross three difficulty levels for varying reasoning depths, and the relationship between performance and question difficulty across top performing models. c) Error analysis based on CoT step evaluation. Most model errors stem from misunderstanding the question, where models misinterpret what the question asks overlook critical multimodal cues.
and time. This setting closely aligns with the central challenge, where overlooking even few critical details can derail the entire reasoning process. Second, we study how frame sampling influences performance by evaluating Qwen2.5-VL. with 16, 64, 256, and 768-frame settings (see Fig. Bb) We find that increasing frame count provides consistent improvements, particularly for longer videos: a 5-point gain for short videos and up to 8 points in long videos. highlighting that models capable of handling extended frame sequences and maintaining long-range temporal coherence are better equipped for Further, in Fig. 44, we compare video models with rision-blint text only and single-imag models, highligting that in-depth temporal reasoning is required to perform well on VideoMath Abenchmark.
mathematical reasoning.
How does question difficulty affect performance? Model accuracy varies notably with question difficulty, while most models solve a moderate proportion of easy questions, they struggle with harder ones (Fig. (Fig.4b). GPT-4o answers 96% of easy questions correctly, yet handles only 46% of the hard ones. VL-3m7VL33-78B solves 60% of easy examples, but manages just 8% at the hard level. Other models follow the same trend, exposing key limitation in current models, their ability generalize weakens under higher cognitive load.
Where do models fail in the reasoning process? Fig. 4 shows a breakdown of model errors from CoT step evaluation across seven error categories: problem misunderstanding, failure to retrieve relevant information, misinterpreting visuals, incorrect concept application, wrong strategy or formula, forgetting previous context, and calculation errors. The most common issue is problem misunderstanding, where models fails to localize the specific information referred to in the question or overlook critical multimodal cues in the video. This reflects the core challenge of our benchmark, where missing even a small verbal or visual detail can derail reasoning entirely. Proprietary models like GPT-o4-mini and Gemini-2.0-FFaas show fewer errors in concept application and strategy selection (12% and 6%), suggesting stronger domain grounding. In contrast, open- source models like InternVL-3 exhibit more broadly distributed errors, with concept application and strategy selection together accounting for 23% of total errors, plus notable computation mistakes. GPT-o4-mint however, has a higher share of visual interpretation errors, showing difficulty with fine- fine-grained cues like charts and diagrams. See Tab. F.7 for error annotation prompts and category definitions and Appendix[AJoo further discussion performance across mathematical concepts and numan-mode comparisons.
9

---

Published as a conference paper ICLR 2026
4 CONCLUSION
In this work, we introduce VideoMathQA, comprehensive benchmark evaluate mathematical reasoning in real-world educational videos. It provides curated video-question pairs with step-by-step reasoning across diverse domains and instructional formats, addressing the lack of benchmarks for temporally extended, cross-modal reasoning in multimodal models. Unlike prior benchmarks based on static images and limited context, VideoMathQA requires models to integrate dynamic visual, textual, and auditory cues solve problems, transfer concepts, and demonstrate deep instructional comprehension.
REFERENCES
Aida Amini et al. MathQA: Towards interpretable math word problem solving with iperation-basel for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.139233 2025.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.052711 2024.
Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al. Perceptionlm: Open-access data and models for detailed visual understanding. arXiv preprini arXiv:2504. 13180, 2025.
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench- video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems, 2024.
Kaituo Feng, Kaixiong Gong. Bohao Li, Zonghao Guo, Yibing Wang. Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv preprint arXiv:2503.21776, 2025.
Chaoyou Fu et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal lims in video analysis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2025.
Kairui Hu, Jindong Liu, Yeyu Duan, Song Li, Haoyu Zhang, Can Xu, Xiaodan Liang, and Dahua Lin. Video-mmmu: Evaluating knowledge acquisition from multi-diseipline professional videos. arXiv preprint arXiv:2501.13826 2025a.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video- mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint 2rXiv:2501.13820 2025b.
Kairui Hu, Penghao WI, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video- mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501. 13826, 2025c.
10

---

Published as a conference paper ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, 2024a. URL https: hattps://githane httptttttththtoeegegggg com Evol www.iiiiggMMMaaan 1mmns-evall
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024b.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprini arXiv22410.05993, 2024c.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22195-22206 2024d.
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng. Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Vidcochat-rl: Enhancing spatio-temporal perception via reinforcement fine-tuning, arXiv preprint WXXv:2504.06658, 2025.
Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx milm: On-demand spatial-tempores understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024.
Pan Lu, Hritik Bansal, et al. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- formfformmo video language understanding. Advances in Neural Information Processing Systems, 2024.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACI. 2022, 2022.
Viorica Patraucean et al. Perception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems, 2023.
Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models, arXiv preprint 1mXiv:2410.23266. 2024.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18233 2024.
11

---

Published as conference paper ICLR 2026
Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video-mmlu: massive multi-discipline lecture understanding benchmark. arXiv preprint 21w.xiiv2204444669,, 2025.
Qwen Team. Qwen3, April 2025. URL http:t/ttt////eeeea..nntttttttta
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with the math-vision dataset. In Advances in Neural Information Processing Systems, 2024a.
Weihan Wang, Zehai He. Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi. Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint 20n::4406.003355 2024b.
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2.55 5: Empowering video milms with long and rich context modeling. arXiv preprint (rXxi:2501.12388 2025.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interlcaved vidco-language understanding, In Advances in Neural Information Processing Systems, 2024a.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-languag languale understanding. Advances in Neural Information Processing Systems, 37: 28828-28857 2024b.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786, 2021.
Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can trust your answer? visually grounded video question answering. In Proceedings of the JEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 13204-13214, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025a.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 10632-10643, 2025b.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint 0rXiv:1910.0144 2019a.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint 1111v:1910 2019b.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Xiang Yue et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024a.
Xiang Yue et al. Mmmu-pro A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409 02813, 2024b.
12

---

Published as conference paper ICLR 2026
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024a.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, arXiv preprint arXiv:2410.02713, 2024b.
Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christopher Arif Setiadharma, Jingkang Yang, and Ziwei Liu. Worldqa: Multimodal world knowledge in videos through long-chain reasoning. arXiv preprint arXiv:2405.0327 0332022. 2024c.
Yilun Zhao, Haowei Zhang, Lujing Xie. Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi-discipline video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8475-8489, 2025.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin. Xi Yang. Yongping Xiong, Bo Zhang, et al. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13691-13700 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.1047, 2025.
Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. In ICLR, 2025.
13

---

Published as a conference paper ICLR 2026
Appendix
We provide appendix for deeper understanding and more analysis related to the main paper, arranged as:
1. Additional Quantitative Analysis (Appendix A)
2. Additional Implementation Details (Appendix (B)
3. Robustness of wise Reasoning Evaluation (Appendix
4. Related Works (Appendix (D)
5. Limitations and Future Work (Appendix E
6. Prompts (Appendix
A ADDITIONAL QUANTITATIVE ANALYSIS
How well do models perform across different mathematical concepts? We analyze model performances across the ten mathematical categories covered in the benchmark and observe notable variation in their ability comprehend and solve different mathematical concepts (see Fig. 2 Current models tend to perform better on questions involving arithmetic and calculus, with average accuracy around 32% and GPT-o4-mini achieving the best performance of 63.5% with CoT evaluation. Most models show moderate performance on categories such as geometric reasoning and puzzles, with average performance ranging between 24% and 30%. contrast, chart reading, topology, graph theory, and statistics and probability are more challenging for all models. Average accuracy across these categories typically falls between 16% and 21 21%, with GPT-o4-mini scoring only 20% in topology and graph theory, and a maximum of 40% chart reading.
How does human performance compare to model performance? Humans achieve an overall accuracy of 80.7%, confirming that the benchmark is solvable but challenging. Performance is lower in categories such as topology (60.0%) and puzzle (70.4%), which typically require longer reasoning. Accuracy is also strongly time-dependent: if all questions taking more than 10 minutes are counted as incorrect, overall accuracy drops 44.3%. This highlights both the difficulty of the benchmark and that extending the 20-minute cap could further improve performance. Compared to evaluated models, humans score about 36 points higher than the best model average. The gap is smallest in arithmetic and graph theory, but largest in visually demanding categories like topology, counting, and chart Reading, where humans lead by roughly 35-50 points.
B ADDITIONAL IMPLEMENTATION DETAILS
For all models, we run direct answering for both MCQ and MBin. For sessoinng-capbbll models, we additionally perform CoT evaluations for both. A lightweight LLM--assed post-processing extracts option choices from CoT responses to compute accuracy. Step-wise evaluation is applied only to MCQ CoT responses. Using ground-truth steps, model responses, and step-evaluation critiques, we conduct error analysis for selected models. We also ablate the effect of subtitles across settings to analyze sensitivity to multimodal input. All prompts used, including those for CoT prompting, postprocessing, step-wise evaluation, error analysis, and subtitle handling are detailed in Appendix F. to F.7
We use 8 A100-80 GB GPUs for all evaluations. For models with < 8B parameters, use data parallelism across 8 workers. For larger models, we utilize tensor parallelism with TP = 8. All MLLM evaluations are conducted using Imms eval (Li al. 2024a), while evaluations for language-only models are performed using VLLMM (Kwon et al. 2023). We provide the implementation of VideoMathQA in Imms eval, along with scripts to both MILM evaluations as part of the submission.
For all evaluated models, we use the recommended number of input frames following the official code base. Specifically, use 128 frames for Aria at. 2024c). 512 frames for Intern Video-2.5 (Wang al.) 2025)
14

---

Published as a conference paper at ICLR 2026
64 frames for LLaVA- (Zhang al. 2024b), 128 frames for Long VA (Zhang et al. 2024a). 32 frames for LLaVA-One Vision (Li et al. 2024b) and 768 frames for Qwen2.5- VL (Bai et al. 2025). We use greedy decoding with a temperature
for all MLLM evaluations. For chain of thought (Coll evaluation, we post-process the responses to extract the final answer option using Qwen3-4B (Team] 2025 LLM. For the CoT step evaluation, we also use the Qwen3-4B model as the LLM judge and report the average over three runs. Further details on the exact prompts used for CoT prompting, post-processing, step-wise evaluation, error analysis, and subtitle handling are provided in Appendix
C ROBUSTNESS OF STEP-WISE REASONING EVALUATION
Validation of Qwen as Judge Model: To assess the reliability of Qwen-3-4B as a judge model for step-wise reasoning evaluation and its alignment with human scoring, we conduct an experimental analysis. We sample 15 questions from the benehmark and generate reasoning chains for each question using three models: GPT-o4- mini, Qwen-2.5- -wen-2.5-VL-722. and Intern VL-3-78B, resulting in a total of 45 samples. We then manually assign step-wise reasoning scores to each prediction using the same rubric used during automated evaluation. We com- pute the average score per model across the 15 questions and compare them against the scores assigned by Qwen-3. We observe small differences in the absolute scores, since manual ratings are harder to calibrate and can vary in strictness or leniency depending on the sample. As result, direct comparison is not always meaningful. However, the relative ranking remains consistent (GPT-4 InternVL-3 > Qwen-2.5-VL), This validates that Qwen-3 provides reliable scoring and aligns well with human evaluation.
Evaluator GPT-o4-mi mini Qwen-2.5-VL IntemVL-33
Qwen-3
7.0
5.6
6.4
Human
6.2
4.7
5.3
Accounting for Alternative Correct Reasoning Paths: Our step-wise evaluation rubric explicitly instructs the judge model to assign a full score if the final answer is correct and the reasoning is logically valid, even if the steps differ from the annotated ground truth. This instruction is stated in the evaluation prompt under Scoring Rubric-2, and is consistently used during evaluation of reasoning chains. To examine how well this instruction is followed during step-wise evaluation, we analyze the frequency and scoring of alternative reasoning paths. We focus on the MCQ Chain-of- Th-Thh with Subtitles setting and evaluate the two strongest performing models: Qwen2.5-VL-722. and GPT-o4-minn: For cases where the final answer is correct, we manually examined whether the predicted reasoning matched the annotated steps. For samples where the model took different reasoning path, we analyse the scores assigned by the judge model.
Qwen2.5-Ve -VLL7-B- answered 148 questions correctly, and among these, 46 followed a different reasoning path, all of which received full score. GPT-o4-mini correctly answered 256 questions, 110 using alternatiee reasoning; 109 of these received full scores. We also observe that 22 of these samples were common cases in which both models followed reasoning paths different from the ground truth. These results indicate that the judge model consistently follows the rubric and assigns full credit when the reasoning is logically valid.
The evaluation prompt asks the judge model to produce a structured scorecard that includes a numeric score along with a critique (see Appendix F). We present the L.LM judge evaluator generated critique for one such example: "The prediction uses coordinate geometry to show perpendicularity via dot product, which is a valid alternative reasoning path. While it differs from the ground truth's 's geometric transformation approach, the steps are logically sound and lead to the correct answer." This shows that the evaluator appropriately recognizes and credits valid alternative reasoning.
We chose Qwen3-4B primarily because it is open-source, reproducible, and accessible for low-resource settings. Importantly, we use Qwen3-4B in thinking mode for step evaluation and prompt the model to assign a score and provide a justification for the score in the form of a critique. This encourages the evaluation to be more grounded, consistent, and interpretable (see Appendix F.I). To validate robustness, we additionally conduct step-wise evaluations using three larger
Qwen3-4B for Step-wise Reasoning Evaluation:
15

---

Published as a conference paper ICLR 2026
Qwen3 models (8B, 14B, and 30B-A3) for four rep- resentative models: GPT-o4-mini, Gemini-2.0-Flann Qwen2.5-VL--22. and interVL-778B As shown in the table, the trends are consistent, and the average scores across different Qwen3 model sizes are very close to those reported using Qwen3-4B. We observe that larger Qwen3 models can be slightly more stringent in evaluation, but the relative performance ranking remains consistent across all model sizes (GPT-o4-ming Qwen2.5-VL-722 InternVL-78B Gemini-2.0-Flashh) This demonstrates that Qwen3-4B provides reliable comparative insights while maintaining reproducibility and computational efficiency.
Model
4B
8B
14B
30B-A3
Avg
GPT-o4-mini
6.9
6.6
6.6
6.9
6.8
Qwen2.5-VL-72B
5.0
4.8
4.8
5.0
4.9
InterVVI--78B
4.9
4.7
4.6
4.9
4.8
Gemini-2.0-Flash
4.7
4.5
4.4
4.6
4.6
D RELATED WORKS
The exploration of video understanding benchmarks has evolved rapidly alongside the developments of multimodal large language models (MLLMss). Early benchmarks such as ActivityNet-QA (Yu et al., 2019). NEXT-QA (iiao et al.] 2021), and CLEVRER (Yi et al. 2019a) primarily supported models that preceded MLLMs, focused on short clips with short answers, targeting action recognition, causal reasoning, or temporal ordering. They offered a limited scope for open-ended reasoning or compositional language generation.
With the introduction of MLLMs, the need for more comprehensive evaluation protocols became apparent. VCGBench (Maaz al., 2023) represented an early attempt to quantify MLLM performance by designing open-ended question answering benchmark built on ActivityNet videos. MVBench (Li et al. (024d) proposed an extensive multiple-choice setup covering wide range of perceptual and temporal reasoning tasks, though it was constrained to short video clips due to limitations in available datasets. Video-MME (Fu et al. 2025) addressed this gap by introducing a unified benchmark covering short, medium, and long videos. NEXT-GQA (Xiao et al. 2024) expanded the evaluation space by adding grounded reasoning questions. TempCompass provided a controlled probe into fundamental temporal properties such as event order and speed. As models matured, research turned to the more demanding setting of long-fou mormformmoo video understanding. EgoSchema (Mangalam et al. 2024) and MovieChat [Song et al. (2024) emphasized narrative comprehension by extending question answering minute-long (1-3 min) egocentric movie clips, while LongVideo VideoBench (Wu et al. 2024b). MLVU Zhou et al. (2055) and LVBench (Wang et al. 2024b) pushed the scale further to hour-long videos.
Beyond video duration, separate line of work investigates benchmarks that emphasize knowledge-intensive and reasoning-cent centric challenges. WorldQA (Zhang al.) 2024c), Video-MMLU (Song et al. 2025), Video- MMMU (Hu et al., (205cc), and MMVU (Zhao et al. 2025) explicitly target high-level subject matter, from STEM lectures professional training content, probing whether MLLMs can extract and apply domain expertise from video. VSI-Bench Yang et al. (2025b) complements this by diagnosing spatial intelligence, testing how well models infer layouts and object relations from egocentric perspectives.
While these benchmarks have advanced multimodal understanding in everyday contexts, they fall short of the deeper complexity posed by mathematical reasoning, where models must navigate non-linear content and demonstrate tightly integrated understanding across vision, audio, text, and background subject knowledge.
Multimodal Mathematical Benchmarks: Recent image-based mathematical benchmarks have significantly advanced the evaluation of multimodal models. MathVista (Lu et al.) (2224) and Math-V V (Wang al. 2024a) include visual questions drawn from textbooks and competitions, while MMMU (Yue et al.] (024a)) and MMMU-Pro (Yue al., 2024b) introduce subject-specific questions with CoT prompts and OCR inputs. DynaMath (Zuu et al. (2025) evaluates the robustness of mathematical reasoning through visual perturbations. Although these benchmarks span diverse range of mathematical topics, they fundamentally limited to
16

---

Published as conference paper ICLR 2026
assessing reasoning over static images, and the temporal dimension intrinsic video, where mathematical information may unfold through lectures, stepwise derivations, or interactive explanations, is not captured.
Benchmark
Domain
Dur. (s)
STEM
CoT
Annotation
CLEVRER Yi al.
2019a
Physics (Syn.)
V
X
Auto
MovieChat- K Song et al.
2024
Movie
500
X
X
Human
EgoSchema Mangalam al. 2024
Egocentric
X
Auto+Human
Video-MME Fu et al. 2025
General
1018
X
X
Human
Perception Patraucean ct al.
Perceptual
X
X
Human
MMBench- Video
Fang
2024
General
100
X
Human
LoncVideoBench
Wu et al. 2024a
General
473
X
X
Human
Video-MMMU Hu et al.
2025b
Scientific
Human
VideoMath
Math
241
Humant
Table Comparison of VideoMathQA with recent video-language benchmarks in terms of domain, duration, STEM focus, step-wise CoT annotation, and source. Annotations for our VideoMathQA provided by graduate-level human experts.
Video-MMMU (Hu al., 2025a) begins to explore video-based academic QA, but includes only a small subset of mathematical questions and focuses on general comprehension. It does not target the depth of reasoning modality integration that mathematical problem solving demands. contrast, VideoMathhAA designed to evaluate deep mathematical reasoning by challenging models to interpret high-resolution visuals, follow non-linear spoken explanations, and integrate vision, language, and domain knowledge over time, while providing detailed step-by-step reasoning annotations to enable comprehensive analysis beyond just final answer accuracy.
E LIMITATIONS AND FUTURE WORK
VideoMathQQ is an initial effort to annotate question-answer pairs along with step-by-step chain-of-thought reasoning temporally rich videos, where the answer cannot be inferred merely from a few static frames or the audio transcript. The selection and annotation of these samples require significant amount of time. For example, on average, it took graduate-level experts with at least Master's degree in Science approximately 30 minutes find suitable video, 40 minutes write good question and answer, and hour to compose detailed step-by-step reasoning. total, annotating one sample took around to 2.5 human hours, amounting approximately 115 working days for 420 samples. This effort is substantial and makes scaling the dataset size difficult. We identify this as a limitation of our work and plan to explore semi-automatic annotation pipelines in the future reduce the annotation effort while maintaining high annotation quality.
PROMPTS
Below we provide all the LLM prompts used this work, including (i) CoT prompting to clicit step-by-step reasoning, (ii) post-processing prompts to extract final answers from free-form responses, (iii) evaluation prompts for step-wise reasoning assessment, (iv) error analysis prompts for categorizing mistakes, and (v) hubtitle-handligg prompts ensure multimodal alignment.
17

---

Published as conference paper ICLR 2026
LLM Evaluation Prompt for Chain of thoughts Step Evaluation
You are a intelligent assistant for grading math question solutions. You will be given: mathematical question (question on) with multipl--choice options (opt ons);; A list of numbered ground truth steps (gt_steps) showing the correct reasoning to solve a math problem; A answer (answer) that is the correct final solution to the question; A model prediction perediction that includes the steps the model followed and possibly the final answer.
TASK: Compare the prediction to gt_steps and assign score out of 10 using the rubric below. You must reward both matching logic and valid alternative reasoning. Avoid overly strict step-by-ster comparison, instead focus if the model follows coherent and plausible mathematical approach.
Scoring Rubric:
1. Relative Step Matching (Main Criterion) Count the total number of ground truth steps: N.
Evaluate how many predicted steps correctly align with gt steps in terms of mathematical logic, reasoning, or computations.
-Score (matching steps N) x 10, rounded to nearest whole number.
AA step MATCHES if serves the same mathematical purpose, even if phrased or ordered differently.
2. Correct Final Answer via Different Reasoning
If the model's final answer is correct, and the steps are logically valid (even if they differ from gt gt.steps, assign a full score of 10.
Ignore number of matching steps in this case unless the reasoning is clearly flawed incoherent.
Reduce the score proportionally if reasoning contradicts parts of the ground truth.
3. Implicit or Inferred Steps
-Do NOT penalize if early steps are skipped, but later logic clearly depends on them.
- -If a model does not state "identify the chart," but proceeds to use correct values, assume it did this implicitly.
-ALWAYS check for implied steps before reducing the score.
4. Ignore Superficial Differences
-Do NOT deduct score for formatting, different notation or variable names, or additional clarifications. FOCUS on the mathematical meaning, not the literal step match.
Output Format
SCORE_CARD: ("matched.steps"" "X/N", "final wawaalaawsvrrcc.rettt.. <0 "<2--3 sentence sunmary>", "score": <0--10>}
1>, "critique":
Be strict when awarding credit. Do NOT be lenient. Carefully evaluate how far the model's reasoning aligns with the ground truth steps before assigning a score.
18

---

Published as conference paper ICLR 2026
Prompt for LLM Based Post Processing Kultiple-Choice Chain-of- -hhough Responses
Given the original multiple-choice options and a model-generated answer containing reasoning and a final answer, identify the option that best matches the final answer and return only the corresponding letter (A, B. C. D, or E).
The options are: {options)
The model response is:
{ {Response)
Only return the letter A, B. C. or E. If none is found, return None.
Prompt for Multiple-Choic Evaluation with Direct Answering
Select the best answer to the following multiple- choice question based on the video. Respond with only the letter (A, B, C, D or E) of the correct option.
Answer with the option's 's letter (A or B) from the given choices directly.
Prompt for Multiple-Choiee Evaluation with Chain-of-Thought
Select the best answer to the following multiple-choice question based on the video. Respond with the letter (A, B, C, or E) of the correct option.
First, please perform reasoning, and think step-by-step to provide the best answer to the following question with the option's 's letter (A, B, C, D or E) from the given choices.
Prompt for Multiple- Multiple-Choice Choice Evaluation with Subtitles
The subtitles of the video are listed below:
(Subtitles)
Select the best answer to the following multiple-choice question based on the video. Respond with only the letter (A, B, C, D E) of the correct option.
Prompt for Multi-Binary Evaluation with Direct Answering
Select the best answer to the following multiple-choice question based on the video. Respond with only the letter (A B) of the correct option.
Answer with the option's 's letter (A B) from the given choices directly.
19

---

Published as a conference paper ICLR 2026
LLM Evaluation Prompt for Error Analysis using CoT Step Evaluation
You are an intelligent assistant for analyzing model-generated math solutions. You will be given the following: A mathematical question (question) with multiple-choiee options (opt ions); A list of numbered ground truth steps (gt_steps) showing the correct reasoning and a correct final solution (answer); A model prediction (predicts ct) that includes the steps the model followed; A critique (critique), which is a short rationale describing the quality of the model's reasoning and its alignment with the ground truth.
TASK:
Your job is to carefully read and analyze the model's prediction. Compare to the ground truth steps and determine whether the prediction contains any of the following types of reasoning errors. Use the critique and reason about where and why the model diverges from correct logic. Then, assign all relevant error category labels from the list below. The error types are not specific each step; instead, identify a set of error types that apply to the overall reasoning.
The model fails to understand what the question is asking or misunderstands its demand. It cannot correctly interpret which quantity, relationship, or part of the video is referenced, often mixing up figures, charts, or examples.
TYPE1: Question Misunderstanding Error:
Information Retrieval Failure: The model cannot locate or recognize the needed data in the video, including charts. It overlooks presented numbers, labels, angles, side lengths, diagrams, or text overlays, so no raw facts are available for further processing.
TYPE2:
TYPE3: Visual Interpretation Error: Although the model attends to the correct visual element, reads it wrongly, misinterpreting axis scales, bar heights, marker positions, legends, or estimating distances and angles improperly.
TYPE4: Concept Application Error The model knows which principle or method applies but is not able to execute properly on the given question. It may recall the right concept yet misalign variables, swap parameters, or break the logical steps needed for that specific problem.
TYPE5: Strategy & Formula Selection Error: The model picks an entirely inappropriate approach,
choosing the wrong theorem, formula, or groblem-solving strategy for the task.
Recall & Memory Error: The model forgets or ignores earlier information that are essential. also covers cases when it contradicts key information from earlier in the question or in its own reasoning. This includes dropping previously used values, repeating steps unnecessarily, or breaking logical flow by not following through earlier steps.
TYPE6:
Computational Error: The model has the correct inputs and method but makes calculation mistakes, incorrect addition, subtraction, multiplication, division, rounding, unit-conversion errors.
TYPE7:
RULES:
Multiple error types may apply to a single prediction. Errors may be global (relevant to all steps) or local (relevant one few steps).
Important note: If the model uses different but valid reasoning and arrives at the correct answer, assign: none, even if the steps do not match the ground truth.
Do not assign an error unless you are confident it reflects real mistake. Important note: If you are not confident about which error applies, do not guess. Use uncategorized instead of forcing type. If clear mistake exists, but does not match any of the listed error types, assign: uncategorized.
OUTPUT FORMAT: Return only comma-separated list of error type labels. Examples: TTYPE2, TYPE4
-one
un unnattegrr zed Do not include any explanations extra text.
20


Reviews:
[{"review_id": "ekG1OEKyHB", "reviewer": "Reviewer_etW5", "summary": "This paper proposes a benchmark called videoMathQA which contains math/reasoning questions about materials presented in a video. The author claims that this work provides realistic reasoning challenges and they highlight that they use 920 hours of human labour for the annotation.", "questions": "See weaknesses", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The presentation of the paper is clear and the paper writing is good.\n\n2. The human labels might be useful as a point of comparison.", "weaknesses": "1. __Limited technical contribution__: \n\n1.1. While the authors claiming that they have filled the gap of video-based reasoning in math or very specific domains, there is work in [1] which have (1). videos in math, biology and various other scientific domains; and (2). question with high-quality human annotations which include reasoning. Therefore, the contribution might be incremental. A discussion and comparison should be made.\n\n[1] Sun et al. \"video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model\". ICML 2025.\n \n1.2. The size of the dataset is questionable especially when there are only 420 MCQs. The authors should justify whether 420 questions are enough to draw statistically significant conclusions when comparing different models. Especially when 10 sub-categories are given, as shown in Table 1, there are a lot of systems having the same score because there are not enough data samples. This makes analysis in the results part unconvincing because the difference might just be getting a couple more questions right. \\\nI would encourage the authors to theoretically compute the number of samples needed to have statistically significance in their results.\n\n1.3. Statistics and verification of human annotator quality missing. Statistics and evidence should be provided to show whether the annotators are reliable, such as inter-annotator agreement for their reasoning steps etc. I do not see a reason why we need the number of human-hour since the efficiency of human annotators can vary a lot.\n\n2. __Lack of justification for the need of Human Reasoning__:\n\nWhile a lot of efforts have been made to human reasoning annotation, it is unclear whether those are helpful performance indicators. While the authors try to do a comparison using LLM-as-a-judge. I have the following concerns regarding this part:\n  - I do not understand the results stated in Appendix C (Note that without referring to Appendices, the justification for this is missing in the main paper). Proper metrics, e.g. Pearson Correlation Coefficient at sample level should be given instead of saying the performance metrics are close to each other. Per-sample PCC is suitable here because we are assigning scores and comparing the absolute values of the scores. \n  - An easier and more objective way would be to perform rollout and check the rate of correctness to assign scores to each reasoning step. This is standard in reasoning literature and is more controllable and convincing to me than using LLM-as-a-judge, especially with such a small LLM. I wonder why the authors did not try this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ye4hNF2BbS", "reviewer": "Reviewer_fxx1", "summary": "VideoMathQA introduces a benchmark for math problem solving in instructional videos, bridging traditional text/image-based QA and full multimodal video reasoning. It includes a few thousand curated QA pairs from formula-rich YouTube videos, each with aligned frames, audio, transcripts, and formulas across 10 math domains (geometry, calculus, statistics, etc.). Each question features step-by-step solutions (chain-of-thought) for detailed evaluation. The benchmark targets three scenarios: Direct Problem Solving, Conceptual Transfer, and Deep Instructional Comprehension. Baselines with models like Math-LLaVA, MathBLIP, and Video-CoT reveal a large gap from human performance, underscoring the difficulty of integrating visual, textual, and temporal reasoning in mathematical contexts.", "questions": "1. Is there any analysis of the model performance without any video and only given the question?\n2. Is there any analysis on how reliable the evaluation scores are? is the model following the video at all, or having the video distracting the model from the correct solution?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces the first benchmark for math reasoning in instructional videos, capturing dynamic visual and spoken content absent in prior static text/image datasets.\n\n2. Built through 920+ hours of expert annotation, covering 420 problems (~4.5K QA pairs) with aligned formulas, video timestamps, and full step-by-step solutions.\n\n3. Spans 10 math domains and varied video styles (lectures, tutorials, handwritten, slides), testing both short-term perception and long-range reasoning.\n\n4. Measures both answer accuracy and chain-of-thought alignment, includes conditions with/without subtitles, and offers detailed error categorization for model failures.\n\n5. Evaluates multiple vision-language models (e.g., Qwen-VL, InternVL, Math-LLaVA), showing large performance gaps to human accuracy, confirming the benchmark’s difficulty and diagnostic value.", "weaknesses": "1. The five-option format simplifies scoring but allows guessing or elimination, limiting evaluation of free-form reasoning and creativity.\n2. Some Conceptual Transfer questions may be solvable without watching the video, letting models rely on prior knowledge rather than true video understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GKnbmaJoUr", "reviewer": "Reviewer_qm1z", "summary": "The paper introduce VideoMathQA, a new benchmark focus on cross-modal reasoning on video modality. This dataset is annotated by raduate-level experts with a long annotation process. It fills the gap of high quality math reasoning dataset in the video modailty scope.", "questions": "1. The overall dataset is not large. After distributing 420 videos across 10 categories, some categories contain as few as 17 videos (topology and graph theory each account for 4%). Within a category, videos are further distributed across different lengths. As a result, certain categories have only single-digit counts of videos at specific lengths. Could this introduce bias and error?\n2. In Fig2 a), the sum of all proportion is **112%**, how could this happen?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. VideoMathQA tests different reasoning scenarios: Direct Problem Solving, Conceptual Transfer, and Deep Instructional Comprehension, which supports hierarchical evaluation and training of models with varying levels of reasoning.\n2. The annotation process uses expert manpower to ensure annotation quality, thereby contributing to the benchmark community that current primarily relies on model auto generation.\n3. It supports a comprehensive evaluation to MLLM on Math Reasoning with the four question type: MCQ, MBin, Cot vs Direct, and step-wised.", "weaknesses": "1. Data scale: The benchmark only contains 420 videos.\n2. Lack the SOTA Proprietary Models' performace: the paper hasn't test GPT-5 and Gemini 2.5-pro\n3. The absence of cross-modal ablation experiments makes it difficult to quantify the actual contributions of each modality to mathematical reasoning.\n4. The paper use Qwen-3-4B as the judge model for Stepwise Reasoning Evaluation, but this is a relatively small size model. So the model's ability in evaluations is questionable, and it also lacks comparative experiments with human eval to demonstrate its feasibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzdS0TQeXg", "reviewer": "Reviewer_bHWm", "summary": "This paper introduces VideoMathQA, a benchmark designed to evaluate multimodal mathematical reasoning in real-world videos, addressing the limitations of existing static image/text-based math benchmarks that lack support for temporal extension, dynamic visuals, and cross-modal integration. The benchmark comprises 420 manually curated video-question pairs spanning 10 mathematical domains and video durations from 10 seconds to over 1 hour. Each pair includes expert-annotated multi-step reasoning (2,945 total steps) with timestamps, enabling fine-grained evaluation of intermediate inference.", "questions": "1. Given the high annotation cost, what semi-automatic or crowdsourcing strategies are you exploring to scale VideoMathQA? Could synthetic data or data augmentation techniques be integrated without compromising the \"real-world\" essence of the benchmark?\n2. Models perform poorly on some categories, such as topology and graph theory. Could you provide qualitative examples of why these domains are more challenging for current models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. VideoMathQA fills a gap by focusing on temporally extended cross-modal reasoning for math, an underexplored area in existing video benchmarks.  \n2. The benchmark leverages graduate-level experts to create detailed step-wise reasoning with timestamps. \n3. The four evaluation strategies address limitations of traditional MCQ and provide nuanced insights.    \n4. The authors systematically investigate factors impacting performance (model size, video duration, subtitles, frame sampling) and conduct error analysis across 7 categories, offering actionable guidance for model improvement.", "weaknesses": "1. With only 420 video-question pairs, the benchmark may lack sufficient diversity to generalize across all real-world math instructional scenarios. The high annotation cost raises concerns about scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0mcyOJBOI7", "reviewer": "Reviewer_oRiV", "summary": "This paper introduces VideoMathQA, a new benchmark designed to evaluate mathematical reasoning in videos. The authors argue that this task presents unique challenges not found in static-image or text-based benchmarks, requiring models to interpret fine-grained visual information, read text, and integrate spoken cues that unfold non-linearly over time.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The effort invested in data quality is a major strength. The three-stage annotation pipeline (Video Selection, QA Annotation, Step-wise Reasoning) is exceptionally rigorous. Using \"expert science graduates\" and \"involving different annotators for verification at each stage\" ensures high fidelity. The fact that this process required \"115 person-days\" demonstrates a significant and laudable effort.\n2. The experimental setup is thorough. The paper evaluates a wide range of models using four distinct strategies: standard Multiple-Choice (MCQ), a more robust Multi-Binary (MBin) evaluation, Chain-of-Thought (CoT) vs. direct answering , and a detailed Step-wise Reasoning Evaluation.\n3. The analysis provides clear insights into current model limitations including: (a) A large gap between human performance and the best model (b) The error analysis (c) Temporal Analysis and (d) The ablations on subtitles and frame sampling.", "weaknesses": "1. The primary limitation, which the authors correctly identify in the appendix, is the size of the dataset (420 video-question pairs). While the quality is extremely high, this small scale limits its utility for training and raises questions about the statistical robustness of the evaluation.\n2. The benchmark appears to be English-centric. The paper does not specify the languages covered, but the examples (e.g., in Figure 1) and discussion imply an English-only dataset. This limits its applicability for evaluating mathematical reasoning in other languages, where instructional content and terminology may differ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)