SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models

## Abstract

Abstract Abstract: Large Multimodal Models (LMMs) have achieved remarkable progress across various capabilities; however, complex video reasoning in the scientific domain remains a significant and challenging frontier.
Current video benchmarks predominantly target general scenarios where perception/recognition is heavily relied on, while with relatively simple reasoning tasks,
leading to saturation and thus failing to effectively evaluate advanced multimodal cognitive skills. To address this critical gap, we introduce SciVideoBench , a rigorous benchmark specifically designed to assess advanced video reasoning in scientific contexts. SciVideoBench consists of 1,000 carefully crafted multiple-choice questions derived from cutting-edge scientific experimental videos spanning over 25 specialized academic subjects and verified by a semi-automatic system.
Each question demands sophisticated domain-specific knowledge, precise spatiotemporal perception, and intricate logical reasoning, effectively challenging models’ higher-order cognitive abilities.
Our evaluation highlights significant performance deficits in state-of-the-art proprietary and open-source LMMs, including Gemini 2.5 Pro and Qwen2.5-VL, indicating substantial room for advancement in video reasoning capabilities.
Detailed analyses of critical factors such as reasoning complexity and visual grounding provide valuable insights and clear direction for future developments in LMMs, driving the evolution of truly capable multimodal AI co-scientists. We hope SciVideoBench could fit the interests of the community and help to push the boundary of cutting-edge AI for border science.

## 1 Introduction

Large Multimodal Models (LMMs) [34, 1, 2, 30, 41, 25] have demonstrated rapid advancements across a diverse range of capabilities, including conversational interaction [17, 52], code generation [20, 12], mathematical reasoning [10, 60], and image understanding [55, 19]. Among the various input modalities, video presents a particularly rich and complex form of multimodal data, uniquely integrating temporal dynamics, spatial perception, and high-level semantic reasoning. Consequently, video understanding has emerged as a critical frontier, pivotal for advancing LMMs towards next-generation applications in fields such as robotics, interactive education, and scientific discovery.

To evaluate the performance of LMMs, numerous video benchmarks have been developed [13, 15, 18, 44, 35, 62, 45, 39, 46, 26, 33]. However, the majority of existing benchmarks concentrate on relatively general domains—such as movies [15, 39], daily activities [32, 45], and instructional content [62]—and primarily emphasize tasks like temporal grounding, common-sense reasoning, and event understanding. While these benchmarks once posed significant challenges, current state-of-the-art LMMs, Gemini 2.5 Pro [14], now exhibit saturated performance, achieving accuracies exceeding 85% on popular benchmarks like VideoMME [15] and Neptune [33].

More recent initiatives [18, 40, 61] have begun to incorporate scientific and educational videos to assess deeper domain knowledge and reasoning. Nevertheless, most of this content remains at a college level, allowing LMMs to achieve success either largely through pure visual recognition with memorized domain knowledge (e.g., asking the virus type given the video) or the visual information does not dominate the reasoning process, thus lacking complex multimodal reasoning. For instance, part of the questions in MMVU [61] leverage visually-irrelevant hypotheses which could enable models to answer without watching the video, e.g., in a video depicting blowing air into a flask of limewater, which reacts with carbon dioxide to form a cloudy calcium carbonate precipitate. The associated question asks:“*Assume that 2.24 liters of gas fully participates in the reaction shown in the video under the standard temperature and pressure conditions, how many grams of precipitate are produced approximately?*”. The assumption about gas volume is actually the most important information to complete the calculation in order to answer the question. Consequently, these benchmarks pose a limited challenge to current proprietary LMMs (Gemini 2.5 Pro achieves 83.6 overall performance on Video-MMMU, and OpenAI o1 achieves 76.1 overall performance on MMVU), and can also be solved via a simple language-based reasoning framework [56, 53], which achieves 82.7 and 83.1 on Video-MMMU and Video MMLU, respectively.

Furthermore, current benchmarks inadequately evaluate sophisticated cognitive skills that necessitate expert-level knowledge, intricate logical reasoning, and precise visual perception. Moreover, the videos from these benchmarks are usually from daily life or simple college-level scenarios, while the advanced research-level scenarios, where complex scientific experiments happen, are ignored.

To address this critical gap, we introduce SciVideoBench, an innovative benchmark specifically designed to rigorously assess advanced video reasoning capabilities. SciVideoBench consists of 1,000 meticulously crafted multiple-choice questions derived from research-level experimental videos in physics, chemistry, biology, and medicine that are published with the corresponding journal publications.
These videos are sourced from more than 25 distinct academic subjects, including Fluid Mechanics, Analytical Chemistry, Neuroscience, and Oncology. Each question is categorized into one of three reasoning types: conceptual, hypothetical, or quantitative. As illustrated in Figure. [1](https://arxiv.org/html/2510.08559v1#S0.F1), SciVideoBench demands that models demonstrate accurate spatio-temporal grounding, possess profound domain knowledge, and execute sophisticated logical reasoning.

We developed SciVideoBench using a multi-stage, agent-human collaborative pipeline. This process involved mining associated experimental manuscripts, leveraging LMMs for initial question generation, and engaging domain experts to validate the question-answer pairs and filter out unanswerable or video-irrelevant questions.

Our evaluation of both proprietary and open-source LMMs on SciVideoBench reveals consistently low accuracy, underscoring the significant challenge this new benchmark presents and, consequently, the substantial opportunity for advancing research-level
video reasoning capabilities. In-depth analysis of the results reveals that factors such as model architecture, reasoning capacity, and perceptual grounding play a critical role in shaping video reasoning performance.
These insights not only offer clear guidance for future research efforts aimed at developing more sophisticated LMMs but also emphasize the broader potential of SciVideoBench.
As the first comprehensive research-level video reasoning benchmark, SciVideoBench not only provides a rigorous testbed for evaluating current video reasoning abilities, but also serves as a catalyst for innovation, fostering the development of highly capable AI co-scientists that can accelerate future scientific discovery.

## 2 Dataset Construction

**Table 1: Benchmark comparison for multi-modal video understanding and reasoning tasks.**
| Dataset | Video Domain | Difficulty Level | Knowledge-Driven | Reasoning-Intensive | # Ave. Duration (s) | Question Type |
| --- | --- | --- | --- | --- | --- | --- |
| MovieChat-1K [39] | Movie | Elementary | ✗ | ✗ | 564 | Open-ended |
| MLVU [46] | General | Elementary | ✗ | ✗ | 930 | Multi-choice |
| MVBench [26] | General | Elementary | ✗ | ✗ | 16 | Multi-choice |
| LongVideoBench [44] | General | Elementary | ✗ | ✗ | 473 | Multi-choice |
| TempCompass [29] | General | Elementary | ✗ | ✗ | < 30 | Multi-choice |
| Video-MME [15] | General | Elementary | ✗ | ✗ | 1018 | Multi-choice |
| VSI-Bench [50] | Embodied | Elementary | ✗ | ✗ | 122 | Multi-choice |
| MMWorld [16] | General | Elementary | ✓ | ✗ | 107 | Multi-choice |
| MMVU [61] | Scientific | College | ✓ | ✗ | 51 | Multi-choice |
| Video-MMMU [18] | Lecture | College | ✓ | ✓ | 506 | Multi-choice |
| Video-MMLU [40] | Lecture | College | ✓ | ✓ | 109 | Open-ended |
| SciVideoBench (ours) | Scientific | Research | ✓ | ✓ | 484 | Multi-choice |

### 2.1 Video Collection and Processing

To build a high-quality benchmark that rigorously evaluates advanced scientific reasoning in videos, we collect 241 research-grade experimental videos from the Journal of Visualized Experiments (JoVE)(^1^11[https://www.jove.com](https://www.jove.com)), a peer-reviewed platform dedicated to publishing methodological videos across a broad spectrum of scientific disciplines. These videos are professionally produced and narratively structured, clearly demonstrating laboratory protocols, scientific phenomena, and technical instrumentation. Their high visual and instructional quality makes them an ideal foundation for constructing a benchmark grounded in authentic scientific practice.

Crucially, each video is accompanied by both a peer-reviewed manuscript and synchronized audio narration. The manuscript describes the experimental protocol and results, while the narration provides temporally aligned explanations of each experimental step as it appears in the video. This tri-modal alignment—video, audio, and text—enables a principled and rigorous approach to question generation and answer verification. The accompanying materials offer detailed procedural descriptions, theoretical context, and experimental outcomes, which are instrumental in formulating meaningful, answerable, and visually grounded questions. This synergy ensures that each question in our benchmark is firmly grounded in both the visual content and its underlying scientific rationale.

We focus our selection on four foundational domains—physics, chemistry, biology, and medicine—which together span a diverse range of procedural complexity and reasoning challenges. Within these domains, videos are carefully selected to ensure the presence of clearly measurable variables (such as reaction time, temperature, or applied force), observable causal relationships and experimental outcomes, as well as logical sequences of actions or instrument usage that facilitate conceptual, quantitative, and hypothetical reasoning.
This targeted curation ensures that each video in SciVideoBench presents rich multimodal cues necessary for rigorous scientific reasoning, making it an ideal testbed for evaluating the capabilities of large multimodal models.

After downloading each video and its corresponding research paper, we use the Whisper [37] automatic speech recognition (ASR) model to extract the audio track and generate a transcript. This transcript is temporally aligned with the visual content to capture step-by-step experimental narration. For the associated paper, we parse and extract the full textual content from the PDF. These multimodal inputs—video, aligned transcript, and paper text—form the basis for downstream question generation. Further details on the preprocessing steps are provided in the Appendix.

Figure: Figure 2: Overview of our annotation pipeline. We manually annotate QA pairs for three example videos using both the video and the associated paper. A multi-agent LLM system generates and refines QA pairs: the QA Generator produces initial questions, the Evaluator answers them with reasoning, the Visual Comparer checks for visual grounding and timestamps cues, and the Refiner ensures questions rely on video content and improves option quality. Human experts verify and refine the final QA pairs. Audio transcripts are omitted for simplicity.
Refer to caption: x2.png

### 2.2 Annotation Pipeline

To generate high-quality, research-level question-answer pairs, we develop a semi-automatic annotation pipeline, as shown in Figure [2](https://arxiv.org/html/2510.08559v1#S2.F2), that integrates video, aligned transcripts, and paired research papers to provide rich scientific and procedural context. This setup enables accurate and visually grounded question generation while reducing manual workload through a multi-agent system.

To ensure quality and guide the semi-automatic annotation process, we first manually annotate a small set of exemplar cases. Four human experts (PhD students from biology, chemistry, medicine, and physics, respectively) review four selected papers (one for each subject) to extract key scientific concepts and procedural steps, then identify corresponding visual cues in the associated videos to construct sophisticated questions. These exemplar QA pairs are further refined and expanded via GPT-4o, resulting in 12 high-quality examples per question type. These are used as prompts to guide the human-in-the-loop multi-agent annotation system.

In the second stage, we assign distinct roles to a set of large language model agents, each responsible for a specific subtask in the annotation pipeline:

• QA Generator produces initial question–answer (QA) pairs from multimodal inputs—video, transcript, and associated paper text—guided by curated exemplar questions. Specifically, we design prompts for Gemini 2.5 Pro to first identify the core scientific question along with key aspects related to theory, experimental operations, and relevant numerical calculations for each video. Using these extracted elements and human-designed exemplars as references, Gemini 2.5 Pro then generates open-ended questions. To expand these into multi-choice format, the model is prompted to create nine scientifically plausible but incorrect distractors for conceptual and hypothetical reasoning, while for quantitative reasoning, distractors are automatically generated by adding appropriate Gaussian noise to the correct answer.

• Evaluator attempts to answer the generated question using the video, the transcript, and paper content, while also producing a detailed rationale that outlines the reasoning process. This step is to ensure all of the questions are answerable given sufficient information.

• Visual Comparer verifies whether the rationale is grounded in the video content by checking that all necessary visual cues are present. It also provides precise timestamps corresponding to each visual reference.

• Refiner refines the QA pairs by replacing generic or descriptive terms in the question with explicit references to visual segments, ensuring that the question cannot be answered without the video content. It also refines the options for each question to ensure the difficulty of the distractors.

• Human Verifier reviews the refined QA pairs and the outputs from the Visual Comparer to confirm that the question is fully grounded in the visual evidence. Specifically, these verifiers first check the timestamps annotation in questions to make sure the question is answerable based on the video content. If critical details (e.g., mass or temperature) are mentioned in the transcript but not visible in the video, the expert triggers a script to overlay this information directly onto the relevant video frames. Furthermore, the answer will be scrutinized to ensure that it matches the video content and the reasoning path.

In the final stage, the human experts perform a final review to manually correct any errors and improve clarity. The expert also ensures that each question is closely aligned with the core scientific contribution or focus of the experiment.

Figure: Figure 3: Discipline and subject distribution in SciVideoBench. Our benchmark covers four major scientific disciplines—Biology, Chemistry, Medicine, and Physics—encompassing more than 25 specialized subjects. This diverse coverage ensures a comprehensive evaluation across a wide range of scientific domains.
Refer to caption: x3.png

### 2.3 Statistics

We collected a total of 241 experimental videos spanning four major domains: Physics, Chemistry, Biology, and Medicine. These videos cover more than 25 distinct scientific subjects, as illustrated in Figure [3](https://arxiv.org/html/2510.08559v1#S2.F3). The videos in SciVideoBench have a competitive average duration of 484 seconds, with the duration distribution shown in Figure [4](https://arxiv.org/html/2510.08559v1#S2.F4)(a). This relatively long temporal scale ensures that the benchmark reflects the complexity and extended reasoning often required in real-world scientific experiments.

Figure: (a) Video duration distribution.
Refer to caption: x4.png

Building upon these videos, we annotated a total of 1,000 challenging questions that demand research-level knowledge for both perception and reasoning. As shown in Figure [4](https://arxiv.org/html/2510.08559v1#S2.F4)(b), the average question length demonstrates that the questions are more linguistically complex than those in existing benchmarks, reinforcing the emphasis on detailed experimental understanding. Furthermore, the option length distribution in Figure [4](https://arxiv.org/html/2510.08559v1#S2.F4)(c) highlights the balanced design of ground-truth answers and distractors, with comparable average lengths, thereby reducing annotation bias and ensuring fair evaluation. To further capture the nature of academic research and experimental analysis, we carefully designed three distinct question types that reflect common reasoning scenarios observed across the videos, as illustrated in Figure [5](https://arxiv.org/html/2510.08559v1#S2.F5). Collectively, these design choices ensure that the question–answer pairs in SciVideoBench require deeper domain expertise and multi-step reasoning, distinguishing it from prior benchmarks that mainly target elementary or undergraduate-level understanding.

- •
245 Quantitative Reasoning involves numeric perception, reasoning, and calculation. For a specific quantitative question, we require that all the numeric information must come from the video, which naturally ask the model to perceive informative values from the video before perform complicated calculations and reasoning.
- •
385 Hypothetical Reasoning focuses on specific experimental operations that play a critical role for the motivation or outcome of the whole experiment. This question type usually involves hypothesized error, what-if analysis, or experimental control logic, which requires both expert-level knowledge on a specific domain and accurate visual perception to capture important details in the videos.
- •
370 Conceptual Reasoning explore the mechanisms, protocols, and scientific principles behind the operations in the experiment that are inferred via visual/operational cues.

Figure: Figure 5: Examples of SciVideoBench, including videos from 4 disciplines (Physics, Biology, Chemistry, and Medicine), which involve 19 different subjects. The research-level QAs challenge LMMs in three different aspects (Conceptual, Hypothetical, and Quantitative) that are of vital importance in scientific experiment video understanding.
Refer to caption: x7.png

## 3 Experiments

### 3.1 Models

Vision-Blind Baselines To investigate the importance of visual information of our benchmark, we first conduct the vision-blind evaluation using GPT-4o [34] and Qwen2.5 series [49].

Proprietary Models We evaluate five proprietary LLMs that have superior performance over other popular video benchmarks, including Gemini-1.5-Pro [1], Gemini-2.0-Flash [1], Gemini-2.5-Pro [14], and GPT-4o [34].

Open-Source Models We comprehensively evaluate 30 open-sourced LMMs with the parameter volume ranging from as tiny as 0.5B to 78B, including Qwen-VL series [3, 41], InternVL series [8, 63, 7], InternVide2.5-Chat-8B [42], LLaVA-OneVision Series [24], LLaVA-NeXT-Video-32B series [25] and LongVA [58].

### 3.2 Evaluation Setting

We adopt the default frame sampling strategy for all open-source models. Specifically, we sample 768 frames for Qwen-2.5-VL [3], 512 frames for InternVideo2.5-Chat-8B [42], 128 frames for LongVA [58], 16 frames for InternVL2 [8], and 32 frames for InternVL3 [63], LLaVA-OneVision [24], and LLaVA-NeXT-Video-32B [25]. For Gemini [1, 11, 14], we set the frame rate to 1 FPS, and for GPT-4o [34] we use 256 frames. The temperature is fixed to 0 to ensure stable and reproducible evaluation results. All evaluations are using the LMM-Eval toolkit [57] and are conducted on 8 NVIDIA H100 GPUs.

### 3.3 Human Evaluation

We also conduct a human evaluation by recruiting three graduate students from each discipline to answer questions in a closed-book setting. Their overall accuracy is only 17.4% (Table [2](https://arxiv.org/html/2510.08559v1#S3.T2)), specifically, the quantitative reasoning is only 14.29%, showing that even advanced students cannot handle the benchmark, which requires research-level expertise.

**Table 2: Evaluation Results of Proprietary Models and Open-Source Models on SciVideoBench.**
| Models | LLM | Size | Overall | Question Type | Discipline |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Conceptual | Hypothetical | Quantitative | Biology | Chemistry | Medicine | Physics |  |  |  |  |
| Random Guess |  |  |  |  |  |  |  |  |  |  |
| - | - | - | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 |
| Human Evaluation (Graduate Students) |  |  |  |  |  |  |  |  |  |  |
| - | - | - | 17.40 | 18.11 | 18.70 | 14.29 | 15.88 | 16.06 | 21.19 | 18.88 |
| Vision-Blind Baselines |  |  |  |  |  |  |  |  |  |  |
| GPT-4o [34] | - | - | 15.80 | 14.05 | 20.00 | 11.84 | 17.53 | 13.95 | 16.13 | 14.33 |
|  | - | 0.5B | 12.40 | 11.35 | 13.51 | 12.24 | 13.20 | 13.94 | 11.21 | 10.97 |
|  | - | 1.5B | 13.40 | 11.62 | 19.22 | 6.94 | 14.43 | 12.73 | 14.95 | 11.91 |
|  | - | 3B | 16.40 | 17.03 | 20.52 | 8.98 | 15.89 | 16.36 | 15.89 | 17.24 |
| Qwen2.5 [49] | - | 7B | 16.70 | 18.65 | 19.74 | 8.98 | 18.34 | 10.91 | 14.95 | 18.18 |
|  | - | 32B | 17.10 | 18.11 | 21.30 | 8.98 | 19.32 | 16.97 | 13.08 | 15.67 |
|  | - | 72B | 18.90 | 21.89 | 18.44 | 15.10 | 19.07 | 18.79 | 20.56 | 18.18 |
| Proprietary Models |  |  |  |  |  |  |  |  |  |  |
| Gemini-2.5-Pro [14] | - | - | 64.30 | 69.73 | 67.79 | 50.61 | 64.79 | 61.82 | 74.77 | 61.44 |
| Gemini-2.5-Flash [11] | - | - | 46.40 | 50.81 | 44.16 | 43.27 | 44.01 | 49.70 | 55.14 | 44.83 |
| Gemini-1.5-Pro [1] | - | - | 27.50 | 27.84 | 28.31 | 25.71 | 27.38 | 26.06 | 27.10 | 28.53 |
| Gemini-2.0-Flash [1] | - | - | 25.70 | 28.38 | 24.94 | 22.86 | 24.69 | 26.06 | 22.43 | 27.90 |
| GPT-4o [34] | - | - | 24.90 | 30.27 | 28.05 | 11.84 | 21.52 | 29.70 | 31.78 | 24.45 |
| Proprietary Models w/ Chain-of-Thought Reasoning |  |  |  |  |  |  |  |  |  |  |
| Gemini-1.5-Pro [1] | - | - | 48.60(+21.10) | 47.03(+19.19) | 48.57(+20.26) | 51.02(+25.31) | 52.60(+25.22) | 44.19(+18.13) | 54.84(+27.74) | 44.78(+16.25) |
| Gemini-2.0-Flash [1] | - | - | 39.70(+14.00) | 39.19(+10.81) | 39.74(+14.80) | 40.41(+17.55) | 41.56(+16.87) | 40.70(+14.64) | 35.48(+13.05) | 37.01(+9.11) |
| GPT-4o [34] | - | - | 35.00(+10.10) | 37.84(+7.57) | 32.73(+4.68) | 34.29(+22.45) | 31.78(+10.26) | 37.58(+7.88) | 36.45(+4.67) | 37.30(+12.85) |
| Open-Source Models w/ Chain-of-Thought Reasoning |  |  |  |  |  |  |  |  |  |  |
| InternVL-3-78B [63] | Qwen2.5 | 72B | 37.90 | 51.62 | 35.58 | 20.82 | 38.39 | 36.36 | 38.32 | 37.93 |
| InternVL-3-14B [63] | Qwen2.5 | 14B | 34.20 | 46.49 | 30.65 | 21.22 | 35.70 | 30.30 | 37.38 | 33.23 |
| InternVL-3-14B-Instruct [63] | Qwen2.5 | 14B | 31.50 | 42.43 | 27.79 | 20.82 | 30.07 | 32.73 | 37.38 | 30.72 |
| InternVL-3-8B [63] | Qwen2.5 | 7B | 25.50 | 34.32 | 24.16 | 14.29 | 25.18 | 24.85 | 28.97 | 25.08 |
| InternVL2-Llama3-76B [8] | Hermes2 | 70B | 24.90 | 27.30 | 22.34 | 25.31 | 25.18 | 24.24 | 28.97 | 23.51 |
| InternVL-3-9B-Instruct [63] | InternLM3 | 8B | 24.00 | 29.19 | 22.08 | 19.18 | 25.92 | 27.88 | 17.76 | 21.63 |
| InternVL-3-2B [63] | Qwen2.5 | 1.5B | 22.20 | 30.00 | 21.82 | 11.02 | 21.76 | 16.36 | 25.23 | 24.76 |
| Qwen2.5-VL-32B-Instruct [3] | Qwen2.5 | 32B | 20.80 | 20.54 | 22.86 | 17.96 | 20.78 | 18.18 | 16.82 | 23.51 |
| LLaVA-OneVision-7B [24] | Qwen2.5 | 7B | 19.90 | 24.32 | 20.52 | 12.24 | 16.87 | 21.21 | 28.04 | 20.38 |
| Qwen2.5-VL-3B-Instruct [3] | Qwen2.5 | 3B | 18.10 | 19.19 | 18.96 | 15.10 | 17.60 | 18.79 | 19.63 | 17.87 |
| InternVL-3-1B [63] | Qwen2.5 | 0.5B | 14.00 | 16.76 | 13.51 | 10.61 | 15.40 | 13.94 | 14.02 | 12.23 |
| LLaVA-OneVision-0.5B [24] | Qwen2.5 | 0.5B | 10.40 | 11.08 | 10.65 | 8.98 | 8.56 | 12.12 | 13.08 | 10.97 |
| Open-Source Models (0.5B - 4B) |  |  |  |  |  |  |  |  |  |  |
| InternVL-3-2B-Instruct [63] | Qwen2.5 | 1.5B | 24.00 | 31.08 | 23.90 | 13.47 | 24.21 | 21.82 | 23.36 | 25.08 |
| InternVL-3-2B [63] | Qwen2.5 | 1.5B | 22.90 | 31.08 | 22.60 | 11.02 | 21.52 | 21.21 | 24.30 | 25.08 |
| InternVL2-4B [8] | Phi-3-mini | 4B | 21.30 | 26.22 | 24.16 | 9.39 | 18.09 | 24.85 | 30.84 | 20.38 |
| InternVL-3-1B-Instruct [63] | Qwen2.5 | 0.5B | 18.90 | 26.76 | 18.44 | 7.76 | 18.34 | 18.18 | 22.43 | 18.81 |
| InternVL-3-1B [63] | Qwen2.5 | 0.5B | 18.50 | 25.95 | 18.44 | 7.35 | 17.60 | 16.36 | 25.23 | 18.50 |
| Qwen2.5-VL-3B-Instruct [3] | Qwen2.5 | 3B | 16.10 | 20.00 | 17.92 | 7.35 | 15.16 | 13.94 | 18.69 | 17.55 |
| InternVL2-1B [8] | InternLM2 | 0.5B | 14.40 | 17.57 | 14.03 | 10.20 | 12.96 | 13.94 | 16.82 | 15.67 |
| InternVL2-2B [8] | InternLM2 | 2B | 13.10 | 14.59 | 15.06 | 7.76 | 11.98 | 12.12 | 12.15 | 15.36 |
| LLaVA-OneVision-0.5B [24] | Qwen2.5 | 0.5B | 12.10 | 15.41 | 12.99 | 5.71 | 11.74 | 13.94 | 13.08 | 11.29 |
| Open-Source Models (7B - 14B) |  |  |  |  |  |  |  |  |  |  |
| InternVL-3-14B [63] | Qwen2.5 | 14B | 35.70 | 53.51 | 35.32 | 9.39 | 35.94 | 33.94 | 38.32 | 35.42 |
| InternVL-3-14B-Instruct [63] | Qwen2.5 | 14B | 35.70 | 53.24 | 36.36 | 8.16 | 35.70 | 34.55 | 38.32 | 35.42 |
| InternVL-3-8B [63] | Qwen2.5 | 7B | 30.50 | 44.59 | 30.39 | 9.39 | 29.10 | 31.52 | 35.51 | 30.09 |
| InternVL-3-8B-Instruct [63] | Qwen2.5 | 7B | 29.40 | 43.78 | 29.35 | 7.76 | 26.16 | 31.52 | 34.58 | 30.72 |
| InternVL-3-9B-Instruct [63] | InternLM3 | 8B | 29.20 | 40.27 | 30.91 | 9.80 | 29.10 | 29.70 | 33.64 | 27.69 |
| InternVL-3-9B [63] | InternLM3 | 8B | 27.20 | 38.65 | 27.79 | 8.98 | 25.67 | 26.06 | 31.78 | 28.21 |
| InternVideo2.5-Chat-8B [42] | Qwen2.5 | 7B | 25.30 | 37.84 | 23.12 | 9.80 | 20.29 | 23.64 | 31.78 | 30.41 |
| InternVL2-8B [8] | InternLM2 | 7B | 19.40 | 24.86 | 18.96 | 11.84 | 17.85 | 21.21 | 19.63 | 20.38 |
| LLaVA-OneVision-7B [24] | Qwen2.5 | 7B | 18.80 | 23.51 | 19.22 | 11.02 | 15.56 | 23.03 | 26.17 | 18.18 |
| Qwen2.5-VL-7B-Instruct [3] | Qwen2.5 | 7B | 16.40 | 18.92 | 17.14 | 11.43 | 16.87 | 15.15 | 16.82 | 16.30 |
| LongVA [58] | Qwen2 | 7B | 14.30 | 16.15 | 14.94 | 10.31 | 13.69 | 15.00 | 15.65 | 14.11 |
| Open-Source Models (26B - 40B) |  |  |  |  |  |  |  |  |  |  |
| InternVL-3-38B [63] | Qwen2.5 | 32B | 38.30 | 53.78 | 38.44 | 14.69 | 36.67 | 40.00 | 42.06 | 38.24 |
| InternVL-3-38B-Instruct [63] | Qwen2.5 | 32B | 37.30 | 52.43 | 37.14 | 14.69 | 35.94 | 39.39 | 40.19 | 36.99 |
| InternVL2-40B [8] | Hermes2 | 34B | 23.80 | 28.38 | 23.64 | 17.14 | 22.74 | 21.82 | 30.84 | 23.82 |
| Qwen2.5-VL-32B-Instruct [3] | Qwen2.5 | 32B | 21.50 | 24.86 | 24.68 | 11.43 | 22.49 | 16.36 | 20.56 | 23.20 |
| LLaVA-NeXT-Video-32B [25] | Qwen2 | 32B | 21.10 | 26.22 | 22.86 | 10.61 | 19.80 | 22.42 | 23.36 | 21.32 |
| InternVL2-26B [8] | InternLM2 | 20B | 19.50 | 21.89 | 20.26 | 14.69 | 18.83 | 18.18 | 22.43 | 20.06 |
| Open-Source Models (> 70B) |  |  |  |  |  |  |  |  |  |  |
| InternVL-3-78B-Instruct [63] | Qwen2.5 | 72B | 38.80 | 57.30 | 39.74 | 9.39 | 37.90 | 39.39 | 46.73 | 36.99 |
| InternVL-3-78B [63] | Qwen2.5 | 72B | 38.50 | 56.76 | 39.22 | 9.80 | 37.65 | 37.58 | 46.73 | 37.30 |
| InternVL2-Llama3-76B [8] | Hermes2 | 70B | 26.30 | 28.38 | 27.01 | 22.04 | 24.94 | 29.70 | 29.91 | 25.08 |
| Qwen2.5-VL-72B-Instruct [3] | Qwen2.5 | 72B | 20.30 | 21.62 | 20.78 | 17.55 | 21.27 | 16.97 | 24.30 | 19.44 |
|  |  |  |  |  |  |  |  |  |  |  |

### 3.4 Blind Baseline Results

To assess the role of visual information in SciVideoBench, we evaluate blind baselines by removing the video input and feeding only textual content to the models. As shown in Table [2](https://arxiv.org/html/2510.08559v1#S3.T2), GPT-4o achieves only 15.80% overall accuracy, with particularly poor results on Quantitative Reasoning (11.84%), barely above random guess. Qwen2.5 exhibits similar trends across scales: small variants (0.5B–3B) remain near chance, while larger models (32B and 72B) reach at most 18.90%. Despite moderate gains from scaling, both GPT-4o and Qwen2.5 fail to exceed 20% accuracy without visual input. Together with the time-specific, observation-heavy expressions, e.g., "starting at 02:31", in the questions shown in Figure [5](https://arxiv.org/html/2510.08559v1#S2.F5), these results underscore that visual information is indispensable for solving SciVideoBench, especially for reasoning types requiring precise observation and measurement from the video content.

### 3.5 Proprietary Models vs. Open-Source Models

As shown in Table [2](https://arxiv.org/html/2510.08559v1#S3.T2), proprietary models substantially outperform open-source counterparts on our SciVideoBench. The strongest proprietary system, Gemini-2.5-Pro, achieves 64.30% overall accuracy, whereas the best open-source model, InternVL-3-78B-Instruct, reaches only 38.80%. The gap is especially pronounced in Quantitative Reasoning, where Gemini-2.5-Pro attains 50.61%—more than double the best open-source score (22.04% by InternVL2-Llama3-76B). Nevertheless, top open-source models demonstrate competitiveness against earlier proprietary models: for example, most InternVL-3 variants outperform Gemini-1.5-Pro, and even InternVL-3-2B surpasses it on Conceptual Reasoning.
At the lower end, performance drops sharply. The weakest open-source model, LLaVA-OneVision-0.5B, achieves just 12.10% overall, with its Quantitative score (5.71%) falling below even the blind GPT-4o baseline (11.84%). On average, proprietary models achieve nearly double the accuracy of open-source models (38.83% vs. 18.14%), highlighting the considerable advantage of proprietary systems in handling the multimodal and reasoning-intensive demands of SciVideoBench.

### 3.6 Results of Different Question Types

As shown in the right portion of Table [2](https://arxiv.org/html/2510.08559v1#S3.T2), Quantitative Reasoning consistently emerges as the most challenging category across models, regardless of architecture or scale. This difficulty is evident not only in proprietary models but also in open-source models, where Quantitative scores are systematically lower than those for Conceptual or Hypothetical reasoning. In fact, several open-source models perform close to or even below the random-guess baseline, for example, InternVL2-4B achieves only 9.39%, and InternVL-3-1B-Instruct reaches 7.76% in Quantitative Reasoning. By contrast, Conceptual Reasoning generally yields the highest scores, slightly ahead of Hypothetical Reasoning, although the gap is modest. These patterns indicate that SciVideoBench places particularly high demands on precise numerical reasoning and multi-step calculation abilities, while Conceptual and Hypothetical questions—though still challenging—tend to be more approachable for current multimodal LLMs.

### 3.7 Results of Different Disciplines

Across different disciplines, the performance trends remain relatively consistent, indicating that no single discipline presents a disproportionately higher level of difficulty in SciVideoBench. For Gemini-2.5-Pro, Chemistry emerges as the lowest-performing discipline, whereas Medicine achieves the highest accuracy; while Gemini-1.5-Pro displays the opposite pattern, with its best results in Chemistry and comparatively lower performance in other disciplines. This contrast suggests that the impact of model architecture and scale can vary across domains, and that strengths in one discipline do not necessarily translate directly to others.

### 3.8 The Impact of Chain-of-Thought Prompting

To highlight the reasoning-intensive nature of our SciVideoBench benchmark and to examine the performance gains attributable to deeper reasoning chains, we further evaluate models using chain-of-thought (CoT) prompts as shown in Figure [6](https://arxiv.org/html/2510.08559v1#S3.F6), following the methodology in [61]. As shown in Table [2](https://arxiv.org/html/2510.08559v1#S3.T2), models with CoT prompting consistently and substantially outperform their vanilla counterparts.
Gemini-1.5-Pro shows the most pronounced improvement, with overall accuracy rising by +21.10%, and Quantitative Reasoning achieving a remarkable +25.31% boost—surpassing even the stronger Gemini-2.5-Pro. Gemini-2.0-Flash and GPT-4o also exhibit substantial overall gains (+14.00% and +10.10%, respectively). Notably, GPT-4o’s Quantitative Reasoning accuracy increases from a near-random 11.84% to 34.29%, underscoring the transformative impact of explicit reasoning steps.
Across reasoning categories, the magnitude of improvement is not uniform. On average, Quantitative Reasoning benefits the most from the thinking mode (+21.77%), while Conceptual Reasoning shows the smallest relative gain (+12.52%). This disparity suggests that the benefits of CoT are particularly strong in scenarios requiring precise multi-step numerical reasoning, whereas tasks relying on conceptual or high-level understanding may saturate earlier.

Figure: Figure 6: Chain-of-thought prompt used in SciVideoBench
Refer to caption: x8.png

Interestingly, for most of the open-source models, leveraging the chain-of-thought prompt hurt the overall performance; however, the Quantitative Reasoning ability improves obviously. For instance, the overall performance of Qwen2.5-VL-32B-Instruct drops 0.7% but the Quantitative Reasoning increases from 11.43% to 17.96%, which is a 57.13% relative boost. This demonstrates again the intricate reasoning cost in our quantitative questions, which requires multi-step calculation and can be obviously improved by the chain-of-thought prompt.

Figure: Figure 7: Chain-of-Thought (CoT) performance gains across proprietary models and open-source models. An obvious difference between proprietary models across all the reasoning aspects can be observed; while for open-source models, quantitative reasoning has a notable performance boost, while the other two reasoning aspects have negative impacts. This phenomenon again demonstrates that the quantitative settings in SciVideoBench require sophisticated multi-step reasoning that can benefit a lot from chain-of-thought prompts. Best viewed in color.
Refer to caption: x9.png

To further investigate the possible reason that CoT prompting has a such different impact for different reasoning type for open-source models, we analyze the predictions of three representative models: InternVL-3-78B, InternVL2-Llama3-76B, and Qwen2.5-VL-32B-Instruct. From Table [3](https://arxiv.org/html/2510.08559v1#S3.T3), we observe that chain-of-thought (CoT) prompting does not consistently improve evaluation performance across reasoning types. Globally, the number of instances where the direct strategy is correct but CoT is wrong slightly exceeds the reverse, indicating that CoT often introduces additional errors. A more detailed breakdown by reasoning type reveals that the negative impact is concentrated in Conceptual and Hypothetical reasoning. In these categories, CoT tends to amplify hallucinations or over-elaborate on causal relations, leading the model away from the correct choice, whereas direct answering can rely more on memorized factual knowledge or surface-level pattern recognition. In contrast, Quantitative reasoning benefits substantially from CoT: across the three models, we see more cases where CoT corrects errors that the direct approach would miss. This suggests that explicit reasoning chains help models externalize intermediate computational steps (e.g., arithmetic or logical comparisons), which are otherwise challenging when only a single final answer is required. A plausible reason why CoT hurts open-source models on conceptual and hypothetical reasoning is that open-source models are less robust in producing faithful long-form causal explanations, often hallucinating or over-elaborating when reasoning about abstract scenarios, but CoT provides a clear scaffold for step-by-step numerical comparisons, which directly reduces errors in quantitative tasks.

Compared to the CoT gains reported on other video reasoning benchmarks (using Gemini-2.0-Flash as an example) —MMVU [61] (+3.0%), Video-Holmes [9] (+12.5%), VideoMathQA [38] (+6.6%), and MMR-V [64] (+2.4%)—the +14.00% improvement on SciVideoBench is striking. This significant gap strongly indicates that SciVideoBench inherently demands more complex, multi-step reasoning than existing benchmarks, making it a more challenging and discriminative testbed for evaluating reasoning capabilities in multimodal models.

**Table 3: Comparison of agreement and disagreement between CoT and Direct answers across three models.**
| Model | Both Correct | Both Wrong Agree | Both Wrong Disagree | CoT Correct, Direct Wrong | Direct Correct, CoT Wrong |
| --- | --- | --- | --- | --- | --- |
| Overall |  |  |  |  |  |
| InternVL-3-78B | 259 | 253 | 242 | 120 | 126 |
| InternVL2-Llama3-76B | 139 | 287 | 340 | 110 | 124 |
| Qwen2.5-VL-32B-Instruct | 96 | 335 | 338 | 112 | 119 |
| Conceptual Reasoning |  |  |  |  |  |
| InternVL-3-78B | 149 | 77 | 41 | 42 | 61 |
| InternVL2-Llama3-76B | 58 | 125 | 97 | 43 | 47 |
| Qwen2.5-VL-32B-Instruct | 37 | 147 | 92 | 39 | 55 |
| Hypothetical Reasoning |  |  |  |  |  |
| InternVL-3-78B | 93 | 102 | 88 | 44 | 58 |
| InternVL2-Llama3-76B | 48 | 114 | 129 | 38 | 56 |
| Qwen2.5-VL-32B-Instruct | 49 | 145 | 106 | 39 | 46 |
| Quantitative Reasoning |  |  |  |  |  |
| InternVL-3-78B | 17 | 74 | 113 | 34 | 7 |
| InternVL2-Llama3-76B | 33 | 48 | 114 | 29 | 21 |
| Qwen2.5-VL-32B-Instruct | 10 | 43 | 140 | 34 | 18 |
|  |  |  |  |  |  |

### 3.9 The Impact of LMM Model Size

We investigate the relationship between model size and performance across four representative open-source model families: InternVL-3, InternVL-2, and Qwen2.5-VL.
In general, a positive correlation between parameter count and overall performance can be observed within each series; however, larger size does not universally guarantee higher performance, and diminishing or even negative returns occur in some cases.

For InternVL-3, the trend from smaller to larger configurations shows substantial gains in overall accuracy (14.0% $\rightarrow$ 35.7%). The largest jump occurs when scaling from 1B to 7B, where overall performance increases by over 11 percentage points, with notable improvements in Conceptual (16.76% $\rightarrow$ 43.78%) and Medicine (14.02% $\rightarrow$ 34.58%). However, gains are not strictly monotonic. For instance, the 9B variant (27.20%) underperforms compared to the 8B-Instruct variant (29.40%) in Overall, despite having more parameters, partly because of different language backbones.
For InternVL-2, performance improvements with scale are modest and less consistent. The overall accuracy rises from 14.4% (0.5B) to 21.3% (4B) but then plateaus, with the 20B model achieving only 19.5%. The 4B model outperforms the 7B variant in all question types except Quantitative, suggesting that architectural or training differences outweigh the size advantage.
For Qwen2.5-VL, the scaling law appears weaker. The 3B, 7B, and 32B models show modest overall performance increases (16.1%, 16.4%, 21.5%), but the 72B variant drops slightly to 20.3%. This plateauing, and in some cases regression, is particularly evident in Conceptual and Chemistry performance, where the 72B model lags behind the 32B variant. However, the 72B model shows strong Quantitative Reasoning, which requires advanced multi-step reasoning.

However, between different model series, a larger model does not always guarantee better performance. For instance, Qwen2.5-VL-72B-Instruct is not even as good as InternVL2-4B in overall performance.
Overall, while model size generally correlates with improved performance, especially in Conceptual and domain-specific disciplines (e.g., Medicine, Biology), the relationship is far from linear or absolute. Variations between close sizes—such as InternVL-3 7B vs. 8B, or Qwen2.5-VL 32B vs. 72B—underscore that scaling must be accompanied by corresponding improvements in data quality, training objectives, and architectural choices to realize consistent gains in our SciVideoBench.

Figure: Figure 8: The impact of LMM backbones on the performance.
Refer to caption: x10.png

### 3.10 The Impact of LLM Backbone Size

We comprehensively compare the performance across LMMs with different LLM backbones as shown in Figure [8](https://arxiv.org/html/2510.08559v1#S3.F8). In general, we can observe a clear positive correlation between the size of the LLM backbone and the overall performance of the corresponding LMMs. Larger backbones such as Qwen2.5-72B and Qwen2.5-32B consistently achieve the highest scores, while smaller backbones (e.g., Qwen2.5-0.5B, Qwen2.5-1.5B) exhibit significantly weaker performance. This trend highlights the critical role of language backbone capacity in driving the multimodal reasoning ability of LMMs.

Interestingly, the correlation is strong in the conceptual and hypothetical reasoning ($\rho=0.86$ and $\rho=0.88$, respectively), where scaling the backbone size almost monotonically improves performance for most of the VLM series (e.g., InternVL-3 series).
In contrast, the quantitative reasoning exhibits a much weaker correlation ($\rho=0.64$): simply increasing the backbone size does not yield proportional gains.
This indicates that the difficulty of quantitative reasoning cannot be resolved by scaling language ability alone, but likely requires more accurate visual perception and advanced numerical reasoning capability.

### 3.11 The Impact of Audio

To investigate how audio, which provides additional information about the experiments shown in the video, will affect the model performance, we evaluate Gemini-2.5-Pro [14] and Qwen2.5-Omni-7B [48] due to their capability of supporting full modality input. From Table [4](https://arxiv.org/html/2510.08559v1#S3.T4), we can observe consistent improvement brought by audio input for both Gemini-2.5-Pro and Qwen2.5-Omni-7B in overall performance (2.70% and 2.80%, respectively). This limited improvement also demonstrates that the visual content dominates the model performance.

**Table 4: Model Performance Improvement by Audio on SciVideoBench.**
| Models | Use Audio | Overall | Question Type | Discipline |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Conceptual | Hypothetical | Quantitative | Biology | Chemistry | Medicine | Physics |  |  |  |
| Gemini-2.5-Pro [14] | N | 64.30 | 69.73 | 67.79 | 50.61 | 64.79 | 61.82 | 74.77 | 61.44 |
| Gemini-2.5-Pro [14] | Y | 67.00 | 74.32 | 67.53 | 55.10 | 68.70 | 61.21 | 71.96 | 66.14 |
| Qwen2.5-Omni-7B [48] | N | 14.70 | 16.69 | 14.97 | 9.74 | 14.39 | 14.36 | 17.85 | 14.25 |
| Qwen2.5-Omni-7B [48] | Y | 17.50 | 20.29 | 18.51 | 12.80 | 18.24 | 16.52 | 18.19 | 17.67 |

### 3.12 Error Analysis

To gain an in-depth understanding of how far the current models are from real human experts, we comprehensively analyze the results of Gemini-1.5-Pro-Thinking and Gemini-2.0-Flash-Thinking. Comparing these reasoning steps with the real rationale from the annotation process, which was carefully verified by human experts, we found three noticeable errors: Incorrect Visual Perception, Inaccurate Reasoning, and Lack of Domain Knowledge. Importantly, most of the wrong responses stem from a complex error combination, e.g., both incorrect visual perception and inaccurate reasoning progress.

Incorrect Visual Perception (70.68%) is the most common factor that leads to an incorrect conclusion. It leads to misinterpret of what is visually shown in the video and identify the wrong moment in the temporal span. For instance, in Figure [15](https://arxiv.org/html/2510.08559v1#A5.F15), Gemini-2.0-Flash overlooks the on-screen textual information indicating the humidity, which leads to the consequence that it obtains incorrect option analysis for option J, maintaining standardized high humidity.

Inaccurate Reasoning Progress (63.25%) also constitutes a main portion of the error cause. It usually involves logical flaws despite correct or partially correct observations. In Figure [15](https://arxiv.org/html/2510.08559v1#A5.F15), Gemini-2.0-Flash fails to establish the connection between the operation of using potassium sulfate and the high-humidity consequence.

Lack of Domain Knowledge (49.40%) happens when models fail to connect the visual evidence with specific expert knowledge that helps answer the question. For example, in Figure [15](https://arxiv.org/html/2510.08559v1#A5.F15), the function of potassium sulfate should be a well-known expert knowledge for researchers; however, Gemini-2.0-Flash is not able to analyze even perceive the existence of the sulfate in the video, which is strong evidence of domain knowledge lacking.

### 3.13 Failure Case Analysis

From Table [2](https://arxiv.org/html/2510.08559v1#S3.T2), we observe that even the best-performing proprietary model, Gemini-2.5-Pro, achieves limited accuracy on SciVideoBench, revealing the capped reasoning capabilities of current MLLMs in complex scientific experimental scenarios. Chain-of-thought (CoT) evaluation provides clearer insight into these reasoning limitations when facing highly specialized, visually grounded questions.

Figure: Figure 9: Failure case example of Gemini-2.0-Flash-Thinking.
Refer to caption: x11.png

For example, in the case shown in Figure [15](https://arxiv.org/html/2510.08559v1#A5.F15), the question asks: “Why is the setup shown during the cooling step at 03:12–03:17 used?” The correct rationale is as follows: the setup is a sealed desiccator containing a saturated potassium sulfate solution, which maintains a constant relative humidity of approximately 98%. This high-humidity environment is crucial because the luminescent properties of the Ag-zeolite composite are highly sensitive to hydration. Cooling the sample in this controlled atmosphere ensures it equilibrates to a well-defined hydration state, avoiding batch-to-batch variation caused by fluctuating laboratory humidity. The correct answer is J (Maintains standardized high humidity).
However, Gemini-2.0-Flash (CoT) fails despite partially recognizing that the desiccator controls humidity. It incorrectly rules out “standardized high humidity” and instead selects G (Keeps the sample dry to avoid hydrolysis). This mistake stems from two major reasoning errors: (1) Ignored on-screen evidence – The model completely overlooked the clear text overlay “98% relative humidity” in the video, which directly confirms the correct answer. (2) Misinterpreted the chemical role of potassium sulfate – The model failed to associate potassium sulfate’s well-established function with generating a stable high-humidity environment, instead generalizing it as generic moisture control.
This example illustrates that solving such questions requires a combination of precise temporal localization (identifying the cooling step), fine-grained spatial perception (reading on-screen labels), and expert-level domain knowledge (understanding the chemical’s function in environmental control). Current models, even with CoT prompting, often break down when any one of these components is missing.

## 4 Related Work

### 4.1 Video Large Language Models

Recent advances in large multimodal models have led to the rapid development of LMMs that empower temporal perception and reasoning ability. VideoChatGPT [31] leverages a simple spatiotemporal pooling mechanism for the visual encoding, while the LLaVA [28, 25] series leverage anyres technique to unify both image and video encoding. Meanwhile, LongVA [58] and LongVILA [6] aim at processing long-term video by extending the context window of LLMs in a multi-stage training pipeline. However, their ability to perform scientific reasoning, such as interpreting experimental procedures, analyzing quantitative measurements, remains underexplored. To address this gap, we propose SciVideoBench, a domain-specific benchmark designed to evaluate video reasoning in real-world scientific scenarios.

### 4.2 Video Reasoning Benchmarks

Recent benchmarks have increasingly emphasized complex temporal reasoning and multimodal understanding to evaluate LMMs beyond the traditional video QA benchmark paradigm [54, 32, 47, 45, 43]. Perception Test [35] examines multiple reasoning modes such as descriptive, predictive, and counterfactual reasoning in real-world scenarios. MVBench [26] proposes 20 temporally grounded challenges that require understanding actions, object interactions, and motion cues across multiple frames. Video-MME [15] incorporates human annotations across videos ranging from 11 seconds to 1 hour, including subtitle and audio streams to promote multi-source comprehension. MMBench-Video targets long-context temporal reasoning by leveraging hour-long videos with open-ended QA. Domain-specific benchmarks have also emerged. WorldQA [59] focuses on long-chain reasoning with multimodal world knowledge, Video-MMMU [18] collects expert-level instructional videos across disciplines for multi-stage knowledge acquisition, and Video-MMLU [40] centers on lecture-level understanding in math, physics, and chemistry. Our work extends this line of research by introducing a scientific video reasoning benchmark grounded in real experimental settings, emphasizing measurement, calculation, and conceptual reasoning involved in specific experiments.

### 4.3 AI for Science

In recent years, artificial intelligence has made tremendous progress across a variety of scientific disciplines, including biology [21, 51, 27], chemistry [36, 4], materials science [22, 5], and mathematics [23]. These breakthroughs have rapidly pushed the boundaries of scientific research and demonstrated performance that rivals, and in some cases surpasses, traditional methods.
For instance, AlphaFold [21, 51] revolutionized protein structure prediction by achieving near-experimental accuracy through deep learning, enabling the release of structures for nearly all known proteins. In materials science, GNoME [22] employed graph neural networks to discover over 700,000 stable crystal structures, dramatically expanding the known materials space. In chemistry, models like OrbNet [36] and ChemCrow [4] integrate quantum chemistry and symbolic tools with language models to enable efficient simulation and synthesis planning.
Despite these advances, current AI systems remain far from achieving the capabilities of a versatile researcher who can perform complex scientific tasks across domains with expert-level depth and accuracy. In this work, we introduce SciVideoBench, a challenging benchmark that focuses on scientific video reasoning, requiring visual perception, domain-specific knowledge, and intricate reasoning. Our benchmark aims to assess how far current LMMs are from expert-level scientific reasoning and to foster the development of next-generation AI systems for science.

## 5 Conclusion

We present SciVideoBench, the first scientific video reasoning benchmark that demands PhD-level knowledge to perform complex reasoning grounded in real-world experimental scenarios. SciVideoBench bridges the gap between general-purpose video understanding benchmarks and advanced scientific reasoning. It spans four major scientific disciplines—Physics, Chemistry, Biology, and Medicine—and covers over 20 distinct subjects, encompassing a broad spectrum of scientific inquiry.
To enable rigorous evaluation, we design three complementary question types—quantitative, hypothetical, and conceptual—that reflect common reasoning challenges in scientific research. We evaluate a total of 21 models, including 6 proprietary and 15 open-source LMMs, to assess their current capabilities relative to expert human reasoning. Our results show that even the most advanced model (Gemini-2.5-Pro) still struggles with these challenging tasks.
To further explore the impact of reasoning augmentation, we evaluate model performance under chain-of-thought prompting. The results demonstrate clear performance gains, highlighting the importance of explicit reasoning in complex scientific domains.
We hope SciVideoBench will serve as a milestone benchmark for advancing LMMs and inspire future research in the AI for Science community.

## References

- Anil et al. [2023]
Rohan Anil, Orhan Firat, Hyung Won Chung, Yanping Huang, Patrick Lewis, et al.
Gemini: A family of highly capable multimodal models.
[https://arxiv.org/abs/2312.11805](https://arxiv.org/abs/2312.11805), 2023.
arXiv:2312.11805.
- Anthropic [2024]
Anthropic.
Claude 3 models.
[https://www.anthropic.com/news/claude-3-family](https://www.anthropic.com/news/claude-3-family), 2024.
Accessed: 2025-05-09.
- Bai et al. [2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Bran et al. [2023]
Amanda Bran, Thomas Black, Alex Lee, et al.
Chemcrow: Augmenting large-language models with chemistry tools.
*arXiv preprint arXiv:2304.05376*, 2023.
- Chen and Ong [2023]
Chi Chen and Shyue Ping Ong.
Graph networks as a universal machine learning framework for molecules and crystals.
*Nature Communications*, 14(1):799, 2023.
- Chen et al. [2024a]
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al.
Longvila: Scaling long-context visual language models for long videos.
*arXiv preprint arXiv:2408.10188*, 2024a.
- Chen et al. [2024b]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.
*arXiv preprint arXiv:2412.05271*, 2024b.
- Chen et al. [2024c]
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al.
How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.
*arXiv preprint arXiv:2404.16821*, 2024c.
- Cheng et al. [2025]
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan.
Video-holmes: Can mllm think like holmes for complex video reasoning?
*arXiv preprint arXiv:2505.21374*, 2025.
- Cobbe et al. [2021]
Karl Cobbe, Vedant Misra, Moritz Aschbacher, Michael Jiang, Emma Brunskill, Christopher Manning, and Mark Chen.
Training verifiers to solve math word problems, 2021.
- Comanici et al. [2025]
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Dong et al. [2024]
Qingxiu Dong, Tianyi Zhang, Lingfeng Gao, Yuxiao Dong, Zhilin Yang, and Jie Tang.
LiveCodeBench: Realistic evaluation of code generation with online interaction, 2024.
- Fang et al. [2024]
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen.
Mmbench-video: A long-form multi-shot benchmark for holistic video understanding.
*Advances in Neural Information Processing Systems*, 37:89098–89124, 2024.
- for Developers [2025]
Google AI for Developers.
Gemini models: Gemini 2.5 pro, 2025.
URL [https://developers.googleblog.com/en/gemini-2-5-video-understanding/](https://developers.googleblog.com/en/gemini-2-5-video-understanding/).
Accessed May 15, 2025.
- Fu et al. [2024]
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.
*arXiv preprint arXiv:2405.21075*, 2024.
- He et al. [2024]
Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al.
Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos.
*arXiv preprint arXiv:2406.08407*, 2024.
- Hendrycks et al. [2020]
Dan Hendrycks, Collin Burns, Steven Basart, Aakanksha Mazeika, Dan Lin, Tejas Paranjape, Tagyoung Song, Jacob Mazeika, Michael Tang, Nikolas Saunders, et al.
Measuring massive multitask language understanding.
In *International Conference on Learning Representations*, 2020.
- Hu et al. [2025]
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu.
Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos.
*arXiv preprint arXiv:2501.13826*, 2025.
- Hudson and Zitnick [2019]
Drew A. Hudson and C. Lawrence Zitnick.
GQA: A new dataset for compositional question answering over real-world images.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 6398–6407, 2019.
- Jiménez et al. [2023]
Carlos Eduardo Jiménez, Or Rubin, Kevin Sanik, Derrick Xin, Neelansh Mittal, Kyle Rector, Daniel Tarlow, Owolabi Legunsen, and Graham Neubig.
SWE-bench: Can language models resolve real-world github issues?, 2023.
- Jumper et al. [2021]
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.
Highly accurate protein structure prediction with alphafold.
*nature*, 596(7873):583–589, 2021.
- Kim et al. [2023]
Hyeonhu Kim, Daniel Uzan, Tian Xie, et al.
Accelerated discovery of stable materials using deep learning.
*Nature*, 620:1018–1024, 2023.
- Lewkowycz et al. [2022]
Aitor Lewkowycz, Nicholas Schiefer, Clayton Freeman, et al.
Solving quantitative reasoning problems with language models.
*arXiv preprint arXiv:2206.14858*, 2022.
- Li et al. [2024a]
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al.
Llava-onevision: Easy visual task transfer.
*arXiv preprint arXiv:2408.03326*, 2024a.
- Li et al. [2024b]
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li.
Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.
*arXiv preprint arXiv:2407.07895*, 2024b.
- Li et al. [2024c]
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al.
Mvbench: A comprehensive multi-modal video understanding benchmark.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 22195–22206, 2024c.
- Lin et al. [2022]
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al.
Language models of protein sequences at the scale of evolution enable accurate structure prediction.
*BioRxiv*, 2022:500902, 2022.
- Liu et al. [2023]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
*Advances in neural information processing systems*, 36:34892–34916, 2023.
- Liu et al. [2024]
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou.
Tempcompass: Do video llms really understand videos?
*arXiv preprint arXiv:2403.00476*, 2024.
- Lu et al. [2024]
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al.
Deepseek-vl: towards real-world vision-language understanding.
*arXiv preprint arXiv:2403.05525*, 2024.
- Maaz et al. [2023]
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.
Video-chatgpt: Towards detailed video understanding via large vision and language models.
*arXiv preprint arXiv:2306.05424*, 2023.
- Mangalam et al. [2023]
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik.
Egoschema: A diagnostic benchmark for very long-form video language understanding.
*Advances in Neural Information Processing Systems*, 36:46212–46244, 2023.
- Nagrani et al. [2024]
Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hornung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, et al.
Neptune: The long orbit to benchmarking long video understanding.
*arXiv preprint arXiv:2412.09582*, 2024.
- OpenAI [2024]
OpenAI.
Gpt-4o: Openai’s newest multimodal model.
[https://openai.com/index/gpt-4o](https://openai.com/index/gpt-4o), 2024.
Accessed: 2025-05-09.
- Pătrăucean et al. [2023]
Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira.
Perception test: A diagnostic benchmark for multimodal video models.
In *Advances in Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=HYEGXFnPoq](https://openreview.net/forum?id=HYEGXFnPoq).
- Qiao et al. [2020]
Zhenxing Qiao, Matthew Welborn, Anima Anandkumar, Frederick R Manby, and Thomas F Miller III.
Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features.
*Journal of Chemical Physics*, 153(12):124111, 2020.
- Radford et al. [2023]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.
Robust speech recognition via large-scale weak supervision.
In *International conference on machine learning*, pages 28492–28518. PMLR, 2023.
- Rasheed et al. [2025]
Hanoona Rasheed, Abdelrahman Shaker, Anqi Tang, Muhammad Maaz, Ming-Hsuan Yang, Salman Khan, and Fahad Shahbaz Khan.
Videomathqa: Benchmarking mathematical reasoning via multimodal understanding in videos.
*arXiv preprint arXiv:2506.05349*, 2025.
- Song et al. [2024]
Enxin Song, Wenhao Chai, Gaoang Wang, Yifan Zhang, Haoyu Zhou, Feng Wu, Hang Chi, Xudong Guo, Tong Ye, and Yan Lu.
MovieChat: From dense token to sparse memory for long video understanding.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 1714–1725, 2024.
- Song et al. [2025]
Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang.
Video-mmlu: A massive multi-discipline lecture understanding benchmark.
*arXiv preprint arXiv:2504.14693*, 2025.
- Wang et al. [2024]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Wang et al. [2025]
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al.
Internvideo2. 5: Empowering video mllms with long and rich context modeling.
*arXiv preprint arXiv:2501.12386*, 2025.
- Wu et al. [2024a]
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan.
Star: A benchmark for situated reasoning in real-world videos.
*arXiv preprint arXiv:2405.09711*, 2024a.
- Wu et al. [2024b]
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li.
Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024b.
URL [https://arxiv.org/abs/2407.15754](https://arxiv.org/abs/2407.15754).
- Xiao et al. [2021]
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua.
Next-qa: Next phase of question-answering to explaining temporal actions.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 9777–9786, 2021.
- Xiao et al. [2024]
Shitao Xiao, Peifei Wu, Yifan Zhang, Tianrui Li, Jinze Yu, Hua Wu, and Haifeng Wang.
MLVU: Benchmarking multi-task long video understanding, 2024.
- Xu et al. [2017]
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.
Video question answering via gradually refined attention over appearance and motion.
In *Proceedings of the 25th ACM international conference on Multimedia*, pages 1645–1653, 2017.
- Xu et al. [2025]
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin.
Qwen2.5-omni technical report.
*arXiv preprint arXiv:2503.20215*, 2025.
- Yang et al. [2024a]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024a.
- Yang et al. [2024b]
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie.
Thinking in space: How multimodal large language models see, remember, and recall spaces.
*arXiv preprint arXiv:2412.14171*, 2024b.
- Yang et al. [2023]
Zhenyu Yang, Xiaoxi Zeng, Yi Zhao, and Runsheng Chen.
Alphafold2 and its applications in the fields of biology and medicine.
*Signal Transduction and Targeted Therapy*, 8(1):115, 2023.
- Yang et al. [2018]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.
HotpotQA: A dataset for diverse, explainable multi-hop question answering.
In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 2018.
- Yu et al. [2025]
Shoubin Yu, Yue Zhang, Ziyang Wang, Jaehong Yoon, and Mohit Bansal.
Mexa: Towards general multimodal reasoning with dynamic multi-expert aggregation.
*arXiv preprint arXiv:2506.17113*, 2025.
- Yu et al. [2019]
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao.
Activitynet-qa: A dataset for understanding complex web videos via question answering.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 33, pages 9127–9134, 2019.
- Yue et al. [2024]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yan, Jun Zhu, Hongsheng Li, Yu Qiao, Hang Su, Liwei Wang, Yuxuan Sun, Wenhu Chen, and Yu Su.
MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 24310–24321, 2024.
- Zhang et al. [2025]
Ce Zhang, Yan-Bo Lin, Ziyang Wang, Mohit Bansal, and Gedas Bertasius.
Silvr: A simple language-based video reasoning framework.
*arXiv preprint arXiv:2505.24869*, 2025.
- Zhang et al. [2024a]
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu.
Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a.
URL [https://arxiv.org/abs/2407.12772](https://arxiv.org/abs/2407.12772).
- Zhang et al. [2024b]
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu.
Long context transfer from language to vision.
*arXiv preprint arXiv:2406.16852*, 2024b.
- Zhang et al. [2024c]
Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christopher Arif Setiadharma, Jingkang Yang, and Ziwei Liu.
Worldqa: Multimodal world knowledge in videos through long-chain reasoning.
*arXiv preprint arXiv:2405.03272*, 2024c.
- Zhao et al. [2025a]
Shulin Zhao, Tianyu Gao, Liangchen Luo, Renze Lou, Boxin Wang, Weiyang Liu, Hai Zhao, and Yue Zhang.
Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models, 2025a.
- Zhao et al. [2025b]
Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan.
Mmvu: Measuring expert-level multi-discipline video understanding, 2025b.
URL [https://arxiv.org/abs/2501.12380](https://arxiv.org/abs/2501.12380).
- Zhou et al. [2018]
Luowei Zhou, Chenliang Xu, and Jason J Corso.
Towards automatic learning of procedures from web instructional videos.
In *AAAI Conference on Artificial Intelligence*, pages 7590–7598, 2018.
- Zhu et al. [2025a]
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025a.
URL [https://arxiv.org/abs/2504.10479](https://arxiv.org/abs/2504.10479).
- Zhu et al. [2025b]
Kejian Zhu, Zhuoran Jin, Hongbang Yuan, Jiachun Li, Shangqing Tu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao.
Mmr-v: What’s left unsaid? a benchmark for multimodal deep reasoning in videos.
*arXiv preprint arXiv:2506.04141*, 2025b.

## Appendix A Appendix overview

This document provides more details of our approach and additional experimental results, organized as follows:

- •
§ [B](https://arxiv.org/html/2510.08559v1#A2) Discilpines and Subjects in SciVideoBench
- •
§ [C](https://arxiv.org/html/2510.08559v1#A3) Detailed configuration of evaluated models.
- •
§ [D](https://arxiv.org/html/2510.08559v1#A4) Gemini-assisted Metadata Generation
- •
§ [E](https://arxiv.org/html/2510.08559v1#A5) More Failure Cases.

## Appendix B Disciplines and Subjects

**Table 5: List of subjects grouped by their corresponding disciplines.**
| Discipline | Subjects |
| --- | --- |
| Biology | Biochemistry, Bioengineering, Biogeotechnology, Bioinformatics, |
|  | Genetic Engineering, Lipidomics, Mechanobiology, Microfluidics, |
|  | Mycology, Neurohistology, Neuroscience, Phycology, Proteomics, |
|  | Regenerative Biology, Structural Biology, Cell Biology, Molecular Biology |
| Chemistry | Analytical Chemistry, Electrochemistry, Green Chemistry, |
|  | Nanomaterials, Organic Chemistry, Photocatalysis, Photovoltaics, |
|  | Physical Chemistry, Radiochemistry, Materials Chemistry |
| Medicine | Biomaterials, Cardiovascular Research, Dentistry, Drug Delivery, Immunology, |
|  | Molecular Imaging, Nanomedicine, Oncology, Ophthalmology, Pharmacology, |
|  | Radiopharmaceuticals, Regenerative Medicine, Surgical Devices, Therapeutics |
| Physics | Acoustofluidics, Additive Manufacturing, Aerosol Science, Applied Physics, |
|  | Ceramic Engineering, Ceramics, Civil Engineering, Condensed Matter Physics, |
|  | Ecological Engineering, Electronics, Fluid Dynamics, Materials Science, |
|  | Mechanical Engineering, Microfluidics, Nanofabrication, Plasma Physics, |
|  | Semiconductor, Sensors, Soft Robotics, Thermal Engineering |

## Appendix C Configuration of Evaluated Models

All of the evaluated models in this paper were released after May 2024. The detailed configuration of the evaluated models are shown in Table [6](https://arxiv.org/html/2510.08559v1#A3.T6). We follow the default frame sampling settings of the official implementation of each model. The temperature is set to 0 to ensure stable predictions.

**Table 6: Models used in our evaluation with confirmed release dates where available.**
| Organization | Model | Release Date | Input Frames |
| --- | --- | --- | --- |
| Proprietary Models |  |  |  |
| Google | Gemini-2.5-Pro | June 17, 2025 | 1fps |
| Google | Gemini-2.5-Flash | June 17, 2025 | 1fps |
| Google | Gemini-2.0-Flash | February 5, 2025 | 1fps |
| Google | Gemini-1.5-Pro | September 24, 2024 | 1fps |
| OpenAI | GPT-4o | May 13, 2024 | 256 |
| Open-source Multimodal Models |  |  |  |
| OpenGVLab | InternVL-3 series | April 11, 2025 | 32 |
| Alibaba | Qwen2.5-VL | January 28, 2025 | 768 |
| OpenGVLab | InternVideo2.5 | January 21, 2025 | 512 |
| LMMS-Lab | LLaVA-OneVision | August, 2024 | 32 |
| OpenGVLab | InternVL-2 series | July 4, 2024 | 16 |
| LMMS-Lab | LongVA (7B) | June 24, 2024 | 128 |
| UCSD/CMU | LLaVA-NeXT-Video | May 10, 2024 | 32 |
|  |  |  |  |

## Appendix D Metadata Generation Using Gemini 2.5 Pro

Figure: Figure 10: The prompt for Gemini 2.5 Pro to generate metadata to support initial annotation.
Refer to caption: x12.png

Figure: Figure 11: The prompt for Gemini 2.5 Pro to generate distractors.
Refer to caption: x13.png

Figure: Figure 12: The prompt for Gemini 2.5 Pro to update distractors.
Refer to caption: x14.png

Figure: Listing 1: Example metadata generated from Gemini 2.5 Pro

## Appendix E More Failure Case Studies

In this section, we showcase more failure case among Conceptual, Hypothetical, and Quantitative Reasoning.

#### Case 1: Misinterpretation of Spectral Evidence.

InternVL-3-14B misread the electroluminescence spectrum by hallucinating multiple peaks and voltage-dependent shifts, concluding that multiple emissive species contributed to the emission. In reality, the spectrum displayed a single stable peak, indicating well-confined recombination. The model failed to recognize confinement as the key property and instead introduced instability, leading to the wrong answer. This reflects a tendency to over-generalize from prior knowledge of spectroscopy rather than grounding reasoning in the observed evidence.

#### Case 2: Misunderstanding Experimental Procedure.

Qwen2.5-VL-32B erred in interpreting the role of a 20-minute incubation step. It concluded that failure of incubation would prevent substrate hydrolysis, even though the substrate was not yet present during this period. The correct rationale emphasized pre-incubation of inhibitors with elastase, ensuring binding equilibrium before substrate addition. The model ignored this inhibitor-binding context, conflating incubation with general reaction progress, which resulted in selecting an irrelevant option.

#### Case 3: Incorrect Solvent Accounting.

InternVL-3-9B miscalculated the total solvent volume in a polymerization setup. It fabricated multiple additions of 1,4-dioxane and neglected the actual solvents present in the protocol (DMSO and DMF). Furthermore, it mishandled the inclusion of solvent-containing stock solutions and confused monomer additions with solvents. This misidentification led to both an incorrect summation and a mismatch between its numerical reasoning and the final selected option. The correct calculation, grounded in protocol details, yielded a total of 4.462 mL, which the model overlooked.

#### Conclusion.

Across these cases, a consistent pattern emerges: models often fail due to misalignment between the experimental context and the reasoning they generate. Errors stem from three main sources: (1) hallucination of experimental features not present in the input (spectral peaks, solvent additions), (2) failure to account for the temporal sequence of steps (substrate not present during incubation), and (3) confusion between categories of experimental components (monomers vs. solvents). These findings highlight the importance of grounding reasoning strictly in the provided experimental evidence rather than relying on generic domain priors. Improved training on step-by-step alignment between protocol details and answer derivation may reduce these systematic failures.

Figure: Figure 13: Failure case example of InternVL-3-14B with chain-of-thought prompt compared with the correct reasoning of Gemini 1.5 Pro. InternVL-3-14B makes the wrong predictions because of the incorrect visual perception of the experiment results, while Gemini 1.5 Pro correctly captures the unchanged state of the peak at different wavelength.
Refer to caption: x15.png

Figure: Figure 14: Failure case example of Qwen2.5-VL-32B-Instruct. The model makes the wrong prediction because of lacking the domain knowledge about the incubation process.
Refer to caption: x16.png

Figure: Figure 15: Failure case example of InternVL-3-9B. The model makes the wrong predictions because of the wrong calculating logic and wrong solvent identification.
Refer to caption: x17.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: How do you formally define and validate “research-level” reasoning? Can you provide cognitive complexity metrics or expert assessments that distinguish your questions from existing college-level benchmarks? What specific reasoning skills (causal inference, counterfactual reasoning, multi-step deduction) are required?
soundness: 3
strengths: 1. **Novel and challenging benchmark addressing a critical gap**
- Targets research-level scientific reasoning rather than college-level knowledge, filling a significant void in current evaluation frameworks (Sec. 1, Table 1)
- Demonstrates clear performance gaps even for state-of-the-art models like Gemini 2.5 Pro (≈27% accuracy vs. 64% human performance), indicating substantial room for improvement (Table 2)
2. **Comprehensive experimental design and evaluation methodology**
- Evaluates 35 diverse models across proprietary and open-source categories with consistent experimental protocols (Sec. 4)
- Implements multi-stage human-AI collaborative annotation pipeline with quality control mechanisms (Sec. 3.2, Figure 2)
- Provides detailed analysis across question types (conceptual, hypothetical, quantitative) revealing differential model capabilities (Sec. 4.3, Figure 5)
3. **High-quality dataset construction with rigorous validation**
- Covers broad scientific scope with 25+ subjects across Biology, Chemistry, Physics, and Engineering ensuring domain diversity (Figure 3)
- Provides clear categorization framework distinguishing conceptual understanding, hypothetical reasoning, and quantitative analysis (Sec. 3.2)
4. **Practical impact and reproducibility considerations**
- Addresses real-world scientific video understanding needs relevant to AI for Science applications (Sec. 2.2)
- Provides detailed error analysis revealing specific failure modes like ignoring visual evidence and missing domain knowledge (Sec. 4.4)
summary: This paper introduces **SciVideoBench**, a benchmark designed to evaluate advanced video reasoning capabilities of Large Multimodal Models (LMMs) in scientific contexts. The benchmark consists of 1,000 multiple-choice questions derived from 241 research-grade experimental videos sourced from the Journal of Visualized Experiments (**JoVE**), spanning over 25 scientific subjects across four major disciplines. Each question is categorized as conceptual, hypothetical, or quantitative, requiring sophisticated domain knowledge, precise spatiotemporal perception, and complex logical reasoning. The authors evaluate 35 models including proprietary systems (Gemini 2.5 Pro, GPT-4o) and open-source alternatives, revealing significant performance gaps with the best models achieving only modest accuracy compared to human experts. The work aims to address limitations in existing video benchmarks that have become saturated and fail to adequately challenge advanced multimodal reasoning capabilities.
weaknesses: 1. **Dataset construction and annotation concerns**
- Human annotation process details are underspecified, particularly regarding inter-annotator agreement and quality control metrics (Sec. 3.2)
- No reported analysis of potential annotation bias or consistency across different scientific domains (Sec. 3.2)
- Limited discussion of how question difficulty was calibrated or validated beyond expert review (Sec. 3.2)
2. **Limited theoretical analysis of reasoning requirements**
- Lacks formal characterization of what constitutes “research-level” reasoning versus college-level knowledge, relying primarily on intuitive distinctions (Sec. 1)
- No systematic analysis of cognitive complexity or reasoning depth required for different question types beyond basic categorization (Sec. 3.2)
3. **Methodological and statistical rigor issues**
- No reported confidence intervals or statistical significance testing for performance comparisons across models (Table 2)
- Missing analysis of question difficulty distribution and potential ceiling/floor effects in evaluation (Sec. 4)
- Limited discussion of potential dataset contamination or overlap with model training data (Sec. 4)

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Given that Gemini 2.5 Pro was used extensively in annotation, were any steps taken to validate that the benchmark doesn't inadvertently favor Gemini models? What would performance look like if a different model had been used for annotation? Is the high performance of Gemini 2.5 Pro on the benchmark because of this issue?
2. What metrics were used to validate annotation quality beyond human review? Was inter-annotator agreement measured? How many questions were rejected during the validation process?
soundness: 3
strengths: 1. The benchmark addresses a critical gap in evaluating LMMs on research-level scientific reasoning, moving beyond elementary/college-level content that current models are beginning to saturate.
2. The semi-automatic annotation pipeline with human-in-the-loop verification, multiple specialized agents (QA Generator, Evaluator, Visual Comparer, Refiner), and expert validation ensures high-quality questions that require genuine multimodal reasoning.
3. The paper evaluates 35 models across multiple dimensions (proprietary vs. open-source, different sizes, with/without chain-of-thought prompting) and provides detailed analysis of failure modes. And the benchmarks shows significant gap between close-source and open-source models' performance, highlighting a important direction for the open-source community to push.
summary: This paper introduces SciVideoBench, a new benchmark designed to evaluate advanced video reasoning capabilities of Large Multimodal Models (LMMs) in scientific contexts. The benchmark consists of 1,000 multiple-choice (more than 4 choices) questions derived from 241 research-level experimental videos from the Journal of Visualized Experiments (JoVE), spanning four major disciplines (Physics, Chemistry, Biology, Medicine) and over 25 specialized subjects. Questions are categorized into three types: conceptual, hypothetical, and quantitative reasoning. The authors employ a semi-automatic annotation pipeline combining multi-agent LLM systems with human expert verification. Evaluation of 35 models (5 proprietary, 30 open-source) reveals significant performance gaps, with the best model (Gemini 2.5 Pro) achieving only 64.3% accuracy, and best open-source model (InternVL-3-78B) only achieves 38.0, indicating substantial room for advancement in scientific video understanding.
weaknesses: 1. The paper lacks one doable ablation that whether the performance of models scale as the number of frames sampled scaled up. This is also essential for a video understanding benchmark.
2. The low human performance makes it difficult to assess whether questions are genuinely research-level or simply poorly calibrated

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Human baseline: What were the participants’ backgrounds and how closely did they match the disciplines of the videos? Were they allowed to use papers, subtitles, or frame-by-frame playback? What motivated the closed-book design?
2. Answerability: Is there a visual-evidence audit (timestamps, annotated frames, or criteria) verifying that each question truly requires watching the video?
soundness: 3
strengths: 1. The benchmark addresses an underrepresented domain: research-level scientific experiments, that goes beyond existing video QA datasets focused on everyday scenes or lectures.
2. The dataset construction pipeline (multi-agent annotation + human verification) is well-documented and demonstrates substantial engineering and curation effort.
3. The evaluation over a wide range of open and proprietary models provides a clear empirical baseline for future LMM research in scientific contexts.
summary: This paper introduces SCIVideoBench, a benchmark for evaluating scientific video reasoning in large multimodal models. It focuses on research-level experimental videos sourced from JoVE and constructs 1,000 multiple-choice questions across four disciplines. The benchmark aims to assess domain-specific, spatiotemporal, and logical reasoning abilities of LMMs, and reports results over 35 models, highlighting substantial performance gaps between proprietary and open-source systems. The work fills a relatively underexplored space between general video QA and specialized scientific reasoning.
weaknesses: 1. The human baseline design is insufficiently described—participants’ domain alignment, access to supporting materials, and the rationale for the closed-book setup remain unclear, raising questions about the interpretability of the 17% human accuracy.
2. Although the text-only baseline provides aggregate evidence for visual dependence, the paper lacks per-item verification (e.g., labeling or statistics on which questions are visually indispensable vs. text-sufficient).

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: Answerability audits. What fraction of items remain solvable from transcripts alone or audio alone? Can you release per-item tags indicating which modality is necessary/sufficient? 
Annotation reliability. Please report inter-annotator agreement, human edit rates on LLM-proposed items, and an artifact probe (e.g., train a small text-only classifier on distractor style to test leakage).  
Human study details. How were graduate students recruited and matched to subjects? What were the instructions/time budgets, and could they scrub video timelines? Any per-discipline human results?  
Audio impact. The appendix lists “Impact of Audio.” Can you quantify modality contributions (video vs. audio vs. both) per reasoning category in the main text?
soundness: 3
strengths: Originality. Focuses on scientific video reasoning with research-grade lab videos (tri-modal alignment: video, audio narration, and peer-reviewed manuscripts), addressing a gap in existing general video QA benchmarks.  
Quality. The human-in-the-loop multi-agent annotation pipeline aims to ensure answerability and visual grounding (timestamps + visual comparer + expert verification).  
Clarity. Scope and contribution are clearly stated; the paper documents evaluation settings and analyses (blind baselines, proprietary vs. open-source, CoT effects).  
Significance. Results reveal substantial headroom (e.g., human grads at 17.4%; big proprietary–open gaps), underscoring the difficulty and potential impact for LMM research on scientific reasoning.
summary: The paper introduces SciVideoBench, a benchmark targeting research-level scientific video reasoning. It comprises 1,000 multiple-choice questions drawn from research-grade experimental videos (sourced from JoVE) spanning physics, chemistry, biology, and medicine, with questions verified via a human-in-the-loop, multi-agent pipeline that enforces visual grounding with timestamps. The benchmark emphasizes conceptual, hypothetical, and quantitative reasoning beyond recognition. Reported results show: (i) very low human graduate-student accuracy (17.4%), (ii) “vision-blind” text-only baselines performing near chance (e.g., GPT-4o ~15.8%), (iii) a large gap between proprietary and open-source LMMs (e.g., Gemini-2.5-Pro 64.3% vs. best open 38.8%), and (iv) sizeable gains from chain-of-thought prompting for some models. The dataset redistributes annotations/metadata/timestamps (not full videos) under JoVE’s copyright constraints.
weaknesses: Scale & format. The benchmark is limited to 1k MCQs. At this difficulty level, multiple-choice can obscure partial competence and invite guessability. Consider expanding the scale and adding open-ended rationales or structured reasoning targets (e.g., numeric derivations, unit-checked answers) to better capture scientific reasoning depth.  
Domain coverage constraints. Dependence on JoVE (English) narrows modality and topical breadth, which the authors also note as a limitation. Broader sources (field studies, simulations, non-English material) would reduce domain bias and improve generality.  ￼
LLM-in-the-loop artifacts. LLMs are used in annotation (generation, rationales, phrasing). While expert verification is present, quantify and stress-test for annotation artifacts (e.g., distractor style regularities) and measure inter-annotator agreement to ensure reliability beyond model-authored templates.  
Text-only solvability & audio confounds. The blind baseline helps, but add stronger text-only and audio-only ablations per question type, plus adversarial filtering to ensure solutions truly require video evidence rather than transcripts or narration alone. (Appendix mentions “Impact of Audio” — consider promoting those analyses.)  
Human evaluation design. The 17.4% graduate-student accuracy suggests extreme difficulty. Provide more detail: subject expertise matching, time limits, interface, allowed replay, and report per-domain human scores and inter-rater stats to calibrate realism vs. trickiness.