ExpVid: A Benchmark for Experiment Video Understanding & Reasoning
Authors: Yicheng Xu
Yue Wu
Jiashuo Yu
Ziang Yan
Tianxiang Jiang
Yinan He
Qingsong Zhao
Kai Chen
Yu Qiao
Limin Wang
Manabu Okumura
Yi Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Model
Video Large Language Model
Nature Science
Benchmark
Primary area: datasets and benchmarks

Abstract: Multimodal Large Language Models (MLLMs) hold promise for accelerating scientific discovery by interpreting complex experimental procedures. However, their true capabilities are poorly understood, as existing benchmarks neglect the fine-grained and long-horizon nature of authentic laboratory work, especially in wet-lab settings. To bridge this gap, we introduce ExpVid, the first benchmark designed to systematically evaluate MLLMs on scientific experiment videos. Curated from peer-reviewed video publications, ExpVid features a new three-level task hierarchy that mirrors the scientific process: (1) Fine-grained Perception of tools, materials, and actions; (2) Procedural Understanding of step order and completeness; and (3) Scientific Reasoning that connects the full experiment to its published conclusions. Our vision-centric annotation pipeline, combining automated generation with multi-disciplinary expert validation, ensures that tasks require visual grounding. We evaluate 20 leading MLLMs on ExpVid and find that while they excel at coarse-grained recognition, they struggle with disambiguating fine details, tracking state changes over time, and linking experimental procedures to scientific outcomes. Our results reveal a notable performance gap between proprietary and open-source models, particularly in high-order reasoning. ExpVid not only provides a diagnostic tool but also charts a roadmap for developing MLLMs capable of becoming trustworthy partners in scientific experimentation.

Published as a conference paper at ICLR 2026
EXPVID: A BENCHMARK FOR EXPERIMENT VIDEO UNDERSTANDING & REASONING
Yicheng Xu Yue Wu'
Jiashuo Yu'
Ziang Yan'
Tianxiang Jiang'
Yinan He Yi Wang
Qingsong Zhao! Kai Chen'
Yu Qiao'
Limin Wang
Manabu Okumura²
'Shanghai AI Laboratory yxu0400e ww..tuuueeeee edu. sg, wangyi @ ab. org. cn
Instiiute of Science Tokyo
Naajing University
ABSTRACT
Multimodal Large Language Models (MLLMs) hold promise for accelerating sci- entific discovery by interpreting complex experimental procedures. However, their true capabilities are poorly understood, as existing benchmarks neglect the fine-grained and long-horizon nature of authentic laboratory work, especially in wet-lab settings. To bridge this gap, we introduce Exp the first benchmark designed systematically evaluate MI.LMss on scientific experiment videos. Cu- rated from peer-reviewed video publications, ExpVid features a three-level task hierarchy that mirrors the scientific process: (1) Fine-grained Perception tools, materials, and actions; (2) Procedural Understanding of step order and com- pleteness; and (3) Scientific Reasoning that connects the full experiment to its pub- lished conclusions. Our vision-centric annotation pipeline, combining automated generation with multi-disciplinary expert validation, ensures that tasks require vi- sual grounding. We evaluate 20 leading MLLMs on Exp Vid and find that while they excel at coarse- grarseeg recognition, they struggle with disambiguating fine details, tracking state changes over time, and linking experimental procedures scientific outcomes. Our results reveal notable performance gap between propri- etary and open-source models, particularly in high-order reasoning. ExpVid not only provides a diagnostic tool but also charts roadmap for developing MLLMs capable of becoming trustworthy partners in scientific experimentation. Data are publicly available at
com wwtthuu.commOpeeeSSttt http//peen...abbpVX....
INTRODUCTION
Scientific progress is driven by careful experimentation. In wet-lab settings such as biology, chem- istry, and medicine, researchers need to execute fine-grained actions with exacting precision, adhere stepwise protocols, and reason from procedures results (Gabrieli et al. 2025 Yagi al. 2025)
Yet understanding and reproducing these procedures are time-consuming for practitioners and opaque to newcomers. Recent advances in Multimodal Large Language Models (MLLMs) (Ope- nAl 2025 DeepMind 2025b: Bai et al. (025c) make it tempting to delegate parts of this workflow to artificial intelligence: perceiving experimental manipulations, checking procedural fidelity, and even connecting observed operations to scientific conclusions. Regarding this, a question remains: how well do current MLLMs understand real experimental footage?
Despite steady progress video-based benchmarks (Li et al. 2024a Hu et al. 2025 Hasson et al., 2025), most existing datasets emphasize general actions or activities or medical computer vision scenarios rather than authentic laboratory experimentation. These settings lack the distinctive chal- lenge of wet-lab work: visually subtle operations (c.g., pipetting microliter volumes), small and of- ten occluded tools, fine-grained materials and states, and long- horizon dependencies that link early preparation steps to downstream results. To our knowledge, there is no systematic evaluation target- ing the spectrum of capabilities needed for assisting research from operational perception through procedural understanding to higher-order scientific analysis in genuine experiment videos.
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2026
We introduce ExpVid, benchmark for scientific experiment video understanding and reasoning. spans 13 disciplines and centers on wet-lab experiments; small number of dry-lab or field en- gineering videos are included for breadth and completeness, while purely computational and most physics experiments are excluded. Each video is paired with a peer-reviewed publication to ensure scientific rigor and support annotations linking video experiments innovations and conclusions. In term of sources, Exp Vid is curated from online peer-reviewed research collection (JoVE), whose exo-view recordings capture real-world laboratory manipulations with detailed narration.
To assess models across both temporal and analysis difficulty granularity, Exp organizes data into three tiers: single-step perceptions within seconds, multi-step understanding over minutes, and full-experiment as scientific reasoning across extended workflows. In this regard, we define task hierarchy that mirrors how scientists work. the operational level, models must recognize tools, materials, quantities, and fine-grained actions in short clips. At the procedural level, models predict over stage-level segments by ordering steps, verifying completeness. and predicting next moves. At the reasoning level, models integrate visual evidence across the full video and relate it to the accompanying paper to answer questions about motivation, significance, and conclusions.
Specifically, we adopt vision-centric annotation method to generate viable question-answer pairs at multiple temporal scales, and then introduce human expertise to secure the correctness. Questions constructed so that visual cues, instead of background knowledge alone, are necessary, along with carefully designed distractors that are semantically and visually plausible. Multidisciplinary experts then validate, refine, and balance the items ensure domain fidelity and diversity across disciplines and procedures. This combination of automated construction and expert verification yields a relatively scalable yet rigorous benchmark tailored the realities of experimental science.
We use Exp Vid to evaluate 20 popular MLLMs (with both open-source and proprietary). The find- ings (in Sec. reveal clear strengths in coarse object recognition and short-horizon reasoning, but persistent challenges in (i) disambiguating visually similar tools and materials under occlusion, (ii) tracking quantities and states across steps, and (iii) connecting procedural evidence to scientifically valid conclusions. These also emphasize that reliable visual grounding and structured reasoning are most urgently needed in real laboratory settings (mostly wet-lab tasks). We believe these chart roadmap for MLLM research toward trustworthy assistants agents that can perceive, verify, and reason about real experiments rather than stylized demonstrations.
In summary, our contributions are given as:
We present ExpVid, to best knowledge, the first benchmark that systematically evaluates MLLMs on scientific experimental footage across three hierarchical levels: fine- iine-grained percep- tion, procedural understanding, and scientific reasoning.
We design a scalable vision-centric annotation pipeline that constructs multi-level tasks from videos, associated ASR transcripts and peer-reviewed papers, followed by rigorous multi- disciplinary expert validation and refinement.
We benchmark 20 leading MLLMs on Exp and provide an analysis of our results. We show Vid can work a foundation for measuring and advancing MLLMs in real laboratory settings.
RELATED WORK
Multimodal Large Language Models (MLLMs). MLLMs extend LLMs to multimodal domains by combining visual perception with linguistic reasoning. Both closed-source models (e.g.. GPT-
5 (OpenAT 2025), Gemini 2.5 Pro (DeepMind) (0256)) and open-source models (Chen et al. 2024b Zhu al.] Bai et al. 2025c Hong et al. 2025) demonstrate strong reasoning capabilities on multi- modal inputs. Some further address ultra-long video understanding, enabling reasoning over hours 2025c Wang et al. 2025b Li et al., 2024b). To advance scientific discovery, 2025a) is tailored for scientific domains. Nevertheless, MLLMs' ability to understand and reason over laboratory experiment videos remains underexplored.
of content (Bai et al.
Intern-S1 (Bai et al.
Video understanding benchmarks. Existing video benchmarks evaluate video models on general video understanding tasks, including for example, action recognition (Caba Heilbron et al. 2015 Sigurdsson et al.) 2016, Mangalam et al. (222). dense captioning (Das et al. 2013 Rohrbach et al.
Chai al. 2024), and temporal grounding (Gao et al. 2017 Lei al. 2021 Liu
2024
2

---

Published as a conference paper at ICLR 2026
Cevel-1: Fine Fine-grate teee--rraned Perception
Material
11. Tool
111. Quantity
11/ Operation
Saudian
Quatinn Whatloola lo beig aoed?
Qaalas Whatschandh oma??
Qasiaa
Mhat asteral procetedd
What bagpead tlectae!
Deposphils puper
Tamiltrnayring Socddan syringe Micnnpipette Gas Gas-dight springe
pris
Remooe toge from
Zebroaiihh embryre.
pris
Attach clip
Itrosphilla larva
Secure Peint
Tribolium pupse
D. 9ot
D. Apply tape to
Level-2: Procedural Understanding
Fly vial preparation, adult removal, and incubation
Step
Poar
[e] Preyore
Jus]
[s]]Perpars
[is] Select
[1]]
[sl]] Drupp
[1g) Incubate
[so) Analgoe
V. S.Step Ordering
Vi. Completeness Veriffcattion
ViL. Sequence Generazion
1/111. Step Prediction
Qreetton
Question
Qurettion
Stantion
What cor aegunnee!
Glven the (segment shep lte), which dupp was noc perfocmed the video?
Roard en (tull step determine the staps performed in the sideo.
Toned the dvp lie), step
about place the video.
Chopped
[3: Cappadilop
Propery Solact Deop faruie
C. Select - - Propare - Drog Inoube
Cornect
Sonnect Ansost
D. Prepare + Doop Incube - Sclect
Ixsert vlal and insert needle.
J. P.Prppree a Selec, Jnject...
Incabate elat at 60% RR
2 Level-3: Scientific Reasoning
Fecal Deposit Analysis of Drosophila for the Assessment of Antidiarrheal Drugs and Plant Extracts
Preparation and storago of food mixture.
Fly vial proparation and Incubatioo.
Post Post-rearing cleanup.
LX. EExperimental (nterimentu(Anaafiiii
Relative normal food, flies fed Psidium guajava extract showed significantly fecal deposits, smaller total deposit area, and lower and effect
both virgin males and females.
X. Sczentifii Discoverry
a standardized Drosophila fecal deposit assay, feeding files with Psidium guajeva extract demonstrated that the platform can used antidiorrheu screening, reductions lecal parameters achieved without affecting solid-food food
Figure 1: Illustration of three-level task hierarchy in ExpVid.
Video-MME (Fu et al. 2025) and MVBench (Li et al. 2024a) provide comprehensive evaluations on short video clips with multi-choice questions, while several works such as MLVU (Zhou al,, 2024), LVBench (Wang et al.) 2024c), VRBench (Yu et al.) (025), evaluate MLLMs on long video comprehension or introduce narrative-driven dataset for multi-step reasoning extended video contexts. These benchmarks advance perception and temporal reasoning, but remain agnostic to domain-specific scientific knowledge and experimental contexts.
Knowledge-driven and scientific benchmarks. Another stream of work emphasizes knowledge- intensive evaluation, requiring models to integrate discipline knowledge beyond perception. Chem-
Bench (Alampara et al. 2023). Math Vision (Wang et al. 2024b), and Math Vista (Lu et al. 2023 are
for specific domains. Broader efforts (Yue et al. 2024 Zhao et al. 2025: Waag et al. 2024d Chen et al. 2024a) target expert-level, multi-disciplinary tasks, with Video-MMMU (Hu et al.) 2025) extending this domain knowledge from videos. Recently, SCI-VID (Hasson al.) 2025) and
SFE (Zhou et al. (025)) further introduce scientific benchmarks, but focus on outcome recogni- tion (c.g., medical images), rather than understanding whole experiments. Yet real-world scientific discovery critically depends on lab experiments, where step-wise operations and tools drive results.
3 EXPVID: A SCIENTIFIC EXPERIMENT VIDEO BENCHMARK
We develop benchmark to assess the performance of MLLMs experimental footage. Specifi- cally, we mostly focus on wet experiments related to biology, chemistry and medicine. Only a few dry ones (e.g., field engineering) included while computational examples, examples from

---

Published as a conference paper at ICLR 2026
Comparison between ExpVid and some MLLM benchmarks. A and M indicate automatic and manual annotation, respectively.
Table I:
Benchmark
#QA Pairs
#Videos
Avg. Sec.
#Task Types
Annotation
Domain
General Video Benchmarks
MVBench (Li ct ul. 2024a
4,000
3,641
16.0
20
A+M
General
Video-MM (Fu et al. 2025
2,700
900
1,017.9
1
M
General
MLVU (hoo et al. 2024
3,102
1,730
930.0
9
M
Narrative
VRBench Yü et al. 2025
9,468
960
5,796.0
I
M
Narrative
Knowledge-driive Benchmarks
MMVU (haa al. 2025
3,000
1,529
51.4
2
M
Multi-dise.
Video-MNIMU [Hu
2025
900
300
506.2
3
Multi-disc.
Vision (Wang ct al.
2024b
3,040
1
Math
Vista (Lu el 2023
6,141
31
Math
MMMU Yue et al. 2024
11,500
2
M
Multi-disc.
ScienceQA (Sankh et at. 2022
21,208
1
M
Science
SeiBench (Wang
2025
789
1
M
Science
MMStar (Chen el al. 2024a
1,500
6
M
Multi-disc.
SFF (Zhou et al. 2025
830
66
M
Science
Exp
7,800
390
489.0
A+M 10
Science
physics are excluded. Since wet experiments commonly own higher operational costs and complex- ity than dry ones, they demand more in intelligent assistance and analysis. In the following, we first describe ExpVid's data curation (Sec. (.1), then present its task hierarchy (Sec.(3.2) and finally detail the annotation (Sec. 3.3) An overview of the benchmark construction pipeline is illustrated Fig.
3.1 EXPERIMENT DATA CURATION
Collection. We collect scientific experiment videos, automatic speech recognition (ASR) tran- scripts, and corresponding papers from the Research section of JoVE (Journal of Visualized Experi- ments), multi-disciplinary, peer-reviewed video journal. JoVE publishes ste--by-step experimental protocols in video format, allowing viewers to observe the fine-grained manipulations and precise procedures. Its exo-view recordings of lab experiments yield high-quality visual content, while asso- ciated ASR transcripts offer detailed procedural descriptions, which well-suited for annotation. The paired peer- peer-reviewed papers further allow us to design challenging reasoning tasks that bridge experimental procedures to research conclusions and scientific findings.
Filtering. For quality control, we apply multi-dimensional scoring process ASR transcripts via DeepSeek-R1 (Guo et al. 2025a). Each transcript is rated on five criteria (0-5 scale): 1) Conti- nuity: whether covers the video without temporal gaps or missing segments. 2) Alignment: whether its timestamps align with the actual video duration; 3) Clarity: its logical coherence, domain- appropriate terminology, and overall readability; 4) Integrity: whether records entire exper- imental workflow, including distinct procedural stages; 5) Focus: whether centers on procedures rather than background, lectures, or unrelated context.
An overall score is obtained by averaging across five dimensions, and only those scored at least overall with no dimension below 3.5 are retained, yielding a high-quality subset. Additionally, videos are constrained to the interquartile range of durations (25th-75th percentiles, 378-728s) to remove outliers. Within each scientific discipline, experiments are ranked by overall scores and man- ually reviewed to exclude videos that predominantly feature computer-screen displays or lack actual laboratory footage. Further, multi-disciplinary experts select 30 top-ranked experiments from each of the 13 disciplines, yielding 390 videos with ASR transcripts averaging 1,026 words. This ensures Exp Vid remains balanced and diverse. Detailed statistics, along with the list of all 13 disciplines, reported in Appendix B.1
Preprocessing. For systematical evaluation across temporal scales, process all videos into three-level hierarchy to probe distinct capabilities.
Level-1: Action-level Clips. We obtain ~ 10k clip-text pairs (with each lasting ~8s on average). Specifically, we segment ASR transcripts by punctuation and align each sentence with its times-
4

---

Published as a conference paper at ICLR 2026
Collection
Annotation
A5K
Level-1
jove Research
Material
Senternce
Target
Captian
MMG Generatton
Quantity
Il Operation
Level-2
Viles Propecsuss
Antolation Sourse
QA Geseralion
Mediciokk ltzath
Neurceckene
y Uheepliness ya Scortng Ranking
Completee
Seginent Step list
11Mehuaed MIQQ
Sky Onderieg
Integrity
Experiments
V. Caeplitmess Verifiatiis
Eastrvennen
Seperee
Target Step Remorea
Full Step List
Template Emibedling
2. Preprocess
Level-3
Level-1
To
Cevel22
had Stage Engroemtation
Panagraphi
Zacarab
Cinteayll Goneration
Sobesifff Diccesy
4. Verification
Mannta
Boi Willinte 5
Stepis
Full
Online Anootation Platfcmm
O Acapptal
Fojected
Mey Estracionn
Segment Step
2003
7800
Level3
FAR
Rebion
5000
Orgentdd
Figure 2: An overview of ExpVid construction pipeline.
tamp to cut the video. This yields clip-ASR sentence pairs that provide step-wise experimental narrations, well-suited for perception-oriict oriented tasks such as action or material recognition.
Level-2: Stage-level Segments. We get ~3.5k segment-text pairs with an average duration of 448.. We divide each experiment into semantically coherent stages (e.g., preparation, main pro- cedures, post handling). We use DeepSeek-RI to generate stage-level boundaries for each ASR transcript, guided by prompts that enforces both logical and causal continuity across operations. Each ASR paragraph is constrained to 20-60s to preserve temporal coherence while avoiding ex- cessive context length. From each paragraph, DeepSeek-R further extracts step-level operation descriptions to form segment step list. Concatenating all segment step lists reconstructs full step list, which serves a suitable basis for procedural understanding tasks.
Level-3: Full Procedure Videos. We directly preserve the full experiment videos (average ~88 minutes). In certain cases, remove concluding slides, figures, and data-analysis segments avoid potential shortcuts (c.g., models exploiting textual conclusions) and ensure evaluation relies on procedural content. This level targets long visual context and structural reasoning, requiring models to integrate information across extended experimental workflows.
3.2 TASK HIERARCHY IN EXPVID
Based on the processed videos of varied lengths, we define ExpVid's three-level task hierarchy, benchmarking MLLMs scientific experiment videos, ranging from short-term perception long- term reasoning. This design allows us to progressively evaluate models' abilities: whether they can recognize fine-grained visual details, predict over coherent experimental procedures, and ultimately reason scientific conclusions over lab experiments. Fig. iillustrates this hierarchy.
Level-1: Fine-grained Perception. It evaluates whether MILMs visually ground essential elements in short clips of individual steps through four Multi-Choice Question (MCQ) tasks:
Material Recognition: Distinguish the target experimental material and distinguish it from other plausible substances commonly encountered in laboratory settings.
5

---

Published as a conference paper at ICLR 2026
Tool Recognition: Identify the tools that appear in the scene and reject visually functionally similar distractors.
Quantity Recognition: Choose the correct numerical attribute (e.g., dosage, temperature) by visually interpreting scales, amounts, or counts.
Operation Recognition: Recognize the specific action being performed in the video and differ- entiate it from confusable but incorrect operations in the similar setup (e.g., Insert - Attach).
Level-2: Procedural Understanding. This type of task evaluate models on their reasoning about logical and temporal order across multiple steps within stage-level clips, including:
Step Ordering: Select the correct step execution order when the original sequence is perturbed into plausible but incorrect arrangements.
Sequence Generation: Given the candidates, find out the ordered steps that appear in the clip,
Completeness Verification: Given the candidates, detect the missing step in the clip.
Step Prediction: Given the first steps an experiment stage, predict the next step n.
Scientific Reasoning. It has two tasks that require models integrate visual experiment processes with domain knowledge to draw conclusions, in the form of fill-in-thebblank questions:
Level-3:
Experimental Analysis: Infer crucial conclusions from experimental data, e.g., compare current results with existing studies, highlight new findings, and explain the corresponding mechanisms.
Scientific Discovery:
Reason over the entire experiment video, move beyond current outcomes, and abstract broader insights, such as linking results or innovations to larger scientific phenomena, interpreting the significance in filling blanks of which domain or potential application values, and proposing improved solutions for the current limitations and new directions for this area.
3.3
VISION-CENTRIC ANNOTATION WITH KNOWLEDGE GUIDANCE
Our annotation pipeline adopts a semi-automatic strategy that combines LLM assistance with human expert verification. To ensure benchmark vision-centric, we deliberately avoid encoding contextual cues from the narration that could directly reveal answers during QA construction. Moreover, dis- tractors are crafted to be semantically or visually plausible, forcing models to rely on visuals rather than purely leveraging LLM priors and textual cues. To minimize the LLM bias, LLM is limited to extracting experimental entities (e.g., subjects, actions, tools) from ASR transcripts and transform- ing them into QA candidates. Human experts then review, refine, and validate these annotations for correctness. Building upon the hierarchy given in Sec. 3.1 and 3.2 we construct them as follows.
Fine-grained Perception. For the four perception tasks Material, Tool, Quantity, and Operation, candidate entities or actions are first extracted from ASR sentences by DeepSeck-RR R1 as targets and aligned with video clips, with Qwen2.5-VI captioner providing visual triggers verify their vis- ibility. Normalization preserves critical states of materials and essential identifiers of tools, while excluding under-specified generie terms. Then, these resulting targets are converted into four- option multiple-choice questions (MCQs), where distractors are generated by DeepSeek-R follow- ing task musk-specific prompt rules: for Material and Tool, distractors reflect visual/functional similarity or common confusions; for Quantity, they lie in a comparable numeric range to mimic perceptual errors; and for Operation, they are plausible but incorrect within the same experimental setting. This design forces models to ground their answers in visual signals.
Procedural Understanding. These four sequential tasks are built on step lists derived from ASR, The first is Step Ordering, where each segment's 's step sequence is converted into four-option MCQ with distractors generated by DeepSeck-Ri RI as plausible but incorrect permutations that still follow experimental logic. The other three formulated by embedding step list into question templates. Sequence Generation and Step Prediction use the full step list as the candidate set, where Step Prediction, additionally, the final step and its video are removed, with only segments containing at least three preceding steps retained; Completeness Verification instead uses the segment step list and randomly removes a non-fina step as the target answer.

---

Published as a conference paper at ICLR 2026
Scientific Reasoning. For Experimental Analysis and Scientificc Discovery, we construct annota- tions for each full experiment video based on its corresponding peer-reviewed paper. The paper is first processed with MinerU (Wang et al. 2024a) to extract key sections (Introduction, Results, Discussion), and GPT-5 is used to summarize findings as anchors for annotation. PhD--evel expert annotators then design two types of fill-i-thhe-blank question based experiment videos and cor- responding paper, under the following principles: 1) Solvable only through visual observation and requiring reasoning across the full experiment. 2) Should not be answerable without the video. 3) Constrained to a single precise answer, minimizing ambiguity and synonym overlap. 4) Should an- notate nulti-blank questions, where each question contains multiple blanks that capture several key information points.
Expert Verification. All annotations are rigorously human-verified through structured expert workflow. For personnel capacity, we maintained pool of roughly 15 Nomain-prepareed annotators per major scientific category (e.g., medicine, biology), with a total team size of about 50 annotators, enabling balanced and timely distribution of tasks across disciplines.
To ensure consistent and scalable annotation across heterogeneous tasks, we built a dedicated online annotation platform (see illustrative examples in Appendix E with task-specific interfaces tailored to each question type. These interfaces guide annotators through the rubric and enforce correct use of the annotation schema. Each annotation requires brief justification, even for approvals, ensuring transparency and quality control.
Annotators follow unified criteria across all tasks:
Video-grounded: All questions must be solvable using the visual evidence in the video.
No leakage or shorteuts: Stems must not reveal the answer; distractors must be scientifically plausible.
Only visually verifiable actions are retained; abstract or unobserv- able descriptions revised or removed.
Concrete, step-level fidelity:
Consistent formatting and clarity: Wording avoids unverifiable details and ensures each ques- tion has a unique, unambiguous answer.
Justified verification;
Annotators provide reasons for all accepted corrected items.
Each annotator first reviews the entire experiment video together with its associated paper (approx- imately 40 minutes per experiment), followed by question-level verification that averages 6-8 min- utes for Level-1, 13 minutes for Level-2, and 18 minutes for Level-3.
The overall verification pipeline includes a one-month pilot phase for rubric alignment and iterative feedback, followed by one month of formal annotation, ultimately yielding 7,800 QA pairs across 10 tasks under 13 disciplines. Additional benchmark statistics are provided in Appendix B.2 B.2
4 EXPERIMENTS
Evaluation models. reasoning
We evaluate MI.LMs covering both open-source and proprietary models, and not. On the open-source side, include Qwen2.5-VL (Bai et al. (022cc), Qwen3-
VL (Bai et al. 2025b), Intern VL3 (Zhu et al.)., Intern VL3.5 (Wang et al. 2025a). GLM4.5V (Hong
al. (025). ), Kimi-VL. (Team et al. 2025), and Intern-S1 (Bai et al. 2025a). For closed-source ones, we benchmark Seed-1 1.5-VL (Guo et al. 2025b), Gemini-2.5- Flash (DeepMind] (025a), Gemini-
2.5-Pro (DeepMind) 2025b). Claude-Sonnet-4 (Anthropic 2025), and GPT-5 (OpenAl 2025). A
full description of the evaluated models' configurations can be found in Appendix F
Metrics.
ExpVid adopts hierarchical evaluation metrics aligned with task design.
Level-1: All fine-grained recognition tasks are formulated as multiple-choice questions (MCQs). Performance measured by Top-1 Accuracy, defined as the ratio of correctly answered questions to the total number of Level-1 questions.
Level-2. This level comprises four task types: Sequence Ordering, Completeness Verification, Step Prediction, and Sequence Generation. The first three are MCQs. In Sequence Ordering, models select the correct step order from four candidates. In Completeness Verification, candidate options correspond to all steps within specific video segment, resulting in instance-dependent
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance of evaluated models on the ExpVid across 10 tasks under three levels.
Level-1
Level-2
Level-3
Model
Think
Tool
Mat.
Quan.
Oper.
Avg.
Ord.
Gen.
Veri,
Pred.
Avg.
Anal.
Disc.
Avg.
Human Performance
17.5
15.9
61.3
55.5
37.6
69.8
31.2
45.6
21.8
42.1
Open-sourrs MLLMs
Qwen2.5 VL-BB Instruct
32.0
33.9
49.0
62.4
42.6
56.2
20.8
20.7
1.3
24.6
25.2
21.4
23.3
MiMo MiMo-YLL-BB
x
34.2
33.7
44.2
62.4
42.4
43.9
28.5
18.5
11.4
27.4
28.7
25.9
27.3
MiMo-VL-7B-RL
36.1
29.1
53.6
67.8
44.3
64.8
32.3
24.9
15.6
34.3
29.3
27.3
28.3
IntemVL3-88
x
27.5
31.0
38.8
65.6
39.4
43.4
20.4
20.2
3,9
23.9
29.2
25.3
27.2
Intem LL3.5-8B
27.3
30.8
45.5
64.8
40.3
25.8
23.7
4.8
34.0
22.6
Intem-S1-mini
33.3
31.2
52.5
61.4
42.5
73.6
14.3
16.8
8.3
28.1
33.5
28.3
30.9
Keye- Beye-VL-BB-Preieer
22.4
38.9
60.8
32.6
19.1
1.7
14.6
9.5
Keye.VI..1.5.. V1.-L.5-8B
21.0
23.4
51.3
64.0
37.0
56.7
9.5
20.0
2.8
22.1
8.4
6.1
7.2
GLM-4.IV99B
GLM-4.5V
35.5
33.6
61.5
62.3
45.6
71.9
34,9
27.2
12.9
36.6
33.3
32.5
32.9
Kimi- VL-ASB-Thinking
34.6
32.6
40.7
59.5
40.8
18.2
23.3
6.2
20.0
24.6
LL3.5LL35538B
35.9
34.0
46.7
65.3
44,0
65.8
36.7
23.0
19.0
36,0
33.1
30.8
31.9
Intem VL3-788
35.1
34.3
75.8
50.9
45.5
19.8
41.9
40.3
2ween25-VL-72BB-44
30.5
34.7
54.5
64.5
43.9
86.3
34.1
23.8
0.3
35,9
31.9
29.3
30.6
Been--VL-235B-nnstuue
X
30.2
34.1
50.0
68.5
45.1
74.0
42.3
16.2
19.6
39.0
40.5
31.8
36.1
Qwen3-Ve 21.-2358-Thiii Think--.
35.7
30.2
48.6
68.8
43.9
69.9
36.6
20.3
24.3
37.8
30.2
36.3
33.1
Qwen3 21.-235B Think
36.6
31.0
48.6
71.0
44.9
68.5
38.3
21.6
27.0
38.9
32.2
38.1
35.1
Intern S1
38.9
35.2
58.9
73.8
49.9
82.2
45.0
24.1
15.4
36.0
43.0
36.3
39.6
Closed source MILMs
Seed-VL-1.5
32.9
24.6
43.9
69.2
40.7
48.6
19.8
27.9
42.5
32.0
29.4
30.7
Claude-Somee-44
x
25.6
31.2
54.3
61.9
40.8
78.7
37.6
16.5
11.6
36.0
29.1
30.1
29.6
Gemini-2.55Flaa
52.7
50.1
65.2
72.6
58.6
86.0
50.5
24.1
40.2
50.1
47.2
41.1
44.1
Gemini-2.5-Flash
52.7
50.7
71.9
73.3
60.2
85.1
54.3
22.3
38.0
49.8
44.8
41.3
43.0
Gcmini-2.5-Pro
53.1
45.9
64.3
80.8
59.2
61.3
26.8
49.6
53.8
50.6
Gemini 2.5 Pro
51.3
44.3
63.8
74.4
56.7
84.2
59.9
26.8
46.9
54.3
50.1
44.8
47.4
GPT-5
37.8
57.5
55.4
option counts (see Fig. 14 left). In Step Prediction, options are drawn from all steps in the full experimental procedure, also leading to variable option sizes (see Fig. 14 right). These MCQ tasks are evaluated using Top-1 Accuracy.
Sequence Generation requires producing ordered sequence of steps and is evaluated using the Jaccard Similarity Coefficient (range: (-1), measuring overlap between the predicted and ground- truth step sets. The overall Level-2 score is computed by averaging Top-1 Accuracy for MCQ tasks and similarity scores for Sequence Generation across all Level-2 instances.
Level-3. This level contains two tasks: Experimental Analysis and Scientific Discovery. All questions are formulated fill-in-the-blank. We employ lightweight language model to compare model outputs with reference answers. Each blank is assigned one point, and performance is measured by Per-blank Accuracy, defined the number of correctly filled blanks divided by the total number of blanks.
recruited 15 undergraduate students without specialized backgrounds in biomedical or related sciences. They represent participants with general knowledge and com- mon sense rather than domain expertise, providing realistic reference point for non-expert human understanding, Notably, for Level-3 open-ended cloze tasks, participants reported being unable to complete the questions without specialized training, so no human baseline is reported for this level.
Human performance.
4.1 RESULTS
We evaluate 20 MLLMs on ExpVid, as detailed in Tab. 2 Frontier closed-source models, notably GPT-5 and the Gemini-2.5 series, clearly outperform the human baseline. Saaiini-2.--Flas-TThie
8

---

Published as a conference paper at ICLR 2026
vs. w/o Frames
Experimental Analysis
Mat.
Accurssy (50
Toal
52.6
Quan.
524
Oper
11_Avg
39.6
58.6
79.6
Scientific Discovery
12.5
30
Gon.
50.5
40.2
Vari.
16.1
24.1
Aug
MiMo-At MIMo-4t-7B-BU-200
w/ frames
20
60
80
100
0
64
128
192
256
Accuracy (%)
Frames
Figure 3: Effect of input video frames.
reaches 60.2 on the Level-1 (I.1) average, and GPT-5 scores 57.5 on the Level-2 (1.2) average, well above the human averages 37.6 and 42.1, respectively.
Closed-source models also maintain a clear lead over open-source ones as shown in Tab. 2 gap that widens with task complexity. basic perception such as recognizing tools, materials, quantities, and operations, closed-source models hold notable lead. The top-performing Gemini-2.5-Fllan (with "think"") scores 60.2 on average. The best open-source models, InternVL3-778B and Intern-S1, achieve commendable but lower scores of 50.9 and 49.9, respectively. This indicates that while the gap exists, leading open-source models are becoming increasingly competitive in fundamental visual perception.
Concerning procedural understanding, the gap becomes more pronounced. GPT-5 leads with an av- erage of 57.5, followed closely by Gemini-2.5.. 2.5--Po at 54.3. The top open-source model, Intern VL3- 78B, lags with an average of 41.9. A deeper look reveals nuances: VLte78 excels at Step Ordering (87.1), even outperforming GPT-5 (85.1). However, it falls short more generative and predictive tasks like Sequence Generation (45.5 vs. GPT-5's 66.9) and Step Prediction (15.5 vs. GPT-5's 51.8). This highlights that while open-source models can master specific structured tasks, they struggle with more holistic procedural reasoning.
In Level-3 (L3) scientific reasoning, GPT-5 achieves leading average score of 56.4, with strong results in both Experimental Analysis (55.4) and Scientific Discovery (57.4), well ahead of all com- petitors. By contrast, the best open-source model, Intern-S1, reaches only 39.6, falling nearly 17 points short of GPT-5. It underscores the advanced reasoning capabilities of frontier closed-source models, which remain clear target for the open-source community.
4.2 MORE ANALYSIS
Scaling Effects in Open-Source Models. A clear and consistent trend found among open-source models is the positive correlation between model scale and performance. The InternVL family serves excellent
study. As the model size increases from Intern VL.3-B (L1: 39.4, L.: 23.9, 1.: 27.2) to Intern VL3.5-388 (L.1: 44.0, I.2: 36.0, L3: 31.9) and finally to InternVL.3-788 (L1: 50.9, L2: 41.9, L3: 37.7), performance improves across all three levels. This demonstrates that increasing model scale directly contributes to enhanced capabilities in perception, procedural understanding, and scientific reasoning tasks, validating scaling as crucial axis for experiment video understanding in the open-source ecosystem.
Potential Unbalanced Capabilities. The results also shed light on the relative difficulty of dif- ferent tasks. Within I.2, models consistently score highest Step Ordering, indicating strong

---

Published as a conference paper at ICLR 2026
ability to rearrange provided information. In contrast, scores for Completeness Verification and Step Prediction are significantly lower across all models, revealing a weakness in identifying missing in- formation and forecasting future actions. The extremely low score of Qwen2.5- ----ttt-nstt Step Prediction (0.3) despite its strong performance on Step Ordering (86.3) exemplifies the brittle- ness and uneven capabilities of current MI.LMs.
Effect of thinking. In Tab. 2 we find Thinking does not consistently improve results. The patterns from Qwen3-VLL VL under different thinking budgets further show that a high thinking budget can even reduce accuracy on several tasks. Regarding this, analyze error cases where Gemini-2.55Flaas with Thinking Budget=8,199 192 fails but the NoThinking mode succeeds. The Thinking model often adopts logic-oriented style: abstracting the problem, reasoning step by step, and proposing "rea- sonable" workflow. Yet it drifts from the actual video sequence and relies on priors. By contrast, the NoThinking model remains video-grounded. directly matching steps to visual order and pro- ducing concise, faithful descriptions. For example, NoThinking answers typically begin with "The video shows... whereas Thinking answers start with ". identify the most logical workflow... revealing reasoning beyond visuals (see Appendix F.4).
Vision centric. We compare G.minn-22.-FFlash with and without frame inputs on all L1 and L.2 tasks (the left of Fig. B). As result, inputting frames consistently boosts performance, with some tasks such as Step Prediction becoming unsolvable without visual cues. Even for tasks like Step Ordering, where models can sometimes infer the correct answer from scientific priors alone, adding video inputs still yields clear gains. This validates the vision-centric design of Exp Vid.
For long-video reasoning tasks in L3, we ablate frame counts in Fig. 3 right. Results show that visuals are indispensable: accuracy is near zero without frames and increases as more are added. However, models benefit differently. IiternVL.3.5 peaks early (~128 frames) and then declines, sug- gesting saturation or distraction from redundant inputs, whereas MiMo-V VI. and Kimi-VL, steadily improve up to 256 frames, reflecting stronger ability to leverage extended temporal context. This indicates MLLMs like VL3.5, trained mainly for image-text alignment, gain little from ex- tended sequences. In contrast, Kimi-V VL and MiMo-VL, which incorporated long-video data during long-conter activation training, continue improve with more frames. Overall, these findings highlight the critical role of vision and the varying optimal frame budgets across models.
Limitation. ExpVid currently focuses on wet-lab experiments, not covering the full spectrum of scientific inquiry. Domains such as physics, which often involve distinct experimental apparatus (e.g., optical tables, particle detectors) and abstract phenomena, or purely computational experi- ments and targe-scale engineering tests, remain underexplored. Reasoning tasks in Level-3 assess outcomes but do not illuminate the underlying reasoning process (e.g., chain-of-thee of-touughhh that links experiments to conclusions.
CONCLUSION
This paper presents ExpVid, the first benchmark dedicated to scientific experiment videos. With its three-level task hierarchy, vision-centric annotation pipeline, and expert-guided validation, Exp Vid gives systematic evaluation of MLLMs across fine-grained perception, procedural understanding, and scientific reasoning. Our empirical studies demonstrates both the progress and the persistent limitations of current models, highlighting directions for advancing trustworthy AI experimental science.
ACKNOWLEDGMENTS
We thank Shanghai Artificial Intelligence Laboratory for institutional support. We also sincerely acknowledge the domain expert annotators for their careful annotation and verification efforts in constructing the benchmark.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work involves the collection and annotation of scientific experiment videos sourced from JoVE, a peer-reviewed video journal. All data are publicly available under JoVE's license, and we do not involve any private, sensitive, or personally identifiable information. The benchmark focuses on laboratory procedures rather than human subjects, and no clinical or personally invasive data are included. Annotation was conducted by PhD-level domain experts with clear guidelines to ensure accuracy. fairness, and scientific integrity. Potential risks such as misuse for non-scientific or unsafe experimental replication are mitigated by providing the dataset strictly for research purposes. We adhere to the ICLR Code of Ethics in all aspects of this work, including dataset release, annotation transparency, and reporting of model limitations.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure reproducibility of our benchmark and experiments. Sec. 3.1 and Appendix [.T describe data collection and filtering criteria, including quantitative thresholds. Sec. 3.1 details preprocessing pipelines for constructing
benchmark. Sec. 3.3 Appendix and outline annotation templates, distractor generation heuristics, and expert verification pro- cesses. Evaluation protocols and metries for all tasks specified in Sec. 4 and Appendix
REFERENCES
Nawaf Alampara. Mara Schilling- Wilhelmi, Martiño fars-Garcia, Indraject Mandal, Pranav Khetarpal, Hargun Singh Grover, NM Anoop Krishnan, and Kevin Maik Jablonka. Probing the limitations of multimodal language models for chemistry and materials research. Nature compu- tational science, pp. 1-10, 2025.
Anthropic. Claude sonnet https:/www.artrrooccccco///aaadd cessed: 2025-09-21.
2025. Ac-
Lei Bai, Zhongrui Cai, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, et Intern-sl: scientific multimodal foun- dation model. arXiv preprint arXiv:2508.1576. 2025a.
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, al. Qwen3-vl technical report. arXiv preprint arXiv:2511.2166 2025b.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint a3X2 2025c.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and paltern recognition, pp. 961-970, 2015.
Wenhao Chai, Enxin Song, Yilun Du Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.0300 2024.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-270$3 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of soun-s multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b.
11

---

Published as a conference paper at ICLR 2026
Pradipto Das, Chenliang Xu, Richard F Doell, and Jason Corso. A thousand frames in just few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2634-26441 2013.
Google DeepMind. Gemini 2.5 flash.
hctee://beepnlssc.ooo//////////2 flash/ 2025a. Accessed: 2025-09-24.
Google DeepMind. Gemini 2.5 pro. https::/deeem//ndn httpss//deeemlaa.oooggg/////eh/p
gemini 2025b. Accessed: 2025-09-21.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, al. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal Ilms video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24108-24118, 2025.
Gianmarco Gabrieli, Irina Espejo Morales, Dimitrios Christofidellis, Mara Graziani, Andrea Gio- vannini, Federico Zipoli, Amol Thakkar, Antonio Foncubierta, Matteo Manica, and Patrick W Ruch. Activity recognition in scientific experimentation using multimodal visual encoding, Dig- ital Discovery, 4(2):393-402, 2025.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp.
5267-5275, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025a.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.0000 070622 2025b.
Yana Hasson, Pauline Luc, Liliane Momeni, Maks Ovsjanikov, Guillaume Le Moing, Alina Kuznetsova, Ira Ktena, Jennifer Sun, Skanda Koppula, Dilara Gokay, et al. Scivid: Cross- domain evaluation of video models in scientific applications. arXiv preprint arXiv:2507.03578, 2025.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints, pp. arXiv-2507, 2025.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025.
JoVE. Jove: Journal of visualized experiments. https:///wwz jove. com 12.
Accessed: 2025-09-
Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 1846-118888 1858, 2021.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal vidco understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195 22206, 2024a.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng. Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan Hc, Chenting Wang, et al. Videochat-flasee Hicrarchical compression for long-context video modeling. arXiv preprint ar Xrviv:2501 2000000500..74.... 2024b.
Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang W Chen. Et bench: To- wards open-ended event-level video-language understanding. Advances in Neural Information Processing Systems, 37:32076-321100 2024.
12

---

Published as a conference paper at ICLR 2026
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very form-form video language understanding. Advances in Neural Information Process- ing Systems, 36:46212-46244, 2023.
OpenAl. Gpt-5. https: httpe/eeaa..ccooggggggmoeee com/ 2025. Accessed: 2025-09-21.
Anna Rohrbach, Mareus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie descrip- tion. In Proceedings of the IEEE conference computer vision and pattem recognition, pp- 3202-3212, 2015.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya, Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289 200011,, 2022.
Gunnar Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsoureing data collection for activity understanding. In European conference on computer vision, pp. 510-526. Springer, 2016.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang. Chenzhuang Du, Chu Wei, et al. Kimi-yl technical report. arXiv preprint 05440,9504.07991, 2025.
Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409, 18839, 2024a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math- vision dataset. Advances in Neural Information Processing Systems, 37:95095- 951699 2024b.
Weihan Wang, Zchai He, Wenyi Hong. Yean Cheng. Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.080.0 2024c.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei. Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint 208. 2025a.
Xiaoxuan Wang. Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scien- tific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10633 2023.
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvide02. 5: Empowering video milms with long and 2025b.
rich context modeling. arXiv preprint arXiv:2501.
Yubo Wang, Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266-9522 2024d.
Takuma Yagi, Misaki Ohashi, Yifei Huang. Ryosuke Furuta, Shungo Adachi, Toutai Mitsuyama, and Yoichi Sato. Finebio: A fine mine-grained video dataset of biological experiments with hierarchical annotation. International Journal of Computer Vision, pp. 1-16, 2025.
Jiashuo Yu, Yue Wu, Meng Chu, Zhifei Ren, Zizheng Huang, Pei Chu. Ruijie Zhang, Yinan He, Qirui Li, Songze Li, et al. Vrbench: A benchmark for multi-step reasoning in long narrative videos. arXiv preprint arXiv:2506. 10857, 2025.
13

---

Published as a conference paper at ICLR 2026
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Puttern Recognition, pp. 9556-9567, 2024.
Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, al. Mmvu: Measuring expert-level multi musti-discipline video un- derstanding, In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 8475-8489, 2025.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang. Tiejun Huang, and Zheng Liu. Mlvu: A comprchensive benchmark for multi-task long video understanding. arXiv e-prints, pp. arXiv-2406, 2024.
Yuhao Zhou, Yiheng Wang, Xuming He, Ruoyao Xiao, Zhiwei Li, Qiantai Feng, Zijie Guo, Yuejin Yang. Hao Wu, Wenxuan Huang, al. Scientists' first exam: Probing cognitive abilities of milm via perception, understanding, and reasoning. arXiv preprint arXiv:2506. 10521, 2025.
Jinguo Zhu, W Wang, Z Chen, Z Liu, S Ye, L Gu, H Tian, Y Duan, W Su, Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL htlps://arxiv. org/abs/2504.100 9.
14

---

Published as a conference paper at ICLR 2026
A THE USAGE OF LARGE LANGUAGE MODELS (LLMS)
In our work, LLMs are employed to assist the automated data annotation pipeline, with the resulting annotations subsequently reviewed and refined by human researchers, In addition, LLMs used to support proofreading of the manuscript. All content presented in this paper is rigorously verified to ensure faithful representation of the authors' original intent and to eliminate any factual inaccuracies or hallucinations that might be introduced by the models.
B DATASET STATISTICS
Experiments by Discipline
Overall Distribution
Mean: 45%
Median: $05
Cavertennn Ingreetios
400
Devesppent
Number Exper tments
Burat on (secents)
Discipline
by
2000
Seconaai
Duratinn
b0e
MediumiQuality (n-99355)
01-5879)
(o-595)
Figure 4: Data statistics in Exp Vid collection and filtering. (a) Number of experiment videos per discipline before filtering. (b) Video duration distribution with mean 489s and median 505s, showing long-tail outliers beyond 2,000s. (c) Boxplot of video duration by discipline (whiskers. 1.5 IQR).
(d) Boxplot video duration by quality based on the multi-dimennionall scoring process.
In this section, we present key statistics of Exp Vid and its curation process.
B. STATISTICS IN DATA COLLECTION AND FILTERING
Fig. shows the overall video duration distribution, the number of experiments across disciplines, and the results of the multi-dimennional scoring process. As illustrated, the source collection (JoVE) initially contains tens of thousands of videos, with biology, medicine, and neuroscience among the largest disciplines. The raw duration distribution centers around 489s on average (median 505s), but includes -ail outliers exceeding 2,000s.
To ensure high quality, we retain only experiments with an overall score of at least 4 and no in- dividual dimension score below 3.5, resulting in 5,879 videos (37.2%). To further align with
15

---

Published as a conference paper at ICLR 2026
task hierarchy and maintain temporal diversity, exclude videos outside the interquartile range (378s-728s). After this coarse filtering guided by LLM-based ASR scoring, à multidisciplinary ex- pert team manually curated the final dataset. To balance disciplines, control annotation cost, and keep a manageable benchmark size, we preserve 30 experiments per discipline across 13 fields, yielding 390 experiments in total.
The 13 disciplines include: Genetics, Environment, Behavior, Cancer Research, Engineering, Chemistry, Biochemistry, Developmental Biology, Bioengineering, Immunology and Infection, Neuroscience, Medicine, and Biology.
Comparison Duration (log scale)
QA by
9.35
Duration (pecodss)
Pairs
730
744
740
791
39.525.4s
409
10.12.33
390
19
Level1
Levd2
Levil
Soop Peofection
Analys
Average Annotation by
Average Annotation
Experiments
0.0
340
DAte ger Experimeent
Aemoctation
273
2.000 Lauoa
3.3
105
Xdax
85
Hatenal
Opeatio
Quanday
Sesteesca Daet
stop oiderial
Prenecion
lerifaation
diedvery
Hateea
quenter
sen oolenid
Pheticioon
Asspot
Level Dine-grained Perception)
Level (Pracedural Understardinn)
Level (Scientific Reasoning)
Figure 5: Data statistics of video duration and annotations in ExpVid. (a) Average video/clip dura- tion and standard deviation across the three levels (log scale). (b) Number of annotations for each task. (c) Average number of words per annotation with standard deviation. (d) Average number of annotations per full experimental video across different tasks, with standard deviation.
B.2 STATISTICS IN CURATED BENCHMARK
We further provide detailed statistics of the annotated dataset in Fig. As shown in Fig. 5 (a), our preprocessing splits videos into three levels with relatively stable durations and small standard deviations. particular, the small variance Level-3 benefits from the filtering process, which controls video length during selection. The progressively longer durations across the three levels naturally support our design for evaluating different capabilities, emphasizing not only linguistic reasoning but also reasoning across temporal scales.
Fig. 5 (c) reports the token counts of annotated tasks. Sequence generation and step prediction at Level-2 contain significantly more tokens than other tasks, since their questions include the prede- fined full step list as context. This indicates that models must reason over multi-step procedures in video while simultaneously handling long textual contexts.
16

---

Published as a conference paper at ICLR 2026
Fig. (d) shows the number of annotations per experiment across disciplines. Since balance the number of experiments per discipline filtering, the small variance here reflects that Exp Vid spans diverse domains while maintaining annotation consistency. ensuring fair evaluation of models' cross-disciplinary capabilities.
C PERFORMANCE BY DISCIPLINE
We visualize the averaged performance on each task by discipline in Fig. 6 The figure shows that, because these disciplines closely related and primarily consist of web-based experiments, the performance differences across disciplines remain limited.
Lovel3
040
6/8
234
024
0.44
6.2
075
0.33
023
832
1.3
639
331
831
1.3
064
682
127
0.6
0.6
8.2
122
08
125
1.2
631
0.30
8.4
0M
944
622
6M
Figure 6: Three level performance averaged across models by disciplines.
17

---

Published as a conference paper at ICLR 2026
D EXAMPLES OF EACH TASK
In this section, we provide representative QA examples for every task across all three levels.
For Level-1, we include video frame snapshots facilitate clearer visual interpretation, shown in Fig. 7 In particular, for the Quantity Recognition task, we provide multiple examples accompanied by the human annotators' accepted reasoning and supporting evidence from the corresponding video frames, illustrating how the task is grounded in fine- ine-grained visual cues.
For Level-2 and Level-3, due the substantially longer temporal scale of the videos, we present key frames for each QA example in Figs. 9
and I0
The complete video examples can be accessed
through the anonymous link provided above.
Material
Tool
Operation
Question
Question
What material apprars researcher's
babiigg experimental step?
the penon doing with
this video segiennt?
A: tracheal commila
At microscopy slide
A: Tapping vertically dislogge material
hrart--lung block
It Penri dish
I: Sifting heriznntally overa containce
perfusion cirouit tabing
coversti
C: Stirting contents inside container Hokiing stationary pouring throughit Answer
lung hoppey sample
Quantity
See text for solution
Currttin
Charsfint
Cuention
How many trocars are used in the procedure?
What speed set on the orbital sbakes?
How many stitches are placed around the tradhea?
A:5
At go RPM
A:3
B:
6a RPM
Bi
RPM
RPM
Resean
Beason
Boseat
Five trocars are clearly visible the video.
The inver left display shows o.g RPM to0, indicating the shoisee is RPM
Three knots are vistble in the video, indicating three stitches were placed.
DVG
many plate?
many glass covertlips aze
A:16
A:4
A:100 mL
6
2
Bi ml.
2:
C-6
C:45 mb
Dr
Dr
D: md.
Resson
Beason
The plate is clearly 3x4 lypott, correspondingte 12 well plate.
The video shows glass coverstip place lato the well.
The visille "oo" marking on the tube corresponds the scale of standarr so mi. couical tube.
Figure 7: Level-1 QA examples, including Material, Tool, Operation and Quantity Recognition.
18

---

Published as a conference paper at ICLR 2026
Sequence Generation
Question
Based the full experimental procedure list shown below, determine numbers of steps performed in video
Weigh out z-Chlorotrityl Calonottittlchhorii chloride resm
Plac resinn ml. frited medium porosity synthesis vosuel
59 Comhine sonicated
from precipitation stop
Cover
refrigernte avernight to maximize precipitation
Collect precipitated material by vacuum filtration
Use tine medium pour size center disk fllter tunnel for filtuation Wash precipitate twice with milliliters cold ether remove residuall organics
"6r"," '62", '63"]
Step Ordering
Qresttin
What the correct sequence of steps performed in the video?
Collect precigitated material by vacumm filtwiionn
flask and refrigerate overnight marimize precipitation
Use fine or medium- pore filter funnel for filtration
Wash precipitate tudie with 5- milliliters cold ether residuaa
&
flask and refrigerate Collect precipttated material by vacuum filtraiinn
maximize
Use fine mediums filter funnel filtration
Wash precipitate tvice with s- millliiters of cold ether to remove residual oopanics
Cover flask and refrigerate overnight maximize precipetation
Use fine medium- filter funmel for filtration
Wash precipitate tvice with so millliteess of cold ether
Collect precipttated material by
filtration
Use fine medium- filter funmel for filtration
Cover and refrigerate overnight macimize precipitation
Collect precipttated material by filtration
precipitate twice with cold ether to remowe residual organics
Answer
Figure 8: Level-2 QA examples, including Step Ordering and Sequence Generation.
19

---

Published as a conference paper at ICLR 2026
Completeness Verification
Ouestion
tollowing step list, step was not performed
A. Cover flack and refrigerate owernight masimise precpitaaiio
& Collect prccipitatdd material by vacuum filtration
C. Use fine medium size center disk filter funnel filtration precipitate
5-10 milliliters
residual organics
Step Prediction
Question
Géven the complete step list of the experiment, please predict the next step that will take place after experisiental steps shown in this video.
Weigh out a Chlorotrityl chloride resnn
Place resin in ço mL tritted medium perasity synthesis vessel
Aftx synthesis vessel variable mechanical shaker
vessel 45 degree angle maximite agitation
i5
59 Combine sonicated mixture with other solution from precipitation stop
60 Cover tlask and retrigerate overnight maximize precipitation
Collect precipitated material by filtration
Use
filtration
63 precipitate twice with 1-
organics
Answer '6""
Figure 9: Level-2 QA examples, including Completeness Verification and Step Prediction. Target steps are clipped in the give video segments.
Experimental Analysis
Queestio
a split root whest experiment where only one root half was spiked, and were later detected in the opposite, unspiked root root and surrounding medium, demonstrating long distance transport from shoots roots and release into the
Answer
["PPO&" "PPOS". "pliloem, "unspiked solution!!
Scientific Discovery
Question
stady that, usinga ..... perfluorinated compounds undergo long. distance transport from shoots the and traverse the rool into the external solution, establishing pallway for redistribution and elimination in wheat.
('split-rool hydroponnc "PFOA and PFOS". "phloen"" 'Casparian strip"
Figure 10: Level-3 QA examples, including Experimental Analysis and Scientific Discovery fill-in- blanks.
20

---

Published as a conference paper at ICLR 2026
E EXPERT VERIFICATION
In this section, we present illustrative examples of the online annotation platform that supports expert verification across all tasks. Experts follow standardized guidelines: watch source videos and related materials, review annotations, and refine them to meet task-specific criteria. For any modifications, they must also provide justifications to ensure transparency and traceability.
Figs. 12 and 13 show representative cases from each level. Experts validate annotations, correct errors, and refine question-anse -nswer pairs ensure accuracy and domain fidelity. Level-3 is distinct in requiring annotators to also consult the corresponding research papers when designing questions. The entire process is iterative: low-quality annotations can be returned for revision until they fully satisfy the benchmark's 's standards.
veee
Figure 11: Expert annotation example of Level-1 task.
21

---

Published as a conference paper at ICLR 2026
1060
jove
Heparin Concentration: 1200 U/ke
Ouddiis
Ve
wash eveles each RT
Figure 12: Expert annotation example of Level-2 task.
22

---

Published as a conference paper at ICLR 2026
Figure 13: Expert annotation example of Level-3 task.
23

---

Published as a conference paper at ICLR 2026
Video Yeutficatiee Optize Count Diaributiinn Randems
Prodiction Optize Crest Dietribution Randons
Anoup lenetry
Buad Acrence Huzber Crartien
Figure 14: Distribution of # options of Completion Verification and Step Prediction.
F EVALUATION DETAILS
In this section, we detail experiment settings, model configurations and inference prompts below.
F.1 EXPERIMENT SETTINGS
For frame selection, we use frames for Level tasks and 32 frames for Level 2 tasks, which approximately correspond
sampling rate of I fps given the average duration of the videos in these tasks. For Level 3 tasks, we adopt either the recommended number of frames or the maximum number of frames that can be accommodated within the model's context window and available GPU memory. Frames are uniformly sampled from the raw videos and resized to 224x224 to ensure fair comparison across models.
For inference, we allocate a maximum of 8192 tokens to each model to ensure that complete answers can be generated in the vast majority of instances. The temperature fixed at 0.1 for all models to reduce randomness in generation.
24

---

Published as a conference paper at ICLR 2026
F.2 CONFIGURATIONS EVALUATED MODELS
The detailed configurations of evaluated MLLMs, including model versions and visual frame inputs, given in Tab.
Details of evaluated MLLMs used in ExpVid. The "# Frames" column represents the default number of input frames in level3 tasks, chosen from {96, 128, 256, 512}. "HF" means Hugging Face inference, "vLLM" indicates vLLM engine, and "API" denotes proprictary API call.
Table 3:
Level3
Organization
Model
Release
Version
Pipeline
Closed-source MLLMs
OpenAl
GPT-5
2025-8
GPT-5
128
Gemini-2.5-FP
2025-5
1e.in1-2.5-22
128
API
Google
2.mini-2..5-Pm
2025-3
Gemini-2.5-Pro
128
API
Authropic
Claude-Sonnet-4
2025-5
Claude-
96
API
ByteDance
Seed1 VI.
2025-3
Sood1 15-VE
256
API
Open-source MILMS
Qwen2.5-V1-778 VL
2025-1
128
VLLMM
Alibaba
Qwen2.5-VL-72B
2025-1
2wen2-5--E--72--Iustuuuu
128
vLLM
Qwen3-VL-235B
2025-9
1Be23-V--23BB-A222
Instructt
128
vLLM
Qwen3- V1.-3388
2025.9
128
VLLLM
VL3-B
2025-4
InternVL3-B8
Intern V1.3.5888
2025.9
InternVi.3.599t
256
HF
Intern VL3,55-388 38B
2025-9
InternVL3.5-302
256
HF
Shanghai Al Lab
Intern VL3-78B
2025-4
InternVL3-78B
HF
Intern-SI-mini
2025-7
Intern-Sl-ming
128
HE
Intern $1
2025-7
Intern-81
128
HF
Keye- Keye-VL-SB-Pree VL-SB-Previeww
2025-6
Keye-VL- Beye-VL-8B-PEee
PB--eeerevee
256
HE
Kwai
Keye- VL--1.5-8B
2025-9
Keye-V1.-1.5-B0
HF
Moonshot
Kimi- VL-A3B -hn--iiiinking
2025-6
Kimi--VL-A3B--iiikinn----
vLLM
Xiaomi
MiMo-VLL-BBBL.
2025-8
MiMo-WW-78-RR-2500
512
VLLMM
ZhipuAI
GLM-4.VV9B-Thinge
2025-7
GLM-4.
1V--9B-Thinking
256
HF
GLM-4.VV
2025-8
GLM-4.5V
256
API
25

---

Published as a conference paper at ICLR 2026
F.3 PROMPT FOR INFERENCE
We provide prompt templates across all tasks and examples below.
Prompt for Level Tasks
Full Prompt
(task instruction] }
{question)
Task Instruction
Solve the multiple choice question based on the video. Provide your final answer a single letter enclosed in [boxed
Question
Materials
Question: Which material appears in this experimental step?
Options:
A: collected pellets
B: agarose beads
C: silica gel packets
D: lyophilized powder
Operation
Question: What is the person doing with the pipette to the cell plate wells?
Options:
A: Removing the medium
B: Pouring fresh medium
C: Injecting PBS solution
D: Mixing the contents
Quantity
Question: How many pellets are gathered?
Options:
A: 10 pellets
B: 8 pellets
C: 12 pellets
D: 15 pellets
Tool
Question: Which tool is being used in this experimental step?
Options:
A: plastic bag
B: ziplock bag
C: desiccator
D: weigh boat
26

---

Published as a conference paper at ICLR 2026
Prompt for Level Tasks
Sequence Generation
Task Instruction
Solve the following question based the video. Provide your final answer list of numbers (comma-separated) enclosed in (boxed(]
Question Based on the full step list, determine the step numbers shown the video.
Full Step List:
Use laryngoscope to expose vocal cords through mouth of 25-30g female Yorkshire pig
2. Spray vocal cords with two puffs of 2% lidocaine topical solution
39. Suture flap skin panel to cervical midline skin incision
40. Close abdominal skin incision
Step Ordering
Task Instruction
Solve the multiple choice question based on the video. Provide your final answer as a single letter enclosed in [
Question What is the correct sequence of steps shown in the video? Options:
A: 1. Thoroughly mix equal proportions of epoxy and hardener
Leave mixture for one hour
B: Place ZIF-8 membrane on 24mm steel disc with 5mm diameter center hole
2. Thoroughly mix equal proportions of epoxy and hardener
C: 1. Thoroughly mix equal proportions of epoxy and hardener Place ZIF-8 membrane on 24mm steel disc with 5mm diameter center hole
D: 1. Thoroughly mix equal proportions of epoxy and hardener Leave mixture for one hour
Completeness Verification
Task Instruction
Solve the multiple choice question based on the video. Provide your final answer single letter enclosed in \boxed(].
Question Given the complete step list, which step was not performed in the video?
Step List:
1. Withdraw 1 milliliter of isoprene solution using syringe
2. Rinse syringe three times with isoprene solution prior to final withdrawal
7. Introduce flow of standard liters per minute of purified air
Options:
A: 1
B: 2 C: D:4
E:5
F:
G:7
Step Prediction
Task Instruction
Solve the following question based on the video. Provide your final answer as single number enclosed in [bxxed
Question Given all steps of the experiment, please predict the next operation that will take place after this video segment.
Full Step List:
1. Cut high purity copper foil into 4x4 em pieces
Draw line 0.5 from one edge of each square foil
44. Calculate permeance in Excel using mass spectrum data after steady state establishment
27

---

Published as a conference paper at ICLR 2026
Prompt for Level Tasks
Full Prompt
(task instruintttttion
(question)
Task Instruction
Solve the following fill-n-thheblank question based on the video. Provide your final answer as list of words phrases (comma-separatel enclosed in {boxed{].
Question
Experimental Analysis
Title: [OPPEhos>ttc))CuJPF:: A General and Broadly Applicable Copper-Based Photoredox
Catalyst
Discipline: Chemistry
Question: In this study, the direct C2 arylation of N-methylpyrrole with 4-indobenzonitrile at
room temperature proceeded using under irradiation to afford the product in yield after
Scientific Discovery
Title:
Novel Object Recognition Test for the Investigation of Learning and Memory in Mice Discipline: Behavior
Question: In the mouse object recognition task, inhibitors of _f_ produced a dose-dependent improvement in memory when administered 30 minutes before training, immediately after training, or 30 minutes before recall, implying that this target modulates
and
28

---

Published as a conference paper at ICLR 2026
F.4 ERROR CASES
We provide an output example of an Step Ordering Question from Gemini-2.5-Flash in both No- Thinking and BrinkinggBudget-23 modes. Since the full outputs lengthy, we present only key excerpts to highlight the distinct answering styles of the two modes.
NoThinking Response (Correct, A):
The video shows precursor vaporization setup experiment. The steps involved are:
1. Withdraw 1 milliliter of iseprene solution using syringe....
2. Rinse syringe three times with isoprene solution pricr...
Let's analyze the options:
Option A: 1, 2, 3, 4, 5, 6, 7. This sequence aligns with the logical progression of the experiment.
The video shows the following sequence of actions:
1. Freparation of the syringe (draving and rinsing the solution)
2. Insertion of the syringe into the syringe pump.
3. Insertion of the syringe needle into the flask.
4. leaating of the flask.
5. Activation the syringe pump.
6. Introduction of the carrier gas.
Comparing this with option A:
1. Withdraw 1 milliliter of iseprene solution using syringe. (Matches)
2. Rinse syringe three times with isoprene solution pricr to final withdrawal. (Matches)
The final answer is |\boxed (A)S....
Thinking Response (Incorrect, D):
In considering the question regarding the correct sequence of steps for the Precursor Vaporization Setup experiment, I've reviewed the provided options and the implicit logic of laboratory procedures for such setups. My thought process has been to deconstruct each optien, evaluate the scientific rationale behind the order of operations, and ultimately efficient workflow.
identity the most logical
initially, recognized that the experiment involves ves vaporizing precursor (isoprene solution) using a heated flask and a syringe pump for controlled delivery, likely in conjunction with carrier gas. This immediately suggested general workšlow: prepare the sample, the apparatus, prepare the reaction environment, introduce the sample, and initiate the process.
up
then systematically analyzed each proposed sequence:
**pption A**: found this option plausible but noted petential issues. Speci fically, preheating the flask after inserting the syringe seemed less efficient, and introcucing the carrier gas the very last step raised concerns about optimal sweeping of the vaporized sample. Considering the logical flow, presents the most coherent sequence: Prepare the sample in the syringe.
1.
2. Place the syringe into the syringe injector.
3. Freheat the flask.
4. Insert the syringe neecle into the flask.
5. Introduce the carrier gas.
6. Turn on the syringe injection.
The final answer is $)boxed(D)$....
29

---

Published as a conference paper at ICLR 2026
G PROMPTS FOR ANNOTATION
In this section, we provide detailed prompts used each annotation task.
G.1
LEVEL-1
Prompts for Level-1 tasks are provided as follows.
G.1.1 MATERIAL EXTRACTION
SYSTEM_PROMPT==
You are an expert in scientific experimental procedure analysis, specializing in extracting **atcriils** from experimental procedure text. Please strictly follow the instructions by users.
USER PROMPT.TEMPLATT =
### Task Objective:
Extract the list of scientific mattriils**** mentioned in the following ASR transcript, preserving
critical states and specifications.
You are given:
An experimental step transcription (ASR caption): semantically accurate.
A visual scene description from a vision-language model (Qwen caption): rough but helps verify visibility of the material.
### Material Definition:
Biological specimens (with preparation state)
Chemicals/reagents (with concentrations/forms))
/olutioos/mixttrrss (when specifically named)
Gases/substrates
### Extraction Rules (Critical):
1. * Prreserve essential descriptors ** that define:
Biological state (e.g.. "anesthetized mouse", "fixed tissue")
Preparation form (e.g., "trimmed hair", "lyophilized powder") Anatomical parts when manipulated (e.g., "mouse's 's head", "renal cortex")
2. Normalization guidelines:
Keep singular/plural as in original context
Remove non-essential modifiers (e.g., "carefully", "gentl"")
Retain:
Mixture states (e.g., "OVA-alum emulsified")
Biological conditions (e.g., "post-mortem brain")
Exclusion criteria:
Instruments/tools (e.g., "shaver". "pipette")
Generic containers (e.g., "tube", "well plate")
Unspecified solutions (e.g., just "solution")
### Output Format:
"materials": ["materiall" "material2",
ASR caption: "(asr_caption)"
Qwen caption: "(qwen_caption)"
30

---

Published as a conference paper at ICLR 2026
G.1.2 MCQ ANNOTATION FOR MATERIAL RECOGNITION
SYSTEM PROMPT =
You are scientific researcher creating multiple-choice questions (MCQs) for material recogni- tion in scientific videos.
Your task is generate plausible distractors for given material based on the experimental context.
USER PWWMPP..TEPPPALTE
You are generating a multiple-choice question (MCQ) for material recognition in scientific experiment videos.
Given:
An experimental step transcription (ASR): "(asr_caption)"
- A target material: [[target_materinn
### Your Task:
Generate #*3 scientifically plausible distractors** (i.e., incorrect but believable options) for the given material.
### Each distractor must meet the following constraints:
1. Do not use distractors that only differ from the target material by quantity concentration.
2. Must be **actual material chemical** used in real laboratory settings.
3. Must be **contextually plausible** in the described procedure it should be reasonable that such a material might appear in this type of experiment.
4. Distractors should fall into *different plausible confusion categories**;
** Visual similarity looks similar in appearance form (e.g., transparent liquids)
Functional similarity used for similar purposes (e.g., washing, dissolving, blocking)
** #Common confusion **; frequently confused due to naming, function, or form
5. Do not invent fake materials** or use vague terms (e.g., "solution", "fluid")..
If the target material includes modifier (e.g.. "PBS buffer", "deionized water"), keep the full original phrase from the ASR as the correct answer.
Output ONLY valid JSON in the following format:
"question": "[question_ "ttpuaationtemmlatee""""
"options":
"A": "<correct answer with proper modifiers>""
"B": "<distractor 1>".
"C". "<distractor 2>"
"D": "<distractor 3
"answer": "A",
materel.material.:: "[targetmaterril)",
"distractor_types"
"B": "<Visu//finctioo///omiissns"""""
"C": "<Visul/ftnctiom///ottsssss
"D": "ccisal//fnectioalloofftssss
Example for "PBS":
- -A: "PBS" (correct)
B: "saline solution" (functional both for cell washing)
C: "Tris buffer" (visual- - similar buffer solutions)
D: "deionized water" (confusion commonly mistaken)
31

---

Published as a conference paper at ICLR 2026
G.1.3 TOOL EXTRACTION
SYSTEM_PROMPT-
You are an expert in scientific experimental procedure analysis, specializing in extracting **tools** ** from experimental procedure text. Please strictly follow the instructions by users.
USER
=
### Task Objective:
Extract the list of scientific **tools** mentioned in the following ASR transcript.
You given:
An experimental step transcription (ASR caption): semantically accurate.
A visual scene description from vision-language model (Qwen caption): rough but helps verify visibility of the tool.
### Tool Definition:
Any instrument, equipment, or container used directly during the experiment (e.g.. pipette, centrifuge, test tube).
### Standardization Rules:
Use lowercase and singular form (c.g., "gloves" + "glove").
2. Remove units or quantity descriptors (e.g.. "1.5 milliliter microcentriftgg tube" "micro- centrifuge tube").
3. Remove generic adjectives modifiers not affecting tool identity (e.g., "sterile", "clean").
Retain essential identifiers (e.g.. "AVB Sepharose column")).
Do hallucinate. Only extract explicitly mentioned tools.
### Output Format:
"tools": ["tooll", "tool2",
ASR caption: "fasr_car "asr_ccaptoo]""
Qwen caption: "(qwen_caption
32

---

Published as a conference paper at ICLR 2026
G.1.4 MCQ ANNOTATION FOR TOOL RECOGNITION
DUSTEM_PROMP" =
You are scientific researcher creating multiple-choic questions (MCQs) for tool recognition
in scientific videos. Your task is to generate 3 plausible distractors for given tool based on the experimental context.
USER PROMPT_TEMPLATE.
You generating multiple-choice question (MCQ) for tool recognition in scientific experi- ment videos.
Given:
ASR: (aasr_caption)
-argett tool: "(target_tool)"
Your task:
Create plausible distractors (wrong options) for the target tool.
### Requirements:
Options must be tools that could reasonably appear in this experimental context.
Distractors should be visually similar, functionally related, or commonly confused tools.
the target tool has modifiers (e.g., "microcentrifuge tube"), use the full phrase. Ensure the target tool name matches the ASR context. ### Output Format:
"question": (qquestion.templatee
"options":
"A": "<correct answer with proper modifiers>""
"B": "<distractor 1>",
"C": "<distractor 2>"".
"D": "<distractor 3>
"answer": "A"
"target_tool" tool": "(target
"distractor types":
"B": "<viss//inntiiim////iissssss
"C": "<Vvsua/ffinttional/cfffuusssss"
"D"; "<cisul//iunctioalloottisiooo
Example for "pipette":
-A: "pipette" (correct)
- B: "syringe" (functional both for liquid transfer)
C: "dropper" (visual similar appearance)
D: "burette" (confusion precise liquid measurement)
33

---

Published as a conference paper at ICLR 2026
G.1.5 QUANTITY RECOGNITION
SYSTEM PIIEEMPP =
You are scientific researcher creating multiple-choice questions (MCQs) for quantity recogni- tion in scientific videos.
USER PROMPT TEMPLATE =
You are generating a multiple-choice question (MCQ) for ** *uuantity recognition** via visual observation.
You given:
An experimental step transcription (ASR caption).
A visual scene description from vision- rision-language model (Qwen caption).
### Task:
Generate exactly ONE quantity-oouused MCQ where the correct answer can only be determined by visually observing the video (e.g., volume, number of containers, temperature, duration).
### Rules:
1. Keep the question minimal and direct, focusing only on the quantity.
The answer must be visually inferable (use Qwen caption to check visibility).
3. Do not rely textual or auditory clues.
### Distractor Guidelines:
Options must be plausible in the context (realistic volumes, times, temperatures, counts).
Keep distractors in the same magnitude range.
Use visually confusable alternatives (e.g., tubes).
Avoid overly fine distinctions (e.g.. 5.0 vs 5.2 mL).
Reflect common visual errors (slight miscounts, occlusion).
### Output Format:
"question": "<Clear, quantity-only question>""
"options":
"A": "<correct answer>".
"B":
"<distractor 1>"
"C":
"<distractor 2>"
"D": "ddistractor 3>""
"answer": "A"
If the ASR caption has quantity-relater info, return: { "question": null
Input:
ASR caption: caption]]
Qwen caption: "(qwen.captie caption)"
34

---

Published as a conference paper at ICLR 2026
G.1.6 OPERATION RECOGNITION (STAGE 1: ALIGNMENT SCORING)
USER PROMPT.TEMPLATT =
You will be given two inputs about the same video segment:
ASR caption (narration of experimental steps)
Qwen caption (vision-langures description of the visual scene)
Your tasks:
1) Decide whether the segment contains experimental operation(s), preferably visible actions (e.g., pipetting, pouring, placing, transferring, cutting, mixing). If no experimental operation is present, or only background talking/intro without hands-on action, set the score to O.
2) If operation(s) present, judge the alignment between ASR and Qwen descriptions, and produce a score from to 5 (higher = better alignment of actionstoolseetiiisssseeeneeee)
Output JSON only with the fields:
"has operation": <true-false>, "visible_actions <true truue-false>> drignment_score" <integer 0-5
Rules:
no operation: set has operation=false, visible action=false, alignment score=).
If operations present: set tree visible_action=tree only if the action is likely visible.
For operations present: alignment score in [1..5]
ASR caption: "(asr "asss_captio]""
Qwen caption: "(qwen_captie caption)"
35

---

Published as a conference paper at ICLR 2026
G.1.7 OPERATION RECOGNITION (STAGE 2: MCQ GENERATION)
USER PROMPT.TEMPLATE =
You are generating multiple-choice question (MCQ) for scientific experiment video under- standing given the ASR caption.
### Task:
Generate exactly ONE action-focuse MCQ. The correct answer must describe specific experimental operation stated the ASR caption. Create 3 distractors that plausible but incorrect variations of the action in the same
tools/materials/setup context.
### Question design rules:
Minimal and direct: focus only the observable action.
2. Visually grounded: the correct answer must be verifiable via video.
Do NOT use audio/textual clues (e.g.. ASR narration). Assume only visual content is available.
### Distractor guidelines:
Options must be plausible actions in the same context.
Keep distractors in the same action/tool category (e.g., if pipetting is correct, distractors can be pouring, injecting, mixing).
Avoid distractors that too ambiguous not visually distinguishable.
Favor common mistakes or visually similar but incorrect operations (wrong hand, placing vs removing).
### Output Format:
"question": "Caction-focused question strictly from ASR>". "options":
"A": "<correct action from ASR>"
"B": "<plausible distractor>",
"C":
"<plausible distractor>",
"D"; "<plausible distractor>""
"answer": "A"
ASR caption: "[ast_caption]"
36

---

Published as a conference paper at ICLR 2026
G.2 LEVEL-2
Prompts for Level-2 tasks are provided as follows.
G.2.1 CLIP SEGMENTATION
SYSTEM PEOMMMTPPPPT=
You scientific video annotation assistant. Your task is to segment scientific experiment video transcript (ASR subtitles) into meaningful procedural clips for multi-step understanding benchmark.
USER
=
The benchmark focuses on medium-length videos containing several consecutive experimental steps. Each clip should:
Include multiple related actions (usually 2+)
Correspond a coherent workflow unit (preparation, execution, wrap-up)
Reflect logical/causal continuity
Be suitable for designing multi-step reasoning questions
Please identify clip boundaries where:
A major shift in experimental phase occurs
The toolset, materials, or purpose changes significantly
A natural grouping of steps can form compact unit
### Output Format:
Return JSON list where each segment has:
"imme: exact timestamp where the segment begins
"end time": exact timestamp where the segment ends
"title": short summary the clip
"description": 1-2 sentences explaining the segment
### Rules:
1. Each segment must be 20-60 seconds long.
2. Start/end times must come directly from ASR (no invented timestamps).
3. Avoid over-segmentation of atomic actions; do not merge unrelated steps.
ASR transcript:
"(cerrptt(( "asrr_caption""
37

---

Published as a conference paper at ICLR 2026
G.2.2 STEP EXTRACTION
SYSTEM_PROMEE
You are an expert in scientific experiment procedure analysis. Your task is to break down com- plex experimental procedures into atomic steps.
USER CCOMPT_TEMPLAN
You are assistant tasked with decomposing scientific experiment procedures into atomic steps. ### Task:
Break down the experimental procedure in the timestamped subtitles into sequence of **atomic steps** Each step should represent a single action and include the corresponding time window. ### Guidelines:
Only use the timestamped subtitles (ignore title/description)..
Each step must be: specific, self-contained, sequential, precise, and timed.
Split compound actions into separate steps.
Use technical language suitable for scientific protocols.
If subtitles are ambiguous, make best effort with available info.
subtitles contain no experimental operation, return * **nul|** ### Output Format:
aatomi_steps":: [
"step :ooppnumerr": 1,
"action": "<concise action description>""
"start tinttttime:: "tstart timtstttttmmttmp"..
"end time": "<end tiindddtmmttmpp>
].
"total_steps": <integer>,
"confidence": "<high medium - low>"
If no operations: return {
null
### Example:
Timestamped Subtitles:
00:15.540 -> 00:19.144 140: Take 200 microliters of
00:19.140 -> 00:20.640: your culture of interest
00:22.590- -> 00:23.940: And just make a spot.
00:45.390 -> 00:46.080: can usually
Expected Output:
[ttomic.steps::
hitep_number": "action": "Take 200 microliters of culture of interest", "start_time"" "00:15.540", "end time": "00:20.640" }.
"step number": 2, "action": "Make sample spots on plate", "start time": "00:22.590", "end_time": "00)51.080") 1.
"totalssteps": 3, "confidence": "high"
Now analyze the given timestamped subtitles and generate atomic steps:
TTill: (title)
Description: [description)
Timestamped Subtitles: (timestamped
38

---

Published as a conference paper at ICLR 2026
G.2.3 SEQUENCE ORDERING MCQ
PESTEM_PROMP =
You are an expert creating multiple-choice questions for scientific experiment step sequencing.
USER PROMPT TEMPLATE =
You are creating a multiple-choice question about the correct sequence of experimental steps.
### Context:
Title: [title}
Description: (description)
### Task:
Given the following correct sequence of atomic steps, create an MCQ with 4 options (A, B, C,
D):
Option A = correct sequence
Options B. D incorrect but plausible sequences
### Correct Sequence: (extps_text]
### Requirements for incorrect options:
1. Maintain scientific plausibility.
2. Keep logical procedural flow (no impossible orders).
Introduce subtle ordering variations (swap/rearrange steps plausibly).
4. Use the same steps only reorder.
### Output Format:
"question": "What is the correct sequence of steps for this experimental procedure?",
"options": {
"A": "1. <correct step 1>2. <correct step 2>3. <correct step 3>..."
"B": "1. <incorrect step 1>2.
<..>"
"C": "1. <incorrect step 1>2, <...>
"D"; "1 <incorrect step 1>2. <...>
}.
"noweet "A"
Generate the MCQ now.
39


Reviews:
[{"review_id": "XyoLABp4Yj", "reviewer": "Reviewer_YjXg", "summary": "The paper introduces ExpVid, a new benchmark designed to systematically evaluate Multimodal Large Language Models (MLLMs) on scientific experiment videos. The dataset consists of peer-reviewed laboratory videos from JoVE (an online journal), annotated with a three-level hierarchy: 1. Fine-grained Perception (e.g., identifying tools, materials, quantities, and actions); 2. Procedural Understanding (e.g., step ordering, sequence prediction, completeness verification); and 3. Scientific Reasoning (e.g., connecting experimental procedures to conclusions). The authors use multiple MLLMs (both open- and closed-source) to evaluate the performance on ExpVid. The results reveal a substantial gap between closed-source models (GPT-5, Gemini-2.5) and open-source counterparts in all tasks.", "questions": "1. Are there other domains where the construction of datasets in the same fashion can be expanded to ensure this work would target a larger audience from the ICLR community?\n\n2. Can you please provide details on any biases that LLMs may have included in the annotation process (if any)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors introduce a dataset for the evaluation of MLLMs for scientific experimental reasoning, a domain previously underserved by benchmarks like MVBench, Video-MMMU, and SCI-VID. The three-level hierarchy (Perception -> Procedure -> Reasoning) mimics the actual cognitive workflow of scientific experimentation and it is interesting. \n2. The annotation pipeline combines automated extraction by LLMs with expert human validation by PhD students. \n3. Empirical evaluation with a large number of LLMs is seen as valuable.", "weaknesses": "1. Although an interesting dataset, using only one source to construct the dataset (JoVE) could introduce biases. It is unclear how the models would perform on other types of data (collected from other sources). Are there other sources that can be used to enhance the dataset? (Other journals with such data?) \n\n2. The reliance of LLMs for annotation could potentially introduce some biases. Did the authors/PhD student annotators carefully look for such biases and ensure the dataset is properly annotated? A more thorough discussion on this would strengthen the annotation process. \n\n3. The focus on wet-lab biology, chemistry, medicine could limit the scope of this paper and may not be of interest to a large audience in the ICLR community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OCARwsLbOH", "reviewer": "Reviewer_Z8zB", "summary": "ExpVid introduces a benchmark for understanding and reasoning over scientific experiment videos, with a three-level hierarchy: (L1) fine-grained perception (materials, tools, quantities, operations), (L2) procedural understanding (ordering, sequence generation, completeness, next-step prediction), and (L3) scientific reasoning (experimental analysis, discovery). The dataset is curated from peer-reviewed research collection videos and paired papers, using a vision-centric, semi-automatic pipeline (LLM-assisted extraction + expert verification). The released scale is 390 videos across 13 disciplines with 7,800 QA items spanning 10 task types. Baselines across 19 MLLMs show frontier closed-source models outperform open-source ones, with the gap widening at higher-order reasoning; e.g., GPT-5 leads L2/L3, and Gemini-2.5-Flash tops L1. Authors report vision is necessary (ablations w/ vs. w/o frames) and “thinking” can sometimes hurt if it drifts from visuals. Limitations include a focus on wet-lab domains and L3 evaluation via an LLM judge rather than human-graded answers.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The task design is clear and hierarchical (perception $\\rightarrow$ procedure $\\rightarrow$ reasoning), aligning with practical laboratory steps and utilizing expert-validated task templates and distractors.\n2. The paper details a transparent curation process, utilizing JoVE videos and expert verification, to create a substantial and diverse benchmark (7,800 QA pairs across 13 disciplines).\n3. The paper provides a comprehensive empirical evaluation by testing 19 MLLMs against human baselines using task-appropriate metrics for comparison.", "weaknesses": "1. The Level-3 (Scientific Reasoning) tasks use a \"fill-in-the-blank\" format (Sec 3.3). This format might be simpler or more susceptible to text-matching than open-ended questions. Please discuss the rationale for this choice and how it was designed to test deep reasoning beyond simple keyword extraction.\n2. The Thinking effect observation is insightful but based on one model configuration and qualitative examples. Please Run a 2×2 across ≥3 model families with or w/o think within matched budgets to verify the claim.\n3. L2 shows high Step Ordering yet low Step Prediction/Completeness, but diagnostics aren’t tied to controllable factors (occlusion, clip length, distractor difficulty). Please provide controlled L2 suites that vary occlusion, temporal gap, and distractor closeness, and report item-response curves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dYj1kLg8X3", "reviewer": "Reviewer_1vC1", "summary": "This paper introduces ExpVid, a new and significant benchmark for evaluating Multimodal MLLMs on scientific experiment video understanding. Its primary contribution lies in addressing a clear gap, as existing benchmarks fail to capture the fine-grained, long-horizon nature of \"wet-lab\" procedures. Its core contribution is a three-level task hierarchy (Perception, Procedure, Reasoning) which allows for a detailed diagnosis of model capabilities, from basic visual recognition up to complex, long-horizon inference and is built from peer-reviewed JoVE videos paired with their scientific publications. This unique data pairing enables high-level reasoning tasks absent in other datasets. Crucially, the authors employ a rigorous \"vision-centric\" annotation pipeline with PhD-level expert validation, ensuring that tasks require genuine visual grounding and cannot be \"cheated\" by models relying on prior data. Finally, the paper provides actionable findings by evaluating 19 SOTA MLLMs, clearly demonstrating their current limitations and offering a valuable roadmap for future research.", "questions": "- The Level-3 tasks are \"outcome-based\" (fill-in-the-blank), which checks if the final answer is correct but not if the reasoning process was. How can we be sure models are not arriving at correct answers through flawed logic? Was a \"process-based\" (e.g., chain-of-thought) evaluation considered, and why was it excluded?\n- The paper makes a point that \"Thinking\" mode can hurt performance by causing models to \"drift\" from visual evidence. This is a critical but under explained finding. Could the authors elaborate on how prevalent this was? Does this suggest a fundamental flaw in how current models balance their internal knowledge against real-time visual evidence?\n- The paper states ASR transcripts were collected from JoVE. Is there any data on the quality (e.g., Word Error Rate) of these transcripts? It is ambiguous whether some model failures, especially on Level 2 and 3, could be attributed to errors in the provided ASR text rather than the model's own reasoning.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The three-level task hierarchy is a major contribution. This structure is exceptionally well-conceived, as it mirrors the real-world scientific process—from basic observation of tools to high-level reasoning about an experiment's conclusions. This hierarchy allows for a much more granular diagnosis of model capabilities than a single, monolithic task, enabling researchers to pinpoint where models are failing.\n- The choice of data source (peer-reviewed JoVE videos) is excellent. By pairing videos with their corresponding peer-reviewed publications, the authors ensure a high degree of scientific rigor.\n- The \"vision-centric\" annotation pipeline is a critical methodological strength. The authors explicitly designed the tasks to require visual grounding, using semantically and visually plausible distractors. By validating these tasks with multi-disciplinary, PhD-level experts, they have addressed the well-known \"language-prior shortcut\" problem (where models guess the answer from text without processing the video).\n- The paper provides more than just a dataset; it delivers a strong empirical study by benchmarking 19 leading MLLMs. The findings are clear and valuable: all models struggle with fine details and long-horizon reasoning, and a significant performance gap exists between proprietary and open-source models.", "weaknesses": "- The benchmark is built on 390 high-quality videos. While the annotation depth (7,800 QA pairs) is impressive, the total number of unique experimental videos is relatively small compared to other large-scale video benchmarks.\n- The data is sourced from JoVE. This is a strength for rigor but a potential weakness for real-world applicability. JoVE videos are highly produced, professionally narrated, and edited for clarity. They do not represent the \"messy\" reality of lab work, which often involves poor lighting, occlusions, camera-phone footage, verbal mistakes, and actual failed steps. Models trained for this benchmark may not generalize well to the real lab videos.\n- The Level-3 (Scientific Reasoning) tasks are evaluated using a \"fill-in-the-blank\" format. This Outcome-Based method effectively tests the outcome of a model's reasoning but fails to evaluate the process. As the authors acknowledge, it doesn't illuminate the chain-of-thought. A model could arrive at the correct answer through flawed logic, or fail for a simple perceptual mistake. This is a missed opportunity for a benchmark on \"reasoning.\"\n- The paper uses non-expert undergraduates as a human baseline. This baseline is difficult to interpret. One would expect non-experts to fail at Level-3 (Scientific Reasoning), making their (unreported) score on this task less informative. A more valuable, albeit much more difficult, baseline would have been to benchmark the performance of PhD-level experts to understand the true gap between models and expert-level scientific comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UsoJdjCnEA", "reviewer": "Reviewer_stVf", "summary": "The paper introduces ExpVid, a new benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on scientific experiment videos, focusing on wet-lab procedures across 13 disciplines. ExpVid is structured around a three-level task hierarchy: (1) Fine-grained Perception, (2) Procedural Understanding, and (3) Scientific Reasoning. The benchmark includes 390 curated experiment videos with detailed annotations from peer-reviewed publications. The study evaluates 19 MLLMs, revealing their strengths in coarse-grained recognition but highlighting limitations in fine-grained tasks and scientific reasoning.", "questions": "1. **Clarification on Input Format:**\n\n   * Could you clarify the exact input format used in ExpVid? Specifically, are the models provided with full videos, or are the videos split into individual frames or intervals of frames? How does this format affect model performance, and have you considered experimenting with different formats to evaluate potential performance variations?\n\n2. **Expert Annotation Workload:**\n\n   * Given the wide range of scientific disciplines covered in ExpVid, could you provide more details on the number of PhD-level expert annotators involved in the project? How were the tasks distributed across experts, and how much time did they spend on annotation? This information would help understand the scalability of the annotation process and any potential limitations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* **Multilevel Evaluation Capacity:** ExpVid introduces a three-tier task hierarchy: fine-grained perception (identifying tools, materials, and actions), procedural understanding (step order and completeness), and scientific reasoning (linking experiments to conclusions). This ensures comprehensive assessment of MLLMs' capabilities across varying complexity levels.\n\n* **Scientific and Realistic Data:** The videos in ExpVid are sourced from the peer-reviewed JoVE journal, ensuring scientific accuracy and reflecting actual experimental processes. The task design closely aligns with real-world experimental workflows, avoiding overly theoretical tests.\n\n* **Vision-Centric Annotation with Expert Validation:** ExpVid uses a vision-based annotation pipeline combined with expert review, minimizing reliance on text knowledge and enhancing the reliability and diversity of tasks. This process ensures accurate and contextually appropriate task creation.", "weaknesses": "1. **Input Format Ambiguity:** The paper does not clearly specify the input format of the experiment videos used in the evaluation. It is important to clarify whether the inputs are full videos, images extracted from videos at intervals, or something else. This detail could significantly impact the model's performance, as different input formats may require different processing approaches. Clearer descriptions of the input format would enhance reproducibility and understanding of the benchmark.\n\n2. **Expert Annotation Scale and Feasibility:** While the paper highlights the use of PhD-level expert annotators, the number of experts and their workload distribution across various disciplines are not provided. Given the breadth of scientific fields covered by ExpVid, it would be useful to understand how many experts were involved, how much time they spent, and how the workload was managed across such a diverse set of tasks. This information would help assess the scalability and practicality of the annotation process, which could be a potential bottleneck for future applications of this benchmark.\n\n3. **Typographical Errors:** A minor but notable typographical error occurs on line 282, where \"e.g.\" is incorrectly formatted as \"e.g.\" without a following comma. Ensuring that such errors are corrected would improve the overall presentation of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XA6CxjQxLC", "reviewer": "Reviewer_QvUU", "summary": "This paper evaluates the extent to which multimodal large language models (MLLMs) (which have the capacity to analyze videos) can be used for important tasks in wetlab settings. This includes identifying different aspects of experimental tasks from video, understanding sequences of tasks that make up a particular procedure, and drawing conclusions from a sequence of scientific procedures. To this end, the paper introduces a new dataset ExpVid, which aims to capture some of the capabilities required for a video-processing MLLM to be useful in a wetlab setting. The paper breaks these capabilities up into a hierarchy: finegrained perception, procedural understanding, and scientific reasoning. ExpVid contains questions aimed at each of these. After describing how the dataset was constructed, automatically annotated, and verified, it provides the results of evaluating both closed and open models. Broadly, it is found that closed models like GPT5 and Gemini have substantially better performance. The paper ends with some further light analysis of the results.", "questions": "- The paper says that an anonymized version of the dataset would be released, but the reviewer could not find any links to this?\n- One of the tasks is ‘quantity recognition’. This reviewer has trouble identifying numbers appearing on devices in a video. How was it verified that this task was feasible? Examples of what this looks like would be very helpful. \n- Line 463: How was the ablation of frames performed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Clarity:** In general, the paper is well-written (but see typos in the Nitpicks section below and the issue of missing examples). It is clear that a lot of work was put into the figures, which are clean and communicate well the points of the paper. The reviewer believes that most participants at ICLR will be able to read and appreciate the results while on the other hand, a good amount of detail on the dataset generation process (usual to experts) was included.\n\n**Problem importance:** The use of MLLMs (not restricted to the video models in this paper) in science is a growing area with several notable start-ups focused on this appearing in the US just this year. Getting a better understanding of the real capabilities of these models in this space is thus highly relevant to current developments in the field. In this sense, benchmarks such as ExpVid, are a welcome contribution.", "weaknesses": "**Without access to the videos, it is hard for a reader to gauge the difficulty of these tasks:** Though the reproducibility statement says that an anonymized copy of the dataset would be made available, the reviewer did not see this (please correct me if I just missed it). Without this, it is hard to get a good sense of the actual content of the dataset and therefore challenging to know how to interpret the results. The reviewer would also recommend adding more examples of questions from the dataset (as much as this is possible for a text and image-based paper) in the body of the paper itself. One way to do this would be to add example questions corresponding to each of the categories introduced in Section 3.2.\n\n**The paper would have benefited from deeper analysis of the results:** It is challenging to include all the details in a dataset paper that has a page limit of only 9 pages. Especially when dataset construction was complicated with many different steps (as with this paper). However, thinking about what is most valuable to the average reader, this reviewer believes there is a strong argument for including more analysis of the results, possibly at the expense of moving some details of dataset construction to the appendix. In particular, the reviewer was curious about current limitations of all models in the tasks that the paper captures. What are limitations that the model-building part of the community needs to address? Where are we succeeding?\n\n**Nitpicks**\n- Line 121: “…provide their corresponding analysis…” $\\mapsto$ “…provide an analysis of our results…”?\n- Line 158: “…while most of it in computation and physics are…” $\\mapsto$ “…while most computational examples, or examples from physics…”?\n- Line 195: “Whether records an entire…” missing ‘it’.\n- Line 261: The term ‘MCQ’ is used before it is defined.\n- Line 264: “Identify the appeared tools from the scene…” should be “tools that appear” or something like that.\n- Line 323: “Encouraging multi-blank settings to probing several key points within one question.” What does this mean?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)