Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents
Authors: Fan Liu
Xiaozhao Zeng
Hao Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Data-driven Scientific Discovery
LLM Agent
Primary area: datasets and benchmarks

Abstract: Recent advances in large language models (LLMs) have enabled agents that automate scientific discovery by interpreting data, generating analysis pipelines, and executing them with computational tools. However, existing benchmarks remain largely limited to unimodal datasets and slice-level tasks, overlooking the fact that real discovery requires multimodal integration, modeling, and hypothesis-driven reasoning. To address this gap, we introduce MoSciBench, the first benchmark for multimodal scientific discovery, constructed from peer-reviewed studies through a principled four-stage pipeline. MoSciBench spans six scientific domains, seven data modalities, and five categories of discovery questions, yielding 88 individual, end-to-end, data-driven tasks. Each task is designed as a cross-modal hypothesis verification workflow, requiring agents to align and integrate heterogeneous datasets before modeling and reasoning. We further evaluate four representative agent frameworks across multiple LLM families. Results show that multimodal discovery is substantially harder than unimodal tasks: even the strongest agents achieve only 48.94\% accuracy, with over 60\% of failures due to cross-modal alignment. Lightweight workflow scaffolding consistently improves performance, reducing alignment errors by 5–10\% and raising accuracy by 5.7\% on average. Our benchmark and evaluation framework thus establish a rigorous testbed for advancing LLM agents toward realistic, multimodal scientific discovery.

Published as a conference paper at ICLR 2026
TOWARDS MULTIMODAL DATA-DRIVEN SCIENTIFIC DISCOVERY POWERED BY LLM AGENTS
Fan Liu', Xiaozhao Zeng', Hao Liu'+ Hong Kong University of Science and Technology (GuangZhou) {fliu236, xzeng638 @hkust pazust-gz... edu..dd..o en 1 liuh@ust.he
ABSTRACT
Recent advances in large language models (LLMs) have enabled agents that auto- mate scientific discovery by interpreting data, generating analysis pipelines, and executing them with computational tools. However, existing benchmarks remain largely limited to unimodal datasets and slice-level tasks, overlooking the fact that real discovery requires multimodal integration, modeling, and hypothesis-drie driven reasoning. To address this gap, we introduce MoSciBench, the first benchmark for multimodal scientific discovery constructed from peer-reviewed studies through principled four-stage pipeline. MoSciBench spans six scientific domains, seven data modalities, and five categories of discovery questions, yiclding 88 individual, end-to-end, data-driven tasks. Each task is designed as cross-modal hypothe- sis verification workflow, requiring agents to align and integrate heterogeneous datasets before modeling and reasoning. We further evaluate four representative agent frameworks across multiple LLM families, Results show that multimodal discovery is substantially harder than unimodal tasks: even the strongest agents achieve only 48.94% accuracy, with over 60% of failures due to cross- modal alignment. Lightweight workflow scaffolding consistently improves performance, reducing alignment errors by 5-10% and raising accuracy by 5.7% on average. Our benchmark and evaluation framework thus establish a rigorous testbed for advane- ing LLM agents toward realistic, multimodal scientific discovery. Our code and data are available at ht htsps: /gihtub. com/ com/usail /us/iusill Motisst/MMSiiBeccc.
INTRODUCTION
Scientific discovery is increasingly data-driven, requiring integration of multimodal data (e.g., satellite imagery, climate time series, and tabular measurements), building models to uncover patterns (e.g.. predicting extreme climate events identifying molecular interactions), and validating hypotheses through iterative analysis Li et al. (2025). Traditionally, constructing such end-to-end workflows, from data preparation to model validation, has been manual and expertise-intensive limiting scal- ability Zheng et al. (2025a;b). Recent advances in LLMs suggest a new paradigm: agents that can interpret diverse data types, automatically generate analysis pipelines, and execute them with scientific tools Guo et al. (2024); Liu et al. (2025c); Lai et al. (2025); Yuan et al. (2026). Realizing this vision, however, systematic evaluation is needed on realistic multimodal scientific tasks.
Existing benchmarks (e.g., ScienceAgent Chen et al. (2024b), DiscoveryBench Majumder et al. (2024)) have advanced L.Lee discovery by formalizing workflows Zhang al. (2025); Tang et al. (2023); Lu et al. (2024) (e.g., dataset preparation, analysis. model design, and validation). In these benchmarks, however, each task is tied single type of dataset, for instance, tabular records or single time series, so agents are only evaluated within isolated modalities Gu et al. (2024). As a result, they remain restricted to unimodal data (e.g., image, time series, or tabular formats). In addition, many tasks defined the level of individual points or slices, where agents handle small fragments, lacking the realism of repository-Ievel discovery Tian et al. (2024). By contrast, real scientific discovery is inherently multimodal, requiring agents access full repositories, integrate heterogeneous files, and reason across them to generate insights, as illustrated in Figure 1. For
"Correspondence Hao Liu.

---

Published as a conference paper at ICLR 2026
instance, climate studies combine satellite imagery with spatiotemporal metadata Liu & Yao (2024), and health research links physiological signals with environmental measures Anders et al. (2024). Capturing this complexity in benchmarks is challenging Zheng (2025), as it requires evaluating agents on cross-modal alignment, modeling, and reasoning capabilities that are essential for practical scientific discovery but largely absent from current benchmarks.
Tank Instruction and Detnser
Instruction Datasets
Disamina peefenese
bntweed
Segment
Repow lavel
Modelling
Reasoning
Alignment M0 1wW.. 210,
Modelling 1
Reasoming
Selact daex "TEMULE 1E
2001 21.10 21.1022........ 22... 6.57..
Comper prin pratip vartill
Cefine
Draad complestt
Accass sasocarden diacki and imang?
Inargres
(a) Existing Benchmark
MoSclBench
Figure 1: Previous benchmarks vs. MoSciBench (ours). Left: Existing benchmarks focus unimodal. small- scale tasks (e.g., single tables or short sequences), offering only fragmented evaluations. Right: MoSciBench supports end-to aud-to-end multimodal discovery by letting agents access full repositories, integrate heterogeneous data, generate and run code, and reason over results to verify scientific hypothesis.
In this work, we introduce MoSciBench, the first benchmark for multimodal data-driven scientific discovery. To ensure realism, we construct tasks from peer-reviewed studies through principled four-stage pipeline: (1) raw data extraction from published repositories, (2) multimodal process- ing and alignment clean, standardize, and integrate heterogeneous datasets, (3) task instruction formulation and annotation to encode discovery objectives in executable and verifiable forms, and (4) human verification and quality control to guarantee consistency and reliability. Each task is designed around scientific discovery goal that requires agents to perform cross cross-modal alignment, modeling, and reasoning, rather than isolated slice-level predictions. For example, a climate task integrates satellite imagery with numerical storm tracks to assess cyclone intensity, while health task combines physiological and environmental signals to identify risk factors for cardiac stress. Overall, MoSciBench covers six scientific domains (climate science, biomedical engineering, cheminformat- ics, health psychology, population genomics, and earth science), five categories of discovery questions (descriptive analysis, correlation, causal inference, prediction, and pattern discovery), and seven data modalities (multi-sensor time series, tabular data, satellite imagery, mass spectra, molecular structures, genotype matrices, and HDF matrices), yielding 88 individual tasks in total.
To systematically evaluate LLM agents on scientific discovery, we present MoSciBench and a comprehensive evaluation framework. Our contributions are threefold:
Benchmark. We introduce MoSciBench, the first benchmark for multimodal scientific discovery, spanning six domains, seven modalities, and 88 individual tasks. Built from peer-reviewed studies through principled pipeline (data acquisition, multimodal processing, task annotation, and multi- pass verification), MoSciBench explicitly targets multimodal, repository-level discovery, making tasks substantially more complex and realistic than prior unimodal benchmarks.
Task formalization. Each task is defined as cross-modal hypothesis verification workflow, where agents must load, preprocess, align, and integrate heterogeneous datasets before modeling and reasoning. Tasks cover five categories central to discovery, descriptive analysis, correlation testing, causal inference, predictive modeling, and pattern discovery. explicitly enforcing multimodal alignment (c.g., linking imagery with time series, or genotype matrices with phenotypes).
Evaluation. We systematically evaluate four representative agent frameworks, combined with both open- and closed-source L.M families, on all 88 tasks. Results reveal three findings: (i) multimodal discovery is significantly harder than unimodal tasks, with even the strongest agents achieving only 48.4 % accuracy; (ii) cross-modal alignment is the dominant bottleneck, accounting for over 60% of crrors; and (ii) lightweight workflow scaffolding consistently boosts performance, reducing alignment errors by 5-10% and raising accuracy by 5.7% on average.
2

---

Published as a conference paper at ICLR 2026
2 MOSCIBENCH CONSTRUCTION
In this section, we introduce MoSciBench, benchmark designed to evaluate LLM agents on multimodal data-driven scientific discovery tasks. These tasks require integrating information from multiple modalities through multimodal data exploration, scientific computation, and reasoning with LLM agents, ultimately aiming to validate scientific hypotheses.
2.1 PROBLEM FORMULATION
MoSciBench evaluates L.M agents multimodal data-driven discovery tasks, each framed an end- to-end workflow requiring cross-modal alignment, scientific modeling, and hypothesis verification. Each task is instantiated with three components: (i) task instruction derived from peer-reviewed study, specifying the scientific background and hypothesis to be tested; (ii) one or more multimodal datasets (e.g.. imagery, time series, tabular records, molecular structures) providing the evidence base; and (ii)) an evaluation protocol that checks whether the agent's 's output is consistent with the gold-standard hypothesis. solve a task, the agent must autonomously align and fuse heterogeneous data sources, build models, perform computations, and reason over the results to test the hypothesis.
Task Instructions. Each task is framed as a scientific question with three elements: the background (e.g., s. the role of precipitation analysis in climatology), the hypothesis to be verified (c.g., identifying the region with the highest average precipitation during 2021-2023), and the expected answer format, shown in Table 1. The answer format provides the key anchor for evaluation, specifying how the hypothesis should be expressed and verified, for instance, categorical outputs (e.g., true/false, class labels), numerical values (e.g., averages, coefficients), short strings, structural patterns. Instructions are concise and open-ended to encourage agents to autonomously decide on exploration, preprocessing, analysis, or modeling steps. Some tasks include optional domain knowledge (e.g.. definitions, formulas, methodological hints) to reduce ambiguity without prescribing solutions.
Multimodal Datasets. Tasks are grounded in datasets spanning seven modalities: (I) time series from multi-sensor streams (e.g., physiological or climate records), (2) tabular data (e.g., survey results or experimental measurements), (3) satellite imagery (e.g., remote sensing products), (4) mass spectra (e.g., metabolomics or proteomics assays), (5) molecular structures (e.g., chemical compounds), (6) genotype matrices (e-g.. population genomics data), and (7) HDF matrices (e.g.. high-dimensional simulation outputs). Each dataset is released in structured directory with previews to expose available variables and formats. Because the dataset multimodal, agents must not only load and preprocess data but also align heterogeneous modalities, for example, linking satellite imagery with spatiotemporal metadata or integrating physiological signals with environmental variables, before conducting scientific analysis.
Evaluation. MoSciBench adopts a eupothesis-centera evaluation: an agent is judged by whether its output correctly verifies the hypothesis. The ground-truth hypotheses and answers are derived from peer-reviewed publications, ensuring objectivity and scientific validity. Each task specifies an expected answer format, which defines how correctness is assessed. Performance is measured by exact match accuracy: a prediction is correct only if it exactly matches the reference answer. For categorical outputs (e.g., strings, integers, class labels), exact identity required. For numerical values, lists, or coordinates, task-specifie tolerances are applied (e.g., numeric precision or ordering) to ensure fairness. This setup makes evaluation automatic, objective, and reproducible, Results are aggregated across tasks and reported as overall accuracy.
2.2 DATA COLLECTION AND TASK ANNOTATION
MoSciBench is constructed directly from peer-reviewed scientific publications spanning six domains (c.g., climate science, biomedical engineering, and cheminformaties) Each benchmark instance follows a principled four-stage pipeline: (i) raw data extraction, (ii) multimodal processing and alignment, (iii) task instruction formulation and annotation, and (iv) human verification and quality control. The overall pipeline construction of MoSciBench is illustrated in Figure 2.
Raw Data Extraction. Papers releasing datasets under permissive licenses and posing explict data- driven scientific questions suitable for hypothesis verification are first selected. From these sources, multimodal datasets, such imagery, time series, and tabular measurements, extracted, along
3

---

Published as a conference paper at ICLR 2026
Table 1: Representative examples of the five task categories in MOSCIBENCH. Each example highlights distinct reasoning type required multimodal scientific discovery. Calegory
Representative Task Instruction
Success Criteria (Example Output from Benchmark)
Descriptive Analysis
Compute the average precipitation in 2021-2023
Output: (3, -78)
across all stations and identify the wettest region.
Corelational Study
Test the time series correlation between air tem-
Ouiput: p-value 0.174 74 (not sig- nificant)
perature and shortwave radiation 20021-2023)
Causal Inference
Assess whether reduction in shortwave radia- tion leads decreased precipitation.
Output: answer: (false)
Predictive Modeling
Predict future daily temperature using the past days data.
Output:
Best RMSE 3. 8208
(Ridge Regression)
Pattern Discovery
Determine the global trend in heatwave-affected areas during 2021-200 10233
Output: Tzend upward
Step Raw Data Extraction
MAI
HDF
Rowe
Umeseries
Mullmodia
Retrieval
Datasets
Sclendili
Research
Boo
Papers
Healh Paych.
Biomediccl Eng.
Earth Sol.
Cheminfommates
Genomics
Sci
Processing Alignment
L
Feature Filtering
Multi-source Consolidation
Format Standardization
ID Matching
Extected
wend Datasets
Standantoed
Mutimodal
Datasets
Datesets
Dateeets
rellet 319. 63W
e.g-
318_58W 53W
timeseries; pres, precitation, temp, reihumidty. spechumiityy
tabular: gild range, country. moan elevation, vegetation, climate type
Step Instruction Formulation Annotation
Step 4: Human Verification and Quality Control
Muttmodal Datssels
Task
JSON
Instrutioos
Extraction
Annotation
JSON
Review
Research
Background
Tosk Instruction
Queerions
Hypotheeis
End-tb-end
Ground Truth
Executable
Stripts
Final Datasets
Figure 2: Overview of pipeline construction MoSciBench.
with dataset provenance, licensing information, and concise summaries the associated scientific questions. Dataset descriptors, including variable names and spatial and temporal coverage, are registered in a metadata sheet to support downstream processing. The details of the data collection source and license can be seen in Appendix A.I.
Multimodal Data Processing and Alignment. The processing pipeline begins with feature filter- ing to remove missing anomalous values and extract a consistent subset of samples from raw multimodal datasets. Next, multi-source consolidation integrates heterogeneous inputs, which are then standardized to ensure comparability by harmonizing units, timestamps, and spatial references. Finally, multimodal alignment is achieved through shared indices: for example, linking individual attributes with physiological time series via subject IDs. aligning satellite imagery with environ- mental variables using geographic grids. This step yields aligned datasets, enabling the verification of scientific hypotheses through data-driven downstream tasks.
4

---

Published as a conference paper at ICLR 2026
Task Instruction Formulation and Annotation. Each research question is translated into concise task instruction that preserves the original scientific intent while avoiding overly prescriptive steps. Tasks are paired with verifiable hypotheses, gold answer specifications, and explicit answer formats (e.g., slot-fiiin,,, true/false, categorical label). Minimal domain knowledge snippets, such as defini- tions, formulas, methodological pointers, are added where necessary clarify terminology without revealing solutions. Further details of the instructions' composition are provided in Appendix A.I.
Human Verification and Quality Control. To ensure task quality, each task instance MoSciBench undergoes multi-pass verification. Verification relies solely on released datasets, ensuring hypotheses can be tested without externl resources and multimodal alignments remain correct. In addition to manual checks, annotators implement end-to-end executable scripts that reproduce workflows and automatically check consistency with gold hypotheses, for example validating numerical results within tolerance, checking correlations causal directions, and assessing predictive performance. Tasks which human verification conflicted with the original gold-standard hypotheses were filtered out, ensuring that the final benchmark maintains perfect consistency between verified annotations and ground-truth hypotheses (100% agreement).
2.3 STATISTICS INFORMATION
Overall Coverage. MoSciBench is designed around five fundamental categories of data-driven scientific dis- covery questions, with a total of 88 instantiated subtasks. To highlight the breadth of coverage, we summarize do- mains, modalities, and task counts Table 2. These subtasks span six major scientific domains, climate sci- ence, biomedical engineering. cheminformatics, health psychology, population genomics, and earth science and incorporate seven complementary data modalities, in- cluding multi-sensor time series, tabular data, satellite imagery, molecular structures, mass spectra, genotype matrices, and HDF matrices.
Table Summary of domains, modalities, and task numbers in MoSciBench. Each domain contains multiple modalities, reflect- ing the diversity of scientific problems in the benchmark.
Themain
Midalitee
Tisk Num.
Climate Science
HDF Iobular, Timeseries
Blemedical Timeseries Test
Chenintormnties
Moss
smactares
Health Psych.
Timesenes, Tahilor
Pop. Gemomies
Rahelor
Earth Sct
Emyz. Talule Timcario
Task Categories. The five categories in MoSciBench capture the major reasoning needs of data-driven science: (1) descriptive analysis (e.g., summary statistics), (2) cor- relational studies (e.g., correlation tests), (3) causal in- ference (e.g., causal relationship analysis), (4) predictive modeling (e.g., regression or classification). and (5) pat- tern discovery (e.g., clustering factor analysis), as il- lustrated in Figure 3. Each of the 88 tasks is grounded in one of these categories. providing relatively balanced distribution across reasoning types. The six domains are evenly integrated, ensuring broad coverage of scientific modalities such as time series, tabular data, molecular structures, and remote sensing imagery. The scale of 88 tasks is deliberately chosen to balance breadth and feasi- bility: each task is framed as an end-to-end, repository- level workflow, where agents must independently perform data-driven computation and reasoning. Given that even a single predictive modeling task require hours to plete, this design ensures the benchmark remains both challenging and practically executable.
Descriptive(25)
Correlational(18)
statisinn
mady
leumazz imgertance audvis
leveaction
effict
aadycc
poych
Discovery(14)
Pattern
Clamite Sei
las
Each S.t.
Causal Inference(11)
duaaing
MoSciBench
caaa
Choniskemaia
liss srics anboo
S
oateutce
scnvel ouseaaite
Prediotive(20)
claiiiatioon
sograsse
malkcuaa charticaton
Figure 3: Distribution of task categories and domains in MoSciBench.
3 EXPERIMENTS
In this section, conduct experiments to address the following research questions: (RQ1): How well do current LLM agents perform end-to-end multimodal data-driven discovery on MoSciBench? (RQ2): What are the main error sources in multimodal scientific discovery, and how can LLM agents be enhanced to address them? (RQ 3): What factors influence the ieftormanne-efficienee trade-offs of L.M agents in multimodal scientific discovery?
5

---

Published as a conference paper at ICLR 2026
3.1 DISCOVERY AGENT
LLMs and Setup. We evaluate both open and closed-source models, including Owen 3wen3-30B-A33
DeepSeek- V3. 1, gpt- 5-mi-,, and 04 o4-mini. All models are under unified configuration with temperature set to 0.0 and zero-shot prompting via API. To prevent excessive computation, the maximum code execution time of each individual task is limited to 1 hour, after which the process is automatically terminated. We report the results of additional model experiments in Appendix A.2.2.
Agent Frameworks. Since there are currently no multimodal discovery agents, we follow widely used single-domain discovery agents Majumder et al. (2024); Chen et al. (2024b) and adapt them to the multimodal scientific discovery setting: (1) NoDataGuess: A naive baseline without data-driven methods. It only provides task descriptions and relies entirely on the LLM's internal memory and reasoning ability. (2) ReAct Yao et al. (2023): Alternates between reasoning steps and code execution in an iterative loop to refine hypotheses. (3) Data Voyager: Employs a modular pipeline with planner, code generator, analysis, and critic components to orchestrate discovery. (4) Reflexion (Oracle): Extends CodeGen with oracle feedback and iterative retries (up to three times) for self-improvement. (5) SelfDebug: A program-executin
agent that iteratively debugs itself by inspecting execution traces, detecting failure symptoms, and rewriting code based internal error hypotheses. (6) RAG- ReAct: A drrmeneed variant of ReAct that supplements the agent's reasoning trajectory with external domain knowledge. Extended baseline descriptions and more detailed analysis of experimental patterns across scientific domains are provided in Appendix A.2.4.
Metric. We evaluate agent performance using three complementary metrics. (1) Accuracy. (Acc) A prediction is considered correct only when it exactly matches the reference answer. This strict criterion applies to all categorical outputs, such as numbers, strings, lists, and coordinates. (2) Code Execution Success Rate (Exec). Measures whether the agent-generane
code executes without errors. (3) Modeling Rationality (MR). A 1-5 LLM-as-judge score assessing the scientific soundness of the workflow, including variable selection, model design, and analytical reasoning. In practice, we use Not--o--a -inmnii as the judge model.
3.2 MAIN EXPERIMENTS (RQ1)
Table 3 summarizes the performance of four LLM agents across six domains and 88 tasks. Our observations (Obs.) are as follows: Obs. O LLM agents perform poorly on multimodal data- driven discovery. Overall accuracy remains modest across all settings, rarely exceeding 50%. Even the strongest configuration, o4 -mim n with ReAct (48.4%) and Reflexion (45.8%), fails to achieve reliable performance. Accuracy is particularly low for smaller models such as Qwen3-30B-B3 (best 23.3%) and gpt --minn (best 17.4%), underscoring the fundamental difficulty of multimodal reasoning and the limitations of current agents in handling multimodal data at scale. Obs. 2 Data- driven approaches are indispensable. The non datadatad baseline (NODATAGUESS) consistently collapses to near-zero accuracy: 0.0% for Qwen3-30B-A3 and 2.6% for DeepSeek-V3. 1, and only 10.5% for o4-miní. By contrast, data- frameworks achieve substantially higher scores, with improvements of 20-40 % across six domains. For example, DeepSeek . 1 with ReAct reaches 36.5% and o4 04mmnn with Reflexion 45.8%. These results confirm that pure reasoning without access to underlying data is ineffective, while explicit data grounding is critical for meaningful discovery. Obs. Stronger base models yield stronger agents. Performance scales directly with the underlying LLM's capability. The strongest model, o4 in achieves the best overall averages (48.9% with ReAct, 46.6% with Reflexion), while DeepSeek- V.. 1 delivers mid-tier performance (36.5% with ReAct), and gpt-5 55 5-mini lags far behind (17.4%). This consistent trend indicates that advances in LLM translate directly into more capable downstream scientific discovery agents, reinforcing the tight coupling between base model strength and effective multimodal reasoning.
3.3 ERROR ANALYSIS AND AGENT ENHANCEMENT (RQ 2)
Error Analysis. To characterize agent failures in multimodal scientific discovery, we classify errors into three categories: alignment (conceptual or implementation misalignments), model- ing (representation, planning, or computation errors), and reasoning (statistical or logical infer- ence errors). These categories collectively span the entire LLM agent workflow. We conduct a detailed analysis to uncover underlying causes and common failure modes, as summarized in Figure 4. Our analysis focuses the hest-performing agent, ReAct with the base model

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison of LLM-based agents across domains. We report Accuracy / Code Execution Success/ Modeling Rationality (MR) six scientific domains. "Overall" refers to the mvero- acTOSs domains.
Method
Cimaat
Binueelical
Cheminformatis
Health Psych,
Gesomics [Ase/ Exce/Mii
Earth
Overali
Moe Exes MR)
(Ace / Eiec 7MR)
Maax/ Exsc MRI
(Ace Ewe//RR
LAce Elve 7MR)
Meee Eawe/AR
Gwen3-30a-A-2
NolaataGuess
0000/-/-
0.003
0.000/-1
0/149/1.00/
0A12/ 1000
0,338/
1.00/3.6
B
1.000
0,021 11000/3 3.77
1,000
Data Voyager
0.000 /O, 364 3.58
0.529 / (0.60/ 3.62
0.867
/.875
B.067
/3900/3447
0.154/0.001 3.85
0,071
/1.776 3.55
IL14S (UARE 3.64
Rellextoo
0.143/0800 3.82
0,000
/1000/
0.133/
1.000
U.133/
0,154/0,5457 44000
0,071
F1L500/ 3.82
IL106/
00M66/0..99/..0 3.78
SefDDebug
0.429
0.998/0.857
0,60
0.133/ 1
0.286/10.000
00.95/
ReAett
(,000/10003330
/ 1,00/333333
0,500/
1,000/2.47
0.26/11001 1200/527
1,000/44.0
BeagSeek--V... 1
NoDaataGurss
0.000/---
01000/---
0,000/-77
0.000--12
0,000///-
00001-7-
0J100) -/-
0420/1000/3.64
0/647/10000 /1.00/100//77
0,4007
0.267/ 1000/33
3.00
0.143 1 (100/33. 3.64
0.365 /1000/
Vogaer
0.566
/1600/3382
0.333 0.750
0.154/0.55/
3.09
0.143
0.285 /0.753/
Reflesdon
0.55700.33/3.0
0/471/
/1,00/
0,267/
1,000
0.133//
1,00
20.33/
0.843
1.500/337
0.207 0.944/ 3.59
SelfDebugg
0.643/ 1.63//1000// 3,64
0.529
3.711
3.000 0,2007
0,333/
100 5.67
1.000/ 3.82 0,308
0,357
1.000/ 3.57
0 1 1.000/ /.66
RAG-ReAct
0.00/1.0003.88
0
3307
0,1007
1.000/377
0.335/ 0385/1000433 1000 567
0,508/
1.000//.
0.286/1
1.00/3379
0.3331 2.33/1100///0 /376
gpt S-mins
NodhataCoess
0-000-/--
/-767/77-
0,033 1-1-
0.000/-1--
0.000
2--
B.086--7-
0J126 /-/-
ReAst
GU71/1000/22
/1.00/1
0.167/
1.000
UL033/ 10000/ 3.93
0.269
3,90
0.379
1.000/339
0LE74/1 L.070 3.92
Data Voyger
0.00/11000/40
0.176 1.000/
0,0007
4..00
1.000
0.154
1.000/ 4,00
4.00 0.179/
0.085 1.000/
6.0001.00033.86
0.067:
/..00/4.00
4000/
4.000/...0
SellDwhng
0:145/..0003.92
0.353/10000 1.000/
0.0007 4.000/4.00
0.000/
00004400
0/14//1.000
0.2177
1.00044000 4.00
1.000/
RAG-RRAAt
0.00//
0.112/10000400
0.06//10000440
4.000
0000//
10007400
0.144/1.00/4.00
11000/4000 0.021
ILE2S /.900/3.9
o4-mini
NoDuaaGuuss
0.077
ReAct
0,00//1.0003.00
1,000/1000/3.1
0,533/ 1.000/3.93
0.533/ 1000 5:80
0,462/
1,00/ 3.92
0.357
1,000 3.57
0.484/1 1,000// 3.75
Data Voyugrr
0.357/0,375 /3.50
0.568 / 1.00033.5
0.133:0.714/3.79
0/555/6667//55 1567
0.231 /0.2007 3.75
0.286
0.6007 3.50
0.321 0..59 3663
Reflesioo
0.429 /0.60/0000
0.706/ 00.86/
0.400/0.609910
0/467/0
34667/0566//580 3
01462
/0.289 3366
0:286 0.5000/ 3.67
0LASS 0..58 /366
SelfDehugg
0.429 /3.9//3.8433.577
0.50//.57//3.0
0,429 0.666/066///
/06100
RAG-ReAet
0646/1.00/3.86
0.588/ 0.09336
0,313/ 1,000/0)00/3380
0/400/ 0000/364
1400/110003399
1014//0000/39
0.460/ 1.000/00000000
ReAct
ReAct
Domain Knowledge
ReAct
Workflow
0.75
Accuracy
0.50
0.25
0.00
Climate Scl.
Biomedical Eng.
Cheminformatics Health Psych.
Pop. Genomics
Earth Scl.
Figure 5: Performance comparison of ReAct with domain knowledge and workflow scaffolding, We evaluate three ReAct variants across six domains. Task-provided domain knowledge shows limited gains. whereas lightweight workflow scaffolding enhances, or at least maintains, the capabilities of LLM agents through explicit task decomposition.
04-mini. Obs. Alignment errors dominate. The majority of errors are alignment-related (31.8 %), including issues such as information mismatches and data processing failures. The fundamental cause lies in the difficulty of cross-data fusion, linking datasets across domains and transforming diverse forms, distributions, scales, and resolutions into shared, computable represen- tations while preserving domain-specifit information. In comparison, modeling errors account for 15.9% and reasoning errors represent only 3.4%. Further details are provided in Appendix A.2.3.
52-
Alignment
Agent Enhancement. We further attempt to enhance LLM agents from two angles: task- provided domain knowledge and lightweight hu- man workflow scaffolding. Specifically, both task-specific domain knowledge and human workflow scaffolding are explicitly incorporated into the agent's context as executable guid- ance. We evaluate two ReAct variants (Re- Act + Domain Knowledge and ReAct+ + Work- flow) across six domains, with results summa- rized in Figure 5. Obs.@ Task-provided do- main knowledge yields limited or even neg- ative gains, whereas lightweight workflow scaffolding consistently enhances performance.
Modzling
Reasening
Success Case
Fato 1M4
18
5.7%
Figure 4: Error analysis of ReAct. Distribution of alignment, modeling, and reasoning errors across the multimodal scientific discovery.
7

---

Published as a conference paper at ICLR 2026
On average, ReAct achieves 48.4% across six domains. Incorporating task musk-provided domain knowl- edge reduces the average to 44.9% a decline of 3.5%, with notable drops in climate science (from 57.1% to 50.0%) and cheminformatics (from 33.3% to 26.7%). This suggests that naively injecting domain knowledge may introduce noise or misalignment, thereby hindering effectiveness in auto- mated multimodal discovery. In contrast, lightweight workflow scaffolding increases the average to 54.1%. an overall improvement of 5.7%, with the largest relative gains observed in climate science (from 57.1% 1 to 71.4%) and earth science (from 35.7% to 50.0%). These results are consistent with our error analysis: since most failures stem from alignment issues, explicit task decomposition and validation checkpoints introduced by workflow scaffolding significantly improve alignment ability, thereby stabilizing agent performance in multimodal scientific discovery. Specifically, the proportion of alignment errors drops markedly compared with vanilla ReAct (e.g., from 31.8% to 27.3% ), while the share of successful cases increases correspondingly (e.g., from 53.4% to over 60%). This shift shows that workflow scaffolding reduces misinterpretatton and handling errors while promoting consistent reasoning, yielding more reliable outcomes across domains.
3.4 TRADE-OFFS (RQ 3)
Performance by Problem Type. We further break down performance across the five categories, as shown in Fig- ure 7. Obs. Performance varies substantially across problem types. Agents achieve the highest accuracy on causal inference tasks (81.8%), likely because these tasks are explicitly defined and often reduce to structured hy-
Secr bdee
Accuray 13
20.2%
pothesis testing, which LLMS causal direction is clear. In contrast, descriptive (52.0%)
reliably follow when the
and predictive tasks (50.0%) show moderate accuracy: Figure 7: Performance across problem
while agents handle summarization and straightforward types.
supervised modeling, they struggle with maintaining consistency and mitigating error propagation in extended workflows. The sharp drop in correlational (33.3%) and pattern discovery tasks (35.7%) reflects deeper limitations, detecting weak associations or latent structures requires sensitivity to faint statistical signals, robustness to noisy inputs, and inductive generalization beyond observed patterns, areas where current LLM agents remain fragile. Overall, these results suggest that LLMs excel when reasoning steps are well-speeified and rule-based, but falter in exploratory tasks that demand subtle statistical rigor, resilience to noise, and open-ended inference.
Cost Analysis. We further analyze API costs across both domains and agents. Table 4 sum- marizes domain-level results, while Table 5 re- ports agent-level breakdowns. Obs. Cost- effectiveness varies substantially across do- mains. Biomedical Engineering achieves the high- est cost-effectiveeness (1.1), benefiting from rela- tively low cost ($0.57) and high task scores (0.65), likely due to the structured nature of biomedical datasets. In contrast, Population Genomics and Earth Science both incur high costs (above $1.0) yet deliver low accuracy (0.46 and 0.36), yielding the lowest CE (0.4). This highlights that domains with large, noisy, or high -igeediiensional modalities (e.g., genotype matrices or geoscientific data) are less efficiently handled by current agents. Obs. Agents exhibit distinct cost-performance trade-offs. As shown in Table 5, NoDataGuess achieves the lowest cost ($0.04) but offers negligible utility. Reflexion is the most expensive agent ($1.34 on average), driven by repeated trial-and-error loops, yet its performance gains are often marginal relative to the extra cost. ReAct (S0.89) and Data ($0.77) lie between these extremes: ReAct generally provides higher accuracy but at higher cost, while Data Voyager achieves more balanced efficiency, avoiding Reflexion's 's overhead while still improving over naive baselines. These results suggest that improving workflow-level efficiency may yield greater gains than simply scaling model size or computation budgets.
Table 4: Domain-level cost, score, and cost- effectiveness (CE) of Agent ReAct with o4-mini.
The table reports results across six scientific domains, highlighting substantial variations in both absolute
cost and cost-effectiveness.
Domain
Cost
Score
CE
Climate Sci.
$0.98
0.57
0.6
Biomedical Eng.
$0.57
0.65
1.1
Cheminf.
$0.86
0.33
0.4
Health Psych.
$0.76
0.53
0.7
Pop. Gen.
$1.15
0.46
0.4
Earth Sci.
$1.02
0.36
0.4
8

---

Published as a conference paper at ICLR 2026
Table 5: Cost comparison across agents with base model o4-mini. The table reports agent-level costs for four representative agents (NoDataGuess, ReAct, Voyage, and Reflexion) over scientific domains. Results highlight substantial cost differences between lightweight baselines and advanced agent strategies under the same base model configuration.
Agent
Clim.
Bio.
Chem.
Psych.
Gen.
Earth
Avg
NoDataGuess
$0.04
$0.05
$0.04
$0.05
$0.04
$0.04
$0.04
ReAct
$0.98
$0.57
$0.86
$0.76
$1.15
$1.02
$0.89
Data Voyager
$0.94
$0.54
S0.50
$0.78
$0.90
S0.92
$0.77
Reflexion
$1.63
$1.06
$0.83
$1.21
$2.41
$0.92
$1.34
Inference Time Computation. We investigate whether increasing inference-time computation improves agent per- formance through two strategies. First, Best-of-N with ReAct (DeepSeek V3. 1) shows gains up N = but declines thereafter, as additional generations increas- ingly amplify erroneous outputs. Second, Reflexion (Ora- cle) with ni-mini also improves with limited retries but quickly plateaus. Obs. Inference-time scaling yields diminishing returns. Because data-driven scientific dis- covery tasks are inherently complex, each rollout carries high risk of errors. With small number of rollouts (e.g., best-of-3), celf-consistency helps reduce variance and improve reliability. However, the number of roll- outs increases, low-quality generations accumulate and begin outweigh the correct ones, while inference time grows almost linearly with the rollout count. This creates a clear trade-off: limited rollouts can stabilize performance and boost accuracy, but excessive repetition ultimately degrades both efficiency and reliability, making adaptive allocation strategies essential for practical deployment.
Figure 8: Impact of inference-time com- putation on agent performance. We compare ReAct with a Best-of-N strategy (DeepSeek-V3. 1) and Refiexion with it- erative retries (04 -mini..
4 RELATED WORKS
LLM Agents for Scientific Discovery. Recent advances in LLM agents have demonstrated capa- bilities such as advanced reasoning Liu et al. (2025b), code-based tool use Ren et al. (2025), and iterative strategies like reflection and planning Liu et (2025a). Building on these abilities, early prototypes such as the AI Scientist Lu et al. (2024) and Agent Laboratory Schmidgall et al. (2025) explored end-to-end research automation, from hypothesis generation and experimental design to code execution and report writing. However, these systems were evaluated primarily within machine learning subfields and offered limited validation against real scientific studies. In particular, current work rarely tests agents realistic multimodal workflows. MoSciBench addresses this gap by introducing a principled, domain-divers, benchmark for systematically evaluating I.LM agents in multimodal, data-driven scientific discovery.
Benchmarks for Data-Driven Scientific Discovery. A growing body of work has introduced bench- marks evaluate LLM agents in scientific and data-driven workflows Lai al. (2026); Zhou et al. (2026). Early efforts focused on statistical analysis and AutoML benchmarks Chan et al. (2024), or on code generation from structured tasks Liu al. (2024). More recently, Discovery Bench Majumder et al. (2024) and ScienceAgent discovery, providing structured tasks, annotated programs, and graded evaluation criteria. Science- Board Sun et al. (2025) further extends this line by creating realistic multi-domain environment where agents interact with professional software to complete end-to-end scientific workflows. While these contributions mark important progress, most tasks remain either unimodal (e.g.. tabular data) or centered on system-leve interactions. In contrast, realistic scientific discovery requires integrative reasoning over diverse modalities such imagery, time series, metadata, and text. Our MoSciBench addresses this gap by introducing the first benchmark explicitly designed for multimodal, Luta-driven scientific discovery with hypothesis verification grounded in peer-reviewed reve-weviewed studies.
Chen et al. (2024b) have formalized the workflow of data-driven

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this work, we introduced MoSciBench, the first benchmark for multimodal data-driven scientific discovery, constructed from peer-reviewed studies across five scientific domains. MoSciBench formalizes tasks as end-to-end workflows that require cross-modal alignment, scientific computation, and reasoning verify hypotheses. Through systematic evaluation, show that current LLM agents struggle with multimodal synthesis, highlighting fundamental gaps in their ability to integrate heterogeneous evidence. By providing scientifically grounded tasks and reproducible evaluations, MoSciBench establishes new challenge space for advancing LLM agents toward more reliable and impactful roles in scientific discovery.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 62572417, No.92370204), National Key R&D Program of China (Grant No. 20.02227707755044) the Guangzhou Basic and Applied Basic Research Program under Grant No. 2024A0413279, Education Bureau of Guangzhou Municipality, and CCF-DiDi GAIA Collaborative Research Funds.
REFERENCES
Christoph Anders, Sidratul Moontaha, Samik Real, and Bert Amniich. Unobtrusive measurement of cognitive load and physiological signals in uncontrolled environments. Scientific Data, 11(1): 1000, 2024.
Cezar Anicai and Muhammad Zeeshan Shakir. A multimodal dataset of cardiac, electrodermal, and environmental signals. Scientific Data, 12(1):844, 2025.
Roman Bushuiev, Anton Bushuiev, Niek de Jonge, Adamo Young, Fleming Kretschmer, Raman Samusevich, Janne Heirman, Fei Wang, Luke Zhang, Kai Dührkop, et al. Massspecgym: A benchmark for the discovery and identification of molecules. Advances in Neural Information Processing Systems, 37:1 10010-1 1000000017,,0 2024.
Francesc Calafell and Simone Andrea Biagini. France dataset, 2019. URL httpt:://///l/ orr/ 10. 608 4 m9. figshare 10008689.v.. vl.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095. 2024.
Wei Chen, Xixuan Hao, Yuankai Wu, and Yuxuan Liang. Terra: A multimodal spatio-temporal dataset spanning the earth. Advances in Neural Information Processing Systems, 37:66329-66356 2024a.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang. Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven seientific discovery. arXiv preprint arXiv:2410.05080. 2024b.
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, et al. Blade: Benchmarking language model agents for data-driven 2024.
science. arXiv preprint ArXiv:2408.09::
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. arXiv preprint arXiv:2402.17453, 2024.
Seyedmajid Hosscini, Raju Gottumukkala, Satya Katragadda, Ravi Teja Bhupatiraju, Ziad Ashkar, Christoph W Borst, and Kenneth Cochran. A multimodal sensor dataset for continuous stress detection of nurses in hospital. Scientific Data, 911):255, 2022.
Asanobu Kitamoto, Erwan Dzik, and Gaspar Faure. Machine learning for the digital typhoon dataset: Extensions to multiple basins and new developments in representations and tasks. arxiv preprint N..:24111.16221 2024.
10

---

Published as a conference paper at ICLR 2026
Siqi Lai, Zhao Xu, Weijia Zhang. Hao Liu, and Hui Xiong. Limlight: Large language models as traffic signal control agents. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. pp. 2335-2348 2025.
Siqi Lai, Yansong Ning, Zirui Yuan, Zhixi Chen, and Hao Liu. USTBench: Benchmarking and dissecting spatiotemporal reasoning capabilities of LLMs urban agents. In The Fourteenth International Conference on Learning Representations, 2026. URL https: /tapss/////eeeeeeeee
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yar Chiu, Vishal Dey, Zitong Lu, et al. Autosdt: Scaling data-driven discovery tasks toward open co-scientists. arXiv preprint arXiv:2506.08 140, 2025.
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang. Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song. Kunlun Zhu, et al. Advances and challenges foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.01990, 2025a.
Fan Liu, Wenshuo Chao, Naiqiang Tan, and Hao Liu. Bag of tricks for inference- computation of Ilm reasoning. arXiv preprint arXiv:2502.07191, 2025b.
Fan Liu, Zherui Yang, Cancheng Liu, Tianrui Song, Xiaofeng Gao, and Hao Liu. Mm-agent: Llm as agents for real-world mathematical modeling problem. arXiv preprint arXiv:2505. 14148, 2025c.
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint Stt1::20080032211 2024.
Zhong Liu and Tian Yao. Strategizing earth science data development. Scientific Data, 11(1):693, 2024.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The scientist: Towards fully automated open-ended scientific discovery. arXiv preprint 2tti::2000006929 2024.
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725. 2024.
Shuo Ren, Pu Jian, Zhenjiang Ren, Chunlin Leng, Can Xie, and Jiajun Zhang. Towards scientific intelligence: survey of llm-based scientific agents. arXiv preprint arXiv:2503.24047 2025.
Samuel Schmidgall, Yusheng Su, Ze Wang. Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants. arXiv preprint ar Xiv:22500 04227, 2025.
Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint arXiv:2505. 19897, 2025.
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv preprint Xrv::2231 .09835, 2023.
Minyang Tian, Luyu Gao, Shizhuo Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624-30650 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
11

---

Published as a conference paper at ICLR 2026
Zirui Yuan, Siqi Lai, and Hao Liu. CoLLMLight: Cooperative large language model agents for network-witd traffic signal control. In The Fourteenth International Conference on Learning Representations, 2026. URL https:
Yanbo Zhang, Sumeer Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, et al. Exploring the role of large language models in the scientific method: from hypothesis discovery. npj Artificial Intelligence. 1(1):14, 2025.
Tianshi Zheng, Zheye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang, and Yangqiu Song. From automation to autonomy: A survey on large language models in scientific discovery. arXiv preprint arXiv:2505.13255. 2025a.
Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh TN Nguyen, Lauren T May, Geoffrey I Webb, and Shirui Pan. Large language models for scientific discovery in molecular property prediction. Nature Machine Intelligence, pp. 1-11, 2025b.
Yu Zheng. Fusing cross-domain knowledge from multimodal data solve problems in the physical world. ACM Transactions on Intelligent Systems and Technology, 2025.
Yuping Zhou, Siqi Lai, Jindong Han, and Hao Liu. An llm-powered cooperative framework for large-scale multi-vehicle navigation. In Companion Proceedings of the ACM Web Conference 2026, New York, NY, USA, 2026. Association for Computing Machinery.
12

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MoSciBench DATA
Data Source. Our dataset is curated from peer-reviewed scientific papers across six domains: climate science, biomedical engineering, cheminformatics, health psychology, population genomics, and earth science. Specifically, we replicate workflows and hypotheses from prior works in each domain, including climate science Kitamoto et al. (2024), biomedical engineering Anicai & Shakir (2025), cheminformatics Bushuiev et al. (2024), health psychology Hosseini et al. (2022), population genomies Calafell & Biagini (2019), and earth science Chen et al. (2024a). All datasets and associated assets are released under CC or other permissive open licenses, ensuring accessibility and compliance with data-sharing standards.
Composition of Instructions. To illustrate how each task is represented in our benchmark, we provide detailed explanation of the task instruction schema. Each field defines specific aspect of the scientific problem, from context and hypothesis to workflow, expected answer format, and evaluation criteria, shown below:
Task Instruction Schema
id Unique identifier of the task instance.
background Contextual description the scientific motivation for the task.
hypothesis The scientific hypothesis to be tested.
workflow Step-hy-step instructions outlining the expected analysis procedure.
gold_hypothesis- - Ground-truth scientific conclusion derived from the data.
scientific_domett The disciplinary area where the task belongs (e.g., biomedical engineering).
problem_ type- The reasoning category of the task (e.g., descriptive, predictive, causal). task type Specific methodological type (c.g., statistical tests).
domain knowledge Key domain knowledge or definitions needed to perform the task. modality The data modality used (e.g., time series, tabular, image).
answer answer_format- The expected format of the answer, ensuring consistency across outputs. evaluation The reference numerical value for evaluation (e.g., percentage of outliers). judge_ type The evaluation criterion (e.g., exact match "=", range-based). based).
A.2 EXPERIMENTS
A.2.1 EXPERIMENTAL SETUP
Agent Frameworks. At present, there dedicated multimodal discovery agents. Existing multimodal foundation models, such as -lniun-e models or pretrained fusion architectures, not suitable baselines, as they primarily target images and text. contrast, MoSciBench covers far richer set of modalities, including long multivariate time series, tabular data, satellite imagery, molecular structures, mass spectra, genotype matrices, and HDF simulation outputs. These formats exceed the input constraints of current models, making direct application infeasible. Consequently, end-to-end L.M agents with tool use remain the only practical approach for executing repository- level discovery workflows in this setting. To ensure faimess and coverage, we benchmark four representative frameworks: (I) NoDataGuess. A naive baseline that provides only task descriptions and relies entirely on the LLM's internal memory and reasoning. (2) ReAct Yao et al. (2023): Alternates between reasoning and code execution an iterative loop to refine hypotheses. (3) Data Voyager. A modular pipeline with planner, code generator, analysis, and critic components to orchestrate discovery. (4) Reflexion (Oracle). Extends CodeGen with oracle feedback and up to three iterative retries for seff-improvemment (5) SelfDebug. program-execution agent that iteratively
13

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison of Qwen-based LLM agents across domains. We report accuracy for six
scientific domains. "Overall Avg" refers to macro average across domains.
Method
Climate Sd.
Biomedical Eng.
Cheminformatics
Health Psych.
Pop. Genomics
Earth Sci.
Overall Avg
Qwen332358
NoDataGuess 0.000
0,000
0.000
0.000
0,000
0.000
0.000
ReAct
0,500
0.529
0.133
0.267
0.385
0.357
0.361
Data Voyager
0,500
0.588
0.200
0,200
0.231
0,357
0.346
Reflexion
0,286
0.471
0.133
0.200
0.231
0.143
0.244
Qwen3-Coder
NoDataGness
0.000
0,000
0.067
0.067
0,000
0.000
0.022
ReAct 0,500
0.529
0.200
0.333
0.462
0.429
0.408
Voyager
0.471
0.300
0.214
Reflexion 0,500
0,588
0.200
0.133
0.154
0.143
0.286
debugs itself by inspecting execution traces, detecting failure symptoms, and rewriting code based on internal error hypotheses. (6) RAG-ReAct. A retrieval-augmented variant of ReAct that supplements the agent's reasoning trajectory with external domain knowledge.
A.2.2 FURTHER EXPERIMENTS ON OTHER BASE MODEL
We further conduct experiments to evaluate the performance of Qwen- LLM agents across six scientific domains. As shown in Table 6, the ReAct framework consistently outperforms all other methods for both Qwen3-235B and Qwen3-Coder. For Qwen3-Coder ReAct achieves highest overall average of 0.408, demonstrating strong effectiveness in climate science (0.500) and biomedical engineering (0.529), while also maintaining competitive results in earth science (0.357). Similarly, for Qwen3-235t ReAct again leads with overall average of 0.361, showing notable robustness in population genomics (0.385) and earth science (0.357). These results highlight ReAct the most reliable framework across domains, underscoring its ability to generalize effectively in complex multimodal scientific settings.
A.2.3 ERROR ANALYSIS AND AGENT ENHANCEMENT
Error Analysis. To characterize agent failures in multimodal scientific discovery, we classify emors into three categories: alignment (conceptual or implementation misalignments), modeling (representation, planning, or computation errors), and reasoning (logical or statistical inference errors). Alignment emrrs include concept misalignments (e.g., selecting the wrong variable, lead-lag mismatches, or entity mismapping) and implementation misalignments (e.g., faulty joins, missing keys, or unit conversion failures). Modeling errors arise from flawed representations that fail to capture essential scientific relationships, as well as from training or implementation issues, including coding mistakes, unstable optimization, feature misordering. Reasoning encompass logical or statistical inference mistakes (e.g., conflating correlation with causation or misusing significance thresholds) and reporting output parsing formats or failed result extraction). This taxonomy clarifies why agents may generate outputs that appear plausible yet deviate from correct multimodal reasoning, and collectively spans the entire agent workflow. We adopt the LLM-as-judge framework to classify each failure instance into these categories, ensuring systematic and reproducible evaluation. Our analysis centers on the best-performing agent, ReAct with the base model c4 minni, as summarized in Figure 4. Alignment errors dominate. The majority errors are aligndent-related (31.8%), such as information mismatches and data processing failures. The root cause lies in the difficulty of integrating datasets across domains and transforming heterogeneous forms, distributions, scales, and resolutions into unified, computable representations while retaining Nomain-speciffe information. In comparison, modeling emror account for 15.9% and reasoning errors represent only 3.4%.
plans
(e.g., incorrect answer
A.2.4 EXTENDED EVALUATION WITH ADDITIONAL AGENTS AND METRICS
14

---

Published as a conference paper at ICLR 2026
To broaden the experimental coverage, we additionally include two baselines that stress code reliability and external knowledge usage: Self-Debug, a code-execution agent with iterative repair capabilities, and RAG-ReAct, retrieval-uugmentee variant of ReAct tailored for scientific discovery. Table reports their performance across six scientific domains. Overall, ReAct-style agents continue to excel in producing executable analyses, Self-Dehug improves robustness through stepwise correction, and RAG-ReAct shows mixed gains, suggesting that naively incorporating external knowledge does consistently benefit data-driven scientific inference. Table reveals several noteworthy patterns across the six scientific domains. First, although most agents achieve nearly perfect Code Execution Success Rates, their Accuracy (Acc) can be substantially lower. For instance, under g RAG-ReAct reaches Exec = 1.000 yet its accuracy in Cheminformaties is only 0.067, indicating that the generated code syntactically valid scientifically misaligned. Similar trends appear under Qwwn3-300 - where Data Voyager attains Exec 0.875 in Cheminformaties but produces only 0.067 correct answers, showing that executable workflows can still follow incorrect modeling assumptions. Second, Self- illustrates the limits of iiccution-focusee agents. It consistently fixes runtime errors (e.g.. Exec = 1.000 in Climate Science under o4 04-mini) yet achieves only moderate accuracy (0.333), because robustness to code failures does not guarantee that the analytical pipeline is conceptually sound. Its strength lies in error recovery rather than hypothesis refinement. Third, ReAct-style agents demonstrate the most balanced performance. Across models, ReAct achieves both high execution reliability (often Exec = 1.000) and competitive accuracy for example, 0.571 in Climate Science and 0.647 in Biomedical Engineering under 04 14minn This suggests that interleaving reasoning with incremental code generation helps maintain scientific coherence and reduces modeling drift. Finally, retrieval-augmented RAG-ReAct exhibits mixed benefits. Although retrieval improves Exec uniformly, its accuracy can drop sharply when external knowledge conflicts with the dataset. For example, under gpt -mini, RAG-ReAct achieves Exec 1.000 but accuracy falls to 0.000-0.067 in multiple domains, revealing that naïve retrieval may introduce noise rather than useful inductive priors.
15


Reviews:
[{"review_id": "bggqf3alCb", "reviewer": "Reviewer_AMy8", "summary": "This paper introduces MoSciBench, a multimodal benchmark designed for scientific discovery that enables agents to access complete repositories, integrate heterogeneous data, generate and execute code, and reason over results to verify scientific hypotheses. The experiments across 88 tasks reveal that cross-modal alignment is a significant bottleneck, while lightweight workflow scaffolding consistently enhances performance.", "questions": "identical to the 'weaknesses'", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tUnlike previous unimodal benchmarks, MoSciBench explicitly targets multimodal, repository-level discovery, significantly increasing task complexity and realism. This benchmark will be valuable for evaluating the progress of AI agents within the community.\n2.\tThe experiments conducted provide valuable insights into enhancing agents in scientific domains, highlighting areas for further development.", "weaknesses": "1.\tThe ground-truth hypotheses and answers in MoSciBench are derived from peer-reviewed publications. How rigorous is this benchmark? Additionally, if an agent can access search engines, how would that impact its ability to find answers?\n2.\tFor Figure 4, are there significant differences in error distributions among tasks with varying requirements? Which specific data models are particularly prone to failure, and is this related to the length or format of the data?\n3.\tDid the introduction of lightweight human workflow scaffolding change the proportion of model invocation tools/code execution used? Without this scaffolding, would models demonstrate a need for additional information or behave differently in their outputs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8I70bmXw5r", "reviewer": "Reviewer_Yaab", "summary": "This paper introduces MoSciBench, a benchmark for multimodal, data-driven scientific discovery powered by LLM agents. The benchmark includes six scientific domains, seven data modalities, and five discovery task types, totaling 88 tasks. The authors systematically evaluate several LLM-based agent frameworks and provide a detailed analysis of error sources and limitations. Overall, the paper addresses an important and underexplored problem — assessing AI agents in real-world, multimodal scientific workflows.", "questions": "1. Although the benchmark aims to emulate “scientific discovery,” most tasks are still formulated as structured, answerable queries with gold labels. The open-ended and hypothesis-generating aspects of real discovery are largely absent.\n\n2. Only a small set of existing agent frameworks are evaluated. It is unclear whether the conclusions generalize beyond the tested models.\n\n3. Reported accuracy numbers (≈50%) are low and not deeply analyzed beyond descriptive statistics. There is limited discussion of why certain domains are more difficult.\n\n4. For evaluation metrics, the reliance on exact-match accuracy is restrictive and might underestimate partial success or reasoning quality.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces MoSciBench, a benchmark for multimodal, data-driven scientific discovery powered by LLM agents. The benchmark includes six scientific domains, seven data modalities, and five discovery task types, totaling 88 tasks. The authors systematically evaluate several LLM-based agent frameworks and provide a detailed analysis of error sources and limitations. Overall, the paper addresses an important and underexplored problem — assessing AI agents in real-world, multimodal scientific workflows.", "weaknesses": "1. The paper tackles a meaningful and challenging goal — end-to-end scientific discovery from heterogeneous data sources — which is timely and relevant to the emerging intersection of AI agents and scientific reasoning.\n\n2. MoSciBench is well designed, covering a diverse set of domains and modalities. The data curation pipeline is clearly described and appears reproducible.\n\n3. The authors perform an insightful breakdown of error categories (alignment, modeling, reasoning), providing useful diagnostic information for the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LMUWi8N96K", "reviewer": "Reviewer_mCwP", "summary": "This paper introduces MoSciBench, the first benchmark specifically designed for multimodal data-driven scientific discovery powered by LLM Agents. MoSciBench consists of 88 tasks, evaluated through a principled four-stage pipeline, which assesses agents on repository-level tasks that require alignment, modeling, and reasoning across seven data modalities and six scientific domains.", "questions": "1. Could you provide a few concrete examples of alignment errors as case studies, and explain what kind of mechanism might cause them to happen?\n\n2. Have you tested whether a longer runtime significantly improves performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. MoSciBench is the first multimodal benchmark, covering 7 data modalities and 6 scientific domains. \n\n2. MoSciBench identifies the bottleneck of cross-modal alignment of  LLM agents in real-world multimodal scientific tasks. The author presents that over 30% of failures stem from misaligned data rather than flawed reasoning, which is a valuable insight.\n\n3. MoSciBench reveals the importance of data grounding in LLM-based scientific discovery. NoDataGuess approach performing close to 0% accuracy indicates that relying solely on LLMs’ internal knowledge is insufficient for solving scientific problems.\n\n4. By evaluating NoDataGuess, ReAct, Reflexion, and the proposed DataVoyager, the paper reveals critical limitations across current agent architectures.", "weaknesses": "1. The baseline coverage is limited. Since all evaluated agents are prompt-based reasoning and code generation frameworks like ReAct and Reflexion, no domain task-specific Agents, multimodal-specific Agents, or retrieval-augmented Agents are included. \n\n2. While alignment errors are identified as the dominant failure mode, the root causes are not explored. Whether this arises from architectural limitations (e.g., lack of explicit alignment modules) or inherent model incapacity is not discussed. Without deeper mechanistic analysis and case studies, the paper does not provide further guidance beyond a generic suggestion of “better alignment is needed”.\n\n3. The evaluation relies exclusively on end-to-end exact match, which may conflate semantically correct solutions with fundamental failures. It is better to report a secondary metric (e.g., partial/subtask credit) or analysis for correct reasoning traces to better disentangle reasoning failure from execution failure.\n\n4. The 1-hour execution cap is not explained. Some failures may be due to time limits rather than methodological flaws.\n\n5. The paper includes several formatting and naming inconsistencies. e.g., in lines 278 and 646, it mixes the spelling of its self-proposing framework “DataVoyager” and “DataVoyage”. In line 295,  there is a missing space before “DeepSeek-V3.1”. In line 689, there is a missing space between the text and the period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ezmxNWvWkq", "reviewer": "Reviewer_BRwp", "summary": "This article designs a multi-model benchmark for data-driven scientific discovery, evaluating agents across five categories of discovery questions and seven data modalities. It also tests on state-of-the-art models, analyzes their performance, and proposes the “ReAct + Workflow” method to enhance agent effectiveness.", "questions": "Table 6 only shows the results for React, but lines 658 - 660 mention that “the ReAct framework consistently outperforms all other methods for both Qwen3–235B and Qwen3–Coder,” which cannot be concluded from Table 6.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  The core contribution of this paper is the introduction of MoSciBench, which is the first benchmark focused on evaluating LLM Agents in performing multimodal data-driven scientific discovery tasks.\n2.  MoSciBench itself covers five categories of discovery questions and seven data modalities, demonstrating good comprehensiveness.", "weaknesses": "1. The article only shows that the performance of ReAct + Domain Knowledge deteriorates, but does not analyze why the performance worsens. Moreover, what would be the effect if Domain Knowledge and Workflow were added simultaneously?\n2. Based on reference [1] and the examples shown in the article, MoSciBench appears to be testing the model's ability to integrate various data for reasoning. However, according to Table 3, the performance of NoDataGuess is very poor; even with o4-mini, it hardly gets any answers correct. It would be beneficial to add an analysis regarding NoDataGuess.\n3. Figure 4 is the same as Figure 9. The text does not introduce Figure 4 and directly uses Figure 9 in line 318. The existence of Figure 4 seems to be meaningless.\n\n[1] Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)