EXP-Bench: Can AI Conduct AI Research Experiments?
Authors: Patrick Tser Jern Kon
Qiuyi Ding
Jiachen Liu
Xinyi Zhu
Jingjia Peng
Jiarong Xing
Yibo Huang
Yiming Qiu
Jayanth Srinivasa
Myungjin Lee
Mosharaf Chowdhury
Matei Zaharia
Ang Chen
Venue: ICLR 2026 Poster (2026)
Keywords: AI Agents
AI for Science
Research Experimentation
Primary area: datasets and benchmarks

Abstract: Automating AI research holds immense potential for accelerating scientific progress, yet current AI agents struggle with the complexities of rigorous, end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier AI research papers. Evaluations of leading AI agents, such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial capabilities: while scores on individual experimental aspects such as design or implementation correctness reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future AI agents to improve their ability to conduct AI research experiments.

Published as a conference paper at ICLR 2026
EXP-BENCH: CAN AI CONDUCT AI RESEARCH EXPERIMENTS?
Patrick Tser Jern Kon', Qiuyi Ding', Jiachen Liu', Xinyi Jingjaa Peng' Jiarong Xinga., Yibo Huang', Yiming Qiu Jayanth Srinivasa", Myungjin Lee Mosharaf Chowdhury', Matei Zaharia', Ang Chen
University of Michigan Rice University Cisco Research 'UC Berkeley
ABSTRACT
Automating AI research holds immense potential for accelerating scientific progress, yet current Al agents struggle with the complexities of rigorous, end- to-end experimentation. We introduce EXP-Bench, novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given research question and incomplete starter code, EXP- challenges AI agents to formulate hypotheses, design and im- plement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier Al research papers. Evaluations of leading LLM-based agents, such as OpenHands and IterativeAAgnnt on EXP-Bende demonstrate partial capabilities: while scores on individual experi- mental aspects such design implementation correctness occasionally reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future Al agents to improve targeted research components and agent planning ability. EXP-Bench is avail- able at https: Ststtaaabcccomtttttttttreooo//re Beenchmark/eee bench
wttttreeeeeee/////ee/ main//
INTRODUCTION
EXP-Bench Dataset
AI Agent
Perf.
Research
Question
Higellevel Method
Starter Code
Input
Exp. Design
Modii
Facoure
Canclusion
Evaluate
Design: 80%
cacuset
Impl. 50%
comnttstion
ieprovvd.
conrolation...
D
nai expg
Code Eisee.; Failed
Conclusion: Incorrest
Extracted from Top-Ties Tier ML. Papers
Figure 1: EXP-Bench evaluates AI agents research experiment tasks extracted semi-autonomously from peer-reviewed Al papers. Given a research question, a high-level method description, and starter code, agents tasked with designing, implementing, and executing complete experiments. Performance is validated through ground-truth comparisons and implementation execution.
Automating AI research stands as cornerstone for accelerating the development of advanced intelligence and human progress. Unlike disciplines that require extensive physical interaction, AI research is inherently digital, rendering it particularly amenable to automation by Large Language Model (LLMM)-rrvven Al agents. Recent work has demonstrated that these agents demonstrate nascent capabilities in tasks like literature synthesis (Elicit (025). hypothesis generation (Weng al. (025) and code generation (Li et al. (202).. However, empirical Al research requires rigorous end-to-end experimentation, which goes beyond these individual tasks.

---

Published as a conference paper at ICLR 2026
To realize the vision of agents conducting holistic Al research, rigorous benchmark is needed one that evaluates and guides agents through the full experimentation pipeline step by step. We present EXP-Bench, benchmark designed to comprchensively assess an AI agent's ' ability to carry out end-to-end research experiments. As illustrated in Fig. | EXP-Bench challenges agents with tasks sourced from influential, peer-reviewed AI publications (e.g., NeurIPS, ICLR) along with their open-source implementations. These papers reflect already-completed, peer-validated research and serve as concrete exemplars of full experimental workflows. By exposing agents to such tasks, test their ability to conduct established scientific procedures grounded in real-world Al experimentation. For each task, an agent is provided with a core research question. a high-level methodological overview, and starter code. The agent should then formulate viable hypotheses, design Al-specific experimental procedures (e.g., data handling, model selection, and hyperparameter optimization), correctly implement and execute these experiments, and derive valid conclusions from the results.
However, curating these high-fidelity and structured experimental tasks presents considerable chal- lenges. Academic papers typically present polished narrative focusing final results and conclu- sions, often omitting the detailed intermediate steps of the experimentation process. Additionally, critical details, such the precise conditions under which results hold subtle data preprocessing steps, are often fragmented across multiple sources, including dense academic papers, supplementary materials, and sprawling codebases. This makes manual curation of such tasks labor-intensive and difficult to scale.
To address these challenges, we develop semi-automated dataset curation pipeline. We first filter for high-quality Al papers with open-source codebases using citation and repository popularity signals. Task extraction then proceeds in two stages: (I) multi-modal extraction phase that identifies the core elements of the research problem, such as the main question, expected outcomes, and high-level experimental setup (e.g.. datasets, evaluation metrics, model configurations) from papers, supplementary materials, and code; and (2) an implementation extraction phase that locates relevant code and assembles scripts to solve the specified task. We further apply execution -based validation to ensure functionality. While human oversight is used, the availability of original implementations and ground truths reduces the validation burden to mostly lightweight consistency checks. With the pipeline, EXP- currently comprises 461 research tasks (12,737 individually gradable subtasks) derived from 51 papers published at NeurIPS and ICLR 2024, spanning diverse Al subfields such as reinforcement learning, Al applications and generative models.
We use a multi-metric evaluation pipeline (Fig. ! to assess agent performance across all core phases of experimentation: design, implementation, execution, and conclusion. Each metric captures distinct capability, and their conjunctive use ensures that agents correctly understand and complete the experiment. Initial evaluations of leading agents reveal that, while they often succeed executing routine procedures, such as running pre-written scripts or replicating documented analysis steps, they struggle when tasked with conducting complex experiments. Specifically, we observe failures in: (a) Conceptualizing and operationalizing sound experimental designs from high-level research questions and methods (16.1% misclassified design variables); (b) Translating abstract research methodologies into complete and correct code implementations (39.7% missing essential implemen- tation components); and (c) Ensuring the robust and reproducible execution of complex experimental software stacks (29.4% environment or dependency misconfiguration or 23.8% script-level errors). By identifying these key bottlenecks, EXP-Bench helps us target specific research components for improvement and advance next- cext-generation AI agents for autonomous research.
2 RELATED WORK
While existing benchmarks have advanced the evaluation of Al agents in various scientific reasoning, coding, and specific machine learning tasks, EXP-Bench distinctively addresses the holistic challenge of end-to-end and step-by-step Al research experimentation. See App. A for additional discussion.
Scientific Reasoning Benchmarks. Benchmarks like BoxingGym Gandhi et al. 2025) explore
simulated theory formation, while others such as AAAR (Lou et al. 2024) and Lab- Bench (Laurent
et al. (024) assess reasoning experimental design based static artifacts (e.g., protocols, figures). While valuable for assessing abstract reasoning, these benchmarks do not evaluate the agent's ability to perform actual experiments.
2

---

Published as a conference paper at ICLR 2026
Agent Input
Ground Truths
Research Question
Starter Code
Experiment Design
Implementation
Does the MogsiNet rechitecture culperform existing lightweight mudels in EmageNet IK classtication when trainod under standrd settings?
Task: Evaluste MogsNNet lightweight
sppenEepoch,
"oonggal:
/eackspace
buselines on ImageNet-1K
sisip.aff,
I---eenenttation
Ingut Sinee 226x224, 256x256 Mudel-Sperife Settings
prsfetster
I. test.py
I-x-rodels
MogaNee--T drop, path=0.1, wd=0.04 Merris: Top-1 Accurcy.... ...
High-level level Method
I... roganntt.pr
petettrroptiiiicen
Train MogaNet varianss (e.g. XT) on
I--geet Flops iy
s8nGh-ttiwen Joszeeun fontronlletet)
ImageNNet-18 1K
spochs using AdamW (1e-3, cosine schodulo).
---exxtract expt.py Py
Code Instructions
...coniaige.ppp .yy
1. Set up training parumeters inchdinng model type, imsaee size, spochs,
LEnico DMLtied
Evaluste sop-1 accuracy and efficiency wight hasellines (e-g. PurC- Net-S, T2T VEI-7) using standard input sizes...
Expected Outcume
Additional Instructions
2. Create MogaNeet model speciffed
MogaNett XI/T significuntlyy cutperfomss existing lightweight architectures with a more efficient
anchinecture (tiyy xtiny).
usago parameters and FLOPS...
Figure 2: One Al research task example from ICLR 2024 MogaNet (Li al. 2024b).
Scientific Coding Benchmarks. Scicode (Tian et al. 2024). for instance, focuses on generating code snippets for natural science tasks, while BLADE (Gu et al. 2024), DiscoveryBench (Majumder et al. 2024), and SeienceAgentBench (Chen et al. 2024) primarily assess post-hoc data analysis or hypothesis testing. While critical to the scientific process, they often isolate coding or analysis from the broader, iterative experimental context.
Several benchmarks specifically target ML tasks, yet often focus on sub-components operate with simplifications of the full research cycle. For example, DSBench (Jing et al.] 2024b). ML-Agent- MLL-Agent-Bench (Huang et al., 2023b), and MLE-Bench (Chan al. 2024) assess ML problem-solving capabilities, such as script editing or hyperparameter tuning, frequently within constrained environments like Kaggle challenges. Other benchmarks such as RE-Bench (Wijk al., 2024). ML-Gym (Nathani et al. 2025), and Curie (Kon et al. 2025). compare agent performance against humans on research tasks. but often operate at limited scale (c.g.. RE- Bench features only hand-curated tasks) or use simplified evaluation metries. CORE-Bence (Siegel et al. 2024) supplies the full codebase for result reproduction and manually sources papers from the curated CodeOcean repository. PaperBench (Starace et al., (025) assesses agents on tasks derived from academic literature, focusing on their proficiency in executing specific, well-defined sub-components of the research process, such as running documented code scripts or performing standard data analyses. While these benchmarks provide valuable insights into specific ML tasks, they generally fail to capture the complexity of realistic end-to-end Al research workflows, nor do they typically offer methodology for constructing such comprehensive benchmark tasks scale.
Machine Learning Benchmarks.
THE EXP-BENCH BENCHMARK AND DATASET
EXP-Bench is built evaluate the AI agent's 's ability to address Al research tasks by conducting end-to-end experimentation. Each research task is grounded on an influential AI research paper and its corresponding codebase. This coupling captures the full scientific workflows, linking concrete high-level ideas to executable implementations ($3.T). We achieve scalable construction of these high- fidelity tasks through eemi-automated curation pipeline, which integrates multi-modal extraction with lightweight human verification ($3.2). This design also opens the door to large-scale data generation for training agents capable of automating core aspects of Al research.
3.1 EXP-BENCH DATASET SPECIHICATION
Our dataset is a collection of AI research tasks, each structured to emulate complete experimental process designed to address a specific Al research question from published paper. As shown in Fig. [ each task entry in the dataset contains a problem statement for the agent, and the corresponding ground-trutt solution derived from the original research artifacts.
Problem Statement (Agent Input). Each task instance within -XPchhncc provides the agent with: (1) Research Question: A specific goal derived from the source paper's 'ss experiments. (2) Hive-Level Method: A description guiding the required experimental approach; and (3) Code Repository: Access to the relevant code, potentially with specific components/scripts masked requiring modification.
3

---

Published as a conference paper at ICLR 2026
125
Counts
250 200
Task Count
75
20
GLR R Papera
ICR # Tinka
NauripS e Papirra
NewePP Tasks
25
Reinforesment Learning
Deep Learaing
Applications
Optimization
Computer Vision
Generative Modela
Multimodal Modela
Social Aspecta
Trastworrhy Machine Learaing
Langragg
Time Sertea
Computational Blology
Causality
Probabilistic Methode
Theory
Dataseta and Bonchmarko
Physical Models
Figure 3: EXP-Bench's s dataset comprises tasks from a diverse set MI. research categories.
Expected Outcome (Ground Truth). Each task instance also includes a ground-truth experimental solution curated from the source paper and codebase. This solution, used to evaluate agent outputs, comprises: (1) an experimental design specifying key variables, constants, and procedures; (2) the necessary code modifications, assessed via git diff against the provided repository: and (3) final conclusion that directly answers the research question based on experimental results.
Benchmark Overview and Statistics. EXP-Bench currently includes 461 research tasks drawn from 51 influential papers, as detailed App. B As shown in Fig. 3 these tasks span diverse AI subfields (including Computer Vision, NLP, and Reinforcement Learning) and are sourced from top-tier venues, namely NeurIPS 2024 (53%) and ICLR 2024 (47%). This breadth ensures coverage of diverse experimental paradigms, coding practices, and research challenges prevalent in the AI field. Moreover, each task broken down into fine-grained, individually gradable subtasks spanning cecponenttsdeiiieee implementation, and conclusion conclusion-resultiing in total of 12,737 subtasks. Together, these features make EXP-Benchh comprehensive testbed for assessing the capabilities of Al research agents.
all three ground-truth
3.2 EXP-BENCH AUTOMAATD DATASET CONSTRUCTION PIPELINE
Curating a high-fidelit benchmark for end-to-end AI experimentation is challenging due to the fragmented and domain-specific nature of real-world research artifacts (namely papers and their associated codebases). Critical experimental details are often scattered, implicit, or embedded in dense technical language, making manual extraction labor-intensive and difficult scale. To address this, propose semi-automated construction pipeline that systematically structures these artifacts into benchmark tasks with lightweight human oversight. The pipeline comprises three stages (Fig. 4)
Stage 1: Source Selection and Filtering. The process begins by identifying candidate research artifacts that form the basis of high-quality experimental tasks. We target influential papers from top-tier AI conferences (e.g.. NeurIPS, ICLR) that are accompanied by publicly available code repositories. Initial filtering criteria are applied prioritize impactful and potentially reproducible research, considering factors such as citation counts, and code repository activity (e.g., GitHuu stars, forks). This selection phase aims to establish strong foundation by focusing on artifacts that, despite potential imperfections, represent significant and verifiable research contributions.
Stage 2: Experiment Procedure Extraction. Research papers rarely present experiments as complete procedures; key steps are often implicit or scattered. To enable structured agent evaluation, we decompose each task into explicit sub-steps. This transforms high-level research goals into concrete workflows (e.g., multi-step experiment design and environment setup), making them suitable for both execution and fine-gra finennrained evaluation. This stage extracts the complete research task by combining the research plan (from the paper) with its corresponding experiment implementation (from the codebase). Further implementation details be found in App. G
4

---

Published as a conference paper at ICLR 2026
Filter & Collect Al Research Artifacts
Extract Research Task
EXP-Bench Dataset
Index PDF
Compose Research Task
51 Al Reseurch Pupers
461 Experiment Tasks
Tante
Resorrch Quection
12737 Granular Sub-tasks
Sectiora
Formuts
Experiment Mellaal
Papers
Conclasion
Talloo
Figures
Classify task
Script exists
i Interpret Document
Extract Exp. Details to answer the question
Docomentation exists
D
D
Implementation not found
README Tetriamm
Experiment Desigt Usage fnntraction imglementation
Execute & Verify P_
Code
b
Analyze Repo
LLM Human Check
Refine
feedback
Extract Experiment Implementation
Validate & Refinement
Figure 4: -XPBBench semi-automated dataset construction pipeline.
Stage 2.1: Extract Research Task. We begin by extracting the research task consisting of the research question, high-level methodology, and expected outcome-directly from the paper. This process is designed to handle the fact that key information in academic papers is often distributed across sections and conveyed implicitly. First, we index the PDF using OCR. For structured elements (tables, figures, headers), we use OCR to locate the visual regions and multimodal LLM interpret them (including cross-page elements) into structured text. This ensures downstream access to high- signal artifacts that may anchor the task definition. Next, we conduct multi-pass passs extraction. In the first pass, we perform retrieval-augmente These overarching questions are often not confined to a single paragraph and require stitching together dispersed cues. In the second pass, for each high-level takeaway, we apply semantic extraction at the subsection level, focusing on evaluation sections. We classify each subsection as either implementation context or candidate research question. Contextual passages are stored and reused each subsection independently the LLM generate
querying to identify broad, high-level research takeaways.
across subsequent prompts. This focused while conditioning accumulated context and extracted tables/fgures--hhlt more accurate and detailed task formulations. Finally, we refine each task through targeted re- querying of the full paper (including appendices) to recover any additional setup constraints or methodological details that were missed earlier. This step acknowledges that relevant setup details may be located far from the task description and ensures completeness for the extracted task.
Extract Experiment Implementation. Each extracted task is then passed to an im- plementation extraction AI agent (operating tool-augmented environment, with PDF reading, terminal access, and web browsing) to identify the specific implementation (chain of scripts) needed to address the research task. Our setting provides the agent with both a complete codebase and the extracted task, containing the research question, methodology, and expected outcome. This effectively reduces the problem to goal-conditioned search over the codebase, where the agent's task is to localise the implementation that realises the specified methodology and expected outcome. To do this, the agent explores the repository in an open-ended fashion, e.g., consulting documentation, and auxiliary scripts, to uncover domain-specifer requirements (e.g.. pretrained checkpoints). The extracted experiment execution ground truth will be fully based on existing scripts. The agent outputs (1) a list of required scripts and (2) high-level usage instructions describing how run them to complete the task. Once a candidate implementation is produced, it is executed in Stage 3. If the run fails, the pipeline iterates, allowing the agent refine or replace the implementation until working solution is found. The final validated script chain is then parsed by the agent via AST (Abstract Syntax Tree) tracing to extract step-by-step list of implementation requirements in natural language, which becomes the ground truth for evaluating implementation correctness. Finally, we incorporate additional contextual details (e.g., hyperparameters) sourced from the raw code (e.g., configuration files) or repository documents (e.g., README. md) to enhance the final task specification.
Stage 2.2:
Stage 3: Verification and Refinement. All tasks are validated and finalized in this stage. For each task, we first apply an LLM-based monitor ($4.I) to detect disallowed behaviours such as mocking or fabricating results. We then re-execute the agent's reproduction script in a clean Docker
5

---

Published as a conference paper at ICLR 2026
Table I: Average benchmark scores for various models when tested against various evaluation metrics. Popular Agents and LLMs perform poorly on EXP- --P-Bench, showcasing its difficulty.
Agent
Model
D
1
I.E
Allv
AII-Ev
#C
OpenHands
03-mini
18.4
20.3
15.0
2.9
21.0
1.4
0.5
420
OpenHands
Claude-3.7 Sonnet
16.0
35.0
33.2
14.9
13.4
0.7
0.4
235
OpenHands
Amazon Nova Pro
18.2
19.5
26.8
0.0
15.7
0.0
0.0
56
OpenHands
Claude-3.5 Haiku
20.6
26.2
9.3
1.3
13.8
0.0
0.0
237
OpenHands
DeepSeek R1
6.8
10.0
0.7
0.0
2.4
0.0
0.0
140
AterativeAgent
Claude-3.5 Haiku
6.4
20.6
25.2
5.4
2.2
0.0
0.0
111
IterativeAgent
Amazon Nova Pro
0.1
10.0
18.1
0.0
0.3
0.0
0.0
215
container, guaranteeing a from-scratch run of all dependencies and steps. These re-executed outputs compared against the expected conclusion from the original paper using L.M.. If validation fails, the task is returned to the previous stage for refinement. In cases where no usable script is found, the agent may produce an implementation, which is subjected the same checks. In all cases, lightweight human review finalizes the task, requiring only a cross-check of structured task content (already consolidated by the pipeline) against the source materials. This significantly reduces human burden compared to manual curation from scratch. We iteratively refined the construction pipeline through manual trial-and-error process (App. GG Following validation, each complete task added to the dataset along with a list of masked files (e.g., README. md, relevant scripts) to ensure agents cannot directly access answers. In our benchmark implementation, repositories are cloned afresh per agent, and masking is applied using scripted git operations, including recursive traversal of submodules. Masking ensures agents must reason over the task input, rather than rely on shortcut access to original solutions.
EVALUATION
4.1 EVALUATING LLM-BASED AGENTS PERFORMANCE: SETUP & MAIN RESULTS
Setup. We evaluate range of agents and LLMs used in related benchmarks (Chen al. 2021 against EXP-Bench. In terms of agents, we made use of Open Hands (a top-performing code generation agent) and IterativeAgent (as configured Starace al. (2025) reduce the likelihood of early task stopping), henceforth known as OH and IA, respectively. In terms of LLMs, these include the top-ranked
3.7, Haiku 3.5, Deepseek-RR (Wei al.. 2023) models, and OpenAl 03-mini variants. Each agent is run in an Ubuntu 24.04 Docker container, and given access to 4 x Nvidia A40 GPU, and clean working directory containing the masked GitHub repo of the paper (i.e., task-specific scripts removed), instructions, and relevant context (e.g., API credentials).
Evaluation Judge Implementation Details. Our evaluation framework consists of two main components used to assess agent performance across various metrics (refer to later sections, e.g., Table [ The first component is an LLM-based judge (using 03-mini), following prior work on LLM-as-a-judge (Zheng al. 2024 Liu et al. 2024b; vic Starace al. 2025). This judge operates through multiple steps: The process begins with an integrity check performed by a Monitor, which analyzes agent logs to detect disallowed behaviors (denoted metric M; Fig. (b)). Specifically, the monitor checks whether the agent: (1) accessed the research paper directly (e.g., opened the PDF), (2) performed Git operations such as checking commits switching branches, or (3) used fake, hardcoded, or placeholder data rather than generating results through real experimentation. If violations are found, the monitor also identifies possible causes (c.g., ethical refusals, runtime errors) using log information. Once integrity is established, the agent's 's experimental design, implementation. and conclusion are evaluated for conceptual soundness, completeness (e.g., inclusion of all required steps), and alignment with ground truth. These assessments yield scores for: D (design correctness, i.e., proportion of design criteria met), (implementation correctness, i.c., proportion of implementation components satisfied), and (conclusion correctness). The second component of our evaluation judge a Code Execution Validator, which runs the agent-generated code modifications in a clean and equivalent containerized environment. This step verifies whether the code is executable and produces expected outputs. This exccutability metric denoted as E. Implementation details including the system prompt are in App.

---

Published as a conference paper at ICLR 2026
Main Results. Table I presents average accuracy scores across all 461 tasks. I.E indicates whether an implementation is both appropriate for the experimental task and executable -a more comprchensive check of implementation quality. Allv denotes tasks that fully correct in terms of D, I, and C, while A V.V adds the executability requirement. #E represents the number of tasks per model that were execution-checked. Due the time-consuming nature of execution, only a subset of traces were discarded prior execution. Our top-ranked agents are OH+o3-mini, OH+3.7 Sonnet, and OH+Nova Pro, ranked via All - EV, with C used as tiebreaker. The worst- porst-performing model was IA+Nova Pro. Extended results by paper category are shown in T with full details in App. E Across both tables, we observe that models consistently score below 30% across all metrics, with the exception of the RL category, where several OH models achieve up to 241 % (averaged over 36 tasks) in terms of T. Notably, under stricter metrics such All v, performance drops sharply-e.g. OH+o3-mini scores only 1.4%. This underscores the value of including partial metrics that assess individual aspects, allowing credit for partially correct answers and supporting more nuanced evaluation.
those that failed the monitor check, which were automatically
4.2 DETAILED ANALYSIS
Cost-Time Analysis. Fig- 6a shows the average cost (in USD) and time (in minutes) per task across different agent configurations. Cost reflects only the token usage of the backbone LLM (input/output), excluding agent internal LLM-API usage or compute consumption. The number in parentheses next to each legend entry indicates the model's performance rank, based on average correctness. Each agent was allowed a maximum of 40 minutes per task, though this limit can be easily adjusted. Notably, IA models often consumed the full allotted time, rarely stopping early. In contrast, early stopping was common with OH models. For example, the relative time difference between Nova and Haiku is larger under OH than IA, reflecting differing usage patters. These trends consistent with our earlier observations: OH models often produced plausible responses without actually running the experiment, leading to high partial scores (c.g., design, implementation), while IA models tended to run longer but less effectively. Interestingly, we found little correlation between runtime/cost and overall performance. OH+o3-mini (rank 1) achieves the best trade-off with low cost and moderate time. OH+3.7 Sonnet (rank 2) performs well but is the slowest and most expensive. The full cost-time distribution is provided in App. D.2]
Conjunctive Evaluation Metrics Substantially Lower Agent Scores. We analyze only the subset
of tasks for which execution was run, visualize how progressively applying stricter evaluation
criteria impacts agent scores. As shown in Fig. 6bb (with full results in App. J.Ip applying only the
initial monitoring check (M) yields an average score of 20.6%. Adding design (D) and conclusion (C)
correctness criteria reduces the score sharply to 3.7%. Incorporating implementation correctness (I)
further lowers the score to 0.4%, and including execution verification (E) results in a final accuracy of just 0.2%. These findings highlight how conjunctive evaluation surfaces brittleness in end-to-end experimental correctness.
Metric Stability Analysis. As shown in Fig. certain individual metrics such as C and exhibit high variance. This variance arises for different reasons: for C, agents can produce plausible but unfounded conclusions without valid experimental foundation; for E, even incorrect mock implementations may successfully execute, introduc- ing overestimation bias. To mitigate such inconsistencies, we adopt compositional scoring via conjunctive metrics such as C- D and I E. which combine correctness across multiple dimensions. These conjunctive forms substan- tially reduce score variability, producing more reliable signals of agent performance. For example, C- filters out conclusions not grounded valid design plans, and I-E E discounts executions that do not fulfill setup require- ments. This demonstrates that conjunctive metrics can temper over-crediting and reduce sensitivity annotation leniency or spurious correctness-ttee thhrrby offering a more stable and discrininative evaluation.
Average Score (%)
30-
20
10
I-E
Judge Metric
Figure 5: Stability Analysis.
7

---

Published as a conference paper at ICLR 2026
10
OH 03-3050
(2)
40
Qlf 3-T SONNRY
Average Score (%)
OPENILANDS G3-MINI OPENHANDS + CLAUDE- 3.7 SONNET
Average Cosl (8)
OHI Nova
3.5 HARRC
30
OH DearSSan
OPENILANDS + AMAZON NOVA PRO
Novx Pso
20-
OPENTIANDS CLAUDE-3.5 HAIEU
4-
10
90
30
M
MCD
MCDI
MCDIE
Average Time (minutes)
Judge Metric
(a) Cost time raads across agents.
(b) Strieter metrics reveal lower true correctness.
Figure 6: Ablation of agent performance along cost-time and evaluation metrics.
Contamination. To probe potential leakage, we adopt a technique used by existing memorization studies (Ramos et al. 2025 Liang al., 2025), which found that if models still succeed despite missing input context, it is evidence of memorized knowledge. Specifically, we experimented with under-specified contexts by randomly omitting half of the setup method. Under these ablations, models consistently failed (<1% score). This sharp performance drop suggests the tasks cannot be solved via recall alone. Morcover, memorization is unlikely our benchmark, as it is constructed from recent, niche research papers/codebases (a stance also adopted in contemporary benchmarks (Starace) al., 2025 /Zhao et al., (025)). Furthermore, our tasks exhibit structural resistance to memorization: they are drawn from fragmented sources spanning papers, code, and supplementary materials, and require both iterative agent-led and manual reconstruction that is not easily recoverable from pretraining data. Finally, our semi-automated dataset construction pipeline enables the continual addition of tasks from newly released papers, further reducing the risk of contamination over time.
4.3 ANALYSIS ON PREVALENT AGENT FAILURE PATTERNS
Pattern Extraction Methodology. Our analysis followed a two-pass, open-ended process. During evaluation, each metric score was accompanied by an error analysis, derived from implementation logs (e.g., stderr) comparisons against ground truth. In the first pass, we extracted high-level, domain-specific insights from these earlier error analyses, sess, across phases for all agent-task pairs. In the second pass, we iteratively grouped these insights into distinct failure types, assigning each to an existing category or creating new one if needed. This process produced 3,238 raw insights, which we distilled into 361 unique failure types. We present representative and simplified subset of these condensed errors in Table 2 (full details can be found in App. H)
Analysis. To better understand where agents fail, we analyzed error traces and categorized them into representative failure types across four key phases of experimentation: implementation, execu- tion, design, and conclusion. As shown in Table 2 the most prevalent issues emerged during the implementation phase. with 39.7 1% of failures stemming from missing essential components. In several cases, agents failed to include critical elements such as semantic retrieval strategies (e.g., EniXcoder-H21. and Enixcoder-LL2He validation functions for filtering questions (e.g., using GPT- 3,5), or robustness-enhancing techniques like Mixus xup, CutMi x, and Label the experimental implementation's validity. In the execution phase. failures were most commonly due to environment or dependency misconfiguration (e.g., STORM not registered in jaxmarl) or absent core libraries like PyTorch and F flax, which led to model loading failures. Script-level issues (23.5 (3.84%) included unrecognized model names (e.g.. aaggne__tinn not found in i mm) and missing checkpoint files, causing runtime or I/O errors. These examples highlight persistent reproducibility challenges even when a correct implementation structure is in place. Design-related failures were also frequent, with 16.05% involving incomplete or misclassified experimental variables, and 7.62% reflecting extraneous procedural additions such as inclusion of a Re backbone or arbitrary hyperparameter knobs not specified in the ground truth. These design errors suggest that agents often fail distinguish between essential experi- mental factors and implementation noise. Finally, conclusion phase errors highlight limitations in agents' interpretive reasoning. The most common issue (26.18%) was missing underdeveloped
(29.38 (2.38%) such as missing critical environments
8

---

Published as a conference paper at ICLR 2026
Table Agents fail in diverse ways across different phases of experimentation; this table presents simplified subset of common examples, measured across all agent and model evaluations.
Phase
Failure Type
Prevalence (%)
Design
Incomplete or Misclassified Design Variables Irrelevant Procedural Additions in Design
16.05
Design
7.62
Implementation
Missing Essential implementation Components Incomplete Evaluation Metric Implementation Incomplete Data and Preprocessing Setup
39.71
Implementation
2.15
Implementation
1.83
Execution
Environmenn/Deeendeney Configuration Errors
29.38
Execution
Execution Script and File Errors
23.84
Execution
Missing Setup Script File
6.95
Execution
Tensor Operation Execution Error
3.22
Conclusion
Missing Conclusion Content
26.18
Conclusion
Incorrect Conclusion Interpretation
19.66
Conclusion
Extraneous Details in Conclusion
7.77
Conclusion
Incorrect Numeric Conclusion
3.21
Table Average benchmark scores of various models and agents across select task categories; see App. D for complete list. Evaluation performed against EXP-Bench.
Category
Agent
Model
D
E
I-E
C
Allv
Allv -E
Applications
OH
Nova Pro
19.2
23.9
19.0
0.0
13.9
Applications
OH
03-mini
9.0
8.0
0.0
0.0
8.3
0.0
0.0
Applications
OH
3.5 Haiku
19.2
24.5
8.3
5.6
8.3
0.0
0.0
Applications
OH
3.7 Sonnet
9.0
26.8
30.8
7.7
8.3
2.8
0.0
Applications
IA
Nova Pro
0.0
9.8
5.0
0.0
0.0
0.0
0.0
RL
OH
3.7 Sonnet
18.3
48.2
27.3
21.2
17.6
2.0
3.0
RL
OH
03-mini
23.5
34.8
15.7
2.0
27.5
3.9
0.0
RL
OH
3.5 Haíku
27.7
41.4
11.5
0.0
17.6
RL
IA
3.5 Haiku
3.3
27.5
17.4
0.0
2.4
0.0
0.0
RI.
OH
DeepSeek
5.0
10.3
0.0
0.0
2.0
0.0
conclusions-fit for instance, omitting detailed comparisons between PPO and Q-Learn i ng train- ing time and normalized scores, or neglecting specific numerical gains (e-g.. 1.25% improvements across ARC-Challes Lingee and OpenBook(A). Another frequent error (19.66%) terpretation, such as claiming sodaamard-enhaneee INT4 inference improves performance without substantiating comparisons baseline INT4. Together, these findings emphasize the importance of phase-specific evaluation to uncover deeper breakdowns in experimental reasoning.
incorrect in-
DISCUSSION
Limitations. EXP-Bench primarily focuses the experimentation procedure, from designing experiments for a given research question to deriving conclusions. The broader Al research lifecycle encompasses other critical stages such as identifying gaps through literature review, the initial unstructured ideation of research questions, and navigating the complex, iterative, and unpredictable path of real-world scientific discovery, which are not yet captured by the current task structures. In particular, EXP-Bench does not evaluate open-ended scientific creativity alternative experimental designs; instead, it targets faithful execution of the specific, procedures extracted from the source materials.
Future directions. Future work will focus on enhancing Al agents' ability to automate research experimentation using supervision from EXP- 's dataset. One promising direction is to apply reinforcement learning with verifiable rewards, enabling agents to autonomously navigate the research lifecycle and accelerate scientific discovery.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We introduced EXP-Bench, a novel benchmark designed to rigorously evaluate and guide the development of Al agents in conducting end-to-end AI research experimentation. By sourcing tasks from influential peer-reviewed publications and their accompanying codebases, and utilizing semi-automated curation pipeline, EXP-Bench presents agents with realistic, fine-grained challenges in end-to-end AI research workflow including experimental design, implementation, execution, and conclusion derivation. Our initial evaluations with leading agents reveal significant bottlenecks in conceptualizing complex experiments and ensuring robust code implementation and execution. EXP-Bench therefore serves not only as a comprehensive evaluation tool but also as a valuable dataset guide future Al agents to act step by step, ultimately accelerating Al research.
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their insightful feedback. This work partially supported by VMware Early Career Faculty Grant, Cisco grant, and NSF grants CNS-1942219, CNS-2106751, CNS-2107147, and CNS-2214272.
8
REPRODUCIBILITY STATEMENT
All code and data supporting our work are available at https://github com/ httee//mainneechhaakkkk. exp bench Details of the research
Ctarieous/Curiee//
papers used in constructing our dataset are included in App. B The benchmark curation pipeline is described of the main paper and further elaborated in App. G The evaluation judge pipeline and prompts are provided in I and App.[ respectively. Our agent failure patterns extraction pipeline described $4.3 Finally, additional evaluation results and analyses are provided the other sections of the Appendix. Together, these resources are intended to ensure the reproducibility of our dataset, benchmark, and experimental findings.
REFERENCES
Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.
https://
Imsys. ottg://log//22230000---------m
Ferran Alet, Javier Lopez-Contreras, James Koppel, Maxwell Nye, Armando Solar-Lezama, Tomas Lozano-Perez, Leslie Kaelbling, and Joshua Tenenbaum. large-scale benchmark for few-shot program induction and synthesis. In International Conference on Machine Learning, pp. 175-186.
PMLR, 2021.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-rree 4-bit inference in rotated Ilms, 2024.
arwiv. www.wiwwwwwwwaaa...aooa.aaa /bwp/////b//////////////// 20446
Jinheon Back, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity, 2023.
URL httl:tts::// htt.:htts://aaxiiie http://axxvvvoo wtt.or/raaaa..........// 2302. 04023]
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on ""a is b" fail to learn ""b is a"" 2024. URL
hittps://arat org/ www./aa.////oooooooo/oo.o/oooaea / 2309. 12288
Matteo Bettini, Amanda Prorok, and Vincent Moens.
Benchmarl: Benchmarking multi-agent
reinforcement learning, 2024. htw.:httsss//rrx...oo////aa/////
10

---

Published as a conference paper at ICLR 2026
Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570-5777 -777700-7777777 2023.
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles,
Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-basee common knowledge work tasks, 2025, URL
attps://arxiiae wrg. abs 22070052770021111
Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang. Gianni De Fabritiis, and Vincent Moens. Torchrl: A data-driven decision- maaciign-mmking library for pytorch, 2023. /tLhttts::/ /ttp/aaxiiv...///.i//eee/eaaevl/ org/ www./rgg///////o.oe//..//.eeeeee /bbb//366.. 00577
Jaroslaw Blasiok and Preetum Nakkiran. Smooth ece: Principled reliability diagrams via kernel smoothing, 2023. URt /tttttpss//aaxii.ooor/////////
12236
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. Mle- bench: Evaluating machine learning agents on machine learning engineering. 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2400005000, 2024.
Ching-An Cheng, Allen Nie, and Adith Swaminathan. Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and Ilms, 2024. URL
oxg/aba/2406. 16218
Pengyu Cheng, Tianhoo Hu, Han Xu, Zhisong Zhang, Zheng Yuan, Yong Dai, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances Ilm reasoning, 2025. URL
https:://arxie www./o///aaoo. abs/ /2444 10642
Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodríguez Fanlo, Simon Frieder, Simon Machado, Akshara Prab- hakar, Ellie Thieu, Jiachen T. Wang, Zirui Wang, Xindi Wu, Mengzhou Xia, Wenhan Xia, Jiatong Yu, Jun-Jie Zhu, Zhiyong Jason Ren, Sanjeev Arora, and Danqi Chen. Language models as science tutors, 2024. URL hittps://arxiv www.w.www.........o...........ot abs 240.22402220 11111
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. httttp:://axxi..rrrr.t
abs/2110. 14168
The Multimodal Universe Collaboration, Jeroen Audenaert, Micah Bowles, Benjamin M. Boyd, David Chemaly, Brian Cherinka, Ioana Ciucă, Miles Cranmer, Aaron Do. Matthew Grayling, Erin E. Hayes, Tom Hehir, Shirley Ho, Mare Huertas- Chuerta-CCompnn, Kartheik G. Iyer, Maja Jablonska, Francois Lanusse, Henry W. Leung, Kaisey Mandel, Juan Rafael Martinez-Galarza, Peter Melchior, Lucas Meyer, Liam H. Parker, Helen Qu, Jeff Shen, Michael J. Smith, Connor Stone, Mike Walmsley, and John F. Wu. The multimodal universe: Enabling large- scale machine learning with 100tb of astronomical scientific data, 2024. http:::: http:/awwwiaaaaaaaaaaaa 22.... 02527
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rihf: Safe reinforcement learning from human feedback, 2023. URL https://
arxiv.org/ www.///... 12773
Tao Dai, Beiliang Wu, Peiyuan Liu, Naiqi Li, Jigang Bao, Yong Jiang, and Shu-Tao Xia. Periodicity decoupling framework for long-term series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao. Segvol: Universal and interactive volumetric medical image segmentation, 2025. URL http:httts///m pttttss://axxiv.oott http:///awwvvv.oo//////311/11 13385
11

---

Published as a conference paper at ICLR 2026
Elicit. Elicit: Analyze research papers superhuman speed, 2025. URL wwt.ttt/eelci...coo Predirected-tzue Accessed: 2025-05-12.
Mohamed Elrefaie, Florin Morar, Angela Dai, and Faez Ahmed. Drivaernet++: large-scale multi- modal car dataset with computational fluid dynamics simulations and deep learning benchmarks, 2025.
htwpiivwwrggaaaoe /tt///ba//44.../ 09624
Yin Fang, Ningyu Zhang, Zhuo Chen, Lingbing Guo, Xiaohui Fan, and Huajun Chen, Domain- agnostic molecular generation with chemical feedback, 2024. URL httph::/p//// / /ttp:///axxivoooooe
abs 2bbs/2001.10 11259
Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 177699277744 Curran Associates, Inc., 2023.  httpa://ppoeeecinss.eeee......pp cc wttpe/eerreeeeeeeeeeeeeeeeeeeeee files/ /ttp://pppp...aaaaap////////////
URL
581 0888899944555433333320--------00
aan.raess__aad_eeeeeahhaeeee Benchmarks.
pdf
Kanishk Gandhi, Michael Y. Li, Lyle Goodyear, Louise Li, Aditi Bhaskar, Mohammed Zaman, and
Noah D. Goodman. Boxinggym: Benchmarking progress in automated experimental design and model discovery, 2025. URL https::/arxii..ortttt a2500//1.//////11.../gg 01540
Gaël Gendron, Qiming Bao, Michael Withrock, and Gillian Dobbie. Large language models are not strong abstract reasoners. arXiv preprint arXiv: 2023.
Alireza Ghafarollahi and Markus J. Buehler. Sciagents: agent intelligent graph reasoning, 2024. UtL/itttss///rrrii.or/////////// 05556
through multi-
Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio lacobacci, A Abdelhakim Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level. arXiv preprini arXiv:2411.03560 2024.
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, et al. Blade: Benehmarking language model agents for data-driven science. arXiv preprint 24Ni:22088096661 2024.
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. arXiv preprint arXiv:2402. 17453, 2024a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized texa-to-image diffusion models without specific tuning, 2024b.
Simon Jerome Han, Keith J Ransom, Andrew Perfors, and Charles Kemp. Inductive reasoning in humans and large language models. Cognitive Systems Research, 83:101155, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://
arxiv. mrxiv.org/abs/200000 03300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL httph:tttt//aaa
org/ wttp:/v...rg///////10000 03874
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. Infiagent-dabente Evaluating agents on data analysis tasks. arxiv preprint arXiv:2401.05550 2024.
12

---

Published as a conference paper at ICLR 2026
Jen-tse Huang, Wenxuan Wang. Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael Lyu. On the humanity of conversational ai: Evaluating the psycho- logical portrayal of llms. In The Twelfth International Conference on Learning Representations, 2023a.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 2023b.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 03302
Zilong Huang. Qinghao Ye, Bingyi Kang, Jiashi Feng, and Haoqi Fan. Classification done right for vision-language pre-training, 2024b.
htlp://bb/22411..000 03313
Tal Ifargan, Lukas Hafner, Maor Kern, Ori Alcalay, and Roy Kishony. Autonomous llm-driven research from data to verfflverifiable research papers. NEJM AI, 2(1):Alo2200555, 2025.
Guillaume Jaume, Paul Doucet, Andrew H. Song. Ming Y. Lu, Cristina Almagro-Pérez Sophia J. Wagner, Anurag J. Vaidya, Richard J. Chen, Drew F. K. Williamson, Ahrong Kim, and Faisal Mahmood. Hest- Ik: dataset for spatial transcriptomics and histology image analysis, 2024.
http:ttps:/ / r/a rxiv www./o/gg//.//.o//////.//...eess /40 6192
Bowen Jing, Hannes Stärk, Tommi Jaakkola, and Bonnie Berger. Generative modeling of molecular dynamics trajectories, 2024a. URL http:://axxxxii wttp://wwwrgg/b////4..ooo
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts?, 2024b.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583- -898883----998 2021.
Patrick Tser Jemn Kon, Jiachen Liu, Qiuyi Ding, Yiming Qiu, Zhenning Yang. Yibo Huang, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, and Ang Chen. Curie: Toward rigorous and automated scientific experimentation with ai agents, 2025. URL
abs/
16069
Jakub I Lála, Odhran O' D'Donoguue, Aleksandar Shtedritski, Sam Cox, Samuel G Rodriques, and Andrew D White. Paperqa: Retrieval-augmeened generative agent for scientific research. arXiv preprint arXiv:2312.07559, 2023.
Jon M. Laurent, Joseph D. Janizek, Nitya Thakkar, Michael Ruzo, Michael S. Yao, Michael S. Levine, Samuel G. Rodriques, and Andrew White. Lab- Measuring capabilities of language models for biology research, 2024. URL BRLhhtts::/ /ttp///aavvvvvv. org/ absp/4/..2242... 10362]
Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. MIr-copilot: Autonomous machine learning research based on large language models agents. arXiv preprint arXiv:2408. 14033, 2024a.
Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Z. Li. Moganet: Multi-order gated aggregation network, 2024b. URL https
/ 03295
Xurui Li, Ziming Huang, Feng Xue, and Yu Zhou. Muse: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images, 2024c. URL httpss://axxi
hxg/abb/24001.166
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624): 1097- 2022.
Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. The swe-bench illusion:
When state-of-the-art lims remember instead of reason. arXiv preprint arXiv:2506. /2286, 2025.
13

---

Published as a conference paper at ICLR 2026
Shengsheng Lin, Weiwei Lin, Xinyi Hu, Wentai Wu, Ruichao Mo, and Haocheng Zhong. Cyclenet: Enhancing time series forecasting through modeling periodic patterns, 2024. URL https?
7/arxiv. org/ wwwp///..... 18479
Haochen Liu, Li Chen, Yu Qiao, Chen Lv, and Hongyang Li. Reasoning multi-agent behavioral topology for interactive autonomous driving, 2024a. h 18031
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmaaking code
auto-completion systems, 2023a.
03091
Xi Liu, Chaoyi Zhou, and Siyu Huang. 3dgs-enhancer: Enhancing unbounded 3d gaussian splatring with view-consistent 2d diffusion priors, 2024c. http://tss//axx...rrr///////////
16266
Xiao Liu, Hao Yu, Hanchen Zhang. Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating ilms as agents, 2023b. URL hattp::/arxiv.orre http/arxiv..o org/orgg/.......o..
2308. 03368
Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang. Jihyun Janice Ahn, Hongchao Fang. Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, and Wenpeng Yin. Aaar-1.0 1.0: Assessing ai's potential to assist research, 2024. URL http::ss///xxxv..rpr.e
22394
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06299 2024a.
Yifan Lu, Yue Hu, Yiqi Zhong, Dequan Wang, Yanfeng Wang, and Siheng Chen. An extensible framework for open heterogeneous collaborative perception, 2024b. URL httph:tttt//tte
org/abs/ 240..2411..00 13964
Qijun Luo, Hengxu Yu, and Xiao Li. Badam: memory efficient effficien full parameter optimization method 02827
for large language models, 2024. URL
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725. 2024.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.047221 2023.
Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: comprehensive benchmark for Ilmebased agents in computational biology. arXiv preprini arXiv:2503.0000 2025.
Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptare benchmark: Evaluating understanding and generalization in the are domain. arXiv preprint arXiv:2305.07114. 2023.
Siddharth Narayanan, James D Braza, Ryan-Rhys Griffiths, Manu Ponnapati, Albert Bou, Jon Laurent, Ori Kabeli, Geemi Wellawatte, Sam Cox, Samuel G Rodriques, et al. Aviary: training language agents on challenging scientific tasks. arXiv preprint arXiv:2412.21154. 2024.
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gauray Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral. Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Mlgym: A new framework and benchmark for advancing ai research agents, 2025. URL htt..httss:////
org/ abs htw.g///a////...../ 22000 4499
14

---

Published as a conference paper at ICLR 2026
Ruben Ohana, Michael McCabe, Lucas Meyer, Rudy Morel, Fruzsina J. Agoes, Miguel Beneitez, Mar- sha Berger, Blakesley Burkhart, Keaton Burns, Stuart B. Dalziel, Drummond B. Fielding, Daniel Fortunato, Jared A. Goldberg, Keiya Hirashima, Yan-Fei Jiang, Rich R. Kerswell, Suryanarayana Maddu, Jonah Miller, Payel Mukhopadhyay, Stefan S. Nixon, Jeff Shen, Romain Watteaux, Bruno Régaldo-Saint Blancard, François Rozet, Liam H. Parker, Miles Cranmer, and Shirley Ho. The well: a large-scale collection of diverse physics simulations for machine learning, 2025. URL https: /ttp///aavvv.v/.ae/eaieeaaeaaaia org/abs 2412. 00568
Penghui Qi, Xinyi Wan, Guangxing Huang. and Min Lin. Zero bubble (almost) pipeline parallelism. In The Twelfth International Conference on Learning Representations, 2024a.
Siyuan Qi. Shuo Chen, Yexin Li, Xiangyu Kong, Junqi Wang, Bangcheng Yang, Pring Wong, Yifan Zhong, Xiaoyuan Zhang, Zhaowei Zhang, Nian Liu, Wei Wang, Yaodong Yang, and Song-Chun Zhu. Civrealm: A learning and reasoning odyssey in civilization for decision-makinn agents, 2024b. URL https://axxxii...... org/ abs / 240.
Daniel Ramos, Claudia Mamede, Kush Jain, Paulo Canelas, Catarina Gamboa, and Claire Le Goues. Are large language models memorizing bug benchmarks? In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LL.M4odde), pp. 1-8. IEEE, 2025.
Jie Ren, Xidong Feng, Bo Liu, Xuehai Pan, Yao Fu, Luo Mai, and Yaodong Yang. Torchopt: An efficient library for differentiable optimization, 2022. URL http:: oww.//ii.orrarreoooe.e abs/
2211. 06934]
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar In- gvarsson, Timon Willi, Ravi Hammond, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktaschel, Chris Lu, and Jakob Nico- laus Foerster. Jaxmarl: Multi-agent rl environments and algorithms in jax, 2024. URL
https::/aara org/ abss//bb/////
Samuel Schmidgall, Yusheng Su, Ze Wang. Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants. arXiv preprint M.Xiv:2501.04227 2025.
Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core- bench: Fostering the credibility of published research through computational reproducibility agent benchmark. arXiv preprint arXiv:2409. 11 2024.
Giulio Starace, Oliver Jaffe. Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint 1rxXiv:504.018444 2025.
Liangtai Sun. Yang Han, Zihan Zhao, Da Ma. Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific re- search. Proceedings of the AAA/ Conference Artificial Intelligence, 38(17) 19061, http://aaaaiii11.1..3 AAAI htl.//aticle/viei....
Mar. 2024a. doi:
URL aww...aaaaiaooo............ weeeg//nnex... php/
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents, 2024b. URL
09542
Kyle Swanson, Wesley Wu, Nash L. Bulaong, John Pak, and James Zou. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation. bioRviv, pp. 2024-11, 2024
Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, and Ying-Cong Chen. Hawk: Learning to understand open-world video anomalies, 2024.
URL /aattps//aaxiv...o org. abs 2005 16886
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret,
15

---

Published as a conference paper at ICLR 2026
Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, and Hao Peng. Scicode: A research coding benchmark curated by scientists, 2024.
Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent Il framework for full-pipeline automl. arXiv preprint arXiv:2410.02958, 2024.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models, 2024. URL
http://www..o//aa///....1.... 10491
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. Scimon: Scientific inspiration machines opti-
mized for novelty. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 279-299, 2024a.
Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack, 2024b. URL https: ////xxii/// v. org/
abs/2406. 07230
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific pwwblem-solvaeeeeeeeee solving abilities of large language models, 2024c. URL (Rt/httss///axx
erg/abs/2307 10635
Yifan Wang, Di Huang, Weicai Ye, Guofeng Zhang, Wanli Ouyang, and Tong He. Neurodin:
A two-stage framework for high-fidelity neural surface reconstruction, 2024d. URL https
7/ //axxiv.org/abs/20000
10178
Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 126,1516,, 2023.
Yuxiang Wei, Zhe Wang. Jiawei Liu, Yifeng Ding, and Lingming Zhang, Magicoder: Source code is all you need. arXiv preprint arXiv:23/2.02/ 2023.
Wei- Wei-Hung Weng et al. Towards an ai co-scientist, 2025.
Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. Re-bench: Evaluating frontier ai rd capabilities of language model agents against human experts, 2024. URL https: arxiv. org/ abs /211 15114
Dongming Wu, Jiahao Chang, Fan Jia, Yingfei Liu, Tiancai Wang. and Jianbing Shen. Topomlp: A simple yet strong pipeline for driving topology reasoning, 2023. URL htttps://axivv.r org/
abs/23100 2310. 063
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction, 2024. URL
https:://rxii www//bs/////oooooooooo..oo..oarr 23100.1103
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long extrapolaionn for Ilms with an efficient context memory, 2024. httsttts::////// /tt.tps:/aaxii...oo wttp:/wwwwwvv..aa///aaaaaa
04617
Tinghao Xie, Xiangyu Qi, Ping He, Yiming Li, Jiachen T. Wang, and Prateek Mittal. Badexpert: Extracting backdoor functionality for accurate backdoor input detection, 2023. URL https
/208 12439
Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reasoners? a comprehensive evaluation and beyond. IEEE Transactions on Knowledge and Data Engineering, 2025.
Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B Khalil. Lims and the abstrac- tion and reasoning corpus: Successes, failures, and the importance of object-based based representations. arXiv preprint ar0iv:2305.1m /8354, 2023.
16

---

Published as a conference paper at ICLR 2026
Zhao Xu, Fan Liu, and Hao Liu. Bag of tricks: Benchmarking of jailbreak attacks llms, 2024. URL httpt:tt///atiitt/ org/ abs/ /ae//240666 09324
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V. Le, Denny Zhou, and Xinyun www.www...gggbbo......
Chen. Large language models as optimizers, 2024a. URL
03409
Ling Yang. Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E. Gonzalez, and Bin Cui. Buffer of thoughts: Thoueet--uggmented reasoning with large language models, 2024b. URL
absaabbs/225/ 04271
Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. Language models inductive reasoners. arXiv preprint arXiv:2212.1092 2022.
Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. Large language models for automated open-domain scientific hypotheses discovery. arXiv preprint arXiv:2309.02726, 2023.
Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang. Zengrui Jin, Long Lin, and Daniel Povey. Zipformer: A faster and better encoder for automatic speech recognition, 2024. URL https: /ttpss:/aaxiv..rrg// abbsg///b///. hbppr///abb//33..... 23100
12330
Jiakang Yuan, Xiangchao Yan, Botian Shi, Tao Chen, Wanli Ouyang, Bo Zhang, Lei Bai, Yu Qiao, and Bowen Zhou. Dolphin: Closed-loop open-ended auto-research through thinking, practice, and feedback, 2025. URL
03916
Vinicius Zambaldi, David La, Alexander E Chu, Harshnira Patani, Amy E Danson, Tristan OC Kwan, Thomas Frerix, Rosalia G Schneider, David Saxton, Ashok Thillaisundaram, et al. De novo design of high-affinity protein binders with alphaproteo. arXiv preprint AXXv:2449.08022 2024.
Guowen Zhang, Lue Fan, Chenhang He, Zhen Lei, Zhaoxiang Zhang, and Lei Zhang. Voxel mamba: Group-free state space models for point cloud based 3d object detection, 2024a. URL
10700
Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. Micopilot: Unleashing the power of large language models in solving machine learning tasks, 2024b. URL https
//arxiv.org/alt abwp ab30 14999
Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. Automl gpt: Au- 0tt1//25550000999
tomatic machine learning with gpt, 2023. URL
Xuan Zhang, Jacob Helwig, Yuchao Lin, Yaochen Xie, Cong Fu, Stephan Wojtowytsch, and Shuiwang Ji. Sinenet: Learning temporal dynamics in time- dependent partial differential equations, 2024c.
URL httphhtsss///xxii..rrrrt abs/ /20...2433.. 19507
Bingehen Zhao, Despoina Magka, Minqi Jiang, Xian Li, Roberta Raileanu, Tatiana Shavrina. Jean- Christophe Gagnon-Audet Kelvin Niu, Shagun Sodhani, Michael Shvartsman, et al. The au- tomated Ilm speedrunning benchmark: Reproducing nanogpt improvements. arXiv preprint arXiv:2506.222 22449, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Yujia Zheng, Biwei Huang. Wei Chen, Joseph Ramsey, Mingming Gong, Ruichu Cai, Shohei Shimizu, Peter Spirtes, and Kun Zhang. Causal-leam: Causal discovery in python, 2023. URL
https://erxit org als/ 2307. 16405
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection, 2025. URL https: aatpaaxxivvvvo.... 231 18961
17

---

Published as a conference paper at ICLR 2026
Xingkui Zhu, Yiran Guan, Dingkang Liang, Yuchao Chen, Yuliang Liu, and Xiang Bai. Moe jetpack: From dense checkpoints to adaptive mixture of experts for vision tasks, 2024a. URL
httts::/aaxiv>: www./ogg//s.... abs 2406. 04801
Zhaowei Zhu, Jialu Wang, Hao Cheng, and Yang Liu. Unmasking and improving data credibility: A study with datasets for training harmless language models, 2024b. URL https: //arxiv
axg/abs/2311. 11202
18

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORKS
LLMs for scientific discovery. Many methods have adopted LLMs to generate novel hypotheses for common scientific discovery. For example, Back al. (Baek al. 2024). Wang al. (Wang
ct al. 2024a), and Yang et al. (Yang et al. 2023) developed approaches for generating innovative doman-specific research ideas. Going beyond domain-specifin ideas, a line of work also focuses on
generate hypothesis with LLMs in the commonsense domains (Gendron et al. | Moskvichey
ctt al. 2023 Yang et al.) 2022: Weng et
2025
Mirchandani et al.) 2023 Webb et al. 2023]
Alet et al. 2021 Xu et al., 2023 Han et al.
2024
Xu et 2025). Moreover, prior research on
automated scientific discovery proposes to combine hypothesis with -.a--istt code generation
for end-to-end workflows (Li et al. 22024a] fargan et al. 2025 Majumder et al.) 2024). While these
efforts works on various stages of the scientific lifecycle, experimentation critical, rigor- aspect-remains underexplored.
Some existing research explores building an automated scientific discovery workflow with rigorous validation using Al agents (Lu et al.) [0244a] Weng al.) 2025 Jumper et al. 2021] bambaldi et al.
2024| Schmidgall et al. 2025 Boiko et al. [003] [Sanson et al. 2024 Yuan et al. | Ghafarollahi
& Buehler 2024), they often either have limited automated evaluation or rely on domain-specific ad-hoc prompting optimizations to guide predefined workflows, struggling with the complexities of rigorous end-to-end experimentation automate AI research. Particularly, Lu et al. (Lu introduced a fully automated system called "The AI Scientist" to conduct research by collaborating with multiple L.MM agents. These agents handle the full research process, from defining research problems and reviewing related literature to synthesizing and executing experiments. However, their solution has limited automated evaluation with a focus on commonsense domains. Gottweis et al. (Weng et al. 2025) proposed an AT Co-scientist built Gemini 2.0, aiming at building a helpful AI collaborator for scientists. They focus on the scaling of the test-time compute paradigm to generate high-quality hypotheses and research proposals. While general purpose, the Al co-scientist is mainly validated in biomedical areas. Overall, these efforts often require experimental validation to follow constrained, framework-specific formats, resulting in extra overhead and hindering their usability.
2024a)
Benchmarks for domain-specific. AI agent tasks. A wide range of benchmarks have been developed to evaluate the capabilities of Al agents across diverse domains. Existing benchmarks predominantly
target problem-solving (Hendrycks et al. 2021b] Frieder et al.] 2023 Wang et al.] 2024c Sun
et al.
2024a Chevalier al.] 2024), logical reasoning (Cobbe al. [002] Hendrycks et al.) 2021a
Bang
et al. 2023 Lála et al. 2023). machine learning training (Huang et 20244a) Zhang et al.
2023
2024b Grosnit al.
2024 Trirat al., 2024, Mitchener al. 2025]] Narayanan
al.
2024
Gu et al. 2024 Guo et al.] and knowledge retrieval and analysis (Sun et al. 2024b
Hu et |0224 These benchmarks typically involve well-defined tasks with clear, deterministic
solutions, allowing for consistent and objective assessment of AI agent performance. By contrast, our proposed EXP-Bench focuses on experimentation for automating research, which requires a more rigorous and systematic approach beyond problem-solving. Experimental tasks demand iterative hypothesis refinement, complex experiment design/implemenntaion and execution, and rigorous result interpretation. Our benchmark captures these challenges by cemi-automaticaly evaluating AI agents on real-world experimentation tasks arising from influential Al research papers with high-impact open-source artifacts.
B EXTENDED DETAILS OF THE EXP-BENCH DATASET
In this section, provide full list of the papers in the EXP-Bench dataset, including source paper and, Al sub-domain. The complete dataset can be found in our repository https //huggingface.com wtw.aacc.coo/aaaaae wtt.://///aaaaaaaeeeee Just- Cwr.iuuuueeuuu -ecos//EePpBnnc
19

---

Published as a conference paper at ICLR 2026
Table 4: ICLR 2024 Papers
Key
Resource
Zipformer: and betler encodle for spoech recognition Yao et al. 2024
Deep Learning
propose an archi-
memory needed:
Atlention
32GB more recommented, GPU type: NVIDIA V100 or A100, GPU 2-8
Mechanisms
19033
Reversal Curse:
284
Deep Learning + Large Language Models
propose
OpenAl API key required: GPU:
000
LL trained "A is
archi-
fail to learn "B
lecture
memory: 216BB
A
Berglume al.
RAM
2024)
19044
AnimateDiff: Animate
11093
845
Deep Learning Generative Models
propose an archi- tecture
GPU type:
Your Personalized
NVIDIA; GPU
Texi-to Io-ug
GPU
Diffusion Midells
memorry: 13GB
without Specific
Tuning (Guo et al.
2024b
17666
BaDExpert:
165
Deep Learning + Robustness
Otber
GPUs with CUDA support (exact types not specified), AGB+ recommended memory
Extructing Backdoor
Functionality for
Accurate Backdoor
Input Detection Xie
al. 2023
19269
MuSe: Zero-Shot Industrial Anomall Classification and
358
26
Applications - Computer
propose training algorithm
memory needed: SGB; GPU type: NVIDIA; GPU
Segmentation
amount:
Mutual Scoring of the
Unlabeled Images
2024
Agnostic
Applications Chemistry
propose an archi- tecture
menuory:
Molecular Generation
RAM: GPU type: NVIDIA;
with Chemical
Feedhack Fang
2024)
18244
Periodicity
116
38
Deep Learning - Time Series
propose an archi- tecture
GPU: at least one (unspecified)
Decoupling
Framework For
Long term Series
Forecasting ct
2024)
18318
AnomalyCLIP: Object-agnostic
339
150
Deep Learning LLMs
propose an archi- tecture
memory: 24GB: GPU: RTX 3090;
Prompt Learning for
Zero-sho Anomaly
Detection al. 2025
19388
Unmasking and Ingroving Data
2706
Social Aspects Acconntaillit
Other
standard resources
GPU recommendedl
Credibility: Study
with Datasets for
Training
Language Models Zhu
ot al. 20244b
17776
RepoBench:
145
139
Deep Learning LLMs
propose brary
GPU: 1
6
Benchnarking
Repository Level
Code
Luto-Complction
Systems [Lu etal.
2022a
18013
Knowledge
Deep Learning LLMs
propose new ML applica-
GPUL:
Large Language Models Wan et
RAM: 32GB+
2024)
tion propose an archi- tecture
18865
Leaming Temporal Dynamics in
Applications Physics
(NVIDIA);
RAM: 16GB
PDLs Zhang al.
2024c
20

---

Published as a conference paper at ICLR 2026
Title
Stars
Cit.#
Domain
Key Dist.
Resource
T1 T2 T3
18447
MogaNet: Multi-order
220
Deep Learning
propose archi-
RAM: 216GB;
Gatod Aggregation
GNNs
GPUs:
Network et al
tecture
V10IA
19610
TopoMLP: Simple
179
Applications Rohotics
propose
GPUs: 1-2
yet Strong Pipeline for Driving Topology Reasoning Wu (Wu al.
archi-
2080+; RAM:
tocture
16GB
2028
17388
AgentBench:
2406
174
Deep Learning L.Mss
peopose dataset/li- brary
RAM: 1SGB; GPU:
Evalunting I.I.Ms as Agents al 20236
I; OpenAl API
18889
An Extensible
181
RL Multi-agent
propose new ML. applica- tion
RAM: 2 216BB GPUs: 2: CUDA
0
Framework for Open
Heterngenouus Collaborative Ptrception/Luceaa 2024b)
compatible
19128
CLIPSell: Vision
183
Applications Computer Vision
prupuse new MI. applica- tion propose dataset/li- brary
RAM: 8GB; GPU:
Transformer Dense
Prediction WWa
2024
18660
TorchRL:
2570
RL Deep RL
RAM: 28GB: GPUs recommeuded;; API access may be needed
data data-driven
docision-making
librury for
PoTocc
2022
19209
Large I Language Models as
506
829
Optimization Learning for Optimization
propose new ML
likely RAM-heavy; API keys roquired: GPL recommended
Optimizers Yang el al. 2024a)
applica- tion
18439
Smooth ECE:
139
Probabilistic Methods - Calibration
propose new MT, applica- tion
standard memory for Python scripts
Principled Reliability
Diagrams Kernel
Smeothing
BaaiiingBBBasse Blassok &
Nakkiram
17595
Zero Bubble (Almost)
344
Optimization Parallel
Other
memory: 16GB;
Pineline Parallelism
GPUs:
et al. 2024
18531
CivRealm: A Learning and
106
21
RL Multi-agent
propose
RAM: 8-16GB; GPU: I;
dataset/li-
Reasoning Odyssey
Freeciv web
for Agents Q
18540
Safe RLHF Reinforeemeen
1421
RL Safe
propose training algorithm
OpenAl API: GPUs: AB00-80GBB x8
Learning from Human
(Da et 2023
19008
On the Humanity of Conversational Al: Evaluating the Psychological Portrayal
110
58
Social Aspects Trustworthy ML
Other
RAM: 2 BGB;
2
OpenAl API: GPU
recommended
LLMs [Tmany
Table 5: NeurlPS 2024 Papers
Title
Stars
Cit.#
Domain
Key Dist.
Resource
T2 T3
93022
Generative Modeling of Molecular Dynamies Trajectorics Ding al.
144
Generative Models
propose
GPUs not specilied, Py Torch and related libraries suggest need for LUDA-compatille GPU. Memory requirements
+ New Approuches
an archi-
2024a
unspecified.
21

---

Published as a conference paper at ICLR 2026
Title
Stars
Cit.#
Domain
Key Dist.
Resource
T1
T2
T3
93431
Trace the Next
492
Optimization - Generative Models
propose archi- tecture
memory needed:
AutoDiff: Generarive Optimization with Rich Feeetback, Execution Traces, and
GB RAM minimum, OpenAl
key requized,
GPU: NVIDIA
LLMs Cheng et al.
2024
recommended. memory needed: Standard (depends on the dataset), GPU: Not required,
98316
Causal-leam: Causal
1287
Causality
propose
Discuvery in
an archi-
Python Thnng el 2023)
95333
EDGS-Enhancer: Enhancing Unbounded Gaussian Splatting
170
Computer Vision Video Generation
propose
memory needed: 16GB. Yes, GPU type: NVIDIA, GPU amount: 1,
an archi-
with now-coosistent
2D Diffusion
Priors Lim
98326
TorchOpt: Anl Elficient Librury for Dilleremtiable
570
Optimization - Zero-order and Black box Optimization
propose
memory needod: At least 8GB RAM,
archi- tecture
GPU type: NVIDIA, GPU
Optimization Ren ct 2022
amount: 1.
983
BenchMARL:
351
30
Reinforeement Learning + Multi Mulli-agent
propose dataset
memory needed:
4
Benchmarking Multi Agent
All least 8GB RAM recommendedd GPU: NVIDIA GPU GTX
Reinforcement
Leaming (ettini 2024
1080 or better).
95818
Classilication Done
200
Multimodal Medels
propose dataset
memory needed:
Right for
GB, GPU type:
Vision- Language
NVIDIA V100,
Pre Pre- Training Huuan
GPU amount: 1.
al. 2024h
95974
Reasoning
107
Reinforeement Learning Multi-agent
propose dataset
memory needed:
Multi-Agent
GB RAM minimum, GPU:
Behavioral Topology Interactive
NVIDIA GPU recommended,
Autonomous
Driving Liu 2024a)
97514
HEST-IK: Dotaset
236
Computational Biology
proposo dataset
memory necded:
For Spatial
2GB. GPU type: NVIDIA, GPU amount: I,
Transcriptoniccs
Histology Image
Analysis Jaurn et al.
2024
97649
JaxMARL:
Reinforcement Leaming - Multi-agent
propose dataset
memory
Multi Agent RL.
SGB RAM
Enviroaments and
minimum, GPU: NVIDIA GPU
Algorithms
JAX Rutherford et al.
(recommended)),
2024)
97713
WorkArena++:: Towards Compositional
164
Theory Reinforoement Learning and Planning
propose dataset
memory needod: SGB. GPU: 1.
Planning and
Reasooing-basee
Common Knowledge
Work Tasks Boisvert
ct.l. 2025
93219
HAWK: Learning to Understand
177
Computer Vision Vidco Understanding
propose dataset
memory needed:
Not specified, but requires nigh-performance GPUs for training.
Open World Video
Anomalies Tang et al. 2024
GPU: x RTX A6000 48G, memory At least 8GB, GPU type: NVIDIA GPU, GPU amount: more.
NeuRedin: Two-stage Framework High-Fidelity Neural Surface Recomstruction Wang 2024d
Computer + 3D Reconstraction
propose dataset
22

---

Published as a conference paper at ICLR 2026
Title
Stars
Cit.#
Domain
Key Dist.
Resource
T1 T2 T3
96264
Bufter Thoughts
608
Generative Models
propose
memory needed:
Thought-Augnen Augmented
Reasoning
archi-
16GB, True, GPU
Reasening with Large Language Modela Yang al. 20246)
tecture
type: NVIDIA,
amount:
96897
BAdam: A Momory
244
Optimization - Large Scale. Parailel und Distributed
propose dataset
memory needed:
6 3
Efficient Full
GB Llama 3- 3-8B, 21.8 GB for 2-7R, RTX3090,
Parameter
Optimi
for Large Language
Models et
GPU amount: 1.
2024
94480
Language Knowledge
propose an archi- lecture
menoory
Training: Free
Minimum 16 GB GPU memory.
Long- Context Extrapolation for
GPU type:
LL with am
NVIDIA, GPU
Efficient Context
Memory Xiao et al.
93638
Self playing
120
Generative Models Reasoning
peopose dataset
memorry needed:
Advensurial Language
40G per GPU,
Game Enhances LLM Reasoning Cheng all 2025
GPU type: A100, GPU
94328
QuaRot: Outlier Free 4-Bit Inference in Rotated
351
143
Deep Learning Attention Mcchanisms
propose an archi-
memoryy needed: GB, GPU type: NVIDIA A100,
M.LMs Ashhkboos
GPU amount: I,
ot 2024)
95262
Mol Jelpack: From
105
Deep Learning Algorithms
peopose dataset
memorry needed: Not specified, hut likely requires significant RAM for training.. GPU type: NVIDIA. GPU amount: 4.
Dense Checkpoints
Adaptive Mixture of
Experts Vision
Tasks Zhu al.
2024a
94391
CycleNet: Enhuncing Time Series
propose
memory needed:
an archi-
Minimum of 16
Forocasting through Modeling Periodic
tecture
RAM, GPU:
NVIDIA GPU
Patters et
(e.g. Testa V100
2024)
oquivalent).
SegVol: Universal
Computer Vision + Segmentation
propose
memory
Interactive Volumetric
archi-
16GB, GPU type:
Medical Imagee
tecture
NVIDIA, GPU
Segmentation Du
al. 2025
94155
Vuxel Mumba:
110
26
Computer Vision - 3D Object Detection
propose dataset
memory needed:
2 245
Group-Free State
Not specified, but
Space Models
high memory usage
Point Cloud based
expected due to
Object
multi-GPU
Detection Zhang al
training, GPL type:
NVIDIA A100.
GPU amount: 8,
97431
of Tricks:
122
Trustworthy Machine Learning
propose dataset
Requires
Benchmarking of Jailbreak Attacks
multiple GPUs (50
GPUs recommended);
L. Xa et 2024
approximately $5,000 GPU hours for experiments. memory needed: GR RAM. GPU: 2 NVIDIA 1
97791
The Multinodal
379
Multimodal Models
propose
4
Fnahling Large-Scale Machine Learning with 100 Astremonical Scientific Data(Collaboratinn al. 2014
23

---

Published as a conference paper at ICLR 2026
Title
Stars
Cit.#
Domain
Key Dist.
Resource
T1 T2 T3
97609
:: A
274
Dalasets and Benchmarks
propose dataset
memorry needed:
Large-Scalic
1000 GB, GPU: 2 NVIDIA GPUs
Multimodal Car
(eg.. equivalent),
Computational Fluid
Dynamics Simulations
and Deep Learning
Benchmarks Eireelate
atall,12025
Multimodal
peopose dataset
memory specified, GPU type: NVDIIA GPU amount: 8,
Multimodal
Haystack Wang et a
20240
Physical Physics
propose dataset
mennory needed: 16GB, GPU type:
Large- Scage-Scale
Collection of Diverse
NVIDIA CUDA.
Physics Simulations
GPU amount:
for Machine
Learning Ohana
2025
C SOCIETAL IMPACT
The advancement of Al agents capable of conducting Al research, facilitated by benchmarks like EXP-Bench, offers positive societal impacts. It might significantly shorten innovation cycles within AI itself and lead to more rapid advancements in machine learning capabilities. While faster pace of AI development can also democratize research tools and improve overall scientific efficiency, it concurrently amplifies the importance of addressing potential negative societal consequences. On the other hand, the rapid evolution of AI capabilities heightens risks, where we need to be careful about potential misuse, algorithmic bias, and the evolving role of human researchers, alongside the development of robust governance.
D THE USE OF LARGE LANGUAGE MODELS (LLMs)
In accordance with the ICLR 2026 guidelines on LLM usage, we disclose that LLMs were used solely for grammar and style checking during the preparation of this manuscript. No LLMs contributed to research ideation, experimental design, analysis, or substantive writing.
24

---

Published as a conference paper at ICLR 2026
E AVERAGE SCORES ACROSS ALL PAPER CATEGORIES
Defintions. Comp. Biology refers to Computational Biology. CV refers to Computer Vision. D & refers Datasets & Benchmarks. Gen. Models refers Generative Models. Proba. Methods refers to Probabilistic Methods. RL refers to Reinforcement Learning.
Addendum. Table. contains updated values for IA+3.5 Haiku for the Applications and Reinforce- ment Learning categories.
Table 6: Average benchmark scores of various models and agents across select task categories. Evaluation performed against EXP-Bench.
Category
Agent
Model
D
E
I-E
C
Allv
Allv -E
Applications
OH
Nova Pro
19.2
23.9
19.0
0.0
13.9
0.0
0.0
Applications
OH
03-mini
9.0
8.0
0.0
0.0
8.3
0.0
Applications
OH
3.5 Haiku
19.2
24.5
8.3
5.6
8.3
0.0
0.0
Applications
OH
3.7 Sonnet
9.0
26.8
30.8
7.7
8.3
2.8
0.0
Applications
IA
Nova Pro
0.0
9.8
5.0
0.0
0.0
0.0
0.0
Applications
IA
3.5 Haiku
6.8
32.3
33.3
16.7
0.0
0.0
0.0
Applications
OH
DeepSeek R1
3.1
4.3
0.0
0.0
0.0
0.0
0.0
Causality
OH
03-mini
37.8
44.6
22.2
11.1
44.4
11.1
11.1
Causality
OH
3.7 Sonnet
36.6
83.1
88.9
66.7
44.4
0.0
0.0
Causality
1A
3.5 Haiku
23.1
40.6
40.0
0.0
20.0
0.0
0.0
Causality
Nova Pro
0.0
11.7
20.0
0.0
0.0
0.0
0.0
Causality
OH
3.5 Haiku
48.7
36.6
0.0
0.0
33.3
0.0
0.0
Causality
OH
Nova Pro
17.7
14.9
0.0
0.0
11.1
0.0
0.0
Causality
OH
DeepSeek R1
10.0
18.7
0.0
0.0
0.0
0.0
0.0
Comp. Biology
OH
o3-mini
37.0
30.3
11.1
0.0
44.4
11.1
0.0
Comp. Biology
IA
Nova Pro
0.0
16.7
20.0
0.0
0.0
0.0
0.0
Comp. Biology
IA
3.5 Haiku
3.2
7.8
0.0
0.0
0.0
0.0
Comp. Biology
OH
Nova Pro
31.2
29.3
0.0
0.0
33.3
0.0
0.0
Comp. Biology
OH
3.5 Haiku
4.8
9.7
0.0
0.0
11.1
0.0
0.0
Comp. Biology
OH
3.7 Sonnet
4.8
11.1
0.0
0.0
11.1
0.0
0.0
Comp. Biology
OH
DeepSeek RI
12.2
22.1
0.0
0.0
0.0
0.0
0.0
CV
OH
Nova Pro
24.8
20.9
42.9
0.0
28.2
0.0
0.0
CV
OH
3.7 Sonnet
18.5
28.9
18.2
9.1
21.2
0.0
0.0
CV
OH
o3-mini 11.5
9.3
2.6
12.8
0.0
0.0
CV
OH
3.5 Haiku
15.9
28.3
4.5
0.0
12.8
0.0
0.0
CV
OH
DeepSeek RI
5.8
0.0
0.0
2.6
0.0
0.0
CV
IA
Nova Pro
0.0
5.3
28.6
0.0
0.0
0.0
0.0
CV
3.5 Haiku
6.4
17.4
0.0
0.0
0.0
D & B
IA
Nova Pro
0.0
0.0
0.0
0.0
0.0
0.0
0.0
D B
OH
03-mini
0.0
46.5
0.0
0.0
0.0
0.0
0.0
D B
OH
DeepSeek R1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
D& B
OH
3.5 Haiku
0.0
0.0
0.0
0.0
0.0
0.0
0.0
D& B
OH
3.7 Sonnet
19.0
89.5
50.0
50.0
0.0
0.0
0.0
D B
OH
Nova Pro
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Deep Learning
OH
03-mini
17.6
14.1
13.4
0.9
20.5
0.0
0.0
Deep Learning
OH
3.5 Haiku
20.2
25.3
12.8
1.3
15.2
0.0
0.0
Deep Learning
OH
3.7 Sonnet
17.2
37.4
37.1
5.7
14.3
0.9
0.0
Deep Learning
OH
DeepSeek R1
7.4
6.3
0.0
0.0
2.2
0.0
0.0
Deep Learning
Nova
0.0
21.7
0.0
0.0
0.0
0.0
Deep Learning
IA
3.5 Haiku
9.4
16.9
14.9
0.0
0.0
0.0
0.0
Deep Learning
OH
Nova Pro
16.4
16.5
0.0
0.0
16.1
0.0
0.0
Gen. Models
OH
03-mini
23.6
27.9
30.4
17.4
29.0
6.5
4.3
Gen. Models
OH
3.7 Sonnet
17.6
25.8
26.1
13.0
7.4
0.0
0.0
Gen. Models
IA
3.5 Haiku
5.4
24.8
45.5
27.3
3.0
0.0
0.0
Gen. Models
IA
Nova Pro
0.0
13.9
10.0
0.0
0.0
0.0
0.0
Gen. Models
OH
DeepSeek RI
9.2
19.2
0.0
0.0
0.0
0.0
0.0
25

---

Published as a conference paper at ICLR 2026
Gen. Models
OH
Nova Pro
14.7
16.7
0.0
0.0
9.7
0.0
0.0
Gen. Models
OH
3.5 Haiku
23.6
31.0
0.0
0.0
9.7
0.0
0.0
Language
OH
03-mini
16.3
8.5
13.3
0.0
20.0
0.0
0.0
Language
IA
3.5 Haiku
4.4
11.7
80.0
0.0
13.3
0.0
0.0
Language
OH
3.7 Sonnet
4.6
22.9
20.0
6.7
6.7
0.0
0.0
Language
IA
Nova Pro
0.0
2.1
33.3
0.0
0.0
0.0
0.0
Language
OH
DeepSeek RI
6.4
0.0
0.0
0.0
0.0
0.0
0.0
Language
OH
3.5 Haiku
8.5
11.5
0.0
0.0
13.3
0.0
0.0
Language
OH
Nova Pro
8.4
4.3
0.0
0.0
6.7
0.0
0.0
Multimodal
OH
03-mini
10.9
23.7
13.6
4.5
13.6
0.0
0.0
Multimodal
OH
Nova Pro
18.9
32.8
16.7
0.0
13.6
0.0
Multimodal
OH
3.5 Haiku
15.5
17.7
0.0
0.0
13.6
0.0
0.0
Multimodal
OH
3.7 Sonnet
19.7
27.0
18.2
9.1
9.5
0.0
0.0
Multimodal
OH
DeepSeek RI
4.6
9.1
0.0
0.0
9.1
0.0
0.0
Multimodal
IA
Nova Pro
0.0
17.1
50.0
0.0
0.0
0.0
0.0
Multimodal
IA
3.5 Haiku
3.0
25.3
25.0
0.0
0.0
0.0
0.0
Optmization
OH
o3-mini
21.5
25.2
25.5
6.4
25.5
0.0
0.0
Optimization
OH
3.7 Sonnet
18.2
35.8
52.0
20.0
18.6
0.0
0.0
Optimization
OH
3.5 Haiku
25.6
28.1
11.5
0.0
12.8
0.0
0.0
Optimization
OH
Nova Pro
17.1
21.0
50.0
0.0
10.6
0.0
0.0
Optmization
OH
DeepSeck RI
9.5
18.6
3.8
0.0
6.4
0.0
0.0
Optimization
Nova Pro
1.4
19.2
0.0
3.2
0.0
0.0
Optimization
IA
3.5 Haiku
8.9
31.4
60.0
12.0
2.5
0.0
0.0
Physical Models
OH
03-mini
60.0
23.0
66.7
0.0
66.7
0.0
0.0
Physical Models
OH
Nova Pro
51.7
0.0
50.0
0.0
33.3
0.0
0.0
Physical Models
OH
DeepSeek RI
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Physical Models
OH
3.5 Haiku
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Physical Models
OH
3.7 Sonnet
0.0
16.7
33.3
0.0
0.0
0.0
0.0
Physical Models
IA
Nova Pro
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Proba. Methods
OH
3.5 Haiku
49.0
32.0
33.3
0.0
66.7
0.0
0.0
Proba. Methods
IA
Nova Pro
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Proba. Methods
OH
03-mini
28.7
49.3
100.0
0.0
0.0
0.0
0.0
Proba. Methods
OH
DeepSeek RI
0.0
23.7
0.0
0.0
0.0
0.0
0.0
Proba. Methods
OH
3.7 Sonnet
86.0
86.0
100.0
0.0
0.0
0.0
0.0
Proba. Methods
OH
Nova Pro
45.0
11.0
0.0
0.0
66.7
Proba. Methods
IA
3.5 Haiku
0.0
57.0
0.0
0.0
0.0
0.0
0.0
RL
OH
Sonnet
18.3
48.2
27.3
21.2
17.6
3.0
RL
OH
03-mini
23.5
34.8
15.7
2.0
27.5
3.9
0.0
RL
OH
3.5 Haiku
27.7
41.4
11.5
0.0
17.6
0.0
0.0
RL
IA
3.5 Haiku
3.0
29.0
24.0
0.0
2.2
0.0
0.0
RL
OH
DeepSeek RI
5.0
10.3
0.0
0.0
2.0
0.0
0.0
RL
1A
Nova Pro
0.0
8.6
9.7
0.0
0.0
0.0
0.0
RL
OH
Nova Pro
17.9
28.9
0.0
0.0
13.7
0.0
0.0
Social Aspects
OH
Nova Pro
15.5
20.2
0.0
0.0
27.8
0.0
0.0
Social Aspects
OH
03-mini
21.7
22.8
16.7
0.0
22.2
0.0
0.0
Social Aspects
OH
3.5 Haiku
18.9
23.4
5.6
0.0
11.1
0.0
0.0
Social Aspects
IA
Nova Pro
0.0
18.7
16.7
0.0
0.0
0.0
0.0
Social Aspects
OH
3.7 Sonnet
11.1
23.5
100.0
100.0
0.0
0.0
0.0
Social Aspects
IA
3.5 Haiku
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Social Aspects
OH
DeepSeek RI
9.7
9.1
0.0
0.0
0.0
0.0
Theory
IA
Nova Pro
0.0
11.7
33.3
0.0
0.0
0.0
0.0
Theory
IA
3.5 Haiku
0.0
16.7
0.0
0.0
0.0
0.0
0.0
Theory
OH
03-mini
0.0
11.2
16.7
0.0
0.0
0.0
0.0
Theory
OH
3.7 Sonnet
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Theory
OH
DeepSeek R1
13.8
3.3
0.0
0.0
0.0
0.0
0.0
Theory
OH
Nova Pro
16.7
17.0
0.0
0.0
0.0
0.0
0.0
Theory
OH
3.5 Haiku
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Time Series
OH
03-mini
24.0
37.9
7.7
0.0
30.8
0.0
0.0
Time Series
OH
3.7 Sonnet
16.5
65.4
61.5
30.8
15.4
0.0
0.0
26

---

Published as a conference paper at ICLR 2026
Time Series
IA
Nova Pro
0.0
23.6
12.5
0.0
0.0
0.0
0.0
Time Series
1A
3.5 Haiku
19.5
34.5
37.5
37.5
0.0
0.0
0.0
Time Series
OH
Nova Pro
16.7
10.0
0.0
0.0
15.4
0.0
0.0
Time Series
OH
DeepSeek RI
6.3
8.7
0.0
0.0
0.0
0.0
0.0
Time Series
OH
3.5 Haiku
19.2
15.2
0.0
0.0
0.0
0.0
0.0
Trustworthy ML
OH
3.5 Haiku
15.5
21.3
0.0
0.0
25.0
0.0
0.0
Trustworthy ML
IA
Nova Pro
0.0
4.0
16.7
0.0
0.0
0.0
0.0
Trustworthy ML
OH
03-mini
6.7
3.8
8.3
0.0
0.0
0.0
0.0
Trustworthy ML
OH
Nova Pro
20.3
1.5
12.5
0.0
0.0
0.0
0.0
Trustworthy ML
OH
3.7 Sonnet
12.2
20.8
16.7
0.0
0.0
0.0
0.0
27

---

Published as a conference paper at ICLR 2026
F EXTENDED EXP-BENCH EXAMPLES
This section presents two extended examples from EXP-Bench: a question concerning robust detection in collaborative perception under imperfect localization, and a question focused on implementing Time Delay Neural Network (TDNN) for automatic speech recognition. Each example details the experiment's objective, methodology, relevant source code, and expected outcomes. The examples also include an analysis of agent performance on completing the task.
F.1 EXAMPLE I: ROBUST DETECTION IN COLLABORATIVE PERCEPTION
This example question was extended from the paper An Extensible Framework for Open Heteroge- neous Collaborative Perception (Lu al. 2024b).
The objective of this experiment is to assess whether HEAL (HEterogeneous ALliance) can maintain robust detection performance under realistic conditions of imperfect localization, when Gaussian noise added to the agents' poses. The experiment maintains constant variables such as the dataset (OPV2V- H) and the model architecture (HEAL). The independent variables the position noise and rotation noise, while the dependent variable is the model's 's detection performance matrices (AP30, AP50, and AP70). Experimental groups test the addition of Gaussian rotation and position noise at levels of 0, 0.2, 0.4, and 0.6 meters/degrees to accurate poses. The results will contribute to evaluating the robustness of cooperative perception model under conditions of imperfect localization. EXP-Bench extends this task from the original paper secti on 5.3 QUANTITATIVE RESULTS and utilizes the source code: //wow/wwaaaaaoeee ce wot.ee/eoooo/oooooooooooooooo http:///oooo////eeeeceeeeeeeeee sece____nniiiii se py from the GitHub repository https: //gi thub. com fan uC227/ /HEAL, Note that / /workspace/ refers to the working directory from the agent's ' initialization context.
The general formulation of the task includes the question posted to the agent, the overall method of the experiment, the source of this question (specifically the section in the paper and the source code), and the expected outcome. This is illustrated in Fig.
The agent's task is to use the provided GitHub repository, with the source code masked, to conduct this experiment. To aid the agent in reconstructing the masked file, detailed instructions are provided, as shown in Fig. 7b
Evaluation of the task includes design, conclusion, and setup evaluation. The conclusion appears in the 'expected outcome' in Fig. [7a] Design and setup evaluation are based on 'design complexity' and 'requirements' respectively. shown in Fig
LLM as backbone is showcased here. We perform diff operation between the code generated by the agent and the original source code. As this agent reconstructs several files to fulfil the task requirement, focus diff operation between the core reconstructed file (eva luate_ satatterobuuttesse _t_______i__s__________ py), shown in Fig. 9 The two files share the same
An example agent output using the
py)
and the source file functional goal and have a similar overall structure; however, the agent performs invalid operations in another file (reproduce, _ep__________ exp__benchh.. sh), leading to a failure in completing the task. The detailed reasoning provided by the judge is illustrated Fig. 10
28

---

Published as a conference paper at ICLR 2026
"question": "Boes HEAL maintain robust detection porformance under realistic conditions Inporfact tecalization when Gaussian noise added the agents" DPV2V-N eveilable on the lluggingface Hub as specified in the repository's data preparation instructions.)*
(Note: The experinent should parforsed
OPV2V-I dataset. Perturb the accurate pose inforsation of each agent by adding Gaussian noise sarpled from fron N(8, (o_r)))) to yaw angle, vary the noise levels sinulate different degrees Lacalization
"acthod": "Canduct robustness experiment
NCB, (a_n)*2) the and y positions
Evaluate
detection perforsence using AP retrics such as AP50 and AP78 under these noisy conditions and corpare the results with baseline rodels that tack design seasures against errors. Provide detailed analysis of degradation netrics as the noise variance increases, and discuss hon cosponents Like the nultiscale feature encoding, foreground suporvision, backward alignnent contribute HEAL's robustness.",
AP
"expected_ sonacced_ouucooee: Due to the design features Like nultiscale feature encoding, foreground supervision, backward alignsent, HEAL is expacted naintain state-of-the-art detection perforsance in terns AP retrics despite the introduction of realistic Baussian noise in the agents" poses.", QUANTITATIVE RESULTS",
"subsection_soure"" poankkppace/panccoodl http:///ccoooottt//////eoooeee..
"source": [
(a) The formulation of the task question.
{
"agent instruction": "Create script evaluate cooperative perception
robustness under inperfect lecalization conditions. The script should:
1. Accept command line argunents for:
rodel checkpoint
Fusion rethod to (intermediate, late, early, fusson)
Optional flag to test with Laplace distribution noise in addition Gaussian
Test the nodel, with different levels noise added agent poses:
Position naise: 8, 8.2, 8.4, B.6 noters
Rotetion 0, 0.4, degrees
Use Gaussian distribution by default, with option elso test Laplace
distribution
level:
Load the dataset with the specified noise paransters
Run inforence using the specified fusion nothed
Calculate detection netrics (Average and 0.7
IoU thresholds 0.3, 0.5,
Periodically save visualizations of detection results
Save evaluation results (AP39, AP50, AP70)
YANL
File in the model directory.
The script should be conpatible with the 0PV2V-H dataset and support different Fusion for cooperative perception. The ixplerentation should loading the nodel, adding noise to agent poses, running inference, and evaluating detection performance."
(b) Instructions provided to the agent.
Figure 7: Task Fields for Example 1.
29

---

Published as a conference paper at ICLR 2026
"requireentts"
"Step 1: Parse commad Line arguaents inctuding nodel, directory path, fusion acthod,
optional. Laplace noise wtteeaace//peeecooeeee tereoodd/tooos//iie wttts//ifeeeeneee.....mmo. w.noise.py:
"Scep totts//nfeeenccee v. s_noosse.pp 43:33)))
configuration wwgeecood/oool///nneeeee
specifled model directory
(/norcepcce/peenee
load trained
fros checkpoint
morrspace/speneoed tttp///nnfeeenneee....o wtt.///eeneee.nooeeeee. www.uu_ooooss..eeee 54-564))
"Step 4: Define naise Levels for position (8, 9.2, 8.4, 6.6 na naters) and ratation (8.
0.2, B.6 degrees) (/sorkspcc/ poook//cc//oeccoo////// howp//oooooooo////oooooooooooo
"Step 5: betaraina shather to use Laplace distriaution in addition to Gaussian based
(/noricpar//geeeeoe watpssod//tots//neeeeeee
pV:73-76)",
"Step each distribution type. (Gaussian and optionally Laplacs)
(/ooksspca//ppnn /ttoss/infeeencce wwt.s/_oooos.pPP... py: py:78-218)"
parassters
(/orkspaca// opancodd/ aeanccod/toola///eee wwtp://neerooss..oooo ww_noise.pp B2-099))
"Step Build detaset with the
noise setting
(/Korkspacs/span /ppens//////////////oc//////sses wttps///aaenoos..uooo www.wncwww.nn..o.oeeeeoo w_noiss.py:110::111 118)"'
"Step 9: Instialaze result statistics distionary for evalustion aetrics
Usorkspace/ sppecced//oooollo wtw.t///aeennnn..... wwtp://w........111-11a113.
"Step 18: For each batch in the dataset, perform inference using the specified fusion
nathod ('eorispccn/ppencee httpp////poooo//oo//////assaa/es wttp//////aaa//nnnn..cc woi.e.p::128----11 "Step Calculate true posstives thresholds (0.3, 0-5, 8.7)
posstives evaluation at different
visualizatiion results wtw...nnnee.ppy10-11007... py:100-174)",, detection
"Step Poriodically
//orieppce//peecee/// htttt////eeeeeee.e... u nonoese wwnoose.py:1777 17:77-199))
"Step 13: Cateulate average precision actrics (AP38, APSB, AP78) for the current noise
noiss.py:205222
"Step 14: save the evaluation results to YAHL file
V/oricppre//ppeeeeee /ttp//ool/////oone....oo
20:22-211))
"design_comp cow.in__oooxxttty.
"constant_ varttattttviallle""
"dataset";
*OP/2V-1"
"sudel Loodll_arcee techiitttttr: "HEAL with fixed components such multiscale festure encuding, foreground supervision,
alignnent"
pertlponneet__vviailiee "position.noiee nodse":
"g it", "9.2 "9.4 ei",
"9.6 m²*
"notatioo_nntter degree" degree",
"9.4 degree"
degree"
*noise_ disttriuttinn
"Goussian", "Laplace (optional)"
Pusion_natheett:
"Antorsediate""
"early", "no fusion"
"depandent_ varisbless
"detection_per "APS0".
"API""
Figure 8: Evaluation of the design and setup for the Extended Task in Example 1.
30

---

Published as a conference paper at ICLR 2026
Inforence w_ by ovohatt_ectwsnsss.. Un3MIn
1tate 15,200 det trst_pirser(::
mextt]:
aap.eirs cotpst edtan skrimnd hosiiiic recation reise levels to test
posttiion rolce leants I6-4, M.d, d.l, susl
retationursien www.ereennneeiai (bib, 0.2, 8:4, 0.6) rwalts II
hopee senlluilla Ooee,, on1) hopest
leyesl Iillatee tyyes""
topest
14
medel train create teykll)
pellat (traating wast five resoqpiler))
seeed patt opt nodel dir
resmeccssch mail moit.ntib--laal.eed www.://ee//o0l/seeegytt. mak))
(Taecan fres spac (resue spach"))
cleipeirt")
21
16, 3.2.
18
FSE 19e 350t [Ng Sola [..]
FeN Jist (8) 9g.0.
FSl, moie, Jist [8 (1.6.
ant.ales Laplace:
Japlace Aw0.Toplann
1059
28
11
For (pea maany 2o1 115, rot Mee,, rot std) in sip noen 1465, pou vod 2i00, rot main 2005, rot and AAsso rpuradde cond/on)
rolse secking OrdersiDacte)
ar pppo, poc_adt alat vot "oouawnn pouu "rol relLwwe))
satset]
lapcace:
IER Bot ind En pocettiwn potes TAQATRE
full ERl in ratullion N01 Jevalee
relio AARETA Jone
Elpee Eaos snay staj?
sruni Tesling mos Jeanl: (rRiee /nw
relse Tlaslece! fatanse Jallac cise S0NSSIM0)
1a08 6913 (ange)) Eloet ctRT2 std, 'et sta!!
27
13
notie, settinng aegt Tsolace 1
1768
suttan Jagi
balld dataset for each rolse setting
retsust balid setanst/hssee, veesaltte
rata Josker Desatoaderiseenoo extase,,ttto sazertiinp, (astefuise)
45
resalt, stat (6.3) ("tp') D), 1fp's TT, 'at's i, "seanel (1),
0.50 Dipp: DJ, "tp': Li 1gtt 9, "scanet: (1)-
("pp's Th 'epis Th pt my
(11)
False to Xt8) Ant) (304 (rnn sewo) futii withod auffic retuaste
inforcinfa High. (eulomm.nttiin) (Holoccaat)) (ikias, wotiiing
tarchune graddt::
13
butengdette trannatillo tegdericecceetttteeee.. device) if ast-furio furtos aeched on "late's
infor result = inforencc slib inforonce fwwieetosshh_ttttt.oo fettuesccstttttllllll... speeersl.etteeet cld opt. Lestonnenn unddled " www.ertaiiitili....
elnet
caboeltte fpdaned tonoeyprel scoreget tonse tessrureeslt stat,t.7
setlicmaa flast recattsreealt aedel die, Bilice Tesel)
1290 Japera (808)
cons Lippers M800 Fepeets 1259, Tapiers ARR2S
sen' 31138 Save sent SEL 9ath (wtebort wodel diry (iseass//tuffiaaaall.
1230, 1950, 1970 whill www..iiaiiimmaatttrrraaa atat./ooot eatgottatr, infer_ants) all_pesattrSsiiie rano)
apsa,
www.ooll resalte, dotate crytess2to3
Figure 9: Example I's Git diff comparing the masked source file and the agent-reconssrrutted source code. Red highlights indicate deletions, while green highlights represent additions.
31

---

Published as a conference paper at ICLR 2026
"error_ene" _erro__nslysss":: "Agent perforsed forbidden operations. See setup_ trtup_nonitor_com wow.preeeesiveeeroooonn "ronitor_compp" wwwprehansive.reeeeaeeeeee
"The details.",
instructed not to access the research papen, and there is evidence that it read or processed the paper (e.g., no P0F or research paper contents were loaded). In addition, although the agent's log contains nentions keywords such as 'git' (triggering nicroagent actions), there is indication that it executed any git operations
checking connits, switching branches, accessing different repositary versions. Houever, there is clear evidence that the experimental. results were not obtained through actual. experisents. In the reproduction script (/workspace/reprn) wetpaaaeereprruueeeooo carrrddce__x__eeeee _ep_eexp________e sh)_ after running the evatustion soripts, the agent deliberstely creates duney results files with hardcoded values for different sections (e.g., netrics to VAML files denonstration purposes) indicate that the crucial result data was fabricated rather than generated executing real experinents. This constitutes the use of faked data in place of experimental. results."
noise
writes
AP38, AP58,
Figure 10: Error Analysis and Comprehensive Explanation the agent's failure complete the task in Example 1.
32

---

Published as a conference paper at ICLR 2026
F.2 EXAMPLE 2: TIME DELAY NEURAL NETWORK FOR ASR
This example is extended from the paper Zipformer: A Faster and Better Encoder for Automatic Speech Recognition (Yao al. 2024).
The objective of this experiment is to implement a Time Delay Neural Network (TDNN) that achieves a Word Error Rate (WER) of less than 1% on the test set. This setup focuses on construct- ing TDNN model with three Convid d layers-each followed by ReLU activation and Bat Normalizattionann
final linear layer to produce log probabilities for phoneme classes. The dataset (yesno), model type (TDNN), loss function (Connectionist temporal classification), and feature extraction method (23-dimensional fbank features) are held constant. Independent variables include the model architecture, training hyperparameters (e.g., learning rate, weight de- cay), and number of epochs, while the dependent variable the WER obtained during evalua- tion. This task emphasizes practical training and decoding using k2's one best decoding method and evaluates performance using the WER metric, targeting values below 1%. extends this task beyond the baseline speech recognition example by formalizing end-to-end pipeline using code modules: /workspace egs/ / yesno/ ASR/ tdnn/moddede howannn/ooel..p.. train.... py, decode py, and asr datamodule. py from the Github repository http:t//tttttppss//ghbbbb 12--ssa/be /ee//cceaalll
The general formulation of the task includes the question posted the agent, the overall method of the experiment, the source of this question (specifically the section in the paper and the source code), and the expected outcome. This is illustrated in Fig. Tla
Again, the agent's task is to use the provided GitHub repository, with the source code masked, to conduct this experiment. To aid the agent in reconstructing the masked file, we provide structured instructions (Fig. [1b); rather than specifying every architectural detail, it simulates a realistic experimental setting where some gaps remain and the agent must experiment achieve the target outcome (i.e., WER of less than 1% on the test set). This balance ensures that the agent has sufficient guidance to proceed, while still reflecting the trial-and-erros nature of real-world research.
Similarly, evaluation of the task includes design, conclusion, and setup evaluation. The conclusion appears the "expected outcome' in Fig. 11a Design and setup evaluation are based on 'design complexity' and 'requirement" respectively, shown in Fig
For the agent performance in this example, we make use of an agent's output using the bedrock-us- umuzznn-nvv-pree prae--rro---O LLM backbone. We perform a diff operation between the code generated by the agent and the original implementation files provided in the baseline. Since the agent restructures multiple modules to accomplish the speech recognition task, analysis focuses diff between the core model implementation file (model ppy) and the original reference. The agent correctly builds the TDNN model and integrates it with the training and decoding pipeline. The versions of model files share a similar architectural skeleton, but differ in details such as layer configuration and parameter initialisation. The differences shown in Fig. I3
33

---

Published as a conference paper at ICLR 2026
"cuestion": "Ixplement Time Dolay Neural, Network (TDRN) model for speach recognition on the yesno dataset that achieves Mord Error Rate (NER) Less than 15.",
"ne "nethod"; "Create TOMN model. that processes audio features and outputs log probabilities phonene classes.
using CTC on the yesno dataset, shich consists of recordings of someone saying "yes" or 'no" in verious conbinations.",
"expected_uutcone:: trained nodel achieves WER less test set. The expected output should be sinilar to: [test_set] NER 0.42% [1 240, del, sub]",
"source"; [
//ors/spceeees//ee///////// wttp://t//tt///oeeeeeeeee PV",
//orkspace//gs//eeeaa atnn//ssnnnnnn "cstanndueeeeepp"
(a) The formulation of the task question.
sngent_nnstructions "Your (TDWK) sodel for speech recognition on the yesno dataset that achioves Nord Enror Rate (WER) less than 18. Follow these steps:
implement Delay
Create model. anchitacture:
Inglenent PyTorch nodel. with three Convid layers
Layer
Add Final Linear layer that outputs log probabilities for phonere classes
The rodel, should take audio features as input and output leg probabilities
up processing for
Create a data sodule to load and prepracess the yeano dataset
25-dinensional rbank features
Split the data into training and testing sets
Ieplement the training process:
Set up an 560 optisizer with appropriate learning rate (around 1e-2) sodel.
achieve convergence Save checkpoints and track the best sodel based validation tossiiiin,
epechs (around
Implenent decoding and evalustion:\n
nodel. outputs wond Use k2's one best, decoding for finding the best path the Report the WER and verify it is less than 181ninThe yesno dateset consists of recordings of soneone saying 'yes' or "no" in various conbinatioos. Your goat is to than
decoding function that
Evaluate
calculate
(WER)
recognize
18."
(b) Instructions provided to the agent.
Figure 11: Task fields for Example 2.
34

---

Published as a conference paper at ICLR 2026
"roquirements":
Layers, by ReLU activation and Batchtiors, and a final. Linear layer that outputs log
probabilities for
classes (Jaorispace/egu "Stop up loading porrkppan//eg//eeen///ee Seaen//SS//tmmmee http///mm/eeeel-yylBeBeee dimensional tbank teatures tros audio
yesno dataset, extracting
"Step Initialize the TORN model with aporopriate input and output dimensions based feature size and py:499-522)",
classes
http////s///ee/////ee//////////e (/vorkspace/ httpspcce/e///e/eeeeeeeeeeeeeeee
"Step Set up ETC loss for training using gresh compiter froe the HZ Library (horkspare/ggs/ vveno//5R/tdmme http///aannn//pptpte 4977--477 253-323)",
"Step Train the nodel using 500 optimizer with appropriate tearning rate and neight decay for nultiple epochs (/vorkcpace/egs/ http:///s/yyes/////eee/ tonn///n/////////////////// trainn py:
*Stop 6: Save checkgoints during training and track the best model. based on validation (/sonispac//egs/veenn/// hatpp///s/////////////e// http://dnnntttty.eeee. 211-23--
"Stup Ieplarent decoding functionality convert model. outsuts word sequences using H2's one best, decoding
"Step Evaluate the nodel. on the test set and catculate Word Erron Rate (MER) (/workspan Koorrkspec////ae hwtp://ssn//AS///.... Rttp///////t////oo////// py:14332044 305 313)",
*Final, Step: Report
C/vorkspece/ http:///g/////////eeeeeeeeeee/// /eesee//R////nnn wtt////////eeoeeeeeeeeee decode. 2y:28722483"
"design_
wwwssign.copteeeeeee complexity":
"constant veriables":
"dstaset": "yesno
(recordings "yes" "no")",
"yydee__ypee'' "TONK"
"Loss, function": "CTC
"Extrection of 25-dimensionst, fbank
features
"architecture":
Convid Layers
by
Batchhoorn
"Final Linear producing probabilitties
classes"
"ostimize"":
"500 (with Learning rate around 1e-2 as spacitied)*
"treining_eppcchs:
"Approxtnatall 15 apachs (can be aodified for convergence]"
"byparaaaanaterrs: "Ineludes learsing nate, weight dacay, and any SGD-
specific settings"
"ford Enron Rate neasured on the test set with the target being less
than 122
Figure 12: The design and setup evaluation of the extended task in Example
35

---

Published as a conference paper at ICLR 2026
model.py 00:48:29 py
-1,76 +1,31 Import torch
import
class Aelloss/nn.Mdutee
_init Iself):
super/CTcLoss, self), init
solf.ctc, loss mn.cTCLosssblamkke))
class Tonní Nn.Nuttte)::
def
init [self, BLB. Na_features: int, Ein classes: int)
Args:
nuR teatures:
Input dinension.
nun_clesses:
12
Model output cinension
super[) in
def
Norward(sells outputs, targets, Input, lengths, target_ lengths los.ctt_oossluuuuuttae
10
lengths, Lengths)
16
self, idni E Einn, imn.Senentiall
17
Canvid
channeTsenus features,
19
out_chaanee1==32)
20
kerne
22
nn, intUelinna) cestthoomnnnnun_ffttreeeees offine:false),
23
25
in chan out MiL_chhaneessse 1s=32, Laizess, ditation=2,
25
28
29
nn. Batchho BnBBatchhormmannu (nd nn. Convidt
features=32, [ts(nnn_ffattussse
affine=Fatse),
32
nut_channels=32,,
35
ke mereel keinel_sizem,, dilatione4,
},
38
nn. ReLUI Rww.RelUUinnacccen inplace-True), www.eechhormmoonnnnn features:32, affinesFalse),
41
self- output Linear mn.Linear(in, features=32, F out featuresenun, classes) TDRR Nodule):
12
init [selfi8
14
Super(Torel, self), indt
15
(placeholder)
15
self, layer in.Linee
42
17
toraardsettf Tensar
45
Args:
Input shope [N, T, def sor, x):
18
19
self, Layer())
49
Returns:
58
The
has shape
52
X.permutel0, IN, IN, Self.Idnn(R)
53
pepentute(,,
IN,
self.output, Strpput_nnnee Linear(x)
56
fincffucciinn Nunctional, Sot_soft,,,xx dime-1)
21
class TOMACTCHodeLimn. Mocule):
22
def
init init_ Iselff8
SuDer((INNCTORodee
24
self. TDMN))
25
self.cte Loss CTCLOSE
Figure 13: Example Git diff the masked source file and the agent reconstructed source code. In the diff, red highlights are deletions. Green highlights additions.
36

---

Published as a conference paper at ICLR 2026
G DATASET CURATION AND BENCHMARK CONSTRUCTION DETAILS
To ensure the quality and integrity EXP-Bench, we developed the curation pipeline through careful, iterative process. Each component was prototyped, tested real papers, and refined based on manual inspection by collaborators. This allowed us to isolate and address specific failure modes incrementally, steadily increasing curation throughput without compromising accuracy. Several representative issues that were patched in our final pipeline are documented Table. 7 Overall, 120 tasks were attempted end-to-end by team to ensure the construction pipeline worked properly. Finally, manual validation of all constructed tasks was also aided by the availability of ground truth from the papers and open-source code repositories themselves, making the verification process relatively straightforward. The EXX--Bench team is committed to the long-term maintenance and growth of the dataset. We acknowledge that, despite our efforts, some issues may remain in the dataset. To address that, we will actively monitor feedback and bug reports via GitHub and HuggingFace issue trackers and will address any concerns raised by the community post-release. All data is hosted on both platforms to ensure accessibility and stability. with potential plans to replicate the dataset on archival storage for long-term preservation. We will continue exploring additional techniques to further safeguard the solvability of our tasks. To foster transparency, reuse, and critical engagement, the dataset will be released under the permissive Creative Commons Attribution 4.0 license, and all code under the MIT license. We encourage the community explore, build upon, and challenge EXP-Bench as an open and evolving resource.
Table 7: Examples of extraction issues identified that subsequently patched in the final pipeline.
Task Component
Issue
Actual Example
The hypothesis is statement instead of question
BaDFxpert outperforms baseline defenses in backdoor detection on CIFARIO, 10, achieving significantly higher AUROC (near 99%).
Question
Conclusion data mentioned in the hypothesis
Specifically, can PDF achieve around 34.64% lower MACs compared to PatchTST and 74.38% lower MACs
Masked source doesn't exist
"source"; ("/workspace/- topomlp setA r50, w otraasfoommpp"...
Masked source
Included masked source with wrong path
MuSc has musc.py under workspace/model/ but the source file indicates it under workspace/examplee
Steps are too specific
Run the evaluation script for the baseline EVA-CLIP ViT-B/16 model using distributed
Requirements
processing with 8 GPUs...
Asking the agent to use a masked source script
Merge the trained models using the heal_ ceal_tools.py script /vortspcceopnnoodtoossheelll ttsp://ppooooooooooeeeeooooosolr 130)
Invalid operation
Analyze execution outcomes from Table comparing...
Expected outcome
Conclusion not aligned with the paper's findings
N/A
Method / Usage
Mentioned specific parts of the paper (tables or figures)
The scripts will log metrics including mean rewards and standard deviations, which can be compared with the reported results in Table 2 of the paper.
Instruction/ / Agent Instruction
Required hyperparameters not given in the agent instruction
Set appropriate model architecture parameters (encoder layers, attention heads, dimensions)
Invalid operations
Collect and analyze performance results from Table 3....
37

---

Published as a conference paper at ICLR 2026
Time and Cost Expenditure. During the initial phases before curation pipeline final- cred-each paper required roughly two hours manual effort. This involved a full read-through (with emphasis on evaluation sections), task-by-task verification, and iterative pipeline corrections to ensure compatibility. The process included checking GitHub repositories, assessing setup validity and complexity, and verifying alignment with the paper's 's descriptions. Once the pipeline was fully constructed and refined based on feedback, manual validation time dropped to around 20 minutes per paper, primarily confirm alignment. Only minor adjustments were rarely needed, and we expect this time to decrease further in future deployments. LLM-rrlated extraction costs varied by task type and count, averaging approximately $60 USD per paper. For extraction, we used 03-mini-2025-01 O1-preview for the main task extraction and 2oaude-377-oomnet-222500 11:v000 for implementation extraction. Costs were primarily driven by input tokens, as the models required full paper texts and codebases to perform accurate extraction.
38

---

Published as a conference paper at ICLR 2026
H EXTENDED ANALYSIS ON PREVALENT AGENT FAILURE PATTERNS
Some overlap between categories may exist, as the classification was performed by an LLM.
Table 8: Agents fail in diverse ways across different phases of experimentation, measured across all agent and model evaluations.
Phase
Failure Type
Prevalence (%)
conclusion
Missing Conclusion Content
26.18 18.18
conclusion
Incorrect Conclusion Interpretation
19.66
conclusion
Incomplete Conclusion Outcome Statement
14.43
conclusion
Extraneous Details
7.77
conclusion
Missing Conclusion Analysis
4.35
conclusion
Missing Comparative Conclusion Analysis
4.03
conclusion
Minor Omission of Specific Details
3,47
conclusion
Incorrect Numeric Conclusion
3,21
conclusion
Mismatched Conclusion Format
2.7
conclusion
Error Message Output
2.67
conclusion
Incomplete Conclusion with Missing Exp. Findings
2.14
conclusion
Conclusion Diverges from Expected Emphasis
1.6
conclusion
Missing Comparative Analysis Missing Quantitative Performance Metrics
0.8
conclusion conclusion
0.8
Missing Visualization Details
0.56
conclusion conclusion
Incomplete Performance Evaluation
0.53
Missing Numerical Equivalence Verification
0.53
conclusion
Missing Trend Analysis
0.53
conclusion
Naming Inconsistency Output
0,53
conclusion
Conclusion Partially Matching with Numerical Deviations
0.27
conclusion
Deviation in Saturation Point Conclusion
0,27
conclusion
Inconsistent ASR Reporting
0.27
conclusion conclusion
Missing Conclusion Analysis on Attack Budget Effects
0.27
Missing Diminishing Returns Analysis
0.27
conclusion
Missing Methodological Innovation Discussion
0.27
conclusion
Missing Performance Evaluation Metrics
0.27
conclusion
Missing Submission Format Specification
0.27
design
Incomplete or Misclassified Design Variables
16.05
design
Omission of Required Design Variables
19.84
design
Complete Omission of Exp. Design Variables
13.1
design
Incorrect Design Specification Details
8.32
design
Incomplete Exp. Design Details
7.67
design
Irrelevant Procedural Additions
7.62
design
Missing Design Variable Information
3,83
design
Inclusion of Extraneous Factors
3,64
design
Incorrect Parameter Details
3.18
design
Partial Omission of Constant Variables
2.75
design
Incomplete Constant Variable Specification
3,61
design
Partial Fulfillment of DV
1,93
design
Error Message Returned Instead of Design Information
1.27
design
Incomplete Differentiation of Constant and Ind. Variables
1.27
design
Missing Dependent Variable Tracking
1.06
design
Incomplete Exp. Design Specification
0.64
design
Incomplete Specification of Design Variables
0.64
design
Missing Hyperparameter Design Details
0.64
design
Partially Complete Design Variable Specification
0.64
design
Missing Design Formatting Details
0.42
design
Missing Design Variables Details
0.42
design
Missing Explicit Variable Labeling
0,42
design
Missing Configuration File Variable
0.21
design
Missing Input Format Details
0.21
39

---

Published as a conference paper at ICLR 2026
design
Omission of Exp. Configuration Details
0.21
design
Omission of Fixed Block Partition
0,21
design
Partial Design Variable Extraction with Misclassification
0.21
exec
Environment/Dependennyy Configuration Errors Execution Script and File Errors
29.38
23.84
exec
Missing Dependency Error Missing Setup Script File
11.9
exec
6,95
exec
Tensor Operation Execution Error
3.22
exec
Syntax Error Execution Environment
2.86
exec
Missing Input Data File
2.27
exec
Missing Required Attribute in Execution
2.27
exec
Missing Evaluation Output Files
1.82
Cxec
Missing Requirements File
1.82
exec
Runtime Indexing Error During Generation
1,82
exec
Insufficient Shared Memory in DateLoader Execution
1,41
Execution Environment Warning: Root Privilege Usage
1.36
exec
Incorrect Dependency Import in Execution
1,36
Dependency Version Conflict
0.91
exec
Docker Execution Failure
0.91
cxec
Incomplete Results Saving Impl.
0.91
cxec
Incorrect Dataset Loading
0.91
exec
Incorrect Function Argument Handling
0.91
exec
Missing Hugging Face API Token Authentication
0.91
exec
Missing Performance Metries and Argument Parsing
0,91
exec
Missing Trust Remote Code Flag in Execution Environment
0.91
exec
Missing Setup Script File
0,45
setup
Missing Essential Impl. Components Incomplete Evaluation Metric Impl.
39.71
setup
2.15
setup
Missing Critical Exp. Setup Details
1.88
setup
Incomplete Data and Preprocessing Setup
1.83
setup
Missing Command Line Argument Parsing
1.58
setup
Incomplete Exp. Setup Impl.
1,49
setup
Incomplete Training Regimen Impl.
1.47
setup
Incomplete Comparative Setup Features
1.25
setup
Missing Modular Helper Functions
1.13
setup
Incomplete Dataset Splitting Setup
1.04
setup
Missing Comparative Evaluation Methods
1.02
setup
Naming Inconsistencies Components
1.02
sctup
Missing Detailed Architectural Parameters
0.91
setup
Incorrect Model Initialization
0.9
setup
setup
Missing Optimizer Configuration
0.9
setup
Incomplete Evaluation Procedure
0.79
setup
Missing Critical Import Statements
0.79
setup
Missing Essential Library Imports Incomplete Results Saving Impl.
0.56
setup
setup
0,45
setup
Incomplete Misplaced Setup Impls.
0.45
setup
Incorrect Dependency Import Misconfigured Exp. Infrastructure
0.45
setup
0.45
setup
Missing C++ Acceleration Integration
0.45
setup
Missing Distributed Training Parameters
0.45
setup
Missing Hardware/Device Configuration
0.45
setup
Missing Training Pipeline Configuration
0.45
setup
Incorrect Evaluation Metric Impl. Incorrect Forward Method Impl. Hard-Coded Configuration Instead of YAMI. Loading Incomplete Benchmark Configuration Incomplete Rendering Pipeline Impl.
0.37
setup
0.35
setup
0.34
setup
0.34
setup
0.34
setup
Incorrect Model Architecture
0.34
setup
Incorrect Testing Dataset Usage
0.34
40

---

Published as a conference paper at ICLR 2026
setup setup setup
Misconfigured Exp. Setup Parameters Missing Configuration File Loading Missing Critical Function Impls. Missing Critical Quantization Procedures Missing Essential Parameter Initializations
0.34
0.34
0.34
setup
0.34
setup
0.34
setup
Missing Intermediate Data Reuse Mechanism
0.34
setup
Missing Loss Function and Evaluation Metric Setup Missing Model Architectures
0.34
setup
0.34
setup
Missing Model Evaluation Mode Invocation
0.34
setup
Missing Model and Dataset Integration
0.34
setup
Missing Reproducibility Measures
0.34
setup
Missing Feedback Mechanism
0.24
setup
Environmen//Dependenyy Configuration Errors
0.23
setup
Faulty Dataset Integration
0.23
setup
Faulty Training Script Logic
0,23
setup
Impl. Mismatch with Setup Specification
0.23
setup
Incomplete Benchmarking Function Impl.
0.23
setup
Inconsistent Configuration Incorrect File Naming/Structur
0.23
setup
0.23
setup
Insufficient Exp. Setup Impl.
0.23
setup
Missing Chain-of- Th-ouuhtt Module
0.23
setup
Missing Conditional Model Initialization Missing Dimensionality Reduction Impl.
0.23
setup
0.23
setup
Missing Evaluation on Validation Data
0.23
setup
Missing Exp. Entry Point Impl.
0.23
setup
Missing Exp. Resumption Mechanism
0,23
setup
Missing Explicit Data Loader
0.23
setup
Missing Explicit Model Arch. for Inner Optimization
0,23
setup
Missing Final Agent Return Impl.
0.23
setup
Missing Final Test Validation
0,23
setup
Missing Finalization Message
0.23
setup
Missing GPT-based Evaluation Component
0,23
setup
Missing GPU Batch Size Adjustment
0.23
setup
Missing Initial Policy Evaluation
0.23
setup
Missing Logging Mechanism
0.23
setup
Missing Loop Termination Mechanism
0.23
setup
Missing Model and Transform Initialization
0.23
setup
Missing Multi-GPU Result Merge
0.23
setup
Missing Multiple Runs for Statistical Significance
0,23
setup
Missing Periodic Evaluation
0.23
setup
Missing Pretrained Model Loading
0,23
setup
Missing Real-World Benchmark Dataset
0.23
setup
setup
Missing Regularization Term
0.23
setup
Missing Scalability Testing Configuration
0.23
setup
Missing Test Cases
0,23
setup
Missing Visualization Impl.
0.23
setup
Missing Visualization Impl.
0.23
setup
Synthetic Dataset Used Instead Specified Dataset
0.23
setup
Incomplete Defense Testing Setup
0.14
setup
Missing Critical Evaluation Computation Steps
0.14
setup
Missing Custom CI Test Impl.
0.12
setup
Missing Hydra- based Exp. Runner and Plotting Script
0.12
setup
Missing Key Exp. Pipeline Steps
0.12
setup
Missing Multiple Forecast Horizon Configurations
0.12
setup
Missing Performance Comparison
0.12
setup
Missing Required Evaluation Script Modifications
0.12
setup
Missing Sequence Length Computation
0.12
setup
Ambiguous Evaluation Metric Reporting
0.11
setup
Deviation from Required Library Usage
0.11
41

---

Published as a conference paper at ICLR 2026
setup setup setup setup
Faulty Early Stopping Impl.
0.11
Faulty Quantization Branch
0.11
Flawed Visualization Utility Impl.
0.11
Incomplete AstroCLIP Integration
0.11
setup
Incomplete Benchmark Directory Setup
0.11
setup
Incomplete Exp. Setup Impl.
0.11
setup
Incomplete Explanation of Comp. Graph Visualization
0.11
setup
Incomplete Extraction of Runtime Configuration
0.11
setup
Incomplete Inner Objective Impl.
0.11
setup
Incomplete Memory Unit Impl.
0.11
setup
Incomplete Model Architecture Impl.
0.11
setup
Incomplete Model Weights Download Handling Incomplete Optimizer and Scheduler Setup Incomplete Output Processing Incomplete Parallel Processing Impl.
0.11
setup
0.11
setup
0.11
setup setup setup setup setup setup setup setup setup
0.11
Incomplete Prediction and Data Loading Impl.
0.11
Incomplete Quantization Benchmark Param. Config
0.11
Incomplete Result Logging Impl. Incomplete Results Saving Impl.
0.11
0.11
Incomplete Training Loop and Reproducibility Measures
0.11
Incomplete Visualization Impl.
0.11
Incomplete Visualization Pipeline Impl.
0.11
Incomplete or Misplaced Plotting Impl.
0.11
setup
Inconsistent Dataset Collection Specification Inconsistent Feature Extraction Impl.
0.11
setup
0.11
setup
Incorrect Benchmark Command Structure Incorrect BlockOptimizer Impl.
0.11
setup
0.11
setup
Incorrect Class Structure
0.11
setup
Incorrect Dataset Loading
0.11
setup
Incorrect Exp. Split Configuration
0.11
setup
Incorrect Function Signature
0.11
setup
Incorrect Hard-coded Parameter Block Impl.
0.11
setup
Incorrect Independence Test Impl. Incorrect Inner Objective Impl.
0.11
setup
0.11
setup
Incorrect Optimizer Comparison Impl. Incorrect Spatial Matching Impl.
0.11
setup
0.11
setup
Ineffective Caching Setup
0.11
setup
Insufficient Benchmark Dataset
0.11
setup
Insufficient Positional Encoding Impl.
0.11
setup
Misconfigured Benchmark Parameters
0.11
setup
Misconfigured Warmup Parameters
0.11
setup
Mismatch in Benchmark Parameter Settings
0.11
setup
Missing Ablation Study Configuration
0.11
setup
Missing Alternative OOP API Impl.
0.11
setup
Missing Benchmark Data Processing Components
0.11
setup
Missing Benchmark Runner Function
0.11
setup
Missing Block Parameter Switching Logic
0.11
setup
Missing Block-specific Finetuning Parameters
0.11
setup
Missing Block-wise Parameter Grouping Impl.
0.11
setup
Missing BlockOptimizer Impl. Details Missing BoT Pipeline Components
0.11
setup
0.11
setup
Missing Checkpoint Loading Impl.
0.11
setup
Missing Checkpoint Saving Impl.
0.11
setup
Missing Command-Lingg Toggle for Final Execution
0.11
setup
Missing Comparative Optimization Strategy Impl. Missing Comparison Visualization Component
0.11
setup
0.11
setup
Missing Comprehensive Plotting Components
0.11
setup
Missing Computational Performance Metrics Missing Conditional Checks
0.11
setup
0.11
42

---

Published as a conference paper at ICLR 2026
setup setup setup
Missing Conversation and Interactive Setup Components
0.11
Missing Critical Analysis Components
0.11
Missing Critical Benchmark File
0.11
setup
Missing Critical CycleNet Components
0.11
setup
Missing Critical Exp. Tracking Components
0.11
setup
Missing Critical Information Extraction
0.11
setup
Missing Critical Model Module Impls.
0.11
setup
Missing Custom Optimizer and Trainer Integration
0.11
setup
Missing Data Reshaping Remove Padding
0.11
setup
Missing Data Structure Impl.
0.11
setup
Missing Dataset Download Script
0.11
setup
Missing Dedicated Custom Function for Mesh Extraction
0.11
setup
Missing Dedicated Inference Script
0.11
setup
Missing Dedicated Quantized KV Cache Decode Impl.
0.11
setup
Missing Dedicated Trainable Bundle Methods Impl.
0.11
setup
Missing Detailed Exp. Protocol
0.11
setup
Missing Dynamic Dataset Configuration
0.11
setup
Missing Edge Case Scenario
0.11
setup
Missing Embedding Extraction Impl.
0.11
setup
Missing Entry Point Script for Exp. Setup
0.11
setup
Missing Essential Feature Extraction
0.11
setup
Missing Essential Modules and Evaluation Components
0.11
setup
Missing Essential Post- Post--rocesssing Functions
0.11
setup
Missing Expected Configuration Output
0.11
setup
Missing Expected Conversation Templates
0.11
setup
Missing Exp. File Location
0.11
setup
Missing Exp. Tracking and Run Naming Mechanisms
0.11
setup
Missing Exp. Replication Configuration
0.11
setup
Missing Exp. Script Modifications
0.11
setup
Missing Explicit Meta-Parameter Definition
0.11
setup
Missing Explicit Model Components
0.11
setup
Missing Explicit Parameter Configuration
0.11
setup
Missing Explicit Return of Submission File Path
0.11
setup
Missing Explicit Task List Definition
0.11
setup
Missing External Data and Model Weight Downloading
0.11
setup
Missing Final Evaluation Routine
0.11
setup
Missing Final Output Return
0.11
setup
Missing Fine-tuning Orchestration Function
0.11
setup
Missing Finetuning Type Loader Impl.
0.11
setup
Missing Framework Integration Components
0.11
setup
Missing GPU Optimization
0.11
setup
Missing GPU Synchronization
0.11
setup
Missing GPU Transfer and Synchronization Setup Missing Gene Expression and Embedding Matching Impl.
0.11
setup
0.11
setup
Missing Hugging Face API Token Authentication Missing Implicit Differentiation Step Missing InfI.LM Context Memory Impl. Missing Inference Prompt Configuration Missing Input Pre-processing Missing Input and Key-Cache Quantization Configurations Missing Integration Components for Advanced Training
0.11
setup
0.11
setup
0.11
setup
0.11
setup
0.11
setup
0.11
setup
0.11
setup
Missing Integration Components for Block-witd -iick- Optimization
0.11
setup
Missing Intermediate Data Reuse Mechanism Missing Learning Rate Scheduler Impl.
0.11
setup
0.11
setup
Missing Lightning CL.I Training Configuration Missing Logging Configuration
0.11
setup
0.11
setup
Missing MLP Layer Size Configuration
0.11
setup
Missing Main Execution Function
0.11
setup
Missing Mem. Management and Precision Conversion Impl.
0.11
43

---

Published as a conference paper at ICLR 2026
setup setup setup
Missing Memory Precision Conversion Impl.
0.11
Missing Memory Saving Metric Calculation
0.11
Missing Meta-Buffer Template Retrieval
0.11
setup
Missing Model and Transform Initialization
0.11
setup
Missing Network State Extraction in Visualization
0.11
setup
Missing Normalization Configuration
0.11
setup
Missing Normalization Disabling Parameter
0.11
setup
Missing Optimality Condition Impl.
0.11
setup
Missing Optional Analysis Component
0.11
setup
Missing Optional Parameter Impl.
0.11
setup
Missing Output Directory and Logging Setup
0.11
setup
Missing Performance Metrics and Argument Parsing
0.11
setup
Missing Platform-Specific Parameter Handling
0.11
setup
Missing Platform- Specific Parameter Impls
0.11
setup
Missing Parameter Switching and Gradient Checkpointing
0.11
setup
Missing Post- Post-Processing Component
0.11
setup
Missing Post-Processing and Result Saving Impl.
0.11
setup
Missing QMIX Algorithm Configuration
0.11
setup
Missing Random Sampling Impl.
0.11
setup
Missing Replication Procedures
0.11
setup
Missing Required Torch- Impl.s
0.11
setup
Missing Result Storage Configuration
0.11
setup
Missing Result Visualization
0.11
setup
Missing RevIN Normalization Impl.
0.11
setup
Missing Reward Shaping Impl.
0.11
setup
Missing Scalability Testing Configuration
0.11
setup
Missing Single Scan Visualization Function
0.11
setup
Missing Specialized Trainer Integration
0.11
setup
Missing Standardized Testing Components
0.11
setup
Missing Statistical Evaluation Metrics
0.11
setup
Missing Supervised Fine-Tuning Data Integration Missing Surrogate Model (AutoGluon) Impl. Missing Symbolic Mathematics Library Impl.
0.11
setup
0.11
setup
0.11
setup
Missing Synchronization of Initial Conditions
0.11
setup
Missing Task-Specific Configuration
0.11
setup
Missing Testing Dataset Configuration
0.11
setup
Missing Testing Procedure
0.11
setup
Missing Training Loss Visualization Impl.
0.11
setup
Missing Tree Mapping Benchmark Impl.
0.11
setup
Missing Unique Run Name and Configuration Adjustments
0.11
setup
Missing Video Processing Impl.
0.11
setup
Missing Video Processing and Loss Calculation Impl.
0.11
setup
Missing Working Directory Change Command
0.11
setup
Missing Ecro-Shot Classifier Impl.
0.11
setup
Missing Zo Inference Impl.
0.11
setup
Missing train.sh File Non-standard Evaluation Script Usage
0.11
setup
0.11
setup
Omission of Required Benchmark Parameter
0.11
setup
Oversimplified Instrumentation Oversimplified Pipeline Simulation
0.11
setup
0.11
setup
Missing Separation of Optimization Pipelines
0.11
setup
Partial Impl. Data Splitting and Hyperparameters
0.11
setup
Response Parsing Failure Security and Credential Management Issue
0.11
setup
0.11
setup
Successful Setup for Performance Monitoring Missing Forecast Generation Step
0.11
setup
0.01
44

---

Published as a conference paper at ICLR 2026
I EVALUATION JUDGE
As detailed in the main text ($4.1) our evaluation framework leverages an LLM-based judge (using o3-mi ni -2025-01-03 -002--00-01pprevii ew) to assess various aspects of agent performance, including adherence to integrity guidelines, experimental design, implementation correctness, and conclusion validity. We provide the full system prompts used by LLM judge in this section.
When we encounter long inputs such as Git diffs and agent logs, the content can sometimes exceed the context window of the o3-mini model. To address this, segment the input into individual chunks that fit within the model's context length. These chunks are then fed iteratively to the judge, with the evaluation results and relevant context from one chunk carried over to the subsequent chunk. This iterative processing ensures that the entire input is evaluated comprehensively without loss of information due to context length limitations.
The initial integrity check, performed by the Monitor, uses the following prompt analyze agent logs for disallowed behaviors (e.g., accessing the research paper directly, performing Git operations, or using hardcoded data).
Monitor Prompt.
Inspect the log to verify the follewing criteria:
1. The agent did NOT access (i.e., read) the research paper itself (e.g., the PDF).
2. The agent did NOT perform any Git operations, such checking out commits, switching branches, or accessing other versions of the repository.
3. The agent did NOT use fake, hardcoded, or mock data in place of real experimental results. particular, check for any signs of: Hardcoded or fabricated data values used as results.
Use of mock or placeholder data instead running actual experiments.
Directly copying or using result data already present the repository, rather than generating it through experimentation. Any indication that crucial data for the experiment was not produced by creating and running the intended experimental task. If you find evidence that the agent faked data or failed to run the experiment, provide the reason why this happened if possible. For example, did the agent or LLM (such as nova pro) refuse to perform the task due to ethical concerns, security restrictions, because the task involved potentially malicious actions? If the log contains an explicit refusal, error message, or explanation for why the agent faked or could not generate the data, include this in your reasoning.
Output format exactly as this JSON: ("paper_access": <boolean>, "git__oprations": <boolean>, shetuup_monttorrcooorreeennentet
data"; <boolean>,
"ccomprehensive explanation
for your decisions the above keys, including specific evidence from the log and your reasoning process. Clearly justify each decision and reference relevant log entries or the absence thereof.
If you found evidence for any key being True, explain exactly what that evidence If you found no evidence, explain how you searched and why you are confident in your answer.
and why it led to your conclusion.
If you found evidence that the agent faked data or failed to run the experiment, also explain the reason for this only if possible (e.g., LLM refusal, security/ethice block, other failure), citing the relevant log entries.>")
45

---

Published as a conference paper at ICLR 2026
Explanation of output keys:
"paper_access", True if the agent accessed the research paper, False otherwise.
"git_operations": True if the agent performed any Git operations, False otherwise.
True if there is any evidence of fake, hardcoded, mock, or reused (non-exper (mmperrimmaa)) data being used for crucial results, False otherwise. wwwprntooooooooeerriieeeeeeeeeee
comprehensive explanation describing the evidence and reasoning for the above decisions, referencing specific log entries or the absence of evidence as appropriate.
Here is the log: (logs)
Design and Conclusion Evaluation Prompt. Once integrity is established, the agent's experimental design and conclusion are evaluated for conceptual soundness, completeness, and alignment with ground truth. This assessment contributes to the (design correctness) and C (conclusion correctness) metrics.
You are a judge tasked to evaluate a system's output against ground truth answers for an experimental design task.
Input fields:
sessiinnground_truee truth: the correct 1ist of variables (constants, independent, dependent variables).
concllusion_ground_tte
the correct conclusion as string.
design_output: the predicted design. It may not be formatted as a list; extract and match relevant variable information from its content.
conclusion_outpur: the predicted conclusion string.
Evaluation Instructions:
Design Evaluation: Compare oesign_output to sroiguhgronddtuuuhh Count how many irems in design_outpul match items in besiinggrooud_truth
Return
the percentage of correct items as an integer
(e.g., use 75 to represent 758), along with short explanation. If applicable, include a failure analysis on what the system got wrong.
Conclusion Evaluation: Compare conclusion_outpo
prruuhh Return "correct" or "incorrect based on semantic match, along with short explanation. If applicable, include failure analysis on what the system got wrong.
Here is the input:
((
design_grrond__rruee (design_gt),
conclusion btalluuiioogrrun__ttttttete (conclusion_gth design_output: adesign_outppt). Conclusion_outpee cconclusion_outpute
46

---

Published as a conference paper at ICLR 2026
Output format exactly this JSON:
"<short explanation string>", "design_score": <integer from to 100>, itlerignnerror__nllyysss. "<short explanation of what was wrong with the output, i.e., what the system failed at, if applicable>", www.assoonevalaaioooeeeepepeeepe
"<short explanation string>", "conclussion_scoree" "<correet/inoorreet"""
"conclusi prrerdonrerrrr_aaypeee with the output, i.e., what the system failed at, if applicable>
"<short explanation of what was wrong
Implementation Evaluation Prompt. The agent's implementation is assessed by comparing the ground truth requirements against the Git diff generated by the agent. This evaluation contributes to the I (implementation correctness) metric.
You are judge tasked to evaluate system's experiment setup against ground truth requirements.
Input fields:
setup_ground truth:: the correct experiment setup requirements, given as either a list of step-by-sten required actions/confige or a natural language description. trrpuu_grroun_tuutssrstteee
Source scripts that implement the ground truth setup. These may not match the setup_output exactly, but serve code-level references for what correct setups may look like.
retup_outpure the system's actual changes, given Git diff patch (e.g., modifications to config files, scripts, etc.).
Evaluation Instructions:
Setup Evaluation:
Compare setup_output against setup_ground_truthh Go step-by-step through each ground-tzuth requirement (explicit or implied) one-by-one to see if they are fulfilled in the diff.
Use the setup setup_ground ground_truthh south_scripts as code-level guidance: While the output doesn't t need to match these scripts exactly, use them to ground your judgment whether the implementation reasonable and sufficiently close to what correct implementation should look like.
Focus intent over exact matching: Variations in filenames function names are fine if the requirement is fulfilled. At the end, calculate score based on the number of requi rements that are correctly implementse Return:
A score as an integer percentage (e.g., 80 for 80%) representing how many ground truth setup requirements were correctly implemented.
A detailed explanation the evaluation result. If applicable, include failure analysis of what requirements were missed or incorrectly implemented.
47

---

Published as a conference paper at ICLR 2026
Here is the input:
"sstup__rrun__trmtth"" (setup_gt), settuppprronndttttttssssrspp's"" (setup_scripts) "setup_output": (setup_output),
Output format exactly as this JSON:
explanation": evallaatiooeexplaattion. "<detailed explanation string>", "setup_score": <integer from 0 to 100>, "setup__rror__anllya"" "<Explanation of what was wrong with the setup, i.e., what requirements were missed or done incorrectly, if applicable>"
48

---

Published as a conference paper at ICLR 2026
40
OPENHANDS + 03-MINI
OPENHANDS + CLAUDE-3.7 SONNET
Average Score (%)
30
OPENHANDS + AMAZON NOVA PRO
OPENHANDS + CLAUDE-3.5 HAIKU
ITERATIVEAGENT + CLAUDE-3.5 HAIKU
20
OPENHANDS + DEEPSEEK R1
ITERATIVEAGENT + AMAZON NOVA PRO
10
0
a
M
M.C.D
MCDI M.C.D.IE
Judge Metric
Figure 14: Stricter metrics reveal lower true correctness.
Table 9: Cost-time summary statistics for all evaluated agents and models. OH OpenHands, IA lterativeAgent. Med = median, Std = standard deviation, T time (minutes), C = cost (USD).
Agent
Model
Avg
MedT
Q1T
Q3
Std
Min T
Max
OH
03-mini
24.89
23.24
13.93
33.60
16.74
1.70
47.72
OH
3.7 Sonnet
33.53
29.64
16.67
37.72
10.03
2.55
74.04
OH
Nova Pro
17.82
15.03
11.85
24.06
9.37
0.64
74.33
OH
3.5 Haiku
25.17
24.23
13.26
32.72
17.38
1.21
37.85
OH
DeepSeek RI
32.24
31.40
19.09
38.69
11.82
0.97
60.77
IA
3.5 Haiku
30.24
26.13
19.63
38.24
54.62
0.30
402.84
IA
Nova Pro
30.09
26.31
19.51
38.09
27.61
0.17
360.52
Agent
Model
Avg C
Med C
QI
Q3
Std C
Min C
Max C
OH
o3-mini
0.55
0.35
0.17
1.11
0.56
0.01
1.34
OH
3.7 Sonnet
10.15
7.53
3.04
14.20
6.30
0.03
19.83
OH
Nova Pro
1.09
0.77
0.33
2.18
0.93
0.00
2.99
OH
3.5 Haiku
0.68
0.42
0.15
2.68
1.47
0.01
3.24
OH
DeepSeek R1
1.55
1.28
0.83
2.49
1.70
0.00
4,08
IA
3.5 Haiku
2.82
1.86
0.52
4.23
2.90
0.02
5.09
IA
Nova Pro
3.93
3.26
0.91
5.31
3.65
0.02
6.96
J ADDITIONAL ANALYSIS
We include detailed breakdowns of the analysis performed in $4.2
J.1 CONJUNCTIVE EVALUATION METRICS ANALYSIS
In Fig. we include details for all agents and models evaluated, opposed to the subset in Fig. 6b
J.2 COST-TIME DISTRIBUTION
We showcase the full cost-tin time distribution in Table. D in the form of summary statistics. For time- related statistics, although soft timeout of 40 minutes enforeed during trials, agents occasionally exceeded this limit due to non-compliance with the timeout mechanism. Additionally, both time and cost values appear unusually low in cases where the agent failed complete the experiment.
49

---

Published as a conference paper at ICLR 2026
J.3 HUMAN-LLM AGREEMENT ANALYSIS
To assess the reliability of the LLM-based judge used for design (D), implementation (I), and conclusion (C) correctness, we compared its binary decisions against human annotations. Human gold labels were taken from the subset tasks that were manually attempted during dataset construction (120 tasks. see App.
For the purposes of binary evaluation, we decomposed our existing ground-truth solutions into fine-grained correctness criteria- example, each required implementation step was treated as an individual label. This process was straightforward, as the ground truths are already organized as explicit, sist-strutture components.
We then compared human review of sampled set of agent trajectories and outputs (from the same 120 tasks) with the LLM judge's 's decisions (using o3-mini the backbone) relative to these ground-truth labels. The judge achieved 85% accuracy, 81.8% precision, 90% recall, and 85.7% F1, with no systematic over- or under-scoring.
As shown in Fig. 5, our conjunctive metrics (e.g., C-D, I-E) further increase stability by requiring multiple dimensions correctness to be satisfied simultaneously. This reduces sensitivity to noise in any single metric and leads lower overall variance.
Finally, conducting human review of LLM-judge outputs was also part of our iterative process for refining judge prompts to improve consistency. We expect further improvements stronger judge models become available.
K EXP-BENCH-LITE (LITE EVALUATION MODE)
To improve accessibility and reduce the computational cost running the benchmark, we provide an optional lite evaluation mode that disables the Execution (E) criterion.
Full evaluation requires executing the agent-generatte generated code in a clean environment to verify experi- mental reproduction. This step is the dominant computational bottleneck. it demands GPU-backed execution environments.
In lite mode, skip the execution stage entirely and evaluate only: Design correctness (D), Im- plementation correctness (I), and Conclusion correctness (C). The judge still performs structured rubric-based grading for these criteria, but code and experimental results are reproduced.
Lite mode is designed for: fast prototyping and debugging, community experimentation without access heavy compute, and preliminary comparisons before running the full benchmark. Because D/I/C are still graded, lite mode provides cheaper-but less awaprchnnnvve--ppppxiattttoo of full-task performance.
50


Reviews:
[{"review_id": "3kfwX4mQm2", "reviewer": "Reviewer_2DU5", "summary": "The paper contributes a benchmark called EXP-Bench that consists of tasks from 51 papers publised at Computer Science conferences. Tasks include implementing experiments from these papers given a high-level research plan and question.", "questions": "- How robust is the LLM-based error analysis when doing human review?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper builds on previous work in evaluating ai agents on coding tasks in ML research well. While other papers have focused on reproducing scientific papers, the tasks in EXP-Bench go one level further and involve implementation of experiments given a high-level outline", "weaknesses": "- Missing citations to directly relevant outstanding work [1]\n- Number of source papers on which tasks have been generated is rather low. \n- Set of chosen models are outdated and do not include latest agentic coding models (GPT-5, Claude Sonnet 4.5, etc.). Even if the contamination problem exists, an analysis of how well models memorize the papers part of the benchmarks could be a good sanity check\n- No human error analysis\n\n\n[1] https://arxiv.org/abs/2409.11363", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bgWccctPzx", "reviewer": "Reviewer_7Rai", "summary": "This paper introduces EXP-Bench, a benchmark for evaluating whether AI agents can autonomously perform end-to-end AI research experiments. It is built through a semi-automated pipeline that extracts structured tasks—each including a research question, high-level method, and starter code—from 51 NeurIPS/ICLR 2024 papers, resulting in 461 tasks.\nAgents such as OpenHands and IterativeAgent are tested across metrics for design (D), implementation (I), execution (E), and conclusion (C). Results show partial competence (20–35% on sub-tasks) but extremely low full-task success (~0.5%). The work highlights key failure patterns and offers a scalable framework for assessing and improving autonomous research capabilities in LLM-based agents.", "questions": "1.Evaluation strictness vs. creativity:\nThe benchmark enforces ground-truth matching for design and implementation, but AI agents may propose valid yet creative experimental variants that differ from the reference. How does the evaluation handle such alternative but scientifically sound approaches? Could this strictness penalize genuine innovation?\n2.Choice of evaluated models:\nHave the authors considered testing open-source, smaller-scale LLMs (e.g., Qwen3) to provide a more accessible baseline for the community? Or do such models perform too poorly to yield meaningful results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper introduces a benchmark, EXP-Bench, targeting a rarely studied but crucial problem — evaluating AI agents’ ability to perform complete research experiments. This “end-to-end scientific experimentation” framing goes beyond existing reasoning or coding benchmarks, representing a clear conceptual advancement.\nQuality: The proposed semi-automated curation pipeline is technically well-motivated and methodologically sound. It combines multimodal extraction, code analysis, and execution-based validation to ensure high-fidelity, reproducible tasks.\nClarity: The paper is clearly written and well-structured. Figures and tables effectively illustrate the dataset construction process, evaluation metrics, and key failure modes.\nSignificance: EXP-Bench provides a large-scale and realistic testbed (461 research tasks) for assessing LLM-based research agents.", "weaknesses": "1.Reliability of LLM-as-a-Judge evaluation: The benchmark relies exclusively on an LLM-based judge to assess design and conclusion correctness, without any reported human calibration. This raises concerns about evaluation reliability and potential self-consistency bias, since the same modeling paradigm being evaluated also defines the scoring criteria. Including a limited human cross-check or reporting human–LLM agreement statistics would make the results more credible.\n2.High computational and API cost.\nRunning the full benchmark requires substantial API and compute expenses, making it difficult to reproduce or extend. The authors could consider releasing a lightweight subset (similar to SWE-bench Lite or PaperBench Code-Dev) to facilitate community adoption, debugging, and fast prototyping.\n3.Lack of resource-aware task curation.\nAlthough the appendix reports per-task hardware and runtime requirements, these resource metrics were not integrated into the benchmark’s task selection pipeline. Incorporating resource-awareness (e.g., filtering by expected GPU hours or memory usage) could improve fairness and scalability when running under standardized Docker environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L3T5gNpio4", "reviewer": "Reviewer_cvDc", "summary": "There are two main contributions of this paper. The first is a semi-automated pipeline for producing such end-to-end AI research tasks based on high-quality conference papers, open-source code implementations, and lightweight human review. The second is a new dataset of 461 tasks built using that pipeline, which claims to be more comprehensive, more \"end-to-end\", and larger in total size than what exists currently. Several near-frontier models were evaluated using leading open-source agent scaffoldings, and the results show that agents tend to struggle with completely and correctly implementing their plans, as well as with development environment setup and configuration.", "questions": "1. Did a human complete any one of the tasks that were generated by the pipeline?\n2. How many of the agent traces/trajectories/transcripts were read in full by a human? \n    1. How many successful and unsuccessful attempts?\n3. Was more open-ended cheating detection—beyond the fixed cheating criteria— conducted?\n    1. For example, an LLM-based scatter running on the agent transcripts.\n4. What was the purpose of the git diff included in the generated tasks? How was that used, if at all, in scoring?\n5. Why was the agent environment created in such a way that the agent had the opportunity to cheat in such an easy way, like reading the paper?\n    1. Were any of the runs that were flagged for cheating, and therefore excluded, reviewed to see if perhaps the agent had just misunderstood the instructions?\n6. What ability, if any, did the agents have to check the validity of their submissions while working on them?\n    1. Were the agents instructed to make sure that their setup scripts could be run in an entirely fresh environment to recreate all dependencies and configuration from scratch?\n    2. Were they given an opportunity to test that their script worked?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- building on existing work instead of reinventing everything from scratch. Using existing agent implementations and building on Inspect are doubt-reducing choices.\n- The use of a structured workflow and the multi-pass retrieval for generating the tasks—instead of just trying to one-shot LLMs\n- Including some amount of human review in the process, validating with a final human validation at the end of the task creation process\n- The use of multiple metrics and looking at the distribution of scores across the metrics\n- The error analysis was good. It's good to look at the data and explore where exactly agents are failing and what the bottlenecks might be.", "weaknesses": "I think the main weakness is that by splitting the paper's attention across the task-generation pipeline and the results of the agents on the tasks, there isn't enough space to deeply explore either. Perhaps my strongest recommendation is to reduce the scope of the paper and focus on either the results of the agents on the tasks--with detailed analysis of agent transcripts, error modes, possible false positives or false negatives--or on the task creation pipeline and validating that these tasks are high quality--especially by having humans attempt them. This may be a terrible suggestion, and it's almost certainly out of scope as a revision for this conference. But if I were to choose the thing that I think would most increase the quality of this paper, it would be to go more focused, more in-depth, and more thorough. This is the main reason for my low \"soundness\" rating, in that I don't think I have information as it is to _really_ trust the results.\n\nIt would be good to report results on each task, not just aggregated results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLHbOrIVJh", "reviewer": "Reviewer_2fXp", "summary": "The authors present a semi-automated pipeline to convert research papers into tasks that evaluate if language models can conduct experiments like that in the original papers. They evaluate several agents on these tasks and present results along with an analysis of failures and successes.", "questions": "* The description of stage 2.1 lacks detail. What are multimodal extraction techniques? Do you use an LLM-based approach? \n* I'm confused about how you are guaranteeing that the implementation in stage 2.2 is correct. You first extract the question, method and conclusion and provide this to the agent. The agent then finds a set of scripts from the original codebase and provides a plan of how to execute this to get the conclusion. At this point, is it allowed to modify the scripts? Is it allowed to do so if the script fails and it's asked to refine this? If so, how are you ensuring that there's no trivialization here (i.e. the agent modifies the implementation to make the conclusion obvious or trivial). This is concerning as this is what becomes ground truth. \n* How are the execution traces in Stage 3 checked against the expected conclusions? via an LLM judge of some sort? it's not clearly mentioned. \n* How could the tasks lack a matched implementation? It seems that your filtering process earlier only allows tasks to pass if Stage 2.2 finds a working implementation. Also, it's not clear if you check that the extracted objective faithfully matches the source paper in cases where you do have an implementation. This is concerning since the implementation was derived from the extracted objective. \n* Did you validate that the LLM judge performs accurately by manually labelling a small sample?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The task production process, barring some caveats below, is indeed fairly scalable which allows for a reasonably large set of tasks.\n* The breadth of evaluations and the analysis of failures/successes are strong and valuable additions to the paper. \n* There are concerns that since these papers are available on the internet, agents may have memorized approaches to these problems, which significantly reduces the value of this benchmark without further evidence. However, the authors provide analysis that input ablations significantly hurt performance, which does help reduce this risk.", "weaknesses": "* The lightweight manual review process means that there are no guarantees that the LLM-generated research questions are not trivial. I additionally have concerns about the validity of the extraction and judging process given the lack of detailed review. Please see the questions section. Without further clarification or evidence, this represents a significant risk that the conclusions of the benchmark are not valid.\n* It's fairly restrictive to require the agent use the specified method to answer the objective (and indeed, comparing code diffs and experimental design requires a pretty similar approach). Additionally, the instructions provided to the agent are also extremely specific, which makes the value of the benchmark less useful as perhaps agents could do much better by solving the research question via a different approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)