ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery
Authors: Ziru Chen
Shijie Chen
Yuting Ning
Qianheng Zhang
Boshi Wang
Botao Yu
Yifei Li
Zeyi Liao
Chen Wei
Zitong Lu
Vishal Dey
Mingyi Xue
Frazier N. Baker
Benjamin Burns
Daniel Adu-Ampratwum
Xuhui Huang
Xia Ning
Song Gao
Yu Su
Huan Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Benchmark
Evaluation
Large Language Model
Language Agent
AI for Science
Code Generation
Task Automation
Primary area: datasets and benchmarks
TL;DR: We present ScienceAgentBench, a new benchmark for rigorously measuring progress towards developing language agents to assist human scientists in data-driven scientific discovery.

Abstract: The advancements of language language models (LLMs) have piqued growing interest in developing LLM-based language agents to automate scientific discovery end-to-end, which has sparked both excitement and skepticism about the true capabilities of such agents. In this work, we argue that for an agent to fully automate scientific discovery, it must be able to complete all essential tasks in the workflow. Thus, we call for rigorous assessment of agents on individual tasks in a scientific workflow before making bold claims on end-to-end automation. To this end, we present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery. To ensure the scientific authenticity and real-world relevance of our benchmark, we extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them. We unify the target output for every task to a self-contained Python program file and employ an array of evaluation metrics to examine the generated programs, execution results, and costs. Each task goes through multiple rounds of manual validation by annotators and subject matter experts to ensure its annotation quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns. Using our benchmark, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands, and self-debug. Given three attempts for each task, the best-performing agent can only solve 32.4% of the tasks independently and 34.3% with expert-provided knowledge. These results underscore the limited capacities of current language agents in generating code for data-driven discovery, let alone end-to-end automation for scientific research.

Published as a conference paper at ICLR 2025
SCIENCEAGENTBENCH: TOWARD RIGOROUS ASSESSMENT OF LANGUAGE AGENTS FOR DATA-DRIVED SCIENTIFIC DISCOVERY
Ziru Chen't, Shijie Chen'r Yuting Ning", Qianheng Zhang", Boshi Wang', Botao Yu', Yifei Li', Zeyi Liao', Chen Wei Zitong Lu', Vishal Dey!, Mingyi Xue Frazier N. Baker Benjamii Burns', Daniel Adu-Ampratwum', Xuhui Huang", Xia Ning 1.266, Song Gao3, Yu Su', Huan Sun
!Department of Computer Science and Engineering, OSU
"Colleee of Pharmacy, OSU
3 DDepartment of Geography, UW-Madison *Department of Psychology, OSU
DDepartment of Chemistry, UWW-Madison Depaartment of Biomedical Informatics, OSU
Website: https: petps://osuse /osu-n p grourr..p... github. io/ Sei enceAgent
ABSTRACT
The advancements of large language models (LLMs) have piqued growing inter- est in developing LLM-based language agents to automate scientific discovery end-to-end, which has sparked both excitement and skepticism about the true ca- pabilities of such agents. In this work, we argue that for an agent to fully automate scientific discovery, it must be able to complete all essential tasks in the workflow. Thus, we call for rigorous assessment of agents individual tasks in scien- tific workflow before making bold claims on end-to-end automation. To this end, we present ScienceAgentBenee for data-driven scientific discovery. To ensure the scientific authenticity and real- world relevance of our benchmark, we extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to vali- self-contained Python
a new benchmark for evaluating language agents
date them. We unify the target output for every task program file and employ an array of evaluation metrics to examine the generated programs, execution results, and costs. Each task goes through multiple rounds of manual validation by annotators and subject matter experts to ensure its anno- tation quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns. Using our benchmark, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompt- ing, OpenHands CodeAct, and self-debug, Given three attempts for each task, the best-performing agent can only solve 32.4% the tasks independently and 34.3% with expert-provided knowledge. These results underscore the limited capacities of current language agents in generating code for data-driven discovery, let alone end-to-end automation for scientific research.
INTRODUCTION
Large language models (LLMs) have shown remarkable capabilities beyond text generation, includ- ing reasoning (Wei et al. 2022. Yao al. (2023), tool learning (Schick et al.) 2023 Wang et al.) 2024a), and code generation (Chen et al.] 2021 Yang et al. 2024a). These abilities have piqued significant research interests in developing LLM based language agents to automate scientific dis- covery end-to-end. For instance, |Majumder et al.) (2024a) urge the community to build automated systems for end-to-end data-driven discovery, increasingly important workflow in many disci- plines (Hey et al. 2009) that leverages existing datasets to derive new findings. More recently, Lu et al. (2024) claim to have built The AI Scientist, agent that is capable of automating the entire search workflow, from generating ideas to running experiments and writing papers. This ambitious claim has sparked both excitement and skepticism about the true capabilities of such agents.
"Correspondence (chen. 8336,, chen. 10216, 6, sun 397 @osu.ed. edu

---

Published as a conference paper at ICLR 2025
Data Processing (23)
Model Development (23)
Data Analysis (59)
Info Visualization (65)
Feature Engineering (20)
Deep Leamingg (14)
Statistical Analysis (9)
Data Visunlization (45)
Feature Selection (2)
Machine Learning (9)
Geospatial Analysa (12)
Map Visuzlizazion (18)
Dara Selection (1)
Computational Amalycx (16)
Molccule Vistalization (2)
(b)
(c)
(d)
Figure I: Top: Distribution of sub-tasks in ScienceAgeniBenchl Each task in our benchmark con- sists of one or more of these sub-tasks and requires successful completion of all sub-tasks achieve the task goal. Bottom: Heterogeneous datasets involved: (a) cell image in Bioinformatics, (b)
molecular activity visualization in Computational Chemistry, (c) a flooding risk map Geographi- cal Information Science, and (d) EEG time series in Psychology and Cognitive Neuroscience.
In this work, we contend that for a language agent to fully automate data-driven discovery, it must be able to complete all essential tasks in the workflow, such as model development, data analysis, and visualization. Thus, we advocate careful evaluations of the agents" performance on these tasks, before claiming they can automate data-driven discovery end-to-end. Such an assessment strategy helps grasp a more solid understanding of an agent's strengths and limitations than purely relying on end-to-end evaluations, e.g., using an LLM-based reviewer to assess generated papers (Lu et al,, 2024). Yet, high-quality benchmarks focusing individual tasks in real-world seientific workflows are lacking for objective assessment and continued development of agents for data-driven discovery.
To this end, we present ScienceAgentBenchh new benchmark for evaluating language agents for data-driven discovery. The construction of ScienceAgentBenee follows three key design principles. (1) Scientific authenticity through co-design with subject matter experts: We ensure the authen- ticity of tasks in our benchmark by directly extracting them from peer-reviewed publications and engaging nine subject matter experts (incl. senior Ph.D. students and professors) from the respec- tive disciplines to validate them. This approach also minimizes the generalization gap for agents developed on our benchmark to real-world scenarios. In total, we curate 102 diverse tasks from 44 peer-reviewed publications in four disciplines: Bioinformatics, Computational Chemistry, Geo- graphical Information Science, and Psychology & Cognitive Neuroscience (Figure Dp. (2) Rigorous graded evaluation: Reliable evaluation for language agents is notably difficult due to the open- endedness and complexity of data-driven discovery tasks. We first unify the target output for every task as self-contained Python program, and then employ array of evaluation metrics that amine the generated programs, execution results (e.g., rendered figures or test set predictions), and costs. We also provide step-by-step rubrics specific each task enable graded evaluation. (3) Careful multi-stage quality control: Each task goes through multiple rounds of manual validation by annotators and subject matter experts ensure its quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns due to LLM pre-training.
We comprehensively evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands CodeAct (Wang et al. 2024c). and self-debug. Surprisingly, with- out expert-prov providee knowledge, Claude-3.5 3.5--onnee using self-debug can successfully solve 10.8% more tasks than using OpenHands CodeAct while costing 17 times less API fees. This result res- onates with recent findings that agent designs should jointly consider costs and performance to maximize their practical utility (Kapoor al. 2024). Still, given three attempts for each task, the best agent can only solve 32.4% of the tasks independently and 34.3% of them with expert providddo knowledge. These results also suggest language agents cannot yet automate essential tasks in data- driven discovery nor the research pipelines end-to-end, in contrast to claims in recent work such as Lu et al. (202)).
Despite their current mediocre performance, we believe language agents hold significant potential in augmenting human scientists' productivity: For each task in our benchmark, it takes trained
2

---

Published as a conference paper at ICLR 2025
Train mulkiiak model un the Cliniox dateect to predict drug's toxicity FDA appeoval status, Save the prodictions, inchading the SMILES ES representation drugs and the probubility of positive libel, to "prod_ resultsvilitiio test_ pred.cwv"
*On the usk The ClinTox dalasel contains drigs approved by *On feutuization; the molecular structure, use Extended Connectivity Fingerprints (FCFPs featurization doepelion.....
(c) Expert-Provided Knowledge
(a) Task Instruction
import deepchen as de
Dataset Directons
from deepchan. Lnne.n functior. functior.molaet. nolaet noldet_Leader Leader MolnetLoader
import
|- el clintox/
clintow train.cav
clees kyClintoxLoader ( Loblnettcederl: def czeate, dataset [solE):
Datasst Previcu:
[START Preview of cllttoiielitt train.cs]]
CLTNTOX PASRS [17DA APPROVED', 1CP POX!]
smi APPROVED,CT TOX
trein loader byClintoxioader ("BCEP', ...
trein dataset
Saaalp/geeaannncaaaeeeoeeeaaaane O0dIN/CGGGHHp0a0a3///--------000 -Dj,110
test Loader MyCLintoxloade: (1BCF?1,
test datasat
model do.nodele e. Hüllittekclaseeteer model. (train dataset)
C3)C,1,0
test scores model. predict Iteat, dataset, i..))
[END elintoccelinoo tmin.csv]
teat scores df, to cav ("pred resul resulta/el intox test tttpoottttttt__et___eeeeee praddcav"))
Dataset Information
(d) Annotated Program
Figure 2: An example Computational Chemistry task in ScienceAgentBench with four components.
annotator at least 2.5-3 hours average adapt existing program from public sources, and potentially much longer for a subject matter scientist to write the program from scratch. In contrast, language agent usually generate meaningful program draft within 10 minutes. In the long run, ScienceAgentBench will serve as benchmark for rigorously measuring progress toward developing language agents to assist scientists in data-driven scientific discovery.
2
SCIENCEAGENTBENCI
In this section, we introduce ScienceAgentBench, which aims to evaluate agents on essential tasks in data-driven discovery workflow. Before automating the entire workflow end-to-end, we envi- sion language agents to first serve as science co-pilots that can write code to process, analyze, and visualize data. Similar co-pilots for software development, we target scientist users who might know how to write such code but want to save hours of programming effort with language agents. Hence, formulate each task code generation problem, whose output easily verifiable and directly usable by a scientist without additional modification efforts.
2.1 PROBLEM FORMULATION
Given a natural language instruction, dataset, and some optional expert-provided knowledge, an agent shall generate program to complete the assigned task and save Python source code file. Each instance in our benchmark contains four components (Figure
(a) Task Instruction, which describes the goal of an essential task in data-driven discovery and its output requirements. To resemble real-world settings, we keep the instructions concise and avoid unnecessary details when describing task goals. This setup also retains the open-endedness of data- driven discovery and encourages the development of practical agents that do not rely on prescriptive directions from scientists. We provide example task instructions in Appendix forr each discipline.
(b) Dataset Information, which contains the dataset's directory structure and preview of its con- tent. For agents without file navigation tools, they need such information to correctly use the dataset in their generated programs. For agents that can navigate file systems, it also helps them save a few turns of interactions to read datasets from the programming environment.
(c) Expert-rrovided Knowledge, which includes explanations for scientific terms, formulas to con- duct analysis, and example usages of programming tools. These pieces of knowledge are provided by subject matter experts, including senior Ph.D. students and professors, and are optional inputs to an agent. Section 4 we show that while with such information, language agents' knowledge gap in involved disciplines be mitigated some extent, they still fall short utilizing it effectively.
3

---

Published as a conference paper at ICLR 2025
(d) Annotated Program, which is adapted from an open-source code repository released by peer- reviewed scientific publication. As shown in Figure 2 each program is self-contained with package imports, function and class implementations, and a main procedure to carry out the task. An agent is expected to produce similar programs that can be executed independently, e.g. by a Python inter- preter, but not necessarily using the same tools those in the annotated programs.
2.2 DATA COLLECTION
Task Annotation. We start by forming a group of nine graduate students to annotate the tasks in four disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience. Within each discipline, we search for peer-reviewed publications that release their code and data under permissive licenses (Appendix[J.. Then, we follow five steps to annotate each task: (1) Identify a reasonably documented code example that is self- contained and convert it into task in our benchmark. (2) Collect and preprocess datasets used in the code. (3) Annotate the reference program by revising the referred code to analyze datasets in benchmark. (4) Implement task-specific success criteria an executable script and GPTT-4o to draft fine-grained rubrics for evaluation. (5) Write the instruction and dataset information for this task. We gathered 110 tasks initially but discarded four because their programs require long execution time or nontrival environment setup. This leaves us with 106 tasks for validation.
Data Contamination and Shorteut Mitigation. In our preliminary studies, we have noticed that some agents, such as OpenHands, may take shortcuts to solve task. For example, when asked to develop a machine learning model, they may directly read and report the ground-truth labels in the test set without writing the training code. Such perfect results are actually cheating and will hurt evaluation validity. In addition, because datasets and programs in our benchmark are open-sourced, they are subject to data contamination in LLM training. To mitigate these issues, we devise two strategies to modify the datasets: (1) For each dataset, we randomly remove five data points from its test set. If an LLM-generated program uses automatic data loaders that appeared in the training corpora, it will produce results misaligned to our setup and fail the success criteria. In some cases, have to skip this step if it would break the completeness of dataset, e.g., if it results in an incomplete geographical map. (2) For tasks involving model development, we re-split the dataset, keep the test set labels only for evaluation, and replace them with dummy values, such -1 for classification tasks. These two strategies effectively mitigate data contamination and agent shortcut concerns by failing agents that recite memorized code or attempt to directly report test set labels. See Appendix F2 Example| F.4 for case study.
Expert Validation. We engage nine subject matter experts. including senior Ph.D. students and pro- fessors from the four involved disciplines, to validate each task and provide additional knowledge. For each task, we present to experts with its instruction, dataset information, annotated program, and task rubrics. The experts are asked to validate the tasks by completing questionnaire (Appendix G). which can be summarized as four steps: (1) Validate if an annotated task represents a realistic task in their data-driven discovery workflow. (2) Review whether task instruction gives accurate high-level description of the program and uses professional languages in their disciplines. (3) Pro- vide up to three pieces of knowledge that might be needed for solving each task. (4) Make necessary revisions to the rubrics for grading the program. Then, following the experts' feedback, we revise 41 task instructions and remove three tasks that are not representative enough for scientific workflows annotation strategy is shown
in their disciplines. With 103 tasks remaining, our publication- to be effective in collecting real-world tasks.
Annotator Verification. To ensure data quality, we work with the nine annotators for another round of task verification. We ask the annotators to verify tasks that are not composed by themselves and execute programs to reproduce the results. During this process, we refine 29 task annotations and discard one more task whose result is hard to replicate with the same program due to randomness. We finalize ScienceAgemtBench with 102 high- tasks for data data-driven scientific discovery.
2.3 EVALUATION
While it is preferable feature, the open-endedness of tasks in our benchmark introduces cru- cial evaluation challenge. Specifically, our evaluation strategy has to accommodate diverse setup requirements of programs generated by different agents. To address this challenge, we implement
4

---

Published as a conference paper at ICLR 2025
Table I: Representative examples of task-specific success criteria in ScienccAgentBench. To keep the table concise, we omit output requirements in the task instructions and show the task goals. We provide more details about how these criteria are established in Appendix[D22
Task Instruction
Subtasks
Success Criteria
Train multitask model the Clintox dataset to predict drug's 's toxicity and FDA approval status.
Feature Engineering Deep Learning
The trained model gets 2 0.77 ROC- score on the test set.
Develop a drug-targer interaction mod- el with the DAVIS dataset to repurpose the antiviral drugs for COVID.
Feature Engineering
The top-5 repurposed drugs match the gold top-5 drugs.
Deep Learning
Analyze the inertial measurement unit (IMU) data collected during sleep and compute sleep endpoints: time of falling asteep. time of awakening, and total duration spent sleeping.
Computational Analysis
Each computed endpoint is close (math. i scl ose in Python) to the corresponding gold answer.
Analyze Toronto fire stations and their service coverage. Visualize the results to identify coveruge gaps.
Map Visualization
The resulting figure gets 2 60 score by the GPT-4o Judge.
a pipeline to set up a conda environment flexibly for any program. Before evaluation, the conda environment is initialized with seven basic Python packages: numpy, pandas, matplotlib, pytorch, tensorflow, rdkit, and tf _eras.. To evaluate each program, we first use pi pipreqs to analyze it and generate file listing all packages used. Then, according the file, we use pip-tool tools crafted rules to update the conda environment and properly configure the packages. We execute each program in the customized environment and calculate the evaluation metrics.
e and hand-
Program Evaluation. We comprehensively evaluate each generated program with four metrics. (1) Valid Execution Rate (VER) checks if the program can execute without errors and save its output with the correct file name. (2) Success Rate (SR) examines whether a program output meets the success criteria for each task goal (TableTT) such as test set performance, prediction-anseee aanser matches, and visualization quality. To automatically check these criteria, we implement them as evaluation programs for each task during annotation. By nature, SR is conditioned on valid execution: If a program has execution errors does not save its output correctly, its SR will be O. Both VER and SR are binary metrics. (3) CodeBERTSconm (CBS) measures how closely the generated program resembles the annotated one with contextual embeddings and calculates the F1 metric for matched token embeddings (Zhou et al. 2023). If = 1 for a program, we change its CBS to 1.0 as well to reflect task success. (4) API Cost (Cost) calculates the average cost (in USD) to complete one task in our benchmark, since it is important for language agents to control their cost and optimize their design for better practical utility (Kapoor al. 2024).
Figure Evaluation. If the task output is a figure, we follow existing work (wu et al. 2024 Yang al. 2024b) to evaluate its quality using GPT-4o as a judge. which is shown correlate reason- ably well with human raters. We use Yang et al. (2024b)ss prompt to request GPT-4o to compare the program- program-produced figure with the ground- -tuthh and respond with a score on its quality. For evaluation stability, we sample 3 responses and use the average score to compute success rates.
Rubric-Based Evaluation. Outcome-based evaluation metrics, which require a program to cor- rectly implement all steps for the task, can sometimes be too stringent. For example, an agent would be underrated by these metrics if it gets all steps right but output formatting wrong. As a comple- introduce rubric-based evaluation to assess the generated ment to the outcome-based metrics, programs at more fine-grained levels. Considering the characteristics of data-driven discovery tasks, structure the rubrics into five stages: Data Loading, Data Processing, Modeling Visualization, Output formatring, and Output Saving. To accelerate the annotation process, we first use GPT-4o generate the rubrics by designating multiple milestones with scores for the five stages. Then, each rubric is refined by an expert (Appendix H). In this work, we leverage the rubrics to conduct
httpss//ggtthobb.cmm//azaaaaaa
5

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of BeienceAgentBench to representative benchmarks. DiscoveryBench Real is evaluating the quality of generated programs indirectly through the natural language hypothesis, while SeienceAgentBenchi 's focus is to rigorously assess the programs and their execution results.
Code Gen
Task
Heterogencous
Shortcut Prevention
Scientific Suhjects
Test
Benchmark
Complexity
Sources
Data Processing
Tasks
TaskBench Shen
2024
No Code Gen
Synthtic
28,271
SWE-Bench Jimency eral
2024
Fite-Level Edit
GitHub
X
2,294
BioCoder-Py
(lanc al
Function-Lewe
GilHub
ML-
TTan m. 202418
Line Level
Galllub
X
260 ]
MI. MI-Agenfeecch [Hiang el
2024b
File-Levee Fdit
Kaggle
Majumddr al. 2024h
BiecovryBBench-Bee Real
Code Gen!
27 Publications
X
6
239
SeCCode [Tab of n A02
Function-Level
Publications
/
5
80
BLADE
Function-Lovel
31 Publications
12
ScienoeAAemiBench (Ours)
File-Lvel Gen
44 Publications
102
human evaluation for generated programs (Section 4.2). We deem that automating this rubric-basee evaluation approach, such as developing an L.L.Mbbssed judge, meaningful future direction.
2.4 COMPARISON WITH EXISTING BENCHMARKS
ScienceAgentennch differs from other benchmarks with a unique ensemble of research challenges (Table 2) (1)Tasks in benchmark require agent to generate a standalone program file from scratch, in contrast to JSON API calls in TaskBench, abstract workflow descriptions in Discovery- edits in other benchmarks. To do so, agent needs
Bench, or a few lines of code completion to have a deep understanding of the task, decompose it into classes and functions appropriately, and implement them. (2) Our benchmark adapts 44 peer-reviewed publications and covers variety of real-world datasets in four different disciplines. Compared to ML-Bench and DiscoveryBench. our ScienceAgentBentt includes more heterogeneous datasets that have complex structures (Figure [.. such as cell images, chemical structure structure-activity activity relationships, and geographical maps with multiple layers. (3) ScienceAgentBench is also one of the two benchmarks that tries to mitigate data contam- ination and agent shoricut issues, which helps establish valid evaluation. (4) Our benchmark has medium scale of 102 tasks. Although smaller than benchmarks with synthetic or easier tasks, this scale is reasonable to evaluate agents, considering the annotation difficulty and evaluation cost.
3 EXPERIMENTAL SETUP
We experiment with three weight LLMs, and Mistral-Large (123B) [MistralA] 2024), and two proprietary LLMs, GPT-4o (OpenAl 2024) and Claude-3.5-Sonnet (Anthropic, (024). For all experiments, we use the same hyperparameters, temperature 0.2 and top-p- 0.95, and perform 0-shot prompting via the APIs. The prompts are We evaluate the LLMs under three different (agent) frameworks:
405B (Dubey et al. 2024).
included in Appendix |
Direct Prompting. Direct prompting is simple framework that does not interact with any program- ming environment. Given the task inputs, it prompts an LLM to generate a corresponding program in one pass. We use this framework to show the basic code generation capability of each LLM.
OpenHands CodeAct. OpenHands 4 Wang et al. 2024c is a generalist agent development frame- work for code generation and software engineering. It provides three kinds of tools in the environ- ment and defines different actions for an agent to interact with the tools: Python code interpreter, bash shell, and web browser. Among the agents developed with OpenHands, we use the best- performing CodeActAgent v1.9 (Wang et al. 2024b) in our experiments. This agent unifies all actions in OpenHands, including the agent-computer interface commands (Yang al.] (024a) to read and edit local files, into large action space of different Python API calls. We experiment with the CodeActAgent v1.9 using different LLMs to test the effectiveness of OpenHands' framework
'OpenHands has built-in I-shot example to demonstrate response formats, tool usages, and other plugins like web browser. We do not provide any examples from our henchmark when evaluating OpenHands. https: /ectps:/giin thub. com A1 H / originally named OpenDevin.

---

Published as a conference paper at ICLR 2025
design for code generation tasks in data-driven discovery. For simplicity, shorten the name of this agent framework as OpenHands CodeAct.
Self-Debug. Self-debug (Chen et al. 2024a) is code generation framework for LLMs to execute their generated programs, access exccution results, and then reflect on the results to improve each program iteratively. In this work, we re-implement self-debug with three modifications. First, do not instruct the LLMS to generate reflections before debugging the code, since self-reflection may not always yield better results (Chen et al. 2024b. Huang et al.) 2024a] Jiang et al.) 2024). Second, we allow early exits if the backbone LLM generates the same program for two consecutive debugging turns. Finally, before rumning each program, we use pipreqs and pip-tools up the environment. We do not initialize the self-debug environment with any of the basic packages or provide the rules to configure some packages that are used for evaluation (Section 2.3 Even though self-debug might not be able to use some packages due to this design choice, we want to ensure fair comparisons with other baselines, which also have no access to these information.
set
To improve evaluation stability, we repeat each task with three independent runs in all experiments. Then we select the best run according to the metries in the following order: maximum SR, maximum VER, maximum CBS, and minimum Cost. We refer to the next metric in this order to break ties. For example, two programs generated for a task both have SR = O, we pick the one with higher VER. Finally, report each metric based the average performance of selected runs. We also include the mean performances out of three runs and standard deviations in Appendix
4 RESULTS AND ANALYSIS
Through comprehensive experiments (Table 3). we show that the latest LLMs and agents can only achieve low-to-moderate task success rates. Given three attempts for each task, -laude-3.5-Sonne with self-debug demonstrates the best performance (34.3% SR) when using expert-provide knowl- edge. This result underline that LLM-based agents are not yet capable of fully addressing realistic and challenging data-driven discovery tasks, such as those in ScienceAgentBench.
4.1 MAIN RESULTS
Direct Prompting vs. Self-Debug: Execution feedback is necessary for LLMs to generate use- ful programs. As shown in Table directly prompting LLMs cannot unleash their full potential in programming for data-driven discovery tasks. Without executing its code, even the best performing LLM, Claude-3.5-Sonnet can only solve 16.7% of the tasks independently and 20.6% with addi- tional knowledge. For most failed tasks, we share similar findings with [Lang et al.] (2024) that LLM-generated programs have correct high-level structures but implementation--eeee errors, such as missing steps wrong API usage. Compared to direct prompting, self-debug can nearly double Claude-3.5-Sonnet's success rate (16.7 - 32.4; 1.94x) without extra knowledge. With expert- provided knowledge, Claude-3.5-Some 3.nnnt using self-debug also shows decent improvement over di- rect prompting. It achieves 13.7 absolute gains on SR (20.6 - 34.3; 1.67x) and 45. absolute gains on VER (41.2 2 - 86.3; x..9x). These results highlight the effectiveness of the simple self-debug framework and the importance of enabling LLMs to execute and revise their code for complex tasks.
OpenHands CodeAct vs. Self-Debug: Agent designs should consider costs and capabilities of LLMs. For four of the five LLMs evaluated, self-debug demonstrates better performance than OpenHands CodeAct, with GPT-4o the only exception (Table 33. By examining the trajectories, we find that GPT-4o is better at leveraging tools in OpenHands than other LLMs. For instance, it the only LLM that search for more details about the provided knowledge with the web browser. In contrast, other I LLMs are still struggling with specialized bash commands in OpenHands to edit programs correctly (Example in Appendix F.T). We hypothesize that GPT-4o may have been trained to better follow instructions for language agents and to better use complex tools like web browser.
When it comes to self-debug, which has a more straightforward design, GPT-4o loses its advantage and underperforms Mistral-Large-2 and Claude-3. 3.5-ude-3.--oomee --nnnnn,, both of which are trained for better code generation according to their reports [MistralA] [2024] Anthropic 2024). Most surprisingly, without the help of expert-provided knowledge, 10.8% more tasks (21.6 + 32.4 SR) than using OpenHands while costing 17 times less API fees ($0.958 $0.057), which is critical factor to consider for practical applications. Overall, our
3.sude-3.--oomae using self-debug can successfully solve
7

---

Published as a conference paper at ICLR 2025
Table Results on SeienceAgentBench. The best performances (with and without domain knowl-
edge) for each framework are in bold. The overall best performances for each metric are underlined.
Without Knowledge
With Knowledge
Models
SR
CBS
VER
Cost +
SR
CBS
VER
Cost 4
Direct Prompting
Llama-3 3. I--nnntttt--00
5.9
81.5
29.4
0.001
4.9
82.1
27.5
0.001
Llama-3. 1.nt-....Intttt-4400000
3.9
79.4
35.3
0.010
2.9
81.3
25.5
0,011
Mistral-Large-2 (2407)
13.7
83.2
47.1
0.009
16.7
84.7
39.2
0.009
GPT-4o
11.8
82.6
52.9
0.011
10.8
83.8
41.2
0,012
Claude-3.5 3.5ee-3.5-Sonet
17.7
83.6
51.0
0.017
21.6
85.4
41.2
0.017
OpenHands CodeAct
Llama-3 1. --nnsttruct-008
6.9
63.5
30.4
0.145
2.9
65.7
25.5
0.252
3lama-3.11Insse 1055
5.9
65.8
52.0
0.383
8.8
71.4
58.8
0.740
Mistral-Large-2 (2407)
9.8
72.5
53,9
0.513
13.7
78.8
50.0
0.759
GPT-4o
19.6
83.1
78.4
0.803
27.5
86.3
73.5
1.094
Claude-3.5-Somae
21.6
83.6
87.3
0.958
24.5
85.1
88.2
0.900
Self-Debug
Llama-2 11.--nntruuct700
13.7
82.7
80.4
0.007
16.7
83.4
73.5
0.008
Llama-3 3.1-Instruct-4053
14.7
82.9
78.4
0.047
13.7
83.6
79.4
0.055
Mistral-Large-2 (2407)
23.5
85.1
83.3
0.034
27.5
86.8
78.4
0.036
GPT-4o
22.6
84.4
83.3
0.047
23.5
85.6
71.6
0.046
Claude-3.5 3.5-o.nnnt
32.4
86.4
92.2
0.057
34.3
87.1
86.3
0.061
results resonate with recent findings on agent design (Kapoor al., 2024 Xia et al. 2024): (1)
LLM-based agents do not always benefit from a large action space with complex tools; and (2) both cost and performance should be considered when designing or selecting agent frameworks. We also provide detailed eтror analysis in Appendix E.2 and identify future research directions.
With vs. Without Expert-Provided Knowledge: Expert-provided knowledge does not always lead to metric improvement. On one hand, we observe that expert-provided knowledge leads to consistent improvements on SR and CBS for most agents (Table 3). These agents can effectively leverage helpful information in the knowledge, such as API names and some concrete steps in the task, to generate a high-quality program draft that closely resembles the annotated gold program and then use execution feedback to address implementation errors.
On the other hand, we notice that there are performance decreases on VER for most agents. These decreases can be attributed two reasons. (1) Expert- knowledge specifies some specific tools that are less familiar to the agents. Originally, they would only use basic tools like rdkit and sklearn in their generated programs, which free of execution errors. With provided knowl- edge, the agents would use those specified tools to generate programs, which often contain incorrect API usage and hallucinated API calls. (2) The agents do not know how to solve some tasks without expert-provided knowledge and would generate some executable but less meaningful programs, e.g., produce an empty figure. While additional knowledge helps them produce more concrete mod- eling or analysis, such programs are error-prone and hard to fix with execution feedback (Appendix F.2.. For these reasons, despite decreases VER, we argue that expert-providee knowledge helps agents to generate more useful programs from scientist user's perspective, reflected by SR and CBS, and future Al agents should improve their abilities to better leverage such information.
Language agents cannot solve complex data-driven discovery tasks yet. Our further analysis on the best performing agent, Claude-3.5- with self-debug and expert-provided knowledge, show that it is not yet capable of addressing complex tasks in data-driven discovery. To estimate the complexity of tasks, we visualize the number of lines in their corresponding gold programs using box plot (Figure 3 Left). More than 75% of succeeded tasks lean to the simpler side because their gold programs have less than 58.6 lines, which is the mean length of all gold programs in the benchmark. In other words, language agents still fail many tasks with complex gold programs.
8

---

Published as a conference paper at ICLR 2025
0.40
Data Processing
Development
Data Arulysis
Info Visualization
Success Label
0.30
0.25
Emrr
0.20
0.15
0.10
0.05
20 40 60 80 100 120 140 160
0.00
Bio
Chem
Psy
Number Lines Gold Program
Task Disciplines
Figure 3: Task performance analysis of Claude-3.5-Sonme with self-debug and expert-provided knowledge. Left: Distribution of lines in gold programs for succeeded and failed tasks. The red vertical line marks the average length (58.6 lines) of all gold programs in the benchmark. Right: Task error rates for each sub-task category in each discipline.
To understand the task failures, we break them down by different disciplines and sub-task categories (Figure 3 Right). For Bioinformatics and Computational Chemistry, the agent mostly fails on tasks involving data processing and model development. This is because data in these highly heterogeneous, including cell images, molecules, and genes, which can be hard to process. Without correctly processed data, the agent would also not be able develop and train a functioning model, not to mention choosing appropriate configurations for various models such as Convolutional or Graph Neural Networks used in the tasks. For Geographical Information Science and Psychology & Cognitive Neuroscience, their tasks usually require besccppiieesseciff tools, such as Geopandas and Biopsykit, to analyze the datasets. However, existing LLMs fall short using these tools and can generate incorrect or hallucinated API usage in the programs. Given these shortcomings. we argue that current language agents cannot yet automate data-driven discovery tasks or the full research pipeline, in contrast claims made in recent work such as Lu et al. 2024).
disciplines
4.2 HUMAN EVALUATION
Evaluation Setup. To further investigate the performance of Slaude-35-Somma with self-debug (the best-performing agent), we conduct rubric-based human evaluation of all the 102 programs generated using expert-provided knowledge. With the task-specific rubrics validated by experts (examples in Appendix [H] and gold programs as references, each generated program is rated by two different evaluators who participated in data collection. To reduce possible noises in ratings, the evaluators only mark whether rubric item is met by the LLM-generated program. For each stage, we add up points for satisfied rubric items and normalize them by total available points to the range of 8-100. Similarly,
calculate the overall score considering all items. The final score of each program is the average of two evaluators' ratings.
Additionally, purpose of this human evaluation is to assign partial credits the generated pro- gram even if not correct (Section 2.3c). Therefore, we do not provide the evaluators with program execution results and hide task success outcomes. Although this setup encourages evaluators ex- amine LLM-generated programs carefully, it also introduces some noise. For example, there are tasks where both a feed-forward neural network and a random forest model can achieve satisfying performance on the test set. While the gold program implements the neural network, the agent chooses to use random forest. Since each rubric is derived from a gold program and reflect its implementation, there are chances that the evaluator overlooks such equivalence. Also, for output formatting, we observe some subjective variance when judging the formats figures, such as colors, scales, and text labels, according the rubrics and gold programs. As a result, successful programs would not always receive perfect human rating.
Results and Analysis. As shown in Figure data loading and processing, the first two stages in data-driven discovery tasks, can distinguish successful programs from failed ones. Except for a few outliers, almost all successful programs receive perfect human rating for data loading. In contrast, 25% of the failed programs have their rating below 50 the first stage. For data processing, the rating distribution of successful programs skews toward the full score, while that of failed programs

---

Published as a conference paper at ICLR 2025
Loading
Processing
Visualization
Ourput formatting
Output Sawing
Overall
100
Averoge Human Rating
40
0
0
0
Success
Figure 4: Rubric-based human ratings for 102 programs generated by Claude-3.5 3.5-onneet with self- debug and expert-provided knowledge. We show the overall distributions and those the five stages in our rubrics (Section 2.3). The blue boxes are distributions for failed tasks, and the orange ones are for succeeded tasks. The open dots represent outliers in the distributions.
skews toward score between 20 and 50. These human evaluation results correspond to intuitive explanation: the dataset were not loaded or processed correctly, it would be impossible to solve task successfully, regardless of the code implementation for consequent stages.
In the third stage, modeling or visualization, human ratings for successful and failed programs are also different: The median score of successful programs is already at the 75th percentile of failed program ratings. This indicates that human evaluators agree with the SR metric and prefer programs passing all success criteria for the task, even though they may have some minor issues. For output formatting and saving, we find no difference between the two groups of programs, indicating that LLMs like Claude-3.5-Sonne can follow such instructions reasonably well.
Overall, human ratings for succeeded and failed programs form two overlapped but distinguishable distributions, which meets our motivation complement outcome-bass metrics with fine-grained evaluation. These ratings agree with our main result and suggest that some LLM-generater programs are close to success but hindered by some bottlenecks, such as data loading and processing. Future research may, for example, improve language agents' capability better process scientific data.
5 CONCLUSION
We introduce ScienceAgemtBench, a new benchmark to evaluate language agents for data-driven scientific discovery. We compile 102 diverse, real-world tasks from 44 peer-reviewed publica- tions across four scientific disciplines and engage nine subject matter experts to ensure data quality. Through comprehensive experiments on five LLMs and three frameworks, we show that the best- performing agent, Claude-3.5-Sonnee with self-debug, expert-provided knowledge. Our results and analysis suggest that current language agents cannot yet automate tasks for data-driven discovery enccAgentBench. we advocate the use of language agents to assist human scientists with tedious tasks in their workflows and call for more rigorous assessments of such agents.
only solve 34.3% of the tasks when using
whole research pipeline. By introducing Sci-
ETHICS STATEMENT
Our benchmark is constructed by adapting open-source code and data, to which we respect their creators' ownership and intellectual property. In Appendix [ we have made our best effort cite the original papers, list the repositories, and provide their licenses. Still, we acknowledge that two repositories are copyrighted and believe their terms for use are compatible with our research purpose (Table J.4 J.5 We welcome requests from the original authors modify remove relevant tasks needed.
Mcanwhile, agents developed with ScienceAgentBencch should consider potential safety issues in deployment, especially when performing Bioinformatics and Computational Chemistry tasks. This work contributes evaluation benchmark assess existing language agents rigorously, which has
10

---

Published as a conference paper at ICLR 2025
limited or no risk in inadvertently synthesizing toxic dangerous chemicals. Yet. aware that the safety of language agents for science is an important research topic (Tang et al. 2024a) and have discussed with our subject matter experts about the risk of synthesizing toxic or dangerous chemicals: (1) Our Bioinformatics and Computational Chemistry tasks focus on property prediction, feature analyses, and molecule visualization, which does not involve synthesis generation of biological or chemical substances. (2) Unlike Coscientist (Boiko et al. 2023), agents evaluated in our submission are not connected to any laboratory hardwares. Thus, is impossible for these agents to produce any dangerous chemicals or substances on their own. Even if they were to be instructed to write code for chemical synthesis in real-world applications, human intervention is still required to grant the access to laboratories, reagents, and equipment. (3) The target outputs for every task in Science AgentBench are unified as self-contained Python programs. Therefore, the evaluated agents only generate code for processing, analyzing and visualizing scientífic data that is already publicly available. They are not instructed to generate chemical reactions or synthesis pathways. We also recommend the developers of these agents to consider such potential risks seriously and provide effective intervention and feedback mechanisms for users.
AUTHOR CONTRIBUTIONS
Z. Chen led the project, formulated the benchmark, organized data collection and human evalua- tion, implemented programs and evaluation scripts for experiments, conducted all experiments on GPT-4o/ClaudeMiistrall, and wrote the manuscript. S. Chen designed and implemented rubric-based evaluation, and helped to run direct prompting, self-debug, and OpenHands experiments Llama 3.1 70B/405B, optimize the evaluation scripts, and revise the manuscript. Y. Ning helped to run some experiments on OpenHands with Llama 3.1 70B/405B. Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang. B. Yu, Y. Li, Z. Liao, and C. Wei worked as the student annotators to collect the bench- mark, verify the tasks, and evaluate generated programs based on rubrics. Z. Lu, V. Dey, M. Xue, F. Baker, B. Burns, D. Adu-Ampratwum, X. Huang, X. Ning, and S. Gao are subject matter experts who validated the tasks and provided sascifie knowledge. In addition, Gao provided substan- tial guidance in Geographical Information Science task collection and constructive feedback on the project during weekly discussions. Y. Su and H. Sun are senior authors who oversaw this project, contributed to the core ideas, and revised the manuscript. H. Sun conceived the research problem.
ACKNOWLEDGMENTS
The authors would thank colleagues from the OSU NLP group and the NSF-ffuned ICICLE AI Institute for constructive feedback. This research was sponsored in part by NSF OAC 2112606, Amazon, and Ohio Supercomputer Center (Center) 1987). The views and conclusions contained herein are those of the authors and should not interpreted as representing the official policies, ei- ther expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
REFERENCES
Mathijs Andeweg and Tom Kuijpers. URL htttpa://lea..aacciii
Model how land subsidence affects flood-
model - how-land 1-n--and--aaa www--annd-uuiiieeeee www.suabseeeee--effe------------ wtt.:is.ccc///////j/eet.....
ing, April 2024.
Anthropic.
Claude 3.5 sonnet. Jun 2024.
c 31aude-355-som
Gordon Bell, Tony Hey, and Alex Szalay. Beyond the data deluge. Science 32365919) 2009. doi: 10. 1ttp://ccceeee110/1111 URL 1126/science. 1170011
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. SUPER: Evaluating agents on setting up and executing tasks from research repositories. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceed- ings of the 2024 Conference 12645, Miami, Florida, USA. November 2024. Association for Computational Linguistics. URL https://aae antho ogy. ogg.o//........./ emn 1- i n. 702
Empirical Methods in Natural Language Processing, pp. 12622-
11

---

Published as a conference paper at ICLR 2025
Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes Autonomous chemical re- search with large language models. Nature, 624:570-578 -78, 2023. doi: https://doi.or//1..1000 $41586-023-06792-00
Cédric Bouysset and Sébastien Fiorucci. ProLIF: library to encode molecular interactions as fingerprints. Journal of Cheminformatics, 13(1):72, 2021. doi: 10.1 10.1166/443322--11-0554646
Daniel Brand, Nicolas Riesterer, Hannah Dames, and Marco Ragni. Analyzing the differences in human reasoning viajoint nonnegative matrix factorization, Proceedings of the Anmual Meeting of the Cognitive Science Society, 42, 2020. URL httpps:/esscholarbi...orrrg
emt
0br9k22
Danila Bredikhin, Ili Kats, and Oliver Stegle. Muon: multimodal omics analysis framework. Genome Biology, 23(1), 2022. doi: 10.1 10.1186/s 111166661009--01122577-74
J. M. Béquignon, B. J. Bongers, W. Jespers, A. P. UJzemman, B. van der Water, and G. J. P. van Westen. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Journal of Cheminformatics, 15(3), 2023. doi: 10.1186/s 13216666333--02200666666
Longbing Cao. Data science: A comprehensive overview. ACM Comput. Surv., 50(3), jun 2017. ISSN 0360-0300. doi: 10. 10.1145/3076253 /tttps////do. org/ 10. 1145/3076255
Anne E. Carpenter, Thouis R. Jones, Michael R. Lamprecht, Colin Clarke, In Han Kang, Ola Friman, David A. Guertin, Joo Han Chang, Robert A. Lindquist, Jason Moffat, Polina Golland, and David M. Sabatini. Cellprofiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biology, 7(R100) 2006. doi: 10.1 00.11866h-2200---1001111
Ohio Supercomputer Center. Ohio supercomputer center, 1987. URL
/19495/E5s1ph7
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hemrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Vosberrt-Vsss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. BERL:ttps://// //apxii://axxvv../ org/ abs 2210 03374
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou, Teaching large language models to delf-d In The Twelfth International Conference on Learning Representations, 2024a. URL
n Dere//ooze forum wid=kuPixIigpP
Ziru Chen, Michael White, Ray Mooney, Ali Payani, Yu Su, and Huan Sun. When is tree search useful for LLM planning? it depends on the discriminator. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pp. 13659-13678, Bangkok, Thailand, August 2024b. Association for Computational Linguisties. URL https://aclanthhll..e org/
2024.ac1-1om. 738.
Simone Ciuti, Tyler B. Muhlym, Dale Paton, Allan D. McDevitt, Marco Musiani, and Mark S. Boyce. Human selection of elk behavioural traits in a landscape of fear. Proceedings of the Royal Society B, 279:4407-4416, 2012. doi: 10. www..//sb............. 1483.
Andrew Dawson. eofs: A library for eof analysis of meteorological, oceanographic, and climate data. Journal of Open Research Software, 4(1), 2016. doi:
Pierre-Pare Paul De Breuck, Matthew L Evans, and Mian-Marco Rignanese. Robust model benchmarking and bias- imbalance in data-driven materials science: case study modnet. Journal of Physics: Condensed Matter, 33(40) jul 202 la. doi: 10. 1.118881131-6644
URL https:
7/dx. www.ddi....ooooo. 088 0 8X/ 1280
12

---

Published as a conference paper at ICLR 2025
Pierre- De Breuck, Geoffroy Hautier, and Gian-Marco Rignanese. Materials property prediction for limited datasets enabled by feature selection and joint learning with modnet. upj Computa- tional Materials, 7(1), 2021b. doi: 00.1038/441224-0210055555
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Thirt--evventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee. Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu. Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park. Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasami. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeata, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov. Nikolay Bogoychev, Niladri Chatterji, Olivier Duchemne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguct, Virginie Do Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching Biiigg Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil,
13

---

Published as a conference paper at ICLR 2025
Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni. Francisco Guzmán, Frank Kanayet, Frank Scide, Gabriela Medina Florez. Gabriella Schwar7, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Saan-Bpptstt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen. Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usumier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito,,, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao The llama 3 herd of models, 2024. URL httpthtss////xxiiii org/ abs/ /et./2407..o 21783
ESRI. 2024a.
Predict
deforestation
in the amazon rain forest, January https://leannh arcgis. com//n//prjjccss Foress
URL
ESRL Assess access to public transit, February 2024b. URL http:ttttt://eee..aaaaagaa com/ en/projects/assess httc://ssssssacccssssssss to to-public-traniit/.
ESRL gust 2024c.
Build a model to connect mountain
lion
habitat, Au-
URL https::/learn. www.aiinnnnnrii...o.aa
bui 1a-llddaoo-e to--modl-toocoooe htw.:oaaoooo-oooomoooooooaaaaaaa mountain--foo---bbttat
ESRL Assess burn scars with satellite imagery. June 2024d. URL https://eee...arrggeee.
com/ /ttp////rooject//ssssssss
Sunny Fleming. Model animal home range, March 2024. URL https://learn.arrt range/
Adam Gayoso, Romain Lopez, Galen Xing, Pierre Boyeau, Valch Valiollah Pour Amiri, Justin Hong, Katherine Wu, Michael Jayasuriya, Edouard Mehiman, Maxime Langevin, Yining Liu,
14

---

Published as a conference paper at ICLR 2025
Jules Samaran, Gabriel Misrachi, Achille Nazaret, Oscar Clivio, Chenling Xu, Tal Ashuach, Mar- iano Gabitto, Mohammad Lotfollahi, Valentine Svensson, Eduardo da Veiga Beltrame, Vitalii Kleshchevnikov, Carlos Talavera-López, Lior Pachter, Fabian J. Theis, Aaron Streets, Michael I. Jordan, Jeffrey Regier, and Nir Yosef. A python library for probabilistic analysis of single-cell omics data. Nature Biotechnology. 40: 166-166, 2022. doi: 10.1038/641587-02--01222
David E. Graff, Eugene I.
Shakhnovich, and Connor Coley, Accelerating high-throughpur virtual screening through molecular pool-based active learning. Chemical Science, 12(22)7866-7881, 2021. doi: 10.1039/DOSCU680E..
Ken Gu, Ruoxi Shang. Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang. Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, and Tim Althoff. Blade: Benchmarking language model agents for data-driven science, 2024. bRLhttts:://at
.rgg abs /t/b///488. 09667
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Web Voyager: Building end-to-end web agent with large multimodal mod- els. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6864-6890, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL wwwp:llloyyo.oogaooe.ee 2024. ael.-llclnlnng 371
Tony Hey, Stewart Tansley, Kristin Tolle, and Jim Gray. The Fourth Paradigm: Data- Intensive Scientific Discovery. Microsoft Research, October 2009. ISBN 978-0-9825442.
04.
Tom Hourigan. NOAA Deep Sea Corals Research and /echaaacccco Program, 1 2023. URL https: 7/ //www. wttpo//aaatteeeeffa333335555
748 74005622--77666 6da20a3c
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024a. URL/ittss:/oppnreriii
fest//Erum?iii I.MdddIIkD333BBBPP
Kexin Huang, Tianfan Fu, Lucas M Glass, Marinka Zitnik, Cao Xiao, and Jimeng Sun. Deeppur- pose: A deep learning library for drug-target interaction prediction. Bioinformatics, 2020.
Qian Huang, Jian Vora, Percy Liang and Jure Leskovec. MLAgentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning. 2024b. UtLhttps: http::::////ppeeeveeeeee wtt.://toooeooooooaooaaaal....l.
jYOW
Ryan Jacobs, Tam Mayeshiba, Ben Afflerbach, Luke Miles, Max Williams, Matthew Turner, Raphael Finkel, and Dane Morgan. The materials simulation toolkit for machine learning (mast- ml): An automated open source toolkit to accelerate data-driven materials research. Com- putational Materials Science, 176:109544, 2020. ISSN 0927-0256. doi: https:/doi..op:
(M16f:.ommmttci.200.0oooo 109544.4 URL https:/ /wwww www.ssseiieeeeerc.cmcmm wtt.sccimcc/e////ccccooo.o briicle/p11/0022755523333
Dongwei Jiang. Jingyu Zhang. Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi. Self-fimjcorreet: Lims struggle with discriminating self- ceff-generated responses, 2024.
URL httph:tt//aaiiiiieee orq/ abs/ 2404 04298
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. putttps://operreeet
feet/forum?ideee 8yNQM66
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikoloy, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman,
15

---

Published as a conference paper at ICLR 2025
Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:5833589, 2021. doi: https://doi.org//0.. hotoo///....0//01111 586-022-08880001888811
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter, 2024. https://
org/ abs /
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Y Huang. Graham Neu- big, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multi- modal agents on realistic visual web tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 881-905, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
Eric Krause. Analyze urban heat using kriging, July 2024. coon//n/projecss////ooooooo httppp//eeettttaaaaaaeeaaaaaaaaa
Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. BioMistrall: A collection of open-source pretrained large language mod- els for medical domains. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Find- ings of the Association for Computational Linguistics ACL 2024, pp. 2848-5866 Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https: //aa://ccc //acl antaanatholloo wow.hhhooyyyoooooo .rgg 2024.0242 findings-acl. acl 348
Zekun Li, Wenxuan Zhou, Yao- Yi Chiang, and Muhao Chen. GeoLM: Empowering language mod- els for geospatially grounded language understanding. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 5227-5240 Singapore, December 2023. Association for Computational Linguis- tics. doi: 01118865//112223..om 2tt.022..mmmlpmmmmmm.m A /at/aaaaaaaaatataaaaaaaaaaaaaale org.22003300 emn 1 p-ma ma n
htth:t
Jenny T. Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. Can gpt-4 replicate empirical software engineering research? Proc. ACM Softw: Eng., ((FSE), jul 2024. doi: 10.1 14536666667777.. URL
3660767
Guanyu Lin, Tao Feng, Pengrui Han, Ge Liu, and Jiaxuan You. Paper copilot: A self-evolving and efficient I1m system for personalized academic assistance, 2024. URRphttss://axxiv.ooo
abs/2409. 04593
Anika Liu, Moritz Walter, Peter Wright, Aleksandra Bartosik, Daniela Dolciami, Abdurrahman Elbasir, Hongbin Yang, and Andreas Bender. Prediction and mechanistic analysis of drug- induced liver injury (dili) based on chemical structure. Biology Direct, 16(6), 2021. doi:
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open- scientific discovery, 2024. URL. Wtt.httts::/aaxxw.orr
abs/2408 2408 06292
Zitong Lu and Julie Golomb. Generate your neural signals from mine: individual-to-indiiidaa eeg converters. Proceedings of the Annual Meeting of the Cognitive Science Society, 45, 2023. URL.
har.chhhaarhhippp .rgg uu wi 5xr.0885t
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabhar- wal, and Peter Clark. Position: Data-driven discovery with large generative models. Forty-first International Conference Machine Learning, 2024a. URL httptt://pperreeaaat weteeiiewwwwweeeeeee.e net/
Eorum?id=5S0 jaZNXtt]
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models, 2024b. URL
https:://rxxiv htt.:/org/a/sss/eoo https::/erriv..rrr/as////// 01725
16

---

Published as a conference paper at ICLR 2025
Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. NeuroKit2: A Python toolbox for neurophysi- ological signal processing. Behavior Research Methods, 53(4):1689-166, February 2021. doi: 10.2758/s13428-11 -220000111....
Mariia Matveieva and Pavel Polishchuk. Benchmarks for interpretation of qsar models. Journal of Cheminformaties, 41(13), 2021. doi: 1138//11331-001 0001-00519--
Maussion, A. Butenko, N. Champollion, M. Dusch, J. Eis, K. Fourteau, Gregor, A. H. Jarosch,
J. Landmann, F. Oesterle, B. Recinos, T. Rothenpieler, A. Vlug, C. T. Wild, and B. Marzeion. The open global glacier model (oggm) v1.1. Geoscientific Model Development, 12(3):909- URL https: http::/:/gmmmccppriicccc.. /rg/
931. 2019. doi: 10.5194/gmd- 12---d-12------111
Deticles/12/909/220
MistralAI. Large enough.
Jul 2024.
URL
mistral-1arge-245
Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L. Chevrier, Kristin A. Persson, and Gerbrand Ceder.
Python materials genomics (pymatgen): A robust, open-source python library for materials anal-
ysis. Computational Materials Science, 68:314-319, 2013, ISSN 0927-0256. doi: https:
10.022 URL httpt:ttttt.scciiiiiiieeeee wttp://wwwwcccciciieee.t...... com/
OpenAl. Hello gpt-4o. May 2024. URL http::::/opeaaai.cmmm/eeeeeeeeee
Bharath Ramsundar, Bowen Liu, Zhenqin Wu, Andreas Verras, Matthew Tudor, Robert Sheri- dan, and Vijay Pande. Is multitask deep learning practical for pharma? Journal of Chemi- cal Information and Modeling, 57(8):2068-2077 2017. doi: 00.100//acsjccimm760044444 URL PMID: 28692267.
https://doi. org 10. 1021 acs jcim. 7bC0146
Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O' O'Reilly Media, 2019. /tttps://www amazon. com/ Deep Learning Life Sciences Mi hiteeeccces.ccccccccc////1110110
Sebastian Raschka, Anne M. Scott, Mar Huertas, Weiming Li, and Leslie A. Kuhn. Automated
Inferncce of Chemical Diseriminants of Biological Activity, pp. 307-338. Springer New York, doi: 10.1007/978- 1010000997---4999--100 701---77662220
New York, NY, 2018. ISBN 978-1 4939-9756-77
1-4939.7756-7.16. URL
https://doi wro.orgg///ooo 110 1007 /978 7932-7756-772111
Robert Richer, Arne Küderle, Martin Ullrich, Nicolas Rohleder, and Bjoern M. Eskofier. Biopsykit: A python package for the analysis of biopsychological data. Journal of Open Source Software, (66):3702, 2021. doi: 10.21 000//sss.00702.. URL http://///oo...00....10000. 03702
Nicolas Riesterer, Daniel Brand, Hannah Dames, and Marco Ragni. Modeling human syllogistic reasoning:the role of "no valid conclusion". Proceedings of the Annual Meeting of the Cognitive Science Society, 41, 2019. htthttps: httttsssss/sshhlllp/////
h/te///schllaassececcceeooe www./orgg/........o wwt.tte////////////ee//////..ett 5xm1m8h8
AShlee Robinson. October 2023.
Chart
coral and sponge distribution factors with python,
URL
Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, edition, 2010.
Jason Schatz and Christopher J Kucharik. Urban climate effects on extreme temperatures in madi- son, wisconsin, usa. Environmental Research Letters, 10(9):094024, 2015.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In sevethh Conference on Neural Information Processing Systems, 2023. URL
ew www.:eet/oorutt i= cm
17

---

Published as a conference paper at ICLR 2025
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dong- sheng Li. and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. In ICLR 2024 Workshop Large Language Model (LLM) Agents, 2024. URL
httpss: http://pppewweeeeeeeeee/ee .et.eeiwwwwee................... forumoorrmm// ? 10
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can ilms generate novel research ideas? large-scale human study with 100+ nlp researchers, 2024. URL URLlattss://axxi..o org/abs/////eaaooooo....o..a 1409.04100
Gregor Sturm, Tamas Szabo, Georgios Fotakis, Marlene Haider, Dietmar Rieder, Zlatko Trajanoski, and Francesca Finotello. Scirpy: Scanpy extension for analyzing single-cell T-cell receptor- sequencing data. Bioinformatics, 36(18):4817-4818, 07 2020. ISSN 1367-4803, doi: 10. 10. www.10999/bioo
1093/hoinfomatis/bbaa... URL
btaaa61
Kyle Swanson, Parker Walther, Jeremy Leitz, Souhrid Mukherjee, Joseph C Wu, Rabindra V Shiv- naraine, and James Zou. ADMET-AL: a machine learning ADMET platform for evaluation of large-scale chemical libraries. Bioinformatics, :0(7):btae416, 06 2024. ISSN 1367-4811 doi: 10. 1093/boonfommttic//aaec116 URL /ttp://///61...og00000000///////
btae416
Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang, Arman Cohan, Zhiyong Lu, and Mark Ger- stein. Prioritizing safeguarding over autonomy: Risks of Ilm agents for science, 2024a. URL https::/arxiv.orgg
04247
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng. Helan Hu, Kaikai An, Ruijun Huang. Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code, 2024b. URL htttss://aaxiiiis org/ abs/
2311. 09835
Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark B Gerstein. BioCoder: benchmark for bioinformaties code generation with large language models. Bioinformatics, 40 (Supplemeett.1xx2266-222222 2024c. ISSN 1367-4811, I. doi: 10. 10003//oinnnommttessueee22. URL huttps://doi.oo htt//oi..oggg../// wtwpr///....19//11.0
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma. Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, and Hao Peng. Scicode: A research coding benchmark curated by scientists, 2024. URL https: 7/ l/axxiv.org/abt /ttpb////4471.33
Isaac Virshup, Danila Bredikhin, Lukas Heumos, Giovanni Palla, Gregor Sturm, Adam Gayoso, Ilia Kats, Mikaela Koutrouli, Philipp Angerer, Volker Bergen, Pierre Boyeau, Maren Büttner, Gokcen Eraslan, David Fischer, Max Frank, Justin Hong, Michal Klein, Marius Lange, Romain Lopez, Mohammad Lotfollahi, Malte Luecken, Fidel Ramirez, Jeffrey Regier, Sergei Rybakov, Anna C. Schaar, Valeh Valiollah Pour Amiri, Philipp Weiler, Galen Xing, Bonnie Berger, Dana Pe'er, Aviv Regev, Sarah Teichmann, Francesca Finotello, F. Alexander Wolf, Nir Yosef, Oliver Stegle, Fabian J. Theis, and Severse Community. The severse project provides computational ecosystem for single-cell omies data analysis. Nature Biotechnology, 41(5), 2023. doi: 10. 1038/ 541587-023-01733-
Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. LLMs in the imagi- narium: Tool learning through simulated trial and emror In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 10583-10604 Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL https: /tcpatttoooaaatooogo.o.o.o horg
2024 acl long. 570.
18

---

Published as a conference paper at ICLR 2025
Hanchen Wang, Tianfan Fu, Yuanqi Du Wenhao Gao, Kexin Huang. Ziming Liu, Payal Chan- dak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho. Pushmeet Kohli, Joan Lasenby. Jure Leskovec, Tie-Yan Liu, Arjun Manrai, Debora Marks, Bharath Ramsundar, Le Song, Jimeng Sun, Jian Tang, Petar Veličković, Max Welling, Linfeng Zhang, Connor W. Coley, Yoshua Bengio, and Marinka Zit- nik. Scientific discovery in the age of artificial intelligence. Nature, 620:47-60, 2023. doi: https://doi.rrg/ 000 1038//41586-023411141
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better L.M agents. In Forty-first International Conference on Machine Learning. 2024b. URL
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Opendevin: An open platform for ai software developers generalist agents, 2024c. URL //tr///aaaa h/w.ixiivv.o../...//aa org/ abs///////22/.//// 16741
Logan Ward, Alexander Dunn, Alireza Faghaninia, Nils E.R. Zimmermann, Saurabh Bajaj, Qi Wang, Joseph Montoya, Jiming Chen, Kyle Bystrom, Maxwell Dylla, Kyle Chard, Mark Asta, Kristin A. Persson, G. Jeffrey Snyder, lan Foster, and Anubhav Jain. Matminer: An open source toolkit for materials data mining. Computationnal Materials Science, 152:60-69, 2018. ISSN 0927-0256. doi: http:///oi..rg/10.10000 2tt.://00....ccooommooo000..011. 05.018 URL https: www. www.sciencedirect.commm htt.://wwaaaaccccccco////aaaaaaa
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022. URL Itttps:
neurips cc wttpppps.ccpppppppppppppppppp..p http://aaprrrfflesseeeeeee wtw.e///aaae/////////222 file/ 9d5 htt.://55000.5e4444444.0.a.aaaa.
F. Alexander Wolf, Philipp Angerer, and Fabian J. Theis. Scanpy: large-scale single-cell gene doi: 10.1186//13099--11-1133222
expression data analysis. Genome Biology, 19(15), 2018.
Felix Wong, Erica Zheng, Jacqueline A. Valeri, Nina M. Donghia, Melis N. Anahtar, Satotaka Omori, Alicia Li, Andres Cubillos-Ruiz, Aarti Krishnan, Wengong Jin, Abigail L. Manson, Jens Friedrichs, Ralf Helbig, Behnoush Hajian. Dawid K. Fiejtek, Florence F. Wagner, Holly H. Sout- ter, Ashlee M. Earl, Jonathan M. Stokes, Lars D. Renner, and James J. Collins. Discovery of structural class of antibiotics with explainable deep learning. Nature, 626:177-1888 2024. doi: 00.1088/44158--02-06668533
Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots, 2024. URL /nttps://axiv..ctt org. / 34C55079900
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learn- ing. Chem. Sci., 9:513-533 2018. doi: 10. 10.1039/C7SC00644A, URL http:/tttp/....../
I0..1039C75CC266A00
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang Agentless: Demystifying llm- based software engineering agents, 2024. URL htttps:::/a.iii.rrrrrr// hbtp://bb//244 httbb//447.0088999
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent computer interfaces enable software engineering language models, 2024a.
Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, and Maosong Sun. Matplotagent: Method and evaluation for based agentic scientific data visualization, 2024b. URL http:: //arx wwwpxivorrrgoo. www.vorggas.a..ee.ooooo 11453
19

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
[d==F_vluYur
Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun. LlaSMol: Advancing large language models for chemistry with large-scale, comprehensive, high-quality instruction tuning dataset. In First Conference on Language Modeling, 2024. URL
mettforummiddl1ee
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang. Huan Sun, Su, and Wenhu Chen. MHmmoTT: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. htth:ts:: http///pppeeeeeee/ee/e/ee//eeeee
het/forummi
Paul Zandbergen. Run geoprocessing tools with python, March 2024. URL https:
Amanda J Zellmer and Barbara S Goto. Urban wildlife corridors: Building bridges for wildlife and people. Frontiers in Sustainable Cities, 4:954089, 2022.
Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han, A com- prehensive survey of scientific large language models and their applications in scientific discovery, 2024. URL httLhhts:://// httpv/ixvv..o org/ abs 2406. 10833
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. (is-oV((sion) is a generalist web agent, if grounded. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 61349-61385. PMLR, 21-27 Jul 2024. URL httpt::tt/poeeeeeeee https////oceeeenng..omm mml. mlr ptt.pp.gggs.rr.ssss/// press/ v235/zheng225 html
Zhiling Zheng, Oufan Zhang, Ha L. Nguyen, Nakul Rampal, Ali H. Alawadhi, Zichao Rong, Teresa Head-Gordon, Christian Borgs, Jemnifer T. Chayes, and Omar M. Yaghi. Chatgpt search group for optimizing the crystallinity of mofs and cofs. ACS Central Science, 9(11); 2161-2170, 2023. doi: 10. 201100//csceemse...000003 URL https: //ct:///o/...o/ wtt//0..1/000..
acscentsci. 3c010c00 087
Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. CodeBERTScore: Evaluating code generation with pretrained models of code. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Pro- cexsing, pp. 13921-13931 Singapore, December 2023. Association for Computational Linguss- tics. doi: 10. 10. 11t186653/11/223.mmllm m59. URL http::/ss///aaaaahooooooooo org/2033. emm.ppmaa mo.maiin.cm 859
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A real- istic web environment for building autonomous agents. In The Twelfth International Confer- ence on Learning Representations, 2024. URL httph::t///oo hattps://openrevee..eee net www./eeeu/ffiiiie..ooooo... oKn.cbytLxx
Abubakr Ziedan, Cassidy Crossland, Candace Brakewood, Philip Pugliese, and Harrison Ooi. Inves- tigating the preferences of local residents toward a proposed bus network redesign in chattanooga, tennessce. Transportation Research Record, 2675(10):825 840, 2021.
20

---

Published as a conference paper at ICLR 2025
APPENDICES
We provide more details omitted the main text as follows:
Appendix A
Limitations and Future Directions
Appendix Related Work
Appendix C
Example Task Instructions
Table C.1
Example Instructions for Bioinformatics and Computational Chemistry
Tasks
Table Example Instructions for Geographical Information Science and Psychol-
ogy & Cognitive Neuroscience Tasks
Appendix D
More Details about Benchmark Construction
Appendix D.I
Details about Annotated Programs
Appendix D.2
Details about Success Criteria
Appendix
E More Details about Main Results
Appendix E.
Mean and Standard Deviations of Agent Performance
Appendix
Error Analysis of OpenHands CodeAct and Self-Debug
Appendix F. Case Studies
Appendix F.
Action Space of OpenHands
Appendix
Case 2: Influence of Expert-Provided Knowledge
Appendix G Expert Validation Details
Appendix
Questionnaire for Domain Experts
Appendix
Program Example for Domain Experts
Appendix
Knowledge Example for Domain Experts
Appendix H Rubric Examples
Appendix H.]
An example rubric of Computational Chemistry task generated by
GPT-4o without expert revision
Appendix H.2
example rubric revised by an expert by adding the available points
to two items
Appendix H.3
example rubric of Geographical Information Science task gener-
ated by GPT-40 without expert revision
Appendix H.4
An example rubric a Geographical Information Science task revised
by an expert by reducing the available points for several items
Appendix 1
Prompt Templates
Table 1.1
Prompt Template for Direct Prompting
Table I.2
Prompt Template for Self-Debug
Table
Prompt Template for OpenHands CodeAct
Appendix
Publications, Repositories, and Licenses
Table
List of Bioinformatics and Computational Chemistry Publications
Table J.2
List of Geographical Information Science and Psychology & Cognitive
Neuroscience Publications
-Table J.3
List of Repositories and Licenses
TTbble J.4
Copyright Information for rastcrio/rasterio
Table J.5
Copyright Information for ackingmaterials/maaminne
21

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE DIRECTIONS
Capabilities and Evaluation of Language Agents for Science. In this work, we have developed a benchmark focusing on tasks in data-driven discovery and formulate them as code generation problems due to two reasons. (1) Data-driven discovery is an increasingly important workflow for science (Hey et 2009). While plenty of computational tools (Cao 2017) and Al models (Wang et al. 2023) have been developed, the sheer amount and heterogenenty of data are already overwhelming for scientists (Bell al., 2009), not mention the programming efforts to access these tools and models for processing, analyzing. and visualizing scientific data. A language agent that can automate such tedious tasks data-driven discovery would help to save hours of effort for scientists. (2) We aim to rigorously assess the capabilities of existing language agents as science co- pilots that can write code to process, analyze, and visualize data. Hence, we formulate each task as code generation problem, whose output shall be easily verifiable using well-established automatic metrics and directly usable by a scientist without additional efforts to modify or implement.
As result, only focus the code generation capability of language agents. We encourage future studies to carefully examine the agents' other capabilities that can help with scientific discovery, such as summarizing literature (Lin et al.] 2024), suggesting ideas (Si et al., (024), planning experiments (Boiko et al. )023). Specifically, we advocate rigorous, comprehensive assessments of one such capability time, as we need deeply understand the strengths and limitations of current language agents for each aspect of scientific discovery. In addition, while we only use well- established evaluation methods in our benchmark, such as CodeBERTScore (Zhou et al., (023) and GPT-4o judge for figures (Wu et al.) 2024, Yang et al.) 2024b), we acknowledge that they are not perfect yet. Future research may leverage the diverse set tasks in our benchmark to develop better automatic evaluation metrics or human evaluation protocols for data-driven discovery tasks and code generation problems.
Diversity of Tasks, disciplines, and Programs. Although we strive to include diverse set of tasks and programs from different scientific disciplines in BeitnecAgnn,Bench, we devise several compromises to make data collection more practical. First, when collecting publications, indeed found more with programs written in R, Stata, or Matlab. However, because our annotators not familiar with these programming languages, we focus collecting Python programs, which all annotators can adapt confidently. Second, for evaluation efficiency, we only collect programs that accomplish the task within 10 minutes. As a result, the final benchmark includes relatively fewer tasks that process large-scale data and develop complex methods. Finally, we choose the four representative disciplines considering their abundance of open-source data and the availability of experts we can easily contact. With these limitations in mind, we have designed a principled, extensible data collection process and expert validation protocol. Future work encouraged to expand ScienceAgentBer
have
with programs in other languages and tasks in other disciplines. We also plan to continually expand our benchmark into more disciplines and facilitate future research in two ways: (1) ScienceAgentßeence will serve as necessary testbed for developing future language agents with stronger capabilities to process scientific data or to utilize sxpert-provide knowledge. (2) Aeenteehg will help future research to design new automatic graded metrics, such as an LLM judge based on task-specific rubrics, to assess language agents for data-driven discovery.
22

---

Published as a conference paper at ICLR 2025
B RELATED WORK
AT for Science. Since deep learning unlocks the power of data, Al algorithms and models have been increasingly used to accelerate scientific discovery (Wang et al. 2023). One of the most prominent examples is AlphaFold (Jumper et al. (021), which can predict protein structures with high accuracy and save biologists months years of effort. More recently, a tremendous number of language models has been developed for different disciplines, including math (Yue et al. (024). chemistry Yu et al. 2024)). biology (Labrak et al. 2024). geography (Li et al., (023). and so on. To automate data-driven discovery end-to-end, it is necessary for language agents to write code to access these Al models and other computational tools (Cao) 2017). Our work aims to develop language agents with this essential ability, which can help scientists save hours of programming effort, and rigorously evaluate such agents to grasp a more solid understanding of their strengths and limitations.
Agents and Benchmarks for Task Automation. Developing agents for task automation is a long- established challenge in Al research (Russell & Norvig 2010). Built upon LLMs, a new generation
of agents has shown new promise automatically perform many tasks in web navigation (Deng
ett al.
2023 He et al.] 2024 Koh et al.] [2024] Zheng et al. 2024 /hoou et al.) 2024). software 2024c Yang et al. 2024a), or scientific discovery 2024).
development (Jimenez al. 2024 Wang al.
[Boiko et al., 2023 Zheng et al. 2023 Lu et al.
To evaluate the performance of these new agents, many benchmarks have been recently proposed. For example, TaskBench (Shen et al.) 2024) is one of the first benchmarks for evaluating language agents with large-scale synthetic tasks. The output of these tasks formatted as JSON API tool calls, which hinders the generalization of agents developed on this benchmark to constantly chang- ing tools or new ones in practice. To resemble real-world use cases with more flexibility, many benchmarks formulate their tasks as code generation and unifies their outputs as Python programs, such as SWE-Bench (Jimenez al.] (0224), BioCoder-Py (Tang et al.) (024c), MI.-Bench (Tang al. 2024b), and MLAgentBench (Huang al., 2024b). Yet, these benchmarks only consist of tasks found on secondary sources, such as GitHub and Kaggle.
To fill the gap in evaluating language agents for scientific tasks, a few benchmarks start to use sci- entific publications as their task sources, including SciCode TTia et al. 2024).. BLADE (Gu et al. 2024), and BenooerryeenchReaal (Majumder et al. 2024b). Among them, DiscoveryBench- Real is the most similar to our ScienceAgent However, DiscoveryBench--R output abstract steps complete task in natural language, which is hard evaluate rigorously and maybe practically less useful. With ScienceAgentBenchh we advocate careful evaluations of lan- guage agents' performance on individual tasks, instead of purely relying on end-to-end evaluations of these agents, e.g., using an LLMM-based reviewer to assess generated papers (Lu et al. (024). In the long run, Acen will serve as high-quality benchmark focusing on essential tasks that involve code generation in real-world data-driven discovery workflows for objective assessment and continued development of future language agents.
asks the agents to
'We refer Zhang al. 2024 for a comprehensive survey on scientific language models.
23

---

Published as a conference paper at ICLR 2025
C
EXAMPLE TASK INSTRUCTIONS
Table C.1: Example instructions of Bioinfomatics and Compuational Chemistry tasks (Section 2.2
Domain
Task Instruction
Train cell counting model the BBBC002 datasets containing Drosophila KC167 cells. Save test set predictions as a single column "count" to "pred results/cell-count pred.csv".
Train a drug-target interaction model using the DAVIS dataset determine the binding affinity between several drugs and targets. Then use the trained model to predict the binding affinities between antiviral drugs and COVID- 19 target. Rank the antiviral drugs based on their predicted affimities and save the ordered list of drugs rereddresult//daiiss dti wwwprrpsiingg.tt"" with one SMILES per line.
Bioinformatics
Plot the Tanimoto similarities of the fingerprint between the frames. Specifically, the interaction fingerprints between selected ligand and protein for the first 10 trajectory frames. Save the png file into pred results/ligand similarity pred.png.
Train a VAE model on the given data and perform 1-vs-all differential expression test for each cell type. Extract top markers for each cell type using the results. Visualize them as dotplot with the cell types organized using a dendrogram. Save the figure to rred__esults/hca.com wtwp_llppeeeepppgg.
Train a multitask model on the Clintox dataset to predict a drug's s toxicity and FDA approval status. Save the test set predictions. including the SMILES representation of drugs and the probability of positive labels, to "pred_ https//lliinoo.ttttppeettee
Generate features for the given diffusion data based on material composition and use the SHAP feature selection approach to select
20 features. Save the selected features as CSV file "mat diffusion features.csv to the folder "pred results/".
Filter the compounds in "hits.esv" and save the SMILES represen- tations the left ones. Compounds to be kept should have no PAINS or Brenk filter substructures and have maximum tanimoto similarity of less than 0.5 any of the active compounds in "train.esv" Save the SMILES of left compounds .txt", with each one line.
Computational Chemistry
restttscooo//ooouudd wtwpoooooonn..lterssllllll .isters
Train graph convolutional network the given dataset to predict the aquatic toxicity of compounds. Use the resulting model compute and visualize the atomic contributions to molecular activity of the given test example compound. Save the figure as "spressesults//qquuticc wtt.otiiitttteeete ceapiitygsssvviiiie sis.pgg...pn"..
24

---

Published as a conference paper at ICLR 2025
Table C.2: Example instructions of Geographical Information Science and Psychology & Cognitive Neuroscience tasks (Section 2.2
Domain
Task Instruction
Analyze and visualize Elk movements in the given dataset. Esti- mate home ranges and assess habitat preferences using spatial analysis techniques. Identify the spatial clusters of Elk move- ments. Document the findings with maps and visualizations. Save the figure as Beeddressus/ElkAAnaaaaa png".
Analyze the impact of land subsidence on flooding based on future elevation data of the study area. Identify flood-prone areas and estimate potential building damage to support urban planning and mitigation strategies. Save the results to "pred results/flodingg analysis.png"
Geo Information Science
Calculate the deforestation area percentage in the Brazilian state of Rondõnia within the buffer zone of 5.5km around road layers. Save the percentage result in CSV file named "preddresullsvddfforsstttio rate.csv" with a column title percentage_ deforcstttion..
Load North America climate data NetCDF file and extract temperature data along the time series, then perform a quadratic polynomial fit analysis on the temperature data, and output the fitting results by year in 'pred_results/ restttt/ooloolyoomaall laaommaallfttteedllle fit_prre.csv'.
Process and visualize the given ECG data by perform R peak detection and outhier correction. Plot overview of the data and save the final figure as "pred results/ccg. processing visI pred result.png".
Analyze the incrtial mcasurement unit (IMU) data collected during sleep and compute sleep endpoints. Load the given data and compute the following sleep endpoints: time of falling asleep, time of awakening, and total duration spent sleeping. The three values should be saved a JSON file ppred_results/ing pred.json". and the keys for them are "sleep.onse"" onset"". "wake_onset", and roealssteppduurtioon" _duratinn". respectively.
Psy & Cognitive Neuroscience
Analyze cognitive theories using pattern similarity. Process CSV files containing model predictions for various syllogistic reasoning tasks. Calculate similarity scores hetween these models and pre-computed tigiccconscietiouusessss and high-openness patterns. The results will contain similarity scores for each cognitive model with respect to the personality trait patterns. Save the results to "pred reaidrreeut//oose www.tepptptttteaa.....eeeaa.eee. hhih niigh-simm.ee dat...pedd....".
Train a linear model to learn the mapping of neural represen- tations in EEG signals from one subject (Sub 01) to another (Sub 03) based on the preprocessed EEG data from Sub 01 and Sub 03. Then use the test set of Subject (Sub 01) to gene- rate EEG signal of Subject (Sub 03). Save the generated EEG signal of Subject 3 to "pred reeeddresutt//inees subo1 sub0ltosubon tosub03. tosub033pred..np pred.npy".
25

---

Published as a conference paper at ICLR 2025
D MORE DETAILS ABOUT BENCHMARK CONSTRUCTION
D.1 DETAILS ABOUT ANNOTATED PROGRAMS
The annotated program for each task is first extracted as is, instead of written by humans or gen- erated by any models, from the open-source repositories of peer-reviewed publications to ensure their scientific authenticity. Then, our annotators make necessary modifications to remove redun- dant lines and load the datasets in our benchmark. Finally, the annotated programs are validated by subject matter experts, as well as other annotators.
D.2 DETAILS ABOUT SUCCESS CRITERIA
The success criteria in benchmark are tailored each task and established by measuring whether an LLM-generated program accurately reproduces the result of the annotated program. Since the annotated programs adapted from open-source repositories of peer-reviewed publications and validated by subject matter experts, their execution results faithfully represent part of the research outcomes in those publications. An agent that is capable of implementing program correctly to reproduce the result would also produce correct program for similar tasks in real-world scenarios.
For example, we have executed our annotated program to train multitask model on the Clintox dataset for five independent runs and consistently observe that the model achieves at least 0.77 ROC-AUC score on the test set. Thus, we use 0.77 as the performance threshold this success criterion and require the agent to train model with the same level of performance to be considered successfully completing the task. Evaluation criteria for other tasks are also established following the same principle of reproducing some data-driver discovery results.
26

---

Published as a conference paper at ICLR 2025
E MORE DETAILS ABOUT MAIN RESULTS
E. MEAN AND STANDARD DEVIATIONS OF AGENT PERFORMANCE
In the main text, we present our results by selecting the best of three independent runs for each task in all experiments (Section B For comprehensiveness, we show the mean performances of each agent and standard deviations below, which demonstrate the same findings as in our main results in Section 4.
Table E.I: Mean performances of each agent and standard deviations Science/AgemtBench with- out domain knowledge.
Models
SR
CBS
VER
Cost
Direct Prompting
Llama-3.
1ermaa,.-Insstrutt--0 Instruct-70B
3.6 (2.0)
(0.4)
22.2 (0.9)
0.001 (0.000)
1rama--.-Innstruc---ee
(0.5)
79.3 (0.1)
32.0 (0.5)
0.011 (0.000)
Mistral-Large-2 (2407)
10.1 (1.2)
82.5 (0.2)
36.6 (0.9)
0.010 (0.000)
GPT-4o
(0.5)
81.7 (0.1)
42.2 (1.6)
0.011 (0.000)
Claude-3.5 3.nnonettt
11.8 (2.1)
82.5 (0.4)
36.0 (1.2)
0.017 (0.000)
OpenHands CodeAct
3lama-3.I-Innstuuc. IInstrucct-7BB
3.3 (0.5)
59.9 (1.6)
17.0 (1.2)
0.234 (0.026)
(0.))
59.0 (4.9)
34.3 (9.2)
0.576 (0.108)
Mistral-Large-2 (2407)
(0.9)
70.4 (1.1)
42.8 (1.7)
0.735 (0.025)
GPT-4o
13.1 (2.6)
80.6 (1,2)
62.8 (2.9)
1.093 (0.071)
Claude-3.5-Somne
14.1 (1.2)
81.2 (0.8)
63.4 (6.5)
1.122 (0.056)
Self-Debug
1 1-m.----tttttt000
7.2 (1.2)
81.2 (0.3)
67.3 (2.4)
0.009 (0.000)
3 1---nssruutt-055
8.8 (1.4)
80.8 (0.5)
67.0 (2.8)
0.054 (0.005)
Mistral-Large-2 (2407)
(1.7)
83.2 (0.4)
70.3 (2.6)
0.043 (0.001)
GPT-4o
14.7
82.6 (0.6)
71.2 (1.2)
0.057 (0.006)
Claude-3.5-Sone 3.5-Sonnet
22.9 (2.0)
84.2 (0.3)
84.0 (1.2)
0.066 (0.005)
Table E.2: Mean performances of each agent and standard deviations on SeienceAgenteench with domain knowledge.
Models
SR
CBS
VER
Cost 4
Direct Prompting
JJamma3.I-Instruct I-Insrruct-70B
(0.5)
81.7 (0.1)
19.3 (1.7)
0.001 (0.000)
2.9 (0.0)
81.3 (0.0)
24.5 (0.0)
0.011 (0.000)
Mistral-Large-2 (2407)
11.4 (1.2)
83.8 (0.2)
28.8 (2.3)
(0.000)
GPT-4o
(1.8)
83.2 (0.4)
35.6 (1.8)
0.012 (0.000)
Claude-3.5-Somee
16.7 (2.4)
84.5 (0.4)
33.0 (1.2)
0.017 (0.000)
OpenHands CodeAct
Llama-3. I.nst-nt 70B
(0.9)
60.5 (0.9)
16.7 (0.8)
0.296 (0.003)
Llama-3. 1.1-a....IIIsnneee I--nntrruct-401
(2.0)
62,9 (6.3)
35.6 (1.7)
0.653 (0.072)
Mistral-Large-2 (2407)
(0.9)
74.1 (2.9)
35.3 (0.8)
0.757 (0.049)
GPT-4o Claude-3.5-Somin
(2.8)
83.7 (0.7)
60.8 (2.4)
1.402 (0.055)
15.7 (2.1)
82.8 (0.3)
68.0 (3.3)
1.095 (0.087)
Self-Debug
Llama-3.1 1 I-sttncctt700
(2.1)
82.0 (0.4)
60.8 (2.1)
0.011 (0.000)
Llama-3. 1.teaa...IIInnnttt I--ntrruct-405B
8.2
82.2 (0.1)
61.1 (3.8)
0.072 (0.002)
Mistral-Large-2 (2407)
18.3 (0.5)
84.9 84.9
62.8 (0.0)
0.051 (0.001)
GPT-4o
(2.00
83.8 (0.4)
61.4 (1.7)
0.063 (0.001)
Claude-3.5
27.8 (2.0)
85.5 (0.5)
81.1 (0.9)
0.072 (0.005)
27

---

Published as a conference paper at ICLR 2025
E.2 ERROR ANALYSIS OF OPENHANDS CODEACT AND SELF-DEBUS
Using Claude-3.5-Sonnee as the base LLM, we sample 50 error trajectories for OpenHands CodeAct and self-debug respectively. From the 100 error trajectories, reasoning and ceff-verification capabilities to make sure their executable programs are also seman- tically correct (29/50 errors for OpenHands CodeAct and 30/50 errors for self-debug). For instance, when having trouble loading the actual scientific data, the agent may write code to simulate some fake data make the program executable but produce incorrect results. Similarly, when the agent cannot implement something correctly, e.g., a graph convolutional neural network, it may just turn to implementing simpler feed-forward network, which underfits the complex data and cannot re- produce the desired performance. These executable but functionally incorrect programs need to be better captured and fixed by improving the agents' reasoning and self-verification future research.
find that both agents need better
The other major issue for both agents is their ability to install and configure the environments with domain-speciffe tools correctly. Our analysis reveals that both the LLM-gennrated installation commands in OpenHands CodeAct (10/50 configuration errors) and human-developed packages used self-debug (9/50 are configuration errors) are not sufficient to set up some domain-specific tools correctly. This finding echoes with concurrent work (Bogin al. (024) that environmental setup for scientific tasks remains challenging for language agents. When the environment is not up correctly, both agents try to get around domain-specific tools in their programs, such as developing a random forest model with scikit-learn instead of deep learning models in deepchem.
Finally, we find that in 23 of the 50 error trajectories, Blaude-3.5-Someee was struggling with the specialized commands in OpenHands to edit programs correctly (Section (..)) especially for longer programs. It would fall into loops of repeatedly generating such commands shown in the case study in Appendix D.1. Such behaviors waste quite few turns on fixing the use of these commands and largely increase the API cost. Future agent research should reconsider the use of such commands and compare closely with some pipeline-based approaches (Xia et al. 2024).
28

---

Published as a conference paper at ICLR 2025
F CASE STUDIES
F.1 CASE 1: ACTION SPACE OF OPENHANDS
In Section our results show that, for four of the five LLMs evaluated, self-debug demonstrates better performance than OpenHands, with GPT-4o as the only exception. For GPT-4o, which may have been trained to better follow instructions for language agents, we indeed observe that it can better leverage tools in OpenHands, such as the web browser commands (lines 33, 60, and 87 in still struggling with the specialized bash command in OpenHands to update the program file without duplication (line 11 Listing F and after five steps of trials, it ends up simply using the Python function open () instead edit the file (line 146 in Listing F:2 Thus, L.LM-based agents do not always benefit from a large action space with complex tool usage, such as the design of OpenHands.
Listing F.T. sharp contrast, as shown in Listing 1:2 Claude-3.5-Somee
Listing F.1: An example of GPT-4o using the web browser in OpenHands. Browsing actions in lines 31 31-34, 58-61, 85-88.
(firat seven turns smitted),
{
"id":
"LimesLamp"s *2024 09 19114:13:52.407656
"scurce": "agent",
"mesaage": *I'm asking Browsingågent for help with this tack.",
"action": "de "delegate",
10
"args":
11
"agent": "Browsingågens",
12
"inputs"; {
13
"tank"; "T: seems that the rrv_rrvv function does not accept a 'peaks' argument. Let's check the Neurokit2 documentasion to find the correct usage of the
function. inknant Ankndf step check NeuroXit2 Rocumentationnnne will browse the NeuroRit2 cocumentation to find the correct usage of the "rap_erv' functien.. should decumentaticn)" l,
('euuooiit2 rsp_Izv function
14
15
"though:": "*
16
17
18
19
"message"; observation',
20
"observation"; "null",
"content":
22
"extras"; 1)
23
24
25
27
"id": 20,
"timestamp": "2024-09-1991411353 634697",
29
"scurce": "agent",
30
"mesaage": *Fxecuting browser actiona: sorotthhttps://neurokti read hedocs.
31
"action": "Erowse_interacttvre
"args":
33
"browsar_actions":
1o/")",
"though=": "In order to accomplish my goal, I need to aearca for the NeuroKit2 'rsp_rtv' Function documentation. will start navigating the NeuroRit2 documentation website.",
35
36
38
39
"id": 21,
40
"timestamp":
41
"scurce": "agent",
42
"message"; "Visited http:://neuccitt2.eeaaaaeeee
43
"cause": 20,
"abservation": "browse",
45
"content"; "e::before> NeuroRit ininiegacy_docs Inin Documentation An'nin's
n_ NeuroXit \alas: seaa:::beeoocesscaaaeeeess.
Dees \u00bb in Documentation In
<::before> Edit GitHub
wtwp:htwwwcmeeeppppeeeellooe///e DOCURENTATION=A
Legacy_doca/ dwwn//dca//ndax.... htt.://///bd....r/////nn.nnn
Anknin)n x Anann Documentation (1safter>lut0ss in at https://
tacs:beforeeMaannng \alnThe
neuropayciology. githulloggggttuu.b..... www.gggihh...ooooo wob.ii///errat ] (https:/ /ttp///ppyccollggy...... githyy.gthh...... sttthu..i//Neete io/Meurccit
Anine v)niniabdag Copyright 2020, hominique Makowski Reviaton bd10557f
29

---

Published as a conference paper at ICLR 2025
AnAn<; bbaoreetcbbroo <ribefore> Read Jocs legacy
<r:bbeoor>>nn
inVarsions \n Legacy_docs
regacy_docs/ VnkaDowwnoade \n [ 2DF http://ro.......eaaeeeeeeseee//d
I UTM 1//neurckit2 tttp////eeettteeeee.......eee/e.
legacy_docat saspey_doc//epab/1 An'non Rezd the Decs htw.urekt...eetheeeeeeeeeeeeeeee
I kpub
Froject 11/readtthsoss...rrr
[projects/neuroitt//1
Bullda View (https://github hwtp://rop///cccn egy/NeuroRit/blott
builds/) \a
Dexnloads
non Gut \n
btsoyydddoo//oo//eeee.e./ Neurokit /eep:///eeecc_oor///eeeeeee
Edit J http:///hhuub.ommeeeeeeeeeee
Aninnearch \n Valnv A\phallosted by
Rezd the poes docs hoss-rachhaosssioppppprrraaaaapp
\u50b7 Privacy Policy ] (https:/?
46
"extras";
*url*: htttpa://neurckit2..o http:p///eeeiii.....eeeeeeeeeeee
48
(webpage information omitted),
49
50
51
1,
52
53
"id": 22,
55
"LimesLamp";
09-19914:113388.3444
56
"scurce": "agent",
57
"message"; "sxecuting brewser actions: click("43)) (rop_rrv()
op 'Enter'1"
58
"action": "browse_interrottive"
59
"args":
"rrowser_actione" "click(' 43") Aa 'zap_esv' Angress 43', 'enter')
61
"though=": "To find the correct usage of the 'rsp_srv" functier, I should search for
the Keurakit2 documentation. The current page search box that T can use for this purpose, ininkcttion))n Cl on the search textbox. AnZ, Fill search Lextbox with the quezy \rrsp_rrv\", An.3, the search.",
initiate
63
64
65
66
"id": 23,
"Limestamp"; *2024 1911::11:0234413311
68
"scurce": "agent", "mesaage"; *Visited https:/ https://neurokiee http://wwww.....a...a.oo//oo//ta
70
"cause": 22,
71
"abservation": "browse",
72
"content"; *I <:tbefore> NeuroKit I (index. \akalegacy_ \a'n_ [index. batt::::beeooreeecaaaaeee
(index.hemi] \200bb
n Search \n <::before> sdit on GitHub httpaa:///itttbb.omm \n\nir\a
nPlease activate Javašcript enable the search functionality. \minft
Searching Antnintmin +inin)pddas Copyright 2020, Daminique Makawski
Revision ba135e7t t
<::oetore> the
legacy_docs <: <::before> inbieforre/nlleeessionn \n legacy_doca thttpe://neurkkit2.
knlnJownloads
228 1//
neurokit2. bepootttrruudtaccrc../////eaaaaa http:////eeemeeoooooooooo/////// docs/ppf/)'s
HIML 1//
Epub 11//
AnAnün Read the
n
Project Home http://hheh/////ssaoaaaoooooooo/
Rot Ids
Downloads 11/
in/no Gi:llub \n
View 1
Edit (https: | http://wnw.oogooeoooooo//////iie peeii///eggaae egaoy decs/doca/
rearch.rat) An\nSearch
\nin + An AnAnoooted by Read the Docs (https://
readthedocs.oo \u00b7 Privacy Policy htttpe://aoc..reedteeeee resdthedecs. wwwdtheeecs...ooooo ptcllccss.o//gggee Lo/page/
73
"extras": "url*:
74
htt.er//we..eetthhcco..////////e hotal html stallqurep_ree htw.e/.cceerdr.esrrrrdeeeeee
75
(webpage Informat' on omitted),
77
78
1r
80
"id": 24,
82
"timestamp": 72024-09-1911::144033.8
83
"scurce": "agent",
84
"message": "Executing browser actions: scroll 300)",
85
"action": "bzowse_ialeract..e
"args":
30

---

Published as a conference paper at ICLR 2025
"browser_ "browssr_actiions: "scro"1 (0, 300)",
88
"though:"; "To find the correct usage of the 'rsp_rrv" functior, need to review the seazch results, Since the search resulta are 1ikely 11sted in the main section, I will scroll down view them.", "arowaargymeeenii_sg..oo..
89
90
91
93
"id": 25,
94
"Limestamp"; *2024 1911::14:05. 356116"",
95
"scurce": "agent",
96
?qurop_ httrrrrrriihttttooooeeeeeeoaoooe http://www.....aa/aa//////////// "cause"; 24,
"mesaage"; "visited https: http::////euokit22
97
"abservation"; "browse",
99
"content"; *I <iibefore> NeuroKit (index.htal)) Wnwnlegacy \n'n_ Neurokit [indeex.htmi tote::::
Docs (index.htmL) \200bb
Search \n <::before> 2dit GitHub ] hettps://github. wtwprr/it///loo/eege...ee.ee
\nkn\r +in!
activate davaSoript enable the
functionallsy \ninff
Searching \a'n'ninaa- +lala'udDa9 Copyright 2020, Dominique Xakowaki Bevision
ba105E7E :eale::befooee
<::before> Read the Docs v: legacy_
Ls:teeooreeAnhneeeee Va
legacy_docs
http/legacy_docs//earrhhc http:://eeee...alllaaaaaaalllal AntaDewnloade
PDF ht//neurokit2..sestthheeee
.10/ .io/ /t/////ownicdd//en/llee lttp/////eccd../////////a
HTXL 1///nerroktt2 wew.:eiittt..rtthhheeess... [ Epub (
wownloadaeee/
Annnnn
An. Project (//
readthedocs wwt/orooeeeeeeooooo
/neurokit2/but /builds// /
Builds wt/e///eeebcc...o/o/o/ooooo jectss
uownlcads s//eeachieecc.ooosppppeetteseeee
dewnloads/) Annnnn Gitllub \a
View
Neurokit
Edit (https://githi)
\n\nSearch \n
\n
*Annnfoated by Read the bocs
\nGDb7 Privaay
Policy (aLLps:/ wtt.ddcc.e.dddeddoos.... seeell/edccs.bb//gge///rve policy hhml)
100
"extras":
"ttcpa://neurrkktt..o scbekk.keyerrdryyssseeeee oeg.
102
(rebpage information omitted),
103
104
106
(19 more turns cmitted),
Listing F.2: An example of Claude- 3.5-oonctttt that struggles with specialized bash commands in OpenHands to edit programs correctly. See line 11 for a failed edit example.
(firat seven turns smitted),
{
"id": "2024-09-1612::450000
"timestamp":
"scuros": "agent",
"message"; "......",
"action": "run_fpython",
10
"arga":
11
"code": "cede Vnimoott pandas as pdinimport aunpy npinfren sklearn. madel_selection import train_test__pll sesst_splittiieeeeeeeeeeesssssse sklearn. Alleaan.pregrocessiii import standardScaler,,
Enceennooorrnoor oklearn. liner, toggsaticdegreesiinntteecc eklearn.metriae import aocuracy sklearn. cempose impert ColummiransFozmeekkan
import
sklearn. inport Pipelinelniren
mixtend. resture_eelectieeeeeeeeeeeeeeeee import Sequent Featureee aa 3rs\nimport. matplotlib.pyplo. as Annnt bencimark/ aatpaaaa/d/atsss//deeeeeee /itaaasss//kkppebppeeesetaaa wtaeeesssrain......... Anlad Frepare features and target \nx data.drop index', 'signal-inhibitionn ), axis=l) \ay datall"ignal-inhbbbit * Annnn Tdent numeric and categorical wtt.mmgmmuerrrrafetttie
pé.readdcsv(' /wockspace/
select_diypes iinelude(('in665)
www.ommmennatteeerries..... select_dtypes ["clude=(""bbectt www.um)...lllla....mm
X. Binarize the target varisble)
nthreshold - y.median \ny_binacy - (y aetyehol in\nt Split the datatnx.:rain, X_test, y_train, y_test train_tess_sppittie y_binary, test_size =0.2, sandom_state=42)
create preprocessing scggelapreprocessrr
celummTranafoomme///
tranaformera=(V>
('num', StandardScaler(),
numeric_teaturest An categoricai_feettressl:
(reat', Id:ooe first', sparse=False),
1Anutt Create a pigeline with preprocessing and Pipeline ('preprocessor', preprocessor 1 In ('aleasifier', iogiaaicoooaacccootttttooeo (trrrrrccooottnndooosssttoo//// \n] (n) in)nt Perform feestture= (1, flostingssalse, In
logistic regressionnnll_.pppeee
feature selectioninafe ies pispeeinne Ix co.oolums)) aocurayy, An
forward=ralse, ovw5,
n_johs--1linnnafs sts.fit(
31

---

Published as a conference paper at ICLR 2025
X_train, y_tzain)
resultslnremulcs pd.
(sfs.
()) aww.results.arrt_.taaa.. ((pllcttpl)oo An'nf Plat the results tell. 61) implt.plo (resulta..ndex,, resulta ('alttallavg_aee avg_acore I, "aaker/woo"aalltttttttcc Backkward Feature Selection: Accuzacy vs Number of reatures" features") /apltt Numbbe Number reatures') ) Anpit.ylabel (t ('roosssrlliiditateee Accuracy') (annracr//nllt..grrddd// htg.alltttt..t....tg.oooooo I' htt.appceeprreereealttteeee
inknt Save the V.otl......avaae tttpsereeeeteee_____ settsre___eee____ee ect www.laala__reedaaaoaaa
y An'nd Print
feature
jafa.k_featurs [aaturonnames_] Anprint (f\" (f) Kumoor of selected reatures:
Best feature subset: [best_ (
len (eeea__sussee (apptti Aaprippii "Bett cross validation accuracy:
An'nn Evaluste the test using the best feature subset pipesteppepeeeinee perssattrtabeet.pppellle. (X_ (X_train, /_train//ny_prre kny_pred best best_pipeline,
predict (X_test) sokntest_accuraag
accuracy_ (y_ test,, y_prad) Imprint "Teet
accuracy with selected Teatures: [test_ cesst__ccuracyyooo : "f
nedst by /wwrk/pace/pret_ppooo. www.duorrrrraaoarrooeeeo http:///rrrodeeeeaaaolllaaaaaa vinualization_ 1eppatio_..pp',,
\n
preeplac="iimpps pandas as pa melinimport numpy np\\
nErom sklearn. preprocessing import Standardscales, (neWotEncoder Venire Anfrem import soggaatickkreeaiinn/nneeee infrrm aklearn.mstrics import Allearnliiees__mdee sklearn. cileanncompose import umnTranstormer Anfcoom mixtend, mixtend. wew.aalreeeeelcetiiiee Import Gequentialleatuureeeleee nimport matplotlib.pyple
import tesi:_lesst_ppit Mafrem sklearn.
aklearn.p'pa paline import Pipel
ae SFSV) new_content<coden
plt', \a
12
that there duplicate sections in the code. Let me clean it up and renove the unnecessary parts:",
"though=": "Thank you for updating the file.
13
"a__oonfi "confirmed", "kernel Weerne_iinitcooee init_code"; "fron agentekills Import
14
15
17
18
"id": 16,
19
"timestamp": "2024-09-1612::45:00 983049",
20
"scuroe": "agent",
"message"; "Code execuled IPython cell.", "cause": 15, "sbservation": "run_ipython",
22
24
"centent"; "......",
25
"extras":
26
"code":
27
28
29
1,
30
31
32
"id": 17,
33
"timestamp": *2024-09-1672" 345720.373166",
34
"scurce": "agent",
"message"; "......",
36
"action": "run_ipythoo", "arga":
37
38
"code":
39
"though=": apologize for the confusion. It aeams that the file still contains duplicate content. Let clean it up duplicates and ensuze the code is cocrecsi",
"is_cenfirmed"; "confirmed",
41
inda_coda"; "from agentskilla import "karnel "karnel_init, init
43
44
45
"id": "timestamp": "2021-99-18122:45.2 :5:1
46
"scurce": "agent",
48
"mesaage": *Code executed ir. TPython call.",
49
"cause"; 17,
50
"obsecvation": "run_ipython", "content": Cell In.1], line 883n cowceeeeennnn
51
nöyntexError: incomplete input (JJupyyer bencimark/datasstts)
wozking direstory: /workspace/
benchmark/ datasats] An (uppyter Python interprezer: peeehannss/peetty//opeeeeee 10th-ppp...L/////ggphhoo]]
52
"extras":
"code"; "......"
54
55
56
57
59
"id": 19,
60
"Limestamp": *2024 09 16122:45:39.261122 "scurce": "agent",
61
62
"message": "......"
"action": "rython,,
32

---

Published as a conference paper at ICLR 2025
"arga":
65
"cnde":
"though="; *I apologize for the syntax ecror, Iet correct and update
the file prepesly:
67
"is_cenfirmed"; "confirmed",
68
*kornel "karnel_init, _nddt_ "from agentakilla import
69
71
72
"id": 20,
73
"timestamp"s "2024-09-1612:: 4800440"
74
"scurce": "agent",
75
"mesaage": "Code executed TPython cell.",
76
"cause"; 19,
"ebservation": "run_dpython",
78
"content";
79
"extras":
80
"code":
81
82
83
1r
85
86
"id": 21,
87
"timestamp": "2024-D9-10122::5555..166
88
"scurae": "agant",
89
"message"; "......",
90
"action": "run_ipython",
"arga":
92
"code":
93
"though=": "1 apologize for the confusion. It seems that the file still contains duplicate content. Let clean it up final time remove all dupl icates and ensure the code is correat:",
94
"is_cenfirmed"; "confirmed",
95
"kernel "kernel_init_cee innd_ "from agentskills import
97
98
99
"id": 22,
100
"timestamp"s -2024-09--172::::::2222222
"scurce": "agent",
102
"meszage": *Code executed it 1Pythen cell.",
"cause": 21,
104
"observation": "sun_ipython",
105
"content": Cell In line 86ta new naw_conzent=in
ndyntaxRrror:
(Tupter
working directory: /workspace/
sancamack/dataseee (aatasets))Me (n Python interprecer: Copeihaandsggeetlyyell ppetiy//peehhancs al 1yb.////b/i///phhooo
-504 SCH_
106
"extras":
"code": "......"
108
109
111
113
"id": 23,
"timestamp": *2024-09-1672" 011
115
"scurce": "agent",
"mesaage": "......",
"action": "run_ipython",
118
"args":
"code": "......",
120
"though:": *I apologize ze for the repeated mi mistake. Let me correct it and update the file properly:",,
121
"is_cenfirmed"; "confirmed", "kernel ciirnel_iitt_coeel "fron agentakills Import **
122
124
126
"id": 24,
"Limestamp": *2024 09 16722:46:11.0300
128
"scurce": "agent", "message"; *Code executed in IPython cell.",
129
"cause": "abservation"; "sun_ipython", "content": "......",
131
133
"extras": 1
134
"code": "......"
135
136
33

---

Published as a conference paper at ICLR 2025
139
"id": 25,
141
"Limestamp": *2024 09 1612::16:26.8742277
142
"scurce": "agent",
143
"mesaage";
"action": "run_ipython",
"arga":
146
"coda": "with open httppaccpppeeepprrrraaaa///////e
[w") as I:Aa f.wzlte( Aninport pandas as od'nimport numpy as npAnfrom
inport train_tess__pp sesssssrit clleaan.preprocesstt.. import Standardscaler, Cnesotincoderlnfree aklearn.1ineor 1inear__dddd import coggstttteeg compose import Columifranetormeekkfff mixtend, fwature_selecttteeeeeeeeeeeeeeee import BeguantiallFaaturssol.ccm matplot lib-pypoo as pitinint Load the data\ndata pa. reac_esv(" /workspace/ htt.as/tt//dppee/pppss.aadd Aaka) Prepare features and Larget \nX axis=l)
irport accuracy_ hatrrrcy_ccorrrrrroore aklearn. pinellne Inport Fipeline intonrron
SFS\nimport
data.drop index', 'Sigall-iniibitionn
datall"igmal-iahbbiee
"J'nnnt Idantify numoric and categorical
decc__ttpessssseeeeeeee__ iicclude="""ntte", " "float64' (amass))aoooommnaaateeerrr.
select_d (bbjttt c).cuuun Inknt Binazize the Larget varisble) nthreshold y.median() bin_binary ly threshold)
wttpahoddd.sstyyeeeeeeee.. esstpp datainX_Crrann, X_test, y_train, y_test traii_tees__ppittk y_binary, test_sice -0.2, 1 Anknt Create preprocessing s:gpetppeepooeessor
Split
Selummironssformer transformera=(Va
('num', StandardScaler()
cemeric_feeturestt Vn eategoricca__eeturees
("eat', (ndkotthooddrrddagp== frrs'', sparse=False), 11
pipeline preprocessing
logistic pospressionnnli_pppeeieee An ('classifies',
(ipeline([Va
l'pregrocessor', preproceszor
Anint (_festures= len
backward Seature selectionknsts sisiir_pppeeiinee Soetting (8.columns),,
forward-Ealse,
K_train, y_train) Aning Get the reaultslnreaults -_jona=ll/innaat ( jel_metric_diet baatesulls.oott__ieeeiiiii
acoring='ascuraeyy,
cv=5,
tronsttarame.tronddll (sfs.
Plot results)
(el.. (10,, 6)) ppptt.poott (results.indes results (auuussllvv__eee avg_scrre markersfo hnppt. Backward reature selection: Accuracy va
Peatures') Xaplt.xlebell l' Numbes Features') mpii.yyabbll'Cer Crsssroooiddtttee (Prue) (ttpa//
ininf Save the plot Loot/pplt.som savefig Accurscy' http://ooo_rriittaaeaepeeaaaae wttprrpe__fttur__eleeeee ier__aelleeeo_aoeeeeeo wow.oooaallllaaaaaeo //orkkppae//prdd the best feature aubset inbeat_aubset 1ist lats. Best feature subset: (best_subeet] Amprintt (f len (beet_ ]\" Amprint "estt cress-validation accuracy: sazore IV") in)nf Evaluate on the test set using the best wee.ssattottbbbet.ppppeel....e.. (K y-train) Any_pred best_pipeline. accuracy
An'nd Print
(oamme__nnpiintt Anprint (f)*
festures:
nubsotlnbestt_pppp
predict "K_tess/llnnttttettt inwatast accuracy_ (y test, y_pred) Amprint (f)'rsst "Test
selected features: (test_acouracy: 4f)) Anf A (F"Flle content updated sucoessCully.''.
persistent issue. me for all by directly aditing the file conzent:", "ia_oonfi "confirmed",
"though=": "I apologize
149
"kernel_inite innt___aii___ "from agentskills import
150
151
152
"id": 26,
154
"timestamp": 19924-99-6112::4622..9000
"scuros": "agent",
156
"message"; *Code executed IPythen cell.",
157
"cause*: 25,
"observation"; "run_ipython",
159
"centent": *File content updated successfully, Auppuyt current working directery: /wockapace/benciiie
hooe/kpppst/benccee///taaaeeee (Jupyter Python interpreter:
"extras":
161
"code":
162
164
1,
...... more turns emitted)
166
34

---

Published as a conference paper at ICLR 2025
F.2 CASE 2: INFLUENCE OF EXPERT- PROVIDE KNOWLEDGE
In Section we also find that expert-provided knowledge does not always lead to metric improve- ment because: (1) Expert-provided knowledge specifies some advanced tools that are less familiar to the agents. (2) When the agents do not know how to solve some tasks without domain knowledge, they tend to generate some exccutable but less meaningful programs.
without sepert-provided knowledge, Claude-3.5-Sonce using self-debug would only use basic tools like pandas and sklearn to develop random forest model, which is not accurate enough to find the best drugs repurpose. To help the agent, an expert provided the following knowledge: "Typical encoders for a drug include ECFP fingerprint or message- passing neural networks convolutional neural networks over the amino acid sequence. " With the help of expert-provided knowledge, the same agent successfully generates a program install the DeepPurpose package and use it do develop an advanced neural network with MPNN drug encoding and CNN target encoding (lines47-57 in Listing F.4.. However, we noticed some signs of data contamination in this generated program, such as the use of automatic data loader in DeepPurpose (lines 33-38 in Listing F.4). This issue effectively captured by our strategies mitigate data contamination and agent shortcut concerns (Section 2.2 The generated program (Listing F.4 is not executable as the automatic data loader does not read the modified local dataset correctly. Despite this issue, from scientist user's perspective, the program generated with expert-providee knowledge tries to develop a more helpful model and resembles the ground ground-truth program (Listing F.5 more closely.
As show in Listing F.3
the 2D molecular graph representation and for farget include 1D
Listing F3: Program generated by Claude-3.5 Solude-355-oomee 3.5-Sonnet using self-debug without expert-provine knowledge.
import nampy as mp
import pandas pd
from sosearnnnodei_eelll
import train_test _eset_ppptppttteeett _ppli
enserble import RandomPorestBegzeee
from aklesss.preprrcesstte import standardicaler
Amport json
Amport
preprocess
10 def Load daad_data (file_path))
(pe((fileppath, (=')
12 return [line.ttrip() for line in
14 def
15
return ro 16
header=Nore) vvaluas
17
def load Laad_arrget_ao Loed_Larget__segeenee sequences (Flle_path);
18
with (fel (=")
19
return loon.
21
def encode_sequeiie
max_Length=100)
22
amino_action "ACDFFGATRTTMPPRRSTTW"""
23
encoding ap.zeros (max_lenght
24
for i, aa in enumerate (mel: aa amino_acidas encoding(1] = amino_acids..ndexx
26
27
return encoding
28
29
toad data
30
drugs_train load_data an
31
triini afad htt.a///waaataaaaaaaaaatttttttta
32 barget_seepe _eee__ssueencaa load_tanget, setpettseeeeesssaeee /stppp/ssss.cccccaee//aaaaaaa//s json')
33
34 Encode drugs (simple fingerprint)
35 def encode_drug (smiles))
36 return (smiles. count (c) for in 'CHOFeS']
38 X_rrgg np.array (Lencode_dree drug (drug) for drug in drugs_train])
39 ttrgets up. acray (encode_ssquencee (zeq) for seq in sesgee__seueeness.alll. ()
40
41 f Freporo data for troining
ap.column_sredd (Iap. repeat (X_drugs, Ettarggss.shapee axis=0),
43
ap.tile (X_targeta, stcdruuga.hhpp(0... 1)11)
arfinity_trafin flatten!!
35

---

Published as a conference paper at ICLR 2025
46
Scale features
scaler StandardBcaler()
48
X_scaled
49
50 split data
51
X_train, X_test, y_train, y_teat train_test_splitt (K_aclledd y, test test_size=G.2, sandon_stato==2)
52
53 Train model
54 model RandonforeestRRggessss (aLestimatoos=50, mex_depth=10, n_jobs=-1, candom_state
=421
55 (odel.fit((xtree y_train)
56
COVIB-19 target
drugs
58 covid_aeq load_data www.beenk///aaaeee benommeee/eestttt/////oooeeeeeee contains the sequence
$ Second 1ine
59 antiviral_druge pd.read_csv http:///a...eccaacc/aa//////aaaa
sarruugstabb/. sepe"\tt)
60
61 Predict binoing afrinities
62 covid_encoding eacode_ seroode_seueeeeee (covid
X_rred up. www.celmmsssccc.ee (aapcckapp....eee.eeeee (ap.erray [encode_drugaaniieiee (smiles) smiles antiviral_drug
('SMILES']]1
64
ap. (priilelcovid (len Lleniantivirel 11)1)
65 x_pred_scaled scales. saales.tranffooik.prr
66
67 predictions model predict pree___aaleed
68
69 sank drugs on predicted affinities
70 rauge antivisel_ predictiens
71 (tw.:dduuggdreeiiieeeeeeete
72 ranked_drugs rankad_drugs.one valuev Afffsitced
73
74
results
15 os. ca.makedirs p'prrd_rssuits'' exist_ok=Trud
open (' prred_rssult//doe http:///aaaaaad....rrrrree..ott. (yr)
77
for smiles drugs SMILES'];
78
(fmiiees))
79
80
print ("Predicttonn savec to pred_resullts/dar/ _ttiii_ddtttt http://wwi...iirrrrrrrrrronnnnno repurposing..txt"
Listing F.4: Program generated by Claude-3.5 1eie-3..-SSonnn using self-debug with expert- pppprttprovided knowl-
edge.
import import Amport subprocaas
Function to install required packages using pip
def install (package): baaprocess.checkkco ([sys.executablee m", "pip", "install", package])
Check and instoil required packages
requi redd packages pac_kkagss l'pandaa', 'rakit', 'DeepPurpose', /git=ntcpaa///gtthbb.cmm
Delley//decriplaetoouee "pendas [lavor*] for package in
11
12
try:
13
if backagg.ssertswihhhte instell_ (package)
15
else:
Import (package. _ppkort__pccckeee split ("="))
17
except ImportErroz:
18
print (f" (f"Ipaccage) not found. Tratalling...")
19
package 'ddkitr;
20
ins:all_pip(" rdkit--yypp))) Use rdkit-pypá for pip instailation
22
install 1_pallppppppp (package)
24 Now that we've ensured ali packeges installed, import the requized modules
25 import numpy as ap
import pandas pd
27 from UeepPurpese import DII as models
ReepPurpose import utils, dataset
29
30 Create directories if they don't exist
31 makedirs (preediis/pree_reennttee exist_ok=irue)
32
preprocess DAVIS pro_____ocss__DAVE 34 X_ __drug, X_target, dataset, load patha//enchhas///ttaattt////t/
36

---

Published as a conference paper at ICLR 2025
binary
37
convert_to_log-Truo
38
39
40
Convert numpy arrays to pandas ratel'rames
41
trzin pd.DstaPrama
42
'drug' : X_drug,
'earget': X_target,
44
'Tabel': Changed "Y' to "Tabel'
45 1)
46
47 Load drug-target interaction model
48 config ut.ls. wew.ggeneer_eooom
49
51 target_encaddgg=
1024, 512], train_epoch-5
53
54 model modela.model how.eooiinntiicciooooooofffif
55
56 Train the model
58
59 Losd antivizel drugs
60 antiviral_drug
sepe
61
62 Load COVID-19 target sequence
63 with open hataci/nsrkkaattetttt/////eeeeee
(z")
sovid_seg f.read() () (0]
65
66 #repere data for prediction
67 a_rug_repurrs repurpee antivirsi_dran ('uggddddgss ('SSIIEs' "ariiass..tllstil tillist
68 x _ttaggetrrpprrposet
[covid_seq] + len www_ararrrrrrrpoosss__
10 Make predictions
pred model. prediit iK_druggrepurpprr X_target_rapurpone
72
73 Create a dataframe with predictiens
74 results pd.
75 'Drug's antiviiall drrgssddrss"_aa "Name"
76 'SMILES'; antivirel_druge SXILES'),
77 '#redicted_Afffiittyy y-pred
))
79
80 sort by predicted affinity (lover values indicate stronger binding)
81 results_sorted results.sort_ee (eadrrrrttted
82
83 results
84 seeults_sortedd ('rtedd"sKILES' ]. lov /ttp:///./......../////////aaaa. header=7alse)
index=ralse
85
86 print ("Pradiation completed. Results aaved to
Listing F.5: Ground-trute program in the benchmark.
DeepPuzpose Amport utile, from DeepPuzpese import DTI as models pathlib import from Shutil import copyfile
import
import Jaon
import numpy ap
import pandas as pd
11 drog_encoding, sarget_encod ng "MPN"" "CNN
12
13
def jatke_dattss fname, affinity_fnamd target):
14 with open (druu (name) as I: drug [l.rs:rip()
15
16
affinity vedrraa__cs//affiniii
header=None)
18
19
SMILES [] Target_seg []
20
21
y =
23
for in range (len (drugl)
range (len )
37

---

Published as a conference paper at ICLR 2025
SMILES. wtw.al..pppedi..o.o. (drug(i)) (dr)
26
srrget__oogaapemeee (target(3)
27
y y-appendfariinttyy.a...com 111
28
29
y uti.s...everr.y..nt.. (npaarryy (y), 'zx", (p')
30
31
return utila. patappooess Ing.array rp.array (Target__gg) rp.arrayly),
drug arugdnng,,,ng, target_encoding
33
apl hod_ttttoo-nno no__pplennoooooooooooo it!)
34
35
36
37
def main():
38
with open o' "een"""ncmmrkkddttae hetp:///ecllkkkaaaaeee/////////e
as It
39
target jaon.load(th
40
target 1ist watattattt..aueeee...
42
train make_ dataset hatp://////o////////////////////
'benchmark/
datasets/ctt/Da/1S att.sstt//t////liiffffttt.a.abbd target)
43
htw../aattatttteoooe////aattta/s (nAVT///ruggaaalllllll..aa "benchnark/
Largel)
45
con utils, gwweratte_co...ggittte.e encodiing drug_encoding,
46
terget_encoding target_encoding,
47
cis_hidden_dims (1044, 512), train_epoch 10,
48
49
50
batch_size 125,
dim_drug mpan_tidder__szee 128,
52
53
mpan_depth ain_targot_filtert 64, onn con_target_kernels [4,8,12]
54
55
56
57
59
60
ierll.tteinntaatte val, val)
61
62
t, t_name (1.ratrip() for in open http://eecccnkkkkaaaaattt////ood
63
64
Af = read
tab', (\e')
5, r_name, r_puhchem_cid dM.SMILS,, values, dé Nama! vvalues, df (f(" ('bbhn 1.values
66
67
-fath Path /t///dddd_d_ppppee http//eeddddddudeeeeeeeeeeeeeeee results/reault/')
68
if not out pul_[path.existe us.nkdirrrout_fpeee
70
y_pred model repurrpoe (X repurpose т, target t. model model, drug_names
target_rame t_neme,
72
resuit_folder "./pred_remulteene peredd_realls//rmnnt// convert_y
True)
73
f_in:
75
lines [1 for in f_in,
77
with open (p.p/addd hit.///dd_rrsuttta wtw.tt_dii_rgprrrrrnn...ss. "gt") fotut:
78
foout. Nutt.riteere join () hee (3:-1)))
79
80
name
82
main()
38

---

Published as a conference paper at ICLR 2025
G EXPERT VALIDATION DETAILS
In this section, we provide details about the expert validation process in Section 2.2 We include the questionnaire (Section G.I) for domain experts and two examples used in (Section G.2 and G.3
G.1 QUESTIONNAIRE FOR DOMAIN EXPERTS
Thanks for providing feedback developing an Al agent to assist you! Given a task instruction and a datasct, the agent will help you write a computer program to fulfill the task you have mind. To develop and evaluate such an Al agent, we have collected a benchmark by adapting some tasks from peer-reviewed publications with open-source codes. Each data sample in our benchmark consists of the following main components: Task Instruction: Describes (1) the goal of task scientific hypothesis and (2) ouiput require- ments.
A14Science benchmark called ScienceAgenuBench. We
Dataset Information:
Contains (1) the dataset directory structure and (2) helpful metadata or a few examples from the dataset.
Annotated Program: The reference solution adapted from each publication's 's open-source code. Evaluation Script: The code to evaluate Al agents' performance by comparing the exccution results of its generated programs with those of the annotated programs.
To ensure that each task is formulated and described correctly and professionally, we would like you to give us a hand by reviewing additional information from you domain expert, including writing down some task-related domain knowledge and revising rubric score the generated programs.
collected data samples. In addition, we are also seeking some
Please follow the guidelines below to review each task. First, please enter the Task ID you are review- ing: [task id]
Guidelines for Data Reviewing
First, bit more background: once you give the AT agent task instruction, it will try automatically complete everything without seeking additional help from you. This similar the scenario where you give the task to junior student in your lab/class who will complete as an assignment.
For each task, please first spend few minutes reading the given task information (instruction. dataset information, and source GitHub repository) and annotated program have rough understanding the task and relevant concepts. Then, please comment on the following two parts. Note that you may iteratively revise your answer to each question to help us improve the task instructions and programs. 1. Program
Is the program valid solution (not necessarily the best solution) to the given task instruction? Here is an example: Igoogle doe doc_link] there are only minor issues, please comment on how the program should be modified below. However, if you believe there is major issue (e.g., the program is doing sth irrelevant or more than two lines of code need to be revised in order to make it correct), please let us know the task ID and do NOT fill the rest of the form.
the program a valid solution to given task instruction?
[] Yes
N Need Modification (comment below)
No (report and continue to the next task)
How should the program be modified? Please mention the line numbers that need be inspected.
[Long Text Answer]
2. Task Instruction
The task instructions were created by non-experts and thus might contain some misused terms, awk- ward expressions, inaccurate descriptions of the task that do not adhere your domain's 's scientific language. Do you see such issues for this task instruction? If so, please revise or rewrite the task instruction for any issues you can find.
Finally, if needed, please help make the task instruction more fluent and natural sounding. Please enter your revised task instruction helow. If there are no changes, please skip this question and leave the answer text blank. [Long Text Answer] "The example for program and instruction validation is provided in Scction G.2
39

---

Published as a conference paper at ICLR 2025
Domain Knowledge
Suppose the Al agent fails fulfill the task based solely the task instruction, perhaps due lack of some background knowledge, we want to provide some additional information to help it suceeed. This is similar to the situation where you give an exam problem to student in your class and they might not be able to do it just based on the problem description, but you can provide some hints to help them. Please write down most three important pieces of knowledge that related to the task and the program.
For example (lgoogle_dog doc link]): P]
Concepts and details in the task description or program that may need further explanation extra attention, e.g., term definition on wikipedia you would send this task, common practice for such tasks in your field.
student (without much domain
expertise) working Information about the python packages and/or functions used in program. For example, you would copy and paste a snippet of package description/functio documentation to help the new student work- ing on the task.
You may assume the Al agent has general sense of your domain, like new graduate student with undergrad-Ieve knowledge but not much about the specific task. Please help by providing some knowledge to write the program for this task. You can search online for more details about the dataset, packages, and functions used in cach task before writing.
Please try not to "Ieak" the annotated program directly. You may imagine that you don't 'o have the direct answer but could provide some helpful information your junior colleague that they derive the program. For example:
Instead of copying/describnng few lines in the program, you may copy the documentations of pack- ages/functions used in that program.
Instead of specifying the variables and parameters, you may suggest range (e.g.. 1e-3 to 1e-4 for learning rate).
Instead of saying columns A,B,C are related to the target attribute Y. you may try to find a knowledge snippet describing what is correlated to Y.
However, in some rare cases, there may be need to provide minimal "leak" of the annotated pro- gram, e.g., the decision boundary of Y is 0.6 instead of 0.5. Still, it would be great you could think about its necessity before annotating such knowledge.
For each piece of domain knowledge related to this task, please write 1-5 sentences. If you believe the lask instruction is self-contained and needs no further explanations, please enter "None". [Long Text Answer]
Scoring Rubric
Once the Al agent generates program, we need an evaluation method to review the generated pro- gram. To do it, we need task-specific scoring rubric, which assigns partial credits for more com- prehensive evaluation of the generated program. Right now we have already got initial draft of the rubric with five major components: (1) data loading, (2) data processing. (3) modeling, analysis or visualization, (4) output formatting, (5) saving output.
Please review our initial draft of the rubric. Imagine that you will use this rubric to score the programs produced by your junior students. Please modify the rubric items that you think are incorrect, should be described with more/less details, should be reweighed with higher/lower credits for each component. Please also add any missing but necessary rubric item you would use to assess a program's 's correctness, or remove redundant rubric items.
Please enter your revised scoring rubric below. there are no changes, please skip this question and leave the answer text blank.
[Long Text Answer] "The example for domain knowledge annotation is provided in Section G.3
40

---

Published as a conference paper at ICLR 2025
G.2 PROGRAM EXAMPLE FOR DOMAIN EXPERTS
Task Instruction: Train graph convolutional network the given dataset to predict the aquatic toxicity of compounds. Use the resulting model to compute and visualize the atomic contributions molecular activity of the given test example compound. Save the figure as pereeddesuuts/aaquuic..ooo serpiiiitt_ssarrveie qssr_.vs.png".
Program:
Amport 08
cs.enviren
From rekit import Chem
from rekit. Chem. import Simitaritydlpps
Amport
pd
import deepchem as de
10 def vie_cont: rie imol, dt, oni_or_edC *ad[");
11
WE ()
12 smi_or_add "smi":
13
for n, arom in enumerate!
14
Chem,
Canoni RankAAooms
15
):
éf. (mat.Get/rop! "_Name"), "Contrib [a]
17
if omi_or_dd_ "sdr";
18
for n, azon in enumerate (range (mol. Get.umieavyttcnn (1) )
19
[atam] éf ef.loc[Chem, [Chem. Nolma les (mol), "Contrib"] [n]
20
return 5lml tarityyaps..Gtt Sem Iarl larltyMapP romte (mol wt) wt)
22
def (a
ONTASET_FILE as.path.joini
24
panchmark/tatasess/apae lo toxicity', sC
25
26
27
28
29
Mtt.m..0Mllsuupp..c.oo (DATASET_FILE)
31
if is not None
32
33
loader www.datt.sFF.aaae..
34
Lasks=("GGC50"
featurizer=de. Conteoleeeesssrriieeee)...
36
sanitizesirue
38
dataset loader www.etereeaaeeeeeeeeeeeeeeeeeeee (DATASET_FISA) shard_eize-5000)
39
40
GraphConwWodel
41
1,
mode= rregression",
43
batch_ soachh_nomalllooFaaeee
45
m.fit (dataset, nb_epoch-10)
46
47
RGST_DATASET_FIT oa pa.path. joint
48
50
teat_mol
52
53
for m in Chem. wtMo (TST
54
55
1101
test_ datesas saaderr.rraae_ddttaeee
57
TEST BEST__ATARRT_FTEE PTTE,
shaed_size=5000
59
60
LDFFLeee "asks=(),
62
featuri zerade. wow.eool per_atto__raggmettccorroo
64
66
sanitize=irue
67
frag_datese: Losder.cresc__dttssee.
69
TEST DAST__AATSET_FIA 5120-502
41

---

Published as a conference paper at ICLR 2025
72
73
de.trans www.teeningrraneeooonooooo former dacastd -
74
frag_datese: tr. tra.sffsrm (frgg_ dataset) )
75
76
pred m.predict ) pred pd.DataFrame
77
pred,
79
168, columms-(""olecui
80
81
82
83
gred_frags m.predict (fra_ddataset]
84
pred pred_frage = pa.DataFrame
prad_frags,
86
mndex-frag_dataeet...
87
88
89
90
mermerge (prerprrrrreee fraga, pred, right_ Trdex--rx, left_index-Truel dCl'Contrib) di "Molecule"] df dF("raagmett")
91
93
vie stsscoottrbbssttte_mtto dr)
94
vis sivefig
95
96
fnn__inchess'ee tight"
97
98
100
main ()
Explanation:
In this example, there three key points in the instruction: (1) GCN training (lines 28-45), (2) calculating atomic contribution (lines 76-91), and (3) visualizing atomic contribution (lines 10-20). In case, you can select "Yes" for first question and move on.
Suppose the given program is not training a GCN at line 33 but, say, a simple feed-forward neural network, may select "Need Modification" and comment "Line 33" in the follow-up question. However, if more than three lines of code have errors, please seleet "No". More clarifications:
(1) The annotated program should be treated as a "reference solution" to the task. As you may have already noticed, these tasks are open-ended and can have multiple valid solutions. So, although the annotated program may import certain classes and packages, don't want to force the agent to necessarily do the same the "task instruction." But, if you find the classes and packages helpful, feel free to mention them "domain knowledge."
(2) The agents will be able to install packages for themselves via pip. For example, if it chooses to use mastml, should use "pip install mastml" to set itself up. During annotation, we tried to make sure that all packages are distributed via pip so that the agent should be able to install, but there might be few mistakes. the program something that is available via pip but critical completing the task, please us know.
42

---

Published as a conference paper at ICLR 2025
G.3 KNOWLEDGE EXAMPLE PROVIDED DOMAIN EXPERTS DURING ANNOTATION
Task Instruction: Train (1) graph convolutional network the given dataset to predict the aquatic toxicity of compounds. Use the resulting model to (2) compute and (3) visualize the atomic contributions molecular activity of the given test example compound. Save the figure as aegeerrrsuutsaauattttttocc.. paaa_."s.png".
Program:
import 08
ce.enviren ("TE LBGACY KERAS";
From rekit import Chem
from rekit. Chen. import Simi-aritydapps
Amport
import deepchem as de
10 862848
11
(3) This pazt defines a function for visuelizing atomic contributions. One reievant piece of domain knoxledge you might junior atudent working on this task is about haw to draw atomic contributions rokít), e,g., mentioning
provida the your
required fonstions.
12
def vis_contriba (mol,
or_ *sdE");
14
-
15
ani_or__dd "smi":
16
for in enumorate!
17
Chem, romol lenon...nnnn ca RankAroms (mol)
18
19
Mt [atom] - [.f.loc[moi..... (mai.GetProp (* "_Name",, *Contrib") "] [a] ami_or_ad$ "scf":
21
for n, 3rom in enumerate ixange (mo) Nww.oeetnaaaaa..o.... [1) 1 Mt [atom] éf. éf.
22
Fff.lo/(Chee [Chem. Molmom (mol), "Contrik") [n]
23
Simi larityyro Simm::: lard tyrapy romMei ght (mol, wt)
24
25
26
main():
28
DATASET_PTTS PTCB oa.path. Join!
29
'benchmark/ 'benchmark/bataseeta http://aaaatttttttaaiaaaaaa http://aattitttiiiittoooooo toxicity',
30
*Tatrahymena pyrt formia a_mma_acccsMdd
31
32
33
#*80##
$ (1) This part leads the data trains GOM. One reievant piace domain
knowledge you might want provide to the AI agent oz junioz student working on thia task is about what rac50 means and why that column is the gold Jabel aquatic toxicity.
35
36
37
Chem. Supp! (DATASET_FA) (FSRR)
39
if is not None
40
loader dd.data. SDFLoader
42
tasks=("IGC50)
featurizermde. wtw.weelrrssrrrieeeet)).. te sanitizesTrue
44
46
datase: ceaeeer.creeee_deeeeee (DATASET_PISA shezd_sise==000)
47
48
model. GraphhCoavWddel
49
1,
"rggeesigg,,,,
51
batch_ cacth_norraall11 12g.Fall1g-aaaaa
52
53
i.ittads nb_epoch-10)
54
55
*8838*
56
TEST DATASET os.path.joing
58
penoomaekk/attaaesssapaee wt..itittttoxxitt....o..o
59
60
61
tast_mal =
63
for in them. Scpplsupplerr (TEST_DXTASET_
43

---

Published as a conference paper at ICLR 2025
1101
66
tast__ test_datesss dotosat Lowder.crecte_cheeeee..... __AAAAST____EEE
68
shard_size=5000
69
70
71
loader
tasks=(),
73
featuri serado, feat Convie ConnvelPeeaaeee Peaturizer
75
76
sanitizemirue
77
78
Trag Trag_datase: datase: loader, creett
TEST_ DATASFTT
80
si2e__30
82
83
tr teanra wt..rann. slatteningtrannooooor (frog_ trag_datases tr.transform(ffrag..d [
84
85
87
# (2) This part
the Lzained GON predict the Lest example's toxicity and calculate the stonic contributions. One zelevent piece of domain knowledge you might
provide the 44 taak to about how atontc contribution may be calculated, 1.e. predicting the toxicity
junior student working
compound those cospound fragmenta (with removed), then making a subtraction to find the contribution of the resoved
88
8y
pred m.predict (test_dateeet)) pred rd. pred,
90
91
92
Lndex=lestt_dlaseee. columns- ("Molecule""""
93
95
96
gredffrap n.predict (atag]dttaset)
97
pred_frage frags po.nataFrame
98
pred_fraçs,
index==fee__dataeee..tt
100
columns" ("szagmens")
102
103
= pd. wpred_ frogo, pred, right_indexest Trdux- loft_index--ruel c'nntriil of (Molecule") of ("Fragment")
105
#88*8#
107
vis vll___vvaaaattiaaaaoaaa tos_tootttrisssttt_moll df)
109
sis.savefig(
"pred_resultaa"
bbox_incheg=1 tight"
112
114 if
name moin
main()
44

---

Published as a conference paper at ICLR 2025
H RUBRIC EXAMPLES
In this section, we show two rubrics generated by GPT-4o (Listing H.1 H.3 and their final versions revised by domain experts (Listing H.2 H.4).
Listing H.l: An example rubric of Computational Chemistry task generated by GPT-4o without expert revision.
"data_loading"
"name"; "Initialize Data Loader for Training", "description ion": "Succesefully initializes the MyClinzoxtoeder object for training data with correct parameters: featurizer"' ECFF', tasks=(" "cr_rox") I, featuro_ amied=" and corract file path 'benchmark/datasatte/ eltooxx/llntte patto_ttraannca "points": 5
"name": "Load Training Dataset",
10
"description": "Successfully loads the training dataset using the train_leader objact and correctlly assigns the dataset to "aatsedttse,t." "points";
I1
12
14
"name"; "Iaitialize Data Loades for Testing",
15
dlescription" "Successfully initializes the kyClinzoxioader object for test data with parametors: festurizers/ ECFP', tasks= ['DAA_ 'CT_TOn ") "eature_ fiel sniles', and correct file path "sanchmart/catseers//ll ciittox_tes..ev'' "points":
16
18
19
"name": "Load Testing Letasse",
20
"description" "Succeasful"y loads the donsact using test_loader object and correctlly assigns the dataset to test_dataset.", "points":
22
24
"dsta_prooessing::
25
26
"name": "Transform nata",
27
"description": "Applies required LzansTormatioos to the training dataset using specified transformers le.g., 'belancing')
28
"pcints": 5
30
31
32
33
"name"; "Initialize Model",
"description". "Successfully initializes Xultitaskciessifier paramsters: numbor of tasks equal to length of CliNToX 1024 (00.0) tearning_ratewd> batch_stze=50.""
layer so
35
"points": 10
36
37
38
"name": "Fit Model",
39
"description": "Successfully fits
using train_dataset.""
40
"points": 10
42
43
"name"; "Precict Using Model",
"description": "Successfully the trained model predict scozes on test_datasee correctly applying any necessary tes__transfommrrs., "pcints":
46
47
48
"output_forrattingg
49
"name": "Borma: Cutput DataFrame",
51
"description": "creates pandas Datai'rame named cast_scores_de df containing smiles', AFERRVEE, "CT_Tox" with correctly assignad test scores.", "points":
52
53
54
"eutput_saviiee "cutput_eavingg::
56
"nams": "Save predict ona CSV",
45

---

Published as a conference paper at ICLR 2025
"description": "Correctly saves test_scores__d at pred_ wttp:tttss/ltttottttttttooo pred..cv" withost an index.",
"points":
60
61
62
"tatal_pointer points";
63
Listing H.2: An example rubric revised by an expert by adding the available points to two items.
"dada_l
"name"; "Initialize Data Loade: for Training",
"description": "Successfully initializes the dyClinzoxLoader object for training data with correct parometors: featurizer= sCFP', coaka=("FBA 'CT_ToX' feature_ rield="sm les', taa...
file bbenhmark//dttaatte
"peints":
10 "name":
"Load Training Dataset",
"description" ion": "Succeasfully loads the training dataset using the train_lozder and correctly assigns "points": 5
traln_dataset,""
11
13
14
"name": "Initialize Data Loader for Testing", "description": "Successfully initializes the syClintoxionder object for data with correct parametars: festurizer tasks= APTROVRYY, Tox_To Zeature field=' smiles', Dintoxx_est.csv'... "peints":
15
file path "yncclaack/@aaaeele//lldee
16
17
18
19
"nams":
"Load Testing Bataget",
20
"descript ion"; "Successtully loads the teat dacaset uaing the test_loader ject and correctly assigas "pcints": 5
test_dataset.",
21
23
24
data__ procopsocos:;"g"
25
26
"name"; "TransCorm Data",
"description": "Applies zequired transformations training using the specified transformers (e.g., 'balancing') 1.", "points":
29
30
31
Lordeeiingo___alllaaao.ooo isationn
32
"name": Minitialize Model",
34
"description" "Successfuliy instializes the Sultitaskclassiiie with parameters: number tasks equal length of CITNTOX TASKS, n_featuree-1024,
[ive__siee==1300] (ropouts=(0.25), jearaing_:ate=0.000 and batch_eiae=50.1" "points": 15
35
36
37
38
"name": Model",
39
"desoription": "Succeasfully fits the model using the train_dataset...
40
"points": 10
41
42
43
"name"; "Bredict Using Model",
44
"description": "Successfully uses the trsined model to predict scores on the test_ best_dataset, correctly applying any necessary test_transtormarss." "points": 10
45
47
48
"oopput_formactingg:
50
"yormaz Output DataFrame", "description "greates a pandas DataFrame named tast_scores containing smiles', "FGA_APEROVED", and rCT_ToX" colunns with correctly assigned teat scores.",
"name":
52
"peints":
53
55
"ourpot_aaving":
46

---

Published as a conference paper at ICLR 2025
"name": "Save Predictions csv", "description"; "Correctly saves the cest_scorra__dt to cav file at pred rree_reeutaaooLiioootttet..op pred.cccy without an index.",
58
59
"peints": 5
60
61
62
*total pooal_ppints" 75
47

---

Published as a conference paper at ICLR 2025
Listing H.3: An example rubric of Geographical Information Science task generated by GPT-4o without expert revision.
ddata "dala_loading":
"name"; "Load Bethymelry Date", "description". "Correctly Loads the bethymetry zaster date from the pata bencimarik/dataa https//ccrrrrsogg//aaaaaaaaa "points": 10
"name"; "Load Coral and Sponge Data",
10
"descript ion"; "Correctly reads the coral and aponge data from the path bencimark/ http:///aaaaspoeooooooaaoooooooo geonjsssnn .",
11
"points": 10
12
13
14
"name":
cCs trensformation",
15
"description"" ion": "Correctly traastorms the CRS of the GeoDaLaFrame to BF8G
:432..
16
"points": 5
17
18
datt_ppoccssiing:
20
21
"name": "Eleva:ion Conversion",
22
"description"; "Correctly slevation values multiplying with -1.",
23
"points": 10
24
25
"name"; "Caiculate Gradient", "description": "Accurately calculatas the gradient (grad_x, grad_y) using rumpy 's gradient function.",
27
28
"points": 10
29
30
31
"name": "Calculate Slope",
"description": "Correctly calculates the slops degrees gradients. "points": 10
33
35
36
"Ca culate Aspect", "description" "Correclly calculates the aspec. degcees and adjuste any negative values.",
"name":
37
38
"pcints":
39
40
41
"nams": "Coordinate Raster Index Convarsion",
42
"descript ton"; "Correctly implements the function to convert coordinates to zaster grid indices.", "pcints": 5
43
45
"name": "Extract Slope and Aspect",
47
"descript ton": "Extracta ope and sapect values GeoDaLaFrame correctly.",
in
48
"peints":
49
51
"name"; "Add Slope and Aapect to GeoDataPrame",
52
"descript ion": "Succesefuliy edds the extrected slope and aspect velues as new columna the GeoDataPrame.
53
"points": 5
55
56
"Group ay Vernacul teaamaatrraa dieerription"; "Correctly groups the GeoDataFrame 'Vernnecularllmeeeaeeeyyy and computes mean values for slope and sspect.", "points":
"name":
57
58
59
61
model "modeling or anaaoorsaall _rr_s_rrrrsssss "oansliattin ization":
63
"Bar Plot for Mean Slope", "description": "Correctly creates ber plot showing the mean slope per species
"name":
64
65
"points": 10
67
"name": "Bar Plot Ioz Mean Aspect*,
48

---

Published as a conference paper at ICLR 2025
"description": "Correctly creates plot showing nspect pez
species.
70
"points": 10
71
72
73
pueput_ffrrsstinng
74
"name";
Descriptions",
76
"desoription" "Properly sets plot titles, axia labels, and ensures x-tloks are rotated for readabiLity.",
77
"peints": 5
78
79
80
puutput_saving"
82
"name"; "Save Plots",
"description". "Saves the plots as "meen_slope_per__pe
and pppee_eeeuuls//ootaaaadppepppeee
84
"points":
85
86
1
points";
88
Listing H.4: An example rubric of a Geographical Information Science task revised by expert by reducing the available points for several items.
*data_ _data__ooddi loadingg ng":
"name": "Load Bsthymetry Date",
"description": "correctly Loads bathymetry raster sttpaare//atttttss/sssssssssss Sponge/Catal ca "pcints":
path
"name"; "Correctly reads the coral and sponge data from the path "description".
and Sponge Data",
10
bancimmrtc/attaatt//aaaa httpa//sooooooorraaaaaaaaoooaaaa gon.ooooon
II
"points": 5
12
13
14
"name"; CRB Trama
"description": "Correctly transforms
:4326. ",
16
"points":
17
18
1.
19
:ata_processinn":
20
"name": "glevazion Conversion",
22
"deseription"; "Correctly converts alevation values by multiplying with -1.",
"points":
24
25
26
"name"; "Calculate Gradient",
27
"description" "Accurately calculates the gradient (grad_x, gradLy) using numpy gradient function.",
28
"points": 5
29
30
31
"name"; "Calculate Slope",
32
"description" "Correctly calculates the slope in degrees gradients. ",
33
"pcints": 10
35
36
"name"; "Calculate Aspect", "description": "Correctly calculates the aspect in degrees and adjusts any negative values.",
37
38
"points":
39
41
"name": "Coordinate to Raster Index Conversion",
42
"descript ion": "Correctly implements the function to convert coordinates to grid indices.",
43
"pointa": 5
45
"nams": Slope Aspect*,
49

---

Published as a conference paper at ICLR 2025
"description": "Extracts slope hspect GeonstaFrame correct-y.",
point "nams": "Add Slope and Aspect GeatiataFrame',
"points"; 10
49
50
51
52
"descript ton"; "Succeastul y adds the extracted alope and aspect valuea aa new SeotataFrame.",
columns
53
"points": 5
55
56
"name": "Group by VermecilarkammCaaaporr''
57
"description ion":: "Correctly groupa the GeonatoFrame by 'Vernacu tnaaaaaategrr and computes mean values for slope and aepect.", "points":
59
60
61
62
63
"name": Plot Slope",
64
"description"; "Correctly creales baz plot showing the mean slope per species
65
"points": 10
66
67
68
showing aapect per "name": "Bar Plot for Mean Aspect", "description"; "Correctly species.",
"peints":
71
72
73
"cutput_ foreur_.ormm:tinng::
74
75
"name";
"Plot Descrigtions",
76
"description": "Properly sets plot titles, axis labels, and ensures x-ticks are rotated readability.',
77
"peints": 5
78
79
80
"output_aaving
W2
"name": "save Plots",
"description": "Saves plots boan__llep_per__ppccicee pen_eppct_pe__speieee. and ppree_eeulls//orrtaaadapggggg "pointa":
84
85
86
1r
Miotal_points" 100
88
50

---

Published as a conference paper at ICLR 2025
1 PROMPT TEMPLATES
In this section, we document the templates used to prompt LLMs for different frameworks (Section
3 direct prompting (Table L.I. self-debug (Table T.2). and OpenHands CodeAct (Table I.3
Table I.1: Prompt template for direct prompting (Section domain knowledge is optional.
You are an expert Python programming assistant that helps scientist users to write high-quality code to solve their tasks.
Given a user request, you are expected to write complete program that accomplishes the requested task and any outputs in the correct format.
Please wrap your program in code black that specifies the script type, python. For example: "python
print ( HHlloo World!!r)
Please keep your response concise not use code block not intended executed. Please do not suggest few line changes, incomplete program outline, or partial code that requires the user to modify. Please do not any interactive Python commands in your program, such Mpip install nampy', which will cause execution
Here's 's the user request you need to work on:
{task instruct i (domain.knowwedee. }
You can access the dataset at "[dataset.pata)" Here the directory structure of dataset:
{dataset_ wolder_treedeeeeeeeeeeeee
Here some helpful previews for the dataset file(s):
(datase previeew
Table I.2: Prompt template for self-debug (Section domain knowledge is optional.
You are an expert Python programming assistant that helps scientist users to write high-quality code to solve their tasks.
Given user request, you expected to write complete program that accomplishes the requested task and save any outputs in the correct format. Please wrap your program in code black that specifies the script type, python. For example: "python
print ( 'Hello World!!!)
The user may execute your code and report any exceptions and error messages. Please address the reported issues and respond with a fixed, complete program.
Please keep your response concise and do not use code block if it's not intended executed.
Please not suggest few line changes, incomplete program outline, or partial code that requires the user to modify.
Please do not use any interactive Python commands in your program, such as "lppp install numpy', which will cause execution errors.
Here's 's the user request you need to work on: [taak_instt ructr on)]
idoma k_kkowl edge]
You access the dataset "(dataset path) v. Here the directory structure of the dataset: '**
(dataset..folder.tee
Here are some helpful previews for the datasct file(s); |datase.preview
51

---

Published as a conference paper at ICLR 2025
Table I.3: tional.
Prompt template for OpenHands CodeAct (Section womain.knowledee is op-
You are an expert Python programming assistant that helps scientist users to write high-quality code to solve their tasks. Given user request, you expected to write complete program that accomplishes the requested task and any outputs
wwworksaaee///eeooo wwt.s/ppcceeeeeeeeeeeeee prw. results/ in the correct format.
Here's 's the user request you need to work on:
(task_inst instiiitstuuttiiii i
[domain.knon enowledgee
You can access the dataset at [[dataset.path)- Here the directory structure of the dataset:
{dat teettaatt...d...tte.
Here helpful previews for the dataset file(s): {dataae preview]
Please save your program ' /wwrkspacc///////c / hxteLpppooocess/pppsspppppaaaaap Then, please the program check and fix any errors.
Please do NOT run program the background. If the program uses some packages that are incompatible, please figure out alternative implementa- tions and do NOT restart the environment.
52

---

Published as a conference paper at ICLR 2025
J PUBLICATIONS, REPOSITORIES, AND LICENSES
In this section, we list all referred publications (Table J.1 J.2 and repositories (Table J.3 during
data collection (Section 2.2 We also include the repositories' licenses in Table J.3
J.4
and J.5
Table J.1: List of Bioinformatics and Computational Chemistry publications referred to during data collection (Section 2.2
Domain
Title
Citation
Automated Inference of Chemical Discriminants Biological Activity
Raschka ct al.
2018
CellProfiler: image analysis software for ideatifying and quantifying cell phenotypes
Carpenter ct al.
2006
DeepPurpose: A Deep Learning Library for Drug. Target Internction Prediction
Huang
2020
ADMET-AI: a machine learning ADMET platform for evaliation large scale chemical libraries
Swanson et al. 2024
Prediction and mechanistic analysis drug-induced liver injury (DILf) based on chemical structure
Bininfomatics
Liu et al.
2021
SCANPY: large-seale single-cell gene expression data analysis
Wolf al.
2018
A Python library for probabilistic analysis of single-cell omics
Gayosso al. 2022
MUON: multimodal vanics analysis framework
Bredikhin al.
2022
Scirpy: Scanpy extension for analyzing single-cell Ticell receptur sequelicnseunnnigggg data
Sturm et al. 2020
The severse project provides computational ecnsystem for single-cel cell omics data analysis
Virshup et ml 2023
MoleculeNet: beachmark for molecular machine learning
Wu al. 2018
Accelerating high throughput virtual screening through molecular pool-based active learning
Graff ct al. 2021
Multiask Deep Learning Practical for Pharma?
Ramsundar
Discovery structural class antibiotics with explainable deep learning
Wong ct al 2024
Papyrus: large-scale curated dataset
Béquignon et
2023
aimed at bioactivity predictions
ProL.I:; library to encode molecular
Bouysset Fiorucc
2021
interactions as fingerprints
Computational Chemistry
Python Materials Genomics (pymalggen:: A robust. open source python library for materials analysis
Ong et al.
2013
Benchmarks for interpretation QSAR motelss
Matveieva & Polishchuk
Matminer: open source taolkit
Ward et al.
2018
for materials data mining
The Materials Simulation Toolkit for Machine learning (MAST-ML): An automated open source toolkit to accelcrate data-driven materials research
Jacobs et
2020
Robust model benchmarking and bias-inbalance data-driven materials science: case study MODNet
De Breuck et all
2021a
Materials property prediction for limited datavets enabled by fenture selection and joint learning with MODNet
De Breuck
202
Bioinfomatics & Computational Chemistry
Deep Learning for the Life Sciences
Ramsundar
2019
53

---

Published as a conference paper at ICLR 2025
Table J.2: List of Geographical Information Science and Psychology & Cognitive Neuroscienee publications referred to during data collection (Section
Domain
Title
Citation
Library Analysis
Meteorological, Oceanographic and Climate Data
Dawson
2016
The Open Global Glacier Model (OGGM) vi.1
Maussion et al
2019
Human selection of elk hehavioural traits in a landscape of fear
Ciuti
2012
Investigating the preferences of local residents toward proposed bus redesign in Chattanooga, Tennessee
Ziedan al.
2021
Urban wildlife corridors: Building bridges for wildlife and people
Zellmer & Goto
2022
Urban climate effects
temperatures
Schatz Kucharik
2015
in Madison, Wisconsin, USA
Geographccal Information Science
Model Animal Home Range
Flemning
2024
Run geoprocessing tools with Python
Zandbergen
2024
Model How land subsidence affects flooding
Andeweg & Kuijpers
2024
Predict deforestation Amazon
ESRI
2024a
NOAA Deep
Research and Technology Program
Hourigan
Chart coral and sponge distribation factors with Python
Rabinson
2023
A Assess access to public transit
ESRT
2024h
Build a model to connect mountain lion habitat
ESRI
2024e
Analyze urhan heat using kriging
Krause 2024
Assess burn sears with satellite imagery
ESRT
2024d
BioPsyKir Python package for
the analysis of biopsychological data
Richer al.
2021
NeumKir2: A Python toolbox for
Acurophysiologicl signal processing
Makowski
2021
Psychology & Cognitive Neuroscience
Modeling Human Syllogistic Reasoning:
The Role of "No Valid Conclusion"
Riesterer al. 2019
Analyzing Differences via Joint Nonnegative Matris Factorization
Reasoning
Brand et 2020
Generate neural signals from mine: individual to individual EEG converters
Giolomb
54

---

Published as a conference paper at ICLR 2025
Table J.3: List of 31 repositories adapted during data collection (Section 2.2 and their licenses. AAdpttinn allowed for non-commercial use; we include their full licenses as Table J.4 and J.5
GitHub Repositories
License
deepchem/ddepchhem
coleygroup/molpal
swansonk J4/admeL.an _ai
mhartn-sichoo/pppymee wuppyrus-caafolld-iisaaaaaa
Oliverrcc/PPapyyuss scripts
mad-lab-tau/Bee
mad-lab-tau/BioP'kkkii PyKit
hhaleralsprojecdpymmagger
MIT
neuropsychology/Neum
arresserrar/yyllogsstiiinnee
brand-d/cogsci-jnnt
UW-cmg/MAST-ML
ZioongLu199/EE622EB
Zitong
Kesudentt//arooggeopll
ppxcchrecckmmodnel
geopandas/geopandas
kexinhuang
telnxjwong/mmttiotttssa
Scilools/ms
OGGMoogem
BSD-3-Clause
scverse/scanpy
scverse/sevi-tools
scversc/muon
severse/scurpy
GooStattraaneoo///yykigggg
psa
laallabbpredicctingaaiiii by machine
machin--eaarninng learning
Apache-2.0
anikaliu/AAMDA--DI...
GPI.-3.0
ajdawson/eots
Tolve-CGesollatios//raaffoom 2022
CC-BY-3.0-AU
rasteriofrasterio
Copyrighted
hackkigmmatcrallsmmtmmnnnee
55

---

Published as a conference paper at ICLR 2025
Table J.4: License for rasterio/rasterio.
Copyright (c) 2013-2021, Mapbox All rights reserved. Redistribution and in source and binary forms, with without modification, are permitted provided that the following conditions are met:
Redistribution: of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
its contributors may used to endorse or promote products derived from this software without specific prior written permission.
Neither the
Mapbox
the
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR PARTICULAR PURPOSE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMP OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, FROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NRGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOF TWATR,RE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
PROPITS; BUSINESS
56

---

Published as a conference paper at ICLR 2025
Table J.5: License for hakiingmaterial/mmmminnnn
matminer Copyright (c) 2015, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject receipt any required approvals from the U.S. Dept. of Energy). All rights reserved.
Redistribution and use in source and binary forms, with or without modi fication, are permitted provided that the fol lowing conditions are met:
(1) Redistribution ions of source code must retain the above copyright notice, this 1ist of conditions and the following disclaimer. (2) Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
(3) Neither the name of the University of California, Lawrence Berkeley National Laboratory, U.S. Dept. of Energy nor the names of its contributors may used endorse promote products derived from this software without specific prior written permission.
THIS TWARE IS FROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILIIT AND FITNESS FOR PARTICULAN PURPOSE ARE DISCLAIMED IN NO EVENT SHALL THE COPYRIGHT GITT OWNER OR CONTRIBUTORS B8 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, CR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT THE
THIS SOFTWARE, EVEN ADVISED THE POSSIBILITY OF SUCH DAMAGE. You are under no obli inligation whatsoever to provide any bug fixes, patches, performance of the source code ("Enhancements") to anyone; however, if you choose to make your Enhancements available either publicly, or directly to Lawrence Berkeley National Laboratory or its contributors, without imposing a separate written license agreement for such Enhancements, then you hereby grant the following license: a non-exclusive, royalty-free perpetual license to install, use, modify, prepare derivative works, incorporate into other computer software, distribute, and sublicense such enhancements or derivative works thereof, in binary and source code form.
upgrades to the features, functionality or
57


Reviews:
[{"review_id": "6MpSK3S21A", "reviewer": "Reviewer_c5sH", "summary": "This paper introduces ScienceAgentBench, a new benchmark designed to test how well language agents can handle tasks in data-driven scientific discovery. The authors collected 102 tasks from 44 peer-reviewed papers across four scientific fields: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience. Each task asks the agents to write self-contained Python programs to perform specific scientific activities like data processing, model development, analysis, and visualization.\n\nTo ensure the tasks are authentic and to prevent issues with data contamination, the authors involved experts from the respective fields and modified datasets so agents couldn't rely on memorized code. They evaluated five large language models using three different frameworks: direct prompting, OpenHands, and self-debug. The results showed that the best-performing agent could complete only about one-third of the tasks. This highlights the current limitations of language agents in fully automating data-driven scientific discovery and suggests that more advancements are needed.", "questions": "1. Have you considered evaluating state-of-the-art frameworks like ReAct or Toolformer incorporating advanced reasoning and tool-use capabilities? Including these could offer deeper insights into the agents' performance on complex tasks.\n2. Since evaluators were also involved in data collection, how did you mitigate potential assessment bias? Would involving independent evaluators improve objectivity?\n3. Could you provide a more detailed analysis of the standard failure modes encountered by the agents? Understanding specific errors might help identify areas for improvement.\n4. Have you compared the agents' performance with traditional methods or domain-specific tools? Including such comparisons could help assess their practical utility relative to existing solutions.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces ScienceAgentBench, a novel benchmark for evaluating language agents in data-driven scientific discovery tasks. By incorporating tasks from four diverse scientific disciplines—Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience—it creatively applies language agents to new domains, filling a gap where existing benchmarks fall short.\n\nThe benchmark is rigorously developed with input from nine subject matter experts, ensuring tasks are authentic and challenging. The authors proactively mitigate data contamination by modifying datasets, enhancing the reliability of their evaluation. They use comprehensive evaluation metrics—including Valid Execution Rate (VER), Success Rate (SR), CodeBERTScore (CBS), and computational costs—to provide a holistic assessment of agent performance.\n\nThe paper is well-organized and written, utilizing figures and tables to enhance understanding. The authors provide insightful analyses of experimental results, highlighting why current language agents struggle with these tasks. By releasing all code and data, they promote open science and collaboration, significantly contributing to the advancement of AI in scientific research.", "weaknesses": "1. The paper evaluates agents using three frameworks but doesn't justify these choices or explore advanced architectures like ReAct or Toolformer. Without including state-of-the-art frameworks that offer advanced reasoning and tool-use capabilities, the study may not fully assess the agents' potential to handle complex scientific tasks. Incorporating such frameworks could provide deeper insights into their capabilities and limitations.\n2. Human evaluators who also participated in data collection may introduce bias due to familiarity with the tasks, affecting the objectivity of the assessments. Additionally, the error analysis lacks depth, as specific failure modes are not thoroughly examined. Involving independent evaluators and conducting a detailed error analysis would improve objectivity and help identify areas where agents struggle.\n3. The paper doesn't compare the agents' performance with traditional methods or domain-specific tools, making it difficult to assess their practical utility relative to existing solutions. Including such comparisons would provide valuable context to evaluate the agents' real-world usefulness and guide future improvements.\n4. Providing expert domain knowledge doesn't consistently improve agent performance and sometimes even decreases it, suggesting agents struggle to integrate this information effectively. Exploring why agents fail to benefit from expert knowledge could lead to better integration strategies and enhance their overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "si5sgrgHcw", "reviewer": "Reviewer_fdqA", "summary": "The ScienceAgentBench framework is introduced in this paper to assess the data-driven scientific discovery capabilities of LLM models. The framework offers both end-to-end and fine-grained metrics in evaluations. Significant room for improvement in scientific tasks was confirmed by implementing the benchmark on various sota models. The benchmark has the potential to serve as a long-term progress indicator for LLM models on scientific reasoning capabilities.", "questions": "Why was VER chosen over CBS when ranking models? High VER but low CBS could still indicate good context understanding, though poor execution. Was it considered to use heuristics / weighted sum to combine all metrics in the final evaluation?\n\nWill setting CBS to 1.0 when SR is 1 introduce bias into the metric? Some argue that this specific treatment can skew the metric's results. While CBS may not be ideal when the model employs a different approach than annotation but still arrives at the correct answer, setting it to 1.0 could lead to inconsistent score interpretations. Additionally, if the ranking is order-based, this specific treatment might not have a significant impact.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The study involved extensive data curation and human annotation, demonstrating the authors' dedication and thoroughness. The inclusion of both end-to-end and fine-grained metrics allows for a comprehensive evaluation of models, particularly when the models can only partially solve a problem. Additionally, the exploration and discussion of various interaction methods with the local environment provides valuable insights.", "weaknesses": "Coding generation-related tasks may not be representative of some other scientific domains. While recent research has focused on such tasks, the authors could briefly acknowledge this limitations, especially since the benchmark's name suggests a more comprehensive evaluation of broader scientific capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UxXce9W7YB", "reviewer": "Reviewer_Ws3K", "summary": "This paper introduces a novel benchmark, ScienceAgentBench, designed to assess language agents' performance in data-driven scientific exploration. It meticulously curates 102 diverse tasks sourced from 44 peer-reviewed publications spanning four disciplines (Bio, Chem, Information Sci, Psy & Cog Neuroscience), subsequently validated by nine subject matter experts. Employing a variety of evaluation metrics, the study examines the efficacy of generated programs, their execution outcomes, and associated costs. By evaluating five LLMs, including both open-weight and proprietary models, across three frameworks—direct prompting, OpenHands, and self-debug—the findings underscore the current limitations of language agents in generating code for data-driven discovery.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) Writing: The clarity of this paper make it well-written and easy to comprehend.\n\n(2) Benchmark: This paper introduces ScienceAgentBench, a framework tailored for assessing language agents in the realm of data-driven scientific exploration. It emphasizes scientific authenticity through collaboration with subject matter experts, establishes rigorous evaluation criteria, and maintains meticulous control over multi-stage quality assurance.\n\n(3) Experiments: The paper evaluates three open-source models and two API-based models, conducting detailed assessments and in-depth analyses to provide comprehensive insights.", "weaknesses": "(1) It appears that the emphasis of this paper leans more towards Data Science or data-driven discovery rather than scientific discovery.\n\n(2) Task Annotation in Section 2.2 seems labor-intensive and time-consuming due to the involvement of identifying code, preprocessing data, implementing code, and writing dataset information. Are there any automated annotation or data collection methods available?\n\n(3) How is the ground truth for each task defined and generated? Are there any automated validation methods that could streamline this process instead of relying solely on multiple rounds of manual validation by annotators?\n\n(4) Could you elaborate on how the evaluation criteria outlined in Table 1 were established?\n\n(5) Regarding the validation of generated Python programs during inference and the utilization of CodeBERTScore to assess token-level embeddings, have you considered employing a self-consistency strategy to validate multiple outputs over time?\n\n(6) How is the validity of outputs generated by GPT-4o for the four heterogeneous datasets depicted in Figure 1 verified?\n\n(7) Given the focus on code generation for data science, have you considered evaluating or providing the performance of code generation models like Codellama and DeepSeek-Coder?\n\n(8) There appears to be inconsistency in the citation format, as observed in instances such as line 249 and line 251. Would it be possible to ensure uniformity in citation formatting throughout the paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)