CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Authors: Zachary S Siegel
Sayash Kapoor
Nitya Nadgir
Benedikt Stroebl
Arvind Narayanan
Venue: Accepted by TMLR ()

Abstract: AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves reproducing the results of a study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language-only and vision-language tasks. We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to a sequential implementation. We evaluated two baseline agents: the general-purpose AutoGPT and a task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 19% on the hardest level of tasks, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is a necessary step toward building agents that can conduct novel research and could verify and improve the performance of other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.

Published in Transactions on Machine Learning Research (01/2025)
CORE-Bench: Fostering the Credibility of Published Research Through Computational Reproducibility Agent Benchmark
Zachary S. Siegel
zachary@ssegel.com
Princeton University
Sayash Kapoor
sayashhddpprnnceton..dd
Princeton University
Nitya Nadgir Princeton University
mm18887@ppinncttnn.ct
Benedikt Stroebl
stroebl@@princeton.ddd edu
Princeton University
Arvind Narayanan
arvnndn@cs.princctto.com
Princeton University
Reviewed on OpenReview: / net/ forum? id= BMMMc4MMGS
Abstract
AI agents have the potential to aid users variety of consequential tasks, including conducting scientific research. To spur the development. of useful agents, need bench- marks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to mcasure the accuracy of AI agents in lackling crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves re- producing the results of study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language- only and vision-language tasks. We provide an evaluation system to mcasure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to soquential implementation. We evaluated two bascline agents: the AutoGPT and task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 19% on the hardest level of tasks, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is necessary step loward building agents that can conduct novel research and could verify and improve the performance other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.
1 Introduction
An article about computational science in scientific publication is not the scholarship itself, is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures. [Buckheit & Donoho 1995)
Computational reproducibility, the ability to reproduce the results of a scientific study using the data and code provided by its authors, is fundamental to scientific research (Medicine) 2019). Yet, recent studies

---

Published in Transactions on Machine Learning Research (01/2025)
Task Prompt
Run P40-PoscColeettion Analysis. using Rscript.
Report
Task Questions
Fron the Experinental IAG Data graph, zeport y-axis label.
"Fron the Experinental IAQ
Data graph, zeport the y-axis label.
2. Fron the Indoor Air Quality Kitchen Auturn plot, corzelation
Virtual Machine
"Gas Resistance" "Fron the Indeor Air Quality X Kitchen Autumn plot, the correlation between hun and gas.": -0.642
hur
and gas.
Agent
co Code Repository
readmeed
expDena..co
I4Q FostColl.
MScDoroKit..
Figure Overview of CORE-Bench. Fach lask in CORE-Bench requires an agent reproduce the results of research paper given its repository. The agent must install libraries, packages, and dependencies and run the code. If the code runs successfully, the agent needs to search through all outputs to answer the task questions. The agent submits report and is evaluated against the results of a successful reproduction. An agent successfully completes task if it correctly answers all questions about a code repository.
have documented severe shortcomings in the state of computational reproducibility across fields including psychology (Hardwicke ct al.
2021 Obels et al. 2020
Hardwicke et al. 2018), economics (Gertler ct al.
2018; McCullough et al. 2006], medicine (Naudel el al.
2018), political science (Stockemer al. 2018 life loannidis et al. 2009), geoscience (Konkol et al. 2019), and computer science (Belz et al. 2021 Raff 2019 Collberg & Proebsting 2016). Even if code and data accompany a study, reproducing study's i results can be challenging for many reasons: the software libraries used might not have their versions specified, researchers could use different. machine architect (ARM vs. x86) or operating systems (Linux vs. Windows vs. MacOS), old libraries could be incompatible with new hardware, there could be inherent variance in the results of study. To quantify this, surveyed evidence for the lack of computational reproducibility across fields, where papers were found to be irreproducible despite available reproduction materials (summarized in Table 1..
sciences (Andrew et al. 2015 Gilbert et al. 2012
Table 1: Computational reproducibility with data and code available across fields. There is a widespread issue in scientific research: even when data and code are provided, a significant proportion studies across diverse fields fail to be computationally reproducible.
Studies revlowed
Studies with comp. rep.
Studies rov levlewed Studles with compp. rep.
eryors
errors
Field
Paper
Field
Paper
Finance
Pérignon et al.
2024
1008
484
Economics
Gertler
2018
203
128
MI.
Sinha
2023
28
10
Medicine
Naudet
Multiple
Trisovic
2022
Political Science
Stockemer et al
2018
21
NLP
Bela et al
2021
5-19
472
Multiple
Wood et al
2018
23
Psychology
Hardwicke al
2021
25
16
Geoseiences
Konkol ct al.
2019
41
39
Psychology
Obels et al.
2020
36
15
Computer Sys.
Collberg & Proebsting
2016
6U1
311
Sociology
Liu & Salganik
2019
14
12
Blology
Andrew
2015
25
ML
Rafi
2019)
255
82
Molecular
Psychology
Hardwicke
Genetics
loannidis al.
2009
10
Economics
McCullough et al
2006
150
135
2

---

Published in Transactions on Machine Learning Research (01/2025)
CodeOcean Files
File Details
metadata
Capsule DOI, citation, licenses, author(s), associated publication
environment
Environment folder
Dockerfile
Instructions to generate Docker image with required dependencies
code
Folder with code
README.me
Project details (title, filenames, etc)
xun
Script to run code with appropriate arguments Code file
test.py
data
Folder with data necessary to run the code
results
Folder with results
REPRODUCING.n md
Environment details and exact Docker command that executes code
Figure 2: Files and folders in each CodeOcean capsule. Each capsule contains Readme, Dockerfile, and instructions on how to use Docker, which we sclectively provide to the agent depending on the difficulty of the task.
Machine learning (ML) is no exception. While introducing the NewrIPS checklist incentivized researchers to share data and code (Pineau et al. 2021), studies still lack computational reproducibility. To illustrate this, we collected the results of ML reproducibility challenges. The challenges consist of events that incentivize independent researchers to reproduce the results of studies in top venues. We analyzed the results of the 2022 challenge and found that only 18 of 28 papers that are accompanied by code and data are completely reproducible. Verifying the computational reproducibility of paper requires expertise. In some (6/28) cases, challenge participants could not fully reproduce results despite conversing with the original paper's authors.
The importance of uncovering and documenting reproducibility issues has been recognized in the ML community. As an example, reproducibility reports warrant publication in the poer-reviewed ML journal Transactions on Machine Learning Research (TMLR) F and earlier reproducibility challenges recommended graduate-level ML expertise for preparing reproducibility reports
Simultancously, language models have made significant strides in coding tasks, solving most tasks in benchnarks such as HunanEval (Chen et al. 2021). However, real-world coding challenges remain difficult for these models. More recently, the emergence of compound Al systems (Zaharia et al. 2024) has allowed for the completion of more difficult tasks. For instance, on SWE-bench, a BitHuub-based coding issue benchmark (Jimenez et al. 2023), language models alone achieve less than 5% accuracy, while agents boost this to over 30%. Such results have prompted claims that we will soon be able to automate most scientific research, especially in computationally intensive fields. For instance, one work builds an early-stage framework that uses large language models to automate the AI research process, from idea generation to paper writing Lu et al. 2024). However, designing evaluation schemes is difficult, and the quality of the Al-generated papers has been questioned (Koppel 2024). Before agents can automate scientific rescarch, they must be able to reproduce existing results.
In this paper, we ask: Can AI Agents Enable Verification of the Computational Reproducibility of Published Scientific Rescarch? Specifically, we evaluate whether agents can confirm whether the code associated with published papers is reproducible, as opposed to checking the content of the papers themselves reproducing without the code. We do not assess whether agents can determine if code is irreproducihle, but high performing agents could be used in such manner. We make two main contributions:
CORE-Bench (Computational Reproducibility Benchmark). CORE-Bench comprises 270 tasks derived from 90 papers across computer science, social science, and medicine with Python or R codebases. We source papers from CodeOcean.com and create tasks at three different difficulty levels based on available information in the repository. The benchmark involves diverse skills including coding, shell interaction,
IIn the 2020-2023 editions of the reproducibility challenge, pecr-reviewed reproucccblyyyy reports were published in the journal ReScieecc and httpps:/rrepronl.oorg
2 2 httpp:://ww.. cs. wttp://cw.mggll...a//peeeeeee/ee
3

---

Published in Transactions on Machine Learning Research (01/2025)
Medical sciences capsules filtered on all criteria
Computer Science, social sciences, medical sciences capsules 2,536
Initial capsules 5,090
Python and capsules 1,657
Social sciences capsules filtered on all criteria
28
Randomly selected Computer Science capsules 355
Removed capsules outside desired sub fields 2,554
Removed capsules not written in Python or R 879
Computer Science capsules filtered on all criteria
Figure 3: Capsule selection process. We filtered the 5,090 capsules CodeOccan by discipline, language, and the ten selection criteria to arrive at the 90 capsules selected for CORE-Bench. We provide a breakdown of capsules by discipline in Appendix A.4.
retrieval, and tool use. While many existing benchmarks include Python tasks (Cassano et al. 2022 ours one of the first to include tasks in R. Successful task completion may require multiple steps such as library installation, script execution, retrieval of code results, and figure interpretation using vision-language models. CORE-Bench's foundation in public repositories enables periodic updates of the benchmark tasks if it becomes saturated. An agent performing well on CORE-Bench would have real-world utility: authors could verify their work's reproducibility before publication, independent researchers could more casily replicate past studies, and conference organizers and journal editors could efficiently assess the reproducibility of submissions
Evaluation results on baseline agents. We evaluated two agents on CORE-Bench: the generalist agent AutoGPT (Significant Gravitas 2024 and task-specific version we built based on AutoGPT called Agent. Results show that generalist agents be casily adapted to specific tasks, yielding signifi- cant performance improvements. Our lask-specific agent achieved 58.52% accuracy on the easiest lask, demoustrating potential for automating computational reproducibility. However, performance dropped to 19.26% on the hardest task, indicating substantial room for improvement. We ran experiments with two different language models: GPT-4o and GPT-4o-min mini. To facilitate these evaluations, we are releasing CORE-Bench alongside an evaluation harness specifically designed for this benchmark, making it easy for developers evaluate their own agents on the benchmark. This harness runs each task in an isolated virtual machine, enabling parallelized testing, ensuring reproducibility, and maintaining clear separation between benchmark and agent code. The harness dramatically reduces evaluation time from over 20 days to mere hours by rumning on hundreds of parallel virtual machines.
2 CORE-Benche Evaluating agents on computational reproducibility
As the capabilities of Al agents continue expand, many claims have been made about their ability to sulonomously conduct research (Lu et al. 2024). But reproducing existing research is easier than conducting new research, especially when new research requires reproducing earlier baselines for comparison.
Recent work has introduced several benchmarks to evaluate language models and agents on various tasks related lo computer programming and scientifie research. These include benchmarks for conducting machine learning experiments (Huang et al.) (022)), research programming (Tian et al.) 2024), scientific discovery
Majumder et al. 2024h, performing scientific reasoning and citation tasks PPress al. 2024 Xu et al. 2024
TThe benchmark and code can found http:///gttaaccccccsssss////eert
4

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Capsule selection criteria. CodeOcean contains capsules from a variety of disciplines and programming languages. To create realistic and robust benchmark, we select capsules from CodeOcean that adhere the ten criteria in this table. These criteria ensure that CORE-Bench represents a diverse yet feasible subset of computational reproducibility tasks.
Criterion
Reason
Corresponds to a publicly accessible research paper.
Necessary for the scope of the benchmark.
From the fields of computer science, medical science, social science.
Allows for assessing changes in accuracy due to distribution shifts.
Allows for assessing changes in accuracy due to distribution shifts.
Written in Python or R.
Improves construct validity. Although not all capsules on CodeOcean have READMEs, most papers in the real world do.
Contains a README file.
Code runs in under 45 minutes on CodeOcean's hardware.
Ensures capsules are reproducible given our time and hardware constraints.
Allows for easy design of an English task prompt specifying how the code should be run for lasks where the agent does not have access to the run file.
Requires relatively simple Bash command to reproduce the code correctly.
Results are adequately labeled with figure, table, or file names in code output.
Eliminates the need to design task questions for disorganized or unlabeled data.
Ensures that all included capsules were verifiable and reproducible by a human.
Results have low variance when running code.
Ensures capsules are reproducible given our resource constraints.
Capsule is under 10 GB.
Capsule results can be reproduced when running the code locally.
Ensures capsules are reproducible.
and solving real-world programming problems (Zhang et al. 2024). With CORE-Bench, we aim to evaluate the ability of agents to automate the rescarch reproduction process, a part of the pipcline that hasn't 't yet received atteniion.
2.1 Benchmark Construction
We decompose the task verifying computational reproducibility into two sub-tasks: code reproducibility and result reproducibility. This paper and benchmark focus on code reproducibility, which means running the code and obtaining the results the capsule supposod produce, not result reproducibility, which checking whether the code results match what is reported in the paper. Code reproducibility is by far the more time consuming part for human. Some papers included in the benchmark (30/90) it is possible code reproducibility difficulty distributions are different for papers that are cesuttirreproduuible For the remainder of the paper, when we refer to "reproducible" we mean "code-reprodueible""
sesuttirreproducibbl and
Verifying code reproducibility requires significant domain expertise and can be intor-intensiee, even for experienced rescarchers. This makes particularly challenging to build a benchmark where the reproducibility of each paper is verified. It can take a few hours test the reprodueibility of a paper in the wild, so verifying about hundred papers from diverse fields would be impractical.
5

---

Published in Transactions on Machine Learning Research (01/2025)
Table 3: Ladder of difficulty. We created tasks at three distinct difficulty levels for each the 90 papers. This translates 270 tasks and 181 task questions across the three benchmark levels (the number of questions is less than the total number of tasks because all three difficulty levels consist of the same task questions). These levels are differentiated by the files of the repository that are given to hidden from the agent. CORE-Bench-Hard is the most realistic and akin to the setup an agent would have when reproducing paper in the real world. Each difficulty level tests the agent on an expanding set of skills as the difficulty increases.
Task level
Information provided to the agent
Agent task
Agent is provided the complete code output from a successful run of the code (instead of having to the code correctly itself).
Perform information extraction
CORE-Bench-Eagg
the code output to correctly answer the task questions.
Agent is provided the Dockerfile required to run the code, alongside text-based instructions for running it in README.
Run the Docker command and perform information extraction over the code output.
CORE-Bench-
Install all required libraries and dependencies, determine (and run) the correct command to reproduce the code from the task prompt, and perform information extraction over the code output.
Agent is provided only the READ- ME file with instructions and Dockerfile.
CORE-Bench-Hare
To address this, we drew papers and repositorios from CodeOcean capsules (Sec Figure 2), which are known to be code-reproduuible with little effort (Clyburne-Sheii et al. 2019). We selected a set of 90 reproducible papers from CodeOcean using the process outlined in Table 2 and Figure 3, We split the dataset into 45 papers for training and 45 for testing. For each paper, we manually created a set of task questions about the outputs generated from a successful reproduction of the paper (Appendix A.3 provides details on task question construction). These questions assess whether an agent. has correctly executed the code and retrieved the results. For instance, an agent could be asked report the test accuracy of model, an axis label of a figure, or another reproduced result. Some tasks have single task question, while others consist of multiple. We ensure each task has at least one question that cannot be solved by guessing (e.g. a question with an open-ended numerical answer), and a task is marked correct only if all of the task questions are answered correctly, which ensures all tasks cannot be solved by guessing.
We focus on evaluating agents can verify the code to be reproducible (not whether the results in a paper are consistent with the code), which is why all tasks CORE-Bench have been verified to be code reproducible. Since we measure an agent's performance by their ability to answer task questions about the expected output of code, do not include irreproducible capsules, because the task questions would be impossible to answer. However, it follows that high performing agents could be used to check if code is irreproducible, since the failure of a high-performing agent to reproduce code would indicate problem with the code, not the agent.
2.2
Why use CORE-Bench?
Skills and modalities. Solving the tasks in CORE-Bench requires many skills, including understanding instructions, debugging code, retrieval, and interpreting results from wide range of disciplines. The skills necessary to perform well on CORE-Bench reflect many skills necessary to reproduce new research.
Tasks require interpreting both text and image output from code. The vision- based questions (e.g. "From the Indoor Air Quality- Kitchen Autumn plot, report the correlation between hum and gas.") require extracting results from attributes of figures, graphs, plots, or PDF tables. The text-based questions (e.g. "Report the test accuracy of the neural network after epoch 10.") include extracting results from command line text, PDF
6

---

Published in Transactions on Machine Learning Research (01/2025)
Task: Capsule #3418007
Evaluation: Capsule #3418007
Task
Manual Run
Tosk Prompt
Provided Task Questions
FL sceze for statistionl genecal 9.956GaNHSEEEE
atatistical
Run 'main.py
only."
'Report the proposed model's AUC fron the ROC
("Roport the proseeed
sigues.")
cueves Zigaze." 0.00
Prediction Interval
Agent Run
Manual Run 2
[?Phppct statiatica?
*nosulte"
I ""ossllt"s [''hepoet statistioal groeral 8. 8.70081665 prspraed wodel's auc frum the RSC curves figuts. 0.9831
Set-up
Run Code
[2' RiPpppatt the
scoce foz
sereral only. 0. 0-76648122093322233
Core files
"'teld":
gereral only,
REPROOUCING.. mn/
festupe www.eciconttttyyo L105508/
"Language" sitle"s "capeule, 14'5 poonpt": "sosults"
["Sepoct the propoaed
recklls AUC Lcon Use curven siganw."
REAUME. end/
Euguzo. 0.98
eivironrent/ tockertila/
.py/ utile.ay/
[ "Booort the Flun: 0.78_ ). "Repoct proposed. in: "capoule,
tet//
[ "rosulte") [S"wouust the genazal onty. 6.109951666481801
code/
F- cun/
tite, fastures, txtt
Maticiing resufts
["Oeprct the posporedd
recel's trem the
figire.
(a) Example of task execution pipeline
(b) Example of evaluation criteria
Figure 4: During task execution, the agent must interpret the task prompt, set up the code in the capsule, run the code, and populate the specified result in the provided JSON file. For evaluation, we manually reproduced each capsule in the benchmark three times. We determine if an agent correctly solves a task if the agent's reported results for all questions fall within a 95% prediction interval for every task question of the results from the three manual runs. Prediction intervals provide a range in which we expect future observations to fall, accounting for stochasticity in the code outputs (Spence & Stanley 2016).
text, and tables or text in HTML. markdown, or Latex Capsules can have vision-based questions, text-based questions, or both (See Table A2 and capsules have codebases in either Python or R (See Table A1.
Real-world computational reproducibility tasks. When constructing our benchmark, we focus on its construct validity, which is about how well a test measures real-world performance (Biderman et al., 2024 Raji et al. 2021 Kapoor & Narayanan 2023). CORE-Bench tasks correspond closely to tasks that roscarchers must accomplish so that improved performance on the benchmark can dircctly lead to improvod computational reproducibility norms.
First step toward research agents. The first step toward completing new scientific research is the ability to reproduce existing scientific work. Building agents that excel at reproducibility is a necessary, and yct more attainable step toward building agents that can conduct novel research.
3 Baseline agents and evaluation setup
We evaluated all agents on CORE-Bench split by difficulty: CORE-Bench-Easy. CORE-Bench-Mediung and Hare-Bench-Hard.
Bascline agents. We developed and evaluatod two variants of the AutoGPT agent (Significant Gravitas 2024) on the benchmark: AutoGPT, which was not prompted or given any tools specific to CORE-Bench, and the CORE-Agent family of agents, which were prompted and modified for enhanced performance on each of the three difficulty levels of CORE-Bench.
1. AutoGPT:
This agent is largely unmodified from the popular general-purpose AutoGPT agent, but we created another tool for the agent called query_ vision srg_vviiionnnnuugeeeeeoe model, which takes input an image and a query, and outputs OpenAl APT's response to the image query. This allows the agent to analyze results in figures and F We included this modification in AutoGPT because the ability to query a vision language model is not specific to CORE-Bench. Other minor changes can be found in Appendix D.I
We plan make pull request include this feature the official PT repository.
7

---

Published in Transactions on Machine Learning Research (01/2025)
2. CORE COeEtt We built upon AutoGPT to create Agent, task-specific variant of AutoGPT, customized for each level of CORE-Benci Our primary change was implementing programmatic check to ensure the correct submission and keys of the file reporting the reproduced results (i.e., report.json) In addition, for each difficulty level, we added specific prompting hints to guide the agent's behavior, as detailed in Table 4 These hints address common pitfalls observed during qualitative analysis of agent performance on the training set. Notably, these adaptations required only a few days of work, with the most time-consuming aspect being the analysis of failure logs to identify effective prompting strategies.
Table 4:
Primary task-specific modifications to AutoGPT This table summarizes the modifications made to create CORE-Agent for each level of difficulty. The modifications listed for CORE-Bench-Easy. CORE-Bench-Medine and CORE-Bench-Hard are hints specific to each difficulty level we added to the default. prompt, while the programmatic check of the output report file applies to all levels. Additional modifications and prompts can be found in Appendix D.2
Task Level
AutoGPT Errors
CORE-Agent Modifications
All Task Levels
Not creating a report.json file not including the correct keys in the file
Programmatic check of report. json to en- sure agent submitted the report file with correct keys
CORE-Bench- Easy
Not consistently read- ing results from PDFs HTMT.
Use pdftotext for text extraction from PDFs
Use pdftoppm for extracting results from tables and figures
Extracting information from the incorrect filc without exploring all files
Check full results directory for image files before querying vision language model
Prioritize reading 'output' or 'manuscript' files
Convert HTML to PDF or PNG before in- formation extraction
Print the entire output directory tree and analyze five most relevant files before us- ing query visiovis_iin. Sarriionlannuage__ee () to extract information from images
CORE-Bench- Medium
execute execute_shell0 shell() tool did not support environ- mental variables
All modifications from CORE- MORRE-Bench-EBas
Use absolute paths instead of environmental variables in precute_shelll () command
CORE-Bench- Hard
Greedily installing de- pendencies in response code failures, with- out plan
All modifications from CORF-Bench-Fasy and MoRR-Be
Determine and install package dependencics before running code
Models. We ran both AutoGPT and Agent using -07-4--2024-05-3 and 1in-4o--in1-2224-00 as LLM backends since the AutoGPT developers recommend the GPT-4 family models. We included
Whee we refer CORE- Agent we refer the agent built that level the benchmark.
8

---

Published in Transactions on Machine Learning Research (01/2025)
Manager
VMs, upload capsule and agent code for each (task, agent) pair
Download and evaluate results from VM
2
Run agent on each
capsule 7036571
copsule- copsule-3137115 3137115
capsule-5367550 5367566
AutoGPT(gpt-oo
Workers
Agent (gpt-4o)
Figure 5: (1) The manager machine creates VM for each (agent, task) pair and uploads both the capsule and the agent code to the VM. (2) The manager machine invokes the agent on each of the VMs, so they all run in parallel. (3) The manager machine downloads the results from the agent off each VM once the agent indicates task completion, deletes the VM, and locally evaluates all of the results.
the smaller 177-4o-mini-2024-07-- to better understand the cost-accuracy trade-off. Duc to budget constraints, we had the agents terminate if they incurred API costs of over $4 per task (as Figure shows, this did not have major impact on accuracy).
Metrics. We report task accuracy the main metrie, which is the proportion of tasks for which all of the task questions have been answered correctly. We also report the average cost of the agent, which is the average API cost of all requests made by each agent.
Evaluation harness. We developed evaluation harness to each task of CORE-Bench virtual machine to ensure each lask is encapsulated and we could parralelize evaluating all tasks (See Figure 5) and Appendix B). Rumning each task on a VM, as opposed to a Docker container, allowed us to standardize hardware access for cach agent. The harness can run hundreds of benchmark tasks in parallel on virtual machine instances, enforcing a clear separation between the benchmark and the agent, and allowing for the easy development new agents (ATST 2024 METR 2024).
isolated
The harness is initialized on Manager machine, which has the code to run the benchmark and stores the CORE-Bench dataset. For each task in the benchmark, the Manager creates Worker instance, copies over the code for the agent and task capsule, and runs the agent on that instance. When the agent completes or fails a task, the Manager downloads the results from the Worker, deletes the instance, and evaluates the results locally. Agent evaluations are performed on the Manager machine. On CORE-Bench, which has 270 tasks and 181 task questions (and a per-task time limit of 2 hours in our evaluation). running each task soquentially could take over days. Using our evaluation harness took a little over two hours.
4 Results
Overall, CORE-Agent with GPT-4o is the top performing agent on all three levels of the benchmark, solving 58.52% of tasks on CORE-Bench-Easy. 55.56% on CORE-Bench-Medium but only 19.26% on CORE-Bench-Hard. We report all results in this section on the test split unless otherwise mentioned, since we used the train split. while developing the agent (see Figure Al for train set results).
Our results demonstrate that generalist agents can be effectively adapted to specific tasks with minimal effort, yielding significant performance improvements. For instance, AutoGPT with GPT-4o scored just 4.44%

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Accuracy (pass@1) of CORE-Agent and AutoGPT with gPt-40-2024-05-13 and gpt-4o-mini-2024-077
by task difficulty on the test set. We ran CORE-Agent three times on the benchmark to calculate confidence intervals (see Table A4), and therefore report average accuracy across the three runs. We only ran AutoGPT once due to cost constraints.
Agent Architecture
LLM
CORE-Bench-Easy CORE-Bench-Medium CORE-Bench-Hard
GPT-4o
58.52%
55.56%
19.26%
Agent
GPT-4o-mini
42.22%
30.37%
14.07%
GPT-4o
33.33%
35.56%
4.44%
AutoGPT
GPT-4o-mini
6.67%
0.00%
0.00%
on CORE-Bench-Hard. The following sections provide a detailed analysis of agent performance and highlight. the potential of adaptable generalist agents for specialized tasks.
4.1 Accuracy varies by difficulty level
Agents generally performed the highest on CORE-Bench-Ease followed by CORE-Bench-Mediun and CORE-Bench-Hand Hard. For instance, CORE-Agent with GPT-4o-min 4i-min scored 42.22%, 30.37%, and 14.07% on the three levels, respectively (See Table 5).
These results are expected, since CORE-Bench-Easy is designed to be the easiest. lask with the code outputs already provided in the environment. CORE-Bench-Mediur is slightly harder, requiring agents use a provided Docker command to replicate the paper's results. CORE-Bench-Hard significantly harder, requiring agents to install all dependencies and libraries and determine the correct command necessary to reproduce relevant results.
4.2
Task specific modifications improve accuracy, especially for weaker models
Comparing performance when fixing the LLM model, we observed that AutoGPT's performance improved substantially with only slight modifications. This adaptability seems to be particularly advantageous for weaker LLMs,, where small changes provide crucial guardrails and task guidance. With the GPT-4o back-end, few modifications to the prompt and the programmatic check of the output format boosted the performance on Bere-BBnch-Eaae -asy performance from 33.33% to 58.52%. The differences were even starker when using GPT-4o-mini: performance improved from 6.67% to 42.22%.
Our results highlight the adaptability generalist agents, demonstrating significant performance gains from minimal, task-specific adjustments. We hypothesize that agents that use stronger models in the future will require even fewer task-specific modifications perform well on a given task.
4.3 Stronger models lead to higher accuracy despite a lower token budget
We ran AutoGPT and CORE-Agent using both GPT-4o and GPT-4o-mini with an API cost limit of $4, Even though the per-token cost of GPT-4o-mini is less than 5% than that of GPT-4o, which allows for longer sessions before hitting the cost limit, GPT-4o still outperformed GPT-4o-mini on both agents. Despite having the same cost limits, GPT-4o-mini powered agents tended to be 3-5x cheaper than GPT-4o agents. In all settings, the average per-task cost was cheapest on BaaE-BBnch-Easy. followed by MoRd-Bench-Medium and CORE-Bench-Hard (Figure 6).
To evaluate the impact of our $4 cost limit on performance, we ran CORE-Agent on the CORE-Bench-Haze with $10 cost limit on the train set. With the new limit, GPT-4o-mini performance remained unchanged, and GPT-4o's performance increased modestly from 26% to 31% Figure 7) Note that GPT-4o-mini outperformed GPT-4o for lower cost limits under around $2.50.
Increasing the cost limit did not greatly increase accuracy because when agents succeeded at tasks, they succeeded quickly (the average cost of successful tasks for CORE-Agent and GPT-4o $0.54, compared to
10

---

Published in Transactions on Machine Learning Research (01/2025)
CORE-Bench-Easy Tasks
CORE-Bench-Medium Tasks
CORE-Bench-Hee Tasks
1.0
1.0
1.0
CORE-Agent
CORE-Agent
CORE-Agent
CoRE-Agent with GPT-4o-mini
CORE-Agent with GPT-4a-mini
CORE-Agenn with GPT-4o-mini
AutoGP1
AutoGPI GP1-40
AutaCPT with GPT-4o-mini
AutaGPT with GPT-4o-minl
AutoGPTT with GPT-4o-mini
Accuracy
0.6
Accuracy
Accuracy
0.6
0.4
0.4
0.4
0.2
1.5
Cost (USD)
Cost (USD)
Cost (USD)
Figure 6: Scatter plot of the cost vs accuracy of agents on the test set.
Success Rate by Cost Cutoff on CORE-Bench-Hard Tasks
1.00
Success Rate
CORE-Agent GPT-4o
0.75
CORE Agent with GPT 4o min
0.25
0.00
2
Cost Cutoff (USD)
Figure 7: Success rates of CORE-Agent with GPT-4o and GPT-4o-mini at varying cost limits on SERE-Bench-Hal tasks on the train set.
$2.59 for failed tasks) but when they failed at tasks, they often hit the cost limit and failed after not making progress. Increasing the cost limit does not resolve cases where agents are stuck.
4.4 Written questions easier than vision questions
Agents consistently performed better on text-based questions than vision-based questions. CORE-Agent with GPT-4o got 58.70% vision questions correct and 87.8 87.88% written questions correct CORE-Bench-Easy the test set. Similarly, CORE-Agent with GPT-4o-mini got 36.96% of vision questions correct and 81.81% of written questions correct. Vision questions are harder because they typically require analyzing results from figures, whereas written answers are often directly found in the terminal output. Agents were sometimes unable to find the relevant figure if multiple output files are generated. Even once found, analyzing the output can be difficult, as past work as also shown Xu et al. 2024 Majunnar al. 2024).
4.5 Python tasks much easier than R
Agents performed much better on Python tasks than R tasks Figure 8). One reason is that R outputs were often more difficult to parse, since many R capsules generate full PDF manuscripts which the agent has to read through. Another reason is that installing the requirements and dependencies for R packages can take much longer than for Python. Computer Science tasks are disproportionately in Python, which partially explains why they are casier than the other two disciplincs.
4.6 Agents struggle to retrieve results from many files and often time out while installing dependencies
We qualitatively analyzed some of the common failure cases of agents on each level of the benchmark. On CORE-Bench-Easy, agents excelled tasks where the code output was written in just one file directly
11

---

Published in Transactions on Machine Learning Research (01/2025)
Success Rate by Discipline
Success Rate by Programming Language
1.0
1.0
CORE-Agert with GPT-4
CORE-Agent with GPT-4o
CORE-Ag CORE-Agart Agert with GPT-40-mini
CORE-
CORE-Agent Agent with
Success Rate
0.6
0.6
Success
0.4
0.4
0.2
0.2
0.0
0.0
Computer Science
Medical Sciences
Social Sciences
Python
Discipline
Programming Language
Figure 8: Performance of CORE-Agent using GPT-4o vs GPT-4o-mini on the test set by discipline and programming language. Error bars are one standard deviation calculated from three trials.
outputted to the terminal. If the code output was written to multiple files, such as different figures, agents struggled to determinc which figure was relevant and had the correct information. Often, agents would use information from the incorrect figure to answer the question (Appendix D.3.1
On CORE-Bench-Medium, AutoGPT struggled to follow instructions to execute the Docker command to reproduce the code and would sometimes get thrown off by competing instructions. For instance, the agent might read the README file, and attempl to reproduce the code manually, without using Docker [ppendix D.3.2). CORE-Agent, however, tended to not struggle on this because of the task-specific instructions, and mistakes were usually caused by retrieval issues as described above.
On HaRd-Bench-Hard, in addition to the retrieval issues described above (which accounted for 23% of failures for CORE-Agent with GPT-4o on the test set), agents struggled with installing the dependencies for running code repositories (accounting for 57% of failures) and rumning the correct commands to reproduce the paper (accounting for 20% of failures). Agents often did not finish resolving dependency version issues before hitting the cost limit, getting stuck attempting to install the same library multiple times (Appendix D.3.3
4.7 Better guardrails are needed deploy safe agents
In one case, the agent attempted to search for the CodeOccan repository online look for the requirements for missing dependencies. Although the agent tried to create an account on CodeOcean, it could not view the CodeOcean website since it required JavaScript (Appendix D.3.4). This points to the need for mechanisms to restrict the actions taken by the agent. We have updated the release version of our evaluation harness to restrict access to the CodeOcean.com domain.
Since AutoGPT can execute arbitrary actions on the web, better guardrails should be developed to ensure agents exhibit safe and expected behavior (He et al., 2024). For instance, there are no existing safeguards preventing simple agent errors such as creating thousands of accounts on a website. For this paper, we did not incorporate web browsing restrictions for agents since their inability to render JavaScript prevented most damaging actions from being taken out. However, as agents advance, developers should implement additional safety checks.
5 Conclusion
Many visions for the future of LLMs and tool use anticipate grandiose reforms of the fields of research and science, including claims that AI agents will automate research completely (Lu al.) (224]. However, a pre-requisite for building existing knowledge is to reproduce research that has already been released.
12

---

Published in Transactions on Machine Learning Research (01/2025)
If an AI agent reproduce rescarch effectively, it can drastically reduce the human labor required to read, understand, and run code as part of an assessment of computational reproducibility. By releasing CORE-Bench, we hope to stimulate the development of agents to reduce the time and effort required for this burdensome yet routine scientific activity. At the same time, we recognize important future work remains in evaluating computational reproducibility agents, particularly when high-performing agents are used to verily the reproducibility of other papers.
Our bascline results show that while automating computational reproducibility is hard, simple task-specific modifications to existing general-purposs agents already help increase accuracy. This is in line with other results showing the importance of task-specific modifications (Yang et al. 2024). Yet, our best bascline agent only has a test-set accuracy of 19,, showing the vast room for improvement. We hope that CORE-Bench can spur research in improving the utility of agents automating computational reproducibility.
Acknowledgments
We thank Eugene Laska, Carole Siegel, Roma Bhattacharjec, and Veniamin Vesclovsky for discussions and inputs that informed our analysis. We acknowledge compute support from Princeton University's Center for Statistics and Machine Learning and OpenAT's researcher access program.
References
AISI. Inspect, 2024. URL https://inapect. http://iappptaaaaaaaaeeeeaaattts uk
Rose L. Andrew, Arianne Y.K. Albert, Sebastien Renaut, Diana J. Rennison, Dan G. Bock, and Tim Vines. Assessing the reproducibility of discriminant function analyses. PeenJ, 3:e1137, August 2015. ISSN 2167-8359. doi: 10.7717/peer..1137 URL https://www ncbi nlm nih. http://mc/aatticcce/////55500000
Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. A Systematic Review of Reproducibility Research in Natural Language Processing. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 381-393, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.1 18653/ v1/2021.each mac.-email.mn0 29, URL attps://aclanttholgg/ wttp seacl-maile 29
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haoman Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the Trenches Reproducible Evaluation of Language Models, May 2024. URL http:://arxvv.or//// [cs].
14782 arXiv:2405.14782
Matteo Brivio and Çağn Çöltekin. [Re] Exploring the Representation of Word Meanings in Context. ReScience C, 952):45, July 2023. doi: 10.5281 10.5281/zenodee /ttp://zoooo11111 1....8173300 8173688. URL htttt:://teeooeeee org/recordd//1133558
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, July 2024. arXiv:2407.2178 [cs].
URL http: /ttp:v/axxvv.... http://awxwv.oooabb////asa org/abs//4000 2207/221777
Jonathan B. Buckheit and David L. Donoho. WaveLab and Reproducible Research. In Anestis Antoniadis and Georges Oppenheim (ods.), Wavelets and Statistics, pp. 51-81. Springer, New York, NY, 1995. ISBN 978-1-
4612-2544-7, doi: 10.1007/978- 1--111--254444
5. URL http::///ood..rgg11..00////////
5
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costinee Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: Scalable and Extensible Approach to Benchmarking Neural Code Generation, 2022. URL https://arxi:: http/abbs/208.02222
Version Number: 4.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cumminng, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Πlya Sutskover, and Wojciech Zaremba. Evaluating Large Language Models Trained on Code, July 2021. URL arXiv:2107.0337. [cs].
03374
April Clyburne-Sherin Xu Fei, and Seth Ariel Green. Computational Reproducibility via Containers in Psychology. Meyaoosschoolou, 3, November 2019. ISSN 2003-2714. doi: 10.15626/MP:2018..0 URL
httpss://ppen. 1mu. www.:inn...eeeee...eeeeeee http:///pppppcoooooo/o///a//a//a
Christian Collberg and Todd A. Proebsting. Repeatability in computer systems research. Communications
of the ACM, 59(3):62-69, 69, February 2016. ISSN 0001-0782, 1557-7317. doi: 10.1145/2812803. URL
https://dll.ate att.://..ccoo.rg///o/o wtt.:aa...rr//o//...111111
Paul Gertler, Sebastian Galiani, and Mauricio Romero. How to make replication the norm. Nature, 554(7693); 417-419, February 2018. doi: 10.1038 01088//4155--01-022100
URL https://ww..natee http://wwwwaatuu..eem www.:nature.coo/rttaaao com/articles/ 841586-018-02108-99 Bandiera abtest: Cg type: Comment Publisher: Nature Publishing Group Subject term: Research data, Research management, Publishing.
Kimberly Gilbert, Rose L. Andrew, Dan G. Bock, Michelle T. Franklin, Nolan C. Kane, Jean-Sébastien Moore, Brook '. Moyers, Sébastien Renaut, Diana J. Rennison, Thor Veen, and Timothy H. Vines. Recommendations for utilizing and reporting population genetic analyses: the reproducibility of ge- netic clustering using the program structure. Molecular Ecology, 21(20):4925 2012. ISSN 1365- 294X. doi: 10.1111/i.136--22
URL http:://ooolllll...aaay..aa.eeop
10. 1111/j 1365- 294X. 2012 05754.57.4.. x 294X. 2012.0002..
_eprint: http:://hllllbbboessocccc//////p
Tom E. Hardwicke, Maya B. Mathur, Kyle MacDonald, Gustav Nilsonne, George C. Banks, Mallory C. Kidwell, Alicia Hofelich Mohr, Elizabeth Clayton, Erica J. Yoon, Michael Henry Tessler, Richie L. Lenne, Sara Altman, Bria Long, and Michael C. Frank. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Sociely Open Science, 588):180448, August 2018. doi: 10.1098/rsos..1 180448. URL httpess/rroyalooceetypbbbb.bbbb org/doi/10. 1098////0..0////0 180448
Publisher: Royal Society.
Tom F. Hardwicke, Manuel Bohn, Kyle MacDonald, Emily Hembacher, Michèle B. Nuijten, Benjamin N. Peloquin, Benjamin E. deMayo, Bria Long, Erica J. Yoon, and Michael C. Frank. Analytic reproducibility in articles receiving open data badges at the journal Psychological Science : an observational study. Royal Society Open Science, 211)2201494, Jamuary 2021. ISSN 2054-5703. doi: 10.1098/rsos.22 201494.. URL
Michael Hassid, Tal Remez, Jonas Gehring, Ray Schwartz, and Yossi Adi The Larger the Better? Improved LLM Code-Genen Sode-Generation via Budget Reallocation, July 2024. URL http://arxiv.orr org. haabb//244440070 00725
arXiv:2404.00725 [cs].
Yifeng He, Ethan Wang, Yuyang Rong, Zifei Cheng, and Hao Chen. Security of AI Agents, 2024. URL https://arxiv http:://axxxvv.oo/bbbbsssee werg/abs/24ee
www.aa//.....e////////e/eeeeee.. /406. 086899 Version Number: 2.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovee. Benchmarking Large Language Models As AI Research Agents, October 2023. URL http://aaaa....r//a//33/...000 23300b
14

---

Published in Transactions on Machine Learning Research (01/2025)
John P. A. Ioannidis, David B. Allison, Catherine A. Ball, Issa Coulibaly, Xiangqin Cui, Aedin C. Culhane, Mario Falchi, Cesare Furlanello, Laurence Game, Giuseppe Jurman, Jon Mangion, Tapan Mehta, Michael Nitzberg, Grier P. Page, Enrico Petretto, and Vera van Noort. Repeatability of published microarray gene expression analyses. Nature Genetics, (1(2))149-155 155, February 2009. ISSN 1546-1718. doi: 10.1038/ng.295, URL https:: pettps://wwtr htttp:://www.naurree www.:w/ature.comm htt.//urrecccoaarrclllsee//////e 295 Publisher: Nature Publishing Group.
Carlos E. Jimenez, John Yang, Alexander Weltig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, October 2023. URL https
Sayash Kapoor and Arvind Narayanan. Evaluating LLMs is minefield, October 2023.
URL
https
Sayash Kapoor, Benedikt Strocbl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That arXiv:2407.01500 [cs].
Matter, July 2024. URL
Markus Konkol, Christian Kray, and Max Pfeiffer. Computational reproducibility in geoscientific papers: Insights from a series studies with geoscientists and a reproduction study. International Journal of Geographical Information Science, 33(2):408 February 2019. ISSN 1365-8816. doi: 10. 10.1080/136555 13658116.
2018.1508681 URL / 1ttp://10...1000116656180000 2018. 1508687 eprint:
2018.1108687
Publisher: Taylor Francis
Jimmy Koppel. Everyone's ' talking about Sakana's AI scientist. But no-one's answering the big question: is its output good?, August 2024. URL httpa////..oooo////ooo//////ttt2
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' l'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexcy Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-Levw Code Generation with AlphaCode. Science, 378 (6624):1092 1097, December 2022. ISSN 0036-8075, 1095-9203. doi: 10.1126 aci.. URL http://arxiv http:///axxiv.oog/abbee /2t///ass///...e 22102022330000011111111
arXiv:2203.0811 [cs].
David M. Liu and Matthew J. Salganik. Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge. Socius, 022378023119849888 January 2019. ISSN 2378-0231. doi: 10. 1177/23780231198840000 URL http://pps//ooooo///////1//7//11 Publications.
Publisher: SAGE
Victor Livernoche and Vidya Sujaya. [Re] A Reproduction of Automatic Multi-Label Prompting: Simple and Interpretable SSot Classification. ReScience C, 92):#333, July 2023. doi: (0.5281/zenodo.53
URL http:///woooooor///rrrr///d/////
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The Al Scientist: Towards Fully Automated Open-Ended Scientific Discovery, August 2024. URL http:://axxxt org/abs/ 2408.06292. arXiv:2408.0622 06290 [cs].
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sncha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent. Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Ba- tra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran. OpenEQA: Embodied Question Answering in the Era of Foundation Models. pp. 16488-164998 2024. CRL ltttps:// thwewf.hheee..c wtw.w/f.co//oooteert/oooooo http:///wwaaaee///////////////// http:///aaaaaaaaaaauaaaaaaaaaaa www__ppeeen_aaeeee Emboddid__m 2a1ii22224__aaa__ 201p2/4_ppp_____ html
Question_Answerrt Answering ii_nnnin__________ the Era_of wow.ooatiinooooelleaaa....
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discovery Towards Data-Driven Discovery with Large Language Models, July 2024. (RL/tttp://are .rg wwtpw/////w/////...o////// 01725 arXiv:2407.01725 [cs].
15

---

Published in Transactions on Machine Learning Research (01/2025)
B. D. McCullough, Kerry Anne McGeary, and Teresa D. Harrison. Lessons from the JMCB Archive. Journal of Money, Credit and Banking, (5(4))1093-1100) 2006. ISSN 0022-2879. URL https://www. jstor org stable/3838995 Publisher: [Wiley, Ohio State University Press].
National Academies of Sciences Engineering and Medicine. Reproducibility and Replioability in Science. 2019. doi: 10.17226/25003, URL httpsss//aap..ttiollaaeeeesssss hwtp:///e....oeeee///////// chapter/7
METR. Vivaria, 2024. URL https://vivaria.mmttt
Florian Naudet, Charlotte Sakarovitch, Perrine Janiaud, Toana Cristea, Daniele Fanelli, David Moher, and John P. A. loannidis. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. BMJ, 360:k400, February 2018. ISSN 0959-8138, 1756-1833. doi: 10.1136/
URL https Publisher: British Medical Journal Publishing Group Section:
//www.bmj com/conteett//36/// k400.
Research.
Pepiju Obels, Daniël Lakens, Nicholas A. Coles, Jaroslav Goltfried, and Seth A. Green. Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology. Advances in Methods and Practices Psychological Science 3(2):229-23) June 2020. ISSN 2515-2459. doi: 10.1177 1.1177/2515248920188887777 URL
10. /ttp//....177//1155520111111
Publisher: SAGE Publications Inc.
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygekzimer, Florence d' Alche Buc, Emily Fox, and Hugo Larochelle. Improving Reproducibility in Machine Learning Research(A (A Report from the NeurlPS 2019 Reproducibility Program). Journal of Machine Learning Research, 22(164):
1-20, 2021.
ISSN 1533-7928. URL http: //jml: wwwporrggggg..oeee pttpppppps/////2///0/ 303. html
Ori Press, Andreas Hochlehmert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. CiteME: Can Language Models Accurately Cite Scientific Claims?, July 2024. URL http:///trret http:://axxvv.org
arXiv:2407.1286) [cs].
Christophe Pérignon, Olivier Akmansoy, Christophe Hurlin, Auna Dreber, Felix Holameister, Juergen
Huber, Magnus Johannesson, Michael Kirchler, Albert J. Menkveld, Michael Razen, and Utz Weitzel. Computational Reproducibility in Finance: Evidence from 1,000 Tests, February 2024. URL https
//papers. ssrn. //paperss.ssrnccom//eeee coth.cccoo/bbbtraat44444722
Edward Raff. A Step Toward Quantilying Indepenennty Reproducible Machine Learning Research, September 2019. URL
06674 arXiv:1909.06674 [cs, stat].
Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Al and the Everything in the Whole Wide World Benchmark, 2021. URL htttss://axxi..orr http:/awwwi.o..or//a/// Version Number: 1.
Sheeba Samuel and Daniel Mietchen. Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience, 13:giad113, January 2024. ISSN 2047-217X. doi: 10.1093/gigasccies 1ia
URL https:://oiiiii http:://///oo..o01110//110// http:///....////gaaaaeeee/e/////
giadd11
Significant Gravitas. AutoGPT, September 2024. URL http:///ggthhabooooo/o////aattta
AutoGPT. original-date: 2023-03-16T00:2110077
Koustuv Sinha, Maurits Blecker, Samarth Bhargav, Jessica Zosa Forde, Sharath Chandra Raparthy, Jesse Dodge, Joelle Pineau, and Robert Stojnic. ML Reproducibility Challenge 2022. ReScience C, 9 July 2023. doi: 10.5281 (22211//enooo.200000
URL
Jeffrey R. Spence and David J. Stanley. Prediction Interval: What to Expect When You're Expecting A Replication. PLOS ONE, 11 (19))c01122874 September 2016. ISSN 1932-6203. doi: 10.1371 10.1371/journall /journal.ponc. 0162874. URL http:///.naaaaal..//.///////aaaa pore:a1111111oaaaaaaaaaoo1...110
Publisher: Public Library of Science.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Daniel Stockemer, Sebastian Kochler, and Tobias Lentz. Data Access, Transparency, and Replica-
tion: New Insights from the Political Behavior Literature.
PS: Political Science & Politics, 51
(4):799 803, October 2018. ISSN 1049-0965, 1537-5935.
doi: 10. 10.001//S100900611800992 URL
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, and Hao Peng. SciCode: Research Coding Benchmark Curated by Scientists, July 2024. URL http:///taae org/ ht////b//2440 /407 13168 arXiv:2407.3316$ [cs].
Ana Trisovic, Matthew K. Lau, Thomas Pasquier, and Mercè Crosas. large-scale study on rescarch code quality and execution. Scientifie Dala, 9(1):00, February 2022. ISSN 2052-4463. doi: 10.1038/
941597-022-01143- URthttps::// //www wttp:/wwww.....tt///t////t//twwt nature. http:///........e.eee////////--. Publisher: Nature Publishing Group.
Benjamin D. K. Wood, Rui Müller, and Annette N. Brown. Push button replication: Is impact evaluation evidence for international development verifiable? PLOS ONE, 18(12): December 2018. ISSN 1932-6203. doi: 10.1371 /tt////uuaalpppa 0209416. URL httpss//journall: peapinalsllll......e..eeommmmlle (d=10.1371/jour
pone. 0209416 Publisher: Public Library of Science.
Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. ChartBench: A Bench- mark for Complex Visual Reasoning in Charts, June 2024. URL http::/arxxv.org/ htt////awww..rrb////////./../. wttpaab..//3/a311 15915 arXiv:2312.15915 [cs].
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasinhan, and Ofir Press. SWE-AGENT: Agent-Computer Interfaces Enable Automated Software Engineering. 2024. URL
https://swe-agen wttp:/aaaentccccppreeeereee cow/paper...... pdf
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. SttauS-bench: A Benchmark for Tool- Agent-User Interaction in Real-World Domains, June 2024. URL http:://arxiw.org//aae hbtp://bs//244..../ 12045 arXiv:2406.12045 [cs].
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models Compound AI Systems, February 2024. URL httt::ttt::/att berkeel httpr//wwww......oooo///////////
cttp:////oooooouuaaaa-aaaaeeeeee ssssottsss
Yaolun Zhang, Yinxu Pan, Yudong Wang, and Jie Cai. PyBench: Evaluating LLM Agent on varions real-world coding tasks, August 2024. URL http://arrxw.orr/abbs////a 16732 arXiv:2407.16732 [cs].
A Benchmark Details
A.1 Original CodeOcean Dataset
To obtain a dataset of 5,090 capsules on CodeOcean and their corresponding environment files, we wrote webscraper that downloads the metadata of every capsule from CodeOcean. We then manually exported each capsule from CodeOcean's web interface obtain the environment files. Finally, we filtered the capsules in this dataset based on the ten criteria. outlined in Table 2
A.2 Examples of capsule selection criteria
Table presents the ten criteria we used to filter the capsules on CodeOcean and construct the tasks for CORE-Bench. We provide example of capsule's file that satisfies criteria six [Listing 1) and an example
17

---

Published in Transactions on Machine Learning Research (01/2025)
#t/usr/bin//n bash set -ex
This is the master script for the capsule. When you click "Reproducible Run", the code
in this file will execute. python -u
py *$e"
Listing 1: Capsule 5507257 run file. Example of simple CodeOcean capsule run file, where code is executed with a single bash command. This run file satisfies criteria six, which requires capsules to have a relatively simple Bash command to reproduce the code correctly.
Table Al: A breakdown of the number of capsules from each discipline by language.
Python
R
Total
Medical Sciences
10
15
25
Social Sciences
24
28
Computer Science
35
2
37
Total
49
41
90
of the output from capsule rejected from the benchmark on the basis of it not being computationally reproducible Listing 2 and Listing 3).
python -u model. py input shape: torch... Size ([1, 1, 234, 256))
python -u model. py
input shape: torch. (I1, 1, 234, 256])
spike probability: 0.4230301678186694 segmentation output shape torch Size ([1, 1, 234, 256])
spike probability: 0. 0.7832288413344207
segmentation output shape torch. Size ([1, 1, 234, 256])
python -u augmentation.pp
python -u augmentation.py
Listing 2: Capsule 826891 code output from our first run on CodeOcean's 's web interface. The spike probability from this run is 0.42303016881806996
Listing 3: Capsule 826891 code output from our
second run on CodeOcean's web interface. The spike probability from this run 0.78322285413744277
A.3 Task question construction
To write task questions for each capsule, we examined the capsule's results folder after a successful reproducible run on CodeOcean's 's web interface and chose outputs from any of the files the results for the agent extract. These outputs could include model's accuracy, the axis label of figure, or any other relevant metric. Then, for each output, manually write prompt instructing the agent report the corresponding value. Since single paper can have multiple outputs, CORE-Bench consists of 90 capsules and 181 task questions. The number of task questions per capsule ranges from one to eight.
We referred to tables or figures in task questions one of three ways:
The metric the table or figure is measuring. For example, From the figure measuring average RTT without ISL, report the x-azis label.
2. The title of the table or figure. For example, From the Indoor Air Quality Kitchen Autumn plot, report the correlation between ham and gos, where Indoor Air Quality Kitchen Autumn is the title of a figure depicting correlation coefficients.
The table figure number from the file name, PDF files, or HTML files in the results folder. For example, From Figure panel A, report the label of the green line.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Table A2:
Number of capsules from each discipline with only vision task questions, only language task questions, or at least one vision task question and at least one language task question.
Only vision task questions Only language task questions
Both
Total
Medical Sciences
5
25
Social Sciences
28
Computer Science Total
24
37
35
11
90
A.4 Breakdown of task questions by discipline, modality, and language
When choosing capsules to include in CORE-Bench, we attempted have similar number of capsules from each disciplinc. We provide breakdown of capsules from each discipline in the train and test sets in TTblle A3 CodeOcean contains 1,259 computer science capsules written in Python or R, 270 social science capsules written in Python or R, and 128 medical sciences capsules written in Python or R. Due to the limited availability of social science and medical sciences capsules that fulfilled all of our criteria (See Table 2), our final benchmark contains more computer science capsules than capsules of other disciplines. CORE-Bench also consists of a similar number Python and capsules (See Table A1 and a similar number of vision-based and language-basdd task questions (See Table A2
Examples of lext-based task questions include:
"Report the accurocy of the multitask learning model at the end of training on the test set."
"Report the AUC the 'sample-level!"
"Report the fl score for the MuskI+ dataset with the kan classifier."
"For the wilhin-variance improvements, report the improvement for the FS TotalGray Vol oulcome with the Day variable." "
"Report the CN prediction accuracy for the Zoo dataset."
"Report the closeness coefficient for location L1."
Examples of vision-based task questions include:
"Report Institutions Sampled for US Table 1."
From the Experimental TAQ Data graph, report the y-aris label." "
"For dataset report the score (%) for the GRU classifier for ACC."
"From the final result plot, report the label for the orange line." "
"Report the name of the model with the highest average energy"
"From the figure depicting calculated transmission coefficient for the 100-nF VISHAY capacitor, report the label of the red line."
19

---

Published in Transactions on Machine Learning Research (01/2025)
Table A3: Number of capsules from each discipline in train and test sets.
Train set
Test set
Total
Medical Sciences
12
13
25
Social Sciences
14
14
28
Computer Science
19
18
37
Total
45
45
90
GPT-4o Success Rate by lask Level
GPI-4o-mini Success Rate lask Level
1.0
1.0
AutoGPT
AutoPPT
Lore:Ageet
CUNb-Agurt
Success Rate
Success Rate
0.6
0.6
0.4
0.4
0.0
0.0
CORE Bench-Enngp
CORE Bench Medium Tesk Level
CORE- Bench-Hard
CORE BeRE--Bench-aayy
CORE CORE-Bench- Sett-Berch-eeiiie Medechum-dimm Task Level
CORE Gench-Haard
Figure A1: Accuracy of AutoGPT and CORE-Agent with GPT-4o and GPT-4o-mini on the train set. Task- specific agents consistently outperformed generalist agents, which were designed to correct for commonly made mistakes.
B Harness Details
Our evaluation harness runs all agents on virtual machines using Azure. For non-GPU capsules, we use a Standard, E2as_ v5 machine type, and for GPU capsules, we use Standard_NCee _a v3 machine type. All VMs run Ubuntu Linux and have an 80 GB disk attachod.
The harness initially creates VM for each task-agent pair and copies the capsule files and agent files to the VM. Once the files are copied over, the harness runs the agent the VM. The capsule only downloads the results and deletes the VM once the agent creates file called task completed. log in the home directory. This log file can be emply or can contain any logging information that the developer wishes to save from the run.
On occasion, the harness may fail to download the results of an agent from a VM due to an Azure error (for example, timing out when attempting to create virtual machine). In this case, you should re-run the experiment with the -resume flag, which will only start VMs for unfinished tasks.
When rumning the benchmark on multiple capsules, please be aware that will incur billing charges for all instances. you need to manually delete a VM capsule (if the harness code gets interrupted), you must delete all associated resources with the VM, (i.c. the network interface, the public IP, the disk, and the virtual network) associated with the instance. It not sufficient to only delete the instance itself.
C Experimental Details
C.1 Agent Accuracy on the Train Set
We plot the accuracy of CORE-Agent and AutoGPT on the train set (See Fig A1). Similarly the test set results, that CORE-Agent consistently outperforms AutoGPT, and GPT-4o outperforms GPT-4o-mini.
20

---

Published in Transactions on Machine Learning Research (01/2025)
Table A4: Accuracy and costs after running AOeE-Agent on the benchmark three times, with 95% confidence intervals. Results are presented by task difficulty on the set with trials on the benchmark.
Task Type
Agent Architecture
LLM Model
Metric
CORE-Bench- Eascy CORE-Bench-Mediun
CORE-Bench-Haad Hard
Accuracy (%)
52%
6.00%
55.56% + 4.51%
19.26% +: 2.60% $2.9643 $0.0888
GPT-4o
Cost (8)
80.6407 + $0.1886
$1.2005 32.3
CORE-Agent
Accuracy (%)
42.22% +: 13.52%
30.37% +: 11.34%
14.07% 1: 2.60% $0.7315 $0.1871
GPT-io-mini
Cost (8) 80.0445 + $0.1083
90.3893 + $0.3891
Table A5: Accuracy (correctly answering all task questions) and full attempt (providing an answer to all task questions) rates for CORE-Agent and AutoGPT on the test set.
Task Type
Agent Architecture
LLM Model
Metric
CORE-Bench-Han Hard
Accuracy (%)
58.52%
55.56%
19.20%
GPT-4o
Full Attempt (%)
97.78%
85.93%
43.70%
CORE-Agent
Accuracy (%)
42.22%
30.37%
14.07%
-Pi-io-mini
Full Attempt (%)
91.85%
82.96%
53.33%
Accuracy (%)
33.33%
35.56%
4.44%
GPT-4o
Full Attempt (%)
08.89%
60.00%
26.67%
AutoOPT
Accuracy (%)
6.67%
0.00%
0.00%
GPT-4o-mini
Full Attempt (%)
26.67%
6.67%
4.44%
C.2 Confidence Intervals Test Set
We ran Agent experiments with GPT-4o and GPT-4o-mini three times to generate 95% confidence interval over the mean accuracy and mean cost (See Table A4p. The accuracy of the top-performing agent had a CI of under 5 percentage points on all difficulty levels. Overall, the accuracy of GPT-4o-mini had a larger CI on results than GPT-4o, suggesting is a less reliable model to use.
C.3 Agent Failures vs Incorrect Responses
There are two ways an agent could fail a lask: by answering lask question incorrectly, or by not answering a task question at all due to getting stuck at some earlier stage. We compare the accuracy rate (answering all task questions correctly) to the full attempt rate (giving an answer to all task questions) in Table A5 Large gaps between the accuracy rate and full attempt rate indicate that the agent could be made more reliable and techniques such resampling responses could improve performance.
C.4 Task Completion Time on Test Set
We report the average amount of time (in seconds) each agent takes to complete tasks all three levels (See Table A6p. CORE-Bench-Eas is by far the quickest level to complete since the code outputs are already given lo the agent, so it does not need to run code. CORE-Bench-Medinn tasks take longer since the agent needs to run the provided Docker command and wait for it to finish. TORE-Bench-Hasd tasks take by far the longest, since the agent needs to install repositories and potentially debug running the code.
C.5 Pass@k on the Test Set
On the test set, the pass@1 accuracy of CORE-Agent with GPT-4o on CORE-Bench-Hand was 20.00% and the pass@3 accuracy was 28.88% (See Figure A2). Similarly, with GPT-4o-mini, the pass@1 accuracy was 13.33% and the pass@3 accuracy was 24.44%. Since the performance could be improved simply by re-running the model, strategies like running the agents multiple times and choosing the best outputs could be promising. Past work has shown retrying or increasing temperature between retrics can be enough drastically improve
performance (Kapoor al. 2024 Hassid et. al. 2024 Brown et al. 2024
Li al. 2022
21

---

Published in Transactions on Machine Learning Research (01/2025)
Table A6: Task completion time (seconds) of CORE-Agent and AutoGPT with gpt-40-2024-05-13 and by task difficulty on the test set. We ran CORE Agent three times on the benchmark calculate confidence intervals, and therefore report average time across the three runs. We only ran AutoGPT once due cost constraints.
Agent Architecture
LLM
CORE-Bench-Base
CORE-Bench-Medium CORE-Bench-Hard
GPT-4o
94.84
571.01
1133.92
Agent
GPT-4o-mini
90.62
1153.55
2329.49
GPT-4o
78.31
595.07
1192.6%
AutoGPT
GPT-4o-mini
33.38
622.94
1256.88
GPT-4o Pass@k Accuracy by Task Level
GPT-4o-mini Pass@k Accuracy by Level
CORE Bench
0.8
Handh-
0.8
Success Rate
Succeds Rate
0.6
0.8
0.7
0.2
0.0
Pass@k
Pass@k
Figure A2: passuk rate for CORE-Agent with GPT-4o and GPT-4o-mini on the test set. Increased performance gains suggests future work could focus on improving reliability.
C.6 P k on the Test Set
To measure the reliability of agents, we report the pass"k metric, which is defined as the percentage of tasks for which all & task trials are successful (Yao et al. 2024]. The pass"1 accuracy of CORE-Agent with GPT-4o on SORE--Bech--aar CORE-Agent with GPT-4o-mini on CORE-Bench-Hard was 13.33% and the pass"3 accuracy was 4.44% (See Figure A3). The results suggest that the underlying stochasticity of the agent caused it not consistently solve the same tasks. Incrcasing the reliability of agents such that they consistently solve problems they are capable of solving is challenging problem.
was 20.00% and the pass accuracy was 6.66%. Similarly, the pass accuracy of
D Agent Details
D.1 AutoGPT Bug Fixes and Changes
In addition to the modifications to AutoGPT described in the main text, we implemented two other changes for both AutoGPT and CORE-Agent. We implemented these changes for both agents and did not consider them task-specific modifications since the changes are not specific to CORE-Bench and would improve the agent in many domains.
1. Truncating tool output: If a tool invoked by AutoGPT generates output that is too long, we updated the code to truncate the output to include the beginning and end, rather than return an error. We found this change helps the agent better use tools when the outputs are long.
Using the shell execute all Bash commands: AutoGPT uses the subprocess module execute commands on the command line. However, the default setting was to set shell=False when invoking subprocess. run, which prevented the agent from using shell-speeific commands such &
22

---

Published in Transactions on Machine Learning Research (01/2025)
GPT-4o Pass Accuracy Task Level
GPT-4o-mini Pass Accuracy Task Level
1.0
CORE: Banch Basy
CORE Bee-----crrrrrr
CORE Bench
0.8
Success Rate
Success Rate
0.6
0.8
0.4
0.2
0.0
0.0
Pass"k
Pass
Figure A3: pass rate for Agent with GPT-4o and GPT-4o-mini the test set. Note that the pass line is identical on CoRe-Bench-Easy and CORE-Bench-Medium for GPT-4o.
when chaining logether two commands. We changed the settings to set shell=True to let the agent. execute all commands.
D.2 CORE-Agent Prompts
We provided tailored prompts to CgeE-Agent for each difficulty level of the benchmark. These prompts were given arguments to the AutoGPT agent, called 'constraints' or "best-practices.
CoRe-Bench-Bas Prompt
-rai-role seasoned digital assistant: capable, intelligent, considerate, and assertive. my dedicated research assistant, you possess extensive skills to complex
research development problems. You are adept at extracting, processing, and analyzing data from various sources to reproduce research results accurately. Using pragaatic approach, you nake the most out of the tools available to you."
do not shy away from writing
best -practice "If you are extracting information from PDF, the preferred utility to use is pdftotext (when you need text only information) or pdftoppm (when you need reaults from tables, figuclearres, eto.)."
beat -practice "When reproducing figures or other results that require you to deal with images, be reminded to check the full results directory for image files before querying the vision language nodel." -prctiice "If you
unsure of what to do, make your best guess."
-- best -prcciic "Before using resources like scripts utilities, verify their presence and functionality in the -praccice "If there exists file called 'output' 'nanuscript' then first read those files task."
directory installation path." "
required results
the questions the
best -praticce you are extracting information Jupyter notebook), convert it to information."
(such as output extract
first
constraint "Use flags or modify commands to bypass any need for confirmations during execution to streamline processes."
constraint "There is no task that you cannot do, so you should not refuse a request by saying you cannot do \ --constraint "You may wish to write and run a Python script to complete the task. especially if the task requires accees to the Internet or other libraries. Hovever assume that I do NOT have API keys to use external services." --costrrint "If you have task that requires you to the wwwyriiisio__annuaae.e _mdeel command to extract information from image files, first output the full tree of files in the directory containing the results and pick the 5 nost relevant files per question given the information Then investigate all the identified files first before choosing which the information you need --oostrriitt "Before you done, make that the keys of the report. json you write they not."
vant to extract.
contains
the question."
the specified by the user. Refine your results
23

---

Published in Transactions on Machine Learning Research (01/2025)
constraint "Also before you done, you do any unnecessary additional but only the aumeric value precise you asked report. The keys the specified by indicate you should report. Refine your results they not."
that the values report jaon
CORE-Bench-Medium Prompt
--ai -role "a seasoned digital assistant: capable, intelligent, considerate, and assertive. AS dedicated research assistant, you possess extensive skills in research and development and do not shy away from writing code to solve complex problems. You adept at extracting, processing, and analyzing data from various reproduce research results accurately. Using pragmatic approach, you nake the most of the tools available best ppactice you extracting information only information) páftoppm (when you tables, figuclearres, etc.)."
you."
PDF, preferred utility
pdftotext (when you need need results
-rraccicctc "When reproducing figures or with images, be reminded to check the full results directory for image files before querying the vision language model."
results that require you to
bebest -practice "If you are unsure of what to do, make your best guees."
-- bestt -pacctccc "Before using resources like scripts or utilities, verify their presence and functionality in the current directory or installation path." -- best -practice "If there exists a file called 'ranuscript' then first read this file to extract the required reaults to answer the questions of the task." A best "If you extracting information from html (such as the output of Jupyter notebook), convert it to a PDF PNG first and then extract the relevant information."
constraint "Use flags modify commands to bypass any need for confirmations during execution streamline processes. constraint "There is by saying you
that you cannot do it" coonstraint "You may wish requires
you should refuse request
Python script complete task, Internet or other libraries. have keys use external services."
especially assume that --onstrrintt "If you have a task that requires you to use the
query_vision law___langgggeeeee model command to extract information from image files, first output the full tree of files in the directory containing the results and pick the most relevant files per question given the information you vant to extract. Then investigate all the identified files first before choosing which one contains the information you need to anawer the question." constraint "Do include environmental variables such as "PND" as an argument for the serecute_shell" command. Instead, determine the value of the variable and directly input it to the comnand. For exanple, by using the absolute path instead of "PWD"."
constraint "Before you done, make that the keys of the report json you write match the they do not." --costrraint "Also before you done, you do value precise you asked report. The keys the specified by indicate you should report. Refine your results they not.
in the task specified by the user. Refine your results if
that the values the report json any unneceseary additional
but only the aumeric
CORE-Bench-HHe Hard Prompt
--ai -role "a seasoned digital assistant: capable, intelligent, considerate, and assertive. As dedicated research assistant, you possess extensive skills in research and development and do not shy away from vriting code to solve complex problems. You adept at extracting, processing, and analyzing data from various reproduce research results accurately. Using pragmatic approach, you nake the most of the tools available you." -raa-cicc "If you extracting information from PDF, preferred utility pdftotext (when you need only information) or pdftoppm (when you results tables, figuclearres, etc.)."
24

---

Published in Transactions on Machine Learning Research (01/2025)
best -practcc "When reproducing figures other resulta that require you deal with images, be reminded check full results directory for image files before querying best pprcctie --at---raciiic "Before using resources like scripts or utilities, verify their presence and functionality in the current directory or installation path." --best "If there exists file called 'nanuscript' then first read this file to extract the required results to ansver the questions of the task." -bbst- "If you are extracting information from html (such as the output of Jupyter notebook), convert it to a PDF or PNG first and then extract the relevant information."
vision language model." you
what do, your guess.".
-- -ebstt -praciie "Before running the code, first determine list of package/dependengg requirements that must be installed by reading through the READNE file or code itself. Then install those dependencies before running the code." " constraint *Use flage modify commands to bypass any need for confirmations during execution streamline processes."
constraint "There is task that you cannot by saying you cannot do it" --costrraint "You may wish write and task requires
you should not refuse request
Python script complete task, Internet or other libraries. However keys use external services."
especially assume that do --oossrraitt "If you have a task that requires you to use the query_vision wawoonnnnnnageemaa model command to extract information from image files, first output the full tree of files in the directory containing the results and pick the nost relevant files per question given the information you want to extract. Then investigate all the identified files first before choosing which one contains the information you need to anawer the question." connsrriit "Do include environmental variables such as 'PHD" as an argument for the command. Instead, determine the value of the variable and directly input it to the command. For exanple, by using the absolute path instead of "PuD'." "
have
'execute_shen
constraiit "To open folder or navigate to different working directory, the open_fooldee folder command rather than 'ed' in execute execute_shell." execute_shell() rather than to the code, since pecute__pthoon_file will have any of the libraries you attempt install. other vords, NEVER execute_python_ffle
constraint "When running Python cade, you should
execute_python__5
constraint "Before you done, write natch the ones in the task specified by the user. Refine your results they do not." --oostrrintt "Also before you are done, make sure that the values the report json you vrite do not contain any unneceseary additional text but only the numeric value or the precise text you are asked to report. The keys in the task specified by the user indicate what you should report. Refine your results if they do not."
the keys of the apportjjoo you
D.3 Agent Trajectory Examples
In this section, we provide some examples of common failure cases for each level of the benchmark.
D.3.1 Using incorrect figure during information extraction (CORE-Bench-Er)
On capsule-4299879, CORE-Agent with GPT-4o extracted the p-value from the wrong image in the results. The plot with the correct title containing the relevant v-value called Figure_Afe pdf, but the agent only looked at Figure__0 2-1 pdf and eigure33-11 -df...f..... Since the agent called query_vision language model the wrong figures, it returned the incorrect p-value.
"Step 1: Executed 'list_folde (folder= results " and successfully listed files ['Figure_A16 pdf 'Figure_Aie A10. 'Table_A7 tex', 'Table A2. tex', Table_2 tex" 'Table_All tex 'Table_3 tex', 'Figure_Al1, par 'Figure_A9 'pdf', table__6.tex', 'Table A10. 'Figure_2 2.pd' 'Table_At A1. tex', Figure_3. pdf'', 'Figure_ALs A 'Figure_A3 pdf', 'Figure_A7 "da
Figure_ A8 , 'Figure_At A 'output', 'Figure_A15 pdf', 'Table_1 tex', 'Table A8 tex', "Figure_A12 pdf', "Table_3a tex', 'Figure_7 p
25

---

Published in Transactions on Machine Learning Research (01/2025)
Figure_6 seuure_..pff. 'Table_A9 tex', 'Figure_ _ 'Pigure_Ate pdf',
Figure_5 pdr pdf', Biiguee_11.pp1 pdf', 'Figure A5 A pdf', 'Figure_ A6. pdf
Step 2: Executed "query hawguaaaggggeooeeee modem (proppttot the sample with the lower homicide rate per 100k in 2000 and the p-value from the predicted distribution of endline trust in police.', imptt_imgge_aathe-(reeee Figure_2. 2. pdf 'results/Figure, pdr are multiple figures the results directory, and the file names not **Status
**Rassoning:* \"There
provide clear information about the figures' content.\"ln" :** 'error 'n",
**Reason:** Error code: 400 ('error': {'message', You uploaded an unsupported image. Please make sure your image is beloя 20 MB in size and of one the following formate: ['png', 'jpeg', 'gif', webp ']. 'type': "Invalid_reuues_erne
'param': None, 'ode' "*Error:** Commandiecutionkere ('Error 'Eroor invalid_ errorro\\'' (V\'message\\': Yoo uploaded unsupported
code: 400
image. Please make aure your image is below 20 MB in size and is of one the following formats: [WVpng\\' Wjpeeg/'', W'gifll' \\zebpp\']. W type\\': AVVinvalid.reuust.orvy W // WW'aramm\'' None, (I'code W aavalid_imge_formatil
Step 3: Executed 'execute_ shell (command_ line= pddftopp -png results/Figure pdf results/Pigure_2
pdftoppm -png results/Figure.3 p.p results/Figure_3 \n" **Reasoning:** \"The PDF files need to be converted to compatible image format (e.g., PNG) before querying the vision language model. \\\n", ""","
**Status:* 'success 'n", \n",
**Output \n", STDERR:
STDOUT:An",
\n"
Step 4: Executed query_vision language_model model (prompt='Find the sample with the lower homicide rate per 100k in 2000 and the p-value from the predicted distribution of endline trust in police.', Figure_2-1.pp png', 'realtts/Figure3 pag.ppgg11 '1) \n" next step is to query the vision language model on the converted PNG images to extract the required information. \'nn, The homicide rate per 100k in the year 2000 is shown in the first image. The solid line represents Antioquia, and the dashed line represents Colombia as whole. In 2000, Antioquia has a higher homicide rate per 100k compared to Colombia.
**Reasoning:* \"The
**Status 'success 'n",
**Output: \n",
"''nn"
Therefore, Colombia (the dashed line) has the lover homicide rate per 100k \n",
find the p-value from the predicted distribution of endline trust in police, relevant measure from the trust category is \"Change in trust in police ( endline baseline). We look for the point eatimate with the Block FE, lagged DV (represented by triangles). An", graph:\nt
in 2000.Ant
refer to the second image. The
Based on the \"Change trust in police
\n"
point estimate for (endline baseline)\ with Block FE and lagged DV crosses the zero line. The corresponding confidence intervals also cross zero.\n", When the confidence intervals cross zero, it usually indicates ton-sigmificance. above the typical threshold for statistical significance (usually 0.05) An" summary:\n",
\n"
this plot, it suggests that the p-value
Colombia had a lower homicide The p-value for the predicted distribution of endline trust in police, based on the provided graph for the \'Change in trust police (endline baseline) with Block FE and lagged DV, is not statistically significant (p-value > 0.05). An", 'e1
\n",
rate per 100k in 2000.An",
Step 5: Executed Eurite_file wfwlefllellllemmeerrrrre 7
json', contents= From the figure measuring homicide rate per 100k in the last 12 months, report the name of the sample with the lover homicide rate 100k in 2000.\": \" Colonbial", \"fig From the figure measuring bootstrapped predictive distribution of endline trust in police assuming mean regression at rate of
26

---

Published in Transactions on Machine Learning Research (01/2025)
regression among unexposed citizens, report the p value from the Heard of Meetings plot.\": \"> 0.05\"}')
D.3.2 Manually reproducing the code instead of using Docker BeARE--Becc--edie
On capsule-8234138 Agee-Agent with GPT-4o-mini tried to manually reproduce the code even when prompted to use Docker, causing the agent to eventually hit the context limit and fail. We found this problem to be more persistent weaker models like GPT-4o-mini, which did not follow instructions well as GPT-4o.
Step 1: Listed the contents of the current directory, resulting in successful output of 162 files and directories. Step 2: Read the RRERODDUCING md' file successfully, which provided instructions for reproducing results using a Docker image from Code Ocean's registry. An", "An", " Step 3: Executed 'pip install II code/requirements txt --no-in resulting failure during
installation packages, specifically with numpy due to metadata generation issues.
Step Executed 'pip install numpy=-1.18.. --no-inputs but the installation failed due to metadata generation error.
Step Attempted install 'matplotlib==3.221 to resolve dependency conflicts, but the installation failed due to missing FreeType dependency
Step 6: Installed "libfreetype6 -dev and related packages successfully, enabling Matplotlib to build.
Step 7: Reinstalled Matplotlib version 3.3.2 successfully after installing FreeType.
Step 8: Executed 'main.py' with GOBI the acheduler argument, resulting in success status but encountering a "ModuleNotFoundrroor for 'joblib'.
[...]
Step 65: Executed oope_ffile (file_ hoeell/dde/e/eeeee py An" R*Raooning :** \"I need to adjust the import statements 'grapher. py ensure it can access the required modules directly, which will resolve the ModuleMotFoundIrrr" for "src'.\"\n"
**Saaus:** 'success 'nn,, has been opened and added to the context ** Output File code/grapher:
u2705
Step 66: Executed 'open_fill sfileffi__ppttii coch=/'code/eee code/grapher py 'n",
**esonning :** ""Next, I will adjust the import paths in 'grapher -py' to enaure it references modules correctly.\"Me " **Status 'success ' An", Output:*. File code/grapher,. has been opened and added to the context \ u2705
Step 67: Executed
code: ') An",
Reasoninn 'grapher. py to
*** \"The next step to modify the import statements ensure it can access the necessary modules directly, which will resolve the ModulelooFoondError for 'src'. '\nn", **Status:** 'success An", ** Dutput::** File code/grapher.pp has been opened and added to the context A
Step 68: Executed fopen_file(fs
code://ooee//geeeeeeee "y
D.3.3 Being unable to install the correct version of dependencies (CORE-Bench-Hard)
In capsule-8807709, CORE-Agent with GPT-4o installed network-diffusion version 0.14.4. However, one of the import statements (from network, diffusion import MultiSpreading) threw an error because the import was only supported in version 0.6). The agent successfully realized may need to install an older library version and performed web search see which version was applicable, but could find the correct. result within the cost constraint.
27

---

Published in Transactions on Machine Learning Research (01/2025)
This example shows how reproducing paper be difficult task, even for human. Determining which library version to install without additional documentation can be hard without external knowledge.
(filenlneeeeeeeeeeeeeee Step 1: Executed B md md and successfully identified dependencies and installation instructions for 'epidemic.py"
Step 2: Executed 'list Liistfoolder (folder="")))) and found: ['config. ini', 'LICENSE' 'README. md', '.gitignore'' .DDS_Store', 'code/config. 'code/ ppy', 'code/LICENSE', wwwp://oppoo....p..oooo..... hppp://paappppppbb....... 'code/sarreet__ometittoa ipynb',
:cdde/auctlaarrre//ooeleeeee.. o 'code/
'code/
ausilaariee//lorrettien.....
hrg.a//aeaaaaaaaaoaaaaaaaaaaaaaa S|g.eaaeeegg.a..........aaoaa.ar 'efficiency_
auxillarieet/hhase...pugg
'code/auxiliaries
/general_model.ppg' s org.a//eop..cciiiicccoaaattttaar
data_ndlib.cav',,
py", 'efficiency_ tests 'metadata/
/tested_functions py", "efficiencyy.eesss/aataaaaa
netadata.yml")
Step 3: Executed 'read_files 'radd_fiiaciilleeeeedcdddiiipa
and successfully retrieved the content, including the import statements, configuration setup propagation model initialization, and node state settings for a multi- spreading experiment.
Step 4: Executed 'pip install -I requirements txt Reault: ERROR: Could not open requirements file: [Errno No such file or directory: 'requirements. txt'
Step Executed 'pip install networkx numpy' and successfully installed networkx-3.3 (numpy was already satisfied).
code/epidemic.pp)) resulting in 'NoduleNotFoudEroor: No module named 'network_difee "iftorrkdifffssonn= Step 7: Read the 'README file successfully, revealing instructions for using the "network_diffusion. package, including setup via Codelcean or Python environment, and details on running various example files. Step 8: Executed "web_search" (query='python network, diffusion module installation') and found the relevant zodule https://pypi... http:///oooeeeeeeotott-aataaaaas Step 9: Executed 'pip install network resulting in the successful installation of 'network network-diifusiooo00111... and its dependencies, but noted dependency conflict with 'numpy".
Step Executed 'execute_ (command Lomeand_Lnee"yttuuillaa
on PyPI
Step 10: Executed 'python3 code/epidemicc.pp resulting in ImportError: cannot import name "MultiSpreading' from 'network_ 'network_diffusionn diffusion'.
Step 11: Executed 'pip show network-diffusion and found that the module network_diffen diffusiiiff is version 04.4. authored by Michalu0142 Czuba and Piotr Br\u00f3dka, licensed under GNU GPL v3, located at '/root/. cache/ cache/
11tpE/B88pp..11/1///////.. 10/1ib/pptthon and requires 'dynetx', 'matplotlib', 'networkx', 'numpy', 'pandas" 'scipy and 'tqdm' "nn", "\n", "* Step 12: Executed '1s pwwkaagssneetorkk http:/oonn.///teteepppaa///eeeae http://tttoo.dffffsioooooot diffusionn and found files: init_..
pppotrr/virttnalevv//aaaaa
-py3.
lib lib/python3 10/ t00/site-acckeee py p__pyccc 'logger. ,y 'mln', 'models', 'seeding', 'simulator.n 'tests', 'tpn', aatils.yy'.
Step 13: Executed '1s' command on
directory, successfully listing files: "__init__.pp .p' __-yyaccee_." 'base_model, sapel.pp. .y' 'dsaa_model.py" " "mic_model.pp" py', 'mit_model py", 'tne_ ine_model.pp' and 'utils
Step 14: Executed 'grep 'class KultiSpreading in the specified directory; output was produced.
Step 15: Executed 'redd hotpaeeeeeeielaaeeeeeeaeeaeeeeee code/ " successfully and found the script initializes and performs a propagation experiment using the 'MultiSpreading' class.
Step 16: Executed 'jupyter abconvert --to notebook --execute code /eee/gssspp... ipynb with result: '/bin/sh: 1: jupyter: f
28

---

Published in Transactions on Machine Learning Research (01/2025)
Step 17: Reviewed 'README. =d successfully; detailed steps for running epidemic. py and installing dependencies were found.
Step 18: Executed 'pip install ipykeruel tqdm successfully, installing packages: 1p.kernel-6.29.
comm comm-0 -0.2.2, sebugpy-188.58 ipython -8.26.0,
jupyter-client-866.2 jupeter-core-5.7. natplotlib-imlne-00 asyncio-1.6.0, 2y.mq-26.12 1.0, 6ernado-6.4.11
nest
Praitlets-5.14.. jedi -0. 19.1,
prompt-toolkit-33oo. stack facc--dtaa-.6.... asttokens -2..1 executing-22 -20011, pure-eval-0.2.2
Step 19: Executed 'python3 code/epidemic.pr resulting in an ImportError:
cannot import name "MultiSpreading" from betworr_diiffsion MultiSpreading class', num_results-1)- successfully; found a relevant URL: https network-- diffusion readthedocs
Step 20: Executed 'web_search hewerreeeetorrrkdittt...
modells html.
[...]
D.3.4 Attempting to look up the capsule CodeOcean (CORE-Bench-Hard)
Also in capsule-8807709, CORE-Agent with GPT-4o, after being unable to locate the requirements. txt file in the repository, attempted to look up the capsule on CodeOcean online. The agent ultimately did not succeed because JavaScript required render CodeOcean, which the agent did not have access to through its web browsing capabilities. However, this example highlights the care that agent developers must take during evaluation.
[...]
Step 60: Attempted to read 'requirements. txt but encountered an error: [Errno 2] No such file directory:
"xt",nn,, "An", Step 61: Executed 'web_searchh (query='netvork diffusion site: com', num aum_results=5)- and found relevant repositories, including \"GitHub
9807709//eqqiremanne
GiulioRossettindllb:: Network Diffusion Library https httpggtthhbbc... com/ GhulioRosetti/ndlib
Step 62: Executed 'web_ (query='codeocen codeocean capaule 8807709 and found links including the main capsule page (here) /oottps://cooooeaan codeocean. com/capsule /8807709) with additional details on compute capsules and their functionalities
Step 63: Executed 'web_search search (query='nttps http:: c/cooo/cenn cot. and retrieved multiple relevant URLs and excerpts related to Code Ocean capsules.
Step 64: Executed 'web_searee http hatpaccaaa.ccccccaalll///000000 and found multiple relevant URLs, including https: http::/ccoocccccccc//cpaa /8807709, which requires JavaScript to Step 65: Executed 'web_search, (qaaarchhuuuu/y//1 c/coeoccc// httpaaaccc.cccapee//////00000 find relevant information about Code Ocean's compute capaules; successfully retrieved multiple URLs and excerpts detailing their features and usage.
Step 66: Executed 'web_search http://hhp//occcccccc/////a///// cat.caaaall//8077099 with success, retrieving multiple relevant Code Ocean URLs and excerpts, including https://codeocean com capssull//80/770 and https: /tte://coooccan.... com/ capaule/80770//tree/ Step 67: Executed 'web_seard" and retrieved multiple relevant URLs, including https wttp:://coocccccccc/ com/ capsule/880771
Step 68: Executed 'web beb__earchh (oobeeaaarclderrrr/ccoooeea
codeocean. com capsule 8807709',
num num_reaa rem_resulls:)' found URL: ctttp::/cooooccaa.coo// http:///cccoccc..cccaaaaaa/a//// cot/capalle/887/000
faatt__filefileenneeee Step 69: Executed 'write_file (filename= requirements txt', contents= anetworkx /nnumpy Comupyy/nnten \nntgdm /mmattllol ( (tn.aaaaaaaaaaaniiipaaaa
\nscipy \nacipy\|nnetror. -
29

---

Published in Transactions on Machine Learning Research (01/2025)
diffuusioudttrpp://gtttuu http://////.aapoooooo//a//a/aao/ /ttt:arr....ffull//arrrri///eeee http:////atttttaageeeeetooototta
heads/master: aiptesgeeetsortt_dffffo diffusion') successfully, resulting in the file requirements txt being written.
Step 70: Executed 'pip install -I requirements. txt' and successfully installed most packages, but encountered LAPACK libraries.
with scipy due missing BLAS/
Step Executed 'execute prcpuutee_pthoo_coooo (code= import network www.fusion/laprlliii dir (network, (itiroookkkfffssoonon)o
Result: "ModulelotFouddrroo No module named
nneworrk diffusion' "nettork_diffusioon
Step 72: Ran 'python3 code/epidemic.pp and encountered ImportError: cannot import name 'MultiSpreading' from metworkk_iiffssio Step 73: Executed "execute_python_com (code=' import netuork,diffusion p print (dir resulting in "ModulelooFFundEroor: No module
named "network_diffusionn"
Step 74: Executed 'executee_yytion_ccmm cocpahoooococeeeeeee
(code= import detwork_diffusion; print module named 'network_diffusion
dir beetwork_diffsiio.nocooooo
resulting "oduueeNotFoudErrr::: No
Step Executed 'import network_diffusiin models as models; print ([cls for cls in dir (models) if resulting "MouulelotFounddrro::: No module named deetwork_diffssion Step 76: Executed
(models, cls), "_.ssass__"" ')]]
difffusion MultiSpreading class GitHub') and found relevant resources, including GitHub repository for NDlib and various documentation and articles network diffusion models. Step 77: Executed web__eaacch (query= M class atte:github.com
nun_results=5)"; found five relevant GitHub pages discussing multithreading in C++ and Java, but no direct references to the 'MultiSpreading" class. Step 78: Executed 'web_searee (query='CodeDcean account creation guide',
num rum_results=1)- and found \"Create an Admin Account Administration Guide\" https://doca.. codeocean. com hatm//ddingguie/eeeeee deployment guide/crett--aainnacccae
I Code Ocean VPC
Step 79: Executed 'read_files (feiffeane config. ini')' successfully, found '[ PATHS] output_dir: ./results'.
Step 80: Executed weeb_search (queryy Create Create an Admin Account Code Ocean VPC Administration Guide', num_results=i) 'nn", progress and challenges, setting up Codelcean seems vital to proceed clearly and efficiently. \"An",
**Raaoninng: \"Given
**Status :** 'success 'n", ++tpput Search results(n",
:** \n"
"'n'",
\"Create an **UBL:**
Admin Account Code Ocean VPC Administration Guide\"\n", https://docs codeocean. com/adiin outpea/ooooooooiouuuuu//////poor - -gddde/raattttteddiiii account \n"
**Excerpt:** \"Go https://codeoces [hoating domain 1/join to create an admin account Before you invite users, we recommend you deploy docker base image first. Visit Deploying Docker Images details. Learn to create the initial admin account on Code Ocean VPC. \"nn .re
[...]
E
Reproducibility Study Details
We report: the number of studies reproduced for each paper Table based on the format. of the results provided by the papers' authors:
The following papers report the percentage of reprodueible studies out of the total number of studics.
We manually calculated the number of reproducible studies and rounded the result: Stockemer al.
2018 Gertler al. (2018), Collberg & Proebsting (2016), Hardwicke et al. (2021) Rafl (0099)
30

---

Published in Transactions on Machine Learning Research (01/2025)
The following papers report the number of results metrics that were computationally reproducible,
rather than the number of papers: Gilbert et al. (2012). Trisovic et al. (2022). Samuel & Mietchen
(2024), Pérignon et al. (2024). Belz et al. (2021).
McCullough et al. (0066) reports approximate number of papers reproduced. Authors state that they analyzed greater than 150 papers, with less than 15 replicated.
We manually analyzed the papers from the 2022 Machine Learning Reproducibility Challenge (Sinha al. 2023). Of the 44 papers submitted the challenge, 28 attempted reproduce papers where both data and code were fully available. 10 of those 28 papers were only partially reproduced. We consider papers to be fully reproduced if all the main claims of the paper completely hold, even if the reproduced quantitative results slightly deviate from the original results. For example, we consider Livernoche & Sujaya (2023) a successful reproduction of the original paper because authors validate the original paper's claims and results fall within the standard deviation reported the original paper. On the other hand, we consider papers to have reproducibility errors if all the main claims of the paper cannot be reproduced, or if result values from the original paper deviate significantly from those of the reproduced paper. For example, we treat Brivio & unsuccessfil reproduction because the highest accuracy score from the reproduced paper deviates significantly from the original paper although the original hypothesis consider the results of additional experiments not contained in the original paper. Of the fully reproduced papers, many codebases contained errors, outdated packages, or limited documentation, requiring researchers to modify the codebase during the reproduction process.
Cöltekin 2023
verified. We do not
31


Reviews:
[{"review_id": "CrGrQawzh5", "reviewer": "Reviewer_SjHV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths: \n\nS1: This is a timely task, and I really appreciate the task definition: being able to replicate results is indeed important, and I think this paper thoughtfully defines a part of that that should theoretically be solvable by agents, and designs a benchmark around it. \n\nS2: Continuing on task setup, the setup of the overlapping layers of difficulty is really solid. That mistakes in Easy are a subset of the ones that can be made in medium, and so on, makes this setup a really good basis for doing diagnostic work on where models fail, and how to make them better. \n\nS3: I really appreciate the authors' willingness to look at particular instances of error cases that these models make. The kind of qualitative analysis in this paper is really welcome. That said, I think it would be great to see a systematic breakdown of what kinds of tasks and stages models fail on. This is clearly something the authors have already done, at least in part - they mention that models sometimes fail in installing dependencies, or fail to locate the correct file from which to evaluate results. I'd love to see the percentages of which models break down at which steps. This would be a perfect opportunity to use e.g. a Sankey diagram per model to display what the common failure modes of models are, so that users of the benchmark can easily diagnose what high-impact fixes for their agents might be. I'd also really appreciate some degree of breakdown of what the specific questions are and evaluation on those - how did models fare on identifying axes vs specific points? The authors currently break this down by modality, but no further. \n\nS4: The counterfactual evaluation for things like compute budget showed really thoughtful experimentation and verifying that the choices made here were reasonable. \n\n## Weaknesses\n\n I have a couple here, but I think only the first one is a real issue. \n\nW1: This paper's intended scope, as defined in the research question on page 3, is much broader than the paper eventually answers. This paper represents a solid step towards that research question, but to be clear, it evaluates models on their ability to reproduce results that are _already known to be reproducible._ To answer the question as posed, there are a number of questions that this paper doesn't answer: what is the models' behavior in cases where the research is not in fact reproducible when attempting to run the provided code? How would such a system differentiate between agent failures and more fundamental reproducibility failures? \n\nThese are questions that could reasonably be argued to be beyond the scope of this paper, but in claiming that this work has \"real world utility,\" in that a model that is strong on this benchmark would help authors \"verify their work's reproducibility,\" the authors assert that those questions are functionally answered. I think scoping down the claim of this paper, and making clear that this is a necessary, but not sufficient step towards automated verification of reproducibility, would render this paper no less strong in its provided benchmark and baselines. \n\nW2: I don't understand the point about how this code being from public repositories mitigates concerns about contamination. Even if they can be periodically updated, what is the number of repositories that are, especially after paper submission? Could the models that underlie the agents for this benchmark not also be retrained? \n\nW3: The difference between CORE-Bench Easy and Medium is very small (and reflected in a correspondingly small accuracy drop), and the difference between Medium and Hard is really large. The paper notes this, but I think the utility of this benchmark and results obtained on it would be helped by understanding _where_ models fail as part of the benchmark - if it's in dependency management, the benchmark could verify the environment before experiments are run; similar steps could be taken for other large steps in the process. While it's useful to know that models fail given the whole repository and instructions, knowing why and when they fail would make this benchmark much more useful as a diagnostic for model performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper defines a new benchmark, CORE-Bench, that measures the ability of LLM agents to effectively imitate the reproduction of results from studies known to be reproducible from three domains, all hosted on CodeOcean.com. The paper additionally proposes two baseline measures, the first using a (mostly) vanilla instantiation of AutoGPT, the second an agent (called CORE-Agent) customized specifically for this task. The authors perform detailed qualitative error analysis on where current models fail.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Scoping down the claims of the paper to be more in line with what the paper actually demonstrates\n2. Some degree of quantification of the error analysis - how common were the common failure points?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pp5WVI2oLG", "reviewer": "Reviewer_t3v5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper is clearly written and easy to follow. The design choices for each difficulty level of the benchmark tasks are explained thoroughly and seem like a natural progression of skills to test for in reproducibility agents. The attached code appears to be well-documented and user friendly (though there is a caveat to this discussed below), and the evaluation harness provided by the authors seems helpful in drastically reducing evaluation time (provided the user has access to enough virtual machines to make full use of the harness). Testing the evaluation framework on an existing agent (AutoGPT) and using the resulting error logs to make improvements to said agent (CORE-Agent) is also convincing evidence that CORE-Bench can be helpful in developing better reproducibility agents.\n\nThis topic is of interest to TMLR's audience. Tools for verifying the reproducibility of scientific work would be of high value for any academic discipline, but especially for ML/AI as the number of published works has exploded in recent years. This paper is an important step towards automatic reproducibility verification, serving as a meta-verification testbed (i.e., a way for verifying that a reproducibility verifier works).\n\n## Weaknesses\n\nThe biggest shortcoming is the barrier to reproducing the results in this paper. None of the provided agents can be run without payment, and running the full pipeline without access to Azure takes many days as the authors themselves state. Given that the focus of this paper is on reproducibility, it seems imperative that the authors provide at least one baseline/test setting which can be reproduced locally and for free, so that their framework can be partially checked with relative ease.\n\nPart of the filtering process when creating CORE-Bench was to select tasks which were in fact reproducible. Another important aspect of reproducibility is verifying when previous work is *not* reproducible. This may be a harder task to quantify, as there is basically only one way for reproducibility to succeed (i.e., the code runs and you get the same numbers that were claimed by the authors), but more ways for it to fail (code doesn't run due to dependencies/bugs, experimental results are different, etc.). The authors discuss this choice briefly at the bottom of pg. 5, but I believe more in-depth discussion is necessary to provide compelling evidence for the claim that the proposed benchmark consists of \"real-world computational reproducibility tasks.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a benchmark dataset, CORE-Bench, for testing AI agents whose job is to check the reproducibility of scientific publications. The benchmark consists of three levels of task difficulty corresponding to different amounts of information/steps of the task which have already been completed that are supplied to the AI agent. Using the results from testing on their benchmark, they make modifications to an existing baseline (AutoGPT) which improve task performance. They also implement an parallel evaluation framework which drastically reduces the amount of time needed to run all of the tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide code which can be run locally, for free, and in a reasonable amount of time which would allow a reader to test a part of your benchmark.\n\n2. Please provide additional justification for the decision not to include any \"negative examples,\" i.e., papers which are not reproducible, in the benchmark.\n\n3. The claim of the header of Section 4.1 is not exactly correct, as the AutoGPT baseline actually performs slightly better on the medium vs. the easy level. Can the authors provide some explanation for this?\n\nConditional on the resolution of these issues, I would recommend accepting the paper.\n\n---\n\nAfter the author discussion period, I believe the authors have adequately addressed my concerns. I recommend accepting the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SDQo5SbOCR", "reviewer": "Reviewer_VqsX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n \n- Computational reproducibility is an essential subject in scientific research. Using AI agents to verify the reproducibility of published research is interesting.\n- The writing is pretty clear and easy to follow.\n- The capsules involve several scientific domains for both Python and R. The paper claimed that CORE-Bench is among the first to include tasks in R.\n\n\n**Weaknesses**:\n- The authors mentioned in 2.1 that including irreproducible papers in the benchmark is unnecessary, which I'm afraid I have to disagree with. In real-life settings, irreproducible papers can be easy to find. Observing how the agents perform with these tasks is also an important aspect of evaluating the agent's performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces CORE-Bench, a benchmark designed to evaluate agents' ability to reproduce the results of scientific studies using provided data and code. The work filtered and selected 90 capsules from CodeOcean, spanning medical science, social science, and computer science, and utilizing Python and R. For each capsule, three task difficulty levels (Easy, Medium, and Hard) are created. The work measures the agent performance with the proportion of tasks in which all questions are answered correctly, defined as results falling within a 95% prediction interval derived from three manual runs. Two baselines, AutoGPT and its task-specific variant CORE-Agent, are tested on this benchmark using GPT-4o and GPT-4o-mini. The results demonstrated expected performance variations across difficulty levels and revealed substantial room for agent improvement in solving computational reproduction tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- **Q1**: What's the exact proportion of R and Python in each discipline? I think Figure 8 can be further improved since we cannot tell any varying performance from this figure in each discipline or programming language if it is disproportional.\n\n- **Q2**: What's the exact proportion of visual and written questions in each difficulty level? I might have missed something in the paper, but I didn't find the exact number. It would be more convincing if some examples were illustrated.\n\n- **Q3**: The paper reports the experimental results of AutoGPT only with different-sized GPT-4o. I would like to know if other LLMs like Claude and Gemini are tested. What's their performance in different levels of the CORE-Bench, and what are any findings compared to the GPT-4o?\n\n- **Q4**: There are various reasons for a task's failure, such as dependency installation and results retrieval. I am wondering if the authors can provide more details. For instance, in each level of difficulty, what are the reasons that lead to failures, and how many tasks are there for each reason?\n\n- **Q5**: What's the time cost for finishing all the tasks at each difficulty level? What are the possible reasons that may lead to more time costs?\n\nI would be more convinced if the above questions could be explained.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: SjHV accurately points out 
> "What this measures is not whether AI agents can automate computational reproducibility, but whether computational reproducibility can be verified by AI agents, which is both a weaker claim, and results in less useful AI agents"

while
> "primarily problems of framing"

The authors failed to appreciate this nuance in their rebuttal.  The work otherwise appears to be technically correct. It is therefore my responsibility to ensure that the revision correctly rescopes to match the results.
Decision: Accept (poster)