From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking
Authors: Gyeongwon James Kim
Alex Wilf
Louis-Philippe Morency
Daniel Fried
Venue: ICLR 2026 Poster (2026)
Keywords: agents
research agents
coding agents
benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce AutoExperiment, a benchmark testing agents' abilities to implement research ideas into code by progressively masking out varied portions of the codebase.

Abstract: Recent progress in autonomous code generation has fueled excitement around AI agents capable of accelerating scientific discovery by running experiments. However, there is currently no benchmark that evaluates whether such agents can implement scientific ideas when given varied amounts of code as a starting point, interpolating between reproduction (running code) and from-scratch replication (fully re-implementing and running code). We introduce AutoExperiment, a benchmark that evaluates AI agents’ ability to implement and run machine learning experiments based on natural language descriptions in research papers. In each task, agents are given a research paper, a codebase with key functions masked out, and a command to run the experiment. The goal is to generate the missing code, execute the experiment in a sandboxed environment, and reproduce the results. AutoExperiment scales in difficulty by varying the number of missing functions $n$, ranging from partial reproduction to full replication. We evaluate state-of-the-art agents and find that performance degrades rapidly as $n$ increases. Agents that can dynamically interact with the environment (e.g. to debug their code) can outperform agents in fixed ``agentless'' harnesses, and there exists a significant gap between single-shot and multi-trial success rates (Pass@1 vs. Pass@5), motivating verifier approaches to our benchmark. Our findings highlight critical challenges in long-horizon code generation, context retrieval, and autonomous experiment execution, establishing AutoExperiment as a new benchmark for evaluating progress in AI-driven scientific experimentation. Our data and code are open-sourced at https://github.com/j1mk1m/AutoExperiment.

Published as a conference paper at ICLR 2026
FROM REPRODUCTION TO REPLICATION: EVALUATING RESEARCH AGENTS WITH PROGRESSIVE CODE MASKING
Gyeongwon James Kim, Alex Wilf, Louis-Philippe Morency, Daniel Fried Carnegie Mellon University Pittsburgh, PA 15213, USA {gyeongwk, awi 1f, morency, dfried]@cs. cmu.
ABSTRACT
Recent progress in autonomous code generation has fueled excitement around Al agents capable of accelerating scientific discovery by running experiments. How- ever, there is currently benchmark that evaluates whether such agents can im- plement scientific ideas when given varied amounts of code as a starting point, interpolating between reproduction (running code) and from-scratch replication (fully re-implementing and running code). We introduce AUTOEXPERIMENT a benchmark that evaluates Al agents' ability to implement and run machine learn- ing experiments based on natural language descriptions in research papers. In each task, agents are given a research paper, a codebase with key functions masked out, and command to run the experiment. The goal is to generate the missing code, execute the experiment in a sandboxed environment, and reproduce the results. AUTOEXPERIMENT scales in difficulty by varying the number of missing fune- tions n, ranging from partial reproduction to full replication. We evaluate state-of- the-art agents and find that performance degrades rapidly that can dynamically interact with the environment (c.g., to debug their code) outperform agents in fixed "agentless" harnesses, and there exists signifi- cant gap between single-shot and multi-trial success rates (Pass @ 1 vs. @ass@5), motivating verifier approaches our benchmark. Our findings highlight criti- cal challenges in long-horizon code generation, context retrieval, and autonomous experiment execution, establishing AUTOEXPERIMENT evaluating progress in AI-driven scientific experimentation. Our data and code are open-sourced https//github.commjmmbl http://.ooo...mmmmmmmmmmmemceoem m/Autoxxpermmm
increases. Agents
new benchmark for
INTRODUCTION
Advances in AI agents are reshaping how we approach scientific research (Lu et al. 2024p. lead- ing many ask whether have unlocked new forms of automated inquiry. However, still lack benchmarks to evaluate the ability of Al agents to generate code for scientific experiments.
While many benchmarks have been proposed evaluate AI agents' capabilities on related code- generation tasks (Hendrycks et al. 2021 Chen et al. 2021 Yin et al. [022] Jimenez et al.) 2023). these have typically not evaluated agents' abilities to implement scientific code, which presents unique challenges. Recently, some benchmarks for scientific code generation have been proposed that require reproducing experimental results: confirming findings by rumning (but not implement- ing) the code (Bogin et al. 2024). Moreover, [Starace et al. (2025) and Hua et al. (2025) proposed benchmarks on replicating results: confirming findings by writing code given only the description in the paper. Yet these benchmarks each have fixed difficulty, and do not evaluate agents' abilities to implement experiments given the research paper and varied portions of the original codebase.
In this paper, we introduce the AUTOEXPERIMENT benchmark, which challenges agents to imple- ment Python functions that have been removed from the original published research repositories, run the code in sandboxed environment, and report the results of the experiment. As input, the agent is given the full research paper containing descriptions of the experiment, the command to run the experiment, and the codebase with given number of target function(s) masked out. To evaluate

---

Published as a conference paper at ICLR 2026
Input
Agent
Evaluation
Paper
</> Partial Codebase
LLM
Butput
Expected
Correct!
stits. By
Exp 1 10.1
10.1
12
Memory
Exp
7.0
8.4
X
Tools
deccinae and papartt
Experiments
D Editor
Pethoon Batoup5 -.3 16-4
Territeed Code
Al1 Correct?
Figure I: AUTOEXPERIMENT challenges AI Agents to implement and run experiments. The inputs the papers with the description of the experiments (but the results), the command sequences to run the experiments, and partial codebases where we mask out a progressively increasing number of functions from the original repository. The Al agent must implement the missing function(s), run the experiments using the command sequences, and correctly report the results of the experiments in order to pass the test cases.
correctness, the agent's findings are compared with the results of the same experiment run with the gold-standard published code from the paper. By progressively masking increasing amounts of the codebase, our benchmark can control the difficulty of the task, guiding clear path from reproduc- tion to replication. Our key findings are:
1. Moving from reproduction to replication is hard, even for frontier models: as we scale the difficulty of the task by progressively masking more of the code, we find that even the performance of agents powered by frontier language models falls off quickly, and that agents begin to rely increasingly on the natural language context from the research papers to bolster performance.
2. Agent performance scales with dynamic interactivity and test-time compute: agents that can dynamically make decisions on how to interact with the environment over time outperform agents in a fixed interaction harness. Agent performance also scales moderately with cest-time-compue in reasoning models, although the trend is less pronounced.
3. The @ass@k gap is large: the pass rate improves over 10% from Pass@] to @ This motivates future work on model-based verifiers for reranking, or agentive search ap- proaches which could recapture some of the performance of the pass @k scores.
THE AUTOEXPERIMENT BENCHMARK
AUTOEXPERIMENT evaluates agents' ability to write scientific code and run experiments. We col- lect four peer-reviewed research papers that have already been replicated as part of the ML Re- producibility Challenge (MLRC) (Rougier et al.) 2017), mask out 85 functions from the publicly released codebases of those papers, and use the experiments and their results as "test cases" evalu- ate the agent's 's implementation of the missing function(s). Papers from MLRC have been replicated, meaning that the method described in the paper has been reproduced and the results have been ver- ified by another researcher. By limiting our selection to MLRC, our benchmark focuses on quality over quantity, guaranteeing that papers have a strong correspondence between natural language de- scription and verified code implementation. We document the benchmark creation process in detail below.
Collecting Research Papers Our key insight is that peer-replicated research papers can act as an effective testbed for creating agents capable of running experiments. The key requirement is that papers should be both reproducible (i.e., running the code with the command sequence for that experiment produces the corresponding result from the paper) and replicable (i.e., given only the contents of the paper, it is possible to write code from scratch to implement the experiments and arrive at the same results). To ensure reproducibility, manually up environments and
2

---

Published as a conference paper at ICLR 2026
the gold standard code with gold standard command sequences, verifying that all results from the research papers are reproduced. Furthermore, to ensure replicability, we select papers that have been replicated as part of the Machine Learning Reproducibility Challenge (MLRC) (Rougier al. 2017]. We manually performed additional filtering to ensure a high quality dataset: we select papers such that 1) the code repositories are well maintained with good documentation and (2) there is strong mapping from research paper contents to the code. We list the details of the research papers in Appendix A
Masking Out Key Functions To construct each sample in the dataset, we collect Python functions that implement core logic for the experiments and replace their implementations with Not edEroo.. We select functions that produce errors when masked out, verifying that these functions are indeed essential in arriving key experimental results. We experiment with masking out n 2 1 functions at time, treating each combination as unique sample in the dataset. Our dataset comprises 85 unique functions, leading to 85 unique samples for n 1, and up to 275, 990 possible samples for = 5. To keep experimental runtime feasible, we select a maximum of 100 samples for evaluation for each setting of n. Although the number of samples constant across increases those samples are drawn from an increasingly diverse set of possible com- positions These statistics are depicted in Table The functions are on average 26.3 lines long and have average of 15.9 function calls to standard library functions other functions in the code.
Evaluation Harness To evaluate the agent's output for each function, we collect a set of test cases: command sequences from the repository README that reproduce experimental results, and that rely on at least one of the masked out functions. We pass the command sequences to the agents part of the input, and ask the agent output the results of the experiment in structured format we can parse easily using a tool call described below. Our evaluation harness checks this output against the results from running the gold-standard code with the same command. We give agents access to the test command (instead of rumming the generated code in an evaluation harness offline) so that an agent can debug the code before submitting its final answer. To prevent cheating, we remove all numerical results from the research paper text.
We consider a sample to have passed our tests if the results for all tests differ relatively by no more than 5% from the gold standard results. The average number of testcases per sample across different values of n is given Table below. Since the test cases are experiments that rely on any of the masked out functions, the average number of test cases increases monotonically with n.
n
1
3
4
5
Samples
85
977
8046
52141
275990
Test cases per Sample
6.08
8.69
10.26
11.12
11.55
Table 1: Number of possible samples that can be generated given the codebase and average number of test cases per sample by where n is the number of functions we mask out at once. Final evaluation done over a fixed budget of a maximum of 100 samples for each setting of n.
In some cases, we take steps to shorten the runtime of the test cases - e.g., if a paper's experiment code involves training a model, reducing the number of training steps. In these cases, we ensure that our "curtailed" setting has high precision with the original test cases in its capacity to discriminate passing from failing solution attempts. See Appendix B for more details. On average, a complete agent run powered by GPT-4o took just over 11 minutes per sample, including both agentic actions and experiment runtime.
Agent Architecture There are five key components that define each agent:
1. Initial Prompt:
The system and user prompts deseribing the task the agent. This prompt specifies the goal of the task, the function that has been masked out, and the command to run the experiments. The exact prompt used can be found in Appendix I.I
'The number of samples we have for different values of n increases combinatorially for each paper. E.g. a paper with 23 functions will contribute (29) samples for each n. Note, the increase happens individually for each paper, the number of samples do not increase combinatorially in total.
3

---

Published as a conference paper at ICLR 2026
2. Tool Definitions:
The exact tool definitions et al. (2024), which give the agents the ability to navigate the repository, manipulate files, and execute scripts. A high level description of these tools can be found in Table 2
similar the ones found in Huang
3. Step-by-Step Prompting Strategy: At each step, the agent
be prompted reason and output actions in different ways. We experiment with different popular strategies Yao et al.) 2023 Song et al.] [0033 Huang al. 2024), described in Appendix [.2 For the main experiments, we use the best performing strategy, ReAct (Yao et al. (023).. This is
weight strategy, which prompts for natural language thought and action at each action.
step. allowing the agent to reason about previous action history and determine the best next
4. History Management: As the agent continues to interact with the environment, the history of its interactions grows. How agents manage that history is the subject of recent research papers (Wang et af., 2024b).
experiment with different common approaches, described in Section[[] For the main experiments, we use Full history management, which performed best when combined with ReAct prompting. This strategy simply maintains the full list of previous actions and corresponding environment observations, which is incorporated into the prompt at each step.
5. Backbone LLM:
We evaluate different underlying LLMs: GPT-4o, GPT-4o-mini, Claude- 3.5 3.5-Sonnet, and 32B-Instruct and All Hands' Beenhands-mm-32b-a but found that these models struggle make any progress due insufficient formatting, tool-calling, and instruction-following skills.
We also tested open-source models Qwen2.5-Codes
Action
Input
Observation
Execute Command List Files
Command Directory
Executes shell command
Lists all files and folders in direc- tory
Execute Script
File name, arguments
Output of executing script Moves file dir to destination
Move
Source, destination
Edit File
File name, edit instruction, save file
Edit based on instruction (LLM call)
Write file Inspect File Lines
File name, content
Writes content to file File content between line numbers
File name, start line no., end line
no.
Understand File Change Directory Final Answer
File name, things look for Directory Final answer
Relevant information (LLM call) Changes the working directory Submits the answer JSON format
Table 2: Tools available to the agent. A complete description of these tools in JSON format can be found in Appendix J.3
Environment and Resources ensure safety and guarantee reproducibility, we create sand- boxed environment using Docker where the agent can edit the code and run command line instruc- tions within conda environments already set up to run code for each repository. We run all exper- iments Linux machine with AMD 24-Core processor and 2x NVIDIA GeForce RTX 3090 Ti GPUs. To limit runtime for the exponentially increasing number of samples, we evaluate on only 100 samples for higher values of n. described in Appendix E. For each sample, agents were limited to action- action-taking steps to limit the resources used by each agent, a maximum runtime of 30 min- utes, and maximum LLM compute budget of $1. The average compute time and cost of each detailed in Appendix K
Appendix contains an example of a full agent trajectory.
Data Contamination Recent LLM benchmarks have shown to be susceptible to data contamina- tion (Xu et al., 2024h. With the release of new models, it is crucial to have contamination free bench- marks that can fairly evaluate the models' capabilities. We evaluate the data contamination risk for AUTOEXPERIMENT First, the agent performance on benchmark at the easiest setting (n 1)

---

Published as a conference paper at ICLR 2026
is relatively low (< 40%). suggesting that contamination is unlikely. We also performed a data Qwen2.5-Coder. Cerooddrr--B--tttuuee and All Hands' openhands- -er-aanddd--------- 1b-3211 The results show that on AutoExperiment, models achieve a contamination score of 35%, 1 and 57%, respectively, which are all far below the threshold to be considered contaminated (85% according developers of et al. (2024)). Moreover, if detect data contamination in the future, a new version of the dataset can be created according to
tamination test (Shi et al. 2024) on models: Qwen2.5-1.5B-Instuue
the detailed process outlined in our paper. As Al research continues to grow (Azad Banu 2024), there will be an increasing source of high-quality research papers with code repositories that can be used create the dataset.
EXPERIMENTS
3.1
FROM REPRODUCTION TO REPLICATION: REMOVING 2 FUNCTIONS
20
GPT-40
mini
Claude 3.5 Sonnet
Claude 3.7 Sonnet
Pass Rate (%6)
2
Figure Performance on AUTOEXPERIMENT with functions removed at a time.
We experimented with masking out multiple functions once from the codebase, using agents powered by different frontier models. Agents perform reasonably well with function removed. Claude-3.7-son 3onnee leads with 36.5% pass rate followed by GPT-4o with 35.3%, Claude-3.5 with 31.8%, and GPT-4o-mini with 27.1%. Interestingly, performance plummets as soon as n with Claude-3.7-sonnet and Claude-3.5 3.5-sonnet having 9.6% pass rate followed by 8.5% for GPT-4o and 2.1% for GPT-44-mini a performance drop of 70-90% on average. As n increases, perfor- mance continues to drop with most models reaching negligible pass rates by n 5.
This precipitous drop is an important result because it shows that although frontier models perform well with one function removed, the benchmark becomes significantly more difficult as it nears the "from-scratch replication" setting, with all functions removed. Because we can control the number of functions removed from the gold codebase, we control a "knob" of difficulty for the benchmark, guiding the path towards full replication.
3.2 THE PASS @K K GAP
One important metric in code generation benchmarks is Pass@k, in which the model generates complete solutions to each problem and we pick the best one (as defined by the ground-truth test cases). This important because can act as an important indicator for the potential of search and verification strategies; simple search strategy (c.g., generate 5 solution attempts) combined with a perfect verifier yields the P performance.
We examined the @ass@k results for GPT-4o and Claude-3.5 Sonnet and found large gap between Pass I and @ for our benchmark: 35.3% - 48.2% for GPT-4o and 31.8% + 42.2% for Claude 3.5 Sonnet. This offers substantial potential that there could be large gains in performance this benchmark from better verifiers alone. In addition, this may motivate research into improving the policy function through RL.
5

---

Published as a conference paper at ICLR 2026
Passg1 Verifee Passg5
Pess Reae 196)
Models
Figure 3: We test agents with GPT-4o and Claude 3.5 Sonnet on n = setting. The gap between @ass@1 I (orange) and Pass@5 (red) performance is large. This is promising for lines of research into verification for use as part of search. We also examine "Verifier", which uses the model as its own verifier select the best answer before submitting that for evaluation. Modeli-as-verrfiees somewhat on this task, but there is still a substantial gap to oracle-level verification. We also report Pass"5 (all responses correct).
help
We also investigated using the model itself verifier ("Verifier"), to see how much of the P gap we can capture through current model verification capabilities (and how much room there to improve models verifiers for this task). We found that models used as their own verifiers could improve performance somewhat but there is still a significant gap to oracle-level performance as shown Figure Appendix describes the prompt used for the model verifier in more detail.
Lastly, we looked at Pass" k, which measures whether all k responses are correct, Overall, the differ- ences between Pass"5, Pass @ I, and Pass@5 metrics are much higher for GPT-4o than Claude-3.5- Sonnet, suggesting that GPT-4o has more variance in its responses and chosen actions.
3.3 SCALING INTERACTIVITY AND TEST-TIME COMPUTE AGENTS
Recently, ol was able achieve state-of-the-arr results on SWE-Bench, benchmark designed to benchmark agents, by using an "agentless" harness in which a fixed set of steps was applied, using an LLM to perform the individual steps (localization, repair, and patch validation) eXia 2024). Motivated by this work, we investigate whether it is helpful for the agent to have the ability to dynamically decide which sequential interactions to take with the environment (our main setup for all other experiments), model applied to each step. We run these experiments with the 03-mini model because using the full ol for each step is very costly in the dynamic setting. We adapted the agentless harness to task, where the pipeline consists of text and code retrieval steps, followed by code-infilling, and lastly, running the experiment and extracting results. In the retrieval steps, (OpenAl eext-mmbedding---smallll to embed chunks of text (from the paper) and code (from the repository), respectively, and retrieve top-k relevant context using cosine similarity. This retrieved context is included in the prompt that tasks an LLM to infill the missing function. Lastly, we run the experiment using the given command sequence and extract the results from the execution outputs with an LLM. The exact prompts and experimental cost are detailed in Appendix H
whether our task can be solved using fixed set of steps with strong
embedding model
Interaction Type
GPT-4o
03-mini
Fixed
8.3
27.8
Dynamic
35.3
33.3
Table 3: Agents powered by GPT-4o and o3-mini backbones benefit from the capability of dynam- ically choosing which action to take next in interactions with the environment, as opposed to the fixed three step "agentless" harness.

---

Published as a conference paper at ICLR 2026
agents far outperformed fixed-interaction agents across both reasoning- (o3-mini) and non-reasoning models (GPT-40). With a fixed interaction harness, a GPT-4o powered agent achieved pass rate of only 8.3%. When given access to dynamic interaction capability, that jumped to 35.3%, a more than 4.0x improvement. Performance gains for 03-mini were smaller, yet still substantial: 27.8% 33.3%. This highlights the importance of agents being able to dynamically interact with the environment. There may exist hand-crafted agentless-style fixed interaction paradigm that optimizes for this benchmark, but from our investigations it seems that this would be less fruitful than allowing agents to dynamically choose how to interact with their environment.
We found that dynamieally-interr
Part of the explanation for this performance improvement may be due to the effectiveness of de- bugging which can be performed by the dynamic agents. Often, the first attempt crashes (69.4% of the time for agents): it results in non-working code that produces runtime or compilation emror. Agents can consequently debug and produce working code. We find that 29.1% of runs recover to working state and 18.6% of runs end up with the correct working code. However, reasoning-only models operating in the fixed interaction paradigm - without access to multiple debugging steps not able to achieve these gains. ol's crash rate 75.0% and 03-mini's 'sss was 66.7%. These results suggest that debugging may partially explain why the dynamic agent outperforms the fixed decomposition setup on this benchmark. Appendix Appendix[prrovide more detail and examples of common errors that agents face.
50
01 a3-miti
CPT-40
GPT-4o mi
Cloude 35 Sontet
Success Rato
Cloude 3 Sonnet
10
10
512
20111
8110
16:38
10
Max Tokans
Interactions
(a) Performance scaling across Max Reasoning To- kens in the Fixed setting.
(b) Performance scaling across Max Interactions in the Dynamic setting,
Figure 4: Scaling the number of reasoning tokens the fixed-interaction setting (left) does not scale well, in comparison to scaling the number of interactions in the dynamic setting (right).
DeepSeek-Al et al,, 2023) can improve performance with "test-time compute" by, e.g., increasing the max number of tokens they use in their chain-of-thouglee before probing for final answer (Muennighoff et al. 2025). We also investigate whether scaling test-time compute helps on our task. We test this in the fixed-interaction setting, because with only one action taken by the agent, we can easily control the number of reasoning tokens used (as opposed to constraining per-action). We reproduce the experimental method from MMuenighoff al. (2025), in which we constrain the model's reasoning tokens to some fixed number, then ask the model to write code given the reasoning trace. The exact prompts and experimental details are in Appendix !
Another recent interesting result is that reasoning models (OpenAl 2024
The results of our experiment are depicted in Figure 4 Similarly to other reasoning bench- marks (Muennighoff et al.) 2025), scaling max tokens in the fixed-interaction setting improved per- formance, but the increase was moderate. Running the harness with ( reasoning tokens achieves 11.1% and 8.3% for ol and m3-mini respectively, and the performance reaches its peak at 22.2% and 27.8% for ol and o3-mini before tapering off. Although performance does improve with reasoning tokens, there seems to be an upper limit on the performance. On the other hand, scaling the max interactions allowed by the agents significantly outperformed the Fixed setting with performance saturating around 38.9%.
The key takeaways from these experiments that (1) AUTOEXPERIMENT provides testbed for dynamically interactive agents, that isn't 't easily solvable by a fixed, agentless interaction paradigm. (2) while scaling max reasoning tokens improves performance moderately, scaling the number of
7

---

Published as a conference paper at ICLR 2026
steps of agentic interactions has much more substantial effect. We believe these experiments highlight the strength of our benchmark in evaluating interactive research agents.
3.4
DEPENDENCE ON NATURAL LANGUAGE
context
Full context
30
Pass Rate (%)
15
Figure 5: Performance across different values of under different natural language (the text of the paper) retrieval conditions. No retrieval only gives agents access to the code. Full retrieval additionally gives agents access the paper through tool calls described in Appendix [.3 (our default setting for all other experiments). We test using agents with GPT-4o as backbone, ReAct prompting, and Full history management. We find that access the paper becomes more important as more functions are masked out.
Another key finding is the relationship between n and the agent's access to natural language context (relevant text from the paper that guides code generation). To understand the importance of natural language context, we run agents on the No context setting, which removes the research paper con- tents from the agents' inputs. On the contrary, our main setting (Full context) provides the contents of the paper as text file that agents can read from. We found that models were often able to perform fairly well on the n 1 task (where only single function needs to be implemented) without access to the natural language (the No context setting). However, as n increases, performance degrades more rapidly in the No context setting, suggesting that natural language guidance becomes more important as greater proportions of the codebase masked out.
At 1, No and Full context perform similarly with 35.3% pass rate for Full context and 34.1% for No context. This is somewhat surprising since we expect the implementation details to come from the paper. In qualitative error analysis, found that this was because information required to implement some of the functions was often inferrable from other functions in the codebase. Also, the relatively low performance of agents in the Full context setting could be explained by the often- observed phenomenon that language models are distracted by irrelevant context (Creswell et al. 2022). Yet as n increases, natural language context from the paper become more important. We find that performance drops more rapidly in the No context setting such that by 4, performance reaches complete failure. We also manually defined "Oracle" retrievals for relevant passages for each function, but did not see significant improvement to using this over the full context of the paper. Qualitatively, we found that the agents often focused on mimicking the style of the code around the target function(s) more than the natural language from the paper. As such, we found that having the right code context is important. We show results on a similar experiment with code context in Appendix G
4 RELATED WORK
Automating Scientific Discovery It has been long- long-standing dream in AI research ma- chines as tools for new scientific discovery (Simon) 1977] Langley 2000). Recent advancements in generative AI agents have led to progress important capabilities such idea generation (Qi
8

---

Published as a conference paper at ICLR 2026
et al. 2023 Wang al.
2024a). peer review Tyser al. 2024 Liu & Shah 2023 and writing
aids (Altmee et al. [2023
Salvagno et al., 2023). As Al grows more capable of running experiments
well (Lu et al. 2024) effective benchmarking of agents' abilities to experiments becomes all
the more important.
Generative Agents Generative Agents, agents powered by large generative models with access to interactive environments (Xi et al. 2023). have shown promising results on a variety of tasks (Ahn al.] 2022] Huang al. 2023 Some core challenges of generative agents are inherited
from their generative backbones, including long context retrieval, reasoning, planning, and self- reflection (Shinn al. 2023 Madaan et 2023). Other core challenges are novel to the agentic setting, including tool use to interact with their environments and managing long histories of inter- Zhang al. 2024).
actions (Wang et al., 2024c
Benchmarking Code Generation Code-generation benchmarks have traditionally evaluated sys- tems' ability to write code from docstring, evaluated by unit tests (Hendrycks al., [02 Chen it al., 2021). This paradigm has been extended to data science notebooks (Yin et al.) 2022] as well as to patch generation for GitHub issues (Jimenez et al., 2023). Most similar our work are benchmarks focused on testing agents' ability to perform ML research. MLAgentBench (Huang) jet al., 2024) focuses on agents' abilities to improve model performance, but does not test on other research hypotheses. Similarly, MLGym (Nathani et al. 2025) tests agents on AI research tasks involving data processing and model training for better performance. SUPER (Bogin et al. 2024) tests agents reproducing results given the full repository (generating commands, not code). and PaperBench (Starace et al. 2025) tests agents on replicating results from scratch. Concurrent to our work is CesearchCodeeeent (Hua et al. 2025), which tests code implementations from recent research papers. However, none have tested agents' abilities to implement research ideas by mask- ing out varying amounts of the gold standard code. As we have shown, this leads an important knob with which to control the difficulty of the benchmark and the reliance of the model on natural language vs. code context.
CONCLUSION
We introduced AUTOEXPERIMENT a benchmark designed to evaluate Al agents on the task of implementing and running scientific experiments from natural language descriptions in research papers. By varying the proportion functions removed from research repositories, AUTOEXPERI- MENT bridges the gap between reproduction and full replication, offering testbed for autonomous research agents. We demonstrate that agent performance degrades sharply with increased autonomy. that dynamic decision making is critical (e.g.. for debugging), and that multi-tria accuracy outpaces single trial significantly, highlighting importance of verifiers. AUTOEXPERIMENT offers a critical lens on the capabilities and limitations of current frontier models, and provides foundation for developing agents capable of autonomous scientific discovery.
LIMITATIONS
Our benchmark focuses exclusively machine learning papers and Python-based code. It unclear how generalizable our findings are to other scientific domains and programming languages. The dataset is relatively small (85 tasks at n==)) and resource-intensive run, though parallelization is possible. All experiments are executed in Conda environments; while agents may attempt package installation, this shifts the task definition slightly toward open-ended setups such as SUPER (Bogin et al.) 2024 though we did not observe this behavior in practice.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, code and data is open-sourced at this link We share details about our development environment in Section 2 Furthermore, all prompts used in this paper are shared in Appendices F and

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Alex Tianyi Xu for building the compute infrastructure used in this project. We are grateful Yiqing Xie and Saujas Vaduguru for their valuable feedback, and to Alex Hsia for assistance with the initial data collection. This material is based upon work partially supported by the National Institutes of Health (awards R01MH125740, RO1MH132225, and UO1MH136535). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do As Can, Not As Say: Grounding Language in Robotic Affordances, August 2022. URL http::/ org/ /tt.////22022 01691 arXiv:2204.016% [cs].
Signe Altmäe, Alberto Sola-Leyva, and Andres Salumets. Artificial intelligence in scientific writing: friend or foe? Reproductive Biomedicine Online, 47(1):3-9, July 2023. ISSN 1472-6491. doi: 10.1016/j.rbmoo 2023.04.000
Ariful Azad and Afeefa Banu. Publication trends in artificial intelligence conferences: The rise of super prolific authors, 2024. URL/attps://axxi/ org. abs wttpaa////22277999.
Ben Bogin, Kejuan Yang. Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. Super: Evaluating agents on setting up and executing tasks from research repositories, 2024. URL http::ttpp://aatttt www.:orggaaaoe.. htrg/abb/24000 07440
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss. Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code, July 2021. URL http: arXiv:2107.033 [cs].
/arxiiv.crg/ab//21101
Antonia Creswell, Murray Shanahan, and Irina Higgins. dneettoo-inferencce: Exploiting large lan- guage models for interpretable logical reasoning. arXiv preprint arXiv:2205.0097/3 2022.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li. Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang.
10

---

Published as a conference paper at ICLR 2026
R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. I.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui L.,, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. URL httphttss://arat
absg///b//// org/ abs /201 12948
B. Efron and R.J. Tibshirani. An Introduction to the Bootstrap. Chapman Hall/CRC Monographs Statistics & Applied Probability. Taylor & Francis, 1994. ISBN 978-0-412-04231-7, URL
https::/bbole http://oooos.oooe.eeeeoeooe.e
google www.com//oooooo
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring Coding Chal- lenge Competence With APPS, November 2021. URL http::htt::aatt wtwp://w..rrr/a///////a// 09938 arXiv:2105.09938 [cs].
Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber. Researchcodebench: Benchmarking ilms on implementing novel machine learning research code, URL httppppp//ax.....//////....//// 25cp//66./250..... 02314
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Benchmarking Large Language Models As Al Research Agents, 2023. _eprint: 2310.03302.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation, April 2024. URL bttp://:://aaxt .rg/ abs/ arXiv:2310.0330 [cs].
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023. .eprint: 2310.06770.
PAT Langley.
The computational support of scientific discovery. International Journal of Human- Human-Computer Computer Studies, 53(3):393 410, September 2000. ISSN 1071-5819, doi: 10.1006/
2hc.2200.03380 URL https: wttp://wwwww/m www wttp://wwwwwiieeeetecccoo wwtp:ec..ooooccco.iiiocc
S107 900903963
Ryan Liu and Nihar B. Shah. ReviewerGPT? An Exploratory Study Using Large Language Models for Paper Reviewing, June 2023. URL Akiv:2306.006222 [cs].
wwtp://..rrraa/////////// 00622
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, August 2024. URL
hhttss://arxti www.xxiv.org//b....e orgooog///aoooo /2002 06292v3
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder. Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- Refine: Iterative Refinement with Self-Feedback, May 2023. URL attp: http://axxvvvoxvo 2230.3 17651 arXiv:2303.199 17651 [es].

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsumori Hashimoto. s1: Simple test-time scaling, 2025.
abs /tt.//51..1110
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vin- cent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. Mlgym: A new framework and benchmark for ad- vancing ai research agents, 2025. h
https: /ttp///axivvvv /ttp://wwwwww//////////2...... 14499
OpenAl. Learning to reason with 1ms, September 2024. URL
learning-ton to reason with 11ms/
Accessed: 2025-05-12.
Biqing Qi. Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. Large Language Models are Zero Shot Hypothesis Proposers, November 2023. URL http: /ttt::/aai /ttp://axvvv.../ wttpaaaww...aaaaa///a/aaaaaaaa/a www.ooggg///////o/////..oo..ee/o 2211 .5965 arXiv:2311.0599 [es].
Nicolas P. Rougier, Konrad Hinsen, Frédéric Alexandre, Thomas Arildsen, Lorena Barba, Fa- bien C. Y. Benureau, C. Titus Brown, Pierre de Buyl, Ozan Caglayan, Andrew P. Davison, Mare André Delsuc, Georgios Detorakis, Alexandra K. Diem, Damien Drix, Pierre Enel, Benoît Girard, Olivia Guest, Matt G. Hall, Rafael Neto Henriques, Xavier Hinaut, Kamil S. Jaron, Mehdi Khamassi, Almar Klein, Tinn Manninen, Pietro Marchesi, Dan McGlinn, Christoph Metzner, Owen L. Petchey, Hans Ekkehard Plesser, Timothée Poisot, Karthik Ram, Yoav Ram, Etienne Roesch, Cyrille Rossant, Vahid Rostami, Aaron Shifman, Joseph Stachelek, Marcel Stimberg, Frank Stollmeier, Federico Vaggi, Guillaume Viejo, Julien Vitay, Anya Vostinar, Roman Yur- chak, and Tiziano Zito. Sustainable computational science: the ReScience initiative. PeerJ Com- puter Science, 3:2142. December 2017. ISSN 2376-5992. doi: 10.7717/pecne 142. URL http:://rxii wwtp:/orggabbb/ooooe 04393 arXiv:1707.0433 [cs].
Michele Salvagno, Fabio Silvio Taccone, and Alberto Giovanni Gerli. Can artificial intelligence help for scientific writing? Critical Care, 2751):75, February 2023. ISSN 1364-8535, doi: 10. 18.1 org 11l1111111110 1185 023 04380-2
$130544-023-43380-2 URL/https:/
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models, 2024. URL
https: www./bo///////oooo
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shumyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, October 2023.
URL
abs/2303. 11366 arXiv:2303.1 1366 [cs].
Herbert A. Simon. Scientific Discovery and the Psychology of Problem Solving. In Her- bert A. Simon (ed.), Models of Discovery: And Other Topics in the Methods of Science, pp. 286-303. Springer Netherlands, Dordrecht, 1977. ISBN 978-94-010-9521-1. doi: 10. 978- 0188----10--52 URL hat.::///08...rr0000.00077733333
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lan Chao, and Yu Su. I.LM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models, March 2023. WtLbttt::// http///aaxiv.v.... org/ abs 2212.22222222 04088 arXiv:2212.04088 [cs].
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's ability replicate ai research, 2025. URL
www.ooggg/bso.. 2504 01848
Keith Tyser, Ben Segev, Gaston Longhitano, Xin- Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, Dov Te' eni, and Iddo Drori. AI-Driven Review Systems: Evaluating LLMs in Scalable and Bwas-Aware Academic Reviews, August 2024, URL arXiv:2408. 10365 [cs].
http::/a httpaaiiww.rggab///////ooo /240 1000088810380000 10365
12

---

Published as a conference paper at ICLR 2026
Qingyun Wang. Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific Inspiration Ma- chines Optimized for Novelty, June 2024a. /R/.ttpp//ax.....cyaa////// 14259 MrXiv:2305.14255 [es].
Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. Beyond the limits: A survey of techniques to extend the context length in large language models, 2024b. URL attps:/aaxxv..rr//aapp.
02244
Zhiruo Wang. Daniel Fried, and Graham Neubig. Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks, 2024c.
hwtp:///ggggaa/aaoooooo.. abs/ /t.p///4401 2401 12869
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong. Ming Zhang, Jun- zhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang. Limao Xiong. Yuhao Zhou, Weiran Wang, Changhao Jiang. Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuan- jing Huang, and Tao Gui. The Rise and Potential of Large Language Model Based Agents: A Survey, September 2023. [cs].
http///awww....//////////// 07864 arXiv:2309.07864
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying 1lm- based software engineering agents, 2024. URL
abs/2401 .144477..8449
01489
Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. Benchmark data contamination of large language models: survey, 2024. URL
wwwp:iiiioorrooooooooooo abs wwwpab/ee.22...o 04244
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineer- ing, May 2024.
15793 M.Kiv:2405.15999 [cs].
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https::/arxiv:. org/abs/ 2210 03629
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua How- land, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sutton. Natural Language Code Generation in Interactive Data Science Notebooks, December 2022.
URL httphttpp/axi...ooo/ org. / /aas//2/1...... 09248 arXiv:2212.09248 [cs].
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö. Arik. Chain of agents: Large language models collaborating on long-context tasks, 2024. URL https:
//arxiv wwwpxiiwwrrr//b//20......
02818
13

---

Published as a conference paper at ICLR 2026
A PAPERS
Papers
Repositories
Functions
ITI-GEN:
Inclusive Texl-to-lmage Generation
Link
23
Studying How to Efficiently and Effectively Guide Models with planations
Link
33
Cartoon Explanations Image Classifiers
Link
14
Joint Multisided Exposure Fairnsss for Recommendation
Link
15
Table 4: Papers in AUTOEXPERIMENT.
All papers submitted to the ML Reproducibility challenge. which require papers and their associated code repositories to be licensed under Creative Commons 4.0 License. This allows for copying, modifying, and sharing of the original work.
B CURTAILED TEST CASES
Metric
Recall
Precision
Unweighted Weighted
0.949
0.959
0.922
0.959
Table 5: Precision and Recall of using curtailed testcases success metric with respect full testcases.
We run the setting with GPT-4o with the curtailed experiments and the full experiments and observe how effective the curtailed experiments are as a proxy for the test case defined by the full experiments. On run with 92, 49 correct and 43 incorrect, curtailed and full experiments, and find that both the precision and recall are very high, meaning that the curtailed testcases are effective proxy for the full experiments. The curtailed experiments took significantly less time and compute resources to run, making the benchmark feasible for large scale testing.
compared the decisions from the
C AGENT ARCHITECTURES & BACKBONE MODELS
GPT-4o
GPT-4o-mini
Full
Sm (5)
SI (5)
SI (1)
Full
Sm (5)
SI (5)
SI (1)
ReAct
38.9
27.0
32.4
27.0
24.3
29.7
21.6
18.9
MLAgentBench
29.7
29.7
24.3
32.4
8.1
21.6
13.5
24.3
Planning-Only 18.9
18.9
18.9
16.2
18.9
24.3
21.6
27.0
Table Success Rates for different prompting strategies and history management combinations.
Many works Yao et al. 2023 Song et al. 2023 Huang et al. 2024 Yang et al. 2024 have inves- tigated prompting frameworks for agents designed elicit planning, action, and reflection capabil- ities. These frameworks are often tightly coupled with how to manage the history of interactions that gets passed the agent's context window, especially as the number interaction steps with the environment grows. The prompting frameworks investigate
1. ReAct Yao et al. 2023 prompts for natural language thought and the next action.
2. MLAgentBench (Huang et al. 2024): prompts for a highly structured response including thought, research plan, and reflection in addition to the next action.
14

---

Published as a conference paper at ICLR 2026
3. Planning-Oni (Song et al. 2023 prompts for step-by-ster plan on how to achieve the final goal and the current status this plan in addition to the next action.
We investigate these across different approaches to managing the history of interactions, including
1. Full, which passes the entire history of interactions,
2. Sliding Window - SI (k) which passes only the most recent & interactions
3. Summary Sm (k), which passes the most recent interactions well summary of older interactions.
We find that the choice of prompting technique and memory management strategy impacts agent per- formance. Among the evaluated methods, ReAct outperforms both Planning and MLAgentBench across most configurations, particularly with the GPT-4o model, achieving success rate of 38.9% with full memory. Notably, full memory management yields the best results with ReAct, though this advantage is contingent the prompting approach and the model's capacity for handling long context (c.g., GPT-4o-min mini struggles with longer context and benefits from summarization instcad of full context with ReAct). For Planning prompts, which typically require longer context windows, summarization strategies (Sm and SI) are essecall- for smaller models like GPT-4o-mini. Overall, it seems that jointly optimizing prompting and memory frameworks may be important to maximize performance given model's capacity for long-context reasoning. We use the paired bootstrap test (Efron & Tibshirani 1994) to compare the performance of the best agent combination (ReAct + Full + GPT-4o) with the worst IMLAgenBencch + Full + GPT-4o mini) and find statis- tically significant difference (p-value 0.004). However, the best agent compared with the 2nd best (React + SI(5) + GPT-4o) yields p-value of 0.243. This warrants further investigation into optimizing agent architectures, which we leave for future work.
D ERROR ANALYSIS
We analyzed agent trajectories to understand the common failure modes AutoFxperiment. For our main agent (ReAct prompting and full memory with GPT-4o backbone on n=l setting), 35.3% of runs resulted in the correct answer, 31.4% produced an incorrect answer, and 33.3% did not produce an answer all.
For instances in which the agent produced incorrect answer, we found that the agent wrote code that compiled and ran without errors, but had implementation flaws that do not reflect the natural paper describes novel metric function with
language description in the paper. For instance, a precise formula, which is to be implemented in the missing function. The agent often fails to retrieve this information from the full paper text and relies solely code context write the missing function. This leads to the implementation having operations that do not match the correct formula.
In the runs that failed to produce answer, agents failed to debug their code as agents often pro- duced code that was incompatible with the rest of the code base. For instance, agents would use variables or call functions that are not defined, or misunderstand the type of variables.
E EVALUATION SAMPLES FOR HIGHER VALUES OF
5
Number of samples 94
98
100
100
Table 7: Number of samples used for evaluation for 2 2.
Because the number of possible samples increases exponentially for higher values of n, we test a subset for each n. The samples were chosen at random, and the number of samples for each n are listed above.
15

---

Published as a conference paper at ICLR 2026
F VERIFIER PROMPT
You are expert software engineer. You are given various code snippets that generated by an LLM that implement function {function_ange J according to dunction_docstring }. Your job is to verify the correctness of the code and pick the best candidate. Here are the code snippets:
Code 1:
{ [contents of code 1}
Code 2: { contents of code 2}
Code 5:
{ contents of code 5}
Please pick the best candidate and return the name of the code snippet. Explain your reasoning.
G CODE CONTEXT RETRIEVAL RESULTS
No
Full
AST
Embedding
13.9
36.1
33.3
41.7
Table 8: Pass rate of agents with No retrieval, Full (All code in context window), AST-based, and Embedding--asee code retrieval
We observe that embedding- retrieval significantly outperforms both in-context (Agent-led) retrieval and other retrieval strategies (Table 8). Agents using embedding-base retrieval achieve pass rate 41.7%, compared 36.1 with in-context retrieval and 33.3% with AST-based retrieval. Simply placing all relevant code into the context window degrades performance, underscoring the limitations of context window-based reasoning. This result highlights code retrieval as a central challenge in our benchmark and key area for future research.
H AGENT-LESS EXPERIMENT
We follow the Agentless approach for SWE-Bench (Xia et al. 2024), which consists of localization, repair, and patch validation steps. One key difference that our task does not require localization since the location of the missing function is given. However, in order to write the masked-out function, retrieval relevant natural language context (from research paper) and code context (rest of repository) is necessary. As such, our "agentless" harness consists of the following steps:
1. Natural language retrieval: we divide the research paper contents into paragraphs and use an embedding based retrieval method using the function header and docstring as query to retrieve top-k relevant text snippets. We use OpenAl's text-embeddingg- -aaiing----mmala 3-slall model to compute embeddings.
2. Code retrieval: LLM-based prompting method is used narrow down the relevant files in the repository. Then, we embed each code snippet and use function header and docstring as query retrieve top-k relevant code snippets. We use OpenAl's text-embedding- 3-small model to compute embeddings.
3. Code infilling: this step is performed by prompting the LLM to fill in the masked-out function using natural language and code context from the retrieval steps. We reproduce
16

---

Published as a conference paper at ICLR 2026
the approach from Muennighoff al. (2025), in which first prompt for reasoning, then prompt for the code implementation. More details of how we can control the number of reasoning tokens in Appendix
4.
Validation and Extraction: we evaluate the generated code using the provided experi- ments (testcases) and extracts the final results.
H.1 CODE RETRIEVAL FILE LOCALIZATION PROMPT
Given repository structure, return top file paths related to the code. Repository structure
{ repository.structue
Code:
{ function _eadeer_eaeeeee _ndd_ doestring
Return only new line-separane list of paths like this
J relative /path/to/ file1
J relative /path/to/ file2
J relative /path/to/ file10
H.2
CODE INFILLING PROMPT
Reasoning prompt
You are helpful coding assistant You are given contents of a Python file with one missing function. Paper context contains snippets a research paper that describes how to implement the code. Code context contains code snippets that are similar to the missing function.
### Paper Context ###
{ paper context] }
#*# Code Context ### { [coee_come J
#### File Content ###
python { file_content }
Think about how you want to implement the missing Python function.
Completion prompt
### Thought ### {thought)
### File Content ### python { file_content }
### Python function ###
python
function content
17

---

Published as a conference paper at ICLR 2026
Give only the missing function implementation. Provide only the code in markdown format. I.e. "" "'python
I TEST TIME COMPUTE EXPERIMENTS
We reproduce the experimental setting from Muennighoff al. (2025), in which control the number of reasoning tokens. In the code infilling step (see Appendix[H] H we first prompt for reason- ing followed by another prompt perform the code infilling. When the reasoning step prematurely terminates before reaching the desired token count, we append the "Wait" token to the response, foreing the model to reason for controlled number of tokens. In this experiment, we report the success rate of the agentless harness with ol and 03-mini as model backbones and maximum tokens ranging from [0, 16384).
Reasoning Models
512
2048
4096
8192
16384
11.1
8.3
19.4
13.8
22.2
19.4
03-mini
8.3
5.6
13.8
25.0
27.8
25.0
Table 9: Scaling number of reasoning tokens in the Fixed setting.
AGENT ARCHITECTURE DETAILS
J.1 INITIAL PROMPT
### Setting
You research assistant that is tasked with running experiments produce results for a scientific paper.
The directory already contains some code that implements the experiments done in the paper and the environment is already set up. But the implementation is implemented yet.
incomplete in that there are functions
You can use the following tools to interact with the environment. ### Tools { tools)}
Your task is to write the missing functions the code and running 'bash refsol. sh' to obtain experiment results Here are the experiments you need to report: (experiment details
J.2 PROMPTING STRATEGIES
ReAct
Think about what action to perform next.
MLAgenBBench
Always respond in this format exactly: Reflection: What does the observation mean? If there an error, what caused the error and how to debug?
18

---

Published as a conference paper at ICLR 2026
Research Plan and Status: The full high level research plan, with current status and confirmed results of each step briefly annotated. must only include progress that has been made by previous steps. If there is any update. enclose the new update text in double asterisks **like this *e. there update, just copy the previous step Research Plan and Status The high level plan from the previous step should be fully retained, unless it is intentionally revised.
Fact Check: List all objective statements in the updates to Research Plan and Status one by one and point out whether it guessed versus directly confirmed by the previous observation directly above. Performance numbers can only be confirmed by running the code and observing the output. Thought: What you are currently doing, what actions to perform and why
Planning-Only
Create high level plan with current status and confirmed results
J.3 TOOL CALLS
We follow standard tool set for coding agents inspired from ML.AgentBench (Huang et al. 2024) and SWE-Agent (Yang et al. (024). summary, the agent can navigate the repository, read- /write/edit files, and execute Python bash scripts.
tool prompt
"type": "function",
"function": "name"; "final_answer",
"description"; "Use this to submit the final answer to the 4 current task", "required"; ("final_answer") "parameters":
"type": "object",
"properties":
"final answer";
"type": "string",
"description": "json format string representing + dicticnary of the final answer"
}
"type": "function",
"function":
"name": "understand_file"
"description"; "Use this to read the whole file and
+ understana certain aspects. You can provide a
+ detailed description on what to look for and what
4 should be returned.",
"required;; ["Eile_name"), "parameters";
"type": "object",
"properties":
"file_name",
"type": "string",
"description": valid file name with relative 4 path to current directory if needed",
h "things_to took__ooooooo
"tupe": "string",
19

---

Published as a conference paper at ICLR 2026
"description": detailed description what to look for and what should returned"
b
"type": "Lunction",
" "function":
"name": dinspect_file. fil_e_leeene,,
"description"; "use this to inspect specific part of a file 4 precisely, or the full content for short files.", "required": "file_name") "parameters";
"type": "object",
"proporties";
"file_name";
"type": "string",
"description": valid python script name with relative path to current directory if 9 needed"
b
"start line number":
"type": "number",
"description": "a valid line number"
b,
"end_line_number::
"type": "number",
"description": valid line number"
b
"type": "function",
"function":
"name": "edit_file" "description": "Use this to do a relatively large but S cohesive edit over python script. Instead of
+ editing the script directly, you should describe the edit instruction so that another can help you
4 this.
"required"; ["file_name*, "edit indit_instrucciions] "parameters";
"type": "object",
"properties";
"file_name"
"type": "string",
"description": *a valid file name with relative path to current directory needed. empty file will be created if does not exist."
b "edit_ instittiontttttinn {
"type": "string",
"doscription": "a detailed stop by stop + dcscription on how to edit it."
"save_ "save_name":
"type": "string",
"description": "a valid sile name with relative S path to current directory it needed"
20

---

Published as a conference paper at ICLR 2026
"type": "function",
"function":
"name": "write_file"
"description"; "Use this to write content to a file. IE the 4 file not exist, new file will created. If file exists, content will overriden", "required" ["file_name", "content"), "parameters";
"type": "object",
"properties":
"file_ "Eile_name":
"type"; "string",
"description": "a valid file name with relative path to current directory if needed"
b, "content"; "type": "string",
"description": "the content to be written to the file"
"type": "function",
"function";
"name": rexcatt__python_crride
"description"; "Use this to execute the python script. The 4 script must already exist.", "required": ["file_name"), " "parameters";
"type": "object",
"properties";
"file "file_name"; nwm.n:mmmm.....
"type": "string",
"description"; " valid python script name with relative path current directory if needed"
h "arguments": "type": "string",
"description": "command line arguments to use if 4 needed"
"type": "function", " "function":
"name"; "cxccute_bace "aststt__t
"description"; "Use this to exccute bash script. The + script must already exist", "required": ["file_name"), "parameters"; { "type": "object", "properties";
"tile name":
21

---

Published as a conference paper at ICLR 2026
"type": "string", "description": valid bash script with relative + path to current directory"
"arguments":
"type": "string",
"description": "command line arguments to use Y needed"
"type": "function",
"funation":
"namo"; "conmand_line",
"description"; "Use this to run any 1inux command 1ine
+ command",
"required"; ("command"), "parameters";
"type": "object",
"properties":
"command";
"type": "string", "description": "valid linux command line command"
b
"type": "function", "function":
"name": "list_files", "description". "Use this to 1ist files in rectory"" "required". ["directory"), "parameters";
"type": "object",
"properties":
"directory"; "type"; "string", "description": "valid path to directory"
"type": "function", "furction":
"name": "move",
"description": "use this to move a file or directory from S source to destination. You can alsc rename files + with this function", " "required": ["source", "destination"), "parameters";
"typo": "object",
"propertics";
"source": "type": "string", "description": "valid path to file or directery"
"destination";
"type": "string",
22

---

Published as a conference paper at ICLR 2026
"description": "valid path to file directery"
b
"type": "function",
"Eunction":
"name": rchange_directonn
"description"; "Use this to navigate the file structure", "required": ["direstory"], "parameters":
"type": "object",
"properties":
"diroctory";
"type": "string", "description": "valid path to directory"
K
COMPUTE RESOURCES
Model Backbone
GPT-4o
GPT-4o-mini
Claude 3.5 Sonnet
Claude 3.7 Sonnet
Compute Cost (S)
0.59
0.15
0.79
0.90
Compute Time (min)
12.7
23.9
9.31
6.67
Table 10: Average Compute Cost and Time of a single sample with model backhones.
L EXAMPLE TRAJECTORY
We show example of a full trajectory of agent with ReAct prompting strategy and Full memory management using GPT-44 as backbone.
Initial Prompt
### Setting
You are a research assistant that is tasked with running experiments to produce results for scientific paper.
The directory already contains some code that implements the experiments done in the paper and the environment already up. But the implementation incomplete in that there are functions not implemented yet.
You can use the following tools to interact with the environment.
### Tools
final : Use this to submit the final answer to the current task command line: Use this to run linux command
inspect_ wil.lllleeeeeeseeeeeeeeeeiileeee Use this to inspect specific part of file precisely, the full content for short files
execute bash script Use this to execute a bash script The script must already exist
23

---

Published as a conference paper at ICLR 2026
Your task is to write the missing functions the code and running 'bash refsol sh to obtain experiment results
Here are the experiments you need to report:
Experiment I: Train model VOC2007 dataset with bcos the model backhone,
BCos attribution method, RRR localization loss, and optimizing explanations Return the IoU after training for epoch in this format {"IoU": 0.03. Replace 0.0 with the actual value
Experiment Train a model on VOC2007 dataset with bcos as the model backbone, GradCam attribution method, RRR localization loss, and optimizing explanations Return the IoU after training for epoch in this format {"IoU ": 0.0). Replace 0.0 with the actual value.
Experiment Train a model on VOC2007 dataset with bcos as the model backbone, IxG attribution method, RRR localization loss, and optimizing explanations Return the loU after training for epoch in this format ("loU": 0.0). Replace 0.0 with the actual value
Return final answer as a json: ("Experiment 1": "Experiment 2":
Missing function (s): c.all. class RRRBBMulippleLoss file losses py
Tips
First, reference the contents of the paper. txt and extract relevant information order to in the missing functions.
Running 'bash refsol sh will run all the experiments.
Step 1
Thought:
I will first check the files present in the directory to gather information the existing code for the implementation and the missing functions. This will help us locate any code files including "losses py which contains the missing __call__ function. Once identified, I can reference the 'paper. txt extract relevant information to implement this function. Let's start by listing the files in the directory.
Action:
munction(argumentte directory """"', name= list_files )
Observation:
README.md pitributionmmethone -py
test bbox size epg py
train -py
train_energy -py
train seg .p
wain_waterbbire -py
Step 2
Thought:
Since the 'paper. txt file is crucial for understanding the scientific context and any descriptions that can help implement the missing function, Il start by reading it to gain insights This can provide with the necessary information or theoretical background required to complete the implementation of the "_all.__ function in losses pyy Let's s proceed by reading the paper. txt
24

---

Published as a conference paper at ICLR 2026
Action:
Function (argttinn " ttaiummetttt"""ilmmmniii "ilmeenmme""""" pappe.. txt "Y, name=' read_file ')
Observation:
# Introduction
Deep neural networks (DNNs) excel at learning predictive features that allow them correctly classify set of training images with ease.
RRR= introduced the RRR loss regularize the normalized input gradients $ hat(A))(k,,w/ $$ mathhal(L)) [eex((loc).k] = |sum ( (um_ (1-M (k,hw)) (hat(A] (k,hw) 1.8$
To extend it to our setting. we take (\har(A)-(kkhr) to be given by arbitrary attribution method (); we denote this generalized version by RRRs. contrast to the loss only regularizes attributions *outside* the ground truth masks. While it thus does not introduce a uniformity prior similar to the loss, it also does not explicitly promote high importance attributions inside the masks.
Step 3
Thought:
The "paper. txt' contains detailed information on various loss functions. including the RRR localization loss, and provides mathematical basis and context for the experiments involving attribution methods for training models. Specifically it mentions various localization losses such as LI distance, PPCE, RRR, and the EPG-derived loss, and explains their application training models.
Given this information, the next step would be to examine 'losses ppy file to understand the current structure and how to integrate the found information, especially since the missing function -_.call_ likely needs to implement one of these described loss functions understanding the overall file structure we can identify where and how to write the "_alll" function, guided by the localization losses mentioned. Let's s now inspect the Jossee.py file to analyze where exactly to implement the missing function.
combination thereof. By
Action:
Function (argunmntsnuummts"ss"r name losses .p")', name=' read file ')
Observation:
class RRRBMMaiipodoss/BMMiiipppp..ss def .init.. O: super __iiit__
self. only positivos = False
self. binarize True
def _.cal__ O:
Calculates the RRR (Remove and Retrain) localization loss given attribution scores and bounding box coordinates.
:param attributions A tensor
:param bb.coordinates : tensor
return A scalar tensor
25

---

Published as a conference paper at ICLR 2026
raise NolmplemenntedEnoo))
Step 4
Thought:
The losses py" file contains several classes implementing different localization losses These include classes for Energy (EPG-based) loss, LI loss, and PPCE loss. There is placeholder for the 'RRRBBMultppelLoss class, which likely needs to be filled in with the RRR localization loss logic based on the insight gained from "paper. txt
To proceed, should implement the missing -_call_ method for the RRRBBMultipleLoss class. The "paper. ixt describes the RRR loss as a Remove and Retrain" loss calculated with normalized input gradients within the mask.
The next step is to fill in this function within the 'RRRBBMulipllLLos' class. Let's proceed with implementing the missing function based the information extracted earlier.
Action:
Function (arguments= ("ille name ":" losses pp""." edit instruutttt "." """Implemen the ,.call_.. method for the RRRBBMulipleeoss class, using information from paper txt. The function computes the RRR loss follows:) (n) nn- Calculate binary mask with bounding box coordinates using get bb maskO.)) Normalize the attributions between and 1. Compute the sum of squared normalized attributions outside the bounding boxes masked by 1-M. n- Return this summed value.) \ \nn) !nHere is the function :\ (n)) nclass RRRBBMultipleLoss BBMultipleLoss))) in def c.all... self, attributions bb. coordinates ):} \n # Create binary mask based on bounding box coordinates! | \n coordinates, attributions .shape)
bb mask self geb..bbbbb.b..bb... _t_._bb___(_______ berebbb_cooooe Normalize attributions between and 1i attributions attributions .max()) !
#
pormalized_attributiie
# Compute the RRR loss based on loss = ((1
the squared attributions outside of the bounding boxi/n bb mask) normalized attributions ** 2).sum())) in save name":" losses "y" name=' edit file ')
return loss \n"."
Observation:
The edited file is saved to losses py. Here diff, please check if the edit is correct and desirable:
+++
raise NotlmplementedEFror))
def
calll.... (self, attributions bb coordinatess) ):
+
# Create binary mask based on bounding box coordinates bb mask = self -geg bb mask( bb coordinates, attributions .shape)
+
# Normalize attributions between 0 and vormalized_attribution # Compute the RRR loss based on the squared attributions outside of the bounding box
= attributions attributions . max()
loss ((1 bb mask) normalized attributions
2) 2)mm)))
+
return loss
Step 5
Thought:
26

---

Published as a conference paper at ICLR 2026
The -cal_ method for the 'RRREBMultpleLoss' class has been implemented successfully. The next step is to run the experiments as specified by executing the bash script refsol This script seems to be responsible for running the experiments and generating results for the specified configurations Let's s proceed by executing the refsol sh* script to obtain the experiment results..
Action:
Function (argummtttt" (""lemmmmtttt"i___mmmmme" "." refsol .sh ")', name=' execute beshstee ')
Observation:
loss ((1 bb bb_mask) * RuntimeError: Expected all tensors to be the same device, but found least two devices, cuda:0 and cpu!
** 2) sum))
Step 6
Thought:
The error indicates that there device mismatch: some tensors are on the CPU while others are on the GPU, To fix this I need to ensure all tensors are on the same device, typically the GPU if CUDA is available. Specifically for the RRR loss, both the normalized.cc aatributiis and the "bb_mask" tensors need to be moved the same device.
I will modify the _call_ method in the 'RRRBBMultplel..os class to explicitly move both tensors to the same device before computing the loss. Let's proceed with this correction.
Action:
Function Function (arguments= f"ilee name " losses py" edit instrrccttt ":" Update the __call__ method the RRRBBMultipieLoss class to ensure that both the the same device.\)n))pi Before computing the
attributions and bb_ bb_mask loss, move bb mask to the device of attributions using "bb_mask.too( attributions devvcce) '\ nn2. Update the loss calculation line use this consistent device. ||n) (nThis should resolve the runtime error with device mismatch." "" save_nanee "awe""ame"." " "", name= edit_file
Observation:
The edited file is saved to losses .py. Here is the diff, please check if the edit correct and desirable:
@ @ -104,6+104,8 @ @
def ...aal... self, attributions bb cwordinatesss ):
# Create binary mask based on bounding box coordinates
bb mbusmas== self get bb mask( bb coordinates, attributions .shape)
# Ensure bb mask is on the same device as bb_mask= bb_mask.tof attributions device) # Normalize attributions between 0 and = attributions attributions .ma())
attributions
# Compute the RRR loss based on the squared attributions outside of the bounding box
Step 7
Thought:
27

---

Published as a conference paper at ICLR 2026
Now that the device mismatch issue in the code is resolved, the next logical step to re-rum the refsol ,hh script. This will execute the experiments again and hopefully, without any issues related to tensor device mismatches. This should yield the loU results for each experiment as described in the task instructions Let's proceed by running the script again.
Action:
Tunction(argumennt f"ille ""m.eeeaame""""""""""""" name":" refsol sh " name=' execute execute.bash.sery scrit
Observation:
Experiment 1
Train data size: 10
Val data size: 100
Test data size: 100
Annotated: 10, Total: 10
Epoch: O, Average Loss: 131.628 1316128551757825
Validation Metrics: ('Accuracy': 0.921999990940099, Precision " nan. 'Recall ':
0.0, 'F-Score": 0.0, 'True Positives 0.0, 'True Negatives': 1844.0, 'False
Positives 0.0, 'False Negatives 156.0, 'BB-Loc': 0124883055498256777,,,
BB-IoU': 0.0094418611755977665, 'Average-Loss' 0.58850425720214853
Experiment 2
Step 8
Thought:
The experiments have been executed successfully. and we have obtained the IoU values for each of the three experiments. These values are present in the validation metries under the key 'BB-IoU', The next step to prepare the final results in the required format:
1. * Experiment (BCos attribution method):** Extract 'BB-IoU':
0.000941861 1755977625
2. ** E=Exerimment (GradCam attribution method):* Extract 'BB-IoU':
0.09110355844335622
3. Expperimentt (IxG attribution method):*s Extract 'BB-IoU':
0.00130122330066170555
I will now format these results into a JSON structure and present as the final answer.
Action:
Function (arroounnrrrummnssss"""aaa final mnswer":"!! "Experimet 1 1\)": (T""Uo"))":
0.0009418611155597722 "Exxerrmmnn 2\\" (l!")oU) !": 0.091 0,091100355844335522)) 0.001300126300066/0016666 nama= somme='finalamass aanwer))
( ["Experiment 3\\": (1|""oU \"ooU\ 1
Observation:
("Experiment 1": ("IoU": 1.0094186611555777663 "Experiment 2": ("IoU":
0.091 103558043335)))) "Experiment 3": ("IoU": 0.0013012633506115566666
28


Reviews:
[{"review_id": "oKTG0kobJS", "reviewer": "Reviewer_i5wK", "summary": "This paper propose a new benchmark——AUTOEXPERIMENT，for evaluating the ability of agents to implement ml experiments. Tasks are created by masking out various numbers of functions from ML research repo codebases and requiring agents to fill in the blanks and match published results. The paper also evaluates several LLMs, presenting detailed analyses of different aspect.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a new benchmark for evaluating the performance of agents on machine learning code replication tasks.\n2. It evaluates the performance of different models and agents under varying task settings (i.e., partial code replication and full replication).", "weaknesses": "1. Novelty of the Benchmark: The benchmark is presented in two settings: partial code replication and full code replication. For full replication, a significant body of related research already exists. The partial replication setting closely resembles many existing code-filling tasks studied in the context of code models.\n2. Rigorousness of Experimental Setup: The paper uses the number of masked functions (n) as its primary analysis target and difficulty metric, concluding that a larger 'n' decreases the likelihood of successful replication. This conclusion is trivial and has already been well-established in numerous code-filling tasks. In practice, different functions vary dramatically in logical complexity, code length, and library dependencies. Using 'n' as a universal measure of difficulty is overly simplistic.\n3. Lack of Diverse Experimental Subjects: The paper's main results are derived from models like GPT-4o and Claude-3.7, which have known capability limitations. As of 2025, many more powerful reasoning models have emerged that demonstrate superior performance on software engineering tasks, but this paper fails to evaluate them. Moreover, the agent setup itself is rudimentary, employing only a simple tool-using agent designed by the authors. This setup does not reflect the current state-of-the-art in agent capabilities and appears to be testing the backbone model more than the agent's autonomous abilities.\n4. Reasonableness of the Benchmark Design: The paper states that with an increasing number of masked functions, the 85 samples from just 4 papers can lead to 275,990 possible combinations for n=5. The methodology of sampling only 100 of these combinations offers no guarantee that these samples are representative of that difficulty level. The design of this benchmark appears to be sloppy and lacks statistical rigor, as this sparse sampling introduces massive variance, making comparisons between different 'n' levels unreliable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNVHf76uDc", "reviewer": "Reviewer_VcpW", "summary": "The authors propose a new benchmark, AutoExperiment, that tests if AI agents can implement ML experiments described in research papers. The benchmark is based on masking out functions for the agent to fill in (scalable by # functions masked). They evaluate various agents and find performance degrades with # fns masked, and that self-reflection helps.", "questions": "See Weaknesses. Also:\n - In what way is \"scientific\" coding different to general coding, or is it the same thing? Are there some specific characteristics of coding that your benchmark captures that other coding benchmarks do not?\n - Is it always a clear, single gold answer to the experiment? In particular, NL descriptions of an experiment are typically underspecified, meaning there might be several ways of implementing an experiment with different design decisions (e.g., choice of hyperparameters), and correspondingly different results. \n - Filling in missing functions seems a somewhat artificial context - in reality, the agent simply has to code up an experiment. Do you think scores on filling in missing functions correlate with scores on coding up an experiment from scratch (the metric we ultimately care about)? Or do your experiments substantiate that correlation?\n - Your scores on AutoExperiment show models still struggle (~< 50%), but other coding benchmark tests have shown agents are strong at coding (e.g., HumanEval, SWE-Bench). How do you explain this apparent inconsistency? \n\nMinor:\n - Expand the MLRC acronym (Section 2) at first mention, not later.\n - I'd prefer the Related Work earlier (Section 2) rather than an afterthought at the end, given the considerable related work in this area", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Appropriate and opportune benchmark - scientific coding is rapidly becoming a major research area, and we still lack good benchmarks\n - Clear and well written\n - Builds on prior work (MLRC) nicely\n - Nicely scalable levels of difficulty in the benchmark", "weaknesses": "- Most significantly, this paper seems very similar to \"ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code\" (https://arxiv.org/abs/2506.02314), significantly weakening the paper's claims of novelty. Please describe the key differences/benefits of your work to this prior paper.\n - The stated findings from the experiments seem obvious (e.g., performance degrades with more masking; debugging helps). What surprising/discoveries did you make in your experiments? What did you learn that was not obvious before? The most significant take-away to me is that agents still struggle with this kind of coding (e.g., most scores are < 50%) and more work needs to be done.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCpNlB3iEx", "reviewer": "Reviewer_4DjD", "summary": "The paper introduces \"AutoExperiment\", a benchmark for evaluating AI agents for conducting scientific replication tasks. The benchmark consists of 85 functions that AI agents must \"fill in\" across 4 machine learning papers. The ML papers are drawn from MLRC. The benchmark is evaluated using ReAct (though other strategies are tried and work more poorly) across a number of closed and open-weight AI models. The authors draw compelling insights from this analysis.\n\nOverall, the ability of AI to reproduce scientific work is an important research problem. The authors do a good job of constructing the benchmark, running the evaluations, and several ablations. \n\nMy main concern is about the small size of the benchmark. I appreciate that even with 85 missing functions, the benchmark is computationally expensive to run. But 4 papers seems like too small a sample size to conduct a benchmarking exercise. I would be more excited about a benchmark with 10+ papers, even if the number of missing functions per paper is smaller.", "questions": "Overall, I think this paper is a decent contribution. However, the small sample size, the lack of a test set, and lack of clarity about my questions above leads me to propose a borderline score for the paper. That said, I would be happy to increase my score if the following points are addressed:\n\n1) The authors commit to adding a held-out test set (perhaps with 4 other MLRC papers). Of course, I realize the authors can't run experiments on this test set. That said, it would be nice to have a held-out set and functions that go beyond the ones that agent developers might use for \"training\"/developing their agents. It would also make the paper's results more reliable, since n=4 is a very small sample size.\n\n2) The authors address the weaknesses I mention above, in particular to improve the clarity of the paper. (I understand if some of these weaknesses are hard to address, such as reporting cost and time taken after the fact, but I hope the others are easier to address.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- ***Solid compute setup***: The computational setup for running the benchmark is decent, and well beyond what many other papers on scientific reproduction using AI agents use. \n- ***Focus on reproducible results (such as by using Docker)***: I appreciate the authors' efforts to make their study reproducible, including using Docker, and making all of their code and analysis openly available.\n- ***Qualitative analysis***: Many points in the paper include some qualitative analysis, which is quite insightful. For example, the authors analyze whether the first try passes or fails to explain what causes the difference between the fixed and dynamic settings.\n- ***Nice ablation***: Many decisions taken in the paper are supported by strong experimental evidence. For example, the decision to run experiments with varying amounts of context about the paper (No vs. Full) is nice. (Are there intermediate versions of this you could run? Just providing the abstract/intro of the paper? Also, does this raise concerns about leakage, since instead of solving the paper's tasks, the agent could simply \"answer\" the questions based on the results in the paper?)", "weaknesses": "- In the introduction, there's a typo (\"agentive\", should this be \"agentic\"?)\n- No train-test split: As far as I understand, there's no \"train\" split in the data. How should researchers using the benchmark optimize and develop their agents without a train set? This raises concerns about leakage (even in the results reported in the paper, since there are many ablations, for example, in the ReAct agents)\n- No reporting of cost, time taken by agents (which can be important for real-world use)\n- Why pick these 4 papers? MLRC has dozens of papers from the last few years, is there a reason these 4 specific papers were picked?\n- It would be nice to see pass^k in addition to pass@k: Papers like TauBench report pass^k (whether all k samples correctly solved the task). This could be interesting to see the reliability of the agents.\n- The verifier used in the paper is quite simple. The authors could test with better verifiers to see if pass@k performance can be distilled to pass@1 (though it's fine to leave this for future work).\n- How did the authors decide on the 5% tolerance for declaring a test case correct? Did you use the human replication to come up with this tolerance level?\n- It's unclear where the curtailed tests are used and where the full tests are used\n- It's unclear if all models are evaluated on the same 100 examples for n>1, or whether these are randomly sampled each time. If the latter, we might need CIs for more reliable results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NY6uoXZUYO", "reviewer": "Reviewer_3pHq", "summary": "This paper presents AutoExperiment, a benchmark to evaluate LLMs and agents’ abilities to complete some functions in codebases that are necessary to replicate ML research papers’ results. With 85 functions adapted from four papers, the authors conduct controlled experiments by varying the number of functions to be implemented. Results show that agent performance drops as more functions are masked for implementation.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed benchmark grounds its tasks on replicating real-world machine learning research papers.\n2. The authors conduct a series of informative experiments, including controlled evaluation by masking different numbers of functions, exploring potential test-time scaling with verifiers, and trying different agent scaffolds.\n3. The paper is clearly presented and easy to follow.", "weaknesses": "1. This paper proposes function masking and implementation as a new evaluation setting. However, it is questionable whether this is indeed necessary compared to related benchmarks (SciCode, ScienceAgentBench, PaperBench) that require LLMs to generate programs from scratch? Function completion essentially makes these research coding tasks less open-ended and easier by nature, which may fail to reflect real-world research coding.\n2. As a benchmark, this work only involves 85 functions from four papers, which is too narrow and can be biased for evaluating LLMs and agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)