AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench
Authors: Edan Toledo
Karen Hambardzumyan
Martin Josifoski
RISHI HAZRA
Nicolas Baldwin
Alexis Audran-Reiss
Michael Kuchnik
Despoina Magka
Minqi Jiang
Alisia Maria Lupidi
Andrei Lupu
Roberta Raileanu
Tatiana Shavrina
Kelvin Niu
Jean-Christophe Gagnon-Audet
Michael Shvartsman
Shagun Sodhani
Alexander H Miller
Abhishek Charnalia
Derek Dunfield
Carole-Jean Wu
Pontus Stenetorp
Nicola Cancedda
Jakob Nicolaus Foerster
Yoram Bachrach
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM Agents; Code generation; AutoML; LLM-driven search; ML Engineering Agents; ML Research Agents; AI Research Agents
Primary area: deep_learning
TL;DR: We develop AI research agents that achieve state-of-the-art performance on real-world Kaggle competitions by searching the space of candidate code solutions.

Abstract: AI research agents are demonstrating great potential to accelerate scientific progress by automating the design, implementation, and training of machine learning models. We focus on methods for improving agents' performance on MLE-bench, a challenging benchmark where agents compete in Kaggle competitions to solve real-world machine learning problems. We formalize AI research agents as search policies that navigate a space of candidate solutions, iteratively modifying them using operators. By designing and systematically varying different operator sets and search policies (Greedy, MCTS, Evolutionary), we show that their interplay is critical for achieving high performance. Our best pairing of search strategy and operator set achieves a state-of-the-art result on MLE-bench lite, increasing the success rate of achieving a Kaggle medal from 39.6% to 47.7%. Our investigation underscores the importance of jointly considering the search strategy, operator design, and evaluation methodology in advancing automated machine learning.

AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench
Edan 1 Karen Hambardzumyan Martin Josifoski" Rishi Hazraf 1,3
Nicolas Baldwin' Alexis Audran-Reiss' Michael Kuchnik' Despoina Magka' Minqi Jiang' Alisia Maria Lupidi' Andrei Lupu's Roberta Raileamu' Tatiana Shavrina' Kelvin Niu'
Jean-Christophe Gagnon-Audet' Michael Shvartsman' Shagun Sodhani'
Alexander H. Miller' Abhishek Charnalia' Derek Dunfield' Carole-Jean Wu
Pontus Stenetorp Nicola Cancedda' Jakob Nicolaus Foerster' Yoram Bachrach'
'quaal contribution Work done while at Meta
Meta University College London 3Örebro University
Abstract
AI research agents are demonstrating great potential to accelerate scientific progress by automating the design, implementation, and training of machine learning mod- els. We focus on methods for improving agents' performance on MLE-bench, a challenging benchmark where agents compete in Kaggle competitions to solve real-world machine learning problems. We formalize Al research agents as search policies that navigate space of candidate solutions, iteratively modifying them using operators. By designing and systematically varying different operator sets and search policies (Greedy, MCTS, Evolutionary), we show that their interplay is critical for achieving high performance. Our best pairing of search strategy and operator set achieves state-of-the-arr result on MLE-bench lite, increasing the success rate of achieving a Kaggle medal from 39.6 % to 47.7 %. Our investigation underscores the importance of jointly considering the search strategy, operator design, and evaluation methodology in advancing automated machine learning.
Introduction
Science is based on searching the open-ended
[0A0 ADD 61 preview
AlRAs ARR 03
AllAveeer
space of hypotheses and lesting them con-
AID cl-preview
AlRbwes
[DO]]] AIDE
AlRhaces R1
trolled experiment [27]. Recent breakthroughs have resulted in artificial intelligence (AI) agents that offer great potential to automate the scien- tific discovery process [49 42 3]. A key obsta- cle to improving research agents is that their de- signs entangle several factors for performance, making it difficult to pinpoint sources of im- provement via controlled experiments at scale (Fig- 7 These factors span algorithm design, concrete implementation, and ability to leverage compute, as performance gains acerue only if no layer in the stack bottlenecks the benefits from additional compute resources. This challenge is exemplified in MLE-bench [4], a benchmark where AI agents compete in Kaggle competitions to solve real-world machine learning (ML) problems. Notably, the state-of-the-ar approach AIDE 23
50%
40%
Mosal Rate
30%
20%
28
45.91
29.77
45.45
10%
Anl Medal
Figure 1: AIRA agents the AIRA-dojo envi- ronment, AIRA operators, and search policies to achieve SOTA performance MLB-Bench lite.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Repeat
Search
Operators
(9)
State
Select Node(s)
Select & Apply Operator
Evaluation
Draft
Debug
Improve
Score
Drwaldd
Law
Code
Figure 2: Overview of AIRA. Given a problem specification, AIRA maintains a search graph whose
nodes are (partial) solutions. At each iteration, the agent (1) selects nodes via selection policy, (2) picks an operator via an operator policy and applies this operator to the node, and (3) scores the resulting solution via a fitesss function. Here, greedy node-selection strategy applies the improve operator to the highest-scoring node.
does not fully disentangle these performance factors, providing limited insight into which components primarily drive the agent's performance and where improvements are needed.
We start by formalizing the design of Al research agents search algorithm composed of two components: The first one is the search policy which is used to navigate the space of candidate solutions, and the second is the operators which iteratively modifies existing solutions to generate new candidate solutions. In this framework (Fig. 2 AIDE is represented as greedy search algorithm that, at each step, applies one of its code operators (i.e., DRAFT, DEBUG, IMPROVE) the current best solution. This allows us to disentangle the effect of the search from that of the operators.
To assess alternative search algorithms, develop more sophisticated agents performing Evolu- tionary and Monte Carlo Tree Search (MCTS) 6 24. We empirically show that AIDE's s operators, rather than the search algorithm, are bottleneck better performance.
Based on these findings, we design an improved set of operators and evaluate them when paired with the above-mentioned search algorithms. Our best persfrrefonnnn agent achieves state-of-the-ar results on MLE-bench lite, increasing the rate of achieving a Kaggle medal from 39.6 % to 47.7 %.
Additionally, we investigate how the generalization gap-the difference between validation and test scores-affects the agents' performance. ML engineering tasks, as in many real- world use cases, an agent can access only proxy metrics (e.g. validation loss) to guide the search process, while the final solution evaluated on a held-out test set. Therefore, a gap between the expected (validation) and actual (test) loss, could potentially mislead the search process. Indeed, we find systematic overfitting: selecting the final solution in a search graph by its rather than validation- score, would increase the medal rate by to 13 % (absolute scale), depending on the search algorithm. These findings highlight the importance of rigorous evaluation protocols during search and regularization for robust and scalable scientific discovery.
Finally, to conduct experiments, we develop AI Research Agent dojo (AIRA-dojo), framework that provides a scalable and customizable environment for AI research agents. First, AIRA-dojj exposes a robust and flexible interface to compute resources, which is essential for building effective agents. The baseline, AIDE, implemented in AIRA-dojo achieves performance increase of 10.68 (absolute scale) over the reported results 4 Second, AIR Adoo- enables users to experiment with custom operators, search policies, evaluation methods, and tasks within comparable setup. This facilitates a rigorous scientific study of AI research automation. Our code is open-sourced at: http://hhwwbblooocccccoooooerere

---

2
Research Agents as Search Algorithms
Al research agents typically approach machine learning problems by generating artifacts codebases designed to solve given task. Executing these artifacts yields trained models, which are then evaluated against a chosen performance metric. The agent iteratively refines its artifacts improve performance on this metric.
Previous research indicates that L.MSs alone are insufficient effectively solve such open-ended tasks 32 In particular, LLM performance significantly improves when augmented with external tools 337. execution feedback [13], and solutions addressing context limitations. Therefore, devel- oping high-quality models typically involves iterative experimentation, where insights from prior experiments inform subsequent refinements. Recent advancements by Jiang al. [23] demonstrate state-of-the-art performance by conceptualizing this iterative experimentation as tree search over potential solutions.
In this section, we formalize and generalize this perspective by modeling research agents as graph- based search algorithms. The proposed framework allows systematic exploration of alternative agent design choices, providing insights into how different algorithms affect the exploration-exxplotttion trade-off fundamental aspect of search [24 44
2.1 Graph-based Search Framework
We consider an agent that operates by searching directed graph Gt
Et) that evolves over multiple iterations = 0, 1.... where each node E V, S S represents an artifact belonging to the set of all possible artifacts S, while each directed edge (vi, vj) E E, represents a transformation from to oz. The root vo is the initial artifact that can represent empty or starting artifact.
Definition (Components of the Search Algorithm). graph-based search algorithm is specified by the tuple (F, Tscl O, T):
Fitness Function. F: S + [0; 1] is a function that estimates the value or quality of a node v E Vi. Since true fitness is typically not available, F is often proxy measure of the value of the node.
Tsl 2% 2% chooses subset of nodes U, C Vi on which operate, typically guided by a heuristic function h : Vi - R, which assigns a scalar estimate to each node. The heuristic may be derived directly from the fitness function F. such as the upper confidence bounds for trees (UCT) used in MCTS (24). or other custom heuristics tailored to the domain [5] [8].
Selection Policy.
Operator Set. O {oz: 2 + +SH comprises L transformation functions that propose new artifacts = 0r ) J DRAFT, DEBUG, and IMPROVE instantiated prompt-based instructions to an LLM. Composite operators (e.g. the result of applying a sequence of base operators) can also be used.
from one or more selected artifacts. In AIDE, examples include
Operator Policy. Top O x U. + O decides which operator to apply to the current node selection. Termination Rule, halts the search when a computational budget is exhausted, progress stalls, or a fitness threshold is reached.
At each iteration, the agent selects existing promising artifacts, applies transformation operators, and updates the search graph, propelling the discovery process forward. We discuss specific instantiations of search algorithms in Sections 2.3 4.2 In all our instantiations: !I) the termination criterion is set the wall-clock time the maximum number of artifacts. whichever happens first; 2 the fitness function F is defined for each node by the operator that generates or modifies (i.c., F is not global); 3 all operators are LL.--driven except for the MEMORY operator, which is defined by hand.
2.2 Operators
We define operators as high-level functions that take in existing artifacts and produce new ones. These
can range from simple rule- bale parsers to LLM calls using prompting techniques 46 53 or more complex agents like Cursor [20].
This broad definition enables a unified comparison across search methods. A search algorithm can be instantiated with any mix of L.M--base, tool-based, nested search operators.
Building on the demonstrated effectiveness of AIDE (23, we adopt a similar operator set as introduced in their framework, consisting of the following: ! DRAFT initializes the search process by generating
3

---

an initial population of candidate artifacts. 2 DEBUG attempts to identify and correct errors in invalid artifacts. 3 IMPROVE refines valid artifacts enhance their performance according to the evaluation criteria. 4 MEMORY chooses how and where information from past artifacts is used in subsequent operations. 5 Furthermore, we introduce CROSSOVER which recombines useful elements from two artifacts to create a new candidate. Section contaiis further details.
2.3 Re-casting AIDE in our Notation
AIDE [33] is an LLM-driven agent that frames problem-solving tree-search over Python scripts. In the tuple (F, Tsel, O Top, from Section 2.1 its components are:
Fitness & selection (F; Tsel). For each node e, fitness is the mean 5-fold cross-validation (CV) score F(v) [0, 1 1 The selection policy is greedy with respect F. This means at iteration the agent selects and operates o² arg maX F(t) but, with probability Ebug may instead revisit buggy node (F = 0) to aid recovery and maintain diversity.
Operator set O AADEE The agent exposes three LLM operators [DRAFT, IMPROVE, DEBUG) and one handcrafted operator MEMORY or as defined by Jiang et al. [3 the SUMMARIZATION operator. The operators are designed to output the following: DRAFT (3 5-seennnnce plan + fenced script that trains, evaluates, and writes submission, csv); DEBUG (short diagnosis and repaired script given traceback); IMPROVE (Creating exactly one measurable change feature, architecture, schedule, etc. in plan + code form); MEMORY (running summary of all previous designs, scores, and notes that is appended to every DRAFT/IMPROVE, prompt).
Operator policy Top- (a) Seeding: invoke DRAFT exactly n' times from the root uoi (b) Logging: after every DRAFT or IMPROVE, call MEMORY; (c) Main loop: for the node chosen by Teel apply IMPROVE if valid and at least ny drafts exist, if less than nd draft nodes exist, DRAFT is called, otherwise apply DEBUG.
This combination of Tael, Top- and OADE defines the baseline agent we denote AIDEGREEDY (also summarized in Section [B). When the agent uses an alternative search policy while keeping OADE unchanged, we write A I>Eeaacch policy to identify change. For example, AIDEMCTS uses the AIDE operator set and MCTS the search policy.
3 Experiment Design
3.1 AIRA-dojo
An agent's environment greatly influences its performance. To systematically study the space of agentic policies (see Fig. 7 we introduce the Al Research Agent (AIRA) dojo-a scalable and customizable framework for AI research agents. AIRA-dojo provides the environment in which agents operate, along with abstractions for operators and policies as described in Section [ and tasks that define evaluation criteria for agent performance. Using these abstractions, we implement and evaluate four search policies: MCTS, Evolutionary, Greedy, and our own implementation of AIDE. We hope that AIRA-dojo's 's scalability and customizability, together with the provided agent and task implementations, will support and advance future research in the community.
The infrastructure design of AIRA-dojo was informed by several reliability and performance con- straints, as discussed in more detail in Section
3.2 Environment
The environment defines the context in which agents operate.
Action Space. We use Jupyter notebooks to execute code from the agents. With this interface, agents can execute arbitrary Python code, perform file reads and writes, and even use the shell via Jupyter magic commands. The environment captures and returns the status, standard outputs, tracebacks for debugging, and the code block execution time to the agents.
In practice CV scores are not necessarily bounded between zero and one: for example, RMSE an unbounded metric.
Eoug is set 1.0 so mimic the hyperparameters used in MLE-bench by Chan et al. 4

---

Isolation. The environment enforces program-level constraints, such as total runtime limit, GPU and CPU usage, memory, and storage, using Apptainer containers [26]. This ensures complete isolation from host systems, preventing agents from affecting the host environment or other agents, and mitigates risks from unintended actions or failures, such as data leakage or interference with other processes. Furthermore, agents possess root-like privileges within their isolated containers, granting them full control over their environment. This allows them not only to configure their environments using standard package management tools such pip, conda, but even apt-get install.
Superimage. A container image provides a base set of tools required for ML tasks. This image includes pre-configured CUDA support, deep learning frameworks like PyTorch and TensorFlo and essential data science libraries. Each coding session starts from the original Superimage state and can diverge based on the agent's 's actions while isolating the state of the other agents.
Together, these design choices create robust testbed that eliminates confounders both the system and implementation levels. This enables reproducible benchmarking and facilitates the development of long-running agentic systems across thousands of parallel runs.
3.3 MLE-bench
MLE-bench 4 contains 75 Kaggle-sourced tasks for evaluating machine learning engineering agents. We evaluate on MLEE-bench lite - curated subset of 22 tasks selected from the full benchmark allowing us to allocate more seeds per task and increase our confidence in our results.
Our experiments showed that existing methods exhibit high variance in this benchmark (see Section D In line with the benchmark guidelines, we assess each agent's 's performance using the Medal Success Rate. Specifically, for each task, agents earn a bronze, silver, gold medal according task-specific percentile thresholds. We report the percentage of attempts in which an agent secures a medal.
3.4 Experimental Details
Environment. Each candidate agent is launched in a freshly initialized, sandboxed process in AIRA- dojo. This guarantees that file systems and environment variables are isolated across evaluations, and there is no cross-agent interference. Every sandbox is provisioned with a fixed hardware quota: dedicated H200 GPU, 24 logical CPU cores, 100 GB of RAM, and I 'TB of additional scratch storage. Internet access is permitted solely for fetching third-party party packages and model checkpoints.
Time Constraints. In line with MLE-bench, the agent is allowed a 24-hour wall-clock window. Within this period, each agent has a maximum runtime of hours per code execution. We chose to reduce this from the 9-hour limit used in MLE-bench after preliminary experiments showed no difference performance and a higher average number of valid nodes in the search trees.
LLMs. We conducted all the experiments with the full-sized DeepSeek R1 77) model, with 128K- token context window ensure input coverage without truncation. Due to wall-clock constraints, this choice was guided by both the model's capabilities and applicable rate limits. In particular, selected DeepSeek R1 as the most capable open-source model available, which allowed to self-host inference servers and maintain high experimental throughput without encountering rate limits. For the main results, we also evaluated o3 (35), one of the most capable closed-source models. To ensure experiment validity and avoid hitting rate limits, we limited the number of parallel runs when experimenting with o3. We always use GPT-4o 34 with Structured Outputs to parse code execution outputs extracting run success, summary text, and validation metric. For the figures, we take [OAI] AIDE o1 artifacts from the MLE-bench 4 GitHub repository. To directly compare with their results, we evaluated o1 o1-preview, but only completed experiments for one agent before deprecation. See Section E for details.
4 AIRA
4.1
Operators OAIRA
As part of AIRA-dojo, we propose a new operator set based on OADDE, denoted OARAA To this end, we focus on maintaining a cleaner context through better-scoped memory, and encouraging structured reasoning and strategic diversity in ideation. The key differences are:
5

---

① Prompt-adaptive complexity. We introduce dynamic complexity eue within the system prompt in order to guide the complexity of artifacts generated by the DRAFT and IMPROVE operators. The complexity is determined by the number of children, De, of the node being processed:
complexity(Me = "minimal" if nhe < 2, "moderate" if 2 S The < 4, "advanced" if ne 2 5
For the DRAFT operator, this cue influences the complexity of the generated ideas. For the IM- PROVE operator, it guides the complexity of the enhancements. This dynamic signal helps prevent premature over-engineering by ensuring the agent provides simple solutions when appropriate, while encouraging more thorough exploration when more advanced solutions may be necessary.
Scoped memory. We modify the MEMORY operator to extract different types of memories depending on the operator used. Specifically, for DRAFT and IMPROVE, it retrieves only sibling memories- the children the artifact the agent is applying the operator to theerereby promoting diversity. This prevents overloading the context and reduces behavior indicative of mode collapse. Conversely, for DEBUG, it retrieves the entire ancestral memory of the artifact's debug chain, enabling review of prior fix attempts and avoiding "undo-redo" oscillations.
Think Tokens. For reasoning models, we use the operators' system prompts explicitly encourage them to use thinking tokens for reasoning and reflection. These thoughts are stripped from the final answer-remaining invisible to other operators (e.g., memory). On average, we observe a 2x increase in completion tokens generated by the AIRA operator set (see Section F).
4.2 Agents
In this section, we introduce three agents that combine the operators with distinct search policies (see Appendix H for the rationale behind their selection). Each agent uses the same proxy-fitness function J. (5-fold CV) and the same termination criterion (based on wall-clock time or artifact cap).
in Section 4.1
This agent employs greedy search (Section 2.3 using the OaIRA operator set. Any performance improvement of AIRAGREEDY over AIDEGREEDY directly reflects the benefit of the new operators, since the only difference between the two is the operator set.
AIRAGREEDY.
AIRAMCTS.
This agent uses Monte-Carke Tree Search (MCTS) (4 with OAIRA operator set. Our implementation of MCTS follows the canonical loop (selection, expansion, evaluation, and backup), but omits simulated roll-outs: the leaf value of expanded nodes the proxy fitness function F::
Selection. From the root node vo. descend by selecting the node with the highest UCT score (v) arg maX (ucr(v) where hucr(v) u) Q(v) + log N (u) + +s), where (-)-) and Q(-) are the visit count and running mean fitness. Here, u is the parent of v. Expansion. Only leaves expanded. The chosen operator from OAIRA is applied n times, creating n children. Buggy children enter an automatic DEBUG loop until fixed or the budget expires.
Evaluation & backup. The leaf fitness is F(vz) is mnck-propagated to ancestors with the standard incremental update of (N, Q)-
AIRAEVO. The evolutionary agent keeps population V; of fixed size and repeats:
Parent selection: Select individuals with probability (u) F(v) Euev, F(u).
Reproduction: With a fixed probability, apply IMPROVE; otherwise, apply CROSSOVER. Buggy parents undergo DEBUG until they are either fixed or the debug attempt limit is reached.
Replacement: Offspring replace the least-fit individuals in Vn.
For both AIRAMCTS and AIRAEVO, we normalize all fitness values using the minimum and maximum values observed during the search process. This normalization ensures a consistent set of hyperparameters throughout the search. To select the final solution, the AIRA and AIDE agents return the one with the highest validation score.
Experiments and Results
5.1 Analyzing the Performance the Current SoTA
The most effective search algorithms strike a balance between exploration and exploitation. In this section, we analyze the -xppliioteeoooitiiinn trade-offs AIDE, the current offte-of-the-art method, and then investigate how additional computation time increases its performance.

---

(20,0]
(00)0] All Monr
4050
Nede Dar
3.%
2.94
35:25
30.18
$9.77
30.30
55.45
35.00
34.35
391 01
105
0%
MCTS
EvD
Figure 3: Searching with AIDE's operators. When limited AIDE's operator set O AIDE agents using more advances search policies (e.g., MCTS, evolutionary algorithms) gain no ad advantage, underscoring the operator set as the bottleneck.
72%
82%
40%
3004
20%
[D006] (ran01111)
[D4C]] 40Emme (Tes)
20%
(000) K1
AlB 13
300 (D0)0] arDesmm 0d
ARM
ARAceay
[DCE0]
Allfccn e3
15%
2.5
10.0
12.5 150 175 250
Vall <bdd time (hr))
WMll cock Drne test
Figure 4: a) AIDE's performance profile over 24-hour search window, Perceived vs. actual medal rate over 24 hours of AIDEGREEDY. The curves show the mean validation (agent-reported) and held-out test medal rates across 20 seeds for all tasks. The widening band illustrates the generalization gap. revealing how apparent gains on the validation can mask overfitting and ultimately undermine the search process. b) Performance profiles of all agents after 24-hour search window.
The three main factors that influence this balance are memory, the operator set, and the selection policy. Memory structures prior knowledge-stor storing promising solutions and tracking which regions of the solution space have been sampled to inform subsequent decisions. This information can bias the search toward exploration by encouraging diversity or toward exploitation by discouraging Operators then apply controlled transformations to existing solutions: for instance, random mutations introduce diversity, targeted refinements exploit known strengths, and recombinations merge features from multiple candidates. Finally, the selection policy balances exploiting high-quality regions of the solution space with exploring less-tested areas by determining how to allocate computational resources. A non-greedy selection policy, such as in MCTS, periodically directs resources toward branches that may appear suboptimal, with the goal of uncovering better solutions.
What is the effect of memory? To quantify the impact of the MEMORY operator, we conduct controlled ablation comparing the performance of AIDE with and without the MEMORY operator enabled. As shown in Fig. 3 both variants achieve nearly identical mean medal rates. This suggests that memory not driving factor behind AIDE's strong performance. What the effect of exploration at the search level with AIDE's operators? AIDE uses a selection policy that does not explore and always greedily selects the most promising candidate. To evaluate the impact of the search policy a modulator of the exploration-exploitation tradcoff when searching using AIDE's operators, we replaced the operators in AIRAMCTS with that of AIDE and varied the UCT exploration constant cucr E {0, 0.25, 0.75J. The cucT, which controls exploration in MCTS, allows us to isolate the effect of search-level exploration on downstream performance. The resulting medal rates (Fig. B showed only marginal differences across all CUCT values, supporting hypothesis that search-level exploration is constrained by the interaction between the operator set and the search policy. We observe the same limitation when performing the same operator replacement in AIRAEVO.
What is the performance profile AIDE? To examine the improvement gains over time, we plot AIDE's anytime performance, which is the average medal rate achieved by the agent if the search was
7

---

[GN] NIDEs ma al-presieu
[Dojo] AlDEzaway R1
AlRAco AltAcco 03
ARAcowey R1 ARAuwey 63
AIRAwcT Alffaecs od
[DQJO] ADD
[Dajo] AlDEnwedy 02
50%
40%
Medd hate
30%
10%
0%
Medal
Silwr
Gold
Figure 5: Medal rates on MLEE-bench Lite. Performance shown for three medal categories: any medal, silver medals and above, and gold medals only. Error bars represent 95% confidence intervals computed using stratified bootstrapping.
to terminate at time L. The results, summarized in Fig. 4a performance (validation score) continues to improve over time, the true test performance plateaus, slightly decreases over time. These findings suggest that overfitting might be fundamental limitation to the agent's performance. We further investigate this in Section 5.3
suggest that while the agent's perceived
5.2 AIRA Beyond Greedy
Based on the observation that search policies yield no performance gain with AIDE's operators, this section is divided into two parts: ! assessing the effectiveness of our improved operators (see Section 4.1 2 examining their interplay with the more advanced search policies. The results are
summarized in Fig. 5 with a detailed breakdown of performance for each task in Section L
Evaluating the enviromment. But first, we highlight the benefits from AIRA-dojo. Specifically, our baseline agent implementation. AIDEGREEDY 01 ol-preview. operating the AIRA-dojo environ- ment, improves the medal rate from 35.2 % to 45.9 % compared to the reported results [4]. This corresponds to state-of-the-as performance with relative improvement of 30 % in medal rate. is notable that AIDEGREEDY o1 ol-preview, outperforms both AIDEGREEDY R1 and AIDEGREEDY 03, despite o3 being the newest model in the series. While evaluating all agents with o1 o1-preview would have been informative, we were only able to complete the AIDEGREEDY experiment before the model was discontinued.
Evaluating the operator sets. AIDEGREEDY and AIRAGREEDY employ the same search policy but differ in their operator sets. Comparing their performance isolates the effect of the operators. AIRAGREEDY outperforms AIDEGREEDY (45.5 % vs. 39.8 %). representing 14 % relative improvement and underscoring the importance of operators for performance.
Evaluating search policies. Equipped with an improved set of operators, we now explore whether combining them with advanced search methods can further enhance performance. The results (Fig. 5) show that for R1, AIRAMCTS achieves the best performance, achieving of-te-----heearrr performance on MLE-bench Lite with average medal rate of 47%. Both AIRAMCTS and AIRAGREEDY achieve silver-or-above medal rates ( 36.5%) outperforming the baseline by 4.5 absolute points. In terms of gold medals, AIR AGREEDY performs best, reaching 27%. For o3, ATR ACTS and A AIR AGREEDY perform comparably in terms of any medal and silver-or-above medal rates, while AIRAMCTS outperforms AIRAGREEDY gold medals by 2.2 percentage points. Although agents using R1 and o3 achieve similar overall medal rates, those using o3 perform better when considering gold and silver-or-above medal rates. Specifically, AIRAMCTS increases the gold medal rate from 25.68 % with R1 to 30.91 % with o3, a relative increase of 20 %.
Collectively, what stands out is that all search policies combined with AIRA operators outperform AIDEGREEDY. Furthermore, the rankings between agents using the AIRA operators with different search strategies differs significantly from that observed in Section 5.1
Finally, Fig. 4b shows the anytime performance of each agent over the 24-hour window defined by MLE-bench. For conciseness, will focus our discussion on performance with R1, as the results

---

nitrmetionefiaa Nace
satter
AD
All
AllAAa
60%
5555
Medal fate
40%
50%
6595
20%
4095
Randomak
3556
Top K
Oracke
0%
305
AlRApoo
AlRAguoey
AIRAacts
Figure 6: a) The validation-fest metric mismatch. Results shown are for agents using R1. Bars depict the absolute performance gap between three configurations: (i) using the validation metric for both intermediate (search) and final (submission) node selection; (ii) using the validation metric only for search and the test metric for selection; and (iii) using the test metric for both search and selection. b) Bridging the validation-fest gap. Medal rate achieved by two final node selection strategies as a function of k: (i) randomly sampling k nodes and reporting the highest test score among them; (ii) selecting the top nodes by validation score and reporting the highest test score among those, As increases, the validation- based strategy closes the gap to the upper-bound performance given by the best test score over the entire search graph.
with o3 follow similar pattern. We observe that the rankings between agents change time. For example, at the 3-hour mark, the performance gap between AIR AGREEDY and AIRAMCTS is notable. This gap narrows by the 10-hour mark, where all agents perform similarly. Meaningful divergence in performance appears only after 19 hours. These trends reflect the interaction between rankings and the resources provided, as shown in Figure 7.
We further examine the effect of extended compute time on performance by running experiments for up to 5 days (Appendix O. showing that our agents continue to improve beyond the 24-hour period. To illustrate differences in search behavior, Appendix [M presents representative search trees from our experiments for each agent, which may help clarify the strengths and weaknesses of each method.
5.3 The Generalization Gap: Searching with Proxy Evaluation
Agents steer their search using the validation scores of candidate solutions, but ultimately, performance is measured on held-out test set. Due to finite sample effects, the validation score is not perfectly predictive of performance on the test set a discrepancy known the generalization gap.
In this section, we measure the impact of this generalization gap on the agents' performance. We further quantify whether this impact is more detrimental during the intermediate node selection in the search process or the final (submitted) node selection.
How large is the impact of the generalization gap performance? We begin by comparing two extremes: VAL/VAL-boL intermediate and final node selection use the validation score (standard practice); TEST/TEST-aa oracle baseline using the true test score at both stages. As shown in Fig. [a] searching based on the test score instead of the validation score improves performance by 9.4 % and 12.4 % for AIRAMCTS and AIRAEVO. The gap is even higher for the agents using greedy search policy, reaching 15 % for AIREGREEDY and 16.6 % for AIDEGREEDY. For the rest of this section, we investigate the nature of this gap in performance and how close it.
How much of the gap can attributed to final node selection? To answer this question, we consider the following two settings: VAL/VAL: and VAL/TEST- the intermediate node selection uses the validation score, while the final node selection is made based on the test scores. The results (Fig. 6aa) indicate that by selecting the best node in a search graph constructed without privileged access, Le., based on validation signal, all agents achieve a performance boost of 9 to 13 absolute points. Crucially, comparing VAL/TEST with TEST/TEST shows that oracle final-node selection eliminates the gap between the standard (VAL/VAL) and oracle (TEST/TEST) settings for both AIRAMCTS and ÁIRAEVO. Even for the agents implementing the simpler greedy policy, selecting the best node in the final search graph closes more than 60 % of the gap. These findings highlight robust fial-nodde-electionn strategies as a promising path higher performance.
Bridging the gap through multiple submissions. A straightforward remedy is to hedge against validation score noise by selecting multiple promising nodes. Specifically, among the top-k nodes

---

ranked by validation score in the search graph, we report the test score of the best best-performing one. As baseline, we repeat the process for a set of random-k nodes, and summarize the results in Fig. (bb) We highlight the following observations: ! the larger gap between top-k and random-k in agents using non-greedy algorithms, ARRO and AIRAMCTS, suggest greater diversity in their search graphs; 2 with as little as 3 top-k submissions agents achieve an additional 10 % of performance; 3 the top validation nodes are informative of the best performing nodes.
6 Related Works
Scaling Search with LLMs. Integrating search with LLMM-based generators has gained traction across
domains such coding (1. math [52]. and planning []6 15 In practice, increased test-time compute
through algorithms like best-of-N [45.. beam search 16. MCTS [5), and evolutionary search [39
17 often improves performance beyond the architectural and size constraints of LLMs [4T]. Our
results highlight that these performance gains depend critically on the interplay between search components (see Section A] an aspect that prior work has largely overlooked.
Automating ML Engineering and Scientific Discovery. Traditional approaches like AutoML [2] 33 and Neural Architecture Search (NAS) [54 29 38 automate ML by searching over predefined, expert-designed configuration spaces, often via force-fore or heuristic methods 2 (28]. In contrast, recent advances in LLMs enable more open-ended design. AIDE 23 treats ML engineering as LLM- guided code optimization via tree search. AI Scientist v2
49 extends this paradigm, automating the entire research pipeline using agentic LLMs. Similar techniques have been applied to software engineering D. algorithm and reward design (39 31
17 7.. and even natural sciences 3 42). Existing systems often combine search procedures, operators, and evaluation in ways that make it challenging to understand which components drive performance improvements. To address this, we separate these components and look how each one-and and their interactions-can be optimized better. Concurrent work, R&D-Agent [SI) which addresses the same problem setting, achieved impressive results on E.E.BBnnch. Our results fall within their reported standard deviation. We note that their experiments use at most 6 seeds, and as discussed in Section]] the limited number of seeds may introduce variation in the conclusions. Finally, our approach differs from methods such as Agent K [14] which uses long-term memory across multiple Kaggle competitions. In contrast, we focus on addressing each competition independently.
AT Research Frameworks. Most existing benchmarks for machine learning or research engineering, such MLGym [32). RE-bench [48), and MLAgentBench [19], come with their own frameworks. Our approach is most similar to Inspect [21) framework which provides an abstraction that makes minimal assumptions about the agent and evaluation design, clearly separating the two. This flexibility in agent design is essential for enabling rigorous comparisons across wide range of agents from simple LLM-based agents with tool access to scaffolds combining algorithmic and LLMM- components within the same environment. The environment plays a critical role in performance (see Section 5.2). However, unlike Inspect, AIRA-dojo focuses strongly on long-running research tasks, which influences our environment design choices. For example, prior work typically uses Docker to containerize agents' workspaces, but Docker is unsupported on most HPC clusters due to its reliance on root privileges and lack of seamless integration with common HPC resource managers. This limitation hinders scalability, addressed through our Apptainer (see Section 3.2)
7 Conclusion
We conceptualize the design of Al research agents as a combination of two axes: search policy and operators. This formulation allowed us perform systematic investigation of the interplay between the two, highlighting how the operator set can act as bottleneck to performance improvements. Based on this finding, we designed an enhanced operator set and constructed agents that combines these operators with several search strategies: Greedy, Monte Carlo Tree Search (MCTS), or Evolutionary Search. Our best- persforerfonnng agent achieves new state of the art on MLE-bench, increasing the success rate of winning a Kaggle medal from 39.6 % to 47.7 %. Further, we analyzed the role of the generalization gap in node evaluation. Our findings indicate systematic overfittin: selecting the final solution from the search graph based on its test score instead of its validation score would increase the medal rate by to 13 % (absolute scale), depending on the search strategy. This highlights that robust final-node selection is a promising avenue for improving performance.
10

---

References
[1] A. Antoniades, Örwall, Zhang, Y. Xie, Goyal, and W. Y. Wang. SWE-search: Enhancing software agents with monte carlo tree search and iterative refinement. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview net//Eorumm1d-C7sPPPaee
[2] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. The journal of machine learning research, 13(1):281-305, 2012.
[3] D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. Autonomous chemical research with large language models. Nature, 624(7992):50--778 2023. ISSN 1476-4687. doi: 1038/ s4 586-0223-0679--. URL htttp:://doi.rrg/ 103///0000000001
1008/ s4188/41186--10 023-
[4] S. Chan, N. Chowdhury, O. Jaffe, J. Aung, Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, A. Madry, and L.. Weng. MLE-bench: Evaluating machine learning agents on machine learning engineering.
The Thirteenth International Conference on Learning Representations, 2025. URL htttps://openreviee
[5] J. M. Crawford and L.. D. Auton. Experimental results on the crossover point in random 3-sat. Artificial intelligence, (1(1-2)31-55, 1996.
[6] K. De Jong. Evolutionarry computation: a unified approach. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 373-388, 2017.
[7] DeepSeek-Al. AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL
12948
[8] Elsken, J. Metzen, and F. Hutter. Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:1711.04522 2017.
[9] G. T. et. al. Gemini: A family of highly capable multimodal models, 2025. URL https
//arxiv www..rrgggbbe.... /231..2222222 11805
[10] L. T. et. al. The llama herd of models, 2024. URL httpp::/aaxxv..rgg////// hangg/bbs/2400 2e0.bb///77222783333
[11] M. Faldor, J. Zhang, A. Cully, and J. Clune. OMNI-EPIC: Open-endedness via models of human notions of interestingness with environments programmed in code. In The Thirteenth International Conference on Learning Representations, 2025. URL htt.hhttp://// httpss:/openrreiie. net/foruunidd-VIXaakjj
[12] M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. Auto-skleamn 2.0; hands-frec automl via meta-learning J. Mach. Learn. Res., 23(1), Jan. 2022. ISSN 1532-4435.
[13] J. Gehring, K. Zheng, J. Copet, V. Mella, Q. Carbonneaux, T. Cohen, and G. Synnaeve. Rlef: Grounding code 11ms execution feedback with reinforcement learning, 2025. URL https://arxiv. .wgp//ii..r////ooooooo.oo abs 02089
[14] A. Grosnit, A. Maraval, J. Doran, G. Paolo, A. Thomas, R. S. H. N. Beevi, J. Gonzalez, K. Khandelwal, Iacobacci, A. Benechehab, H. Cherkaoui, Y. A. EI-Hili, K. Shao, J. Hao, J. Yao, B. Kegl, H. Bou-Ammar, and Wang. Large language models orchestrating structured reasoning achieve kaggle grandmaster level, 2024. URL https:
org/ abs/ 03562
[15] S. Hao, Y. Gu, H. Ma, J. Hong. Z. Wang, D. Wang, and Z. Hu. Reasoning with language model is planning with world model, Dec. 2023. URL htttps://aclanthllogg 0rg/ emnlp-man mwwn.aainn..... 507
[16] R. Hazra, Z.. Dos Martires, and I.. De Raedt. Saycanpay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20123-20133, 2024.
[17] R. Hazra, A. Sygkounas, A. Persson, A. Loutfi, and P. D. Martires. REvolve: Reward evolution with large language models using human feedback. In The Thirleenth International Conference on Learning Representations, 2025. (ttpsttss: //tppe/evie/...eee///eee/ UR//attips//ppeeeeeeee wwwpwewwwwwwi.eeeorrn..t. net/forum? id=cJPUpLL@0w

---

[18] J. N. Hooker and V. Vinay. Branching rules for satisfiability. Journal of Automated Reasoning. 15(3):359-385 1995.
[19] Q. Huang, J. Vora, Liang, and J. Leskovee. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.033000 2023.
[20] A. Inc. Cursor: The ai code editor, 2025. URL httph:t/p////// www.ww.wrrrrccoom com/
[21] U. A. S. Institute. Inspect AI: Framework for Large Language Model Evaluations, May 2025. URL phttps://ggitee
[22] Z. Jiang, H. Lin, Y. Zhong, Q. Huang, Y. Chen, Z. Zhang, Y. Peng. X. Li, C. Xie, S. Nong.
Y. Jia, S. He, H. Chen, Z. Bai, Q. Hou, S. Yan, D. Zhou, Y. Sheng, Z. Jiang, H. Xu, H. Wei, Z. Zhang, P. Nie, L. Zou, S. Zhao, L. Xiang, Z. Liu, Z. Li, X. Jia, J. Ye, X. Jin, and X. Liu. Megascale: scaling large language model training more than 10,000 gpus. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation, NS USA, 2024. USENIX Association. ISBN 978- 778-1-939133-39-1
[23] Z. Jiang, D. Schmidt, D. Srikamth. D. Xu, I. Kaplan, D. Jacenko, and Wu. AIDE: AI-Driven Exploration in the Space of Code. arXiv preprint, 2025. URL https://arxxii org/ abs/ 2502. 13138
[24] L. Koesis and Szepesvári. Bandit based monte-carlo planning. European conference on machine learning, pages 282-293. Springer, 2006.
[25] A. Kokolis, M. Kuchnik, J. Hoffman, A. Kumar, P. Malani, F. Ma. Z. DeVito, S. Sengupta, K. Saladi, and C.-J. Wu. Revisiting Reliability in Large-Scale Machine Learning Research Clusters In 2025 IEEE International Symposium High Performance Computer Architecture (HPCA), pages 1259-121111 1274, Los Alamitos, CA, USA, Mar. 2025. IEEE Computer Society.
doi: 10.1 109/H/CA66900 w0t.C//AAA.....22...... 202..022.00 .00096. URL https::///o...... http:///o....cccoppuuerrrrooooo/ 10 10..109//PCAA1100 2025 2025.00096
[26] G. M. Kurtzer, celerget, M. Bauer, I. Kaneshiro, D. Trudgian, and D. Godlove. hpeng/singularity: Singularity 3.7.3, Apr. 2021. URL https://doi. org/ 10. 001.52221122z. 1667//110oo.4667718
[27] Langley, H. A. Simon, G. Bradshaw, and M. Zytkow. Scientific discovery, 1987.
[28] L. Li, Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1553 2018
[29] H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. International Conference on Learning Representations, 2019. URL http:tt:///ppereeiieeeeee wwwprreviwiieeeeffrrrnree id=S1eYHoC5FX
[30] Liu, K. Wang, Y. Chen, X. Peng, Z. Chen, L. Zhang, and Y. Lou. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.02977 2024.
[31] C. Lu, S. Holt, C. Fanconi, A. J. Chan, J. N. Foerster, M. der Schaar, and R. T. Lange. Discovering preference optimization algorithms with and for large language models. In The Tiihtt-eightt Annual Conference Neural Information Processing Systems, 2024. URL sstps://openreviae net/ htte/fforumiidd=er ptt.a///eee..eeeooooooaarrraaaa. JQJJOZLL,
[32] D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V. Moens, A. Budhiraja, D. Magka, V. Vorotilov, G. Chaurasia, D. Hupkes, R. S. Cabral, T. Shavrina, J. Foerster, Y. Bachrach, W. Y. Wang, and R. Raileanu. MLGym: A New Framework and Benchmark for Advancing AI Research Agents, 2025. URL U httptttttt/ttiiii org/ hbtp//aa//22...2 14499
[33] R. Olson and H. Moore. TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning, pages 151-160. Springer International Publishing, Cham, 2019, URL
https:///ol: org/a/.rr111... 10/ 1ww.000000000 000///77-3-- -0000000--00 3-0---0000---------- 00000-501885-8
[34] OpenAI.
Gpt-4o system card, 2024.
URI
https: httpp:///eee https:://edd.o/// openai........... com/
gpt- 40- 4o--o--yytee--ooo cerstem-card.poo
12

---

[35] OpenAI. Introducing o3 and 04 mini, 2024.
URL
introducing-on o3- and- o4- /4-mini/
[36]
OpenAl. Gpt-5 system card, 2025, URL https://cdn. ptttps:/ccdn.ooeaa... wwt. wtt.://t----sssssteccaaa- pdf
[37] Y. Qin, Hu, Lin, W. Chen, Ding, G. Cui, Z. Zeng. X. Zhou, Y. Huang, C. Xiao, C. Han, Y. R. Fung, Y. Su, H. Wang, C. Qian, R. Tian, K. Zhu, S. Liang, X. Shen, B. Xu, Z. Zhang, Y. Ye, B. Li, Z. Tang. J. Yi, Y. Zhu, Z. Dai, L. Yan, X. Cong, Y. Lu, W. Zhao, Y. Huang. J. Yan, X. Han, X. Sun, D. Li, J. Phang. C. Yang. T. Wu, H. Ji, G. Li, Z. Liu, and M. Sun. Tool learning with foundation models. ACM Comput. Surv. 57(4), Dec. 2024. doi: 10. 10.1145/3704435. URL
//t.://./...r////o 10. 1145/3704435
[38]
E. Real, A. Aggarwal, Huang, and Q. V. Le. Regularized evolution for image classifier archi- tecture search. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-Fuu First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium Educational Advances in Artificial Intelligence, AAAI' 19ЛAАА 19/EAAI 19. doi: 10. 1609/aaa... /tt.//aaa...3301..333144444 URL
AAAI Press, 2019. ISBN 1-77---- https://doi. org/ 10rgg/10.10 1609 aaai v33101 33014780
[39] B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi. Mathematical discover- ies from program search with large language models. Nature, 625(7995):468 475, 2024. ISSN 1476-4687, doi: 10. 10.1088/441186--22-0062222 941586-023-06924-
URL
[40] C. Si, D. Yang, and T. Hashimoto. Can ilms generate novel research ideas? a large-scale human study with 100+ nip researchers. arXiv preprint MXXi:24409004099, 2024.
[41] C. Snell, Lee, K. Xu, and A. Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv, orgp:/iiii.orreeeooe abs/ /200 2408.03314
[42] K. Swanson, W. Wu, N. L. Bulaong, J. Pak, and Zou. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation. bioRxiv, 2024. doi: 10. 1001/2222 11.11.623004 UtLhtts:: biorxiv...
1.620004
[43] C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown Auto-weka: combined selec- tion and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '13, page 847-855, New York, NY, USA, 2013, Association for Computing Machinery. ISBN 9781 9781450321747. 2487575 2487629
doi: 00,114452488755.24887722 URL httph:ttt//:.o.... http://oii..gg/0..1111.1.10./
[44] M. Črepinšek, S.-H. Liu, and M. Mernik. Exploration and exploitation in evolutionary al- gorithms: A survey. ACM Comput. Surv. 45(3), July 2013. ISSN 0360-0300. URL
https://doi.. /ttps:://oo..or/10...o 1111//0//11111///774444111111111 2480752
[45] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL httphhttp///pmmmm
[46] Wei, X. Wang, D. Schuurmans, M. Bosma, Ichter, F. Xia, E. H. Chi, Q, V. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htw.tttttt//peeeiiiiieeeeo wtw.://www..ttoouuiuiiiomcm _iiididd_____________
[47] C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Sai- fullah, S. Dey. Shubh-Agrawal, S. S. Sandha, S. V. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, and M. Goldblum. Livebench: challenging, contamination-fiee LLM bench- mark. In The Thirteenth International Conference Learning Representations, 2025.
13

---

[48] H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15111 2024.
[49] Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha. The Al Scientist- v2: Workshop-Lever Automated Scientific Discovery via Agentic Tree Search, 2025. URL https://arxiv org/ abs/ 2204// 08066
[50] J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Infor- mation Processing Systems, 1755058--5052, 2024.
[51] X. Yang, X. Yang, Fang, B. Xian, Y. Li, Wang, M. Xu, H. Pan, X. Hong, W. Liu, Y. Shen, W. Chen, and J. Bian. R&d-agent: Automating data-driven ai solution building through lmm-poeered automated research, development, and evolution, 2025. URL https 7/arxiv. org/ abs 2505 14738
[52] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
[53] S. Yao, Zhao, D. Yu, N. Du, Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023. URL http::tttp:p/eeeeetiii
net/forum?id-Wtt WE vluYUL-XX
[54] B. Zoph and Q. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017, URL https://openrevier httppeerrriiiee.nee/ruuruor net/forum? id-r1Ue8Hcxg
14

---

Appendix
A
Preconditions for Scaling
The hierarchical diagram in Fig. fillustrates key insights into preconditions for improving agent performance through additional compute. Agents benefit from scaling search only when performance gains are not limited by the environment in which the agent operates, the quality of the evaluation signal guiding the search, the capability of the operators performing the search, or the search policy itself.
Agant imptlmmmmennt comommttiionn
Search
Agthn1 Disiign
Operators
Evaluation Signal
Environment
Compute Resources
Specifically, if the agent is provided with 10x more com- pute resources (e.g., 10 GPUs for hours), the environ- ment must allow the agent to effectively leverage these re- sources. Without sufficiently high-quality evaluation sig- nal-the ability accurately assess solution quality the direction improvement will be unclear, which would undermine the search process. Finally, successful search requires capable operators that effectively perform their functions and search policy that allocates compute efficiently to the most promising regions of the search space and appropriate operators.
Figure 7: Key factors dictating agent performance and scalability as com- pute resourees increase.
Overall, this work underscores the importance of a strong foundation in algorithm design and infrastructure to ensure that scaling search translates into downstream performance improvements.
B Re-Casting AIDE in our Notation
Section 2.3 shows how AIDE can be reformulated within our framework. Table summariiess the tuple that specifies the AIDE agent.
Table I: Instantiation the tuple (F, O, Top T) for AIDE.
Component
AIDE choice
J
5-fold CV score
Tsel
Greedy + Ebug exploration
O
OADE
Top
Fixed rule
T
Wall-clock or No. Node cap
C The Effect of Compute: Searching Beyond 24h
In our analysis of the agents' anytime performance (see Section 5.2). we observe that their rankings evolve over time, with significant differences in performance emerging only after 15 hours of execution.
the experiments presented the main paper, we adopt the experimental setup proposed in the MI.E-bench paper, where agents are given 24 hours to complete the task. However, our aim is to develop agents capable of effectively utilizing computational resources well beyond the 24-hour limit, and thus, evaluating within this restricted timeframe offers only limited insights into their potential long-term capabilities.
To examine the impact of access to computational resources, we conducted an experiment with the same setup described Section 5.2] extending the total runtime 90 hours -eesrly four days. The results, summarized in Fig. X demonstrate similar pattern, where notable differences in behavior emerge after roughly 15 hours. Specifically, the performance of the AIRA agents (AIRAGREED) and AIR AMCTS) continues to improve, whereas the performance of AIDEGREEDY plateaus. Another notable observation that agents employing our proposed operators, AIR AIRAGRED and AIR AMCTS, exhibit comparable improvement profiles until around the 50-hour mark, at which point AIRAMCTS
15

---

60%
50%
40%
Medal Rate
30%
[DOJO] AIDE R1
20%
AIRAgreedy R1
AIRANCTS R1
10%
0%
10
30
40
60
80
90
Wall-clock time (hrs)
Figure 8: Medal achievement rates over a 90-hour search horizon. Each point represents the mean percentage of medals earned across 10 independent runs per task on the MLE-bench lite suite. Results are based on a complete replication of the baseline experiments and extended 90 hours of search.
60%
60%
50%
50%
50%
100/0 4055 KUURR
200
20%
20%
20%
38
128
24
(a) AIDE
(b) MCTS
(c) Greedy
Figure 9: Test vs. Validation Medal Rates over 90 hours. Comparison of medal rates between test and validation scores for up to 120 hours of runtime. Each subplot shows the medal rate if the validation score matched the test score (potential performance) versus the actual medal rate given the true test score, revealing the performance gap between validation and test performance.
begins to overfit and its performance subsequently deteriorates. Conversely, AIRAGREEDY continues to improve, ultimately achieving peak medal rate of approximately 53%. This peak represents an absolute improvement of 6% over the highest performance attained with 24 hours of compute, translating into relative gain 12%. However, the agent's performance subsequently declines as overfitting begins to manifest. We see in Fig. Fig-9tth 1hhat the validation performance continues to climb higher and overfitting becomes more severe. Ultimately over longer search horizon, we see AIRAGREEDY and AIRAMCTS achieve similar test-validation gap AIDEGREEDY. This is consistent with our findings Section 5.3 and highlights a fundamental limitation that affects agents' ability to operate effectivvly over much longer time horizons.fu
Overall, this analysis underscores the necessity of jointly considering search strategy, operator design, and computational resource availability (see Fig. (7). Furthermore, it highlights the fundamental impact of the generalization gap, identified and studied in Section 5.3
16

---

D Evaluation on the Full MLE-bench Set
To get better estimate of performance, as well as the effect of task difficulty and recent, in this section, we report the performance of agents on the full MLE-bench set of tasks. Specifically, we focus on our greedy agent, AIRAGREEDY, using 03 the backbone LLM. We also report the results for AIDEGREEDY, the strongest version of the previous state-of of-the-art agent, and for OpenAl's proprietary agents using o3 and gpt- 5-tinking backbones, using their reported numbers. The average performance and the confidence intervals are computed over 20 random seeds.
repotts the performance for the overall performance, the complexity the MLE-bench paper [4] and the MLE-bench 30 split of interesting and diverse papers proposed in the gpt-5 system card [36]. We achieve an overall medal rate of 31.6%, and outperform the baseline on all splits. In all splits except from low, we achieve a relative increase of more than 100%. We also observed a 7.3 point performance increase on the low split (from 47.7 55.0) by using high reasoning effort and further improving the infrastructure (e.g., optimizing the file system and caching models used solutions avoid litt-lini errors).
splits suggested by
Table 2: Medal rates (mean standard error) across agents. The results annotated with an asterix are reproduced from prior work. For OpenAI o3 and GPT-5 Thinking, we report the values from the system card (https:// : http://addoppoa.......m.mma com/ gpt- 5- systsm-stsmmme card. pdf). For AIDE (ol (ol-preview) we use results from the MI.E-Bench repository for all metrics, except for MLF-Bench-30, which we compute from trajectories on the MLF-Bench leaderboard (https: /tttps:/giithie com/ openai/ mle bench/h
Model
All
Low
Medium
High
MLE-bench-30
OpenAl agent, o3, no browsing
OpenAl agent, gpt-5-thinking, no browsing
AIDE Greedy (ol-preview)
16.9 1.1"
34.3 2.4
8.8 L1
10.0 19
12.5
(Ours) AIRA Greedy o3
31.6 = 0.8
55.0 + 1.5
22.0 1.2
21.7
25.8 + 1.3
To provide comprehensive view of performance, report the per-task medal rate in Table 3
E Implementation Details
Our experiments were run on a single-agem setup powered by an NVIDIA H200 GPU, 24 CPU cores of Intel(R) Xeon(R) Platinum 8488C, 100 GB of RAM, and extra TB of local-disk scratch space. Each job had hard wall-clock cap of 24 hours, an execution time limit of 4 hours, and a 5 min grace period. We deployed full-sized DeepSeek-RR R1 model with sglang, and accessed via the litellm API with generation parameters temperature=0, 6 and top-p=0.95.
For experiments done with MCTS, set num children=5, and uct_c=0. .2 unless stated oth- erwise. For evolutionary search, we set the number of candidates_por ieterddtee_ppeeeeeeratiiii generation=5 for con- sistency. For AIRAMCTS, AIRAEVO, and AIRAGREEDY, limit debug cycle total of 10 nodes or 12 hours total time spent debugging (whichever comes first), to prevent spending all the resources on single node.
E.1 MCTS Statistics and Backup
Each node v stores two rumning statistics: the visit count N(v) E N and the empirical mean fitness Q(v) E R. When freshly expanded leaf is evaluated, we initialize N(v)) 1 and Q(vx) = F(Ve), where F is the proxy fitness used throughout the paper. The value F(vr) is then back back- propppprppe along the path P (ve, .oe,..... vo) the root:
Q(u) + Q(u) + F(ve) -(ve)-Q(uu)
Vu E P: N(u)) < N(u) +1,
N(u)
This incremental update maintains the invariant Q(u) N(a
N(W F(v:), i.e. Q(u) is always the mean fitness of all leaf evaluations that have propagated through If node is re-visited later, the same update is applied, allowing Q to converge the true expected value under continued exploration. No simulated roll-outs are performed; the leaf value is taken directly from F.
17

---

Table 3: Per-task performance, grouped by complexity.
Competitin Name
Beunze Medal Rate (%)
Silver Medal Rate (8)
Gold Medal Rate (8)
Year
MLE-bench-30
dateciinp-nnall--in
2013-whule chillange rigi-shule--rdax
85.0
mlsp2101
70.0
45.0
0.0
2013
random: acts of pozza
40.0 11:2
15.0
0.0
2015
dencesing ditt-cocument cocuments
90.0
80.0
00.0
6.9
2013
65.0 + 10.9
15.0 =
0.0 = 0.0
2017
90.0 +
10.0 = 6.9
90.0 = 8.9
2017
spooky sosthe--deniiiaaate
85.0 + 6.2
85.0 +
0.0 =
0.0
2017
eet noraalzztionn challenge english lanuage
30.0 1 11.3
36.0 1 10.9
0.0 1 0.0
2017
cal-claaiiieaiio
45.0 + 11.4
25.0 = 9.9
60
D.O
2017
Sisaw toaie comment cant-llleee clasttiction chillenge
30.3 + 10.5
5. 5.0
0.0
0.0
2018
+ 0.0
=
0.0
2018
2o predict transparent conductors
75.0 1
70.0 10.5
50.0
breed ideniicctioon
1
1:
0.0
scial ideutiicttion
hiskopathddpi wewphhhggg--aan---ddeeee
plam- patbology- 20211-12wc7
95.0
95.0
96.0
raneer catheter line
35.0 +
35.0 = 10.9
35.0
10.9
abular playround series may 2002
1 0.0
0.0 1 0.0
0.0 1.
0.0
2002
MEDIUM COMPLEXITY
Sacebook- meniiing-iis -ii---yyonn--etttiiiio
0.0
2013
muli-mestl- gratteeeccoggioon
0.0 1 0.0
0.0
0.0
2013
billion wood
statil staeil--eheag--l wislee---aaslllleccllllleeeee
0.0
renerflox-spooch meceriion challenge
gas salt
challenge
whale citegatizztion playgrond
35.0
25.0
Sumpo-calareapplii
0.8
2019
inturaliaa10 9---ghh
58.0 11-4
450 11.4
503
2019
0.0
00
2019
kuzeshiji recognition
15.0
00 1
0.0 1 0.0
2019
30.0 10.5
10.0
6.9
0.0 = 8.0
2019
0.0 0.0
0, 8.0
0.0 = 0.0
2020
ima--100-fggee?
0.0 0-0
00 0.0
0.0 D.0
2070
google ques. challenge
95.0 5.0
90.0 5.0
45.0 E 114
2020
herbrrum 2020 lgvec
70.0 1 10.3
30.0 1 10.5
0.0 1 0.0
2020
osic ssl-pulmotaay Mileaaay--leee-peeeeee -ileeiiggtoprrsiic
0.0 + 0.0
0.0 0.0
0. 0.0
2020
wwett-eetimmi---e cel--mmiinn---straciio -estraccinnn
35.0 10.8
35.0 10.9
002 0.0
60.0 + 11.2
45.0 11.4
20.0 9.2
2000
alaskal steganalysis
1
0.01
glani-patlddgy- 202g-2221
100.0
100.0
9
15.0
10.0
2021
1-gvek
2001
question answering
2021
treakthrougt
2021
hubmap- souu----dee---emmeatiiii
2021
honel-d-2221-6 -fgsek
85.0 8-2
40.0 11.2
00
D.
2001
00
2021
AldCode
0.0
0.0
0.0
2002
us paieni-phaase-ee to-gla teglaas-mnntiing
65.0 11.4
45.0 = 11.4
15. 8.2
2022
uu andison gi trac--mage segneettiiiiin
0.0 0.0
0.0 9.0
0.0
8.0
2022
10.0 + 6.9
5.0 5.0
0.0
B.O
2002
herborium 2022 igve?
5.0 5.0
0.0 1 0.0
0.0 0.0
2002
pettinder pae pilarit score
35.0 1 10.9
30.0 1 10.5
15.0
8.2
2002
secube seutiinss in Sosp-
0.0 0.0
0.0 0.0
0.0
0.0
2023
25.0 9.
25.0
25.0:
2034
0.0 0.0
0.0
0.0
2034
COMPLEXITY
+
100.0
3d Sd-object-dees Setpppc-ddeeeirr---- Setecttint-frrrsttmome wwa.---rrrtttoor---ooooo weticee
25.0
15.0 1:
0.0
2019
stafford covod vaocite
65.0 4: 10.9
65.0 = 10.9
65.0 = 10.9
2020
bms melecular transaali
0.0
0.0 1 9.0
0.0 1 0.0
2021
0.0 + 0.0
0.0 = 0.0
0.0 + D.O
2021
predict volcanic emptoon inge-ee
100.0 0.0
100.0 1 0.0
100.0 1 D.0
2001
siim covid (0-demection
0.0
0.0 0.0
0.0
D.O
2001
rua miccal bruis tumor radiogetonic clssslication
30.0 10.5
20.0 9.2
5.0 5.0
2021
0.0 0.0
0.0 0.0
0.0 0.0
2002
weutthhae--eiimttreee. 2022
0.0 0.0
0.0 0.0
0.0 0.0
2022
0.0.:
0.2
2023
all Meattfy
reduce
5.0
casvine-dullangg-in -ittttiii
18

---

E.2 Draft
You Kaggle Grandmaster attending high-stakea competition. Carefully consider the taak description, the size and format of the available data, 4 as well as the available compute resources.
Your goal is to provide EXACTLY ONE IDEA AND ONE CODE IMPLEMENTATION of the idea, different from those previously explored, that leverages the available resources and is likely to lead to strong performance Be specific about each step the proposed approach, including data processing and optimization method, as well as the
the competition.
feature engineering, the modeling - evaluation (USE 5-FOLD CROSS-VALIDATION).
You MUST PROVIDE a solution IDEA/PLAN in natural language and CODE in python that DOES NOT INVOLVE any exploratory data analysis.
TASK DESCRIPTION
((task_desc))
PREVIOUSLY EXPLORED IDEAS
marrkdow
{(memory)}
DATA OVERVIEW
[idata_overviex))
*CDNSTRAINTS
Be aware of the running time of the solution, it should complete within {{execution timeout]]
Prefer vectorized operations over Python loops when processing large datasets. deprecated
Use "torch. oprch. "damii" (the recommended optimizer) instead "AdamW" from "transformers".
Replace the deprecated
rounds' argument in 'lightgbm. train() with the 'lightgbm. beightgbm.early_tttee (tttopinggttoppingg..n..
'lightgbm.train() callback.
If using "timm" models, remember not to prefix suffix the model with - datasets such as "cifar" as this was deprecated.
λs much as possible, keep the stdout clean.
**DATA**: The data is already prepared and available in the read-only '//dtta" directory. You should not unzip any files.
**COMPUTE**: You have access to a Python environemnt with NVIDIA H200 (P)(s) and 24 CPUs available, and the following packages installed: {(packages}}. If you + need to, feel free to use additional libraries that fit the problem. Consider the previously explored ideas,
make sure the idea you propose considers 4 a DIFFERENT ASPECT OF THE SOLUTION, but keep the EVALUATION CONSISTENT.
Brainstorm about possible approaches and WHY THEY ARE LIKELY TO BE EFFECTIVE AND INCREASE THE PERFORMANCE for the given task, and the available data and compute resources.
Remember, and this is important, the first idea should simple and easy + implement, while the last one should be more complex and sophisticated. {% if traft_complexit
== 'simple' %/
In this iteration **focus PROPOSING SIMPLE IDEA:** that serve as SIMPLE YET EFFECTIVE BASELINE for the taak. For example, consider battle-tested methods or (potentially pre-trained. models that are known to work well task at hand.
0% elif sraft_complexity = 'normal' x
In this iteration **focus PROPOSING MORE COMLPEX IDEA:** one that can beat the previous baselines at the cost of complexity and compute. For example, consider leveraging more complex and/or larger (potentially pre-trained) models, apecialized feature engineering, or basic ensambling and/or hyper-parameter optimization.
0% elif draft_complexity = 'complex' %} In this iteration **focus PROPOSING AN ADVANCED IDEA:** that beat the previous baselines at the cost of complexity and compute. For example, consider using specialized (potentially pre-trained) models, leveraging advanced feature engineering or data augmentiation strategies, advanced ensambling and/or or paraneer optimization.
19

---

1% endif X}
RESPONSE FORMAT FOR INPLEMENTATION
Provide **SINGLE** Markdown code block (wrapped in ".") for the implementation Python script that:
- containing a *MSEFF-CONTAINED-
1. Implements the idea **END-TO-END**
2. **PRINTS THE 5-POLD CROSS - score of the evaluation metric
3. **SAVES THE TEST PREDICTIONS* in suubmsssion. cav' file in the current
+ directory
Start by making sure you understand the task, the
compute resources and the - idea. Then generate a detailed implementation plan that will structure and guide
+ you step-by-ster through the implementation process. Make sure to reflect on the plan ensure that the implementation is efficient and faithful to the idea, requirements (e.g., the evaluation score printed, the submission file follows the correct format and saved in the correct location, etc.) are satisfied.
and that
For large datasets, avoid for loops and aim for efficient and fast data loading and feature engineering.
Format the proposed solution follows:
Idea to implement
<the proposed idea/plan>
python
<the implementation of the proposed idea/plan>
E.3 Improve
Introduction:
You are Kaggle Grandmaster attending high-stakes competition. Carefully consider the task description, the size and format of the available data, w as well as the available compute resources.
Your goal is to provide EXACTLY ONE IDEA AND ONE CODE IMPLEMENTATION of the idea,
different from those previously explored, that improves upon existing solution to the task.
Be specific about each step the proposed improvement, including data processing and feature engineering, the modeling and optimization method, as well as the evaluation (USE 5-FOLD MADSS-VALIDATIN).
You MUST PROVIDE improvement IDEA/PLAN in natural language and CODE in python that DOES NOT INVOLVE any exploratory data analysia. DESCRIPTION
{(task_desc))
# PREVIOUS SOLUTION:
## Code:
{(prev_coee}}
## Terminal Output:
(iprev_termina_.ooe
PREVIOUSLY EXPLORED IMPROVEMENT IDEAS
markdown
((memory))
DATA OVERVIEW
CONSTRAINNTS
Be aware of the running time of the solution, it should complete within ((execution_ timeottt)_
Python loops when processing large datasets. Use "torch. "torch.optinn optim. Adami" (the recommended optimizer) instead deprecated + "AdamW" from "transformers".
Prefer vectorized operations
20

---

Replace the deprecated "early_stoppe baarly_soopping_rooe rounds" argument in "lightgbm. train - the "Lightgbm.eard" early_stopiing (stopping_rounde
callback.
If using "timm" models, remember not to prefix or suffix the model names with 4 datasets auch as ciffar' as this was deprecated.
As much as possible, keep the stdout clean.
*DAAA*** The data is already prepared and available in the read-only '/dataa
directory. have access to a Python environemnt with NVIDIA H200 GPU(s) and + 24 CPUs available, and the following packages installed: ([packages)). If you + need to, feel free to use additional libraries that fit the problem.
should not unzip any files.
COMPUTE***
Consider the previously explored ideas, and make sure the improvement idea you propose considers DIFFERENT IMPROVEMENT OF THE SOLUTION, but keep the
EVALUATION CONSISTENT.
Brainstorm about possible improvements and WHY THEY ARE LIKELY EFFECTIVE AND INCREASE THE PERFORMANCE for the given task, and the available data and compute resources.
(% if improve_comm complextcyylliittyycccocy =- 'simple' %}
In this iteration, suggest *minimal, low-risk* tweak that keepa the current solution's s core intact-no architecture overhauls or fundamental methodology changes.
Think: a Yeature-engineering twist, a lightweight data-augmentation trick, or hyperparameter changes.
Check the MEMORY section first and avoid duplicating earlier ideas. (% elif perove_complexite
'normal' %}
In this iteration, propose a #moderate upgrade* that builds on the baseline without
+ deviating dramatically.
Options include (but not limited to) haper-parameter tuning, a amall ensemble of
similar models, sturdier preprocessing pipeline, feature engineering improvements, and data augmentation.
Check the MEMORY section first and avoid duplicating earlier ideas. (% elif improve_come corrove_cmmllxtttt In this iteration, recommend a #substantial extension* that pushes the method's 's boundaries while preserving its core logic.
'complex' %}
fine-tuning specialized pre-trained models, Consider advanced ensembling/sttckking 4 or exhaustive hyper-parameter searches.
Check the MEMORY section first and avoid duplicating earlier ideas. 5% endif
*ERSSONSE FORMAT FOR IMPLEMENTATION**
Provide a **SINGLE** Markdoun code block (wrapped in >*) containing a *+SELF-CONTAINERS Python script that:
Implements the idea **END-TO-END**
**PRINTS THE 5-FOLD CROSS-VALES VALIDATION score of evaluation metric
3. **SAVES TEST PREDICTIONS**
suumissioon.cm cav" file the current
+ directory
Start by making aure you understand the task, the data and compute resources and the idea. Then generate detailed implementation plan that will structure and guide you step-by-step through the implementation process. Make sure reflect the plan to ensure that the implementation efficient and faithful to the idea, and that all the requirements (e.g., the evaluation score is printed, the submission file follows the correct format and is saved in the correct location, etc.) are satisfied.
For large datasets, avoid for loops and aim for efficient and fast data loading and w feature engineering.
Format the proposed solution as follows:
# Improvement Idea to implement
<the proposed improvement idea/plan /pla>>
pyyhon
<the implementation of the proposed improvement>
21

---

E.4 Analysis
Introduction:
You Kaggle grandmaster attending competition.
have written code to solve this task and now need to evaluate the output of the code execution.
You should determine if there were any bugs as well as report the empirical findings.
Task Description:
([task_desc))
Implementation:
{(code)}
# Execution output:
((executiion
E.5
Debug
Introduction:
You are a Kaggle Grandmaster fixing code bugs in a high-stakes competition solution.
Carefully review the previous debugging attempts, the buggy code and its terminal output in addition to the given task/data details, and available compute resources.
You must not change the core idea methodology of the solution, but only fix the bugs in the code.
Task Description:
markdoww f(task_desc))
(% if memory %} Previous debugging attempts: "markdown {(meoory}}
{% endif %}
Buggy Implementation:
{ (tree_bbugy_ccod))
Execution Output (Error):
{ <execution_outprt
Data Overview:
(data__verviee))
Compute Environment:
22

---

GPU: NVIDIA H200
CPUs: 24
Available Packages: {packages))
Additional libraries allowed as needed.
Instructions:
**Do NOT** alter the core method or underlying idea. Only correct
existing bugs.
Outline your bug-fix plan clearly in 3-5 concise sentences.
Provide a single, complete Python code block wrapped in markdown (- ppyhonn that:
Implements the fix fully.
Calculates and clearly prints the evaluation metric using validation set (use 5-FOLD CV if suitable).
Generates "submission.cr csv file with test set predictions stored **current directory** (..suumission.com csvis csv'),
fully self-contained and executable as-is (The entire bug- free solution is given).
Important Reminders:**
Absolutely do **NOT** skip any part of the code.
Always ensure predictions the provided unlabeled test set saved in "./submission.csv" This is crucial for grading.
# Other remarks
Huggingface is set to OFFLINE mode by default. If you firmly believe that the issue is not having the requested model in the cache, please it to ONLINE mode by setting both the environment variables
"HF_HUB
and "TRANSFORMERS OOFTRMEERSOFFFIE-OOOO on top of your code, by importing and using "os. environ[.
Do not set/force Huggingface OFFLINE mode that will NOT fix any issue.
When model cannot be found in the "timm" library, it might be useful to 'print hii.mmlssttmmmssssemmmem
If using "timm" models, remember not to prefix suffix the model names with datasets such as 'cifar' as this was deprecated. Brainstorm about possible ways to fix the bug and WHY THEY ARE LIKELY TO any other
FIX THE BUG for the given implementation. Additionally, bugs further down the line are observed, please fix them well.
Generate bug-fix plan that will structure and guide your step-by-step reasoning process. Reflect it to make sure all the requirements are satisfied.
Format the proposed bug-fix plan and code as follows:
Bug Fix Plan
<bug-fix plan>
python <the fixed python code>
E.6 Crossover
CROSSOVER Operator For or more valid artifacts, the Crossover operator performs merge of existing solutions. It specifically aims to create solution drawing insights from the best aspects of its parents, while also taking into account the shared task context (see Section 2.3) that informs and guides the merging process. For each crossover operation, the two parent nodes are sampled from the distribution of all nodes' fitness scores.
Introduction:
23

---

You are Kaggle Grandmaster attending a high-stakes competition.
Your goal is to combine two previously developed solutions to further
4 increase performance on the given task.
Carefully consider the task description, the two provided solutions, the available data, and compute resources.
You need to devise a plan to merge or integrate these solutions and then implement it.
# TASK DESCRIPTION
{{task_decc)}
0% if memory %3
# PREVIOUSLY EXPLORED CROSSOVER/ COMBINATIOOOO IDEAS markdown
{ {memory)>
(% endif %
PREVIOUS SOLUTION
## Code:
python
{ {(prev_code1}>
# PREVIOUS SOLUTION 2:
## Code:
"python
[(prev_code2))
# INSTRUCTIONS:
Your main task is to:
1. Propose **Crossover Plan** in natural language explaining how to combine Solution and Solution 2.
2. Provide **Python Code Implementation** of this plan.
Consider any previously explored ideas from the MEMORY section. Brainstorm how the two provided solutions can effectively combined and * THIS CROSSOVER IS LIKELY TO BE EFFECTIVE** and increase performance for the given task, data, and compute resources. Aim for consistent evaluation method (e.g., 5-FOLD CROSS-VALIDATION, unless the task specifics dictate otherwise).
CDNSTRAINTS**
Be aware of the running time of the solution, it should complete within eexxecution_immoute Prefer vectorized operations over Python loops when processing large datasets.
Use "torch. AdamWi'
" (the recommended optimizer) instead of the deprecated "'dawW" from "transformers".
Replace the deprecated brightgbm.trrinn
argument in
with the Frightgbm.early.oo wttp:/ttppgggttttpggggooooooo
callback. If using "timm" models, remember not to prefix suffix the model names with datasets such as "cifar" as this was deprecated.
24

---

much as possible, keep the stdout clean.
**DATA**: The data is already prepared and available in the read-only '/data" directory. You should not unzip any files.
COMPUTE**: You have access to a Python environment with 1 NVIDIA H200 (PU(s) and 24 CPUs available, and the following packages installed: ((pacaggs))). If you need to, feel free to use additional libraries that fit the problem.
Start by making
you understand the task, the provided solutions, the data and compute resources, and your proposed crossover idea. Then, generate detailed internal implementation plan that will structure and guide you step-by-step through the implementation to reflect the plan ensure that the implementation is efficient, faithful to the crossover idea, and that all requirements (e.g., the evaluation score is printed, the submission file follows the correct format and is saved in the correct location, etc.) are satisfied.
process. Make
RESPONSE FORMAT FOR IMPLEMENTATIONN*:
Provide a **SINGLE** Markdown code block (wrapped in ">) containing a #SELF-COOTAINNE=> Python script that: 1. Implements the idea *MEND-TO-END4
2. **PRINTS THE 5-FOLD CROSS-VALIDATIONN+ score of the evaluation
+ metric
3. **SAVES THE TEST PREDICTIONS** in 'submission.cav' file in the current directory
Format proposed solution as follows:
# Crossover Plan
<Your proposed crossover plan/strategy>
python
<the implementation of the crossover solution>
Completion Tokens Per Method
In Fig. 10 we summarize each method's average number of completion tokens per operator. The analysis suggests that our changes to the operators (see Section 4.1) lead to substantially longer thinking chains.
G Infrastructure Lessons
The infrastructure design of AIRA-dojo was informed by a few reliability and performance constraints:
1. LLM Service. As we scaled experiments, we observed that external L.LM services slow down, risking timeouts. While the exact rate limits existence nevertheless places an upper bound on the number of parallel actors and introduces point of failure. Self-hosting LLMs is therefore required for reliable scaling.
by the API provider [ their
https://platform..cm http:///w...aaaaaaeooo/////////e
25

---

draft
debug
Mean Completion Toleess
7000
11303
0090.
EC453
$4321
5000
4000
2544
2000
1000
A00
P40 k1
ceentty R3
MCTS
HoR 82
Eve Rd
Gmerdy Rd
MCTS
cready R3
HCTS
Method
Method
Method
Figure 10: Number of completion tokens per operator for each agent. Each value is averaged across the independent runs on the MLE-bench lite suite in the main experiments (see Section 5.2)
2. Environments. Early experiments resulted in agents corrupting Python environments e.g.. via pip install. Virtualization technology, such as containers, was therefore a natural fit for state isolation.
3. Checkpointing. In line with prior work 22
observe that both hard (machine failure, filesystem failures) and soft failures (slowdowns) are commonplace. Consider, for example, 10 experiments, each with 100 agents and each running for 24 hours that results in 10 x 100 x 24 = 24000 hours of required uninterrupted uptime, which significant when compared Mean Time Failures of 1000 hours per node (25). We therefore introduced checkpointing support to mitigate the effects.
10 25
H Rationale for Selection of Search Policies
To systematically evaluate search policies we selected three complementary approaches. AIDE's greedy tree-search policy serves as an efficient baseline that prioritizes exploitation. Monte Carlo Tree Search (MCTS) extends this by allowing us to directly modulate the exploration-exploitatio tradeoff through single parameter. In contrast, the evolutionary graph-based search policy leverages population-ban sampling and recombination, representing a fundamentally different strategy from both greedy and tree-based methods.
I Limitations and Future Work
There are several important dimensions for performance that we leave for future work to explore.
Agentic operators. Our experiments suggest that the effectiveness of search is heavily dependent on the capability of the operators. To manage complexity, in this work we experiment with LLM-based operators. However, one could readily use full- full-fledged agents operators. For example, natural extension would be to include an ideation agent as an operator
40). and to replace the implementation
and debugging operators with SWE-Agent [30 50
Finetuning. LLMs are critical components of the operators. Future work could investigate supervised fine-tuning or reinforcement learning as methods to enhance operator effectiveness.
Scaling the search. To maintain comparability with MLF-bench, adopt the benchmark's time (24- hour limit) and compute constraints (1 GPU). However, solving challenging problems likely requires substantially greater resources, and evaluating performance under these restrictions provides limited insights (see Section C for an experiment extended to 120 hours-5 days). Studying the scaling behavior of search policies and operators, and developing agents capable of effectively leveraging more computational resources over longer time horizons, is an important direction for future research.
Data contamination. Finally, there is the issue of data contamination. It is possible that results are influenced by the presence information related to the evaluated Kaggle tasks, or similar tasks, within the LLM training data. Creating continuous stream of fresh and novel tasks remains important research challenge [47]
26

---

Variance in the Performance Estimation
Al agent performance on complex benchmarks like MLE-bench exhibits substantial variance that can severely impact the reliabiliity of comparative evaluations. While MLE-bench's 's official recom- mendations suggest using at least seeds for evaluation, this may be insufficient for reliable agent rankings and performance estimation, particularly given that agents/LLMs can be quite high-varian high-variance inherently.
Sample
Sample 2
Sample 3
Sample
Mcan with Seeds
95% C
seeds
5 seeds
53%
43%
LALAU AlDhowsy
seeds
355%
6.26
9.%%
25%
376
Figure 11: Potential algorithm rankings under different seed counts. This figure demonstrates how the same algorithms could be ranked differently if fewer seeds were used, illustrating the instability that results from estimating performance with insufficient samples. While confidence intervals theoretically capture this uncertainty. researchers often underestimate how dramatically rankings can shift with limited seeds. Each panel shows plausible alternative rankings that could emerge from the same underlying algorithm performance distributions. We recommend using minimum 10 seeds task for moderate ranking stability, with 20 seeds preferred to avoid misleading conclusions about relative algorithm performance.
Figure 11 demonstrates how dramatically agent rankings shift with insufficient seed sampling. Each run represents sample from the underlying performance distribution (estimated from 20 seeds), and with only few seeds, observed performance differences may be statistical artifacts rather than genuine capability differences. Given MIFE-bench's 's computational intensity- with 75 competitions requiring substantial resources per run-mos rescarchers face critical trade-off. If options were evaluate on all 75 competitions with 3 seeds each or evaluate on 22 competitions with 10 seeds each, the latter provides more reliable conclusions. While the former experimental setup offers broader coverage, the individual competition results are unreliable, making it difficult to determine whether an agent genuinely excels at specific types of ML engineering tasks. Thus, we prefer enable confident identification of an agent's 's strengths and weaknesses across representative subset of competitions.
For future work recommend a minimum of 10 seeds per competition (with 20 seeds much more preferred), stratified bootstrapping for confidence intervals rather than standard error estimates, and focus competition subsets (e.g., MLEE-bench Lite) with higher seed counts rather than full evaluation with few seeds.
27

---

ARoo
21000
0.000
A
KAU
0000
Mepo prppeeee
Ioay prevecc
Dramy
Variance Decomposition
(b) Task Outcome
Figure 12: Sources of performance variance across tasks and seeds. (a) Variance decomposition shows that substantial portion of the observed variance in agent performance arises from between- task variability (across tasks). However, certain methods' medal achievements are equally as variable within task. (b) Distribution of medal outcomes across tasks for each agent, showing how frequently a method consistently performs well (always medals), consistently fails (never medals), or exhibits inconsistent performance (sometimes medals). These figures underscore that agent evaluation MLF-bench is impacted both by stochasticity in task-level performance and by systematic variation task difficulty agent specialization.
K
Test- Validation Gap Results
we present the performance profile of AIDEGREEDY. In Fig. 13 we show the performance profiles of AIRAMCTS, AIRAEVO, and A AIR AGREEDY. Compared to Fig. 4a the agents using
In Fig. 4a
AIRA operators display smaller gap between test and validation scores.
Per-Task Results
This section presents the non-aggregated, per-task results obtained from the MLE-bench Lite suite in The results are summarized in Fig.
Section 5.2
M
Sample Search Trees
We present samples of search trees from various methods used in the aoooky--uthorrideeiifficc.
task, as shown in Figures 15
to 17 The colors of the
nodes indicate the validation scores, while the labels within the nodes display the test scores. We represent winning nodes with medal emojis and above-medin ranking solutions (relative to the human leaderboard) with an ok emoji. The nodes red are buggy nodes.
28

---

60%
60%
Medal Rate
40%
40%
Medal
30%
[Validation)
30%
(Vaildation)
AIRA o3 (Test)
AIRAucts c3 (Test)
R1 (Valldation)
R1 (Validation)
AIR R1 (Test)
AIRAucts R1 (Test)
10%
10%
Wall-clock time (hrs)
Wall-clock (hrs)
(a) Evolutionary
(b) MCTS
70%
50%
40%
30%
AIRA o3 IValidation)
AlRAw (Test)
20%
AlRAaves (Vaidatioo)
AlRAg R1 (Test)
10%
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Wall-clock Lime (hrs)
(c) Greedy
Figure 13: Perceived actual medal rate over 24 hours of searching with the AIRA operators using different search policies. The curves show the mean validation (agent-reported) and held-out test medal rates across 20 seeds with R1 and 10 seeds o3 for all tasks. The widening band illustrates the generalization gap, revealing how apparent gains on the validation set can mask overfitting and ultimately undermine the search process.
29

---

117
Jry 04
180
PACE SE0
Figure 14: Par-task performance. Methods utilizing RI are averaged over 20 seeds per task and methods utilising o3 averaged over 10 seeds per task
30

---

0.64
0.54
0:22
0,27
6
II
6.29
0.29
0.28
0.31
0:54
0,28
c.21
0.37
0.54
0.51
11
0.29
Figure 15: AIRAGREEDY
31

---

0.56
DEV
C20
C20
0.33
C2T
028
2AB
0.2
Figure 16: AIRAEVO
32

---

64
6:39
030
231
D33
024
62
02
233
12
Figure 17: AIRAMCTS
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Formalize Al agents as two axes, search policy and operators: Section
Design an improved set of operators: Sections 2.2 and 5.2
Develop AIRA-dojo: Section 3.1
How the generalization gap affects performance: Section 5.3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section is dedicated Limitations section in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
34

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides a detailed description of the implementation of agents as well as the key features of AIRA-dojo in sections and 4 A complete list of experimental details is provided in the Appendix for reproducibility. We have made the AIRo-dojo codebase publicly available at htttt:://// :ttp:::///ttttiee giggttt..b... comm/accebokreeeerccch hatpe/aara--iia dojo
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
35

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: AIRA-dojo is open-sourced and available https: betpss//github... com/ facebookressaarch/ aira-dojo
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ http://cpppblcgguuuudcuu httpa///ggguiee////eeuuuuibisss
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: General experiment details are presented Section Specific details related to each experiment are provided in Section 5 Our experiments do not involve any training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report % stratified bootstrap confidence intervals and repeat each experiment using 10 different random seeds per task to ensure robustness and statistical significance. For experiments claiming new statc-of-the-art numbers on MLE-bench, specif- ically in Fig. and Fig. 5 we use 20 seeds per task to enhance reliability.
Guidelines:
36

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96 % CI, if the hypothesis of Normality of errors
verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 3.4 presents these experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eaarppp.ccpc/p/b//////
Answer: [Yes]
Justification: We have reviewed the linked NeurIPS Code of Ethics and can confirm that the research conducted in the paper conforms to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
37

---

Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite and credit creators and original owners of all the assets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
38

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Section Section the AIRA-dojo codebase, which is open-sourced at lhttpp htt.ab/..co/aaccoooeeeeeerrrraaa
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
39

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Our method uses LLMs (some operators are LLM based), the implementation of our agents did involve the of LLMs in original, important non-standard way for AI driven automated machine learning engineering, to tackle MLE-bench / Kaggle tasks. However, did not use any LLMs in the creation of the research ideas, hypotheses, or data analysis.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy httpps///eerrppeccc////ooeeeeeep for what should or should not be described.
40


Reviews:
[{"review_id": "xCMYqpn8cr", "reviewer": "Reviewer_z7Ae", "summary": "This paper provides a new perspective where the design of agents is 1) Search policy, 2) Operator that modifies the current solution. \nA formalization is provided where AIDE is re-interpreted under this new lens. \nThey also provided AI Research Agent (AIRA) Dojo, a software/framework that allows people to implement new operators and search policies.\nThe paper also found a generalization gap.", "questions": "N/A", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is indeed well-written and easy to follow.\n2. The re-framing is a good attempt to provide insight into this field.\n\nWeaknesses:\n\nI believe this is a borderline paper. I can either vote to accept or reject, but will not take a strong stance to vote either way. The paper is well-written, and the experiments are solid. However, the paper puts a lot of emphasis on the **reframing** of AIDE and agents. Then, whether I personally is convinced by the **reframing** is the key to whether I should vote to accept or reject this paper. \n\nI do not find this reframing convincing. My biggest confusion is -- for an agent that has access to tools (such as terminal/internet), what is the search policy and operator? The paper seems to want to merge two separate conceptual models: agent and inference-time search. I think they are different. The authors seem to think they are the same. I'm open to having a longer discussion with the authors during the rebuttal.\n\nThe paper can also use better citations, such as [1] and [2]. The overfitting phenomenon was also discovered by Anthropic last year [3], but proper citation was not provided.\n\n[1] https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/ and the paper: Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P. S., Wagner, A. Z., ... & Balog, M. (2025). AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131.\n\n[2] Pan, Jiayi, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. \"Learning adaptive parallel reasoning with language models.\" arXiv preprint arXiv:2504.15466 (2025).\n\n[3] Pan, Jane, He He, Samuel R. Bowman, and Shi Feng. \"Spontaneous Reward Hacking in Iterative Self-Refinement.\" arXiv preprint arXiv:2407.04549 (2024).", "was_revised": true, "final_justification": "My score is final -- which is an indication of my preference. Since all the other reviewers believe this is a good paper, as research is a community-driven effort, I won't object to sharing this paper with the broader public to inspire future work.", "rebuttal": "We thank Reviewer z7Ae for describing the paper as “well-written and easy to follow,” the experiments as “solid,” and the reframing as a “good attempt to provide insight into this field.”\n\nThe reviewer's concerns center on the relation between research agents and inference-time search, and how ReAct-like LLM-driven agents with access to tools fit into the framework. We acknowledge that understanding how these types of agents fit within the framework is very useful and important for a reader. Given that the reviewer has confirmed the technical soundness of our experiments, we will focus on addressing the conceptual concerns and clarifying the practical significance of our work.\n\n> **Reviewer's Concern:** \"The paper seems to want to merge two separate conceptual models: agent and inference-time search.\"\n\n\nResearch agents aim to automate the scientific research process by **taking actions** that allow them to effectively **(re)search** the possible space of solutions, by iteratively coming up with and testing hypotheses that best **inform the subsequent actions**, until they reach a satisfactory solution (or run out of time). Conceptualizing this process as search is natural, and allows for a systematic study of the exploration-exploitation tradeoffs inherent to search and extensively studied in the literature (see next section). Our work formalizes this perspective by modeling research agents as graph-based search algorithms. Crucially, this abstraction does not impose constraints on the implementation of the agent and the agents’ action space (see next section)—a node is a conceptual object that can span many actions and an operator can be another fully-fledged agent that performs many steps (see Complex operators in Section 7.1). In fact, in our implementation, nodes are composed of multiple operator calls (e.g., the core operator such as debug or improve and the analysis operator that extracts information from the execution output).\n\nWe do not use the term “inference-time search” in the paper, and if the reviewer uses it to refer to \"performing search\" internally by generating more thinking tokens, then we are not arguing that these are the same. In fact, this is not agentic, and is strictly complementary: we are relying on reasoning models in our experiments, and we are agnostic to how they use the \"thinking tokens\". We hope that this discussion clarifies the relation between research agents and search in the context of the proposed formalization.\n\n\nDoes the reviewer have any additional questions regarding this concern, or believe that a reader would benefit from including some clarification in the paper?\n\n> **Reviewer’s Concern: \"For an agent that has access to tools (such as terminal/internet), what is the search policy and operator?\"**\n\nConsider a standard ReAct-like LLM-driven agent with access to tools solving a coding problem on MLE-Bench. There are many ways to conceptualize such agents in our framework, depending on the definition of the artifacts set that define the nodes and the operator. Here we provide **an example instantiation of a ReAct agent in our framework**:\n\n**Nodes:** (Partial) Solution. We could keep this as in the paper’s experiments where each node represents a candidate solution (code file) and some meta-data. \n\n**Operators (O):** A single operator that generates the next solution (based on all of the previous information in the context; note that the context will grow relatively quickly) wrapping the logic of the ReAct agent. The agent could take several actions (note that we make no assumption about the low-level action space)—observe the current state, reason about the next action, execute a tool (terminal/internet), and update the state—until it generates the next candidate solution.\n\n**Search Policy (π):** \"Linear search\" over N steps. At each step, the search policy selects the last node for expansion, or decides to terminate the search. This is a trivial search policy. **This type of agent does not use the search policy as a sophisticated means of modulating the inherent exploration-exploitation tradeoff.** There is a single operator, so the operator policy is trivial in this case.\n\n**Key insight:** Even \"simple\" tool-using research agents implement a search: a systematic exploration of the solution space through a sequence of actions toward finding a solution to the problem. The only difference is that they use “simple linear search\" rather than more sophisticated search policy like MCTS. \n\n**Why is this significant?**\n\n**The proposed formalism enables systematic analysis. Our experimental results expose bottlenecks in research agent design and identify directions for improvement for future work.** \n\nSpecifically, the framework allows us to describe the agent’s behavior in terms of fundamental components that shape its behavior: 1) the **search policy:** deciding where resources will be allocated; 2) the **operator set**, the high-level functions that take in existing artifacts and produce new ones, and the **operator policy**; 3) the evaluation mechanism used to score artifacts and guide the search towards promising directions. This allows us to compare agents that are seemingly not comparable and disentangle the impact of specific design choices. In particular:\n- **AIDE** = greedy search with specific draft, improve, and debug operators\n- **AlphaEvolve** = evolutionary search with specific mutation and crossover operators\n- **ReAct agents** = linear search with a specific code-generation operator\n\nIn our work, we focus on the previous state-of-the-art AIDE and LLM-based operators: changing the search policy while keeping the operators fixed, and vice-versa. **Our comprehensive analysis exposes a complex interplay between the search components**. For example, components can act as bottlenecks to other components (e.g., operators or overfitting can bottleneck potential gains from search strategies), the overfitting manifests differently for different search algorithms, and the exploration-exploitation tradeoff can be modulated by the operators or/and by the search policy. \n\n**We appreciate the reviewer's offer for extended discussion.** \n\n**Missing Prior Work Citations**\n\nWe thank the reviewer for flagging these missing citations, and have incorporated the suggested citations into our revised manuscript:\n- **[1] AlphaEvolve & [2] Pan et al. (Adaptive Parallel Reasoning):** These works illustrate the framework's applicability:\n  - **AlphaEvolve** = evolutionary search policy + sophisticated code-mutation operators *(Submitted on 16 Jun 2025)*\n  - **Pan et al.** = advanced parallel search policy with dynamic resource allocation *(Submitted on 21 Apr 2025)*\n\n- **[3] Pan et al. (Reward Hacking):** Thank you for making us aware of this paper. We have added this reference to our discussion of the generalization gap. The phenomenon they term \"Spontaneous Reward Hacking\" relates closely to the overfitting we observe. Our MLE-Bench results provide additional empirical evidence of this phenomenon in automated AI research, confirming and extending their insights to this domain as well as seeing the effect of different search policies on this overfitting.\n\n\nOverall, we contend that this work provides **conceptual** (the formalism) and **practical** (AIRA-Dojo) tools for advancing AI research automation. The insights from the paper support the value of these tools. \n\nThe reviewer’s “borderline” rating and “poor” significance are centered around the conceptual framing and generalization. The reviewer acknowledges our solid experiments and clear writing, and we hope that the conceptual clarifications above help to address the generalization concern and contextualize the significance of our contributions. If that is the case, we respectfully encourage the reviewer to consider revising their assessment.\n\nWe welcome further dialogue of any remaining concerns about the conceptual framing, or how agent architectures fit into the framework. We sincerely thank the reviewer for helping us make the paper better."}, {"review_id": "vcn5xRiYgt", "reviewer": "Reviewer_QQcd", "summary": "This paper frames AI research tasks as a search problem and breaks down the design of AI research agents, such as AIDE, into two core components: search policies and operators. The authors build on this formulation to conduct a thorough ablation study, systematically evaluating the impact of different design choices for these components. The best configuration identified in the study achieves state-of-the-art results on the challenging MLE-Bench, improving the success rate for earning a Kaggle medal from 39.8% to 47% on MLE-Bench Lite. This result underscores the practical significance of their findings and highlights the value of carefully designing and evaluating agent components to advance automated machine learning research.", "questions": "- Figure 1a’s x-axis label “any medal” is confusing and unclear.\n- Line 93 states that “fitness function F is defined for each node by the operator that generates or modifies it,” but Line 211 claims “each agent employs the same proxy-fitness function F.” These statements appear contradictory. Which is correct?\n- Line 95 notes that “all operators are LLM-driven except for the MEMORY operator, which is defined by hand.” Does “LLM-driven” mean the operators use LLMs during execution or were generated by LLMs? Additionally, does \"the MEMORY operator is defined by hand\" imply that all other operators are LLM-generated (I guess not)?\n- Line 105’s description of the MEMORY operator is unclear. If it does not involve an LLM, how is it implemented? If it does, this contradicts Line 95’s claim.\n- Line 184 indicates that [DOJO] results use the R1 model, while [OAI] results use the O1 model. The use of different models introduces a confounder, making it difficult to draw conclusive comparisons.\n- Line 194 suggests that a larger number of children (nc) requires more advanced code complexity. Why is this the case? Additionally, how can a DRAFT node have children, given its role in initiating solutions?\n- Line 259 omits the exploration constant value c=0.5.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. The paper achieves a notable advancement by establishing a new state-of-the-art performance on MLE-Bench Lite, boosting the Kaggle medal success rate from 39.8% to 47.8%. This improvement highlights the practical impact of the proposed approach and sets a strong benchmark for future work in automated machine learning.\n\n2. The authors provide an insightful perspective by decomposing the design of AI research agents into two key components: search policies and operators. Their comprehensive ablation studies offer valuable insights into how these components contribute to agent performance, paving the way for more systematic design of AI research agents.\n\n3. The release of the codebase will be a significant strength, enabling other researchers to build upon this work and explore new designs for AI research agents. This commitment to open science enhances the paper’s impact and fosters further innovation in the field.\n\n### Weaknesses\n\n1. A primary concern is the absence of statistical significance tests to support the authors’ claims. For instance, in Line 263, the authors conclude that the search policy has minimal impact based on the small performance margin shown in Figure 4a. However, in Line 289, they argue that a slightly larger margin in Figure 3a suggests a meaningful interplay between search policy and operator design. Both conclusions are questionable, as the margins may fall within the error bars (as seen in Figure 3a) and lack statistical validation. Furthermore, the exploration of operator designs is limited, with only a brief comparison between OAI and DOJO, which is confounded by the use of different models, undermining the conclusiveness of the analysis. Including rigorous statistical tests and a broader exploration of operator designs would strengthen the paper’s findings.\n\n2. The analysis in Section 5.3 raises concerns about the interpretation of the validation-test gap. The authors suggest that the model accessing such data represents an upper bound on performance. But both the validation and test sets are equally held out from the training data, with the only difference being whether they are included during the AI research agent search. This framing misrepresents the gap’s significance, as the test set is not inherently \"ground-truth held out data\". Additionally, the method described in Line 321 relies on test set data to report the best-performing scores, which is not feasible in real-world scenarios where test data is inaccessible during model development. I suggest removing this section from the paper.", "was_revised": true, "final_justification": "I think the authors resolve most of my concerns by providing more experimental results and detailed explanations to my questions, especially the newly added significance tests, which enhance the convincingness of the empirical conclusion, in my opinion.\n\nI am still a bit concerned about the \"validation-test\" gap, but I believe that my other major concerns are addressed, and the remaining concern does not affect the major argument of the paper. I will increase my score to 5 and recommend the acceptance of the paper.", "rebuttal": "We appreciate Reviewer QQcd's thoughtful review and are pleased that they recognized our state-of-the-art performance on MLE-Bench Lite, our insightful decomposition of AI research agents into search policies and operators, and the value of our open-source commitment. We're also encouraged by their acknowledgment that our comprehensive ablation studies offer valuable insights into how these components contribute to agent performance. We address their concerns below.\n## **Statistical Significance Testing and More Analysis**\n**We completely agree** that statistical tests would add more rigour to our conclusions. We have now performed Mann-Whitney U tests [1] to demonstrate the probability of improvement [2] of a method X over a method Y. When confidence intervals do not include 0.5, it indicates a statistically significant result [2]. We see that when comparing agents that utilize the AIDE operators, the probability of improvement is not statistically significant for any comparison besides MCTS c=0.5 where AIDE Greedy is better. Additionally, when comparing the AIRA operator agents, we see a statistically significant improvement with all AIRA agents except for the evolutionary search policy. **We will add these statistical significance results to the revised paper** to strengthen claims about performance differences between methods. These tests do not take the magnitude of improvement into account, only whether a method is more likely to perform better on a random task and random seed.\n\n**To address the comparison between OAI and DOJO, we conducted an experiment reproducing AIDE_greedy with o1-preview in AIRA-dojo**.\nSpecifically, our implementation of AIDE, [DOJO] AIDE_Greedy o1-preview, improves the medal rate from 35% to 46% over the reported results, [OAI]  AIDE_Greedy o1-preview.\nThis corresponds to state-of-the-art performance with a relative improvement of 30% in medal rate. \n\nWhile evaluating all agents with o1-preview would have been informative, we were only able to complete the AIDEgreedy experiment before the model was discontinued.\nHowever, **we additionally conducted experiments with OpenAI’s newest model in the series, o3, and observed similar results.**\n\n|Agent X|Agent Y|P(X > Y) Medal Rate|95% CI|\n|-|-|:-:|:-:|\n|AIRA_Evo o3|DOJO AIDE_Greedy o3|0.520|[0.491, 0.550]|\n|AIRA_MCTS o3|DOJO AIDE_Greedy o3|0.539|[0.511, 0.566]|\n|AIRA_Greedy o3|DOJO AIDE_Greedy o3|0.541|[0.511, 0.568]|\n|AIRA_Evo R1|DOJO AIDE_Greedy R1|0.516|[0.495, 0.535]|\n|AIRA_MCTS R1|DOJO AIDE_Greedy R1|0.536|[0.517, 0.557]|\n|AIRA_Greedy R1|DOJO AIDE_Greedy R1|0.528|[0.509, 0.548]|\n|DOJO AIDE_Greedy R1|OAI AIDE_Greedy o1-preview|0.524|[0.505, 0.543]|\n|DOJO AIDE_Greedy o1-preview|OAI AIDE_Greedy o1-preview|0.555|[0.530, 0.581]|\n|AIDE_EVO R1|AIDE_MCTS c=0.75 R1|0.519|[0.492, 0.546]|\n|AIDE_EVO R1|AIDE_MCTS c=0.0 R1|0.510|[0.484, 0.537]|\n|AIDE_EVO R1|DOJO AIDE_Greedy R1|0.493|[0.469, 0.516]|\n|AIDE_MCTS c=0.75 R1|AIDE_MCTS c=0.0 R1|0.491|[0.468, 0.514]|\n|AIDE_MCTS c=0.50 R1|DOJO AIDE_Greedy R1|0.476|[0.453, 0.497]|\n|AIDE_MCTS c=0.0 R1|DOJO AIDE_Greedy R1|0.483|[0.461, 0.503]|\n\n### **Section 5.3 Validation-Test Gap Analysis**\nWe appreciate your feedback regarding Section 5.3. We acknowledge that the motivation and implications of this section could be stated more clearly. We apologize if this did not come across as intended. We will clarify its importance and criticality for scaling research agents here, and we will update the writing to better reflect this for all readers.\n\n**As the reviewer points out, both the validation and the test set are sampled from the same distribution, and the main difference between the two is the fact that the agent uses the validation set during the search process. This is a critical distinction, which our experiments show leads to overfitting to the validation set.** Specifically, every time the agent queries the validation score and uses it to guide the search, it risks overfitting to the validation set due to finite sample effects. Because of that, the performance on the validation score seizes to be a good measure of performance, as suggested by Goodhart’s law.\nHowever, the test set, which is completely held-out, allows us to accurately estimate the actual expected error of the model, and study the gap between the perceived and actual expected error—the overfitting to the validation score during the search process. \n\n**How this gap evolves over time is critical for scaling AI research agents.\nThe main point of this section is to empirically study the validation-test gap in the context of AI research agents.** We quantify the gap over time, show how it differs across different search policies, how it affects the effectiveness of different agents, and demonstrate overfitting. Unless researchers address this overfitting in the search process, scaling agents will be impossible. To the best of our knowledge, our study is the first to study this fundamental phenomenon in machine learning in the context of research agents.\n\n**Regarding Line 321 feasibility:** The section explores the maximum achievable (actual) test performance by selecting the top-k solutions based on the (perceived) validation scores. **This is useful and actionable information in many settings.** First, we could keep a held-out test and evaluate the top-k models on a held-out set and choose the best. Second, we could deploy several models in production, evaluate their performance in a small scale and choose the best (e.g., perform an A/B test). Finally, we could show several candidate solutions to machine learning experts that can to some extent account for overfitting, until we develop methods to do this automatically. \n\nThis setting is even supported in Kaggle, where 2 submissions are allowed for the final private test set evaluation where the best submission is ultimately selected. This section demonstrates that after model development according to validation, if one were allowed k submissions and the best were selected, this would be the achievable score. \n\n**Nonetheless, we would like to emphasize again that the point of the section is not to show improved performance due to selection strategies. The section aims to study the relationship between validation and test scores within a search process and provide valuable insights that can assist future agentic development and could be readily actionable in some domains.**\n\nWe hope that we clarified the value of this section to the review. We are happy to address any additional concerns that the reviewer might have. We will revise the section to make these points more clear to future readers\n## **Minor Corrections**\n- **Figure 1a x-axis:** Agreed—we will remove the \"any medal\" label as the y-axis already specifies medal rate.\n- **Line 259 missing constant:** Thank you for catching this. We will add c=0.5 to the revised manuscript.\n## **Clarifications**\n- **L93 vs L211 Fitness Function:** Sorry for the confusion—this is a good catch. The fitness function can be: (1) an operator that assigns values to nodes, (2) ground truth evaluation for verifiable domains, or (3) a byproduct of operator execution (e.g., 5-fold CV from DRAFT). For Kaggle problems with hidden test sets, each DRAFT/IMPROVE solution performs 5-fold CV, which is extracted as the fitness function. **We will clarify this distinction in the description.**\n- **L95 & L105 \"LLM-driven\" vs \"defined by hand\":** \"LLM-driven\" means the operator's execution involves an LLM call in some way. The MEMORY operator's logic (retrieving previous node information) is hand-coded and deterministic, requiring no LLM call. It outputs plans and validation results in structured format in which each node’s information appended on top of each other separated by a line divider.\n- **L184 Model Confounding (R1 vs O1 (preview)):** We acknowledge this limitation. Different models were used to ensure fair comparison with the original AIDE baseline. **To address this concern, we have included o1-preview results in DOJO and ran o3 on all search/operator combinations**—these results will be in the revised paper, showing infrastructure impact alone is significant. Whilst, ideally we would run o1-preview with all the search methods, the model was deprecated from OpenAI’s services before we could run the experiments.\n- **L194 Complexity and DRAFT Children:** We will clarify this description. nc defines how many children a node currently has. Our complexity rule ensures siblings created sequentially become increasingly complex—ensuring simple solutions are created first and complex solutions later. This is not stating that having more children means the node **needs** to be more complex. This is just a heuristic way to control the complexity of the planned solutions throughout the search tree.\n- **DRAFT nodes do have children:** They are initial nodes from the blank root node, with the IMPROVE operators creating their children. E.g. If you called IMPROVE on the same draft node 5 times, it would have 5 separate children stemming from it.\n## **Summary**\nWe believe the clarifications concerning the overfitting experiments and the additions (statistical tests, additional experiments, and improved descriptions) will significantly strengthen the paper and its core contributions. We appreciate the reviewer's constructive feedback in helping us improve the work.\n\nWe would appreciate the reviewer’s consideration of increasing their score, as we believe we have adequately addressed the main limitations they identified. If the review has additional concerns about our responses, we kindly ask them to let us know. \n- *[1] Mann, Henry B., and Donald R. Whitney. \"On a test of whether one of two random variables is stochastically larger than the other.\" The annals of mathematical statistics (1947): 50-60.*\n- *[2] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.*"}, {"review_id": "lXoB6TJlyM", "reviewer": "Reviewer_qCYX", "summary": "The paper presents an analysis of AI research agents that solve the MLE benchmark, a set of Kaggle competitions. The authors discuss the important details of designing such agents, dividing the process into two major parts: high-level actions that are available to agents, and a search policy. The paper also investigates how the choice of proxy metric affects the agent's performance, and whether the gap between validation and test accuracies can be closed. Additionally, the authors propose to extend the action space of the current approach, which moderately boosts metrics.", "questions": "[1] In AIRAEVO explanation, it is said that there exists a Crossover operation. How is it implemented?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality**: This paper presents a comprehensive empirical analysis on AI research agents, with adequate experiment design that supports the claims. To strengthen the paper, an ablation for each additional operator might be included. The reported results seem reliable, with confidence intervals provided.\n\n**Clarity**: The paper is easy to follow and clearly written, the difference between the approaches is highlighted, and each experiment is clearly presented.\n\n**Significance and Originality**: The paper outlines major directions for improvement for future work, and therefore is beneficial for the scientific community and agentic research in particular.", "was_revised": true, "final_justification": "As my review states, I find this paper interesting and the results reliable. Thus, I am willing to accept this paper.", "rebuttal": "We sincerely thank Reviewer qCYX for their thorough evaluation and positive assessment of our work. We are particularly grateful for their recognition of our \"comprehensive empirical analysis\" with \"adequate experiment design\" and \"reliable results,\" as well as the clarity of our presentation. Most importantly, we appreciate that the reviewer found our work to \"outline major directions for improvement for future work\" and be \"beneficial for the scientific community and agentic research in particular.\"\n\n\n### **Clarification on CROSSOVER Implementation**\n\n> **Q: In AIRA_EVO explanation, it is said that there exists a Crossover operation. How is it implemented?**\n\nThank you for this important question, it is an oversight of ours that we did not properly explain. The CROSSOVER operator combines elements from two parent solutions to create a new candidate solution. Specifically, it takes two existing code artifacts and prompts the LLM to intelligently merge their complementary strengths—for example, combining the feature engineering approach from one solution with the model architecture from another.\n\nThe implementation works as follows: when CROSSOVER is selected during evolutionary search, two parent nodes are chosen based on fitness-proportional selection by the selection policy. The operator is then provided the two parent codes with instructions to create a hybrid solution that leverages the best aspects of each parent while maintaining code coherence and functionality.\n\n**We will add the detailed explanation of the CROSSOVER implementation below to the Appendix of the revised paper with concrete examples to provide full technical details for reproducibility, as a companion to the open-source implementation.**\n\n```\n\\paragraph{\\textsc{Crossover} Operator} For two or more valid artifacts, the Crossover operator performs a merge of existing solutions. It specifically aims to create a solution drawing insights from the best aspects of its parents, while also taking into account the shared task context (see \\Cref{subsec:aid_ops}) that informs and guides the merging process. For each crossover operation, the two parent nodes are sampled from the distribution of all nodes' fitness scores. \n```\n\nWe also point the reviewer to the rebuttal for reviewer DZn3 to see the prompt template for the crossover operator.\n\n### **Ablation Study for Individual Operators**\n\nWe greatly appreciate the reviewer's suggestion to include \"an ablation for each additional operator\" to strengthen the paper. This would indeed provide valuable insights into the individual contributions of each operator in our AIRA set.\n\nWhile the current rebuttal timeline prevents us from conducting this specific ablation, we will aim to include these results in the camera-ready version of the paper."}, {"review_id": "Xqrg9e2ayo", "reviewer": "Reviewer_DZn3", "summary": "The paper proposes a framework to construct LLM-based code agents for solving MLE-bench tasks. The primary contributions of the paper are algorithmic and empirical. Key contributions include the computational environment for agents (AIRA-Dojo), the agent design framework (AIRA) and the experimental results showing improved performance on MLE-bench (lite). AIRA builds on ideas in AIDE. The main technical ideas involve inducing a search space over programs using LLM-based operators like DRAFT, DEBUG, IMPROVE, MEMORY, etc. The sandbox infrastructure creates the environment to safely and reliably conduct the (long-running) search. Experiments demonstrate a new state of the art on the benchmark. The paper also includes a careful empirical study of the search algorithm's design choices.", "questions": "1. What is the performance of SOTA reasoning and code LLMs (that only do test-time search) on this task? How does it compare to AIDE and AIRA?\n\n2. What kinds of mistakes or errors were observed in the programs generated on test datasets? What is the subjective and / or objective code quality?\n\n3. What is the implementation of CROSSOVER from Line 106? I was unable to find it in the paper or appendix.\n\n4. On Line 184, what figures are being referenced? (\"In the figures, we take ...\"). (I think it refers to the prior AIDE eval metrics reported in the graphs but it was confusing on first reading.)", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper studies an important and interesting problem. Progress here would have a large impact and be of interest to the community.\n- The paper is reasonably well written. Key concepts are illustrated nicely.\n- The proposed algorithmic approach is intuitively clear and technically sound. I wasn't able to find any major issues with the technical setup. The induced search space is clearly described and the operators well described in the appendix (with the exception of CROSSOVER).\n- The experiments are well-constructed and nicely detailed. The primary result (Figure 1, Figure 3) shows that AIRA performs better than the previous SOTA (AIDE) on the MLE-bench lite benchmark. The careful investigation into the underlying components of performance in Section 5.2 and 5.3 contains valuable insights into the underlying performance factors. The appendix contains a good amount of additional detail.\n\n**Weaknesses**\n\n- The paper discusses its limitations reasonably well in Sec 7.1 I agree that leakage of the benchmark during the (unseen) base LLM training might be a concern. While AIDE is a good choice of baseline, I was unable to find any evaluations of SOTA reasoning and code LLMs (Gemini 2.5, Opus 4, etc.). (This would be zero-shot, without any search beyond the TTS used by the LLM itself during inference). While I don't expect to see outperformance, it would strengthen the paper's empirical standing and make it easier to assess its relative impact in the broader (and rapidly growing) area of using LLMs to construct search-based agents.\n\n- The algorithmic novelty is a little low. The paper mostly builds on the ideas in AIDE and uses well-known planning algorithms. However, this is a minor issue given the careful and detailed empirical investigation.\n\n- The paper does not include an error analysis of the generated programs or any code quality metrics. It would be useful to better understand the error distribution during program generation (beyond those having to do with inference infrastructure).", "was_revised": true, "final_justification": "I thank the authors for their detailed responses to the reviewers. The authors have addressed my main questions satisfactorily. After reading the other reviews and comments, I did not see any new significant technical issues raised in the other reviews. Since the authors have agreed to include the error analysis in the final version, I remain positive about the paper and continue to recommend acceptance.", "rebuttal": "Thank you for your thoughtful and constructive review. We appreciate your recognition that our work \"studies an important and interesting problem\" with \"large impact\" and that our approach is \"technically sound.\" \n\nWe're also grateful for your positive assessment that our \"experiments are well-constructed and nicely detailed\" and that our investigation into the underlying components \"contains valuable insights.\"\n\n### **Response to Major Questions**\n\n**Q1: Performance of SOTA reasoning and code LLMs as zero-shot baselines**\n\nThis is an excellent point for contextualizing our agent's performance gains. We have now evaluated o3 for all search and operator combinations. We can extract a model’s single-shot performance by using the first node of each search tree/graph, which represents the initial DRAFT operation without any iterative refinement.\n\n| Method              | Medal Rate |\n|---------------------|------------|\n| o3 Single-shot      | 11.36%     |\n| R1 Single-shot      | 11.44%     |\n| [OAI]  AIDE_GREEDY  | 35.00%     |\n| [DOJO] AIRA_MCTS    | 47.05%     |\n\nThese results demonstrate the significant value of our search framework over simple one-shot generation, showing that the iterative search and operator design are crucial for achieving SOTA performance. We will include these results as well as the o3 results for all search methods in the camera ready version of the paper.\n\n**Q2: Error analysis and code quality metrics**\n\nWe appreciate the valuable suggestion. While the limited rebuttal timeframe prevents a full quantitative error analysis, our preliminary findings show the most common issues were ***data access KeyError exceptions**, **submission file formatting errors**, **third-party API KeyError exceptions e.g. lightgbm version issues**, and **execution timeouts** from inefficient data processing or computationally intensive models. We aim to provide a more systematic categorization of these errors and their frequencies in the camera-ready version.\n\n**Q3: Implementation of CROSSOVER operator**\nWe apologize for this omission. The CROSSOVER operator combines elements from two parent solutions to create new candidate solutions by:\nParent Selection: Two parent nodes are chosen using fitness-proportional selection based on validation scores\nRecombination: The LLM intelligently merges complementary strengths from both parents (e.g., feature engineering from Parent A + model architecture from Parent B) while maintaining code coherence\nWe have added a subsection with implementation details and the prompt template into the appendix. The full technical specification will also be available in our open-source implementation.\n\n```\n\\paragraph{\\textsc{Crossover} Operator} For two or more valid artifacts, the Crossover operator performs a merge of existing solutions. It specifically aims to create a solution drawing insights from the best aspects of its parents, while also taking into account the shared task context (see \\Cref{subsec:aid_ops}) that informs and guides the merging process. For each crossover operation, the two parent nodes are sampled from the distribution of all nodes' fitness scores. \n```\n\nSee the template below.\n\n```\n    # Introduction:\n    You are a Kaggle Grandmaster attending a high-stakes competition.\n    Your goal is to combine two previously developed solutions to further increase performance on the given task.\n    Carefully consider the task description, the two provided solutions, the available data, and compute resources.\n    You need to devise a plan to merge or integrate these solutions and then implement it.\n   \n    # TASK DESCRIPTION\n    ```\n    {{task_desc}}\n    ```\n    \n    {% if memory %}\n    # PREVIOUSLY EXPLORED CROSSOVER/COMBINATION IDEAS\n    ```markdown\n    {{memory}}\n    ```\n    {% endif %}\n    \n    # PREVIOUS SOLUTION 1:\n    ## Code:\n    ```python\n    {{prev_code1}}\n    ```\n    \n    # PREVIOUS SOLUTION 2:\n    ## Code:\n    ```python\n    {{prev_code2}}\n    ```\n    \n    # INSTRUCTIONS:\n    \n    Your main task is to:\n    1. Propose a **Crossover Plan** in natural language explaining how to combine Solution 1 and Solution 2.\n    2. Provide a **Python Code Implementation** of this plan.\n    \n    Consider any previously explored ideas from the MEMORY section.\n    Brainstorm how the two provided solutions can be effectively combined and **WHY THIS CROSSOVER IS LIKELY TO BE EFFECTIVE** and increase performance for the given task, data, and compute resources.\n    Aim for a consistent evaluation method (e.g., 5-FOLD CROSS-VALIDATION, unless the task specifics dictate otherwise).\n    \n    **CONSTRAINTS**:\n    - Be aware of the running time of the solution, it should complete within {{execution_timeout}}\n    - Prefer vectorized operations over Python loops when processing large datasets.  \n    - Use `torch.optim.AdamW` (the recommended optimizer) instead of the deprecated `AdamW` from `transformers`.  \n    - Replace the deprecated `early_stopping_rounds` argument in `lightgbm.train()` with the `lightgbm.early_stopping(stopping_rounds=…)` callback.\n    - If using `timm` models, remember not to prefix or suffix the model names with datasets such as `cifar` as this was deprecated.\n    - As much as possible, keep the stdout clean.\n    \n    **DATA**: The data is already prepared and available in the read-only `./data` directory. You should not unzip any files.\n    \n    **COMPUTE**: You have access to a Python environment with 1 NVIDIA H200 GPU(s) and 24 CPUs available, and the following packages installed: {{packages}}. If you need to, feel free to use additional libraries that fit the problem.\n    \n    Start by making sure you understand the task, the provided solutions, the data and compute resources, and your proposed crossover idea. Then, generate a detailed internal implementation plan that will structure and guide you step-by-step through the implementation process. Make sure to reflect on the plan to ensure that the implementation is efficient, faithful to the crossover idea, and that all requirements (e.g., the evaluation score is printed, the submission file follows the correct format and is saved in the correct location, etc.) are satisfied.\n    \n    **RESPONSE FORMAT FOR IMPLEMENTATION**: \n    Provide a **SINGLE** Markdown code block (wrapped in ```) containing a **SELF-CONTAINED** Python script that:\n    1. Implements the idea **END-TO-END**\n    2. **PRINTS THE 5-FOLD CROSS-VALIDATION** score of the evaluation metric\n    3. **SAVES THE TEST PREDICTIONS** in a `submission.csv` file in the current directory\n    \n    Format the proposed solution as follows:\n    \n    # Crossover Plan\n    <Your proposed crossover plan/strategy>\n    \n    ```python\n    <the implementation of the crossover solution>\n    ```\n```\n\n### **Response to Minor Points**\n\n**Algorithmic novelty:** We agree that our work's key value lies in the systematic framework design and empirical investigation that leads to SOTA results. This systematic evaluation is enabled by our novel formalization, which also facilitates the research on the interplay of search and operators. \n\n**Line 184 clarification:** Thank you for spotting this ambiguity. We have corrected the sentence to read: \"For all figures showing [OAI] results, we take AIDE O1 artifacts from the MLE-bench [4] GitHub repository, where O1 stands for O1-preview [32].\""}]

Author rebuttal:

Meta-review: (a) Summary of claims and findings
This paper studies AI research agents in the context of MLE-Bench, framing them as search policies applied over candidate solutions with operators that iteratively modify code. The authors introduce AIRA (a formal framework for describing agents in terms of search policies and operators) and AIRA-Dojo (an open-source environment for experimenting with different agent designs). They perform extensive ablations across search strategies (greedy, evolutionary, MCTS) and operator sets, and demonstrate that the interplay between these components is critical. Their best configuration achieves a new state-of-the-art on MLE-Bench Lite, raising Kaggle medal success from 39.8% to 47%. The paper also highlights a validation–test gap that reflects overfitting during agentic search.

(b) Strengths
- Clear and systematic framework for analyzing research agents, with a useful decomposition into search policies and operators.
- Strong empirical contribution: extensive experiments and ablations that expose how search strategy and operator design interact.
- Achieves a new state-of-the-art on MLE-Bench Lite, a challenging and competitive benchmark.
- Provides open-source infrastructure (AIRA-Dojo), which will benefit the community.
- Identifies and analyzes generalization gaps and overfitting phenomena, offering insights relevant beyond this benchmark.
- Writing is generally clear and accessible, and the results are presented in a reproducible way.

(c) Weaknesses
- Algorithmic novelty is somewhat limited, as the work builds heavily on AIDE and uses standard planning algorithms. The main contribution lies more in empirical systematization than in new methods.
- Some conceptual framing (e.g., unifying agents and inference-time search) was not fully convincing to one reviewer.
- Evaluation does not include direct comparisons to the latest reasoning/code LLMs as zero-shot baselines, though rebuttal results partly addressed this.
- The validation–test gap section remains somewhat debated in terms of interpretation.

(d) Reasons for decision
The strengths outweigh the weaknesses. The paper is technically solid, makes a clear empirical contribution, and introduces infrastructure that will likely spur further work. Although the novelty lies more in systematization and analysis than new algorithms, the comprehensive study, new SOTA results, and open-source release together represent a meaningful step forward for the community. The additional insights about overfitting and generalization in agentic search add to its significance. These merits place the paper above a standard poster; it makes a strong case for spotlight as a well-executed empirical and infrastructural contribution with visible community impact.

(e) Discussion and rebuttal period
Reviewers initially disagreed on the conceptual framing and significance. The rebuttal clarified how tool-using agents fit into the proposed framework, added missing citations, and provided new experiments (including statistical significance tests and SOTA LLM baselines). These efforts addressed most concerns, with reviewers updating their scores upward or softening objections. Some residual reservations about framing and validation–test interpretation remain, but overall consensus shifted to positive, with multiple reviewers recommending acceptance. Given the strong experimental depth, open-source contribution, and new SOTA benchmark results, I recommend acceptance as spotlight.
Decision: Accept (spotlight)