MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement
Authors: Jaehyun Nam
Jinsung Yoon
Jiefeng Chen
Jinwoo Shin
Sercan O Arik
Tomas Pfister
Venue: NeurIPS 2025 poster ()
Keywords: Machine Learning Engineering Agents
LLM Agents
Primary area: applications

Abstract: Agents based on large language models (LLMs) for machine learning engineering (MLE) can automatically implement ML models via code generation. However, existing approaches to build such agents often rely heavily on inherent LLM knowledge and employ coarse exploration strategies that modify the entire code structure at once. This limits their ability to select effective task-specific models and perform deep exploration within specific components, such as experimenting extensively with feature engineering options. To overcome these, we propose MLE-STAR, a novel approach to build MLE agents. MLE-STAR first leverages external knowledge by using a search engine to retrieve effective models from the web, forming an initial solution, then iteratively refines it by exploring various strategies targeting specific ML components. This exploration is guided by ablation studies analyzing the impact of individual code blocks. Furthermore, we introduce a novel ensembling method using an effective strategy suggested by MLE-STAR. Our experimental results show that MLE-STAR achieves medals in 64% of the Kaggle competitions on the MLE-bench, significantly outperforming the best alternative.

MLE-STAR:
Machine Learning Engineering A Agent via Search and Targeted Refinement
Jaehyun Nam Jinsung Yoon', Jiefeng Chen'
Jinwoo Shin?, Sercan Ö. Arık', Tomas Pfister'
Google Cloud, ²KAIST
jaehyun. naw@kaist ac. kr,
htw.aa/s............oooooooggogo jinsuggyonn@goggee.com com
Abstract
Agents based on large language models (LLMs) for machine learning engineering (MLE) can automatically implement ML models via code generation. However, existing approaches to build such agents often rely heavily on inherent LLM knowledge and employ coarse exploration strategies that modify the entire code structure at once. This limits their ability to select effective task-specific models and perform deep exploration within specific components, such experimenting extensively with feature engineering options. To overcome these, we propose MLE- STAR, novel approach build MLE agents. MLE-STAR first leverages external knowledge by using search engine to retrieve effective models from the web, forming an initial solution, then iteratively refines by exploring various strategies targeting specific ML components. This exploration is guided by ablation studies analyzing the impact of individual code blocks. Furthermore, we introduce novel ensembling method using an effective strategy suggested by MLE-STAR. Our experimental results show that MLEESTAR achieves medals in 64% of the Kaggle competitions on the MLE-bench, significantly outperforming the best alternative.!
Introduction
The proliferation of machine learning (ML) has driven gesh-performncc applications across diverse real-world scenarios, from fundamental tasks like tabular classification [1, 2, 3] complex ones such as image denoising [4]. Despite these advances, developing such models remains labor-intensive process for data scientists, involving extensive iterative experimentation and data engineering [5. 6). To streamline such intensive workflows, recent research has focused on employing large language models (LLMs) [7, 8, 9] as machine learning engineering (MLE) agents [10, 11. 12]. By harnessing the coding and reasoning capabilities inherent in LLMs [13, 14], these agents conceptualize ML tasks as code optimization problems. They then navigate the potential code solutions ultimately producing executable code (e.g., Python script) based on provided task description and dataset (see Figure 1).
Despite their promise pioneering efforts, current MLE agents face several obstacles that limit their effectiveness. First, due to their strong reliance on inherent LLM knowledge, they are often biased toward familiar and frequently used methods (e.g., the scikit-learn library [15] for tabular data), neglecting potentially promising task-specific methods. Additionally, these agents [10, 12] typically employ exploration strategy that modifies the entire code structure once each iteration. This often results in agents pivoting prematurely to other steps (e.g., model selection or hyperparameter tuning) because they lack the ability perform deep, iterative exploration within specific pipeline components, such as experimenting different feature engincering options extensively.
'This work was done while Jaehyun was a student researcher at Google Cloud. We release open-source codebase of MIE-STAR at httpp://githie com/ google e/adk- samples.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Planning
Task Description
Input
Training
Output
Dataset
Debugging
Machine Learning Tasks
MLE Agent
Solution Script
Figure 1: Problem setup. ML Engineering agents are designed to process a task description and datasets across various modalities (e.g., tabular, text, image, audio, etc.) with the objective of determining the optimal solution for a given machine learning problem, such as classification,, regression, sequence-to-ssqquene generation, image denoising, text normalization, etc.
Contributions. We propose MLE-STAR, novel ML Engineering agent that integrates web Search and TArgeted code block Refinement (see Figure for an overview). Specifically, generating initial solution code, MLE-STAR utilizes Google Search to retrieve relevant and potentially state-of-the- approaches that could be effective towards building a model. Moreover, to improve the solution, MLE-STAR extracts a specific code block that represents a distinct ML pipeline component, such as feature engineering or ensemble building, and then concentrates on exploring strategies that are targeted to that component, using previous attempts as feedback to reflect on. Here, identify the code block that has the greatest impact on performance, MLE-STAR performs an ablation study that evaluates the contribution of each ML component. This refinement process is repeated, modifying various code blocks (i.e., other ML components). In addition, we introduce a novel method to generate ensembles. MLE-STAR first proposes multiple candidate solutions. Then, instead of relying on simple voting based on validation scores, MLE-STAR merges these candidates into a single improved solution using an ensemble strategy proposed by the agent itself. This ensemble strategy iteratively refined based on the performance of the previous strategies.
To verify the efffcctiveness, conduct comprchensive evaluations of MLE-STAR using the MLE- bench's Kaggle competitions [16]. The experimental results demonstrate that MLE-STAR, requiring only minimal human effort (e.g., defining initial prompts that generalizable any tasks), signifi- cantly outperforms previous methods [12], including those requiring manual labor to collect strategies from Kaggle [10]. In particular, MLE-STAR achieves a substantial gain in medal achievement, improving it from 36.6% to 63.6% when compared to the top-performing baseline. Additionally, we show that our proposed ensemble technique provides a meaningful improvement MLE-STAR.
Related work
LLM agents. Recent advances in LLMs have led to an active research in autonomous agents. General-purpose agents like ReAct [17] and Hug [18] typically use external tools to analyze various problems. Specialized agents, such Voyager [19] for Minecraft or AlphaCode [20] for code generation, excel in specific domains, often using execution feedback to iteratively improve their approach. Extending these, we introduce MLE-STAR, an LLM agent that specialized in ML tasks.
Automated machine learning. Automated machine learning (AutoML) aims to reduce reliance on human experts by automating end-to-end ML pipelines [21, 22, 23]. Mito-WEKA [24], TPOT [25], and recent advances such as AutoGluon [26], have made progress through exploring within predefined model or hyperparameter spaces. AutoML research also specializes in areas such as neural network design [27, 28, 29, 30), and feature engineering [31, 32, 33, 34, 35]. However, these methods rely on predefined search spaces, which often require domain expertise to define. To address this, L.M--based MLE agents [10, 12], including MLE-STAR, are emerging. since they employ effective exploration strategies directly in the code space, without the need of manually-curate search spaces.
MIE agents. Leveraging coding and reasoning capabilities of LLMs [13, 14], research has been conducted on use of LLMS as MLE agents [11, 36, 37], which generate solution code, to automate ML workflows. While MLAB [38] and OpenHands [39] take general actions by calling tools to perform ML tasks, several studies specialize in ML automation. AIDE [12] generates candidate solutions in a tree structure to facilitate code space exploration. However, its heavy reliance on the LLM's internal knowledge can lead to outdated or overly simple model choices, and its refinement may prematurely shift focus between pipeline stages. DS-Agent [10] uses case-based reasoning [40, 41] to discover strategics for solution generation by utilizing manually curated cases (primarily from

---

What modele are effective?
Retrieved Information
Evallation
Val. Soore
Inital Solution
(Model description)
0.91
Integrate
Search as tool
(Example code)
Sort
0.92
0.90
Task description
Generate Python script
MLE-STAR w/ Aperrtever
Dataset
0.85
2nd Inlagrate
Discard
(a) Initialization
component impacts the moet?
Python scrist for ablation study
Exccution result
Impulation method impacts the most, and was not refined before!
the encoding strategy.
Modify specific components
Original: 0.93
Median impuling: 0.96 Romaving feature 0.94
[atapam]
Previous solution
atepleingutee jatzategym
(31
MLE-STAR Aaa
MLE-STAR Ausrsacto
Extracted code block
(b) Target code block extraction
Outer Loop
Suggest plan
Implement plan k
How about Plan k?
Target code block
Code B1ock
Improved Solution
Evaluation
Cusgec Coda klock
Plan Score
Replace the block when done
Replace
Plan - Scora K
MLE-STAR w/
Update trajectory & Feedback,
Aplutnur
Implement best plan
(c) Code block refinement
Figure 2: Overview of MLE-STAR. (a) Using search as tool, MLE-STAR retrieves task-specific models and uses them to generate an initial solution. (b) In each refinement step. MLE-STAR performs an ablation study to extract the code block that have the greatest impact. Previously modified code blocks are also provided as feedhack for diversity. (c) The extracted code block is iteratively refined based on plans suggested by the LLM,, which explores various plans using previous experiments as feedback (i.e., inner loop), and the target code block is also selected repeatedly (i.e., outer loop, where the improved solution of (c) becomes the previous solution in (b)).
Kaggle). However, DS-Agent suffers from scalability issues due to its reliance on a manually built case bank, which requires significant human effort and can lead to solutions that are overfit to the source patterns. Also, it restricts applicability to novel task types (like complex multi-modal problems). Our method addresses these limitations. Instead of attempting to explore the broader code space or relying on a static case bank, MLE-STAR strategically explores implementation options for specific ML pipeline components. It also improves scalability by using LLMs with search as tool to retrieve effective models that fit the task beyond the constraints of a fixed case bank.
3 MLE-STAR
We introduce the proposed framework for MLE agents, MLE-STAR, that effectively leverages the coding and reasoning capabilities of LLMs to solve ML tasks. In nutshell, our approach is based on first generating an initial solution by using web search as a tool (Section 3) and then refining solutions via nested loops. The outer loop targets one code block, which corresponds to the specific ML component extracted through an ablation study. The inner loop itcratively refines only this block until the outer loop moves to the next target (Section 3.2). We propose a novel ensemble method that improves the performance using the plan proposed by LLMs, which is iteratively refined (Section 3.3). To mitigate potential undesirable behaviors from LLMs, such as using test sample statistics for missing value imputation, we introduce specific modules (detailed in Section 3.4). The prompts and algorithms used in each step can be found in Appendix A and B, respectively.
Problem setup. Formally, our goal is to find an optimal solution 8* = arg max.es l(s),, where S is the space of possible solutions (i.e., Python scripts) and h. : S + IR is a score function (e.g., validation accuracy) [12]. To obtain s", we propose a multi-agent framework A, which takes datasets D (that might contain multiple files) and task description Treak (which includes task types, data
3

---

modalities, score functions, etc.) as input. Here, A consists of n LLM agents (A1.... An.). Each agent A; possesses specific functionalities, which are elaborated upon following sections.
3.1 Generating an initial solution using web search tool
Candidate model search. MLE-STAR starts by generating an initial solution. For high performance in ML tasks, selecting the appropriate model is paramount. However, relying solely on an LLM for model suggestions can lead to suboptimal choices. For instance, we observe that LLMs propose models like logistic regression [15] even for competitions like which is a text classification task. potentially because LLMs favor familiar patterns from their pre- training data over up-to-date information. To mitigate this, we propose using web search tool for MLE-STAR first to retrieve M effective, ffate-of-the-e art models for the given task. This retrieved context is then used to guide the L.LM generating a more informed initial solution. Formally:
(Tradel: Tedda Aretriever (Teaaz),
(1)
where Taodal represents the description of retrieved model, while Toode provides corresponding example code. This example code is needed since the LLM can be unfamiliar with the model and cannot generate the executable code without proper guidance. Then, MLE-STAR involves evaluating of the performance of model i. To achieve this, candidate evaluation agent Ainit first generates code, si sinte: using the retrieved model to solve the given ML task. This process is formally defined as:
sinee Ainit (Trasks T Teode).
We evaluate the performance of each using a task-specific metric on dataset D. We denote the resulting score by h(s), which encapsulates the entire process done in 8: splitting 'D into training and validation sets, training the model specified in using the training data, and calculating on
the validation data. The performance for sinst is thus h(s h((in1)) As a result, set of code scripts Sinit [sinse. .... i and their performance scores (h(sinst)," ... (ss (mtt)) are obtained.
Merging candidate models for initial solution. After the evaluation of the M retrieved models, a consolidated initial solution so is constructed through iterative merging procedure. Specifically, we first define π be permutation of the indices such that the scores are sorted in descending order:
so with the top- performing script, and record the current best score, i.e., + 8(1), /beaat + h(so),, where 8(k) =(k)
h(s ((sTl))) 2 h(st((t)) 2 > h( (M(T)M)) ). Then, we initialize the initial solution
denote the script Sinit for simplicity. Finally, we sequentially attempt to incorporate the remaining scripts 8(k) for k = 2, M into 5g. For each k. MLE-STAR creates candidate merged script by leveraging an agent Anerger that attempts to integrate (()) into the current sy. Formally,
8g Anerger (S0, (()):: Meeet + h(sp)
(3)
where, Anarger guided to introduce a simple average ensemble to merge multiple models. Finally, we merge the models until the validation score /beat no longer improves (see Appendix B).
3.2 Refining a code block for solution improvement
The iterative refinement phase begins with an initial solution so and proceeds for a predetermined number of T outer loop steps, indexed by t = 0, 1....... T 1. At each step t, the goal is to improve the current solution 8; to obtain S1+1, optimizing for performance metric h. This process involves two main stages: targeted code block extraction and code block refinement.
Targeted code block extraction. To effectively explore specialized improvement strategies, MLEE STAR identifies and targets specific code blocks within the MI. pipeline represented by 8t. This selection is guided by an ablation study performed by an agent Asss. Specifically, the agent Auu generates a code a, designed to perform an ablation study on st. This script creates variations of st by modifying or disabling specific components. To encourage exploration of different pipeline parts across iterations, Aabi receives the summaries of previous ablation studies (Ti, y-1 as input:
a Aaba (St, (Trb )(=))
MLL-STTRR works across any data modalities (e.g., tabular. image, text. audio) and task types (e.g.. classification, amage-to-image, soguence-to-sequence it is not restricted to specific inputs objectives.

---

Suggesi ensemble plan
Evaluation
Ensemble with the best
Task Descrptin
Implement plan
Update trajectory
In paraliel
Plan 1 Score
Feedbaak
Plan Score
MIE-STAAR
Final
MLE-STAR
Ausa
Piae k Score
Anneembler
Figure 3:
Ensembling solutions. MLE-STAR iteratively proposes effective ensemble strategies based on previous attempts, integrating multiple solutions generated in parallel into single solution.
Then, a₂ is executed, producing output results ra (see examples in Appendix C). A summarization module Asumarize processes the script and its results to generate a concise ablation summary Tasi
Tab Asumarriza Tt).
Finally, MLE-STAR selects a critical block which has the most significant impact on performance. To achieve this, extractor module Auxtractor analyzes the Tak to identify the code block e within 8, whose modification had the most significant impact on performance. To prioritize refinement of blocks not previously targeted, the set of already refined blocks {c:) provided as context:
Ct. Po = Aextractor (Taba Se: {-
(6)
Here, MLE-STAR also generates the initial plan po for code block refinement the same time, since Tabi can provide a good starting point by analyzing the modification of corresponding component.
Code block refinement. Once the targeted code block is defined, MLE-STAR explores various refinement strategies to improve the metric h. This involves an inner loop exploring potential refinement for ca. An agent Acoder first implements Po. transforming into refined block ci. i.e., of Acoder (Ct, Po). ) A candidate solution sf is formed by substituting c into st:
= replace(cc. G.),
(7)
where, replace denotes the code replacement operation. Finally, the performance h(s?) is evaluated.
To discover potentially more effective or novel refinement strategies, MLE-STAR iteratively generates and evaluates further plans. For k = i,... K 1, planning agent Aplannor proposes the next plan Pk. This agent leverages the previous attempts within the current outer step as feedback:
Pk = Aplameer (ct, ((p5, (())))
(8)
For each plan Pk. the coding agent generates the corresponding refined block, i.e., of = Acoder (Ce, Pk). creates the candidate solution sf se.peacc,, )f), and evaluates its performance h(sf)) After exploring refinement strategies (indexed
1), the best-performing candidate solution is identified: ks arg ma..e KK1 h(t)). The solution for the next outer step, 8 is updated sf only improvement over 8; found. This iterative process continues until t T.
0....
3.3 Further improvement by exploring ensemble strategies
To further improve upon the best single solution generated, we introduce a novel ensembling procedure (Figure 3). Standard practice might involve generating multiple candidate solutions and selecting the one with the highest score [42] according to metric h. However, analogous to model ensembling, we posit that suboptimal solutions might contain complementary strengths, and combining multiple solutions could lead to superior performance compared to relying any single one. Therefore, we automatically discover effective strategies for
employ the planning capabilities of MLE-STAR ensembling. Specifically, let {s1)/ be set of L distinct solutions obtained (e.g., from parallel runs of the process described earlier). Our goal is find an effective ensemble plan that merges these solutions, which mirrors the structure of the targeted code block refinement stage. We start with an initial ensemble plan 6o (e.g.. a simple strategy like averaging the final predictions obtained from the models trained using each solution 81), proposed by MIEESTAR itself. After the performance h((=0 for the initial plan en calculated, for fixed number of iterations, = 1,... R, the planning agent Aens A specialized in suggesting ensemble plans, proposes subsequent ensemble plans er-
5

---

This agent uses the history of previously attempted ensemble plans and their resulting performance
as feedback, i.e., er = Aans Aans_plannor plannor ((t))= 1 {[eg, (( ))1 Each e, is implemented via Aensenbler to obtain sens
(9) Finally, after exploring R ensemble strategies, the ensemble result that achieves the highest performance is selected as the final output, yielding the final ensembled result & S
snn (eneebbler [st]==1)
r" = arg (a..r h) This procedure allows MLE-STAR to autonomously explore
and identify potentially novel and effective ways to combine multiple complex solutions.
3.4 Additional modules for robust MLE agents
Debugging agent. We detail the design of our debugging agent within MLE-STAR, If the execution of Python script s triggers an error, resulting in a record Toug (e.g., a traceback), MLE-STAR employs a debugging module Adebugger to attempt correction. This process iteratively updates the script:
8 Adebugger (8, Toug)-
(10)
The debugging step is repeated until either the script executes successfully, a predefined maximum number of debugging rounds is reached. If the bug cannot be resolved, MLE-STAR proceeds to the next task using the latest version of the script that is known be executable.
Data leakage checker. We observe that -LM-geenrated Python scripts might have the risk of introducing data leakage, for example, by improperly accessing information from a test dataset during training dataset preparation (see Figure 6). To address this, we introduce a checker agent, Aleakage. which analyzes the solution script prior to its execution. Recognizing that full-script analysis can be inefficient for lengthy code, we adopt targeted approach. First, we extract the code block Caata where data preprocessing is done. Second, Cdata is passed to the checker. If Aleskage detects potential data leakage, it generates a corrected version Caaza' Caata Aieakaae (Caata). Finally, the original script s is updated by replacing the identified segment with its corrected version: f- s.replace( Cdata: Casta)- no leakage detected in Casta by Aleakage. the script s remains unmodified. All generated solutions passed through a data leakage checker, Aleakage prior to their execution for evaluation.
We observe that LLM-generaale scripts sometimes neglect using provided data sources, focusing solely on simple formats like CSVs (see Figure 7). To ensure the utilization of all relevant provided data, MLE-STAR introduces a data usage checker agent, Adata- Specifically, before MLEESTAR starts refinement, Adata checks the initial solution along with the task description Traek If relevant provided data is not adequately used, Adata revises the initial script as:
Data usage checker.
8g f- Adsta(S0 Trask).
(11)
4 Experiments
In this section, we validate the effectiveness of MLE-STAR using 22 Kaggle competitions from MLE-bench Lite [16]. Our results demonstrate that MLE-STAR significantly outperforms baselines, including those employing various LLMs (Section 4. Furthermore, we show that using better models and leveraging our proposed ensemble strategy effectively improves performance (Section 4.2). We also provide the example solutions generated by S-AA, in Appendix D.
Common setup. All experiments are conducted on 22 Kaggle competitions from MLE-bench Lite [16) using three random seeds, unless otherwise specified. Here, we use an agent Atest, which takes the task description and the final solution as input, and outputs the code that incorporates loading test sample and creating a submission file (see Appendix E for details). MLE-STAR begins by retrieving four model candidates. MLE-STAR refines for four inner loops, while exploring four outer loops. For ensemble, MLE-STAR generates two solutions in parallel, and explore ensemble strategies for five rounds. Following the MLE-bench's setup, we set a maximum time limit of 24 hours for a fair comparison (see computation analysis in Appendix F). We primarily consider AIDE [12] as our main baseline, given its state-of of-te-of-theart performance on MLE-bench It is important to note that other baselines often limit their generalizability across various task types (e.g., audio elassification, ggquence-to-sqquenee frequently showcasing results only on simpler modalities like tabular [11, 37]. For instance, DS-Agent [10] requires a manually constructed case bank, and their current GitHub repository lacks cases for audio classification, woweennett--squeneee image classification, etc.

---

Table 1: Main results from MLE-bench Lite. Each experiment is repeated using three seeds,
except for ol ol-preview (AIDE) and GPT-4o (AIDE), which use 16 and 36 seeds, respectively. All results are taken from the GitHub repository of MLE-bench paper [16]. except for the model using Gemini-2.0- 2.0-Flh and Gemini-2.5-Por Scores represent the mean and one standard error of the mean.
Made
Valid
Above
Any Medal (9)
Model
Submission (%)
Submission (%)
Median (%)
Bronze (%)
Silver (%)
Gold (%)
MLE-STAR (Ours)
gemini-2.5-pro
100.0 un
100.0100 95.5gaa
83.3 63.61e9
6. 6.1
21.2
36.4 30.3
63.61s0
gemini-2.0-faas
95.5gae
9.1
4.5
43.9
AIDE [12]
gemini-2.0
87.9 _7
78.8
39. 4g60
4.5
9.1
12.
25.8
ol-preview
=9,7=a3
90.3+16
58,2a26
4.8
11.1
20.7
36.6226
gpt-4o
82.1
65.7+12
29.9+14
3,4404
5.8+0s
9.3+18
18.6+14
72.7=5a
51,5es2
18.2+47
0.0eaa
4,5+2s
6.1e2s
10,6+3x
claudo-3-5-sonnct
81.8
66.7asa
33.3
3.0
6.1
10.6
19.7
ML.AB [38]
gpt-4o
84,8
63.6e59
7.6x33
3.0m21
1.5eis
1,5
6.1
OpenHands [39]
gpt-4o
81.8=47
71.2+56
16.7+46
3,0e21
3.0e21
6.1
12.1
Table 2: Comparison with DS-Agent.
Table 3: Performance with Claude-Sonnet-4.
Task
Metric
DS-Agent
MLE-STAR
Task
Metric
2.0-Flash
Sonnet-4
WBY MCC
MAE (L)
166
DDD
RMSE (L)
0.0681
0.0155
RMLSE ())
0.2964
0.2911
DBI
Log Loss (L)
0.4535
0.3114
ST
Accuracy (1)
0.7982
0.8091
SAI
Log Loss (L)
0.2797
0.2610
ES
AUROC (t)
0.8727
0.9101
WCR
AUROC (f)
0.9903
0.9888
4.1 Main results
Quantitative results. As demonstrated in Table I, MLE-STAR significantly enhances the perfor- mance of various bascline models. For instance, when applied to Gemini-2.0-FP 2.0-Fahhh MLE-STAR im- proves AIDE's any medal achieving rates in Kaggle competitions from 25.8% to 43.9%, representing an improvement of over 18 percentage points, and rate of above median from 39, 99.4% to 63.6%. Notably, MLE-STAR with Gemini-2.0-Fian reasoning model (i.e., ol ol-preview) in terms of achieving gold medals in 10% more tasks. Moreover, using Gemini-22 2.5-i 2.5-Pro, MLE-STAR shows a medal achievement of over 60%.
also substantially outperforms AIDE using powerful
Comparison to DS-Agent. While DS-Agent [10] shows competitive results on ML tasks, it necessi- tates human effort to curate its case bank from Kaggle. Consequently, a direct comparison between DS-Agent and AIDE or our method is not feasible, as collecting tasks across diverse modalities, such as audio classification or image denoising, requires additional effort. Nevertheless, we utilize four tabular classification tasks, i.e., wild-blueberry-yine -ield (WBY), media-campaign-cost (MCC), spaceship-titanic (ST), and enzyme-substrate (ES), the same ones employed during DS-Agent's development stage [10], for comparison. All experiments are done for 5 seeds following the original setup. As shown in Table 2, MLE-STAR significantly outperforms DS-Agent even without human efforts. See Appendix G for additional results, including comparison with AutoGluon [26].
4.2 Ablation studies
Performance with an advanced reasoning model. To assess if more advanced reasoning model could enhance MLE-STAR's performance, we conduct an experiment with the recently released advanced reasoning models. First of all, as shown in Table 1, Gemini-2.5-Pro [43] yields better performance than using Semini-2.0--laate MLE-STAR with Gemini-2. 2emiii-2.0-Fllash Fla-h scored above the median across all three seeds, failing to achieve any medals. However, when using Gemini-2.5-. MLE-STAR achieves two gold medals and one silver medal. These results demonstrate that MIFE-SARR is designed to harness the advancements of rapidly improving reasoning-based LLMs.
For example, in denoising siiee-dittyyooooo docty-do competition,
7

---

Table 4: Ablation ensemble strategy. Experiment results MLE-bench Lite, repeated three seeds using Gemini-2.0-Flash. Scores represent the mean and one standard error of the mean.
Made Submission (%)
Valid Submission (%)
Above Median (%)
Any
Bronze (%)
Silver (%)
Gold (%)
Ensemble strategy
Medal (%)
AIDE [12)
None
87.9
78.8a50
39.4
4,5
2.1
12. 124
25.8
MLE-STAR (Ours)
None
95.5
95.5
57.6gs1
7.623s
4.5
25.8
37.9
Best-of-N
95.5
95.5
62.1
6.1s30
7.6:33
28.8
42.4ts1
Average ensemble
95.5
95.5=26
60.6est
6.1
12.1=40
25.8
43.9
Ours
95.5
95.5
63.61ss
9.1
4.5
30.32<:
43.9
Table 5: Sensitivity analysis on the number of ensemble rounds. Experiment results on 4 tasks from MLE-bench Lite, repeated three seeds using Gemini-22 2.0-Flash We report the mean score.
DDD
DBI
SAI
WCR (AUROC; 1)
Ensemble Round
(RMSE: 4)
(Log Loss; L)
(Log Loss; 4)
0,07147 0.06805
0,45351
0,28164
0,98943
3
0,45351
0.27967
0,98898
0.06805
0.45351
0.27967
0.99028
Table 6: Ablation on proposed components. Experiment results on 4 tasks from MLE-bench Lite,
repeated three seeds using Gemini-2, 2.0-Flash. We report the mean score and bold the best one.
Targeted
Search
DDD
DBI
SAI
WCR (AUROC: 1)
Refinement
Tool
(RMSE: 4)
(Log Loss;
(Log Loss; 4)
0.10818
0,45689
0,29141
0,98532
x
0.09303
0.65242
0.30529
0,96509
0.06805
0.45351
0.27967
0.99028
In addition, we conduct additional experiments using Claude-Sonnet-4. Here, we select four different type of competitions: mmage-to-image (denoising-divv -d-ttrydocumment:: DDD), image classification (dog-bre (dog-breed-ide iireeddidnttiiicaton; DBI), text classification ww--:0--wwaae-chlllen-----ii
aatil---attoo-ddniiiatttioo SAI), and audio WCR). We run each competition for three seeds. As shown in Table 3, Claude-Sonnet-4 also shows promising results, indicating that framework also compatible and generalizable in terms of LLM type.
classification
Effectiveness proposed ensemble method. As highlighted in Table 4, MLE-STAR demonstrates significant performance improvement over the competing baseline, i.e., AIDE, achieving over 12% higher rate of obtaining any medal even without additional ensemble strategy. Notably, by ensembling multiple solution candidates, approach yields even greater performance gains, i.e., MLE-STAR consistently improves the success rate for achieving any medal (and specifically gold medals), also surpassing the median human expert's 's performance by larger margin compared to scenarios where this ensembling method is not used. While simpler strategies, such as selecting the solution with the best validation score averaging final submissions, also offer benefits, MILE- STAR shows stronger effectiveness, e.g., leading to a higher number of gold medals.
Furthermore, conduct sensitivity analysis on the number of ensemble rounds. Here, utilize four datasets as same as Table 3. Table indicates that while we utilize five rounds for ensemble strategy exploration, comparable performance can be achieved with fewer rounds.
Effectiveness of proposed components. Here, we focus on two key components of the proposed approach: targeted refinement strategy and integrating search as a tool. As shown in Table 6, we verify the effectiveness our targeted refinement and the benefits of the search tool integration. Specifically, our findings indicate that refining only the specific code block identified by the ablation study agent more effective than refining the entire codebase. Moreover, our experiments consistently show that utilizing search tool significantly enhances the overall performance.

---

Baseine ine
MLE-STAR (Ours)
(stample coda))
80
Incorporating RealMLP...
60
Human Expert
MIE-STAAR
Usage (96)
40
pytatess Lepart Cllassstier
20
seain, df coad oave /Asput/trsis.on oos"
VGG
ResNet
U-Net
MobiteNet
Emicentttl
VIT
y (target")
wodel Teal Clansi fler (tevi togel peniiee stateee2,
slaectd
hidden, s1renn 1411
Model Architectree (Ordered Release Year)
modell FIRDX beatin y. y_tnain)
Figure 4: Model usage (%) on image classifica- tion competitions. Other models (1 (11.7%), which are used by MLE-STAR, are omitted.
Figure 5: adding model description, MLE-STAR inte- grates its training into the framework.
Human intervention. By manually
Improperly imputed missing values Conbined DataFrame for Consistent Preprocessing
Python script before refined data usage checker data pd. read cov h/tnptt/tsssse wttt//rrannccoo// cav
de pd.conewt de, test text_de))
Python script after refined by data leakage checker
Ixpute Nonerical features with median
def process xys files (dF) Peatures using XYZ files def volume (filspath):
num Lools ['Age', "BoonService', 'FoodCouxt')
open (filepath, 'x') readdines
all [cel] (snull() () median_val all
(map (float, 12) 0) O (3:]1) (3:11) b_vec (map
[col] (nedian
appiit (1:11)
all (ooo] fillna (nedian val)
Drap 1ines (ppstt
Python script after refined by data leakage checker
volune do np.cross zeturn abs (volune)
vec))
Preproceas Training Set and fit statistics
K_train proceszed, stata
for index, dé ittrrton ():
proprocess data (X_ train, fit statestton))
file_path = os.path. pot.path.joon Join (row['1d') ggoonettyyyyooo xxyeettyyxxyy)))) de [ondex,, 'atonic, volume 'I volume (fsle_path) ruturn df data process eeccess_xy____e_oe xyz rrles (data) Process the train data
Pregrocees Test Set using statiatics from Teain Set
proprocess data IX test, fit sttsstatsssttt)
Figure 6: MLE-STAR's data leakage checker Figure 7: MLE-STAR's data usage checker cap- tures previously unused information.
introduces appropriate prepmecssing.
5 Discussion
Qualitative observations on selected models. Figure illustrates the model usage of two MLE agents: AIDE and MLE-STAR. AIDE primarily employs ResNet [44] for image classification. How- ever, ResNet, released 2015, is now considered outdated and can result in suboptimal performance. In contrast, MLE-STAR primarily utilizes more recent and competitive models like Efficient- Net [45] or ViT [46], leading to the performance gain, winning 37% of the medals, more than AIDE, which wins 26% of the image classification challenges.
Human intervention. MLE-STAR readily adopts even more recent models with minimal human intervention. While MLE-STAR automatically constructs a model description [Tnodal, Teada) using search as tool, natural extension involves leveraging human expertise for this construction. As shown in Figure 5, by manually adding model description for RealML.P [47], MLF-STAR successfully integrates its training into the framework, a model not previously retrieved. In addition, users can also specify the target code blocks by replacing the ablation summary with manually written instructions.
Misbehavior of LLMs and corrections. We observe that while the code generated by the LLM executed correctly, their content is sometime unrealistic, exhibiting hallucination. For example, Figure illustrates an impractical approach where test data is preprocessed using its own statistics. Since test data must remain unseen, correction in the code is necessitated, for which, MLE-STAR em- ploys a data leakage checker Aleakage to identify such issues in the generated Python script. If problem is detected, MLE-STAR refines the code. As shown in the Figure, MLE-STAR successfully identifies the issue and modifies the code by, first extracting statistics from the training data and then preprocessing the test data using these calculated statistics. In addition, the improvement process

---

Table Improvement failure when not using data leak- age checker Aleazage on praceship-titanie competition.
20.8
Validaation Improvement (94)
17.7
Metrie
Accuracy (t)
Validation
0.8188 - 0.8677
15
12.6
Test 0.8033 - 0.7343
Table 8: Ablation study of data usage checker Assta on nomad2018 prrdddiciig competition.
Model
Adats
RMSLE (1)
Step
Step
Step 2
Stop
Step 4
MLE-STAR
X
0.0591 0.0559
MLE-STAR
V
Refinement
Figure 8: Solution refinement trajectory.
can fail generalize when Aleskage is not employed, exemplified in Table 7. In this example, the validation accuracy (i.e., the target objective) improves, but the test accuracy drops significantly. This is attributed to the LLM performing feature engineering using the target variable Trans Transported. which is not accessible in the test set, leading to data leakage and subsequently, poor test performance.
We also observe that LLMs often generate Python scripts that overlook some of the provided data sources. For example, the nomad2018-prediette 8-rredicting competition, Gemini-2. 2.ori--.--FFash solely loads train.csv, neglecting the use of geometry.xy2 (see Figure 7). To address this, MLE-STAR employs Adata, which reexamines the task description to ensure that all given data is utilized. As shown in Figure 7, this design enables MLE-STAR incorporate previously neglected data. As a result, performance is significantly improved, as shown in Table 8.
Progressive improvement via MLE-STAR refinement. This section details the progressive im- provement of solutions achieved by MLE-STAR, as measured by validation metrics. Given the task-specific nature of evaluation metrics, we report the average relative error reduction (%) across the all 22 challenges MLE-bench Lite [16]. This metric measures the extent to which MLE- STAR reduces the error of an initial solution. Figure 8 demonstrates a consistent improvement as MLE-STAR proceeds through its refinement steps, which each step focusing on refining a single code block via an inner loop. Significantly, the magnitude of improvement is notable in the early refinement stages. We posit that this stems from MLE-STAR 's ablation study module which helps to target the most influential code blocks for modification first
Discussion on potential plagiarism. Following MLE-bench, we utilize Dolos [48], a source code plagiarism detection tool, to analyze generated solution code by MLE-STAR, against the top associ- ated notebooks (i.e., Jupyter notebook) from each Kaggle competition. Our analysis, summarized in Table 13 (see Appendix L), shows that no final solution code and notebook pair exceeded 60% similarity score (i.e., a criteria suggested from the MLEEbbech paper), indicating no detected instances of plagiarism. This also shows that MLE-STAR's solution is sufficiently new compared to the existing solutions in Kaggle.
Conclusion
We propose MLE-STAR, novel MLE agent designed for various ML tasks. Our key idea utilize search engine to retrieve effective models and then explore various strategies targeting specific ML pipeline components to improve the solution. The effectiveness of MLLESSTAR is validated by winning medals in 64% (where 36% are gold medals) of the MLE-bench Kaggle competitions.
Limitation. We acknowledge that MLE-STAR requires higher cost due to increased token usage. We include corresponding cost analysis in Appendix K. However, it is worth to note that still, with Gemini-2.0-Flast. the cost of MLE-STAR is only about S0.24 per each ML challenge. In addition, since Kaggle competitions are publicly accessible, there is a potential risk that LLMs might have been trained with the relevant discussions about the challenge. Nevertheless, we show that MLE-STAR's solution is sufficiently novel (using LLM as judge) compared to the discussions on Kaggle (see Appendix H), and also show that its similarity compared to notebooks on Kaggle does not exceed 60%, alleviating such plagiarism issue (see Appendix L).
10

---

Acknowledgements and disclosure of funding
We would like to thank Raj Sinha, Subin Kim, Changyeon Kim, Dongjun Lee, Jihoon Tack, and anonymous reviewers for their helpful feedback and discussions. This work was partly supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) IRS-2019-11190075, Artificial Intelligence Graduate School Program (KAIST)) and ITRC (Information Technology Research Center) grant funded by the Korea 50%).
government (MSIT)
References
[1] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2016.
[2] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and
Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 2018.
[3] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeister, and Frank Hutter. Accurate predictions small data with tabular foundation model. Nature, 2025.
[4] Linwei Fan, Fan Zhang, Hui Fan, and Caiming Zhang. Brief review of image denoising techniques. Visual computing for industry, biomedicine, and art, 2019.
[5] Noah Hollmann, Samuel Müller, and Frank Hutter. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. Advances in Neural Information Processing Systems, 2023.
[6] Jaehyun Nam, Kyuyoung Kim, Seunghyuk Oh, Jihoon Tack, Jaehyung Kim, and Jinwoo Shin. Optimized feature generation for tabular data via Ilms with decision tree reasoning. Advances in Neural Information Processing Systems, 2024.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
[8] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.113977 2023.
[9] Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint 4rX 2024.
[10] Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-agent: Automated data science by empowering large language models with case-based reasoning. International Conference on Machine Learning, 2024.
[11] Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An Ilm agent for data science. arXiv preprint 4rXiv:2402.1867 18679, 2024.
[12] Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Ja- cenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.131 13138, 2025.
[13] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-hench: Can language models resolve real-world github issues? International Conference Learning Representations, 2024.

---

[14] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. International Conference Learning Representations, 2025.
[15]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn:
Machine learning in python. Journal of Machine Learning Research, 2011.
[16] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents machine learning engineering. International Conference Learning Representations, 2025.
[17] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. International Conference on Learning Representations, 2023.
[18) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 2023.
[19] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305.16291 2023.
[20] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition- code generation with alphacode. Science, 2022.
[21] Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning. Journal of Machine Learning Research, 2022.
[22] Erin LeDell and Sebastien Poirier. H20 AutoML: Scalable automatic machine learning. ICML Workshop AutoML, 2020.
[23] Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search system. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2019.
[24] Lars Kotthoff, Chris Thornton, Holger Hoos, Frank Hutter, and Kevin Leyton-Brown. Brown. Auto- weka 2.0: Automatic model selection and hyperparameter optimization in weka. Journal of Machine Learning Research, 2017.
[25] Randal S Olson and Jason H Moore. Tpot: A tree-based pipeline optimization tool for automat- ing machine learning. ICML Workshop on AutoML, 2016.
[26] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabulat Robust and accurate automl for structured data, arXiv preprint arXiv:2003.06550 2020.
[27] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. International Conference on Machine Learning. 2018.
[28] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. Interna- tional Conference on Learning Representationss 2017.
[29] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. AAAI Conference Artificial Intelligence, 2019.
[30] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 2019.
12

---

[31] Wei Fan, Erheng Zhong. Jing Peng, Olivier Verscheure. Kun Zhang, Jiangtao Ren, Rong Yan, and Qiang Yang. Generalized and heuristic- free feature construction for improved accuracy. SIAM International Conference on Data Mining, 2010.
[32] James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automat- ing data science endeavors. IEEE International Conference on Duta Science and Advanced Analytics, 2015.
[33] Liyao Li, Haobo Wang, Liangyu Zha, Qingyi Huang, Sai Wu, Gang Chen, and Junbo Zhao. Learning a data-driven policy network for pre-training automated feature engineering. Interna- tional Conference on Learning Representations, 2023.
[34] Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automated feature engineering and selection. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2019.
[35] Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. International Conference on Machine Learning, 2023.
[36] Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants. arXiv preprint arXiv:2501.044 .04227, 2025.
[37] Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424, 2024.
[38] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. International Conference Machine Learning, 2024.
[39] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, al. Openhands: An open platform for ai software developers as generalist agents. International Conference on Learning Representations, 2024.
[40] Janet L Kolodner. An introduction to case-based reasoning. Artificial intelligence review, 1992.
[41] Ian Watson and Farhi Marir. Case-based reasoning: A review. The knowledge engineering review, 1994.
[42] Yuki Ichihara, You Jinnai, Tetsuro Morimura, Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment. Trunsactions on Machine Learning Research, 2025.
[43] Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 2025.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[45] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning. 2019.
[46] Alexey Dosovitskiyy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
[47] David Holzmüller, Léo Grinsztajn, and Ingo Steinwart. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. Advances in Neural Information Processing Systems, 2024.
13

---

[48] Rien Maertens, Charlotte Van Petegem, Niko Strijbol, Toon Baeyens, Ame Carla Jacobs, Peter Dawyndt, and Bart Mesuere. Dolos: Language-agnosten plagiarism detection source code. Journal of Computer Assisted Learning, 2022.
[49] Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents to becoming data science experts? International Conference Learning Representations, 2025.
[50] Yiming Huang. Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, et al. Da-code: Agent data science code generation benchmark for large language models. arXiv preprini arXiv:2410.0733/ 2024.
[51] Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Evaluating agents on data analysis tasks.
Su, Jingjing Xu, Ming Zhu, et al. arXiv preprint arXiv:2401.00 0050075 2024.
[52] Ziming You, Yumiao Zhang, Dexuan Xu, Yiwei Lou, Yandong Yan, Wei Wang. Huaming Zhang, and Yu Huang. Datawiseagent: A notebook-centric Ilm agent framework for automated data science. arXiv preprint 25Xiv:503.000444 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the introduction and abstract accurately reflect the contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not have a theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: Yes]
Justification: We provide implementation details Section 4. We also provide all prompts in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: The benchmark is already open open-soureed, but we do not currently submit code when submitting.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
public/guides/Codest Swwiiidee/CoeSSumissi..oom sonooiiisso for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips.coom httppp..cppbblicc/cc cetpuulicgguuudeee guwde///eeeee/e /CodeSummi ssi ssionnolllii soloicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the details in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are conducted with multiple seeds, and we report one standard егror of the mean.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by eπror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide compute resources we used Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: :/tee://ee....eeeeeeee httppp::///arppppppppppiiiee///e ctw.ihhesccelllleccccc ines?
Answer: [Yes]
Justification:
We do not have any ethical concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our framework does not introduce risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited all papers and datasets in Reference.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
The method development does not involve LLMs any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : http///rps..cc/oeeeeeeeeeeeee / for what should or should be described.
21


Reviews:
[{"review_id": "jn0Slo0Xvx", "reviewer": "Reviewer_aeRQ", "summary": "This paper introduces an agent framework, MLE-STAR, for orchestrating large language models to perform machine learning engineering tasks. This framework consists of a collection of sub-routines to perform specific tasks (e.g., search for an initial model and its implementation) and a protocol for invoking these sub-routines. It significantly outperforms baselines on MLE-Bench (a standard benchmark).", "questions": "Much of the methodology seems to be very focused on the MLE Bench task specifcially. Are there generalizable principles/takeaways here? Is this additional scaffolding something that will still be helpful as agents get better?\n\nSince AIDE performance is reported for different models, why aren’t MLE-STAR results reported for different models (besides the Gemini-2.5-Pro ablation)?\n\nHow does the cost and running time of MLE-STAR compare to baselines like AIDE?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ This agent scaffolding introduced by this paper performs substantially better than prior methods.\n+ The paper is for the most part clearly written and easy to follow.\n\nWeaknesses:\n- Lack of additional ablations: There seems to be a lot of room for insight from additional ablations here. For example, the authors claim that an LLM using its internal knowledge might be a limitation and use this to motivate searching for up-to-date models as an initial step. This seems like a very valuable ablation to run. I’d similarly like to see the benefits of each step in the proposed agent framework and the impact of choices such as the number of proposals at each step etc. I think that such additional analysis would significantly enrich this work and offer more broad takeaways.\n- Unclear takeaways: This paper introduces an agent framework that is very specific to the MLE task. Many pieces of the framework like adding a data usage checker feel like manual patches for specific issues.", "was_revised": true, "final_justification": "Thank you to the authors for response. I raised my score.", "rebuttal": "Dear Reviewer aeRQ,\n\nWe deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n**[W1] Lack of additional ablations (e.g., removing search at initialization, impact of choices such as the number of proposals at each step).**\n\nWe appreciate your valuable feedback and have conducted additional experiments to further address your comments, building upon the ablation studies already presented (e.g., the effectiveness of our ensemble strategy in Table 4; performance improvements with advanced LLMs in Table 3).\n\nIn this rebuttal, we've focused on two key components of the proposed approach:\n- **Targeted refinement strategy:** Our findings indicate that refining only the specific code block identified by the ablation study agent is more effective than refining the entire codebase. \n- **Integrating search as a tool:** Our experiments, conducted with the same configuration as outlined in the manuscript, consistently show that utilizing search as a tool significantly enhances the overall performance.\n\nThese results, summarized in the table below, verify the effectiveness of our targeted refinement and the benefits of the search tool integration. We will incorporate these ablation studies in the revised manuscript.\n\\begin{array}{lcccccc}\n\\hline\n\\text{Method}&\\text{Targeted refinement}&\\text{Search as a tool}&\\text{DDD (RMSE)}&\\text{DBI (Log Loss)}&\\text{SAI (Log Loss)}&\\text{WCR (AUROC)}\\newline\n\\hline\n\\textbf{MLE-STAR (Variant 1)}&\\text{X}&\\text{O}&0.10818&0.45689&0.29141&0.98532\\newline\n\\textbf{MLE-STAR (Variant 2)}&\\text{O}&\\text{X}&0.09303&0.65242&0.30529&0.96509\\newline\n\\textbf{MLE-STAR}&\\text{O}&\\text{O}&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&\\textbf{0.99028}\\newline\n\\hline\n\\end{array}\n\nFurthermore, we have conducted a sensitivity analysis on the number of ensemble rounds, and the below table indicates that while we utilized five rounds for ensemble strategy exploration, comparable performance can be achieved with fewer rounds. We will incorporate this detailed discussion in the revised manuscript. Also, we commit to including additional analyses of other hyperparameters, such as the number of inner and outer loops, in the revised manuscript.\n\\begin{array}{ccccc}\n\\hline\n\\text{Round}&\\text{DDD (RMSE)}&\\text{DBI (Log Loss)}&\\text{SAI (Log Loss)}&\\text{WCR (AUROC)}\\newline\n\\hline\n1&0.07147&\\textbf{0.45351}&0.28164&0.98943\\newline\n3&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&0.98898\\newline\n5&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&\\textbf{0.99028}\\newline\n\\hline\n\\end{array}\n\n**Experimental settings:** We used the same four datasets – denoising-dirty-documents (DDD), dog-breed-identification (DBI), spooky-author-identification (SAI), and the-icml-2013-whale-challenge-right-whale-redux (WCR) – used in Table 3 of the current manuscript. The experiments were repeated three times for each dataset and reported the average metrics (lower the better except WCR).\n\n---\n**[W2, Q1] Unclear takeaways: Very specific to the MLE task (e.g., data usage checker seems to be manual patches). Are there generalizable principles here? Is this additional scaffolding something that will still be helpful as the agent gets better?**\n\nIndeed, our method is a specialized multi-agent system for machine learning engineering (MLE) tasks. These tasks are incredibly diverse, encompassing areas like audio classification, image denoising/classification, and tabular regressions/classification. Therefore, effectively tackling MLE problems is crucial for addressing a wide range of real-world challenges. In this regard, we believe that our framework offers clear and significant advantages because it handles any MLE task where metrics are well-defined and data is readily available. Furthermore, our key ideas – leveraging search as a tool and refining targeted code blocks – have broader applicability to general coding agents. For example, a debugging agent could identify an erroneous code block and then focus on updating that specific section. We see these generalizable ideas as a promising future research direction.\n\n---\n**[Q2] Why aren’t MLE-STAR results reported for different models?**\n\nFirst, we would like to share the overall results of the entire MLE-bench Lite using Gemini-2.5-Pro. As shown in the table below, Gemini-2.5-Pro shows significantly better performance than Gemini-2.0-Flash, recording a medal achievement rate of over 60%.\n\\begin{array}{lccccc}\n\\hline\n\\text{Model}&\\text{Above Median}&\\text{Bronze}&\\text{Silver}&\\text{Gold}&\\text{Any Medal}\\newline\n\\hline\n\\text{Gemini-2.0-Flash}&63.6&\\textbf{9.1}&4.5&30.3&43.9\\newline\n\\textbf{Gemini-2.5-Pro}&\\textbf{83.3}&6.1&\\textbf{21.2}&\\textbf{36.4}&\\textbf{63.6}\\newline\n\\hline\n\\end{array}\n\nIn addition, we conducted additional experiments using Claude-Sonnet-4 to further address your comments. As shown in the table below, other models besides Gemini also show promising results, proving compatibility and generalizability in terms of model types.\n\\begin{array}{lcccc}\n\\hline\n\\text{Model}&\\text{DDD (RMSE)}&\\text{DBI (Log Loss)}&\\text{SAI (Log Loss)}&\\text{WCR (AUROC)}\\newline\n\\hline\n\\text{Gemini-2.0-Flash}&0.06805&0.45351&0.27967&\\textbf{0.99028}\\newline\n\\text{Claude-Sonnet-4}&\\textbf{0.01547}&\\textbf{0.31135}&\\textbf{0.26101}&0.98878\\newline\n\\hline\n\\end{array}\n\n**Experimental settings:** For the above table, we used the same four datasets – denoising-dirty-documents (DDD), dog-breed-identification (DBI), spooky-author-identification (SAI), and the-icml-2013-whale-challenge-right-whale-redux (WCR) – used in Table 3 of our manuscript. The experiments were repeated three times for each dataset and reported the average metrics (lower the better except WCR).\n\n---\n**[Q3] How is the cost and running time of MLE-STAR compared to baselines like AIDE?**\n\nWe would like to first clarify that MLE-STAR demonstrates comparable running times to the best alternative, AIDE. As detailed in Appendix F of the current manuscript, MLE-STAR averages 14.1 hours on MLE-bench Lite, while AIDE averages 15.4 hours on the same resource environment. This indicates our method does not require more execution time.\n\nWe acknowledge that MLE-STAR requires higher cost due to increased token usage. As shown in the below table MLE-STAR requires 4.5 times more input and output tokens compared to AIDE. This is primarily because our method generates longer scripts during the targeted refinement and ensembling stages (i.e., the average line of the solution code for MLE-STAR in MLE-bench Lite is 414, while that for AIDE is 147). Still, with Gemini-2.0-Flash (cost is \\\\$0.15/0.60 per 1M input/output tokens), the cost of MLE-STAR is only about \\$0.24 per each ML challenge.\n\nDespite the higher token cost, it is important to note that MLE-STAR consistently and significantly achieves better results than AIDE, as demonstrated throughout the current manuscript. We will incorporate this detailed analysis into the revised manuscript.\n\\begin{array}{lcc}\n\\hline\n\\text{Method}&\\text{\\\\# Input tokens}&\\text{\\\\# Output tokens}\\newline\n\\hline\n\\text{AIDE}&\\text{194K}&\\text{38K}\\newline\n\\text{MLE-STAR (Ours)}&\\text{874K}&\\text{175K}\\newline\n\\hline\n\\end{array}"}, {"review_id": "UXE2vGcj2h", "reviewer": "Reviewer_PksL", "summary": "MLE-STAR introduces a machine learning engineering agent that uses Large Language Models (LLMs) augmented by external web search and targeted refinement. The agent constructs initial solutions by retrieving effective state-of-the-art model implementations via web searches, then iteratively refines these solutions through focused ablation studies of ML pipeline components. Additionally, MLE-STAR proposes an LLM-driven ensemble strategy to merge multiple candidate solutions. Evaluations on the MLE-bench Lite (22 Kaggle competitions) show substantial improvements over existing state-of-the-art agents (e.g., AIDE), with a medal achievement rate of approximately 44%.", "questions": "1. Can the authors explicitly quantify the incremental advantage of external web search versus relying solely on internal LLM knowledge? This was claimed without any empirical evidence.\n2. How is novelty ensured against inadvertently using existing Kaggle solutions? Could more rigorous novelty checks (e.g., code similarity metrics) be included?\n4. Have comparisons against traditional AutoML frameworks (e.g., AutoGluon, TPOT) been conducted? Could these results be explicitly included in the main manuscript?\n5. Can the authors run their baselines and quantify performance across the full MLE-bench, rather than just the Lite version of the benchmark? To avoid any overfitting to the MLE-bench lite argument speculation.", "limitations": "Authors have adequately addressed key limitations regarding potential data leakage. Additional discussion on computational resource demands, sensitivity analyses, and generalizability to non-Kaggle scenarios could further strengthen the manuscript.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* Quality: Clearly defined, technically rigorous methodology with robust experimental validation.\n* Clarity: Well-organized, clearly articulated, easy-to-follow manuscript.\n* Significance: Demonstrates substantial performance gains on a well-established benchmark, strongly impacting AutoML and LLM-agent research.\n* Originality: Novel combination of external web search and fine-grained iterative refinement within an LLM-driven AutoML framework.\n\n## Weaknesses\n\n\n\n* Quality:\n  * \"MLE-STAR first leverages external knowledge by using a search engine to retrieve effective models from the web, forming an initial solution.\" This claim is supported by clear methodology; however, explicit ablations quantifying the specific incremental benefit of this external retrieval method versus relying solely on the LLM’s internal knowledge are missing.\n  * \"Iteratively refines it by exploring various strategies targeting specific ML components.\" Supported by detailed ablation studies and iterative refinement steps clearly outlined and empirically evaluated. However, specific evidence demonstrating the advantage of targeted refinement compared to whole-script revision is lacking.\n* Significance: Potentially limited broader applicability beyond structured Kaggle-like competitions; unclear generalizability to more open-ended ML tasks.\n* Originality: \n  * Incremental rather than groundbreaking novelty, primarily integrating existing techniques.\n  * Motivations for specific methodological choices, such as fixed numbers of iterations (outer/inner loops) and ensemble candidate counts, are insufficiently justified. Additional sensitivity analyses on these hyperparameters would significantly strengthen the methodological rigor.\n\n### Missing References\n* Zheng, L. et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS, 2023.\n* Lu, C. et al. “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.” arXiv 2024.\n* Bersenev, D. et al. “Replicating a High-Impact Scientific Publication Using Systems of Large Language Models.” bioRxiv 2024.", "was_revised": true, "final_justification": "The authors’ rebuttal clearly addressed my main concerns, especially regarding the benefit of web search and targeted refinement, with new ablation results and sensitivity analyses. Expanded baselines and improved novelty checks also strengthened the work. While some concerns about generalizability and resource demands remain, the authors have promised further discussion. Overall, the significant improvements and clarifications outweigh the remaining minor limitations. I have increased my score and now recommend Accept.", "rebuttal": "Dear Reviewer PksL,\n\nWe deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n**[W1, W2, Q1] Ablation study on using search as a tool and the advantage of targeted refinement.**\n\nFirst, we have already shown some qualitative results in Figure 4 of the current manuscript that AIDE, which relies on internal LLM knowledge, often utilizes less competitive models compared to MLE-STAR. To further address your comments, we removed the search tool from Gemini-2.0-Flash (Variant 2) and ran the experiments with the same configuration as in the current manuscript. As shown in the table below, using a search as a tool consistently enhances the performance.\n\nIn addition, we further validate the advantage of targeted refinement. Specifically, we removed the ablation study agent (Variant 1), allowing the agent to refine the entire script at every refining step.\n\nAs the below table illustrates, refining specific code blocks consistently outperforms refining the entire script. We hypothesize this is because focusing on extracted code blocks allows our agent to explore more diverse strategies for selected ML pipelines.\n\nWe will incorporate these results and discussion in the revised manuscript.\n\\begin{array}{lcccccc}\n\\hline\n\\text{Method}&\\text{Targeted refinement}&\\text{Search as a tool}&\\text{DDD (RMSE)}&\\text{DBI (Log Loss)}&\\text{SAI (Log Loss)}&\\text{WCR (AUROC)}\\newline\n\\hline\n\\textbf{MLE-STAR (Variant 1)}&\\text{X}&\\text{O}&0.10818&0.45689&0.29141&0.98532\\newline\n\\textbf{MLE-STAR (Variant 2)}&\\text{O}&\\text{X}&0.09303&0.65242&0.30529&0.96509\\newline\n\\textbf{MLE-STAR}&\\text{O}&\\text{O}&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&\\textbf{0.99028}\\newline\n\\hline\n\\end{array}\n\n**Experimental settings:** We used the same four datasets – denoising-dirty-documents (DDD), dog-breed-identification (DBI), spooky-author-identification (SAI), and the-icml-2013-whale-challenge-right-whale-redux (WCR) – used in Table 3 of our manuscript. The experiments were repeated three times for each dataset and reported the average metrics (lower the better except WCR).\n\n---\n**[W3] Unclear generalizability to more open-ended ML tasks.**\n\nWe believe our method's applicability extends beyond Kaggle-like competitions, since it is designed to tackle any ML task, including open-ended problems, as long as metrics are given as natural language and data is available. Our agent can automatically and rigorously define metrics, and the data doesn't require extensive pre-cleaning.\n\nFurthermore, while MLE-bench is derived from Kaggle competitions, it encompasses a diverse array of tasks (e.g., audio classification, image denoising/classification, tabular classification/regressions). The effective performance of our method across these varied challenges underscores its broad applicability to real-world problems.\n\n---\n**[W4] Incremental rather than groundbreaking novelty.**\n\nOur MLE-STAR framework introduces two critical and groundbreaking novelties not found in prior techniques.\n\nFirst of all, orchestration of diverse agents is central to multi-agent systems’ success. MLE-STAR uniquely integrates agents with distinct roles, such as the retriever, ablation study, planner, ensemble, and debugging agents. The seamless and effective connection of these specialized agents within our novel framework represents a significant and non-trivial advancement in multi-agent system deployment.\n\nSecondly, our targeted refinement strategy is another critical innovation. Unlike previous machine learning agents that refined entire code blocks—limiting exploration of refinement strategies—we have introduced an ablation study agent. This novel component automatically analyzes the effectiveness of individual components within the ML pipeline, enabling exploration of diverse refinement strategies for a certain code block, leading to state-of-the-art performance in MLE tasks.\n\n---\n**[W5] Analyses on hyperparameters.**\n\nFollowing your suggestion, we have conducted a sensitivity analysis on the number of ensemble rounds. The below table indicates that while we utilized five rounds for ensemble strategy exploration, comparable performance can be achieved with fewer rounds. We will incorporate this detailed discussion in the revised manuscript. Furthermore, we commit to including additional analyses of other hyperparameters, such as the number of inner and outer loops.\n\\begin{array}{ccccc}\n\\hline\n\\text{Round}&\\text{DDD (RMSE)}&\\text{DBI (Log Loss)}&\\text{SAI (Log Loss)}&\\text{WCR (AUROC)}\\newline\n\\hline\n1&0.07147&\\textbf{0.45351}&0.28164&0.98943\\newline\n3&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&0.98898\\newline\n5&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&\\textbf{0.99028}\\newline\n\\hline\n\\end{array}\n\n**Experimental settings:** Same as the above [W1, W2, Q1].\n\n---\n**[W6] Missing references.**\n\nWe would like to first note that the focuses of [1, 2, 3] differ significantly from our problem settings.\n- [1] primarily compares LLM-as-a-Judge and human preference alignment.\n- [2, 3] concentrate on automating the end-to-end scientific research process, from idea generation to experimental execution, using multi-agent frameworks.\n\nIn contrast, our work focuses on machine learning engineering tasks. Our agent’s objective is to optimize solution performance given predefined task descriptions and datasets. Specifically, to achieve this, we propose new MLE-specific components, such as refining specific code blocks or ensembling solutions. On the other hand, [2] use a general approach aimed at writing good papers.\n\nWhile our core focus diverges, we do leverage LLM-as-a-Judge for specific sub-tasks, such as identifying data leakage (see Section 3.4 of the current manuscript) and assessing solution novelty (see Appendix H of the current manuscript). Furthermore, we acknowledge that [2, 3] utilize multi-agent systems like MLE-STAR. We will ensure proper citation and discussions of these relevant works in the revised manuscript.\n\n---\n**[Q2] How is novelty ensured against existing Kaggle solutions? Could more rigorous novelty checks be included?**\n\nWe would like to clarify that we have already performed novelty checks in Appendix H of the current manuscript. Specifically, using Gemini-2.5-Pro as a judge, we found that all the final solutions generated by our method are sufficiently novel compared to the top discussions of each competition.\n\nMoreover, to fully address your concern, we utilized Dolos, a source code plagiarism detection tool, to analyze submissions against the top associated notebooks from each Kaggle competition, as proposed by MLE-bench’s authors. Our analysis (please refer to [W4] of the Reviewer RCp1 due to the space limit) shows that no submission-notebook pair exceeded a 60% similarity score (i.e., a criteria suggested from Appendix A.4 of the MLE-bench paper), indicating no detected instances of plagiarism.\n\n---\n**[Q3] Have comparisons against traditional AutoML frameworks been conducted? Could these results be explicitly included in the main manuscript?**\n\nFirst, we have already shown that MLE-STAR significantly outperforms AutoGluon in Table 9 of Appendix G in the current manuscript. Meanwhile, it is worth noting that such an AutoML framework is often restricted to data or task types (e.g., tabular data) and therefore cannot be a direct competitor. For example, AutoGluon should reformulate multi-class classification problems as multiple binary classification problems. Furthermore, such AutoML frameworks require continuous updates by domain experts to reflect the latest models, but our method automatically reflects these updates by utilizing search as a tool. We will explicitly include these results and discussions in the revised manuscript.\n\n---\n**[Q4] Can the authors quantify performance across the full MLE-bench?**\n\nWe appreciate the suggestion and are actively expanding our experimental evaluation across the full MLE-bench. Our intermediate results, using three seeds for each competition, are promising.\n- MLE-STAR has achieved three gold medals in both the stanford-covid-vaccine (SCV) and predict-volcanic-eruptions-ingv-oe (PVE) competitions. These are all high-complexity tasks (i.e., hard tasks), demonstrating our method's robust generalizability.\n- While MLE-STAR did not achieve a medal in cassava-leaf-disease-classification (CLD), its performance nearly reached the median threshold (0.89090).\n\nThe table below presents a preliminary comparison with AIDE + o1-preview which consistently show MLE-STAR significantly outperforming AIDE + o1-preview. We are continuing additional experiments to update our results and will incorporate a comprehensive evaluation with full MLE-Bench in the revised manuscript.\n\\begin{array}{lccccc}\n\\hline\n\\text{Competition ID}&\\text{Task Complexity}&\\text{Medals}&\\text{Metric}&\\text{AIDE + o1-preview}&\\textbf{MLE-STAR + Gemini-2.5-Pro (Ours)}\\newline\n\\hline\n\\text{CLD}&\\text{Medium}&\\text{No medal}&\\text{Accuracy (\\uparrow)}&0.86186&\\textbf{0.88391}\\newline\n\\text{SCV}&\\text{Hard}&\\text{Gold}&\\text{MCRMSE (\\downarrow)}&0.40033&\\textbf{0.23377}\\newline\n\\text{PVE}&\\text{Hard}&\\text{Gold}&\\text{MAE (\\downarrow)}&3747830&\\textbf{2832981}\\newline\n\\hline\n\\end{array}\n\n---\n**[L1] Additional discussion on computational resource demands, sensitivity analyses, and generalizability to non-Kaggle scenarios could further strengthen the manuscript.**\n\nThank you for your valuable feedback. We will enhance our discussion about limitations to include an analysis of the costs associated with utilizing commercial LLMs and integrate sensitivity analyses into the revised manuscript. Furthermore, we will expand on the applicability of MLE-STAR beyond Kaggle-like competitions, emphasizing that its ability to automatically define metrics and process uncleaned data makes it broadly relevant to any machine learning task."}, {"review_id": "Y4irYr9jY4", "reviewer": "Reviewer_RCp1", "summary": "This paper investigates the automated machine learning engineering with LLM agents. Specifically, this paper proposes MLE-STAR, an agent framework that leverages domain knowledge from the web search to develop SOTA machine learning models. Extensive experiments on MLE-Bench-Lite are provided to demonstrate the effectiveness of the proposed agent framework. MLE-STAR achieves medals in 44% of the Kaggle competitions from MLE-Bench-Lite, which is a notable progress.", "questions": "Will the authors open-source the MLE-STAR project? This is an important factor that may impact my recommendation. As this paper focuses on application instead of making technical contributions, if the authors would not open-source their project, I think this paper is less impactful for the community. If the authors can promise the open-sourcing, I would consider raising my score as accept.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The presentation quality of this paper is good and easy to follow for me.\n2. The investigated problem is novel and interesting. Automated machine learning engineering with LLM agents may benefit the scientists in their scientific discoveries.\n3. Experiments on ML-Bench-Lite are conducted to verify the effectiveness of the proposed agent framework.\n4. MLE-STAR achieves medals in 44% of the tasks, which is a notable progress.\n\nWeaknesses:\n\n1. Please make rigorous claims in your paper. For example, in the abstract, the authors claim that\n\n> Our experimental results show that MLE-STAR achieves medals in 44% of the Kaggle competitions on the MLE-bench, significantly outperforming the best alternative.\n\nThe experiments are conducted on **MLE-Bench-Lite**, instead of the complete version of MLE-Bench. Please revise this claim.\n\n2. How is the web search enabled in MLE-STAR is not detailed in the paper. Do the authors simply utilize the DeepResearch function from Gemini to perform this? Or do the authors build an automatic workflow on their owns? Please provide more details on this.\n\n3. As this paper focuses on novel applications of LLMs, please consider adding a new experiment to see whether MLE-STAR can achieve medals in real-time Kaggle competitions, instead of offline ones provided by MLE-Bench.\n\n4. Why the authors utilize LLM as judge to analyze the novelty of the generated code script? MLE-Bench provides a tool that can identifies potential plagiarism. Please consider use that tool to make in-depth analyses.", "was_revised": true, "final_justification": "I think this paper would be a strong submission for NeurIPS conditioned on the promised open-sourced project. Thus, I decide to increase my score to 5.", "rebuttal": "Dear Reviewer RCp1,\n\nWe deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n**[W1] Please make rigorous claims in your paper (e.g., MLE-bench to MLE-bench Lite in abstract).**\n\nThanks for your suggestion. We will clarify 'MLE-bench' to 'MLE-bench Lite' in the revised manuscript. For your interest, we are currently conducting experiments for the entire competition, and we will update the results once the experiments are complete.\n\n---\n**[W2] Unclear details on web search implementation: How is web search enabled in MLE-STAR? Is DeepResearch from Gemini utilized, or is there an automatic workflow built by the authors? More details are needed.**\n\nWe would like to clarify that MLE-STAR enables web search by connecting Google Search as a tool to the Gemini model (tool is named as grounding with google search).\n\n---\n**[W3] Please conduct an experiment to evaluate MLE-STAR's performance in real-time Kaggle competitions, distinct from the offline evaluations provided by MLE-Bench, given the paper's focus on novel LLM applications.**\n\nWe appreciate your constructive feedback. We would like to first clarify that our primary evaluation relies on the offline metrics provided by MLE-bench, since it allows for rigorous comparison with existing baselines. Also, these finished competitions provide a more robust and reliable assessment of medal achievements, because their leaderboards are finalized (note that the better solutions are often updated at the end of the competition). Nevertheless, we agree that evaluating our method in real-time Kaggle competitions is a compelling idea, and we will include such results in the revised manuscript.\n\n---\n**[W4] Why do the authors utilize LLM as a judge to analyze the novelty of the generated code script? MLE-Bench provides a tool that can identify potential plagiarism. Please consider using that tool to make in-depth analyses. Please consider using a tool that can identify potential plagiarism, which is provided by MLE-bench.**\n\nFollowing your suggestion, we utilized Dolos, a source code plagiarism detection tool, to analyze submissions against the top associated notebooks from each Kaggle competition, as proposed by MLE-bench’s authors. Our analysis, summarized in the below table, shows that no submission-notebook pair exceeded a 60% similarity score (i.e., a criteria suggested from Appendix A.4 of the MLE-bench paper), indicating no detected instances of plagiarism.\n\\begin{array}{lccc}\n\\hline\n\\text{Competition ID}&\\text{Submission \\\\#1}&\\text{Submission \\\\#2}&\\text{Submission \\\\#3}\\newline\n\\hline\n\\text{aerial-cactus-identification}&0.35&0.42&0.39\\newline\n\\text{aptos2019-blindness-detection}&0.27&0.29&0.27\\newline\n\\text{dog-breed-identification}&0.32&0.30&0.32\\newline\n\\text{dogs-vs-cats-redux-kernels-edition}&0.33&0.27&0.31\\newline\n\\text{histopathologic-cancer-detection}&0.32&0.36&0.35\\newline\n\\text{leaf-classification}&0.27&0.31&0.30\\newline\n\\text{plant-pathology-2020-fgvc7}&0.31&0.35&0.39\\newline\n\\text{ranzcr-clip-catheter-line-classification}&0.30&0.33&0.34\\newline\n\\text{siim-isic-melanoma-classification}&0.24&0.27&0.24\\newline\n\\text{denoising-dirty-documents}&0.25&0.27&0.27\\newline\n\\text{detecting-insults-in-social-commentary}&0.34&0.16&0.18\\newline\n\\text{jigsaw-toxic-comment-classification-challenge}&0.30&0.27&0.30\\newline\n\\text{random-acts-of-pizza}&0.37&0.39&0.37\\newline\n\\text{spooky-author-identification}&0.20&0.21&0.28\\newline\n\\text{new-york-city-taxi-fare-prediction}&0.35&0.33&0.34\\newline\n\\text{nomad2018-predict-transparent-conductors}&0.24&0.27&0.23\\newline\n\\text{tabular-playground-series-dec-2021}&0.26&0.19&0.29\\newline\n\\text{tabular-playground-series-may-2022}&0.24&0.32&0.20\\newline\n\\text{text-normalization-challenge-english-language}&0.22&0.19&0.20\\newline\n\\text{text-normalization-challenge-russian-language}&0.18&0.23&0.22\\newline\n\\hline\n\\end{array}\n\nNote that the results of the mlsp-2013-birds competitions have been omitted because MLE-STAR was unable to generate submissions using Gemini-2.0-Flash. Additionally, the results of the icml-2013-whale-challenge-right-whale-redux competition have also been omitted because the authors of MLE-bench did not provide the relevant notebooks.\n\n---\n**[Q1] Will the authors open-source the MLE-STAR project?**\n\nYes, we will definitely open-source the MLE-STAR project. For reference, we wanted to provide the code as an anonymous link, but this was not possible due to rebuttal policy restrictions. We promise to make the code public."}, {"review_id": "4Opy86lJOF", "reviewer": "Reviewer_3VFF", "summary": "This paper considers the problem of code generation for machine learning tasks using language models.\nHowever, language models tend to use methods they are familiar with, possibly caused by the training data distribution.\nSo the paper proposes to do the following:\n\n- Use web search to retrieve task-relevant model descriptions and code examples.\n\n- Generate an initial solution script using the retrieved information. Evaluate the performance of all the scripts on some validation set. Merge all the scripts into a single solution.\n\n- Perform ablation studies to identify the most impactful part of the pipeline and iteratively refines the solution.\n\n- Combine multiple solutions using an LLM-driven ensemble strategy to produce the final output.\n\nEmpirically, the proposed AI agent outperforms the baseline methods (AIDE, and DS-Agent when comparison is possible) on MLE-bench Lite using different backbone LLMs.", "questions": "The agent identifies one block and refines it at a time. Are there cases where multi-block refinement or full model resets (e.g., changing the model entirely) would be more effective?\n\nDoes the paper consider metrics beyond model performance, such as code readability and modularity?", "limitations": "The limitations are discussed in Section 6.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nThis paper contributes an agentic system that simulates an MLE's task, involving selecting a model, running evaluations, doing ablation studies and improving the model with the results.\nIt shows solid results on the MLE-bench Lite dataset.\n\nThe paper is well written and notationally clear. The overview figures (Figure 2 and 3) are helpful to understand the pipeline.\n\n### Weaknesses\n\n**Need a more comprehensive ablation study**: The ablation studies have analyzed the effectiveness of using an advanced reasoning LLM (which unsurprisingly improves the performance) and the effectiveness of different ensemble methods.\nHowever, we may need more analytical studies to understand the effectivness of other critical components.\nFor example, is web search necessary -- what is the performance if the agent generates a set of models by calling an LLM?\nHow accurate is the ablation agent; how often does it find a part that shows no improvement after refinement?\n\n**Clarity on novelty**\nThere is prior work on writing an academic paper [1], which also involves web search and an agent system to perform trial-and-error.\nIt would be helpful to clarify the novelty of this work.\n\n[1] Lu, Chris, et al. \"The ai scientist: Towards fully automated open-ended scientific discovery.\" arXiv preprint arXiv:2408.06292 (2024).", "was_revised": true, "final_justification": "The authors have addressed all the concerns in the review. The additional ablation results have further justified the algorithm design. The authors have also agreed to add discussions to make the contributions clearer.", "rebuttal": "Dear Reviewer 3VFF,\n\nWe deeply appreciate your invaluable time and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n**[W1-1] What is the performance if the agent generates a set of models by calling an LLM?**\n\nThanks for your constructive feedback and we conducted an additional ablation study, following your suggestion. Specifically, we removed the search tool and ran the experiments with the same configuration as the current manuscript. As shown in the table below, using search as a tool consistently enhances the MLE-STAR performance. We will incorporate this ablation study in the revised manuscript.\n\\begin{array}{lccccc}\n\\hline\n\\text{Method}&\\text{Search as a tool}&\\text{DDD (RMSE)}&\\text{DBI (Log Loss)}&\\text{SAI (Log Loss)}&\\text{WCR (AUROC)}\\newline\n\\hline\n\\textbf{MLE-STAR (Variant)}&\\text{X}&0.09303&0.65242&0.30529&0.96509\\newline\n\\textbf{MLE-STAR}&\\text{O}&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&\\textbf{0.99028}\\newline\n\\hline\n\\end{array}\n\n**Experimental settings:** We used the same four datasets – denoising-dirty-documents (DDD), dog-breed-identification (DBI), spooky-author-identification (SAI), and the-icml-2013-whale-challenge-right-whale-redux (WCR) – used in Table 3 of our manuscript. The experiments were repeated three times for each dataset and reported the average metrics (lower the better except WCR).\n\n---\n**[W1-2] How accurate is the ablation agent; how often does it find a part that shows no improvement after refinement?**\n\nWe would like to first clarify that Figure 8 (in Section 5 of the current manuscript) already demonstrates that the validation score progressively improves throughout the refinement steps, indicating our ablation study agent effectively targets the influential code blocks early on, as evidenced by the significant improvements in the initial refinement stages.\n\nTo further analyze, we conducted additional analysis and found that 38.4% of refinement steps showed no improvement, with this occurring more frequently in later stages. This finding aligns with Figure 8, which shows smaller improvements as the outer loop progresses. We will incorporate this discussion in the revised manuscript.\n\n---\n**[W2] Novelty: There is prior work on writing an academic paper [1], which also involves web search and an agent system to perform trial-and-error. It would be helpful to clarify the novelty of this work.**\n\nWe appreciate the opportunity to clarify the novelty of our work compared to Lu et al. [1]. While Lu et al. also involve web search and agent systems for trial-and-error, their primary focus is on automating the end-to-end scientific research process, from idea generation to experimental execution. This differs significantly from our work in several key aspects.\n\nUnlike Lu et al. [1], our research directly targets improving performance on ML tasks. We specifically consider the whole ML pipeline and the exploration of task-specific models, which are not central to their works. Furthermore, their trial-and-error mechanism primarily serves debugging and paper writing, whereas ours is explicitly designed for performance optimization.\n\nIn this regard, our MLE-STAR framework introduces two key novelties. First of all, orchestration of diverse agents is central to our multi-agent system's success. MLE-STAR uniquely integrates agents with distinct roles, such as the retriever, ablation study, planner, ensemble, and debugging agents. The seamless and effective connection of these specialized agents within our novel framework represents a significant and non-trivial advancement in multi-agent system deployment.\n\nSecondly, our targeted refinement strategy is an important innovation. Unlike previous machine learning agents that refined entire code blocks—limiting exploration of refinement strategies—we have introduced an ablation study agent. This novel component automatically analyzes the effectiveness of individual components within the ML pipeline, enabling exploration of diverse refinement strategies for a certain code block, leading to state-of-the-art performance in MLE tasks.\n\nWe will incorporate this discussion and highlight the difference with Lu et al. [1] in the revised manuscript.\n\n[1] Lu et al., The AI Scientist: Towards Fully Automated Open-ended Scientific Discovery, arXiv 2024.\n\n---\n**[Q1] The agent identifies one block and refines it at a time. Are there cases where multi-block refinement or full model resets (e.g., changing the model entirely) would be more effective?**\n\nThanks for the valuable suggestions. Rather than identifying just one block and refining it at a time, selecting one or multiple blocks dynamically at a certain step seems to be an interesting future direction. This can be also implemented through the ablation study agent to select one or more code blocks to refine.\n\nOn the other hand, changing the full code update might not be beneficial. In this rebuttal, we have conducted an additional ablation study to further validate the advantage of targeted refinement. Specifically, we removed the ablation study agent, allowing the agent to refine the entire script at every refinement step.\n\nAs the below table illustrates, refining specific code blocks consistently outperforms refining the entire script. We hypothesize this is because focusing on extracted code blocks allows our agent to explore more diverse strategies for selected ML pipelines. We will incorporate these results and discussion in the revised manuscript.\n\\begin{array}{lccccc}\n\\hline\n\\text{Method}&\\text{Targeted refinement}&\\text{DDD (RMSE)}&\\text{DBI (Log Loss)}&\\text{SAI (Log Loss)}&\\text{WCR (AUROC)}\\newline\n\\hline\n\\textbf{MLE-STAR (Variant)}&\\text{X}&0.10818&0.45689&0.29141&0.98532\\newline\n\\textbf{MLE-STAR}&\\text{O}&\\textbf{0.06805}&\\textbf{0.45351}&\\textbf{0.27967}&\\textbf{0.99028}\\newline\n\\hline\n\\end{array}\n\n**Experimental settings:** We used the same four datasets – denoising-dirty-documents (DDD), dog-breed-identification (DBI), spooky-author-identification (SAI), and the-icml-2013-whale-challenge-right-whale-redux (WCR) – used in Table 3 of our manuscript. The experiments were repeated three times for each dataset and reported the average metrics (lower the better except WCR).\n\n---\n**[Q2] Does the paper consider metrics beyond model performance, such as code readability and modularity?**\n\nWe appreciate your valuable suggestions. Currently, our assessment focuses solely on model performance. However, we agree that code readability and modularity are also meaningful metrics. We plan to incorporate them into the revised manuscript as follows: (i) introducing them as additional optimization objectives alongside model performance, and (ii) adding a refinement stage that utilizes an LLM to enhance the readability of our final solution code."}]

Author rebuttal:

Meta-review: The reviewers converged on a positive assessment of the paper, highlighting that MLE-STAR substantially outperforms prior LLM-based ML engineering agents on MLE-Bench Lite and introduces a compelling combination of external web search, targeted refinement via ablation, and ensemble strategies. Strengths cited include clear presentation, strong empirical performance, and a meaningful contribution to the growing field of agentic systems for ML engineering. The authors’ rebuttal further strengthened the case, providing new ablations demonstrating the benefit of search and targeted refinement, sensitivity analyses on hyperparameters, plagiarism checks with "Dolos", and clarifications on resource usage and implementation. These additions addressed most major reviewer concerns, leading multiple reviewers to raise their scores to Accept.

The main disagreements were around novelty and generalizability. Some reviewers felt that the contributions were incremental, highly tailored to MLE-Bench, or patched for specific issues. Others argued that the orchestration of specialized agents and the targeted refinement strategy represent significant and broadly useful innovations. Concerns about cost, resource demands, and generalizability beyond Kaggle-like tasks remain, but reviewers generally considered these outweighed by the empirical impact and strengthened methodological justification.
Decision: Accept (poster)