Title: MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine Learning Engineering Agents

LLM Agents
Abstract: Agents based on large language models (LLMs) for machine learning engineering (MLE) can automatically implement ML models via code generation. However, existing approaches to build such agents often rely heavily on inherent LLM knowledge and employ coarse exploration strategies that modify the entire code structure at once. This limits their ability to select effective task-specific models and perform deep exploration within specific components, such as experimenting extensively with feature engineering options. To overcome these, we propose MLE-STAR, a novel approach to build MLE agents. MLE-STAR first leverages external knowledge by using a search engine to retrieve effective models from the web, forming an initial solution, then iteratively refines it by exploring various strategies targeting specific ML components. This exploration is guided by ablation studies analyzing the impact of individual code blocks. Furthermore, we introduce a novel ensembling method using an effective strategy suggested by MLE-STAR. Our experimental results show that MLE-STAR achieves medals in 64% of the Kaggle competitions on the MLE-bench, significantly outperforming the best alternative.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of code generation for machine learning tasks using language models.\nHowever, language models tend to use methods they are familiar with, possibly caused by the training data distribution.\nSo the paper proposes to do the following:\n\n- Use web search to retrieve task-relevant model descriptions and code examples.\n\n- Generate an initial solution script using the retrieved information. Evaluate the performance of all the scripts on some validation set. Merge all the scripts into a single solution.\n\n- Perform ablation studies to identify the most impactful part of the pipeline and iteratively refines the solution.\n\n- Combine multiple solutions using an LLM-driven ensemble strategy to produce the final output.\n\nEmpirically, the proposed AI agent outperforms the baseline methods (AIDE, and DS-Agent when comparison is possible) on MLE-bench Lite using different backbone LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThis paper contributes an agentic system that simulates an MLE's task, involving selecting a model, running evaluations, doing ablation studies and improving the model with the results.\nIt shows solid results on the MLE-bench Lite dataset.\n\nThe paper is well written and notationally clear. The overview figures (Figure 2 and 3) are helpful to understand the pipeline.\n\n### Weaknesses\n\n**Need a more comprehensive ablation study**: The ablation studies have analyzed the effectiveness of using an advanced reasoning LLM (which unsurprisingly improves the performance) and the effectiveness of different ensemble methods.\nHowever, we may need more analytical studies to understand the effectivness of other critical components.\nFor example, is web search necessary -- what is the performance if the agent generates a set of models by calling an LLM?\nHow accurate is the ablation agent; how often does it find a part that shows no improvement after refinement?\n\n**Clarity on novelty**\nThere is prior work on writing an academic paper [1], which also involves web search and an agent system to perform trial-and-error.\nIt would be helpful to clarify the novelty of this work.\n\n[1] Lu, Chris, et al. \"The ai scientist: Towards fully automated open-ended scientific discovery.\" arXiv preprint arXiv:2408.06292 (2024).\n\nQUESTIONS:\nThe agent identifies one block and refines it at a time. Are there cases where multi-block refinement or full model resets (e.g., changing the model entirely) would be more effective?\n\nDoes the paper consider metrics beyond model performance, such as code readability and modularity?\n\nLIMITATIONS:\nThe limitations are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the automated machine learning engineering with LLM agents. Specifically, this paper proposes MLE-STAR, an agent framework that leverages domain knowledge from the web search to develop SOTA machine learning models. Extensive experiments on MLE-Bench-Lite are provided to demonstrate the effectiveness of the proposed agent framework. MLE-STAR achieves medals in 44% of the Kaggle competitions from MLE-Bench-Lite, which is a notable progress.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The presentation quality of this paper is good and easy to follow for me.\n2. The investigated problem is novel and interesting. Automated machine learning engineering with LLM agents may benefit the scientists in their scientific discoveries.\n3. Experiments on ML-Bench-Lite are conducted to verify the effectiveness of the proposed agent framework.\n4. MLE-STAR achieves medals in 44% of the tasks, which is a notable progress.\n\nWeaknesses:\n\n1. Please make rigorous claims in your paper. For example, in the abstract, the authors claim that\n\n> Our experimental results show that MLE-STAR achieves medals in 44% of the Kaggle competitions on the MLE-bench, significantly outperforming the best alternative.\n\nThe experiments are conducted on **MLE-Bench-Lite**, instead of the complete version of MLE-Bench. Please revise this claim.\n\n2. How is the web search enabled in MLE-STAR is not detailed in the paper. Do the authors simply utilize the DeepResearch function from Gemini to perform this? Or do the authors build an automatic workflow on their owns? Please provide more details on this.\n\n3. As this paper focuses on novel applications of LLMs, please consider adding a new experiment to see whether MLE-STAR can achieve medals in real-time Kaggle competitions, instead of offline ones provided by MLE-Bench.\n\n4. Why the authors utilize LLM as judge to analyze the novelty of the generated code script? MLE-Bench provides a tool that can identifies potential plagiarism. Please consider use that tool to make in-depth analyses.\n\nQUESTIONS:\nWill the authors open-source the MLE-STAR project? This is an important factor that may impact my recommendation. As this paper focuses on application instead of making technical contributions, if the authors would not open-source their project, I think this paper is less impactful for the community. If the authors can promise the open-sourcing, I would consider raising my score as accept.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMLE-STAR introduces a machine learning engineering agent that uses Large Language Models (LLMs) augmented by external web search and targeted refinement. The agent constructs initial solutions by retrieving effective state-of-the-art model implementations via web searches, then iteratively refines these solutions through focused ablation studies of ML pipeline components. Additionally, MLE-STAR proposes an LLM-driven ensemble strategy to merge multiple candidate solutions. Evaluations on the MLE-bench Lite (22 Kaggle competitions) show substantial improvements over existing state-of-the-art agents (e.g., AIDE), with a medal achievement rate of approximately 44%.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* Quality: Clearly defined, technically rigorous methodology with robust experimental validation.\n* Clarity: Well-organized, clearly articulated, easy-to-follow manuscript.\n* Significance: Demonstrates substantial performance gains on a well-established benchmark, strongly impacting AutoML and LLM-agent research.\n* Originality: Novel combination of external web search and fine-grained iterative refinement within an LLM-driven AutoML framework.\n\n## Weaknesses\n\n\n\n* Quality:\n  * \"MLE-STAR first leverages external knowledge by using a search engine to retrieve effective models from the web, forming an initial solution.\" This claim is supported by clear methodology; however, explicit ablations quantifying the specific incremental benefit of this external retrieval method versus relying solely on the LLM’s internal knowledge are missing.\n  * \"Iteratively refines it by exploring various strategies targeting specific ML components.\" Supported by detailed ablation studies and iterative refinement steps clearly outlined and empirically evaluated. However, specific evidence demonstrating the advantage of targeted refinement compared to whole-script revision is lacking.\n* Significance: Potentially limited broader applicability beyond structured Kaggle-like competitions; unclear generalizability to more open-ended ML tasks.\n* Originality: \n  * Incremental rather than groundbreaking novelty, primarily integrating existing techniques.\n  * Motivations for specific methodological choices, such as fixed numbers of iterations (outer/inner loops) and ensemble candidate counts, are insufficiently justified. Additional sensitivity analyses on these hyperparameters would significantly strengthen the methodological rigor.\n\n### Missing References\n* Zheng, L. et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS, 2023.\n* Lu, C. et al. “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.” arXiv 2024.\n* Bersenev, D. et al. “Replicating a High-Impact Scientific Publication Using Systems of Large Language Models.” bioRxiv 2024.\n\nQUESTIONS:\n1. Can the authors explicitly quantify the incremental advantage of external web search versus relying solely on internal LLM knowledge? This was claimed without any empirical evidence.\n2. How is novelty ensured against inadvertently using existing Kaggle solutions? Could more rigorous novelty checks (e.g., code similarity metrics) be included?\n4. Have comparisons against traditional AutoML frameworks (e.g., AutoGluon, TPOT) been conducted? Could these results be explicitly included in the main manuscript?\n5. Can the authors run their baselines and quantify performance across the full MLE-bench, rather than just the Lite version of the benchmark? To avoid any overfitting to the MLE-bench lite argument speculation.\n\nLIMITATIONS:\nAuthors have adequately addressed key limitations regarding potential data leakage. Additional discussion on computational resource demands, sensitivity analyses, and generalizability to non-Kaggle scenarios could further strengthen the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an agent framework, MLE-STAR, for orchestrating large language models to perform machine learning engineering tasks. This framework consists of a collection of sub-routines to perform specific tasks (e.g., search for an initial model and its implementation) and a protocol for invoking these sub-routines. It significantly outperforms baselines on MLE-Bench (a standard benchmark).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ This agent scaffolding introduced by this paper performs substantially better than prior methods.\n+ The paper is for the most part clearly written and easy to follow.\n\nWeaknesses:\n- Lack of additional ablations: There seems to be a lot of room for insight from additional ablations here. For example, the authors claim that an LLM using its internal knowledge might be a limitation and use this to motivate searching for up-to-date models as an initial step. This seems like a very valuable ablation to run. I’d similarly like to see the benefits of each step in the proposed agent framework and the impact of choices such as the number of proposals at each step etc. I think that such additional analysis would significantly enrich this work and offer more broad takeaways.\n- Unclear takeaways: This paper introduces an agent framework that is very specific to the MLE task. Many pieces of the framework like adding a data usage checker feel like manual patches for specific issues.\n\nQUESTIONS:\nMuch of the methodology seems to be very focused on the MLE Bench task specifcially. Are there generalizable principles/takeaways here? Is this additional scaffolding something that will still be helpful as agents get better?\n\nSince AIDE performance is reported for different models, why aren’t MLE-STAR results reported for different models (besides the Gemini-2.5-Pro ablation)?\n\nHow does the cost and running time of MLE-STAR compare to baselines like AIDE?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-13"}