R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science

## Abstract

Abstract Recent advances in AI and ML have transformed data science, yet increasing complexity and expertise requirements continue to hinder progress. While crowdsourcing platforms alleviate some challenges, high-level data science tasks remain labor-intensive and iterative. To overcome these limitations, we introduce R&D-Agent, a dual-agent framework for iterative exploration. The Researcher agent uses performance feedback to generate ideas, while the Developer agent refines code based on error feedback. By enabling multiple parallel exploration traces that merge and enhance one another, R&D-Agent narrows the gap between automated solutions and expert-level performance. Evaluated on MLE-Bench, R&D-Agent emerges as the top-performing machine learning engineering agent, demonstrating its potential to accelerate innovation and improve precision across diverse data science applications. We have open-sourced R&D-Agent on GitHub: https://github.com/microsoft/RD-Agent.

## 1 introduction

Over the past decade, artificial intelligence (AI) and machine learning (ML) have transformed the data science landscape, unlocking new possibilities for tackling problems across industries. From personalized e-commerce recommendations (Ko et al., 2022; Isinkaye et al., 2015) to AI-assisted healthcare diagnostics (Khanna et al., 2022; Shaheen, 2021), organizations have leveraged vast datasets and ever-improving algorithms to deliver extraordinary outcomes. Yet, as data complexity grows, so too does the demand for specialized expertise – experienced data scientists who can craft the right models, interpret nuanced patterns, and iterate effectively to reach optimal solutions.

Responding to this skills gap, crowd-sourcing platforms such as [Kaggle](URL) have flourished by providing a collaborative forum where thousands of experts and enthusiasts converge. These communities highlight both the value of diverse perspectives and the limitations of current workflows; despite the collective prowess of global contributors, high-level data science problems still require extensive trial-and-error, deep domain knowledge, and significant time investments.

Recently, agents based on large language models (LLMs) (Achiam et al., 2023; Team et al., 2023; Liu et al., 2024) have markedly improved the efficiency and effectiveness of a wide range of tasks (Huang et al., 2023; Li et al., 2022; Liu et al., 2023; Wang et al., 2024). These agents have demonstrated remarkable capabilities in areas such as natural language processing, machine translation, and complex problem-solving and reasoning.
Several benchmarks are proposed by different teams for measuring how well AI agents perform at machine learning engineering, including MLE-bench (Chan et al., 2024), DSBench (Jing et al., 2024), DiscoveryBench (Majumder et al., 2024), InsightBench (Sahu et al., 2024), etc. According to the reported results in these studies, the state-of-the-art (SOTA) solutions utilizing the latest LLMs still exhibit subpar performance in machine learning engineering and far below the capabilities of human experts.

Crucially, data science projects often hinge on iterative insight – the repeated cycle of exploring, testing, and refining approaches as new knowledge emerges. Even when two projects share the same overarching objective, variations in data distribution, sample size, or domain constraints can demand radically different paths towards optimal solutions. Expert data scientists use feedback from each experimental iteration – whether on feature importance, model fit, or resource limitations – to systematically adapt their methods. Because each iteration can be expensive and time-consuming, these projects place a premium on well-conceived exploration strategies rather than relying on random or brute-force attempts.

In light of these challenges, we propose that a successful machine learning engineering agent must actively learn and adapt through iterative exploration. It should synthesize domain insights, generate deeper hypotheses, and refine its approach based on partial findings rather than rely on a single solution path. With these goals in mind, this paper introduces R&D-Agent, a novel framework designed to drive more effective exploration in data-driven projects across a wide array of industries. By boosting both the efficiency of research exploration and the precision of development, R&D-Agent aspires to narrow the gap between automated intelligence and expert-level data science, accelerating innovation where it is needed most.

In particular, the design of the R&D-Agent is guided by two key principles:

- 1.
*Dedicated R&D Role*: the framework employs two specialized agents – the “Researcher” and the “Developer” – which correspond to the two types of feedback provided in each exploration step: solution performance and execution error information. The Researcher agent processes performance feedback to drive efficient exploration (i.e., idea generation), while the Developer agent leverages execution logs to iteratively refine solution implementation (i.e., code generation). Dedicated agents, designed for specific tasks and equipped with corresponding feedback, facilitate more efficient exploration.
- 2.
*Mutually-Enhanced Multiple Traces*: the framework supports multiple exploration traces running in parallel while facilitating advanced mutual enhancements. For instance, you can launch a new exploration trace from an existing checkpoint or merge checkpoints from different traces to generate a more powerful composite solution. These capabilities not only enable the parallel exploration of a single data science task but also enhance the performance of individual exploration traces, thereby significantly boosting overall exploration efficiency.

Notably, the framework’s design offers significant flexibility for integrating diverse research contributions. This capability not only boosts the efficiency of research exploration but also enhances development precision – all through simple API calls provided by our R&D-Agent framework. Ultimately, this synergy results in a more robust data science automation system.

## 2 R&D-Agent

Figure: Figure 1: Framework of R&D-Agent.
Refer to caption: /html/2505.14738/assets/x1.png

The framework of R&D-Agent is demonstrated in Figure [1](#S2.F1).
The system is composed of two main components: the research agent, the development agent.
The research agent will propose a research idea to the development agent, and the development agent will implement the idea and test the performance of the proposed solution.

### 2.1 Dedicated R&D Role

Assigning dedicated R&D roles helps tackle complex problems across a range of tasks, including machine learning engineering.
When it comes to LLM-based agents, the same design principles apply.
This approach mirrors how a team generally assigns different roles to people, such as researchers and developers, allowing us to use established research and development experience. In turn, the system can gather knowledge and insights from its practice, which match human intuition and experience, and can even inspire domain experts. These lessons can then be applied to new tasks.
On the other hand, there are various LLM foundation models with different strengths. For example, models like o1 are very good at reasoning and coming up with creative ideas, while models like GPT-4.1 are excellent at following instructions and implementing solutions. By assigning each agent the model that best fits its role, we can build a more effective team and achieve better results.

The research agent focuses on learning from experience and exploration of ideas. It proposes research directions to the development agent, analyzes the feedback it receives, and then refines its ideas. Through this cycle of learning and exploration, the research agent continuously improves and discovers better solutions.
The learning process relies on past or external experience. As new knowledge is gained, it is collected and organized into a knowledge base. This knowledge base helps the system refine its ideas or come up with new ones.
For exploration process and search strategies, we propose a novel multi-trace idea explorations, which enable parallel, diverse, and collaborative exploration of the solution space.
This part will be elaborated in section [2.2](#S2.SS2).

The development agent focuses on developing and enhancing the engineering aspects of the proposed idea. The proposed idea usually only covers the key thoughts of the solution and is expressed in natural language as a high-level description that needs to be implemented.
In many cases, important engineering considerations are not fully addressed (e.g., the solution must finish running within a given resource budget); the development agent develops and enhances these aspects to ensure a more complete and practical solution.
To improve development efficiency, the process is divided into two phases: 1) developing a runnable solution and 2) running the solution. In the first phase, the development agent creates a runnable solution by iteratively debugging on a sampled dataset, similar to how human developers work. In the second phase, the development agent runs the solution on the full dataset to evaluate its performance.
In data science, training models usually involves large datasets. By having the development agent iterate on a smaller, sampled dataset first, the process becomes much faster and more efficient. This approach allows the agent to quickly test and refine solutions before running them on the full dataset, greatly speeding up overall development.

### 2.2 Multi-Trace idea explorations

In complex data science and machine learning engineering tasks, a single linear exploration path is often insufficient for discovering high-quality solutions. R&D-Agent introduces a multi-trace exploration mechanism, which enables parallel, diverse, and collaborative exploration of the solution space. This section elaborates on the motivation behind this design and the architectural principles that underpin it.

Motivation and Design Principles: One of the fundamental challenges in automated machine learning engineering is the risk of converging on suboptimal solutions due to the limitations of a single configuration. An exploration trace is inherently constrained by its initialization — including the choice of backend LLM, prompt structure, available tools, and supporting knowledge base. These constraints can heavily bias the exploration path, leading to stagnation or premature convergence. To address this, R&D-Agent supports the parallel execution of multiple exploration traces, each configured with heterogeneous parameters. These include variations in prompt strategies, model backends, domain-specific tools, exploration heuristics, and even knowledge scopes. This diversity increases the likelihood of uncovering valuable insights from different perspectives and avoids the narrow search trajectories imposed by uniform assumptions.

Beyond diversity, R&D-Agent is built to scale. Its multi-trace system enables both logical and physical parallelism. Each trace operates as an independent research agent, executing asynchronously across compute nodes, containers, or threads. This design allows the system to scale horizontally across distributed environments, maximizing resource utilization and dramatically reducing time-to-solution. Such parallelism is especially important in high-complexity tasks where brute-force or single-threaded search would be computationally prohibitive.

What’s more, parallelism alone is not sufficient. Without coordination, multiple traces can waste resources on redundant explorations or persist with unpromising directions. To solve this, R&D-Agent introduces cross-trace collaboration protocols that govern how traces interact, evaluate progress, and make adaptive decisions. Each trace maintains a performance profile based on metrics such as solution quality, novelty, resource cost, and error resilience. A centralized module will track these profiles and makes dynamic decisions, such as terminating unproductive traces, spawning new ones with modified configurations, or initiating trace fusion. Importantly, traces are also able to share intermediate results — such as effective feature sets or partial models — creating a collective learning process where one trace’s success informs others.

This principled combination of diversity, scalability, and collaboration forms the foundation of R&D-Agent’s multi-trace exploration, driving efficient and robust progress in data-centric R&D.

Multi-Trace Fusion for Stronger Solutions:
An essential outcome of multi-trace exploration is the ability to combine the strengths of multiple traces into a single, high-performing solution — a process we refer to as Multi-Trace Merge. Rather than selecting the best trace in isolation, R&D-Agent provides a mechanism for compositional integration of partial results from several promising traces. This strategy allows the system to capitalize on the complementary advantages discovered by each trace.

The fusion process operates at multiple granularities within the data science workflow. For instance, feature generation techniques from one trace may be combined with model architectures from another, and post-processing heuristics from a third. Each trace’s components are evaluated and scored based on utility, novelty, compatibility, and performance impact. A configurable fusion strategy — such as greedy selection, weighted voting, or optimization-guided fusion — is then used to assemble the final solution.

One of R&D-Agent’s key strengths is its flexible and customizable fusion design. Users can define domain-specific control and fusion rules at each stage of the process:

- •
During trace evolution, users can specify constraints for early stoping and spawning new traces based on performance thresholds, time used, or exploratory steps.
- •
During information exchange, users can determine which intermediate outputs (e.g., code fragments, error logs, metrics) are shared across traces.
- •
During the fusion phase, users can customize component compatibility rules, aggregation functions, or even plug in learned scoring models.

This flexibility ensures that R&D-Agent can adapt to a wide range of application domains and engineering preferences. Whether in finance, healthcare, or industrial AI, the system’s composability and extensibility allow practitioners to shape exploration according to domain-specific requirements.

By supporting modular integration and cross-trace learning, the Multi-Trace Merge mechanism not only improves the final solution quality but also accelerates convergence and strengthens the agent’s adaptability. This design is crucial in moving from isolated, trial-and-error automation toward intelligent, collective R&D exploration.

## 3 experiment

### 3.1 experiment setting

A typical scenario in applied data science is the Kaggle competitions. R&D Agent leverage it’s ability in MLE-Bench, which evaluates the agent’s ability to solve Kaggle challenges involving the design, building and training of the machine learning models in GPUs. We align the settings with the benchmark to give R&D Agent 24 hours with a virtual environment, GPU, the dataset, and the competition instructions targeting a solution.

The environment we provided to R&D Agent includes a 12 vCPUs, 220GB of RAM, and 1 V100 GPU with Azure OpenAI services. The targets of the experiments are summarized to:

- •
Evaluate the R&D ability with dedicated R&D role
- •
Evaluate the advantages of multi-trace in idea explorations

### 3.2 Results

**Table 1: Performance comparison on MLE-Bench. Best results in each column are in bold.**
| Agent | Low == Lite (%) | Medium (%) | High (%) | All (%) |
| --- | --- | --- | --- | --- |
| AIDE (Jiang et al., 2025) |  |  |  |  |
| o1-preview | 34.3 $\pm$ 2.4 | 8.8 $\pm$ 1.1 | 10.0 $\pm$ 1.9 | 16.9 $\pm$ 1.1 |
| R&D-Agent |  |  |  |  |
| o1-preview | 48.18 $\pm$ 2.49 | 8.95 $\pm$ 2.36 | 18.67 $\pm$ 2.98 | 22.4 $\pm$ 1.1 |
| o3(R)+GPT-4.1(D) | 51.52 $\pm$ 6.21 | 7.89 $\pm$ 3.33 | 16.67 $\pm$ 3.65 | 22.45 $\pm$ 2.45 |
| o3(R)+GPT-4.1(D)-Multi.Trace | 50.54 $\pm$ 2.51 | 9.86 $\pm$ 3.89 | 20.00 $\pm$ 8.16 | 24.00 $\pm$ 0.94 |

Table [1](#S3.T1) summarizes the performance of different agents on the MLE-Bench, a benchmark suite categorizing Kaggle-style competitions by complexity. Each row of the table corresponds to a specific agent configuration: the rows in the part below present variants of the proposed R&D-Agent (two using o3 for research and GPT-4.1 for development, the other using only o1-preview for both roles), while the part above reports results from AIDE o1-preview (Jiang et al., 2025), the previous public best performer. The columns show the success rates (in percentage) of each agent on competitions grouped into three complexity levels—Low (Lite), Medium, and High—alongside the overall performance across all tasks.
The benchmark’s complexity categories are defined by the estimated time required for an experienced ML engineer to create a basic solution: Low (Lite) for under 2 hours, Medium for 2–10 hours, and High for over 10 hours (excluding model training time).
For each cell, the mean and standard deviation are reported, reflecting variability across multiple experimental runs.
To ensure statistical reliability, performance numbers for R&D-Agent o1-preview are averaged over five random seeds and six seeds for R&D-Agent o3(R)+GPT-4.1(D).

In this short technical report, we present selected key results—showcasing the top-performing configurations on the latest models—to highlight the effectiveness and potential of R&D-Agent.
The results show that R&D-Agent achieves much better performance than the AIDE baseline when using the same LLM backend, particularly in the Low (Lite) and High categories.
This shows that the fundamental system design in R&D-Agent, which aligns more closely with the approach to solving a machine learning engineering problem, results in more robust and high-quality outcomes.
In addition, we explored a hybrid strategy using backend LLMs to enable rapid, cost-effective exploration that meets real-world requirements. In comparison with other unpublished experimental results, our approach—combining o3 and GPT-4.1 by assigning o3 as the research agent (leveraging its creative ideation strengths) and deploying GPT-4.1 as the development agent (capitalizing on its superior instruction-following capabilities)—not only meets real-world requirements but also produces results that match or exceed our strongest baseline. This demonstrates the value of assigning dedicated research and development roles.

To further evaluate the effectiveness of RD-Agent’s multi-trace exploration and fusion capabilities, we designed a dedicated experiment under the MLE-Bench setting that explicitly leverages parallel exploration, information exchange, and final trace fusion.

In this setup, we used the heterogeneous configuration of o3 as the Research Agent and GPT-4.1 as the Development Agen, allowing each to specialize in their respective strengths—ideation and execution. We launched two independent exploration traces, each allowed to run for up to 11 hours under the same task constraints. During this stage, each trace pursued distinct solution strategies, guided by diverse prompts, varying knowledge base configurations, and different toolchains. To increase diversity and reduce redundancy, we implemented an information exchange protocol: before launching the second trace, it was provided access to the exploration history and failure cases from the first trace. This ensured that the second trace could avoid repeating ineffective strategies and instead focus on novel directions.

In the final two hours, the agent initiated a fusion phase. During this phase, it merged:

- •
Code modules (e.g., feature engineering, model training routines),
- •
Ideas (e.g., hypotheses about task analysis or model design),
- •
Performance and error feedback from both traces.

This resulted in a composite solution that preserved the most promising elements of each exploration while resolving inconsistencies. When time allowed, the system continued to iterate on the fused solution until the 24-hour time limit was reached. Finally, all valid solution candidates—including those from individual traces and the fusion—were evaluated. The agent selected the final submission based on a composite scoring function, which considered validation performance, solution robustness, and overfitting risk, as derived from the score curves and model diagnostics.

In ongoing work, we are exploring additional settings to further validate and extend this strategy. These include:

- •
Alternative early-stop policies, such as terminating traces based on stagnating performance rather than fixed time budgets;
- •
Domain knowledge injection into one of the traces, simulating expert hints or historical insights to guide exploration;
- •
Adaptive fusion timing, where fusion is triggered dynamically based on trace progress rather than a fixed time split.

The ablation studies on these configurations are currently under development and will be released in a future version of this report.

## 4 Conclusion and Future Work

In this technical report, we introduced the R&D-Agent framework for a specific scenario, machine learning engineering, and showed how it can automate the process of building data-driven AI solutions. We also shared some very preliminary results to provide an overview of our current research.
R&D-Agent is a flexible framework that can support different types of solutions, and our promising early results suggest it is a valuable direction worth exploring. In addition, while machine learning engineering is just one area of research and development, we believe R&D-Agent can be used in many other scenarios as well. In future work, we plan to provide more technical details about our approach and present more comprehensive experimental results.

## 5 Acknowledgements

We would like to thank our colleagues Yuge Zhang, Zehua Liu, Lewen Wang, Yang Liu, and Shizhao Sun for their ongoing contributions, as well as their invaluable feedback and suggestions during our regular discussions.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Chan et al. (2024)
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry.
Mle-bench: Evaluating machine learning agents on machine learning engineering.
2024.
URL [https://arxiv.org/abs/2410.07095](https://arxiv.org/abs/2410.07095).
- Huang et al. (2023)
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec.
Mlagentbench: Evaluating language agents on machine learning experimentation.
*arXiv preprint arXiv:2310.03302*, 2023.
- Isinkaye et al. (2015)
Folasade Olubusola Isinkaye, Yetunde O Folajimi, and Bolande Adefowoke Ojokoh.
Recommendation systems: Principles, methods and evaluation.
*Egyptian informatics journal*, 16(3):261–273, 2015.
- Jiang et al. (2025)
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu.
Aide: Ai-driven exploration in the space of code.
2025.
URL [https://arxiv.org/abs/2502.13138](https://arxiv.org/abs/2502.13138).
- Jing et al. (2024)
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu.
Dsbench: How far are data science agents to becoming data science experts?
*arXiv preprint arXiv:2409.07703*, 2024.
- Khanna et al. (2022)
Narendra N Khanna, Mahesh A Maindarkar, Vijay Viswanathan, Jose Fernandes E Fernandes, Sudip Paul, Mrinalini Bhagawati, Puneet Ahluwalia, Zoltan Ruzsa, Aditya Sharma, Raghu Kolluri, et al.
Economics of artificial intelligence in healthcare: diagnosis vs. treatment.
In *Healthcare*, volume 10, pp.  2493. MDPI, 2022.
- Ko et al. (2022)
Hyeyoung Ko, Suyeon Lee, Yoonseo Park, and Anna Choi.
A survey of recommendation systems: recommendation models, techniques, and application fields.
*Electronics*, 11(1):141, 2022.
- Li et al. (2022)
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.
Competition-level code generation with alphacode.
*Science*, 378(6624):1092–1097, 2022.
- Liu et al. (2024)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- Liu et al. (2023)
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al.
Agentbench: Evaluating llms as agents.
*arXiv preprint arXiv:2308.03688*, 2023.
- Majumder et al. (2024)
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark.
Discoverybench: Towards data-driven discovery with large language models.
*arXiv preprint arXiv:2407.01725*, 2024.
- Sahu et al. (2024)
Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, et al.
Insightbench: Evaluating business analytics agents through multi-step insight generation.
*arXiv preprint arXiv:2407.06423*, 2024.
- Shaheen (2021)
Mohammed Yousef Shaheen.
Applications of artificial intelligence (ai) in healthcare: A review.
*ScienceOpen Preprints*, 2021.
- Team et al. (2023)
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: a family of highly capable multimodal models.
*arXiv preprint arXiv:2312.11805*, 2023.
- Wang et al. (2024)
Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al.
Opendevin: An open platform for ai software developers as generalist agents.
*arXiv preprint arXiv:2407.16741*, 2024.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 2
presentation: 2
questions: None.
soundness: 2
strengths: The paper shows leading “Any Medal” performance on MLE-Bench when using the most recent GPT-5 model.
summary: The paper claims to implement an agent system that reaches state-of-the-art performance on MLE-Bench.
weaknesses: 1. Regarding “Gold” performance, it still does not perform as well as ML-Master that uses Deepseek-R1 released approximately half year earlier than GPT-5. It is recommended to test the performance of the proposed agent system using Deepseek-R1 for fair comparison.

2. The paper claims “All the existing methods can be summarized as a partial optimization from our framework’s simple baseline”. It is obviously an over-claim since many existing methods were published/released earlier than this work, and those existing methods should not be considered an adaptation or optimization from this work. Instead, polite attributions to the previous existing methods should be included in this paper.

3. Continuation to its above over-claim, the paper shows limited novelty in building its agent system. Instead, a more accurate statement seems to be: Most of this paper’s key components were invented or proposed by previous methods cited or not cited in this paper. If this statement is inaccurate, please provide more details to justify.

4. The paper focuses on the optimization on MLE-Bench. However, it may be more convincing to use multiple benchmarks for evaluating the performance of the agent system. Different data science benchmarks like Economically important tasks, DA-Code and DSBench are used by different research papers and reports including OpenAI agent technical report (https://openai.com/index/introducing-chatgpt-agent/).

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Can you try other models/LLMs? One needs to decouple the framework from the capabilities bestowed by the LLM. All competitors should use the same LLM to make it a like-for-like comparison of frameworks.

2. Please can you compare to the other related work (see above)

3. Does it work beyond MLE-bench type tasks? Like other DS tasks to see where it’s effeccitve and fails. Is there a specific task structure this architecture works for or is it generalisable
soundness: 3
strengths: Originality
- Interesting 6 component multi-agent system. Mainly an integration of existing approaches rather than new approaches

Quality
- very comprehensive and rigorous evaluations conducted

Clarity
- Well-written paper: well exposed, well documented in the appendix and clearly positioned 

Significance
- Shows it outperforms other methods on ML research tasks
summary: R&D-Agent is an agent framework for autonomous ML engineering. The main contribution is the elaborate multi-agent architecture.
weaknesses: - Limited novelty: Individual components use established techniques (MCTS, tree search, memory systems, iterative debugging). This paper is also not the first to integrate many of these into agents for ML engineering and data science. e.g. the MCTS figures even look similar to Climb-DC and and the path structure mechanism like DataInterpreter

- Many missing baselines: DataInterpreter, Climb-DC, MLCopilot, LAMBDA, AutoML-Agent --- please can you outline the differences to these works

- Only closed models used (e.g. GPT-5): Use open models to also understand what’s the source of gain the foundation model or architecture.

- No failure analysis: where and why does the framework work well and when does it fail?

- Gain ablations: given the many components, more ablations should have been done to see which components contribute the most

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Check Weaknesses Section
soundness: 3
strengths: 1. The core idea of separating the research and development phases is interesting and conceptually aligns with how human ML practitioners operate.
2. The experimental results on MLE-bench are promising and suggest potential for structured LLM-driven engineering pipelines.
summary: This paper introduces R&D Agent, an autonomous agent framework that formalizes the machine learning engineering process by explicitly separating a research phase (planning, exploration, path structuring, memory context management, and reasoning pipelines) from a development phase (coding, workflow construction, and evaluation strategy). The framework demonstrates promising performance on the MLE-bench benchmark.
weaknesses: Deeper analysis is needed:
1. While the paper includes baseline comparisons and ablation studies, it is unclear what limits the overall system performance. Is the bottleneck primarily in the research phase (e.g., limited exploration despite the proposed reasoning techniques), or in the development phase (e.g., LLMs struggling to reliably use external libraries or tools)? A more detailed investigation could clarify this.
2. The benchmark used in the paper, although containing 40 competitions, appears to consist of only a few broad categories (e.g., classification). It would be helpful to understand whether the agent’s strategies generalize across task types. For example, what is the medal rate when grouped by task category, and are the generated methods qualitatively similar across those categories?

Discussion and Suggestions:
1. Currently, many papers propose new frameworks or agent architectures to tackle ML or reasoning tasks, often relying on similar principles such as task decomposition or multi-phase design. In this crowded space, the key contribution should not only be introducing yet another framework but also providing insightful analysis into why and how certain design choices lead to better performance.
2. In particular, deeper interpretability and diagnostic analysis would strengthen the paper. While ablation studies help isolate the effectiveness of specific modules, they often do not explain what actually changes in the agent’s behavior or generated algorithms after ablation. For example, in this paper, it remains unclear how the removal of a certain research-phase component affects the agent’s exploration strategy or code-generation quality. Providing qualitative examples or process-level comparisons would make the paper more informative and impactful.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 4
questions: See weakness above.
soundness: 1
strengths: - A large number of experiments on the whole set of MLE-Bench are conducted.
- The writing is of good quality.
- Ablation studies on a subset of MLE-Bench are conducted to provide the effectiveness of the proposed framework.
summary: This paper aims to improve the capabilities of autonomous data science agents. This paper proposes R&D Agent, which consists of six main steps: (1) Planning (tricks on time savings); (2) Exploration path structuring (similar to AIDE); (3) Scientific reasoning pipeline (enhanced planning); (4) Memory context (share knowledge across branches); (5) Coding workflow (debugging based on a small subset and then coding, another trick time savings); (6) Evaluation (consistent data splits for consistent performance comparison). Then, this paper conducts experiments on MLE-Bench to demonstrate the effectiveness of the proposed framework.
weaknesses: - The novelty of this paper is limited. All the contributions are more like engineering efforts and small tricks. There is no fundamental technical contribution in this paper. Also, all the techniques are already investigated by the community. I believe this paper would not bring novel technical insights for the community.

- From the perspective of empirical findings, although this paper performs large-scale experiments on MLE-Bench, the experiment design is not convincing enough for ensuring fair comparison. Here are detailed comments:

1. First of all, the comparison in Table 1 **must** build on top of the same settings and the same foundation models; otherwise, it is not correct to claim SOTA. For example, in Table 1, GLAB, OpenHands and AIDE are developed based on GPT-4o/o1-preview, while R&D-Agent is based on more advanced LLM models GPT-5. This is an unfair comparison. We cannot figure out whether the performance improvement comes from the foundation model or from the agent framework. **Furthermore**, the comparison between MLE-Master and R&D-Agent is also unfair. The prompt of ML-Master is optimized based on DeepSeek-R1; thus, we can find that ML-Master w/ DeepSeek R1 consistently outperforms other foundation models, even more advanced GPT-5. Therefore, this paper should provide empirical results of R&D-Agent with DeepSeek-R1, a consistent setting with ML-Master, to ensure fair comparison. **Last but not least**, I believe the current SOTA data science agent is MLE-STAR. However, this paper only provides comparison with MLE-STAR in the appendix and with unfair setups once again. As shown in Figure 5, R&D-Agent is built on top of GPT-5, while MLE-STAR is built on top of Gemini-2.5-pro. This is a clearly unfair comparison. Beyond this, please make sure MLE-STAR is also evaluated on the whole set of MLE-Bench, if the authors what to claim SOTA.

2. The ablation variants in Table 3 are only run once to report the results. This is clearly unacceptable, especially considering the complicated and stochastic automated data science setting. If the computational resources are limited, the authors can only conduct the experiments on MLE-Bench-Lite. But reporting results across multiple runs is a basic requirement.

- Although this paper briefly discusses related works in Table 1 and Appendix A, I think it is still necessary to comprehensively discuss related works and fundamental differences from them in a more detailed manor.

- MLE-Bench is an offline benchmark, which still suffers from the risks of data leakage. It would be better to run R&D-Agent in an online Kaggle competition to see whether the agent can really win a medal. This would be a more interesting experimental setting, compared with overly reliance on MLE-Bench.