AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions

Authors: Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tianyu Zheng, minghao liu, Xinyao Niu, Xiang Yue, Yue Wang, Jian Yang, Jiaheng Liu, Wanjun Zhong, Wangchunshu Zhou, Wenhao Huang, Ge Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, language agents, multi-agent

Abstract:
Data science competitions on Kaggle, which represent real-world programming challenges, require sophisticated problem-solving approaches. While LLM-based agents demonstrate potential in various fields, their application to data science tasks often falls short due to difficulties in adapting to data changes in multi-stage reasoning and the need for precise reasoning. To address this, we propose AutoKaggle, a robust and user-centric framework that solves Kaggle problems through a collaborative multi-agent cooperative system. AutoKaggle implements an iterative development process that combines code interpretation, debugging, and comprehensive unit testing covering over 30 tests, ensuring code correctness and quality through LLM-based evaluation. It prioritizes user experience by generating detailed reports that elucidate feature engineering processes, data transformations, model selection criteria, and the reasoning behind each decision. It offers customizable workflows, allowing users to intervene and modify each stage of the process, thus combining the advantages of automated intelligence with human expertise. Additionally, we build a universal data science tool library, including carefully verified functions for data cleaning, feature engineering, and modeling, which form the foundation of this solution. We evaluate the framework on 8 carefully selected Kaggle competitions, achieve 83.8\% in average completion rate and 42.8\% average rank in Kaggle.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730586049754

thread_type:
official_review

<thread_object>
creator: reviewer_u3ch

<content>
summary:
This paper presents a scaffolding framework which uses LLMs to create a multi-agent system, used to attempt Kaggle problems. They use a "phase-based" multi-agent approach, together with a library of hand-crafted ML tools, and extensive hand-crafted unit-tests tailored to the Kaggle problems.

Applying this framework to 8 Kaggle problems (4 pre-GPT-4 training cut-off, 4 afterwards), they achieve a significant solve rate, and an average of 42% on the Kaggle leaderboard.

The paper also explores ablation of various modules (various tools, and the unit-testing module).

strengths:
A system which can score 43% on Kaggle leaderboards is a significant milestone on the path to automated coding and datascience. Additionally, since many challenges which such a system would face would also arise in more general task-completion (e.g. long-term planning, establishing coherency, managing context, preventing execution from looping) and so would transfer to improve AI agents in general.

Great collection of Classic and Recent challenges, and baselines seem reasonable (though see my Q about the Strong Baseline).

It's helpful to have this variety of scores (though see my Q about CS).

Architecture is clearly laid out, and the paper is overall very easy to read.

Clear exploration and explanation of the underlyring readon why the feature-engineering tools reduce the framework's score (many features, leading to more complexity than the agents can handle).

weaknesses:
Whenever CoT is used as an interpretability tool, I think it's always wise to mention unfaithfulness e.g. https://arxiv.org/abs/2305.04388

There are two places where a long list is hard to read:
#1 ~L78: AutoKaggle integrates a comprehensive machine learning tools library, covering three core toolsets: data cleaning, feature engineering, and model building, validation, and prediction.

#2 ~L186: The data science process is divided into six key stages: understanding the
background, preliminary exploratory data analysis, data cleaning, in-depth exploratory data anal-
ysis, feature engineering, and model building, validation, and prediction

Perhaps "model-building, -validation, and -prediction" would be easier to read.

~L146: I'm surprised not to see mentioned what seems to me to be the main thing underlying the motivation of multi-agent systems: finite context length, requiring summarisation and specialisation.

It's not clear how much of the headline 43% on the leaderboard is down to the skill of the human-in-the-loop, which severely undermines the claim. Without a comparison to how well the human takes unassisted (in terms of success rate or time taken), or to how well AutoKaggle performs without HITL, it's impossible to reliably state how effective the framework is.

Unspecified HITL also undermines the various claims of a "fully automated framework" (e.g. L175)

Not much detail on these unit tests. Who writes them? What's the coverage like? Are there any guarantees? If (as I suspect) the "meticulously designed" unit tests are written by humans, then we have a similar situation as with the unspecified human-in-the-loop: the framework is not "fully automated", and it's impossible to rigorously determine how much effect the human hand-holding has on the framework's suggess. This should, at minimum, be clearly, explicitly and boldly acknowledged.

Additionally, it is unclear to me how much of the ML-tools library was developed alongside particular Kaggle Competition attempts. If the tools were developed on a case-by-case basis, to address hurdles found in the challenge, then there is significant data leakage from the evaluation dataset to the framework, leading to overfitting to the competitions chosen during development, and much of the headline 43% comes from tools handcrafted by human developers on a case-by-case basis. For a fair validation of how well this framework performs in "fully automated" mode, the library would need to be "frozen" while the framework was tested on a held-out set of Kaggle Competitions.

Very minor point: ~L350, I agree that there is a risk of data leakage for competitions from before Oct '23, however to say that GPT-4o's training data includes Classic Kaggle is an assumption: better to say simply that there is a risk of data leakage.

If you're considering data leakage, it would be worth flagging that the 42% includes Classic problems: using only the newer problems, performance is slightly below human average.

questions:
~L140, you say that CoT improves reasoning at the expense of introducing hallucinations. Is there any evidence that CoT makes models any more or less likely to hallucinate?

~L141, you say that the ReAct paradigm addresses hallucinations - that's not my understanding of what ReAct does or how it works, my understanding is that it combines thoughts and actions, yes, but that this has nothing to do with hallucinations or refining outputs.

~L360: What is the difference between "Success - Non-compliant" and "Success - Compliant"?

~L403: What's the justification / motivation for the complex / compound "Comprehensive Score"? How does it compare to other measures, what specifically does it achieve or avoid?

~L431: could you say more about this "strong baseline" - I don't understand its construction.

If adding FE tools drops performance because FE adds too much complexity, then why does "All tools" (which presumably includes FE tools) recover this performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642740447

thread_type:
official_review

<thread_object>
creator: reviewer_zznf

<content>
summary:
This paper presents AutoKaggle, a multi-agent framework specifically designed to handle the complexities of Kaggle data science competitions.  The framework organizes the competition workflow into six distinct phases—background understanding, exploratory data analysis, data cleaning, in-depth exploratory analysis, feature engineering, and model development and validation—allowing agents to work systematically through each stage. Key agents, including Reader, Planner, Developer, Reviewer, and Summarizer, collaborate within this structure, with iterative debugging and unit testing to ensure robustness and accuracy in code generation. AutoKaggle integrates a machine learning tools library to streamline tasks, enhance code reliability, and provide users with educational insights through comprehensive reports at each phase. Evaluated across multiple Kaggle competitions, the framework achieved an average completion rate of 83.8% and ranked in the top 42.8% in Kaggle.

strengths:
- AutoKaggle introduces a tailored phase-based workflow with multi-agent collaboration specifically designed for data science competitions. The system’s demonstrated high average completion rate and competitive ranking in Kaggle highlight its effectiveness, particularly in tabular classification and regression tasks, showing its strength in handling structured data challenges.

- AutoKaggle empowers the Developer agent to perform iterative debugging and unit testing, bolstering the robustness of code generation. Additionally, the integration of a comprehensive machine learning tools library improves the system's efficiency and accuracy, making it better suited for tackling complex Kaggle competitions

weaknesses:
- Limited novelty.  While the paper addresses data science problem-solving using LLM-based agents, it lacks a clear description of the specific challenges it intends to solve that existing methods have struggled with. Extending from a single-agent to a multi-agent system is insufficiently justified in this field, as the necessity and performance gains of such an approach are not clearly demonstrated. Existing works, as mentioned in the introduction, have also tackled similar problems with LLM-based agents, questioning the incremental contribution of AutoKaggle.

- Multi-agent system design. The multi-agent system, including agents like Reader, Planner, Developer, Reviewer, and Summarizer, is insufficiently explained in terms of its collaborative structure. It is unclear whether these agents operate in an assembly-line fashion or if they engage collectively in each phase under the "Cooperative Engagement" label in Figure 1. Further clarification on their integration and interdependence within each workflow phase is needed.

- Role clarity of Planner and Summarizer. Given AutoKaggle’s sequential, phase-based workflow, the necessity of a Planner agent is ambiguous. Can you quantify the contribution (such as on completion rates or error reduction) of this Planner agent in your system? Similarly, the Summarizer’s role in contributing to critical performance metrics such as completion rate or Best Normalized Performance Score, is not explicitly justified, leaving its impact on performance uncertain.

- Unit Test and Debugging. Dose the Developer agent generate dataset-specific unit tests that align with each unique code snippet or not? How the Developer agent adjusts unit tests based on code variations to ensure logical consistency and accuracy across different tasks? 

- Lines 275-276 mention the importance of detecting logical errors in code, yet the method for achieving this is underexplored. Can you explain more details about detecting the logical error? More detail is needed on how logical errors are detected and avoided, as conducting exploratory data analysis or statistical checks after data cleaning or feature engineering alone may be insufficient. 

- Table 2 illustrates the system's performance across different debugging attempts (DT), showing how increased debugging impacts metrics like Completion Rate (CR) and Comprehensive Score (CS). The data indicate that both CR and CS improve as DT rises, reflecting enhanced task completion and accuracy with more debugging opportunities. What the 'performance plateaus' mean  in line 524-525?

- The paper does not provide information on the cost of running AutoKaggle, which is essential for evaluating its performance and practical applicability. It's benifit to provide cost and total runtime to understand the performance. 

- The chosen baselines are not entirely convincing. Recent similar works, AIDE[1] and MLE-Agent[2] have shown remarkable capability in Kaggle competition settings. A comparative analysis with these recent works, particularly focusing on AutoKaggle’s unique advantages in effectiveness, efficiency, or other performance metrics, would highlight its distinct contributions to the field.

- A broader evaluation across various task types such as time series prediction, image classification, and text classification, are necessary, as these are critical and challenging categories in Kaggle competitions. The current experiments focus primarily on tabular datasets, leaving it unclear whether AutoKaggle is capable of handling more complex, domain-specific tasks. Can AutoKaggle complete such tasks? 

- What is the requirement of the LLM? Can AutoKaggle works well with gpt-3.5 or other open-sourced models?


[1] AIDE: the Machine Learning Engineer Agent(https://github.com/WecoAI/aideml)

[2] MLE-Agent: Your intelligent companion for seamless AI engineering and research (https://github.com/MLSysOps/MLE-agent)

questions:
Please refer to the questions in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720475984

thread_type:
official_review

<thread_object>
creator: reviewer_1dns

<content>
summary:
This paper introduces AutoKaggle, a pipeline to automatically solve Kaggle Competitions. The authors use 5 subparts in a row: a reader, a planner, a developer, a reviewer, and a summarizer. They use LLMs with RAG to develop code-based solutions, with code running, units tests. They evaluate their method on 5 Kaggle competition benchmarks.

strengths:
**Interesting problem.** With the LLMs (+RAG) becoming mature, the open source study of their integration into broader tools that can directly be applied to data science tasks, is the natural next step.

**Overall good presentation.** Even if some details are lacking to grasp the authors' exact contribution (notably in the figures), the overall presentation clearly demonstrates the problem and the approach set up to tackle it. 

**Interesting metrics and ablation studies.**

weaknesses:
**Lacking evaluation.** The evaluation is lacking comparison to existing AutoML baselines (*e.g. [1]) or explanations on why the authors are not comparing their method to any existing solution. If running such comparison is not possible at all, then the authors should provide explanations on why this is not feasible. 
While detailed reports are provided on their methods and the different components, as this works apply existing techniques, its evaluation is its core contribution. 
The authors should report (at least) the standard deviation, but e.g. violin plots to compare AutoKaggle's results of other kaggle competitors could help clearly situate where this automatic pipeline stands.

**Evaluation on a (previously) unknown dataset.** It seems that AutoKaggle has been designed to solve these datasets, so one cannot evaluate how much this method would transfer to another, previously unknown dataset.
It would be nice to provide the reader with how much out of the box your method is, maybe with a user study. It seems like its your core contribution, so having independent people trying AutoKaggle and commenting on how easy the setup and interaction is on a left out dataset would help people looking for such solutions.

**Figure 2 could be improved.** The figure could be split to separate the overall pipeline from details on some of its components. Most importantly, what part is using an LLM, what part is using a human expert ? This figure represents 70% of what the reader is looking for, it should provide first the overall intuition, and then enough details on specific core components that you want to highlight.

**You related work section is actually a background section.**
Your current related work covers some domains that are integrated within AutoKaggle. It thus feels more like a related work of your background section (what AutoKaggle builds upon). Is there any *e.g.* AutoML method that you can compare to ? Any method that addresses the same issue ?


[1] https://github.com/automl/CAAFE

questions:
* Did you do any finetuning over the used models, notably LLMs or are you using frozen models ?
* Why cannot you compare to any existing baselines ?
* Have you optimized the creation of your pipeline using these 5 kaggle competitions, or have you left out some of them, to evaluate on competitions you did not know at design time ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732468256688

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We extend our sincere appreciation to each of the three reviewers for the time dedicated to reviewing our manuscript and for the constructive feedback provided. Your insights and critiques are invaluable.

**Common Problems**

In reviewing the feedback from the reviewers, we identified and addressed two recurring themes:

1. Lack of Evaluation and Baseline Comparison for AutoKaggle 

We have added comparative experiments with the AIDE framework and included results and corresponding analysis of the AutoKaggle framework based on o1-mini in Section 3.2-Main Results. AIDE[1] is the top-performing framework in MLE-Bench[2]. The new main experimental results are presented in Section 3.2-Main Results(Table 1). The results demonstrate that AutoKaggle significantly outperforms the AIDE framework in terms of valid submission rate and comprehensive score across the eight datasets we evaluated.

1. Special Optimization of AutoKaggle for Evaluation Datasets 

AutoKaggle is not specifically optimized for any particular dataset. It is designed to provide a general end-to-end solution for all types of tabular datasets. Our goal is to simplify and optimize the data science workflow, enabling data scientists to handle daily tasks more efficiently. The development of AutoKaggle was based on three toy datasets from [3], rather than the Kaggle competitions we used for testing.

**Other Revisions**

In addition to incorporating a comparison with AIDE, we have made the following revisions to the article based on the reviewers' feedback:

1. Revised Evaluation Metrics: We updated the metrics to include Made/Valid Submission (sourced from MLE-Bench) and Comprehensive Score (sourced from [4]). The calculation method for the Comprehensive Score has been adjusted to 0.5 × Valid Submission + 0.5 × ANPS to better align with the evaluation of our framework.
2. Added Appendix B-Error Analysis: This section provides a detailed analysis of the error distribution encountered by AutoKaggle during the completion of data science tasks. It also describes the code correction methods used within AutoKaggle.
3. Added Appendix E-Case Study: We included a case study using the Titanic competition from Kaggle. This section details the phased workflow of AutoKaggle and presents some intermediate results to enhance understanding of the technical details of AutoKaggle.
4. Enhanced README in Anonymous GitHub Repository: We improved the README file to better explain how to use AutoKaggle. Additionally, sample results have been provided in the multi_agents/example_results/ directory for review.

More questions have been answered in the point-to-point responses. We hereby assure you that all of them will be resolved in the manuscript.

Collectively, we anticipate that, within our point-to-point responses, the detailed explications of our method and the inclusion of supplementary baseline comparisons will clarify the misunderstandings and substantively address the feedback.

We extend our heartfelt gratitude once again to all the reviewers for their meticulous and insightful critiques.

[1] AIDE: https://github.com/WecoAI/aideml

[2] Chan J S, Chowdhury N, Jaffe O, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering[J]. arXiv preprint arXiv:2410.07095, 2024.

[3] Hong S, Lin Y, Liu B, et al. Data interpreter: An LLM agent for data science[J]. arXiv preprint arXiv:2402.18679, 2024.

[4] https://github.com/geekan/MetaGPT/tree/2b160f294936f5b6c29cde63b8e4aa65e9a2ef9f/examples/di
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents AutoKaggle, an LLM creating a multi-agent system together with a library of hand-crafted ML tools in order to solve Kaggle problems. In my opinion, the reviewers did a very thorough job and have presented salient arguments about the suitability of this paper for IJCAI in its current form. While it is interesting to see that an agentic LLM can help with Kaggle competitions, one reviewer points out that a comparison to existing AutoML approaches is missing. Indeed, the authors added a comparison to AIDE. However, AutoML is not only "an LLM agent that generates solutions for machine learning tasks" (as written on the AIDE github page) but also portfolio and other approaches to automatize (parts of) ML. So, a larger discussion (and comparison) is in place. As it turn out Kaggle is partnering with the International Conference on Automated ML,see https://www.kaggle.com/automl-grand-prix. Another reviewer points out that some design choices and arguments for motivation are missing; currently, it reads more like what the authors have done, but it is not well placed into the research landscape. More importantly, an evaluation across different types of modalities (time series, text classification, image object detection, ...) is missing or argued why this is currently not so important. One reviewer also pointed out that an ablation study is missing, showing that the hand-crafted tools are not doing the job in the end. So, while the direction is super interesting, it is too early for publication, but we would like to encourage the authors to push  for one of the next venues. Please note that the overall judgment should not be taken as a statement regarding the usefulness of your research.