AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions

## Abstract

Abstract Data science tasks involving tabular data present complex challenges that require sophisticated problem-solving approaches.
We propose AutoKaggle, a powerful and user-centric framework that assists data scientists in completing daily data pipelines through a collaborative multi-agent system.
AutoKaggle implements an iterative development process that combines code execution, debugging, and comprehensive unit testing to ensure code correctness and logic consistency.
The framework offers highly customizable workflows, allowing users to intervene at each phase, thus integrating automated intelligence with human expertise.
Our universal data science toolkit, comprising validated functions for data cleaning, feature engineering, and modeling, forms the foundation of this solution, enhancing productivity by streamlining common tasks.
We selected 8 Kaggle competitions to simulate data processing workflows in real-world application scenarios.
Evaluation results demonstrate that AutoKaggle achieves a validation submission rate of 0.85 and a comprehensive score of 0.82 in typical data science pipelines, fully proving its effectiveness and practicality in handling complex data science tasks. 1 1 1 All code is available at https://github.com/multimodal-art-projection/AutoKaggle 2 2 2 The project homepage is https://m-a-p.ai/AutoKaggle.github.io/

## 1 Introduction

In recent years, with the rapid development of large language models (LLMs) (OpenAI, ; ), automated data science has gradually become possible.
LLM-based agents have shown great potential in the data domain, as they can automatically understand, analyze, and process data (Hassan et al., ; Lucas, ; Zhang et al., ), thereby promoting the democratization and widespread application of data science.

However, existing research still has significant shortcomings in addressing complex data science problems.
Many studies are limited to simple, one-step data analysis tasks (Zhang et al., ; Hu et al., ), which are far from the actual application scenarios of data science.
While recent work (Jing et al., ) attempts to evaluate data science capabilities through more comprehensive tasks, it still focuses on relatively constrained scenarios that represent only portions of a complete data science pipeline.
Other research relies on pre-built knowledge bases (Guo et al., ), raising the barrier to use and limiting the flexibility and adaptability of solutions.
Moreover, current research focuses excessively on improving task completion rates and optimizing performance metrics, while neglecting the interpretability and transparency of intermediate decision-making steps in logically complex data science tasks.
This neglect not only affects users’ understanding of solutions but also diminishes their credibility and practicality in real-world applications.

To address these issues, we propose AutoKaggle, a universal multi-agent framework that provides data scientists with end-to-end processing solutions for tabular data, helping them efficiently complete daily data pipelines and enhance productivity.
AutoKaggle has the following features:

*(i)* Phase-based Workflow and Multi-agent Collaboration. AutoKaggle employs a phase-based workflow and multi-agent collaboration system. It divides the data science competition process into six key phases: background understanding, preliminary exploratory data analysis, data cleaning (DC), in-depth exploratory data analysis, feature engineering (FE), and model building, validation, and prediction (MBVP).
To execute these phases, five specialized agents (Reader, Planner, Developer, Reviewer, and Summarizer) work collaboratively to execute these phases, from problem analysis to report generation.

*(ii)* Iterative Debugging and Unit Testing.
AutoKaggle ensures code quality through iterative debugging and unit testing.
The Developer employs three main tools (code execution, debugging, and unit testing) to verify both syntactic correctness and logical consistency.

*(iii)* Machine Learning Tools Library. AutoKaggle integrates a comprehensive machine learning tools library covering data cleaning, feature engineering, and model building, validation, and prediction. The library includes expert-written code snippets and custom tools, enhancing code generation efficiency and quality.
By combining predefined tools with self-generated code, AutoKaggle handles complex tasks while reducing reliance on LLMs for domain-specific knowledge.

*(iv)* Comprehensive Reporting. AutoKaggle generates detailed reports after each phase and at the competition’s conclusion, showcasing its decision-making process, key findings, actions, and reasoning.
This feature makes the data processing workflows transparent, increasing user trust in AutoKaggle.

AutoKaggle provides a universal and comprehensive solution for a wide variety of data science tasks.
By simply providing a task overview, it can automatically complete the entire process from development to testing, making it exceptionally easy to use.
AutoKaggle is highly adaptable, allowing users to customize it according to their specific needs.
Moreover, it offers clear interpretability throughout the automated data science process, enhancing users’ understanding and trust in the system.

We chose competitions from the Kaggle platform to evaluate our framework.
Kaggle data science competitions simulate the real challenges faced by data scientists, covering the complete process from data cleaning to model deployment.
These competitions require participants to execute a series of complex and interdependent tasks. These include: data cleaning and preprocessing, exploratory data analysis, feature engineering, and modeling.
Each step demands professional knowledge and meticulous planning, often necessitating multiple iterations.
This complexity makes Kaggle an ideal platform for assessing the effectiveness of data science automation tools.
In the 8 Kaggle data science competitions we evaluated, AutoKaggle achieved 0.85 in valid submission rate and 0.82 in comprehensive score. We summarize our contributions as follows:

- •
We propose AutoKaggle, a novel multi-agent framework for Kaggle data science competitions, achieving high task completion rates and competitive performance above the average human level in our evaluations.
- •
We introduce a phase-based workflow integrated with multi-agent collaboration, incorporating iterative debugging and unit testing, which systematically addresses the complexities of data science tasks and ensures robust, correct code generation.
- •
We develop a machine learning tools library and integrate it into our framework, enhancing code generation efficiency and quality for complex data science tasks.
- •
We implement a comprehensive reporting system that provides detailed insights into the decision-making process at each phase, making AutoKaggle both a solution provider and an educational tool for data science competitions, thereby contributing to the democratization of data science skills.

Figure: Figure 1: Overview of AutoKaggle. AutoKaggle integrates a phase-based workflow with specialized agents (Reader, Planner, Developer, Reviewer, and Summarizer), iterative debugging and unit testing, a comprehensive machine learning tools library, and detailed reporting.
Refer to caption: x13.png

## 2 Related Work

### 2.1 Large Language Model Agents

A concise framework of agents consists of brain, perception, and action modules (Xi et al., ). The perception module processes external information, the brain plans based on that information, and the action module executes these plans (Xi et al., ; Zhou et al., ). LLMs, acting as brain modules, exhibit impressive zero-shot abilities and are applied in fields like data science and music composition (Brown et al., ; Hong et al., ; Deng et al., ). While the chain-of-thought method improves reasoning (Wei et al., ), it often leads to hallucinations due to internal representations (Yao et al., ). The ReAct paradigm addresses this by integrating thoughts and actions, refining outputs through interaction with external environments (Yao et al., ; Madaan et al., ; Shinn et al., ; Zhou et al., ).

### 2.2 Multi-agents

While an individual agent can achieve basic natural language processing (NLP) tasks, real-world tasks have higher complexities. In human societies, people chunk complex tasks into simple sub-tasks that different people can easily handle. Inspired by this division of labor principle, multi-agent systems enhance performance (Talebirad & Nadiri, ) using cooperative interactions (Li et al., ) to achieve shared goals. Another interaction method is adversarial interactions (Lewis et al., ), where several agents compete with each other for better results, or one agent critiques and reviews the generation of another agent (Gou et al., ).

### 2.3 Data Science Agents

In order to address the well-defined requirements of data science tasks, a feasible approach is to design hierarchical systems (Hong et al., ; Zhang et al., ; Chi et al., ) to complete tasks such as task understanding, feature engineering, and model building. In each hierarchy, separately design two agents for the code planning and code generation respectively (Hong et al., ), and use unit tests (Zhang et al., ) to verify the quality of code generation. Beyond self-debugging by autonomous multi-agents, human-in-the-loop (Hong et al., ; Zhang et al., ) mechanisms also provide oversight and corrections to LLM outputs, reducing hallucinations in each hierarchy. Tang et al. () introduces ML-Bench, a benchmark for language agents for machine learning tasks.

In summary, multi-agent systems and LLM-based agents have demonstrated significant potential across domains such as NLP and data science. While individual agents excel in basic tasks, integrating multiple agents is crucial for tackling complex real-world challenges. By combining task-specific agents with human-in-the-loop mechanisms and unit testing, these systems improve code quality and address issues like hallucinations. Our framework, AutoKaggle, advances these efforts by integrating LLM-based reasoning with multi-agent collaboration, ensuring adaptability, correctness, and user control in data science competitions.

## 3 AutoKaggle

### 3.1 Overall Framework

In this section, we introduce AutoKaggle, a fully automated, robust, and user-friendly framework designed to produce directly submittable prediction results using only the original Kaggle data. Given the diversity of data science problems, the range of potential solutions, and the need for precise reasoning and real-time understanding of data changes, effectively handling complex data science tasks on Kaggle is challenging. Our technical design addresses two primary issues: *(i)* how to decompose and systematically manage complex data science tasks; and *(ii)* how to efficiently solve these tasks using LLMs and multi-agent collaboration.

The core concept of AutoKaggle is phase-based multi-agent reasoning. This method leverages LLMs to reason and solve tasks within a structured workflow, addressing different facets of the data science process through the collaboration of multiple agents. AutoKaggle comprises two main components: a phase-based workflow and a multi-agent system, which complement each other, as shown in Figure 1.

Phase-based Workflow. The data science process is divided into six key phases: understanding the background, preliminary exploratory data analysis, data cleaning, in-depth exploratory data analysis, feature engineering, and model building, validation, and prediction. Data cleaning, feature engineering, and model building, validation, and prediction are fundamental processes required for any data science competition. We designed two additional data analysis phases to provide essential information and insights for data cleaning and feature engineering, respectively. Given that our initial input is only an overview of a Kaggle data science competition and the raw dataset, we added a background understanding phase to analyze various aspects of the competition background, objectives, file composition, and data overview from the raw input. This structured approach ensures that all aspects of the problem are systematically and comprehensively addressed, with different phases decoupled from each other. It allows thorough unit testing at each phase to ensure correctness and prevent errors from propagating to subsequent phases.

Figure: Figure 2: Iterative debugging and testing.
Refer to caption: x14.png

Multi-agent System. The system consists of five specialized agents: Reader, Planner, Developer, Reviewer, and Summarizer. Each agent is designed to perform specific tasks within the workflow. They collaborate to analyze the problem, develop strategies, implement solutions, evaluate results, and generate comprehensive reports. Detailed setup and interaction processes of agents are described in Appendix C.1.6.

We summarize the pseudo-code of AutoKaggle in Algorithm 1. Let $\mathcal{C}$ represent the competition, $\mathcal{D}$ the dataset, and $\Phi=\{\phi_{1},\phi_{2},\dots,\phi_{6}\}$ the set of all phases in the competition workflow. For each phase $\phi_{i}$, a specific set of agents $\mathcal{A}_{\phi_{i}}$ is assigned to perform various tasks. The key agents include Planner, Developer, Reviewer, and Summarizer.

### 3.2 Development Based on Iterative Debugging and Testing

In AutoKaggle, the Developer adopts a development approach based on iterative error correction and testing. It ensures the robustness and correctness of generated code through iterative execution, debugging, and testing.

Figure 2 shows the overall process of iterative debugging and testing. Specifically, the Developer first generates code based on the current state $\mathbf{s}_{t}$, the plan $P_{\phi_{i}}$ created by the Planner, and the historical context $\mathcal{H}$: $C_{\phi_{i}}=\text{GenerateCode}(\mathbf{s}_{t},P_{\phi_{i}},\mathcal{H})$. $C_{\phi_{i}}$ is the generated code for phase $\phi_{i}$, and $\text{GenerateCode}(\cdot)$ represents the code generation function executed by the Developer. The historical context $\mathcal{H}$ includes previous phases’ code, outputs, and other relevant information from other agents’ activities.

After the initial code generation, it enters an iterative debugging and testing process. This process can be described by Algorithm 2.

Developer utilize three primary tools: code execution, code debugging, and unit testing.

*(i) Code Execution.* The Code Execution tool runs the generated code and captures any runtime errors. When an error is detected, the system restores a file to record the error messages.

*(ii) Code Debugging.* The Code Debugging tool analyzes error messages and attempts to fix the code. It utilizes error messages along with the current code and historical context to generate fixes: $C^{\prime}_{\phi_{i}}=\text{DebugCode}(C_{\phi_{i}},E_{\phi_{i}},\mathcal{H})$. $C^{\prime}_{\phi_{i}}$ is the debugged version of the code.

Following previous work (Tyen et al., ), we designed the debugging process into three main steps: error localization, error correction, and merging of correct and corrected code segments. We set a maximum of 5 attempts for the Developer to self-correct errors. Additionally, we’ve introduced an assistance mechanism. We record all error messages encountered during the debugging process. When the number of correction attempts reaches 3, the Developer evaluates the feasibility of continuing based on historical information. If past error messages are similar, it suggests that the Developer might lack the ability to resolve this particular error, and continuing might lead to a loop. In such cases, we allow the Developer to exit the correction process and regenerate the code from scratch.

*(iii) Unit Testing.* Unit testing runs predefined tests to ensure code meets requirements. For each phase $\phi_{i}$, a set of unit tests $T_{\phi_{i}}$ is defined: $T_{\phi_{i}}=\{t_{1},t_{2},\dots,t_{k}\}$. The unit testing process can be represented as: $R_{\phi_{i}}=\text{ExecuteUnitTests}(C_{\phi_{i}},T_{\phi_{i}})$. $R_{\phi_{i}}$ is the set of test results, with each result $r_{j}\in\{0,1\}$ indicating whether the corresponding test passed (1) or failed (0).

In complex and accuracy-demanding tasks like Kaggle data science competitions, merely ensuring that the code runs without errors is not enough. These competitions often involve intricate data processing and sophisticated algorithms, where hidden logical errors can significantly affect the final results. Therefore, it is necessary to design meticulous unit tests that not only verify the correctness of the code but also ensure it meets the expected logical and performance standards. Otherwise, hidden errors may accumulate through successive phases, making the completion of each subsequent phase increasingly difficult. For example, unnoticed logical defects during the data cleaning phase may lead to poor feature extraction, thereby affecting the model building in subsequent phases.

To mitigate these risks, unit tests for each phase must be carefully designed to cover a wide range of scenarios, including edge cases and potential failure points. This involves not only checking the correctness of the output but also ensuring that the intermediate steps conform to the expected logic. For instance, in the data cleaning phase, unit tests should verify whether missing values are handled correctly, outliers are appropriately managed, and data transformations are accurately applied.

By implementing comprehensive unit tests, we can catch and correct errors early in the development process, preventing them from propagating to later phases. This systematic testing approach ensures that the code at each phase is not only error-free but also functionally correct and aligned with the overall project goals.

In conclusion, the iterative debugging and testing method employed by Developer ensures the generation of robust, error-free, and effective code for each phase of the competition. By employing advanced error handling, iterative debugging, and comprehensive unit testing, the system can adapt to various challenges and consistently produce high-quality code outputs.

### 3.3 Machine Learning Tools Library

Generating machine learning code from scratch using LLMs can be challenging due to the intricacies of various tasks. These models need to encompass specialized knowledge across a range of processes, from data processing and feature engineering to model building, validation, and prediction. In many cases, leveraging expert-crafted machine learning tools is more efficient than relying solely on LLM-generated code. This is because LLMs often lack domain-specific expertise, potentially leading to suboptimal or inaccurate code. Furthermore, when tasked with complex operations, the generated code may suffer from syntactical or logical errors, increasing the likelihood of failures.

Our machine learning library is categorized into three core toolsets: data cleaning, feature engineering, and model building, validation, and prediction, each serving a specific role in the workflow. The data cleaning toolkit comprises seven tools, including FillMissingValues, RemoveColumns
WithMissingData, DetectAndHandleOutliersZscore, DetectAndHandleOutliersIqr, RemoveDuplicates, ConvertDataTypes and FormatDatetime, all designed to ensure clean, consistent, and reliable data preparation. The feature engineering module encompasses eleven tools aimed at enhancing model performance, such as OneHotEncode, FrequencyEncode, CorrelationFeatureSelection, and ScaleFeatures, employing various techniques like correlation analysis and feature scaling to optimize data representation. The model building, validation, and prediction category provides TrainAndValidationAndSelectTheBestModel to support the full model development lifecycle, including model selection, training, evaluation, prediction, ensemble integration, and hyperparameter optimization, facilitating robust model deployment and effective performance. Each tool comes with comprehensive explanations, input/output specifications, anomaly detection, and error handling guidance.

This comprehensive library is crucial for efficient multi-agent collaboration in tackling complex Kaggle competitions. Each tool provides standardized, reliable functionality, enabling AutoKaggle to seamlessly share and process data, enhance feature quality, and optimize model performance, ultimately improving overall workflow efficiency and ensuring coordinated, high-quality solutions in a competitive environment. Moreover, our machine learning library reduces the burden on AutoKaggle in detailed programming tasks, enabling them to focus more on higher-level task planning and code design. This shift of focus allows AutoKaggle to navigate complex tasks more effectively, ultimately improving their overall performance. More details of our machine learning tools can be found in Appendix C.3.

## 4 Experiments

### 4.1 Experimental Setup

Task Selection.
We select eight Kaggle competitions that predominantly use tabular datasets, focusing on classification and regression tasks. These competitions are categorized into two types: classic Kaggle and Recent Kaggle. Classic Kaggle competitions are those that begin before October 2023 with at least 500 participants, whereas Recent Kaggle competitions begin in 2024 or later. As our analysis relies on GPT-4o, which is trained on data available until October 2023, it includes most of the Classic Kaggle competitions. To evaluate the generalization capabilities of AutoKaggle, we therefore focus on competitions initiated after 2024. Additionally, we classify these competitions into three difficulty levels: easy, medium, and hard. For each dataset, we access the corresponding competition’s homepage on Kaggle, extract content from the overview and data description sections, and compile this information into a file named overview.txt. This file, along with the original competition data files, forms the primary input for AutoKaggle. More details of our datasets can be found in Appendix B.

Notably, we do not incorporate the nine tabular datasets from MLE-Bench (Hong et al., ) due to their substantial size, which would significantly increase computational runtime. Resource limitations prevent us from adhering to MLE-Bench’s experimental setup, which specifies a 24-hour participation window per agent and a 9-hour code execution timeout.

**Table 1: Made submission, valid submission and comprehensive score on 8 Kaggle tasks. Each experiment is repeated with 5 trials. The best performances on individual tasks are underlined, and the best performances across all tasks are bolded.**
|  |  | Classic | Recent |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | Setting / Task | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Task 7 | Task 8 | Avg. |
| Made Submission | AutoKaggle gpt-4o | 1 | 0.80 | 0.80 | 1 | 0.80 | 0.80 | 0.80 | 0.80 | 0.85 |
| AutoKaggle o1-mini | 1 | 0.60 | 0.60 | 1 | 0.60 | 0.80 | 0.60 | 0.60 | 0.73 |  |
| AIDE gpt-4o | 1 | 0.40 | 0.20 | 0.60 | 1 | 0.80 | 0.80 | 0 | 0.60 |  |
| \hdashline |  |  |  |  |  |  |  |  |  |  |
| Valid Submission | AutoKaggle gpt-4o | 1 | 0.80 | 0.80 | 1 | 0.80 | 0.60 | 0.80 | 0.80 | 0.83 |
| AutoKaggle o1-mini | 1 | 0.60 | 0.60 | 1 | 0.60 | 0.60 | 0.60 | 0.60 | 0.70 |  |
| AIDE gpt-4o | 1 | 0.40 | 0.20 | 0.40 | 1 | 0.80 | 0.80 | 0 | 0.58 |  |
| \hdashline |  |  |  |  |  |  |  |  |  |  |
| Comprehensive Score | AutoKaggle gpt-4o | 0.888 | 0.786 | 0.831 | 0.862 | 0.810 | 0.728 | 0.848 | 0.812 | 0.821 |
| AutoKaggle o1-mini | 0.879 | 0.680 | 0.729 | 0.863 | 0.709 | 0.735 | 0.742 | 0.735 | 0.759 |  |
| AIDE gpt-4o | 0.872 | 0.597 | 0.542 | 0.561 | 0.918 | 0.793 | 0.848 | 0 | 0.641 |  |

Evaluation metric. We evaluate the capability of the AutoKaggle from four perspectives: Made Submission, Valid Submission, Average Normalized Performance Score and Comprehensive Score.
The first two metrics refer to MLE-bench and are primarily used to assess the ability to generate a submission.csv file. The last two metrics come from Data Interpreter (Chan et al., ), we made modifications to adapt them to the evaluation of our framework.

*(i) Made Submission (MS).*
Made Submission refers to the percentage of times a submission.csv file is generated.

*(ii) Valid Submission (VS).*
Valid Submission indicates the percentage of those submission.csv files that are valid—meaning they can be successfully submitted to the Kaggle website, produce results without errors, and have no issues related to data scale or category mismatches.

**Table 2: Ablation study on machine learning tools. Evaluated with completion rate and comprehensive score. Best performance are underlined.**
|  |  | Task 1 | Task 2 | Task 3 | Task 5 | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| VS | No Tools | 0.80 | 0.60 | 0.50 | 0.40 | 0.58 |
| DC Tools | 0.80 | 0.70 | 1.00 | 1.00 | 0.88 |  |
| DC & FE Tools | 0.80 | 0.60 | 0.60 | 0.60 | 0.65 |  |
| All Tools | 1.00 | 0.80 | 0.80 | 0.80 | 0.85 |  |
| \hdashline |  |  |  |  |  |  |
| CS | No Tools | 0.781 | 0.697 | 0.666 | 0.602 | 0.687 |
| DC Tools | 0.781 | 0.721 | 0.928 | 0.909 | 0.835 |  |
| DC & FE Tools | 0.787 | 0.684 | 0.735 | 0.713 | 0.730 |  |
| All Tools | 0.888 | 0.786 | 0.831 | 0.810 | 0.829 |  |

*(iii) Comprehensive Score (CS).*
In the evaluations, performance metrics are divided into two categories: bounded metrics, which range from 0 to 1 where higher values indicate better performance, and unbounded metrics, where lower values denote superior performance. To normalize these different types of metrics, we utilize the normalized performance score (NPS), defined as follows:

$$ $\text{NPS}=\begin{cases}\frac{1}{1+s},&\text{if }s\text{ is smaller the better }\\ s,&\text{otherwise.}\end{cases}$ (1) $$

For multiple trials of a task, we calculate the Average Normalized Performance Score (ANPS) as the average of the successful attempts:

$$ $\text{ANPS}=\frac{1}{T_{s}}\sum_{t=1}^{T_{s}}\text{NPS}_{t}$ (2) $$

where $T_{s}$ represents the total number of successful attempts for a task, and $\text{NPS}_{t}$ is the NPS value for the $t$-th attempt.

To comprehensively evaluate both the pass rate and the average performance, we define the Comprehensive Score (CS) as the average of VS and ANPS:

$$ $\text{CS}=0.5\times\text{VS}+0.5\times\text{ANPS}$ (3) $$

Experiment Details.
We evaluated AutoKaggle’s performance based on both GPT-4o and o1-mini models.
Notably, different models were assigned to specific agents based on their functional requirements.
The Reader, Reviewer, and Summarizer, which perform tasks requiring minimal logical reasoning and coding capabilities, were implemented using the GPT-4o-mini model.
The Planner, responsible for task decomposition and planning that demands sophisticated logical reasoning, operates on either the GPT-4o or o1-mini model.
Although the Developer’s tasks traditionally necessitate advanced logical reasoning and coding skills, the Planner’s effective task decomposition methodology has moderated these requirements, therefore it is based on GPT-4o model.

In our experiments, Each task undergoes five trials, with each phase in the workflow allowing for a maximum of three iterations.
During an iteration, the Developer may debug the code up to five times.
If unsuccessful, they proceed with the same phase, deriving insights and adjusting strategies based on previous attempts.
Failure to resolve issues after three iterations is considered a definitive failure.

Baseline. We employ AIDE (Schmidt et al., ) as our baseline, which is the best-performing framework in MLE-bench evaluation results. We use AIDE’s default settings, only modifying agent.base.model to the GPT-4o model.

### 4.2 Main results

The comprehensive performance of AutoKaggle across 8 Kaggle data science competitions is presented in Table 1.
In order to facilitate understanding, we uniformly name the eight tasks as task 1-8. The real task names and detailed dataset information are available in Appendix B.

Figure: Figure 3: Average normalized performance score for different settings/tasks.
Refer to caption: x15.png

Made submission and Valid submission.
We first evaluated the success rate of valid submission.csv file generation across different experimental configurations.
The AutoKaggle framework, implemented with GPT-4o, demonstrated superior performance with an average valid submission rate of 83% across all 8 Kaggle tasks, surpassing the AIDE framework by 28%.
These results underscore the robustness of our framework in executing comprehensive data science workflows.
While the AIDE framework successfully processed Tasks 1-7, which involved single-variable classification or regression on tabular data, it failed to generate valid submissions for Task 8, a multi-variable classification problem.
This differential performance demonstrates our framework’s versatility in handling diverse tabular data tasks.

Another interesting observation is that within the AutoKaggle framework, the GPT-4o model achieved better results than the o1-mini model, despite the latter’s purported superior reasoning capabilities.
This performance difference emerged solely from varying the model used in the Planner component.
We hypothesize that this counterintuitive result stems from o1-mini’s tendency toward excessive planning complexity, which proves disadvantageous in our streamlined, phase-based workflow architecture.
This same consideration influenced our decision to maintain GPT-4o as the Developer’s base model, as our experiments indicated that an o1-mini-based Developer would significantly increase code verbosity, expanding 100-line solutions to approximately 500 lines through the introduction of superfluous components such as logging systems.

Comprehensive Score.
Subsequently, we compared the overall performance of different settings across 8 Kaggle tasks.
AutoKaggle with GPT-4o achieved the highest comprehensive score in 5 tasks and demonstrated the best overall performance.
Figure 3 illustrates the comparison of different settings based on the average normalized performance score metric, where AutoKaggle with o1-mini achieved the highest overall score.
This indicates that although the o1-mini-based Planner generated overly complex plans that increased development difficulty, successfully executing these plans according to specifications led to superior performance outcomes.

Figure: Figure 4: Left. Debugging time and Right. Average performance in competitions.
Refer to caption: x16.png

### 4.3 Ablation study

Apart from the modules involved in the ablation study, all other experimental settings are identical to those in the formal experiment.

Study on Machine Learning Tools. To evaluate the effectiveness of the machine learning tools module and the impact of tools across different phases on the results, we conduct ablation experiments. We begin without any tools and progressively add them at each phase until all machine learning tools are implemented. The results are presented in Table 2. Notably, the completion rate increases by 30% with the use of data cleaning phase tools, and by 27.5% when all tools are utilized, compared to the scenario with no tools. However, the completion rate exhibits a decline during the feature engineering phase, particularly in the house prices and academic success competitions. This decline can be attributed to the relatively large number of features involved, alongside the complexity and high encapsulation of the tools used in this phase, which necessitate the addition and removal of features, thereby complicating their usage. Furthermore, this complexity poses challenges for Developers in debugging erroneous code. As illustrated in Figure 4 (a), the frequency of debugging instances is greater when employing tools from the feature engineering phase.

**Table 3: Ablation study on unit tests. Better performance are underlined.**
|  |  | Task 1 | Task 2 | Task 3 | Task 5 | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| CR | w/o Unit Tests | 0.20 | 0 | 0.20 | 0 | 0.10 |
| w/ Unit Tests | 1.00 | 0.80 | 0.80 | 0.80 | 0.85 |  |
| \hdashline |  |  |  |  |  |  |
| CS | w/o Unit Tests | 0.478 | 0 | 0.482 | 0 | 0.240 |
| w/ Unit Tests | 0.888 | 0.831 | 0.786 | 0.810 | 0.829 |  |

Figure 4 (b) provides a clearer comparison, demonstrating that while the best normalized performance scores across four scenarios are similar, the completion rate significantly increases with the use of the tool. This suggests that although the machine learning tool library we develop does not substantially elevate the solution’s upper limit, it functions as a more stable tool that enhances AutoKaggle’s completion rate. This outcome aligns with expectations, as the machine learning tool library is a redevelopment based on widely used libraries such as pandas and scikit-learn. It does not introduce new functionalities but instead combines and re-packages existing ones, incorporating error handling and manual testing to ensure compatibility with our framework.

Study on Unit Tests. To evaluate the effectiveness of the unit tests module, we conduct ablation experiments. The results are presented in Table 3. In the absence of unit tests, the completion rate significantly decreases, making it nearly impossible to complete the tasks. This emphasizes that for tasks like data science, which demand high levels of precision and logic, it is not enough for each phase of the code to merely execute without errors. Comprehensive unit testing is required to ensure that the code is logical and achieves the objectives of each phase.

**Table 4: Error Types of AutoKaggle in the Problem-Solving Stage**
| Error Type (Count) | Description |
| --- | --- |
| Value Error (49) | Fail to match the expected type or range of the input values |
| Key Error (44) | Attempt to access a dictionary element using a key that does not exist |
| File Error (8) | Attempt to access a file that does not exist in the specified location |
| Model Error (8) | Incorrect setup in the parameters or structure of a model, leading to operational failures |
| Type Error (25) | Mismatch between expected and actual data type, leading to operational failure |
| Timeout Error (6) | Failure to complete a process within the allocated time period |
| Index Error (3) | Attempt to access an element at an index that is outside the range of a list or array |
| Assertion Error (1) | An assertion condition in the code is not met, indicating an unmet expected constraint |
| Name Error (2) | Use of an undeclared variable that is not recognized by the system |
| Attribute Error (2) | Attempt to access an attribute or method that does not exist for an object |
| Indentation Error (1) | Incorrect indentation disrupts code structure, preventing proper parsing |

Study on Debugging Times.
We conduct ablation experiments to investigate the impact of the number of allowed debugging times on the results. The experimental setup permits five code debugging attempts within each phase, with each phase being executable up to three times. Consequently, we analyze scenarios with allowable corrections set at 0, 5, and 10. The results are shown in Figure 5. It can be observed that when AutoKaggle is required to pass without any errors, there is only one successful record on the Titanic task. Allowing five debugging attempts significantly improves the completion rate, and further increases in allowable debugging attempts lead to rises in all metrics. This demonstrates the efficacy of our code debugging module. However, the performance plateaus when the number of allowable debugging attempts is set to 10 and 15, suggesting that the agent’s self-correction abilities are limited. There are complex errors that it cannot resolve independently, and further increasing the number of allowable debugging attempts does not address these errors.See more details in subsection 4.4.

Figure: Figure 5: Comprehensive Score across different debugging times.
Refer to caption: x18.png

Study on Competition Date.
To further evaluate the generalization capabilities of our AutoKaggle framework, we conducted an analysis stratified by competition dates.
Tasks 1-4 corresponded to competitions potentially included in the training data of models such as GPT-4o and O1-mini, while tasks 5-8 were derived from competitions launched in the current year.
This temporal stratification enabled us to assess the framework’s performance on out-of-distribution tasks.
For classic Kaggle tasks, AutoKaggle with GPT-4o achieved a valid submission rate of 0.90 and a comprehensive score of 0.842.
On recent tasks, these metrics were 0.75 and 0.800 respectively, demonstrating only marginal performance degradation.
These results indicate that our task decoupling approach and predefined execution pathways enable effective handling of novel competitions, even in scenarios where the underlying model lacks prior exposure to the domain.

### 4.4 Error Analysis

In each subtask phase of AutoKaggle, errors may occur, with data cleaning and feature engineering experiencing the highest error rates at 25% and 22.5%, respectively. Notably, failures during the feature engineering phase result in direct competition failures in 31.25% of cases.

In the context of the proposed AutoKaggle framework, which aims to assist data scientists in solving complex tabular data challenges through a collaborative multi-agent system, Table 4 provides an overview of the different types of errors encountered during the iterative development process. AutoKaggle’s workflow includes code execution, debugging, and comprehensive unit testing, and the listed errors are indicative of the various challenges encountered while automating these stages. The most frequently observed errors are Value Errors (49 occurrences), related to mismatched input types or ranges, and Key Errors (44 occurrences), resulting from attempts to access non-existent dictionary keys. Additionally, Type Errors (25 occurrences) and Model Errors (8 occurrences) highlight operational issues due to data type mismatches or incorrect model configurations, respectively. The table also details other errors such as Timeout, FileNotFound, and Index Errors, each contributing to the debugging process. Understanding these error types is crucial for improving AutoKaggle’s robustness and aligning automated workflows with human interventions, ultimately enhancing productivity in typical data science pipelines.

In addition, we provide a detailed debugging process for developers. Below, we illustrate this using a FileNotFoundError as an example of the debugging workflow:

- •
Error Localization: The developer initially encounters issues executing a Python script involving file-saving operations with libraries like Matplotlib and Pandas. The specific error, FileNotFoundError, is traced to nonexistent directories or incorrect file paths. Through an iterative analysis, the problematic sections of the code are identified, focusing on the need to properly manage directory paths and handle filenames.
- •
Error Correction: To address these issues, several modifications are suggested. First, the importance of ensuring that directories exist before performing file operations is highlighted by incorporating os.makedirs to create any missing directories. Additionally, a filename sanitization approach is recommended to prevent errors related to invalid characters in file paths. A custom sanitize_filename function is introduced to ensure filenames contain only valid characters, thereby avoiding issues caused by special symbols or whitespace.
- •
Merging Correct and Corrected Code Segments: The final step involves merging the corrected segments back into the original code to create a seamless and robust solution. The revised script includes improvements such as verifying directory existence, creating necessary directories, and applying filename sanitization to ensure compatibility across different operating systems. The corrected code is delivered with a focus on enhancing reliability, particularly in file-saving processes, making it resilient against common pitfalls like missing directories or invalid filenames.

## 5 Conclusion

In this paper, we introduce AutoKaggle, a robust framework designed to leverage phase-based workflows and multi-agent collaboration for solving complex Kaggle data science competitions.
AutoKaggle employs an iterative development process, incorporating thorough code debugging, unit testing, and a specialized machine learning tools library to address the intricate requirements of data science tasks.
Our framework enhances reliability and automation in managing sophisticated data workflows, while maintaining user control through customizable processes.
Extensive evaluations across various Kaggle competitions demonstrate AutoKaggle’s effectiveness, marking a significant advancement in AI-assisted data science problem-solving and expanding the capabilities of LLM-based systems in tackling real-world challenges.

## Appendix A Algorithm

Figure: Algorithm 1 AutoKaggle Workflow

Figure: Algorithm 2 Development based on Iterative Debugging and Testing

## Appendix B Detailed Dataset Description

Here is the detailed description of our dataset. Note that we use task labels to represent the different datasets. Task 1 refers to Titanic (Cukierski, ), Task 2 refers to Spaceship Titanic (Addison Howard, ), Task 3 refers to House Prices (Anna Montoya, ), Task 4 refers to Monsters (Kan, ), Task 5 refers to Academic Success (Walter Reade, ), Task 6 refers to Bank Churn (Walter Reade, ), Task 7 refers to Obesity Risk (Walter Reade, ), and Task 8 refers to Plate Defect (Walter Reade, ).

Our framework deliberately avoids selecting competitions with excessively large datasets. The reason for this is that larger datasets significantly extend the experimental runtime, making it impractical to dedicate a machine to a single experiment for such prolonged periods.

First, we intentionally avoided selecting competitions with datasets that were too large, as larger datasets can significantly extend the experimental runtime, making it impractical to use a single machine for extended experiments. Second, we adhered to real-world competition settings by generating submission files and submitting them manually for evaluation. Simply splitting the training data would result in a test set with a distribution very similar to the training data, which could inflate performance metrics—similar to the difference often seen between validation scores and real test scores. Third, our dataset clearly identifies the contest type, i.e., tabular data. Fourth, since datasets for large language modeling include publicly available Kaggle contests, we selected only those released after 2024. Our framework requires agents to independently interpret contest tasks, understand the data, and determine appropriate optimization strategies without relying on predefined guidance.”

**Table 5: Selected Kaggle tasks. For each task, we show its number, category, difficulty level, number of teams, train size and test size in dataset.**
| Category | No. | Task Name | Task | Level | Teams | Train | Test |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Classic | 1 | Titanic | Classification | Medium | 13994 | 891 | 418 |
| 2 | Spaceship Titanic | Classification | Easy | 1720 | 8693 | 4277 |  |
| 3 | House Prices | Regression | Medium | 4383 | 1460 | 1459 |  |
| 4 | Monsters | Classification | Easy | 763 | 371 | 529 |  |
| \hdashline |  |  |  |  |  |  |  |
| Recent | 5 | Academic Success | Regression | Medium | 2684 | 76.5K | 51K |
| 6 | Bank Churn | Regression | Easy | 3632 | 165K | 110K |  |
| 7 | Obesity Risk | Classification | Easy | 3587 | 20.8K | 13.8K |  |
| 8 | Plate Defect | Regression | Medium | 2199 | 19.2K | 12.8K |  |

## Appendix C Implementation Details

### C.1 Agent Details

#### C.1.1 Agent Base

The base agent is a father class of other agents (Reader, Planner, Developer, Reviewer, and Summarizer) in the AutoKaggle. This agent can act with various tools for tasks related to data analysis, model evaluation, and document retrieval etc.

#### C.1.2 Reader

Reader is designed for reading documents and summarizing information. It processes overview.txt in each competition, subsequently providing a well-organized summary of the competition’s background

#### C.1.3 Planner

Planner is designed for creating task plans and roadmaps. The agent’s main function is to structure and organize tasks into executable plans, primarily by leveraging available tools and previously generated reports.

#### C.1.4 Developer

Developer is responsible for implementing and debugging code based on the structured plans generated by the Planner. The Developer’s key function is to translate the high-level task roadmap into executable code, resolve any arising issues, and perform unit tests to ensure the functionality of the solution.

#### C.1.5 Reviewer

Reviewer is responsible for evaluating the performance of other agents in completing tasks and providing constructive feedback.

#### C.1.6 Summarizer

Summarizer is responsible for generating summaries, designing questions, and reorganizing both questions and answers to produce structured reports based on the competition phases.

### C.2 Unit Tests

In data science competitions, code generated by agents may be executable in the Python interpreter, but this execution does not guarantee correctness.
To ensure that data dependencies are properly handled, a Unit Test Tool is necessary.
In our research, where the framework operates iteratively, we aim to separate tasks corresponding to different states in data science competitions.
Each phase builds upon the results of the previous one, making it crucial to confirm that logic remains sound, data processing is accurate, and information transfers seamlessly from one state to the next.
Our Unit Test Tool plays a key role in supporting the self-refine phase of LLM agents.

We developed unit tests (in the accompanying Table 6) based on issues identified during the execution of weak baseline, strong baseline and our AutoKaggle.
If the code fails to run in the Python interpreter, an error message is relayed to the agent Reviewer.
If the code passes this initial stage, it progresses to the Unit Test Tool, where all required tests are executed in a loop.
If a test fails, the reason is logged as short-term memory and passed to the next review state.
The review and planning stages work in an adversarial interaction: the review phase compiles the reasons for failed unit tests, while the planner addresses these failures in subsequent iterations.

**Table 6: Overview of unit tests for state DC, FE, and MBVP. These unit tests handle to detect missing values, outliers, duplicates, and other data consistency issues.**
| State | Unit test name | Unit test description |
| --- | --- | --- |
| State DC | test_document_exist | Test if cleaned_train.csv and cleaned_test.csv data exist. |
| test_no_duplicate_cleaned_train | Test if there are any duplicate rows in the cleaned_train.csv. |  |
| test_no_duplicate_cleaned_test | Test if there are any duplicate rows in the cleaned_test.csv. |  |
| test_readable_cleaned_train | Test if the cleaned_train.csv is readable. |  |
| test_readable_cleaned_test | Test if the cleaned_test.csv is readable. |  |
| test_cleaned_train_no_missing_values | Test if the cleaned_train.csv contains missing value. |  |
| test_cleaned_test_no_missing_values | Test if the cleaned_test.csv contains missing value. |  |
|  | test_cleaned_train_no_duplicated_features | Test if the cleaned_train.csv contains duplicate features. |
|  | test_cleaned_test_no_duplicated_features | Test if the cleaned_test.csv contains duplicate features. |
|  | test_cleaned_difference_train_test_columns | Test if the cleaned_train.csv and cleaned_test.csv have the same features except for target variable. |
|  | test_cleaned_train_no_missing_target | Test if the target variable is in cleaned_train.csv. |
| \hdashline |  |  |
| State FE | test_document_exist | Test if processed_train.csv and processed_test.csv data exist. |
| test_processed_train_feature_number | Test if the feature engineering phase is performed well in processed_train.csv. |  |
| test_processed_test_feature_number | Test if the feature engineering phase is performed well in processed_test.csv. |  |
| test_file_size | Test if processed data is larger than a threshold. |  |
|  | test_processed_train_no_duplicated_features | Test if the processed_train.csv contains duplicate features. |
|  | test_processed_test_no_duplicated_features | Test if the processed_test.csv contains duplicate features. |
|  | test_processed_difference_train_test_columns | Test if the processed_train.csv and processed_test.csv have the same features except for target varibale. |
|  | test_processed_train_no_missing_target | Test if the target variable is in processed_train.csv. |
| \hdashline |  |  |
| State MBVP | test_document_exist | Test if a submission file exists. |
| test_no_duplicate_submission | Test if there are any duplicate rows in the submission file. |  |
| test_readable_submission | test if the submission file is readable. |  |
| test_file_num_submission | Test if the submission file and sample_submission.csv have the same number of rows. |  |
| test_column_names_submission | Test if the submission file and sample_submission.csv have the same column names. |  |
| test_submission_validity | 1) Test if the submission file and sample_submission.csv have the same data index. 2) Test if the submission file and sample_submission.csv have the same numerical range. |  |

### C.3 Machine Learning Tools Details

**Table 7: Overview of Tools for state DC, FE, and MBVP. This table presents various tools categorized by their functionality.**
| State | Tool name | Tool description |
| --- | --- | --- |
| State DC | FillMissingValues | Fills missing values or removes columns with missing values based on a threshold. |
| RemoveColumns<br>WithMissingData | Removes columns containing missing values from a DataFrame based on a threshold. |  |
| DetectAndHandleOutliersZscore | Detects and handles outliers in specified columns using the Z-score method. |  |
| DetectAndHandleOutliersIqr | Detects and handles outliers in specified columns using the Interquartile Range (IQR) method. |  |
| RemoveDuplicates | Removes duplicate rows from a DataFrame. |  |
| ConvertDataTypes | Converts the data type of specified columns in a DataFrame. |  |
| FormatDatetime | Formats datetime columns to a specified format. |  |
| \hdashline |  |  |
| State FE | OneHotEncode | Performs one-hot encoding on specified categorical columns. |
| LabelEncode | Performs label encoding on specified categorical columns. |  |
| FrequencyEncode | Performs frequency encoding on specified categorical columns. |  |
| TargetEncode | Performs target encoding on specified categorical columns. |  |
| CorrelationFeatureSelection | Performs feature selection based on correlation analysis. |  |
| VarianceFeatureSelection | Performs feature selection based on variance analysis. |  |
| ScaleFeatures | Scales numerical features in the specified columns of a DataFrame. |  |
| PerformPca | Performs Principal Component Analysis (PCA) on the specified columns of a DataFrame. |  |
| PerformRfe | Performs Recursive Feature Elimination (RFE) on the specified columns of a DataFrame. |  |
| CreatePolynomialFeatures | Creates polynomial features from specified columns of a DataFrame. |  |
| CreateFeatureCombinations | Creates feature combinations from specified columns of a DataFrame. |  |
| \hdashline |  |  |
| State MBVP | TrainAndValidation<br>AndSelectTheBestModel | Trains, evaluates, and selects the best machine learning model based on the training data and labels, returning the best performing model along with the performance scores of each model and their best hyperparameters. |

Examples of Tool Schema.
In this paper, we provide two schema formats for each machine learning tool: JSON and Markdown.
Here, we take the FillMissingValues tool as an example and provide schemas in both formats.

Tool use.
During execution, we extract the machine learning tools specified in the plan generated by Planner and use them as queries to search the entire documentation of machine learning tools.
Since the plan includes multiple tools, we retrieve several tools based on their similarity to the queries.
The Developer then uses the retrieved tools to carry out the task.

### C.4 Tool Utilization

In the multi-agent framework designed for autonomous data science tasks, tools serve not only as automation resources but also as integral components of the workflow. The framework enables agents to dynamically access and execute tools as they transition through various problem-solving states, ensuring adaptability and efficiency.

The tool utilization process in this framework is structured around a systematic approach. Tool information is first stored in the system’s Memory, which is implemented as a vector database. This Memory holds detailed explanations regarding each tool’s functionality, usage, and context. A configuration file is used to map specific tools to the states in which they are required, allowing agents to reference and identify the appropriate tools at each stage of the problem-solving process. To determine which tools are required in each state, the table 7 provides an overview of tools categorized by their functionality. As an agent moves into a particular state, it consults the configuration file to determine the relevant tools. From the figure 1 shown, the agent subsequently queries the Memory to retrieve detailed explanations for the tool’s use, and finally, executes the tool with precision based on the retrieved information.

This dynamic interaction between the Memory, configuration file, and agents facilitates seamless tool integration, empowering agents to operate autonomously while maintaining flexibility and ensuring accurate tool application throughout the autonomous process.

### C.5 User Interaction

At each stage of problem-solving, two Human-in-the-Loop methods are employed.
Before the Planner formulates a plan, human can interact with the command line. The input consists of meticulously manually crafted rules, each one carefully cataloged in a handbook. Memory module subsequently retrieved these predefined rules, integrating this human-driven knowledge in prompt engineering to guide the Planner’s next steps.
After generating the plan, humans can review and and refine the Planner’s output. They inspect areas where the logical flow seems inconsistent, focusing particularly on points where the output diverges from reality to address hallucination issues.

## Appendix D Case Study: Titanic

### D.1 Background Understanding

In this step, the system employs a LLM (GPT-4o) to extract and summarize the key information from the Titanic Kaggle competition. Upon completion of this process, a markdown file is automatically generated containing essential competition details, which include the competition overview, dataset information, and evaluation metrics. Below is an excerpt of the generated markdown file:

### D.2 Preliminary Exploratory Data Analysis

In this state, an autonomous exploratory analysis is conducted to understand the Titanic dataset. The LLM will plan a process: 1) the dataset is loaded and inspected to reveal its structure, including data types, basic statistics, and missing values. A univariate analysis follows, where the distributions of key numerical features like age and fare are explored, as well as the frequencies of categorical features such as passenger class and gender. 2) A bivariate analysis is performed to examine the relationship between each feature and the target variable, survival. This step involves visualizing the survival rates across different categories and examining how numerical features vary between survivors and non-survivors. 3) A correlation analysis investigates relationships between numerical features, visualized through a correlation matrix and additional pairwise comparisons. This analysis provides a deeper understanding of feature interactions, highlighting those that may have the most significant impact on survival. Below is an excerpt of the generated file:

### D.3 Data Cleaning

We demonstrate the data analysis capabilities of our framework using the age column from the Titanic competition’s training set as an example. In the pre-EDA phase, the distribution of the age histogram is as shown in Figure 6. During the data cleaning phase, we filter out missing values using unit tests. You can see a comparison of the age box plots before and after the outliers have been processed in Figure 7. In the deep-EDA phase, the distribution of the age histogram is as shown in Figure 8.

Figure: Figure 6: The histogram of age before outliers are processed
Refer to caption: x19.png

Figure: Figure 7: Comparison of age box plots before and after treatment of outliers.The image on the left is before the outliers are processed and the one on the right is after the process is done
Refer to caption: x20.png

Figure: Figure 8: The histogram of age after outliers are processed
Refer to caption: x22.png

### D.4 In-depth Exploratory Data Analysis

In this state, the AutoKaggle delves further into the Titanic dataset. 1) The process begins with an extended univariate analysis to explore the distribution of both numerical and categorical features. Key statistical summaries are generated for numerical features such as age and fare, while bar charts and frequency tables are created for categorical features like passenger class and gender. 2) A bivariate analysis investigates the relationship between individual features and the survival outcome. Box plots and violin plots are used to analyze how numerical features vary between survivors and non-survivors, while count plots are generated for categorical features to visualize survival rates across different groups. 3) A correlation analysis is conducted to explore the relationships between numerical features, visualized through a correlation matrix and heatmap. This helps to identify any strong correlations between features and the target variable, survival. 4) A multivariate analysis is performed to explore interactions between key features such as passenger class, gender, and age, in relation to survival. Visualizations, such as stacked bar charts and facet grids, are used to highlight these complex interactions, providing deeper insights into the data.Below is an excerpt of the generated file:

### D.5 Feature Engineering

In this phase, the AutoKaggle add several new features to enhance the predictive power of the dataset. 1) A FamilySize feature is created by combining the SibSp and Parch columns, representing the total number of family members aboard, including the passenger. This feature captures the familial context, which could influence survival likelihood. 2) An AgeGroup feature is derived by categorizing passengers into age groups, simplifying the continuous age variable into meaningful categories such as ”Child” and ”Senior.” This transformation helps identify potential age-related survival patterns. 3) Categorical features like Sex, Embarked, and Pclass are then encoded into numerical form to ensure they can be used in the model. One-hot encoding is applied to Sex and Embarked, while label encoding is used for Pclass, respecting its ordinal nature. 4) The cabin data is processed by extracting the first letter of the Cabin feature to create a new Deck variable. This feature provides information about the passenger’s location on the ship, which may correlate with survival outcomes. Missing cabin data is handled by assigning an ’Unknown’ category, ensuring completeness of the feature.

### D.6 Model Building, Validation, and Prediction

In this phase, we conduct a comprehensive analysis of the Titanic passenger dataset with the aim of predicting passengers’ survival probabilities. Initially, the data undergo preprocessing that included filling missing values, deleting columns with high missingness, and handling outliers. Subsequent feature engineering efforts introduce new attributes such as family size, solitary travel, age groupings, and fare per person, and involved encoding for gender and embarkation points. Furthermore, a random forest model is employed, optimized via grid search, and evaluated using cross-validation. Predictions are then made on the test set, and a submission file is prepared.

### D.7 Research Report

After completing the entire data science pipeline, AutoKaggle reviews the interim reports from each phase and finally compiles a Research Report to document its key findings, major decisions, and their rationale at every phase.
This helps users better understand the solutions provided by AutoKaggle.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * Did you do any finetuning over the used models, notably LLMs or are you using frozen models ?
* Why cannot you compare to any existing baselines ?
* Have you optimized the creation of your pipeline using these 5 kaggle competitions, or have you left out some of them, to evaluate on competitions you did not know at design time ?
soundness: 3
strengths: **Interesting problem.** With the LLMs (+RAG) becoming mature, the open source study of their integration into broader tools that can directly be applied to data science tasks, is the natural next step.

**Overall good presentation.** Even if some details are lacking to grasp the authors' exact contribution (notably in the figures), the overall presentation clearly demonstrates the problem and the approach set up to tackle it. 

**Interesting metrics and ablation studies.**
summary: This paper introduces AutoKaggle, a pipeline to automatically solve Kaggle Competitions. The authors use 5 subparts in a row: a reader, a planner, a developer, a reviewer, and a summarizer. They use LLMs with RAG to develop code-based solutions, with code running, units tests. They evaluate their method on 5 Kaggle competition benchmarks.
weaknesses: **Lacking evaluation.** The evaluation is lacking comparison to existing AutoML baselines (*e.g. [1]) or explanations on why the authors are not comparing their method to any existing solution. If running such comparison is not possible at all, then the authors should provide explanations on why this is not feasible. 
While detailed reports are provided on their methods and the different components, as this works apply existing techniques, its evaluation is its core contribution. 
The authors should report (at least) the standard deviation, but e.g. violin plots to compare AutoKaggle's results of other kaggle competitors could help clearly situate where this automatic pipeline stands.

**Evaluation on a (previously) unknown dataset.** It seems that AutoKaggle has been designed to solve these datasets, so one cannot evaluate how much this method would transfer to another, previously unknown dataset.
It would be nice to provide the reader with how much out of the box your method is, maybe with a user study. It seems like its your core contribution, so having independent people trying AutoKaggle and commenting on how easy the setup and interaction is on a left out dataset would help people looking for such solutions.

**Figure 2 could be improved.** The figure could be split to separate the overall pipeline from details on some of its components. Most importantly, what part is using an LLM, what part is using a human expert ? This figure represents 70% of what the reader is looking for, it should provide first the overall intuition, and then enough details on specific core components that you want to highlight.

**You related work section is actually a background section.**
Your current related work covers some domains that are integrated within AutoKaggle. It thus feels more like a related work of your background section (what AutoKaggle builds upon). Is there any *e.g.* AutoML method that you can compare to ? Any method that addresses the same issue ?


[1] https://github.com/automl/CAAFE

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the questions in Weaknesses.
soundness: 2
strengths: - AutoKaggle introduces a tailored phase-based workflow with multi-agent collaboration specifically designed for data science competitions. The system’s demonstrated high average completion rate and competitive ranking in Kaggle highlight its effectiveness, particularly in tabular classification and regression tasks, showing its strength in handling structured data challenges.

- AutoKaggle empowers the Developer agent to perform iterative debugging and unit testing, bolstering the robustness of code generation. Additionally, the integration of a comprehensive machine learning tools library improves the system's efficiency and accuracy, making it better suited for tackling complex Kaggle competitions
summary: This paper presents AutoKaggle, a multi-agent framework specifically designed to handle the complexities of Kaggle data science competitions.  The framework organizes the competition workflow into six distinct phases—background understanding, exploratory data analysis, data cleaning, in-depth exploratory analysis, feature engineering, and model development and validation—allowing agents to work systematically through each stage. Key agents, including Reader, Planner, Developer, Reviewer, and Summarizer, collaborate within this structure, with iterative debugging and unit testing to ensure robustness and accuracy in code generation. AutoKaggle integrates a machine learning tools library to streamline tasks, enhance code reliability, and provide users with educational insights through comprehensive reports at each phase. Evaluated across multiple Kaggle competitions, the framework achieved an average completion rate of 83.8% and ranked in the top 42.8% in Kaggle.
weaknesses: - Limited novelty.  While the paper addresses data science problem-solving using LLM-based agents, it lacks a clear description of the specific challenges it intends to solve that existing methods have struggled with. Extending from a single-agent to a multi-agent system is insufficiently justified in this field, as the necessity and performance gains of such an approach are not clearly demonstrated. Existing works, as mentioned in the introduction, have also tackled similar problems with LLM-based agents, questioning the incremental contribution of AutoKaggle.

- Multi-agent system design. The multi-agent system, including agents like Reader, Planner, Developer, Reviewer, and Summarizer, is insufficiently explained in terms of its collaborative structure. It is unclear whether these agents operate in an assembly-line fashion or if they engage collectively in each phase under the "Cooperative Engagement" label in Figure 1. Further clarification on their integration and interdependence within each workflow phase is needed.

- Role clarity of Planner and Summarizer. Given AutoKaggle’s sequential, phase-based workflow, the necessity of a Planner agent is ambiguous. Can you quantify the contribution (such as on completion rates or error reduction) of this Planner agent in your system? Similarly, the Summarizer’s role in contributing to critical performance metrics such as completion rate or Best Normalized Performance Score, is not explicitly justified, leaving its impact on performance uncertain.

- Unit Test and Debugging. Dose the Developer agent generate dataset-specific unit tests that align with each unique code snippet or not? How the Developer agent adjusts unit tests based on code variations to ensure logical consistency and accuracy across different tasks? 

- Lines 275-276 mention the importance of detecting logical errors in code, yet the method for achieving this is underexplored. Can you explain more details about detecting the logical error? More detail is needed on how logical errors are detected and avoided, as conducting exploratory data analysis or statistical checks after data cleaning or feature engineering alone may be insufficient. 

- Table 2 illustrates the system's performance across different debugging attempts (DT), showing how increased debugging impacts metrics like Completion Rate (CR) and Comprehensive Score (CS). The data indicate that both CR and CS improve as DT rises, reflecting enhanced task completion and accuracy with more debugging opportunities. What the 'performance plateaus' mean  in line 524-525?

- The paper does not provide information on the cost of running AutoKaggle, which is essential for evaluating its performance and practical applicability. It's benifit to provide cost and total runtime to understand the performance. 

- The chosen baselines are not entirely convincing. Recent similar works, AIDE[1] and MLE-Agent[2] have shown remarkable capability in Kaggle competition settings. A comparative analysis with these recent works, particularly focusing on AutoKaggle’s unique advantages in effectiveness, efficiency, or other performance metrics, would highlight its distinct contributions to the field.

- A broader evaluation across various task types such as time series prediction, image classification, and text classification, are necessary, as these are critical and challenging categories in Kaggle competitions. The current experiments focus primarily on tabular datasets, leaving it unclear whether AutoKaggle is capable of handling more complex, domain-specific tasks. Can AutoKaggle complete such tasks? 

- What is the requirement of the LLM? Can AutoKaggle works well with gpt-3.5 or other open-sourced models?


[1] AIDE: the Machine Learning Engineer Agent(https://github.com/WecoAI/aideml)

[2] MLE-Agent: Your intelligent companion for seamless AI engineering and research (https://github.com/MLSysOps/MLE-agent)

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: ~L140, you say that CoT improves reasoning at the expense of introducing hallucinations. Is there any evidence that CoT makes models any more or less likely to hallucinate?

~L141, you say that the ReAct paradigm addresses hallucinations - that's not my understanding of what ReAct does or how it works, my understanding is that it combines thoughts and actions, yes, but that this has nothing to do with hallucinations or refining outputs.

~L360: What is the difference between "Success - Non-compliant" and "Success - Compliant"?

~L403: What's the justification / motivation for the complex / compound "Comprehensive Score"? How does it compare to other measures, what specifically does it achieve or avoid?

~L431: could you say more about this "strong baseline" - I don't understand its construction.

If adding FE tools drops performance because FE adds too much complexity, then why does "All tools" (which presumably includes FE tools) recover this performance?
soundness: 2
strengths: A system which can score 43% on Kaggle leaderboards is a significant milestone on the path to automated coding and datascience. Additionally, since many challenges which such a system would face would also arise in more general task-completion (e.g. long-term planning, establishing coherency, managing context, preventing execution from looping) and so would transfer to improve AI agents in general.

Great collection of Classic and Recent challenges, and baselines seem reasonable (though see my Q about the Strong Baseline).

It's helpful to have this variety of scores (though see my Q about CS).

Architecture is clearly laid out, and the paper is overall very easy to read.

Clear exploration and explanation of the underlyring readon why the feature-engineering tools reduce the framework's score (many features, leading to more complexity than the agents can handle).
summary: This paper presents a scaffolding framework which uses LLMs to create a multi-agent system, used to attempt Kaggle problems. They use a "phase-based" multi-agent approach, together with a library of hand-crafted ML tools, and extensive hand-crafted unit-tests tailored to the Kaggle problems.

Applying this framework to 8 Kaggle problems (4 pre-GPT-4 training cut-off, 4 afterwards), they achieve a significant solve rate, and an average of 42% on the Kaggle leaderboard.

The paper also explores ablation of various modules (various tools, and the unit-testing module).
weaknesses: Whenever CoT is used as an interpretability tool, I think it's always wise to mention unfaithfulness e.g. https://arxiv.org/abs/2305.04388

There are two places where a long list is hard to read:
#1 ~L78: AutoKaggle integrates a comprehensive machine learning tools library, covering three core toolsets: data cleaning, feature engineering, and model building, validation, and prediction.

#2 ~L186: The data science process is divided into six key stages: understanding the
background, preliminary exploratory data analysis, data cleaning, in-depth exploratory data anal-
ysis, feature engineering, and model building, validation, and prediction

Perhaps "model-building, -validation, and -prediction" would be easier to read.

~L146: I'm surprised not to see mentioned what seems to me to be the main thing underlying the motivation of multi-agent systems: finite context length, requiring summarisation and specialisation.

It's not clear how much of the headline 43% on the leaderboard is down to the skill of the human-in-the-loop, which severely undermines the claim. Without a comparison to how well the human takes unassisted (in terms of success rate or time taken), or to how well AutoKaggle performs without HITL, it's impossible to reliably state how effective the framework is.

Unspecified HITL also undermines the various claims of a "fully automated framework" (e.g. L175)

Not much detail on these unit tests. Who writes them? What's the coverage like? Are there any guarantees? If (as I suspect) the "meticulously designed" unit tests are written by humans, then we have a similar situation as with the unspecified human-in-the-loop: the framework is not "fully automated", and it's impossible to rigorously determine how much effect the human hand-holding has on the framework's suggess. This should, at minimum, be clearly, explicitly and boldly acknowledged.

Additionally, it is unclear to me how much of the ML-tools library was developed alongside particular Kaggle Competition attempts. If the tools were developed on a case-by-case basis, to address hurdles found in the challenge, then there is significant data leakage from the evaluation dataset to the framework, leading to overfitting to the competitions chosen during development, and much of the headline 43% comes from tools handcrafted by human developers on a case-by-case basis. For a fair validation of how well this framework performs in "fully automated" mode, the library would need to be "frozen" while the framework was tested on a held-out set of Kaggle Competitions.

Very minor point: ~L350, I agree that there is a risk of data leakage for competitions from before Oct '23, however to say that GPT-4o's training data includes Classic Kaggle is an assumption: better to say simply that there is a risk of data leakage.

If you're considering data leakage, it would be worth flagging that the 42% includes Classic problems: using only the newer problems, performance is slightly below human average.

Meta-review: The paper presents AutoKaggle, an LLM creating a multi-agent system together with a library of hand-crafted ML tools in order to solve Kaggle problems. In my opinion, the reviewers did a very thorough job and have presented salient arguments about the suitability of this paper for IJCAI in its current form. While it is interesting to see that an agentic LLM can help with Kaggle competitions, one reviewer points out that a comparison to existing AutoML approaches is missing. Indeed, the authors added a comparison to AIDE. However, AutoML is not only "an LLM agent that generates solutions for machine learning tasks" (as written on the AIDE github page) but also portfolio and other approaches to automatize (parts of) ML. So, a larger discussion (and comparison) is in place. As it turn out Kaggle is partnering with the International Conference on Automated ML,see https://www.kaggle.com/automl-grand-prix. Another reviewer points out that some design choices and arguments for motivation are missing; currently, it reads more like what the authors have done, but it is not well placed into the research landscape. More importantly, an evaluation across different types of modalities (time series, text classification, image object detection, ...) is missing or argued why this is currently not so important. One reviewer also pointed out that an ablation study is missing, showing that the hand-crafted tools are not doing the job in the end. So, while the direction is super interesting, it is too early for publication, but we would like to encourage the authors to push  for one of the next venues. Please note that the overall judgment should not be taken as a statement regarding the usefulness of your research.

Decision: Reject