AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML
Authors: Patara Trirat
Wonyong Jeong
Sung Ju Hwang
Venue: ICML 2025 poster (2025)
Keywords: Automated Machine Learning
Multi-Agent Framework
Large Language Models
Primary area: applications
TL;DR: This paper introduces AutoML-Agent, a novel multi-agent framework for building automated machine learning pipelines from natural language instructions, with informed planning and verification strategies.

Abstract: Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline, such as optimal model search and hyperparameter tuning. Existing AutoML systems often require technical expertise to set up complex tools, which is in general time-consuming and requires a large amount of human effort. Therefore, recent works have started exploiting large language models (LLM) to lessen such burden and increase the usability of AutoML frameworks via a natural language interface, allowing non-expert users to build their data-driven solutions. These methods, however, are usually designed only for a particular process in the AI development pipeline and do not efficiently use the inherent capacity of the LLMs. This paper proposes *AutoML-Agent*, a novel multi-agent framework tailored for full-pipeline AutoML, i.e., from data retrieval to model deployment. *AutoML-Agent* takes user's task descriptions, facilitates collaboration between specialized LLM agents, and delivers deployment-ready models. Unlike existing work, instead of devising a single plan, we introduce a retrieval-augmented planning strategy to enhance exploration to search for more optimal plans. We also decompose each plan into sub-tasks (e.g., data preprocessing and neural network design) each of which is solved by a specialized agent we build via prompting executing in parallel, making the search process more efficient. Moreover, we propose a multi-stage verification to verify executed results and guide the code generation LLM in implementing successful solutions. Extensive experiments on seven downstream tasks using fourteen datasets show that *AutoML-Agent* achieves a higher success rate in automating the full AutoML process, yielding systems with good performance throughout the diverse domains.

AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML
Patara Trirat" Wonyong Jeong'
Sung Ju Hwang 12
Abstract
1. Introduction
Automated machine learning (AutoML) acceler- ates AI development by automating tasks in the development pipeline, such as optimal model search and hyperparameter tuning. Existing Au- toML systems often require technical expertise to set up complex tools, which is in general time- consuming and requires a large amount of human effort. Therefore, recent works have started ex- ploiting large language models (LLM) lessen such burden and increase the usability of AutoML frameworks via a natural language interface, al- lowing non-expert users to build their data-driven solutions. These methods, however, are usually designed only for a particular process in the Al development pipeline and do not efficiently use the inherent capacity of the LLMs. This paper pro- poses AntoML-Agemt, novel multi-agent frame- work tailored for full lull-pipeline AutoML, i.c., from data retrieval to model deployment. AutoML- Agent takes user's task descriptions, facilitates collaboration between specialized LLM agents, and delivers deployment-ready models. Unlike ex- isting work, instead of devising single plan, we introduce eetrieval-augmented planning strategy to enhance exploration search for more optimal plans. We also decompose each plan into sub- tasks (e.g., data preprocessing and neural network design) each of which solved by a specialized agent we build via prompting executing in parallel, making the search process more efficient. More- over, we propose multi-stage verification to ver- ify executed results and guide the code generation LLM in implementing successful solutions. Ex- tensive experiments on seven downstream tasks using fourteen datasets show that AutoML-Agent achieves a higher success rate in automating the full AutoML process, yielding systems with good performance throughout the diverse domains.
Automated machine learning (AutoML) has significantly reduced the need for technical expertise and human labors in developing effective data-driven solutions by automating each process in the AI development pipeline (Yao et al., 2018; 8; Ren et al., 2020; He et al., 2021), such as feature engineering, model selection, and hyperparameter optimiza- tion (HPO). However, current AutoML systems (Gijsbers et al., 2024) often necessitate programming expertise to configure complex tools and resources, potentially creating barriers for larger pool of users with limited skills and knowledge, such domain experts (Sun et al. (2023); $A).
To make AutoMI. frameworks more accessible to non-expert users, many recent studies (Trirat et al., 2021; Viswanathan et al., 2023; Li et al., 2023; Hollmann et al., 2023b; Liu et al., 2025; Zhang et al., 2023; Shen et al., 2023; Zhang
et al., 2024a; Hong et al., 2024a; Guo et al., 2024a; Yang et al., 2025; Chi al., 2024) have suggested use natural language interfaces with large language models (LLM) for machine learning (ML) and data science (DS) tasks. Nev- ertheless, these previous LLM-based AutoML frameworks only considered a limited number of tasks due to their re- stricted designs, either only for a process in the pipeline (c.g.. feature engineering (Hollmann et al., 2023b; Li et al., 2024; Malberg al., 2024), HPO (Liu et al., 2024a; 2025; Zhang et al., 2024a), and model selection (Zhang et al., 2023; Shen et al., 2023)) or for specific group of downstream tasks (e.g., natural language processing (Viswanathan et al., 2023) and computer vision (Yang et al., 2025)). In addition, most methods overlook the inherent capability of LLMs to search for promising models by performing actual training of the candidate models during the search process, making it pro- hibitively costly and slow.
For an AutoML framework to be truly practical, it should perform end-to-end AutoML, considering both the data aspects (retrieval, preprocessing, and feature engineering) and model aspects (selection, HPO, and deployment). This is because a process in one aspect can affect subsequent processes in the other, potentially leading to suboptimal so- lutions when combining results from different frameworks. Meanwhile, the AutoMI. framework should be computation- ally efficient, using strategies to minimize the computational overhead during search. However, there are two main chal- lenges in building such a framework.
DeepAAuto.ai ai KKAIST, Scoul, South Korca. Correspondence to: Sung Ju Hwang <sjhwang @ deepauto.ai>.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AutoMI-AAeen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
High Complexity of Planning Tasks The planning of the entire AutoML pipeline introduces extra complexities com- pared to task- problem-specific planning, primarily due to the inter-dependencies among the steps in the pipeline. For example, types of retrieved datasets affects how to design preprocessing steps and neural networks. Then, the designed network affects which particular hyper-parameters need to be optimized depending on the given downstream task. Such inter-step dependencies result in the enlarged search space since it should consider all possible combinations of inter- related steps. Besides, enabling the framework to operate across various downstream tasks exacerbates these chal- lenges, as each has task-specific requirements.
Input: User's prompt with taak descrption, requireennss and constraints
need aokuion deleching spar cresrggs with the Siet Spam Colleciie detarel offaient possloyy stramsher messagen applcation The clocoilaaiio securacy
lightsight
User
above 95% with a ioiency bolow 100 miNosscnas por reesaage soolle devices.
AutoML-Agent
Agent
Manager
Prompt Anaalysss
Data Handling
Model Development
Deplyymant
Output: Deployable model optimized for the given task and target platform
Accurncy 58
Optimied Model
Inference Latency
Figure 1. AutoML-Agemt receives user's instructions and delivers optimized deployable models.
Challenges in Accurate Implementations To develop modular and extendable framework that effectively handles diverse ML tasks, it is crucial to enhance the flexibility of the LLM agent in its code generation ability, such as by decoupling the template code from the code for specific datasets. However, using LLMs to autonomously gener- ate complete ML pipelines may lead to hallucination issues, including code incompletion, incorrect or missing dependen- cies, and potential undiscovered bugs (Hong et al., 2024b). Furthermore, L.MS often struggle with code generation when prompted with ambiguous task descriptions. Thus, we need accurate analysis of the requirements, and a code- generation platform that can adaptively generate code based on disambiguated requirements.
requirements from the user's description and multi-stage verification that provides feedback between each step in the framework ensure the quality of instructions when guiding the LLM for code generation. These modules aim to improve the correctness and clarity of the task description for code implementation. Our contributions are as follows.
We propose a novel multi-agent LLM framework for Au- toML. designed to automate the entire AI development pipeline. To the best of our knowledge, this is the first at- tempt to employ LLMs in a task-agnostic AutoML frame- work that spans from data retrieval to model deployment. We address the challenges due to the complexity of the planning problem in full-pipeline AutoML by introduc- ing retrieval-aus -uugeette planning with role-specific plan decomposition and prompting-batt
To address these challenges, we propose novel multi-agent framework, AutoML-Agemt, for full-pipeline AutoML from data and model search to evaluation, with strategies to tackle the complexity of the planning problem as well as accu- rate implementation of code. As illustrated in Figure 1, AutoML-Agent accepts a user's task description and coordi- nates multiple specialized agents to collaboratively identify an optimal ML pipeline, ultimately delivering a deployment- ready model and its inference endpoint as the output.
plan execution, en- hancing the flexibility and efficiency of the search process. To enhance the accuracy of our full-pipeline implemen- tation, we integrate structure-base prompt parsing and multi-stage verification to ensure the quality of resulting solutions and instructions prior to actual code implemen- tation, thereby improving overall performance.
Specifically, to tackle the complex planning problem, we introduce a new retrieval -argeeee-- planning strategy equipped with role-specific decomposition and prompting- based execution. This strategy produces multiple plans based on retrieved knowledge for given task description, facilitating the exploration of promising plans. Moreover, it enables LLM agents to discern global (pipeline-level) and local (process-level) relationships among steps through plan decomposition, which helps them focus on their immediate sub-tasks while aligning with the user's goal. The retrieval- augmented component also simplifies extending LLMs to various downstream tasks using relevant knowledge. The prompting- execution enhances search efficiency by exploiting LLMs' in-context learning capabilities without any further training, which could introduce additional cost. To enhance the accuracy of the implementation. we adopt structure-based prompt parsing that extracts MI-rrelevant
We demonstrate the superiority of the proposed AutoML- Agent framework through extensive experiments on seven downstream tasks using fourteen datasets.
We have made the source code available at https: // github. com/ ceoo//deepautoooo coaiito-ii//a ai/automl-aaate aggent..
2. Related Work
AutoML is a transformative approach for optimizing ML workflows, enabling both practitioners and researchers to efficiently design models and preprocessing pipelines with minimal manual intervention (Ren et al., 2020; He et al., 2021; Gijsbers et al., 2024). Despite several advancements in AutoML (Jin et al., 2019; Feurer et al., 2022; Tang et al., 2024), most of them are designed only for particular ele- ments of the ML pipeline. Only a few works (Bisong, 2019; Mukunthu et al., 2019; Microsoft, 2021) support multiple
2

---

AutoMI--Agent A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 1. Comparison between Aggnt and existing LLM-basaa frameworks.
Key Functionality
Framework
Planning
Verification
Full Pipeline Task-Agnostic
Training- Free Search
With Retrieval
AutoML-GPT (Zhang et al., 2023)
Promp(2Model (Viswanathan et al., 2023)
HuggingGPT (Shen et al., 2023)
CAAFE (Hollmann al., 2023b)
MoCopilo (Zhang et al., 2024a)
x
AgentHPO (L.iu et al., 2025)
X
X
X
Data Interpreter (Hong et ul., 2024a)
X
X
X
DS-Agent (Guo et al., 2024a)
X
X
SELA (Chi et al., 2024)
X
X
Agent (Grosnit et al., 2024)
AutoMMLab (Yang et al., 2025)
Auto.ML-Agent (Ours)
V
steps of the pipeline. Also, due to the traditional program- ming interfaces, these systems often have complex config- uration procedures and steep learning curves that require substantial coding expertise and an understanding of the underlying ML concepts, limiting their accessibility to non- experts and being time-consuming even for experienced users. These limitations hinder the widespread adoption of traditional AutoML systems.
3. AutoML-Agent
This section presents details of the proposed multi-agent framework, AutoML-Agent, including agent specifications, prompt parsing module, a aetrieval-augmentn strategy. a prompting-basee based plan execution, and a multi- stage verification. As depicted in Figure 2, all agents coordinated by an Agent Manager to complete the user's instructions by delivering the deployment ready model.
planning
LLMs (e.g., GPT-4 (Achiam et al., 2023)) have recently shown promise in addressing these limitations with the com- plex problem-solving skills across disciplines via human- friendly language interfaces, including AI problems (Xi et al., 2025; Narayanan et al., 2024; Fourney et al., 2024; Tang et al., 2025; Kon et al., 2025). This shift towards natural language-driven interfaces democratizes access and allows users to articulate their needs in more intuitive manner (Tornede et al., 2024). Nevertheless, existing LLM- based frameworks can only assist in specific step of the MI. pipeline, such as feature engineering (Hollmann et al., 2023b), data analysis (Hu et al., 2024b), model search (Hong et al., 2024a; Guo et al., 2024a; Chi et al., 2024), or HPO (Liu et al., 2024a; 2025; Zhang et al., 2024a). For example, Agent K (Grosnit et al., 2024) leverages LLMs to orchestrate ML pipelines; however, it is tailored for Kaggle competition settings and relies on training- based approach with high search overhead. A few attempts (Viswanathan et al., 2023; Yang et al., 2025) support the entire ML pro- duction pipeline, yet only for a specific type of downstream tasks. Besides, these methods either naively use the LLMs or overlook the inherent capabilities, making their search processes time-consuming for the AutoML pipeline that requires sophisticated planning and verification.
3.1. Agent Specifications
We now provide brief descriptions of the agents in our multi- agent AutoML framework.
Agent Manager (Amgr) acts the core interface between users and other specialized LLM agents within the frame- work, orchestrating the search process. It is responsible for interacting with the user, devising a set of global plans for subsequent processes with retrieved knowledge, distributing tasks to corresponding agents, verifying executed results with feedback, and tracking the system progress.
Prompt Agent (Ap) is an LLM specifically instruction- tuned for parsing the user's instructions into a standardized JSON object with predefined keys. The parsed information is then shared across agents in the framework during the planning, searching, and verification phases.
Data Agent (A.,) is an LLM prompted for doing tasks re- lated to data manipulation and analysis. The analysis results from the Data Agent are used to inform the Model Agent about data characteristics during the model search and HPO.
Model Agent (AA,.) is an LLM prompted for doing tasks related to model search, HPO, model profiling, and candi- date ranking. The results produced by the Model Agent are sent back to the Agent Manager for verification before proceeding to the Operation Agent.
In contrast, we overcome these limitations by incorporat- ing a new retrieval-augmented planning strategy, coupled exccution
with plan decomposition and prompting-be techniques, alongside structure-based prompt parsing and stlti-stage verification. Table summarizes the key differ- ences between AutoMI-AAgen and existing frameworks.
Operation Agent (AA)) is an LLM prompted for implement- ing the solution found by the Data and Model Agents that passes the Agent Manager's verification. The Operation
3

---

AutoMI-AAgen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
(1) Innttallizzatinn
Input: use'' prompt with task description, requirements, and/or constraints
(c) Multi-Stage Verification
Request Verification
fail
fail
Implementation Verification
pass
Agent Manager
Deployable Model
User
poss
foil
(2) Planning
Prompt
Execution Verification
Parsing
External Resources (APls)
Prompt Agent
(a) Retrieval-Augmentee Planning
Solution Summary
Training-Free Free Model Search and HPO
Pseudo Data Analysis
Plan Implementation
(3) Execution
Data & Model Hub
Data Agent
Retrieval
Model Agent
Retrieval / Design
Operation
Code Generation
Preprocessing
HPO
Agent
Runtime Debugging
Analysis
Profiling
Ranking
(b) Prompting-Base Baased Plan Execution (parallelizable)
Full- hull-Pipeline Skeleton Code
Figure Overview of our AgenM framework. (1) Initialization stage aims to receive a valid user instruction using request verification. (2) Planning stage focuses on extracting ML related information by parsing the user instruction into standardized form, uses it to devise plans accordingly. (3) Execution stage executes each action given by the devised plans. Finally, based on the best execution results, AutoML-Agent outputs codes containing deployable model to the user.
Agent responsible for writing effective code for actual runtime execution and recording the execution results for final verification before returning the model to the user.
In the following subsections, we provide the descriptions of each step more in detail.
3.3. Instruction Data Generation and Prompt Parsing
After we define all agents with their corresponding profiles as described above (see $C.1 for the complete prompts), the Amar then assigns relevant tasks to each agent according to the user's task descriptions. Note that we can implement AA and Am with more than one agent per task based on the degree of parallelism.
Data Generation For Ap to generate accurate JSON ob- jects, we need to instruction-uun cettcttioonuteeeeee the LLM first because can output a valid JSON object but with incorrect keys that are irrelevant to subsequent processes. Following Xu et al. (2024), we first manually create set of high-quality seed instructions then automatically generate larger instruction dataset D {T, R having N instruction-respone pairs. Here, 4 is the i-th instruction with the correspond- ing response RG We use the JSON format substantially extended from Yang et al. (2025) for response R& with following top-level keys to extract the user's requirement from various aspects of AutoML pipeline.
3.2. Framework Overview
We present an overview of our AutoMI-Agent framework in Figure 2 and Algorithm 1. In the (1) initialization stage, the Agent Manager (Amgr) receives the user instruction and checks its validity through request verification (Figure 2(c) and Line 3). In the (2) planning stage, the Prompt Agent (Ap) parses the verified user instruction into standard- ized JSON object. Then, Amgr generates plans to solve the given AutoML task using retrieval-augmented planning (Fig- ure 2(a) and Line 11). In the (3) execution stage, the Data
User. The user key represents the user intention (e.g.. build, consult, or unclear) of the given instruction and their technical expertise in AI.
Problem. The problem key indicates the characteristics and requirements of the given task, including area (e.g., computer vision), downstream task (e.g., image classifi- cation), application business domain, and other con- straints like expected accuracy and inference latency. Dataset. The dataset key captures the data characteristics and properties, including data modality, requested prepro- cessing and augmentation techniques, and data source. Model. The model key captures the expected model char- acteristies and properties, including model name (e.g.. ViT), family (e.g.. Transformer), and type (e.g., deep neu- ral networks).
(A.)) and Model (Am.) Agents decompose these plans and execute them via plan decomposition (PD) and prompting- based plan execution (Figure 2(b) and Line 13-16), whose results are then verified against the user's 's requirements via execution verification (Figure 2(c) and Line 20). Finally, Amgr selects the best plan and sends it to the Operation Agent (Aa) to write code (Line 22). After code generation, implementation verification (Figure 2(c) and Line 24) is con- ducted to ensure that the code deployment-ready the verification steps fail, AutoML-Agent performs revision steps (orange lines in Figure 2) generate new solutions.
any of

---

AutoMI-Ageen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Algorithm 1 Overall Procedure of AutoML-Agent
( ("user" (Tmeere buiks. "esperis" "medkum"' ppobleen" 1 "aoes "computee velont
los
model me casilty gender and age harna ladal images as the insu, the modil.
'Biid mase maanformrs
mmage
Initialization: Agent Manager Amgr. instruction-tuned Prompt Agent Ap, Data Agent Ad, Model Agent Ams Operation Agent Ao. deployment-ready model M
Prompt Paring
Sagget writtied" I'gende"aaee "aaell "doerr prise" "hhc images doceet casrty gender and age" "ooum""
"dacaset" Ef "hamm" "modsity" Prmages
"model" "rame"" "Vienn Trandormer "temiy" "yandformer "tpe"" "oeece netordee or li
and system state 5
Instruction
JSON Response
Input: User instruction I
Figure 3. An example of prompt parsing process of an instruction- response pair {(T, R.)}-
1: while S * END and M = Ø do
2:
if S = INIT then
3:
- Amgr (ReqVer())) P run ReqVer ($3.6)
3.4. Retrieval-Augmenter Planning
4:
FF- Ø then > check if is valid
Recent studies (Guo et al., 2024b; Huang et al., 2024; Mas- terman et al., 2024; Zhang et al., 2024b; Hu et al., 2024a) highlights that effective planning and tool utilization are essential for solving complex problems with LLMs, espe- cially in multi-agent framework. By bridging two tech- niques in single module, we propose a retrieval-augmented planning (RAP) strategy to effectively devise robust and up-to-date set of diverse plans for the AuotMI. problems.
5:
R AT( parse user instruction ($3.3)
6:
S < PLAN
7:
else
8:
return F return feedback F to the user.
9:
end if
10: else if S = PLAN then
11:
+ Amgr (R)) run RAP ($3.4)
12:
for p; in P do > run PD for Aa and Am ($3.5)
13:
of + PD(R, Ad, P.)
Let P {PI...., pP} be a set of plans. Based on past knowledge embedded in the LLM, knowledge retrieved via external APIs (such as arXiv papers), and R, RAP gener- ates multiple end-to-end plans for the entire AutoML pipeline having different scenario pi. This strategy enables AuteML-Agent to be aware of newer and better solutions. Specifically, we first use the parsed requirements R to ac- quire a summary of the relevant knowledge and insights via API calls, including web search and paper summary. Amgr then uses this information to devise different plans, i.c., P P= Amgr (RAP(R)) Note that Angr devises each plan independently make the subsequent steps parallelizable. The benefit of this strategy is that it enhances exploration for better solutions while allowing parallelization. Examples of generated plans are provided in $D.2.
14:
O + Aa(s)) ) run pseudo data analysis ($3.5)
15:
8 <- PD(R, Am. Pi: )
16:
0 + ( b> run search and HPO ($3.5)
17:
end for
18:
O + (10%, OT),),")))) P combine outcomes ($3.5)
> run execution verification ($3.6)
20:
if Amgr (er( is pass then
21:
Is f Amar (O) >> get code instructions
22:
M + AA" b write code for the best plan > run implementation verification ($3.6)
23:
24:
if Amgr (emp
is pass then S END > stop the process
25:
26: end if
27: end if
28: end if
29:
end while
3.5. Prompting-Base Based Plan Execution
30:
return M
Given the generated P. we now describe how Aa and Am execute each P; using prompting techniques without actu- ally executing the code. Examples are presented in $D.4.
Knowledge. The knowledge key extracts additional knowledge insights helpful for solving the given prob- lem directly provided by the user, potentially associated with the expertise level.
Plan Decomposition Due to the high complexity of the end-to-end plan, we first need to adaptively decompose the original plan p; E P into a smaller set of sub-tasks s; relevant to the agent's roles and expertise to increase the effectiveness of LLMs in solving and executing the given plan (Khot et al., 2023). The plan decomposition (PD) pro- cess involves querying the agents about their understand- ing of the given plan specific to their roles. Formally, sf PD( (R, P:). where si is the decomposed plan for Data Agent, containing sub-tasks for the given plan p.. Then, the agent executes the decomposed plan towards the user's requirements instead of the original lengthy plan. We define the sub-tasks si" of Am below due to its reliance on Data Agent's 's outcomes. Examples are presented in $D.3.
Service. The service key is relevant to the downstream implementation and deployment. It provides information such as a target device and an inference engine.
Prompt Parsing Then, we can use the generated dataset D to train an LLM and use it as Ap. These standardized keys are important for a better control over the LLM agents' behavior within our framework and necessary for effective communication between agents. Moreover, these keys pro- vide contextual information for generating high-quality AutoML pipeline from various perspectives. After the in- struction tuning, we use the A, to parse the user's instruc- tions (or task descriptions) and return the parsed require- ments R (T) to Amgr, as shown in Figure and $.
5

---

AutoMI--Agent A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Pseudo Data Analysis In AutoML-Agent Ag handles sub- tasks in s including the retrieval, preprocessing, augmen- tation, and analysis of the specified dataset. During the data retrieval phase, if the dataset is not directly supplied by the user, we initiate an API call to search for potential datasets in repositories, such as HuggingFace and Kaggle, using the dataset name or description. Upon locating a dataset, we augment the prompt with metadata from the dataset's 's source; if no dataset is found, we rely the inherent knowledge of the LLM.' We then prompt As to proceed by acting as if it actually executes sy. according to the dataset characteristics and user requirements R. The summarized outcomes of these sub-tasks, O2,, are then forwarded to the Am- Hence, 0 = A.(s)).
Request Verification (Req Initially, we assess the clarity of the user's instructions to determine if they are rel- evant and adequate for executing ML tasks and addressing the user's objectives. If the instructions prove insufficient for progressing to the planning stage, Amgr will request ad- ditional information, facilitating multi-turn communication.
This request verification step, however, often overlooked in existing studies, placing an undue burden on users to for- mulate more detailed initial prompt challenging task particularly for those who are non-experts or lack experi- ence. Prompts for Req Ver are shown in $C...11
Execution Verification (ExecVer) After executing the plans in $3.5, Amgr then verifies whether any of the pipelines produced by Ad and Am (i.c., O) satisfy the user's s requirements via prompting (see $C.5.2). If the results are satisfied, the suggested solution is selected as candidate for implementation. This step effectively mitigates computa- tional overhead in the search process by allocating resources exclusively to the most promising solution.
Training-Fres Model Search and HPO Like Ad. Am uses API calls to complete all sub-tasks s." instead of di- rect code execution. However, in contrast to Ad, the plan decomposition for Am incorporates outcomes from the Aá, enabling it to recognize characteristics of the preprocessed dataset, i.e., s PD(R, Am, Pt, O.).. Here, the Am 's prompt is enhanced with insights gathered by Amgr about high-performing models and relevant hyperparameters for the specific ML task. This technique allows the Am to execute the sub-tasks in s more efficiently. Using this augmented prompt, the A. follows similar procedure to Ad, undertaking model retrieval, running HPO, and summa- rizing the results of these sub-tasks, which include expected numerical performance metrics such as accuracy and error, as well as model complexity factors like model size and inference time. To facilitate the subsequent verification step, we also prompt the agent to return results with the top-k most promising models. Formally, O = As"))).
Implementation Verification (Imp Ver) This implemen- tation verification phase closely resembles the Exe: however, it differs in that it involves validating outcomes de- rived from the code that has been executed and compiled by Ag. We present the prompt for this verification in $C.5.3. If the outcomes meet the user's requirements, Angr provides the model and deployment endpoint to the user.
Note that if any exccution or implementation fails to satisfy the user requirements (i.e., does not pass the verification process), these failures are systematically documented. Sub- sequently, the system transitions to the plan revision stage. During this stage, Angr formulates a revised set of plans, incorporating insights derived from the outcomes of the unsuccessful plans.
Plan Implementation To enhance the efficacy of Ao in code generation, Amgr first verifies all executed results O == -[O0 )] from As and Am- Amgr then selects the best outcome O E O and generates the instruction 1s for A, to write the actual code accordingly. Formally, M = A.("",, where M is the deployment-ready model.
4. Experiments
We validate the effectiveness of our full-pipeline AutoML framework by comparing AutoML-Agent with handerafted models, state-of-the-art AutoML variants, and LLM-bbseed frameworks across multiple downstream tasks involving di different data modalities.
3.6. Multi-Stage Verification
Verification, especially with refinement or feedback, is es- sential for maintaining the correct trajectory of LLMs (Baek
4.1. Setup
Tasks and Datasets As summarized in Table 2, we select seven downstream tasks from five different data modali- ties, including image, text, tabular, graph, and time series. These datasets are chosen from different sources. Also, we incorporate various evaluation metrics for these tasks, e.g., accuracy for classification and root mean squared log error (RMSLE) for regression.
et al., 2025; Madaan et al., 2023; Gou et al., 2024). Our framework incorporates three verification steps to guarantee its accuracy and effectiveness: request verification, execu- tion verification, and implementation verification.
When we mention relying on the LLM's inherent knowledge, we are referring to a fallback mechanism intended to produce the most plausible output given limited context. However, in the absence of actual data, the pipeline will ultimately fail during final implementation verification due to runtime errors.
For each task, we prepare two sets of natural language task descriptions to represent munstraint-awar and constraint-
6

---

AutoMI--Agent A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 2. Summary of downstream tasks and datascts.
erate about 2.3K instsoctionnresponne pairs using EvolIn-
Data Modality
Downstrem Tisk
Datasel Name
Evahaatio Metric
struct (Xu et al., 2024). Here, use LoRA (Hu al., 2021) to fine-tune the model with the generated dataset. For ($3.4), we set the number of plans P = 3 and the number of candidate models A: = 3. All experiments
Image
(Computer Vision)
Image Chasification
Shopee-IEi IFT
Butterfly Image
Accuracy
Text
Ecommerce Test
(NLP)
Classiflearion
Temaal Emailment
Accuracy
Tabular Classification
Bazanz Quality
Software Defects
F1
are conducted on an Ubuntu 22.04 LTS server equipped with eight NVIDIA A100 GPUs (CUDA 12.4) and (Rtel(R) Xe)n(R) Platinum 8275CL CPU @ 3.00GHz. For running the generated models, we employ the same execution envi- ronment DS-Agent (Guo et al., 2024a), with all necessary libraries included in the skeleton scripts.
Tabular (Classic MI.)
Crab Age
Tahular Regression
RMSLE
Crop Price
Smoker Status
Tahular Clustering
Student Performance
(Time Series Analysis)
Time Series
Time Series Forecasting
Weather
Flectricity
RMSLE
Graph
(Grggh Mining)
Node Classificttion
Citeseer
Cora
Accuracy
4.2. Main Results
free requirements (see (B) along with a full- -iplliipe skele- ton script. As a result, we extensively evaluate 28 generated models. Note that this setting differs from previous stud- ies (Guo al., 2024a; Huang al., 2023), which require dataset-specific, partially completed code preparation.
We report the average scores from five independent runs for all evaluation metrics in Figure 4.
Success Rate Figure 4(left) and Table 6 present the results for the SR metric. For the constraint- free cases, which can be considered easier tasks, all methods have higher SR than ones in the constraint-awarc setting. Notably, AutoML- Agent also consistently outperforms the baselines in the constraint-awart setting, achieving an average SR of 87.1%, which underscores the effectiveness of our framework. We conjecture that the knowledge retrieved during the planning process helps the agents identify which areas to focus on in order to meet the given constraints. Regarding DS-Agent, although we use the provided example cases for the relevant tasks, it appears to fail on certain tasks due to its heavy reliance on curated case banks and the inclusion of partially completed code, which is unavailable in setting.
Evaluation Metries For a comprehensive evaluation, we measure the agent's effectiveness in both code generation and task-specific performance aspects by using comprehen- sive score (CS) (Hong et al., 2024a) to simultancously eval- uate both the success rate (SR) of code generation and the normalized performance score (NPS) of the built pipelines. That is, CS 0.5 x SR + 0.5 x NPS. Here, NPS 1 is a transformation of loss-based performance score 8, e.g., RMSLE. More detailed explanations are included in $B.4.
As described above, we evaluate all frameworks under two different settings. To measure SR of each method, we use a for perfect
grading scale ranging from 0 for total failure conformity to the user's requirements. For the constraint- free setting, a method can get a score of 0.5 (pass modeling) or 1.0 (pass deployment). For the constraint-awe aware setting, a method can get a score of 0.25 (pass modeling), 0.5 (pass deployment), 0.75 (partially pass the constraints), or 1.0 (pass all cases).
Downstream Performance We present the performance comparison between the successfully built models in Figure 4(center) and Table 7. To ensure meaningful results and to examine how the performance of LLM-generated mod- els compares to state-of-the-art AutoMI. techniques and manual ML pipelines crafted by experienced experts, we select top-performing models by evaluating results reported in Papers with Code benchmarks and Kaggle notebooks for the same tasks and datasets, where applicable, as the Human Models baselines. From the results, we can observe that Auto:M--Aeet significantly outperforms other agents, including Human Models, in the NPS metric. In particu- lar, AutoML-Agent achieves the best performance across all tasks under the constraint-aware setting. These find- ings highlight the superiority of AutoML-Agent in adapting to various scenarios, attributed to the attrieall-augmente planning (RAP) strategy. This approach enables agents to discover effective pipelines for given constraints. These em- pirical observations substantiate the efficacy of the proposed RAP, providing up-to-date solutions for various tasks.
Baselines As we propose a framework for the novel task of full-pipeline AutoML with LLMss there is no direct base- line available for comparison. We thus compare AutoML- Agent against the task-specific manually designed models (see BB.3): Human Models, the variants of state-of- sate-offtheearl the-art AutoML: AutoGluon (Erickson et al., 2020; Shchur et al., 2023; Tang et al., 2024), a seate-the-of-ar LLM for data science: DS-Agent (Guo et al., 2024a), and general-purpose LLMs: GPT-3.5 (Brown et al., 2020) and GPT-4 (Achiam et al., 2023) with zero-shot prompting.
Implementation Details Except for the Ap that is implemented with Mixtral-8x7B Miixtrall&&77--Isttrre 1).1) (Jiang et al., 2024), we use GPT-4 (004-40-0024-05- 13) as the backbone model for all agents and -Lsed basclines to ensure an impartial performance evaluation. To instruction tune the Ap ($3.3), we automatically gen-
Comprehensive Score the weighted quality of each agent based on the CS met- ric. Overall, AutoMI-AAeet outperforms all other baselines,
Figure 4(right) and Table 8 present
7

---

AutoMI-AAeet: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Human Models
AutoGluon
GPT-3.5
GPT-4
DS-Agent
AutoML-Agent (Ours)
SR
NPS
CS
Butterfly
Butterfly
Butterfly
Ave
Shopee
Avg.
Shopee
Avg.
Shopee
Citesee
Ecomm
Citeseer
Ecomm
Citeseer
Ecomm
(a) Constraint-
0
Cora
0.4
Entail
Cora
0.4
Entail
Cora
Entail
0.2
0.2
0
Electricity
Banana
Electricity
Banana
Electricity
Banana
Weather
Software
Weather
Software
Weather
Software
Student Smoke
Crab
Student
Crab
Student
Crab
Crop
Smoker
Crop
Smoker
Crop
Butterfly
Butterfly
Butterfly
Avg.
Shopee
Avg.
Shopee
Avg.
Shopee
(b) Constraint-Awart
0:8
0.8
4
Citeseer
Ecomm
Citeseer
Ecomm
Citeseer
Ecomm
0,6
0.6
0,6
Cora
0.4
Entail
Cora
0.4
Entail
Cora
0.4
Entail
0.2
0.2
Electricity
Banana
Electricity
Banana Electricity
Banana
Weather
Software
Weather
Software
Weather
Software
Student
Crab
Student
Crab
Student
Crab
Smoker
Crop
Smoker
Crop
Smoker
Crop
Figure 4. Performance comparison across all datasets using the SR, NPS, and CS metrics under (a) constraint-free and (b) constraint-aware settings. Higher scores indicate better results.
especially in more complicated tasks. Interestingly, it is evident that general-purpose purpose LLMs still works relatively well on classical tasks like tabular classification and regres- sion, while more sophisticated methods, such as DS-Agent and our AntoML-Agent work significantly better com- plex tasks. This finding aligns with previous research (Guo al., 2024a), which suggests that tabular tasks typically involve straightforward function calls from the sklearn li- brary (Pedregosa et al., 201 I), and therefore do not demand advanced reasoning or coding abilities from LLM agents, unlike more complex tasks.
Bni RAP
Dicarpatioo
8.4
(a) Variations of AutoML-Agent.
(b) Varying P.
Figure 5, Results of (a) ablation study and (b) hyperparameter study in the CS metric.
verification, which provides feedback to agents, thereby enhancing both their planning and coding capabilities. Inte- grating all components significantly empowers LLM agents to effectively incorporate external knowledge from various sources build nall-pipeline AutoML system.
4.3. Additional Analysis
Ablation Study To validate the effectiveness of each com- ponent in AutoML-Agent, conduct the following ablation studies. The results are presented in Figure 5a and Table 9. First, we investigate vetrieval-augmentee planning (RAP) alone, where retrieved knowledge from external APIs is di- rectly used without plan decomposition and verification. As expected, this ablation leads a decline in performance, and in some cases, even fails generate runnable model. This outcome highlights the importance of the decomposition and verification modules. Second, we evaluate RAP with plan decomposition, where the plan is decomposed for each specific agent. While this variant demonstrates better down- stream performance, it still fails to produce runnable models in certain downstream tasks due to the lack of code verifica- tion. Finally, we assess the full framework with multi-stage
Hy Hyperparameter Study To further verify the effective- ness of devising multiple plans in our retrieval- augmented planning strategy ($3.4), we conduct a hyperparameter study by varying the number of plans P in the constraint-free set- ting. As shown in Figure 5b and Table 10, the number of plans does not significantly affect the success rate, likely due GPT-4's robust planning capabilities. However, based the NPS and CS metrics, observe that the number of plans has a notable impact on downstream task performance. Also, these results also suggest that adding more plans does not necessarily lead to better results, as the model may gen- crate multiple similar plans, resulting in similar outcomes. Consequently, we select as the default number of plans.
8

---

AuteMI-AAent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
AAggett
Pre-Injection
Post-tnjjecton
SSELA
AuthoMl---gget SSLA (11me)) AnteMM--Agett (1mm)
3500
3000
0.8
0.5
0.5
2500
0.6
2000
Sconnds
0.4
0.4
0.4
1500
1000
0.2
500
1ida:
praticdia
Sucters
quality
price
40300
preduction
Sustam
Quality
prioo
produtke
Exiters
prire
(a) System Prompt Variations.
(b) AutoML-Agent with Noisy Information.
(c) AutoML-Agent vs. SELA.
Figure 6. Results of (a) sensitivity analysis, (b) robustness analysis, and (c) comparison with SELA (Chi al., 2024) in the CS metric.
Prompt Sensitivity To understand how different levels of system prompt affect the framework performance, we test five prompt variations for each agent, differing in tone
Thew
Movey
and task specificity (see $C.2). Overall, the experimental results in Figure 6a show that agents are not highly sensitive to exact phrasing as long as their roles were clearly defined, which is also reinforced through the user prompts- rompissmmkking the framework robust to variations in system prompts.
16:.0
343
Figure 7. Average time and monetary cost breakdown.
Noise Robustness Since the framework can encounter noisy information from external knowledge sources or APIs during the RAP process, we further evaluate the robustness of Autent to such noise under two simulated scenar- ios. First, in the pre-summary injection scenario, unrelated or low-quality examples are injected before insight extrac- tion and planning. Second, in the post-summary injection scenario, noisy examples are injected after insight extrac- tion but before planning. That is, they are mixed with useful insights. Here, the noisy inputs are generated by an addi- tional adversarial agent prompted to produce "unhelpful" and "fake" insights intended to disrupt the planning process.
ficiency, our method achieves comparable or superior perfor- mance, with an average score of 0.612 compared to SELA's 0.599. While SELA slightly outperforms AutoML-Agen on few datasets, the gains are marginal and come a substan- tial computational cost. This trade-off between efficiency and minor performance gains further justifies our decision to avoid computationally expensive training-based search. Resource Cost As we primarily use closed-source LLMS in this paper, we analyze the costs in terms of time and money. Figure 7 presents the average time and monetary costs across different tasks and datasets for a single run, un- der the constraint-free (upper) and constraint-awarc (lower) settings. On average, it takes around 525 seconds and costs 0.30 USD (using GPT-4o) to search for a single model that will be deployable after training. The significant amount of time spent in the planning stage also suggests the difficulty in devising plans for full- pill-pipe AutoMI.
The results in Figure 6b demonstrate that AutoML-Agent is robust to such noise. Its built-in error correction and multi-stage verification mechanisms significantly mitigate the impact of noisy inputs, ensuring that final model perfor- mance remains largely unaffected. Interestingly, we observe that in some cases, noise injection can even improve perfor- mance. We conjecture that this may be because the Agent Manager is implicitly prompted to more effectively distin- guish between useful and irrelevant information.
5. Conclusions
This paper presents AutoML-Agent, novel LLM-based multi-agent framework designed for AutoML, covering the entire pipeline from data retrieval to model deployment. AutoML-Agent tackles the full-pipeline planning complexity and implementation accuracy challenges the LLMs for task agnostic AutoML by leveraging the newly proposed aitrivval-augmente planning strategy and multi-stage verifi- cation. In addition, we enhance the plan execution efficiency by integrating role-specific decomposition and prompting- based execution techniques into the framework. Our experi- ments on seven ML tasks demonstrate that AutoML-Agen outperforms existing methods in terms of success rate and downstream task performance.
Comparison with Training-Base the effectiveness of our RAP strategy ($3.4) in acquiring plan
Search To examine
useful knowledge for planning and prompting-basee execution ($3.5) in enhancing search speed, we compare our AutoML-Agent with the recent Monte Carlo tree search- based SELA method (10 rollouts) using six datasets. These datasets represent the easiest and hardest tasks, as identi- fied by SELA's results. Figure 6c and Table 12 show that AutoML-Agent achieves search times about 8x faster than SELA, highlighting significant computational efficiency in line with our focus on practical applicability, Despite this ef-

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Acknowledgements
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurlPS, pp. 1877-1901, 2020. 7
We thank the anonymous reviewers for their insightful comments and suggestions, which helped improve the quality of this paper. This work was supported by Insti- tute of Information & communications Technology Plan- ning & Evaluation (ITTP) grant funded by the Korea gov- (MSIT) 2No.2019-0-00077 Artificial Intelligence Graduate School (oogam((AISTT) IITP grant funded by MSIT WEE-RSS2222-12222
Chi, Y., Lin, Y., Hong, S., Pan, D., Fei, Y., Mei, G., Liu, B., Pang, T., Kwok, J., Zhang, C., et al. SELA: Tree-search enhanced LLM agents for automated machine learning. arXiv preprint arXiv:2410.172566 2024. 1, 3, 9, 20, 45
Meta-learning Applicable to Real-world Problems), the National Research Founda- tion of Korea(NRF) grant funded by MSIT (No. RS-2023- 00256259).
Davenport, T. and Bean, R. Five key trends in Al and data science for 2024. MIT Sloan Management Review, 2024.
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. AutoGluon Tabular:: Robust and accurate AutoML for structured data, arXiv preprint arXiv:2003.06500. 2020. 7, 20
Impact Statement
We expect AuteMI-Agent to offer significant advantages by promoting Al-driven innovation and enabling individuals with limited AI expertise to effectively utilize AI capabil- ities. However, we acknowledge the potential misuse of AutoML-Agent by malicious users, such generating of- fensive content, malicious software, or invasive surveillance tools when exposed to harmful inputs. This vulnerability is not unique to our approach but represents a common challenge faced by existing LLMs with substantial creative and reasoning capabilities, which can occasionally produce undesirable outputs. Although we strictly instruct the LLM to focus on generating positive results for machine learning tasks, there is possibility of unforeseen glitches that could introduce security issues within the system. Therefore, we recommend running AutoML-Agent within a Docker con- tainer to ensure isolation from the host's file system. Ad- ditionally, due to its integration with external services for generation and API-based LLMs like GPT-4, privacy concerns may arise. Users should carefully review any data included in API prompts to prevent unin- tended data disclosures.
Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F. Auto-Sklearn 2.0: Hands-free AutoML via meta- learning. Journal of Machine Learning Research, 23(261): 2022.
Fey, M. and Lenssen, J. E. Fast graph representation learn- ing with PyTorch Geometric. In ICLR Workshop on Rep- resentation Learning on Graphs and Manifolds, 2019.
Fourney, A., Bansal, G., Mozannar, H., Tan, C., Salinas, E., Niedtner, F., Proebsting, G., Bassman, G., Gerrits, J., Alber, J., et al. Magentic-One: A generalist multi- agent system for solving complex tasks. arXiv preprint arXiv:2411 1.04468, 2024. 3
Gijsbers, P., Bueno, M. L., Coors, S., LeDell, E., Poirier, S., Thomas, J., Bischl, B., and Vanschoren, J. AMLB: an AutoML benchmark. Journal of Machine Learning Research, 25(101):1165 65, 2024. 1,2
References
Gou, Z., Shao, Z., Gong, Y., yelong shen, Yang, Y., Duan, N., and Chen, W. CRITIC: Large language models can self-correct with tool-interactive critiquing. In ICLR, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023. 3.7
Grosnit, A., Maraval, A., Doran, J., Paolo, G., Thomas, A., Beevi, R. S. H. N., Gonzalez, J., Khandelwal, K., Ia- cobacci, I., Benechehab, A., al. Large language models orchestrating structured reasoning achieve Kaggle grand- master level. arXiv preprint arXiv:2411.03562 2024. 3, 45
Back, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. ResearchAgent: Iterative research idea generation over scientific literature with large language models. In Pro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Laguugee Technologies, pp. 6709- 6738, 2025. 6
Guo, S., Deng, C., Wen, Y., Chen, H., Chang, Y., and Wang, J. DS-Agent: Automated data science by empowering large language models with case-based reasoning. In ICMI., 2024a. 1, 7, 8, 13, 18, 19, 20
Bisong, E. Building Machine Learning and Deep Learning Models on Google Cloud Platform. Springer, 2019. 2
10

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Guo, T., Chen, X., Wang. Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01688. 2024b. 5
Jin, H., Song, Q. and Hu, X. Auto-Keras: An efficient neural architecture search system. In KDD, pp. 1946- 1956, 2019. 2
Karmaker, S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., and Veeramachaneni, K. AutoML to date and beyond: Challenges and opportunities. ACM Computing Surveys, 54(8):1-36, 2021. 13
He, X., Zhao, K., and Chu, X. AutoML: A survey of the state-of-the-art, Showledge-Baree Systems, 212:106622, 2021. 1,2
Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. TabPFN: A Transformer that solves small tabular classification problems in second. In ICLR, 2023a. 20
Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. Decomposed Prompting: A modular approach for solving complex tasks. In ICLR, 2023. 5
Hollmann, N., Müller, S., and Hutter, F. Large language models for automated data science: Introducing CAAFE for context-aware automated feature engineering. NeurlPS, volume 36, pp. 44753-44775 2023b. 1.3
Kon, P. J., Liu, J., Ding, Q.. Qiu, Y., Yang. Z., Huang, Y., Srinivasa, J., Lee, M., Chowdhury, M., and Chen, Curie: Toward rigorous and automated sci- entific experimentation with Al agents. arXiv preprint arXiv:2502. 16069, 2025. 3
Hong, S., Lin, Y., Liu, B., Wu, B., Li, D., Chen, J., Zhang, J., Wang, J. Zhang, L.. Zhuge, M., et al. Data Inter- preter: An LLM agent for data science. arXiv preprint arXiv:2402. 18679, 2024a. 1, 3, 7, 13
Lála, J.. O' O'Donoghue, O., Shtedritski, A., Cox, S., Ro- driques, S. G., and White, A. D. PaperQA: Retrieval- augmented generative agent for scientific research. arXiv preprint arXiv:2312.07555 2023. 14
Hong. S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang. Z., Yau, K. S., Lin, Z., Zhou, L.,, Ran, C., Xiao, L., Wu, C., and Schmidhuber, J. MetaGPT: Meta programming for multi-agent collaborative frame- work. In ICLR, 2024b. 2
Li, D., Tan, Z., and Liu, H. Exploring large language models for feature selection: A data-centrie perspective. arXiv preprint arXiv:2408. /0025, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Li, H., Jiang, H., Zhang, T., Yu, Z., Yin, A., Cheng, H., Fu, S., Zhang, Y., and He, W. TrainerAgent: Customizable and efficient model training through -.o-eee multi- agent system. arXiv preprint arXiv:2311.0660 2023,
Hu, M., Mu, Y., Yu, X. C., Ding, M., Wu, S., Shao, W., Chen, Q.. Wang, B., Qiao, Y., and Luo, P. Tree-Planner: Efficient close-loop task planning with large language models. In ICLR, 2024a. 5
Liu, S., Gao, C., and Li, Y. AgentHPO: Large language model agent for hyper-parameter optimization. In The Second Conference on Parsimony and Learning (Proceed- ings Track), 2025. 1.3
Hu, X., Zhao, Z., Wei, S., Chai, Z., Ma. Q., Wang, G., Wang, X., Su, J., Xu, J.. Zhu, M., et InfiAgent- BeBenhh Evaluating agents data analysis tasks. In ICML., pp. 19544 19572, 2024b.
Liu, T., Astorga, N., Seedat, N., and van der Schaar, M. Large language models enhance bayesian optimization. In ICLR, 2024a. 1,3
Huang, Q., Vora, J., Liang, P., and Leskovec, J. Benchmark- ing large language models as Al research agents. arXiv preprint -xXv:2310.03302 2023. 7
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. i'ransformer: Inverted Transformers are effective for time series forecasting. In ICLR, 2024b. 20
Huang, X., Liu, W. Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y., Tang, R., and Chen, E. Understanding the planning of LLM agents: A survey, arXiv preprint 2mXiv:2402.027166 2024. 5
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe,, S., Alon, U., Dziri, N., Prabhumoye, S.,, Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-Refine: Iterative refinement with self-feedbact In NeurlPS, 2023. 6
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. I., Hanna, E. B., Bressand, F., et al. Mixtral experts. arXiv preprint arXiv:2401. 04088, 2024.7
Malberg. S., Mosca, E., and Groh, G. FELIX: Automatic and interpretable feature engineering using LLMs. In ECML/PKDD, pp. 230-246, 2024.

---

AutoMI-Ageen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Masterman, T., Besen, S., Sawtell, M., and Chao, A. The landscape of emerging Al agent architectures for reason- ing, planning, and tool calling: A survey, arXiv preprint arXiv:2404.1158, 2024. 5
Tornede, A., Deng, D., Bimer, T., Giovanelli, J., Mohan, A., Ruhkopf, T., Segel, S., Theodorakopoulos, D., Tornede, T., Wachsmuth, H., and Lindauer, M. AutoML in the age of large language models: Current challenges, future op- portunities and risks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. 3, 13
Microsoft. Neural Network Intelligence, 1 2021. URL https: / htt///ttthiaat com/mio////o/o/ caam/mmcrossan mwwcmoiiottinniiiimmmmmoiiiiiiii 2
Trirat, P., Shin, Y., Kim, S., and Kim, M. Generating machine learning model with few sentences. Korea Software Congress, pp. 688-690, 2021.
Mukumthu, D., Shah, P., and Tok, W. Practical Automated Machine Learning on Azure: Using Azure Machine Learn- ing to Quickly Build AI Solutions. O'Reilly Media, 2019.
ISBN 9781492055 1920055555 2
Viswanathan, V., Zhao, C., Bertsch, A., Wu, T., and Neu- big, G. Prompt2model: Generating deployable models from natural language instructions. In EMNLP: System Demonstrations, pp. 413-421, 2023. 1.3
Narayanan, S., Braza, J. D., Griffiths, R.-R., Ponnapati, M., Bou, A., Laurent, J., Kabeli, O., Wellawatte, G., Cox, S., Rodriques, S. G., et al. Aviary: training language agents on challenging scientific tasks. arXiv preprint arXiv:2412.21 1544, 2024. 3
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025. 3
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas. J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M., and Édouard Duch- esnay. Scikit-Learn: Machine learning in python. Journal of Machine Learning Research, 12(85))2825-288) 2011.
Xu, C., Sun, Q.. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q. and Jiang, D. WizardL MM: Empower- ing large pre-trained language models to follow complex instructions. In ICLR, 2024. 4, 7, 13
Ren, P., Xiao, Y., Chang, X., Huang. P.-Y., Li, Z., Chen, X., and Wang, X. A comprehensive survey of neural architecture search: Challenges and solutions. arXiv preprint arXiv:2006.02900 2020. 1,2
Yang. C., Wu, Q. Wang, J., and Yan, J. Graph neural networks are inherently good generalizers: Insights by bridging GNNs and MLPs. In ICLR, 2023. 20
Yang. Z., Zeng. W., Jin. S., Qian, C., Luo, P., and Liu, W. AutoMMLab: Automatically generating deployable models from language instructions for computer vision tasks. In AAAI, pp. 22056-22064, 2025. 1.
Shehur, O.,. Turkmen, A. C., Erickson, N., Shen, H., Shirkov, A., Hu, T., and Wang, B. AutoGluon- -imeesrress.eesisssss... Au- toML for probabilistic time series forecasting. In Inter- national Conference Automated Machine Learning. PMLR, 2023. 70
Yao, Q., Wang, M., Chen, Y., Dai, W., Li, Y-F,, Tu, W.- W., Yang, Q., and Yu, Y. Taking human out of learning applications: A survey automated machine learning. arXiv preprint arXiv: 0t1.///...333666. 3306,, 2018.
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. In NeurlPS, volume 36, pp. 38154-388 2023. 3,
Zhang, L.,, Zhang, Y., Ren, K., Li, D., and Yang, Y. MLCopi- lot: Unleashing the power of large language models in solving machine learning tasks. In The 18th Conference of the European Chapter of the Association for Computa- tional Linguistics, 2024a. 1,
Sun, Y., Song, Q., Gui, X., Ma, F., and Wang, T. AutoML in the wild: Obstacles, workarounds, and expectations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2023. 1, 13
Zhang, S., Gong, C., Wu, L., Liu, X., and Zhou, M. AutoML-GPT: Automatic machine learning with GPT. arXiv preprint 2rxxiv2305.02499 2023. 1,3
Tang, J., Fan, T., and Huang. C. AutoAgent: A autoated and zero-code framework for LLM agents, 2025. URL 2502.05957.3
abb//
Zhang, W., Shen, Y., Wu, L., Peng, Q.. Wang. J., Zhuang, Y., and Lu, W. Self-Contrast: Better reflection through inconsistent solving perspectives. In ACLC pp. 3602-3622, 2024b. 5
Tang, Z., Fang, H., Zhou, S., Yang. T., Zhong, Z., Hu, T., Kirchhoff, K., and Karypis, G. AutoGiuon-Multimoda (AutoMM): Supercharging multimodal AutoML with foundation models. arXiv preprint arXiv:2404.16233, 2024. 2, 7, 20
12

---

AutoMI-AAenn: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Appendix
The appendix of this paper is organized as follows.
Appendix A discusses our motivation, challenges associated with smaller backbone models, differences from general LLM-agent frameworks, and limitations and promising future work.
Appendix B describes details of experimental setup.
Appendix C provides the complete prompts used by our AutoMI-Agent Appendix D and Appendix E showcase intermediate and final results.
Appendix F reports the detailed results with standard deviations.
A. Additional Discussions
This section discusses the motivation behind AutoML-Agent in relation to non-expert users, the challenges associated with smaller models, comparisons with general LLM-Agent frameworks, and the limitations of our approach.
A.1. Motivation and Necessity
In the of generative AI, many companies adopting Al technologies. However, these companies often lack Al experts, leading to software engineers-whn are non-experts in A/ attemppiin to implement such solutions and facing significant challenges. Researchers from various domains outside of AI (e.g., economics, chemistry, and healthcare) are increasingly seeking to apply AI models in their work but struggle due to their lack of specialized Al expertise. Besides, the existence of non-expert users in the ML landscape is well-documented in relevant studies (Karmaker et al., 2021; Sun et al., 2023; Tornede et al., 2024; Davenport & Bean, 2024). For example, an early AutoMI. survey paper (Karmaker et al., 2021) highlights the growing demand for ML tools among stakeholders across various domains. emphasizes that AutoML tools aim make MI accessible non-experts, improve efficiency, and accelerate research, addressing the pressing need for user-friendly AI solutions. Similarly, as thoroughly illustrated in Section 3 of recent study (Tornede et al., 2024), the use of LLMs as interfaces for AutoML and as components of AutoML systems offers significant opportunities for non-ML experts seeking to apply off-the-shelf data-driven solutions to their problems.
To further illustrate the necessity and practicality of AutoML-Agent, consider an academic researcher aiming to evaluate several ML models for novel dataset within constrained timeline. Traditionally, this involves significant manual effort in setting up pipelines, selecting models, and debugging code-steps that be both error-prone and time-consuming. Similarly, in industry settings, MI. engineers often need to rapidly prototype models for specifie business requirements, such as creating lightweight and efficient spam detection system for mobile applications. AutoML-Agent enables such users to focus on high-level problem formulation by generating deployment-ready models that adhere to specified constraints, like latency or accuracy, directly from natural language task descriptions. This capability reduces setup time and errors, enabling users to focus on innovation rather than implementation logistics.
A.2. Challenges with Smaller Models
While we acknowledge the importance of validating our framework on smaller-scale models, these models exhibit systemic limitations that hinder their utility for complex tasks requiring code exccution and extensive planning. During the early stage of development, we found that small language models, including LLaMA-2-7B, Qwenl.5-14p gemma-7B-it, WizardLM- 70B (Xu et al., 2024), and even sode-specialized like CodeQwen 1.5.wenn1.-7BB. failed to generate executable code for tasks requiring extensive planning or interdependent processes (e.g., full-pipeline skeleton script presented in $B.1). Commonly, these smaller models exhibit similar issues, such as cutting the given codes, changing comments without code completion, partial completion, and returning code as it is given. As shown in SF below, even vanilla GPT-3.5 and GPT-4o struggle with generating executable code for complex tasks, involving extensive planning or interdependent processes. Similar findings have been corroborated by prior studies (Guo et al., 2024a; Hong et al., 2024a), suggesting that these challenges are systemic to smaller language models and not unique to our framework.
A.3. Key Distinctions from General LLM-Agent Frameworks
While it is true that generic I.LM-agents are versatile and can theoretically execute certain AutoMI. tasks, they lack critical domain-specific capabilities essential for achieving robust and reliable performance across the structured and interdependent
13

---

AutoMI--Agent A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
processes of a full AutoML pipeline. Generie LLM frameworks are typically designed support broad problem domains and rely heavily on user-defined instructions or augmentations, often leading to inefficiencies, suboptimal performance, and increased failure rates in complex tasks like end-to-end AutoML. In contrast, AutoML-Agent is purpose-built with explicit mechanisms, such as retrieval- retrieval-augmentee -aggmented planning, specialized agents for sub-tasks, and multi-stage verification, all tailored to address the unique challenges of AutoMI.. Through these enhancements, we can increase plan execution efficiency and support diverse ML tasks with more accurate pipeline implementation.
Furthermore, as highlighted in $F, even when general-purpose LLM leverages the same backbone model, they consistently underperform compared specialized methods like DS-Agent or AutoML-Agent in both success rates and downstream task metrics. AutoML-Agent not only outperforms these frameworks but also uniquely integrates the entire pipeline into seamless, modular workflow, significantly reducing errors and enhancing usability for diverse downstream tasks. This distinction underscores the importance of designing specialized frameworks like AutoML-Agent, which prioritize task-specific optimizations over generality, ensuring both efficiency and reliability in automating full AutoML pipelines.
A.4. Limitations and Future Work
Even though we offer flexible module capable of accommodating various machine learning tasks and data modalities, the absence of skeleton code for entirely new tasks may increase the risk of code hallucination. Moreover, in the current version, there remains a noticeable gap in code generation quality when using different backbone models, e.g., GPT-4 vs. GPT-3.5. This discrepancy is not unique to our approach but represents a broader challenge faced by existing LL---aased frameworks. Future work focused on developing a more robust framework that can generate reasonable solutions with reduced dependence on specific LLM backbone is promising direction.
Additionally, to further improve the literature retrieval component during the planning process, future iterations could incorporate tools, such as PaperQA (Lála et al., 2023). A PaperQA-style scientific querying system could enhance citation relevance and enable more context-aware prompt generation, ultimately leading to improved planning decisions.
Our work also encounters limitations code generation when applied to machine learning tasks requiring significantly different development pipelines from those evaluated in our experiments, which primarily focused on general supervised and unsupervised settings. Tasks such reinforcement learning and recommendation systems introduce particular chal- lenges. Consequently, extending AutoML-Agent to these domains will necessitate the development of additional agents to handle specialized steps in the target pipeline, such as actor-environment interaction and reward modeling in the case of reinforcement learning.
In particular, extending AutoML-Agent to reinforcement learning would require incorporating domain -seciffc modules for instance, agents for environment interaction, action space design, and reward processing. 2 One could envision an "RL agent""" interfacing with an environment (e.g.. OpenAl Gym), while the planner generates RL-specific procedures such as policy training and evaluation. Although this extension is certainly feasible, it could be substantial enough that RL-specific version of AutoML-Agent, perhaps termed AutoRL-Agent, would represent a significant contribution worthy of separate publication or project.
B. Details of Experimental Setup
This section outlines the detailed experimental setup used in this paper, including the complete instruction prompts for both constraint-free (Table 3) and constraint-aware (Table 4) settings, full-pipeline skeleton script ($B.1), dataset ($B.2) and baseline descriptions ($B.3), well evaluation metrics ($B.4).
SSe https: http://wwwwwuttommmm autom" .org/autor1 -survey.
14

---

AutoMI-AAeen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 3. User instruction (i.e., task description) for experiments under the constraint-free setting.
Task
Dataset
Instruction Prompi
Image
Butterfly Imagee
I need a very accurate model to classify images in the Butterfly Image Classification datasct into their respective categories. The dataset has been uploaded with its label information in the labels.csv file. Pleuse provide clussitication model that categorizes images into one of four clothing calegories. The image path, along with label information, can be found in the files train_lahels.cc ca and test Jabels.cv.
Classification
Shopee-IET -ET
Text
Ecommerce Text
We need state-of-the-ar model for text classilication based the Ecommerce Text dataset. The model should cupable of accurately classifyng text into four categaries: Electronics, Hoosehold, Books, and Clothing & Accessories. We have upliaued the entire dataser without splitting here.
Classficaatin
Textual Entaiilmnt
You are solving this machine learning tusks of classffiction; The datavet presented here (the Textual Entaiment) comprises a series of labeled text pairs. Given two texts (textl and text2), your task is to predict the relationship contradiction (1) or entailment (2). The evaluation metric is securacy. Build language model to get good performance.
text pair neutral (O),
Tabular Classification
Banana Quality
Build model classify banama quality as Good or Bad based their numerical information about hananas of different quality (size, weight, sweetness, softness, harvest time, ripeness, and acidity). We have uploaded the entire datasct for here in the
banana_quality.co esv file.
Software Delects
You solving this data science tasks of binary classification: The dainset presented here (the Sollware Defects Dataset) comprises a lot of numerical features. Please split the datasct into three parts of train, valid and test. Your task is to predict the defects itom. score. Please binary classification model to get a good
which is binary label with O and The evaluation metric performunce on this task.
Tabular Regression
Crab Age
You are solving this data science tasks regression: datasct presented here (the Crab Age Datasct) comprises a of both categurical and numerical features. Pleue split the dataset into three parts of train, valid and test. Your task is to predict the age item. The evaluation metric the RMSLE (moot mean squared log error). Now train regression model to get a good performance on this task.
Crop Price
need a regression model to predict crop prices hused on features like soil composition, envirommental factors, historical yield data, and crop management practices from the dataset I uploaded here.
Tabular Clustering
Smoker Stutus
You are solving this data science tasks of unsupervised clustering: The datasel presented here (the Smoker Status Datasel) comprises lat of numerical features. Please use the features in the lest.csv file. Your task is in create the clustered items, which is binary label with 0 and ttwo clusters). The evaluation metric is the Rand index or Rand score, can be tested against "smoking" labels. Now an unsupervised clustering model to get a good performance on this
Higher Education
want unsupervised clustering model to group student performances into eight groups. The datasct named 'Higher Education Students Performance Evaluation' (id=856) can be dowaloaded via ucimtropo Library. The clustering quality can be check against lurget variable OUTPUT Grade.
Students Performance
Time-Series Forecasting
Weather
I want you to create a model for node classification on the Cora dataset to prodict the category of cach paper. You need to directly find Cora dataset from relevant library. I want you to create model for node classification on the Citeseer datnset to predict the category of each paper. You need to directly find the Citeseer dataset from relevant library.
Electricity
Node Classification
Cora
Build model to perform timo-series forecasting using the Weather datasct uploaded bere, evaluating its accuracy with the RMSLE metrie. Note that the input is a sequence of past observations with fixed size (INPUT_SEQ LEN INPUTDDIM=21,, The model should predict the next future sequence with lixed size (PRED SEQ LEN-9, PRED DIM-211.
Citeseer
You are solving this machine leamnng tasks of time series forecasting: The dainset presented here (the Electricity dataset) comprises
real-world time series data. Please split the dataset
three parts of train, valid and test. The input is a sequence of past observation with fixed size (INPUT SEQ LEN=96, INPUT DIM=321). Your task is to predict the next future sequence with fixed
size (PRED. SEQ LEN= PRED DIM=321), The evaluation metric root mean squared log error (RMSLE). Now train time series forccasting model to get good performance on the given fixed sequeaces.
15

---

AutoMI-AAenn: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 4. User instruction (i.c., task description) for experiments under the constraint-aware setting. Bold texts indicate constraints used for evaluation.
Tack
Dataset
Instruction Promut
Image
Bunerlly Image
I need a higaly accurate machine learaing model developed to classify images within the Buterrly Image Classification dataset into their conrect species entegories. The dataset has uploided with label information this task, leveraging transer kaming from pre-trained ResNet-50 model to improve securacy. Ogtimize the model using cross-validation on the training split to fine-tane teperiramdeer and sim for an accurncy at least 0,95 on the lest split. Provide the final ininned model detailed report of the training pencess, hyperparmmcer settings occumey merrics, and a confusion matis to evaluse performance scross different categories.
Classification
lahels. file. Please convolutional
network (CNN) architecture
Shopee-IET
Please provide a clasilication model tua calegorizes images into found in the files train.labels.vv und test.bbels.csv The model should achiere at least 85% uccuracy on the test set and be implemented using Py Forch. Additonally, please include data augmentation techniques and a confusion maltis the evalution.
four clothing calegories. imuge puth, along
labol information, can
Text Classificttion
Eoommerce Text
We require the development of an advanced neural network model for text classiiccaiion talkored to the Ecommerce Text dataset, with the ohjective of achieving at least 0.95 classification accarscy. The model should be spocificalll trained to distinguish test into four defined categories: Flectronics, Household, Books,
Clothing & Accessories, To facilitate we have uploaded complete dataset in its entirety without any prior division into mmining, validition, or test sets.
Textual Entailment
You are selving this machine learning task of classffiction: The dataset preseated here (the Textual Entailment) comprises a series of labeled text pains. Given two texts, your task is predict the relaiooship of the text neutral (0), contradiction (1), entailment (2). The evaluaiion metric aucuracy. Build language model to get good performance. coouring the size does not exceed million parameters and the inference time millisreonds per prodiction
300
Tabular Classiscatiin
Banana Quality
Build a machine leamnng model, potentially XGBuu or LightGMM, to chasily buman quality as Good or Bad based on their numetical information about bunanas of different quality (size, weight, sweetness, sotiness. harvest time, apeness, and acidity). We have uploxded the entire dataset for you here in the banana quality csv file. The model must achieve least 0,95 accuracy.
Software Defects
You are solving this data science tisk binary classificution: Please splis the dataset into three parts train, valid, and test. Your task is so predict the deferes item. which is a binary label with and 1. The evalnation metric is the Fl score. Please tmin a binary classificaion model exceed minutes
dataset presented
(the Software Defects Dataset) comprises a lot munerrccal fostures.
a good performance on this task, ensuring that the model training timse does not prediction time each Instance under milliseconds.
Tabular Regression
Crab Age
You solving this data scieace task regression: The dataset presented here (the Crub Age Duinseti comprises lot of both categorical and numerical feanures. Please split the datoset into three parts train, valid, and test, Your task to prodict the age item. evaluation metric the RMSLE (cont squiree
mern squzred log errory. Now train a regression modal to get a good performance on this task, ensuring that the model's is training time does not exceed 30 minutes and that it can make prodictions en the test set within 5 scconds need
Crop Price
regression model predict crop prices hased festures like soll composition, environmestal factors, historical yield data, and crop management practices from the datisel uploaded here. You should optimize the model to achieve RMSLE E less than 1.0
Tabular Chastering
Smoker Status
You solving this data science takk of unsupervised clustering: The datasel presented here Smoker Status Dataset) comprises a lot numerical features. Please use the features in test.sv. Your task is to create the clustered items, which is a biary label with O and I ttwa clusters). The evaluntion metric the Kand index Rand score which can be sested against 'smoking' labels. Now train unsupervised clustering model to good performance this task, ensuring that the Rand index
0.75 and model training time does exceed minutes.
Higher Eduscation Stidents Performance
I want an unsupeviseed cinstaring model to group student per refomances into eight groups. The dataset named 'Higher Education Stidents Perfommance Evalustion" (id-856) can he downlouded via ucimtrepo litrary. The clastering quality can he checked against the target variable OUTPUT Grade. Th model should achieve Rand Score of lrast 0.8 and complete clustering within minutes.
Time Series Forecasting
Weather
Build state-of the art time-series fonecasting model for the Weather datteet uploaded here, evalusting its accuracy with the RMSLE metric. Note that input a sequenze past observations tixed size UNPUT SEQ LE- INPUT DDM-211) The model should predict the next future sequesce fixed size (PRED SEQ LEN=96, PRED DIM-21). We target RMSLE Jower than 0.05.
Electiicity
You are solving this machine loarning task of time series forecasting The dotaset preseated here (the Electiliit dataset) comprises worl-wordd time series train, valid, and
Please split dataset into three INPUT DIM-321), 1). Your task is to prodict the next funure sequeses with fixed size (PRED SEQ LFN-99 PRED DIM-321), The evaluation matric root moam squared log crur (RMSL)) Now train a time series forecasting model to gel a good performance on the given fixed sequenoes. Ensure the model achieves RMSLE E of less Than 0.1 and that the training time does not excoed I hour GPU.
input sequence of past observation fixed (INPUT SEQ LEN=96
Node Classification
Coen
I want you to develop mode classification model using the Graph Convelutional Network (GCN) algorithm to predict the eategory of each poper in the Cora dataser. Star by importing Com dataset using the 'llanetod'' dateset from the torch goometric datasets' module in PyRoch Geometria, Ensure you preprocees
include mnde featunes and lahels correctly Triin the model minng a snitablc optimizer and loss functions Thes, evallatt its accurscy on the test scr. The aecuracy on the test set should he over 0.90. I want
Cieeseer
develop mode classification model using the Graph Convolutionl Network (GCN) algurithm to prodict (he calegory of each peper in the hat.gggttcccccccaaacooeoto Ensure you preprocess the daia to include mode features and libels cerrellly. Train the model using a suitable optimizer and loss function. Then, evaluate its accuracy on the test set. The accuracy on the test set should be over 0.50.
Citeseer datteet Start by importing
Clieseer datnect using the 'Planctoid' datnset from
PyTorch Goometric.
B. Skeleton Python Script
Skeleton Python Script (e.g., text. classfsssffccc
The following code is for "text clasaificaticon task using Torch. import os, zandom, Lime, jsoa
define CPU lccation os-envi ron ["CUDA_ DEVTOR [OVTTRR_RRRRR] "FCT_RUS_IT os.environ TOCDAALWISBLL__DEEEEE
"3*
import torch
import torch.an as na
import torch.optim as optim
impoxt numpy as np
import gradio 88 gr
10D0: import other required library here, including libraries for datasets and (pre-trained) models like HuggingFace and Ragyle APIs. If the required module is not found, you can directly install it by zunning
16

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
pip install your_modizle', torchtext Amport datasets, data, vocab from corch.utils.dae data import vataloades, Dataset from akkesrn.metrteen import aecuraay_aaaro, fl_g
SEED - 42
random. (SEED)
torch, march.m _eed (SEED)
ng. ng.random.senn seed (3ERD)
Define devics for model operatiane device
(*ouda" corch.cuca. worch.coci..s_.vallleeee.... is_availeble "cpu")
DALASET Lexperseents/daaaeeeee
path saving loading dateset (s) for user's uploaded for preprocesaing, training, hyperporanter tuning, depioyment, and eva avaluaticn
Deta preprocessing and feature engingering
def preproceas_das _a
this Function is
preprocessing festure engineering
preprccessing
Should zeturn the preprocessed dačá processed_das
train (model, train_icadez,
r000: this function is for model treining loop and optimization on 'train' and 'valid' datasets fine-tuning given pretrained model applicable)
rono: this function
Should return the well trained oz finetuned model. model
evaiuate modeel (medel, test_losdez];
In this cask, use Accuracy and Fi metrics to evaluate the text classification performance. "perfereanee_scon scores should be dictfonary formar having metric names dict tonary keys
TOD0: the first part of this function is for evaluating trained or fine Luned model on the "Lest" dalaset with respect to the relevant downstream task's per metrics Jefine the 'y_true' for ground truth and 'y_pred' for the predicted classes here.
performance_szece
'ACC': accuracy_ (y_true, y_pred),
'FI': (y_score y_pred)
TOD0: the second part this function for moasuring trained mode! compiexity samples with respect to the relevant complexity netrics, such as inference time end model size 'comploxity_ scoress should dictionary having setric names dicticnary keys
Should zetura model's perfozmance scozes return performance_scoct
complexity_eoores
def propare_noae model for her_ddepllymeettt ():
1000: this function is for preparing an evalusted model using modei compression and conversion to deploy the on particular platform
Should zeturn the deployment zeady sodei return deploysble_model
def sapioyy_mdelll [I:
TonO: this fonction deploying an latted model with the Gradia Python library
Should return the url endpoint generated by the Gradio library uri_enspoint
fonction orchestrate model proparation, model doplayment, and model evaluation main():
ioading, data preproceseing, feature engineering, model training,
Kain function to execute the text classification pipeline. u*e
TOD0:
Step Retzieve
available) userfa storage given)
dazaset None
T000: Create trsin trsin-velid test split of the data splitting the 'detsset' into teade_, valid volid_loader, and test_ loader. the trade_ containg 708 the 'dataset', valid loader contains 204 the 'dataset the teat_Loader containa 169 of the 'dataset',
17

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
toade_, valid_ test_ (None, Nane, Noze) # correspending to 706, 208, 108 of 'dataset"
r000: step with the split dstaset, date preprocessing and feature engineering (if applicable) using the "preprocess_ data" function you defined
processed_data preprocess_datar
1020:
required modei. may retrieve model available hub library along with pretrained weights (if aay).
Step
Tr pretrained or predefined mooel is not available, please create the model according to the given userfa requirementa belov using Pytorch and reievant librories.
mode
Nore
1000: Step 5. trein the retrieved/ioaded model using the defined "tzain_aodei" function
rowe: on top madai training, please hyperpazamter optimization based on suggested hyperporamtera and their vo lneu bafore proceeding to the evalustion step to ensure model's optinality
moda:
train_model
r000: evaluate tzained model using defined "evaluate_mode?" Function
modei__ perfornce:,,,,,,e,, complexity " evaluate_model]/
compress and convert the trained model according to given deplayment platform vafing the dafined ltree_ooee__oordpppoeooee depl oysbly pyrable_mmddn prepare_mode for_ddel_o___pplooo_ oymentttttttnt
TOD0:
fonction
deploy the model using defined mdddloy_mode1" url_endpoint = deploy_madel
return processed_data, model, deployable_noden uri_encpoint, poce_pperformanee Lodee_complexiee
processed model, depleyable_ modell, uel arl_endpoint, print ("Model Performance on Test Set:", pondel_performatt print (*Model complexity:", complextm)l
model_ complextyy - main
B.2. Dataset Descriptions
As presented in Table 5, we select seven representative downstream tasks, covering five data modalities. We describe the datasets their statistics as follows.
Butterfly Image (Butterfly). This dataset includes 75 distinct classes of butterflics, featuring over 1,000 labeled images, including validation images. Each image assigned to single butterfly category. The dataset is accessible https: / wwww wwwgle.gg......m com/ dame/t/tssets/////mmmm hwtp:e/ttt....ttthtteaaeeaaaeeee
/ttpt//uttttttteeeeeeeeereeettat www-i-maaeelll classi inassificas fication Shopee-IET (Shopee). This dataset is designed for cloth image classification, where each image represents clothing item, and its corresponding label indicates the clothing category. The available labels include BabyPants, BabyShirt, wom- encasualshoes, and womenchiffontop. The dataset is available at https: /www kaggle..gg..... com/ competitions demo- shopee- iet compet i i on data.
Ecommerce Text (Ecomm). This dataset is a classification based E-commerce text dataset comprising four categories: Electronics, Household, Books, and Clothing Accessories, which together cover approximately 80% any commerce website. includes 50,425 instances and can be found at https: //ww: xaggle. com/ datasets/ saurabhshahane! wttpe//eeeeeeeeeeeeeeeeeeeeeee tewteeeeeeetteeeeeeeeeeeeee ass f cationn
Textual Entailment (Entail). This dataset consists of labeled pairs of text, where the task is to predict the relationship between each pair as either neutral (0), contradiction (1), or entailment (2). It is divided into training set containing 4,907 samples and testing set with 4,908 samples. We use the dataset provided by Guo et al. (2024a).
Banana Quality (Bamana). This tabular dataset consists of numerical information on 8,000 samples of bananas, covering various quality attributes such as size, weight, sweetness, softness, harvest time, ripeness, acidity, and overall quality, The primary objective of the dataset is classify each banana sample as either good or bad. The dataset is available https: / /www. kaggle. com/ datasets 131 1 ff banana/ a
Software Defects (Software). This dataset consists primarily of numerical features and has been divided into three parts: training, validation, and testing. The goal is to predict a binary defect label (0 or 1). The training set contains 82,428 samples, the validation set contains 9,158 samples, and the test set contains 91,587 samples. We use the dataset provided by Guo et al. (2024a).
Crab Age (Crab). This dataset contains mix categorical and numerical features, and has been divided into three parts: training, validation, and test sets. The task to predict the age of the crabs. The training set consists of 59.981
18

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 5. Summary of downstream tasks and henchmark dataset statistics.
Inta Modillty
Downsrram Trok
Datned Name
Features
Train
Yall
Feet
* Classes
Source
Liceose
Exnisation Metrie
Main Datanets
Iauge
Komppeer Vicos)
Image Chacotication
Bunarfly Image
Stepee-lEd IET
224s224
Varying
4,549
0.40
1,200
160
651
10
75
Kagele Dunoct
Kigglt Competiion
CD0 Cystem
Accenary
Oatural Language Procoaing)
Test
Text Claesifiation
Texial Eusilters.
Exoemnenc Text
N/
NA
35,296
3,925
10.04
982
4,008
5,044
Kagel: Dunoctt
Kagele Dunect
CC 40
N/A
Accenary
Tabelar Classifcation
Soltware Defests
Bonane Quality
21
73,268
5,600
18.315
1,500
91,387
500
Kugel: Cumpatiion
Kigale Donset
Apacae 2,0
NIA
F1 RI
Tabula
(Chedir Machine I loining)
Tabslar Chostring
Sucker Stzus
Higher Education Studeam Perfeemmno
22
100,331
101
25,665
79
14,354
14
Kigale Competiiin
Rescech Tatoor (UCIML) ML
(2.RY-O 4.0
N/A
RI
Tobslor Regresdion
Chop Price
Crab Age
53,316
1,540
133329
440
66,646
230
NIA
NIA
Keggle Ditset
Kogal: Cimpattion
CD0
MIT
RMSLE F.
RMSLE
Gooph (Gruph Lesening)
Nod: Clasification
Clinner
Ore
3,703
1,433
2,708
3,327
2,708
3,327
2,718
3,327
Reecech Fattett (Planetund)
Reazech Datect (Planstund)
C0.8Y-4 40
N/A
Acceracy
Time Series
(Time Series Analysio
Time Scries Forecasing
Wexther
Blcutiitt
321
21
36,857
18,412
10,539
5,200
5,270
2
NGA
NGA
Reseach Datzeet (TSLib)
Reseach Danseet (TSLib)
CC CYBY4C 4.0
CC 4.0
RMSLE
Additooall Oateort fiom SPIA
Binury Cheailizatiu
Click Prufiction Small
Snoker Stutes
19174
21500
4794
7500)
Kugale Compettiion
OpenMi
Tahube
(Chasiic Machine Learting!
Malti-Clsa Clascitation
Wize Quality White
Facton.
11
2350
240
58%
990
400)
OpanMI.
OpesMI.
N/A
Regresiion
Colliges
House Prices
90
3389
700
176
1413
292
NGA
NGA
Kegale Competiiion
OpenMI.
RMSB
samples, the validation set includes 6,664 samples, and the test set contains 66,646 samples. We use the dataset provided by Guo et al. (2024a).
Crop Price (Crop). This new dataset contains 2,200 samples with key features such as nitrogen, phosphorus, and potassium ratios in the soil, temperature (in ?C), humidity (in %), soil pH value, and rainfall (in mm), all of which are essential for predicting crop yield values, Crop yield prediction is crucial in modern agriculture, particularly as data-driven methods become more prevalent. This dataset is available at https: //t::://wwwww/ kaggle. com/ dat asetss va rarsii i tanal wwwtr///oopppppp -eep-pprieeeeeeeee- ce- -eelieeeeeeeeee predi ct i on- dat aset.
Smoker Status (Smoker). This dataset contains numerous numerical features. The goal is to categorize smoking status of each instance into a cluster. The training set consists of 143,330 samples and the test includes 143,331 samples. We use the dataset provided by Guo et al. (2024a).
Higher Education Students Performance (Student). The dataset, collected in 2019 from students in the Faculty of Engineering and Faculty of Educational Sciences, was created to predict students' end-of-term performances using machine learning techniques. It is a multivariate dataset with 145 instances and 31 integer-type features, focusing on classification tasks within the domain of social sciences. We adopt this dataset for unsupervised clustering instead of classification. This dataset can be found at https: archive ics. uci edu/ wdw/ hatp//aaaee///////e/ /856//6///// h hie educat selucationtsi stsnrcatttisstuuunnenoee tper worrformmat formanceeeeem..mmommooooeo +enneeeeeeelleeeeeeee luaaiio.oo
Weather. The weather dataset consists of 21 metcorological factors collected every 10 minutes from the Weather Station the Max Planck Biogeochemistry Institute in 2020, containing 52,603 samples without any pre-splitting. It is accessible at https: / /t//ttttit hig..b.t....... com/
com/thuml ttuml/ / "Time-Series Seterress-ibraaa Library.
Electricity. This dataset comprises hourly electricity consumption data for 321 customers collected from 2012 to 2014 totaling 26,211 samples. The dataset records the electricity usage of these clients on an hourly basis and is provided without any pre-split. The dataset available at https: / //gittthuss com/thuml wwweeeieeeeeeeiiiieeee
Cora and Citeseer. The citation network datasets, "Cora" and "CiteSeer," consist of nodes representing documents and edges representing citation links between them. Both datasets provide training, validation, and test splits through binary masks. The Cora dataset contains 2,708 nodes, 10,556 edges, 1,433 features, and 7 classes, while CiteSeer consists of 3,327 nodes, 9,104 edges, 3,703 features, and 6 classes. We use the version provided by Fey & Lenssen (2019).
B.3. Baselines
Human Models We select top- nop-performing models based on evaluations from Papers with Code benchmarks or Kaggle notebooks, where the similar tasks and datasets are applicable. The chosen models for relevant downstream tasks are described below.
Image Classification. The human models for image classification tasks are obtained from a
Kaggle notebook
available at https: //www/wwww/c// http://wwwwwwww..e/pe/pp www.coow////oooooto/.ooooocccooo www.///o///oammeaa/aaaaaaee hww.amddaasaaaaalll.... 93/notebook, utilizing a pretrained
ResNet- 18 model.
19

---

AutoMI-Ageen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Text Classification. For text classification tasks, two models are employed. A Word2Vec-based XGBoost model is applied to the e-commerce text dataset https://wwe wwwgle com/ code sugataghsshthhh commerce -teetecllaeeeeee www.saalassssiiiionnnomm -twpeatiioon---d---------------- worf--dd--ooodeeccffoorreoo while the XLM-RoBERTa model is used for the textual entailment dataset Blperpratteeeeeeeeeeeeeteeeereee Tuning, vbookshel /basiiaicccc of-ccc-oooooo -eeercset-------aaaeeeeee abeer-andd- www.mmmooooee roberta -pytorch.
con/coee// code/
Tabular Classification. Due to the absence a similar model in the repository, we use the state-of-the-art TabPFN model (Hollmann et al., 2023a) designed for tabular classification tasks.
Tabular Regression. For tabular regression tasks, we adopt two models specifically designed for the given datasets, which are available at https:/ /www. www.weagge.... com/ htt.c////ooooe///mmmmmmommmnnaaa
for the crab age dataset and at https:/ /wwww for the crop yield dataset.
crop-yiell wew--yyelllpedeee www.oooonnnnneeeeeeeeeeeeeeeeeee Tabular Clustering, For unsupervised clustering tasks, we use manually hyperparameter-tuned KMeans clustering, following the approach outlined https://wwy http://wwww.gggee... latggge.commae wwwm///od///ameeeelloaaa stt.d///smmlccttinhs///////
sedrrvisddcllsssee www.lsssstrnng...m as the baseline.
Time-Series Forecasting. In this task, we use the state-of-the-arr iTransformer (Liu et al., 2024b), which is designed for
the same task and datasets as the baseline model.
Node Classification. For node classification tasks, also employ a tfate-of-the-ae the graph neural setwork-based model, PMLP (Yang et al., 2023), as the handcrafted baseline for both datasets.
AutoGluon We adopt AutoGluon as the baseline because is a state-of-the-arr AutoML framework capable of handling various downstream tasks and data modalities, with the exception of graph data. There are three variants of AutoGluon: AutoGluon-TT (Shchur et al., 2023) for time series, MooGlluon-Tbblle (Erickson al., 2020) for tabular machine learning, and MonoGGllun-MMltimoeee (Tang et al., 2024) for computer vision and natural language processing tasks.
GPT-3.55 and GPT-4 For GPT-3.5 and GPT-4, we use the gpt-3.5-turbo-0225 and gpt-4-2024-05-13 models via the OpenAl API. We implement the zero-shot baselines using the prompt below.
Zero-Shot Prompt for GPT-3.5 and GPT-4 Baselines
You are a holpful intalligent assistent. New please halp salve the following machina learning task. [Task]
(user instruction)
[[File_name]] (ellenmemepp)) hpyyhho
Stull-pipell skeleton script] s.x
Start the python code with "'yyhhon"" Flease ensure the complereness of the code so that it can be run without additional modt flcacions.
DS-Agent We reproduce the DS-Agent (Guo al., 2024a) baseline using the official source code. However, it is important to note that our framework encompasses the entire process from data retrieval/loading to deployment, whereas DS-Agent focuses solely on the modeling aspect, assuming complete data and evaluation codes provided. In this paper, we utilize the deployment stage of DS-Agent along with its collected case banks and Adapter prompt for the same tasks, as the source code for manual human insights collection during the development stage is unavailable.
As the most recent LLM-based AutoML study for tabular data, additionally compare SELA (Chi al., 2024) using subset of the suggested datasets that vary in difficulty. SELA integrates Monte Carlo Tree Search with LLM agents to enhance the automation of MI. pipelines. The main difference between SELA and AutoML-Agent is that SELA iteratively improves results based on experimental feedback, making it time-consuming, whereas AutoML-Agent bypasses the training process, making our method more efficient while still maintaining comparable downstream performance, as shown $4.3.
SELA
B.4. Evaluation Metries
Success Rate (SR) We employ the success rate (Guo et al., 2024a; Hong et al., 2024a), which evaluates whether the models built by an LLM agent are executable in the given runtime environment. Success rate is used to assess code execution.
For the constraint-free setting, we apply three-level grading scale as follows.
20

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
0.00: Code cannot be executed.
0.50: Code provides runnable ML/DI model.
1.00: Code provides a runnable model and accessible deployment endpoint (e.g., Gradio).
For the constraint-awen -warre setting, we use a five-level grading scale to evaluate whether the code executes successfully and satisfies the given constraints. The grading criteria are as follows.
0.00: Code cannot be executed.
0.25: Code provides a runnable ML/DL. model.
0.50: Code provides a runnable model and an accessible deployment endpoint (e.g., Gradio).
0.75: Code provides a deployed, runnable model that partially meets constraints (e.g., target performance, inference time,
and model size).
1.00: Code provides a deployed, runnable model that fully meets constraints.
Normalized Performance Score (NPS) In this paper, each downstream task is associated with specific evaluation metric, which may vary between tasks. These metrics include accuracy, Fl-score, and RMSLE. For metrics such as accuracy and Fl-score, we present the raw values to facilitate comparison across identical data tasks. For performance metrics where lower values indicate better performance. such as loss-based metrics, we normalize performance values s using the following transformation: NPS a This transformation ensures that metrics like RMSLE are scaled between and 1, with higher NPS values indicating better performance.
Note that achieving downstream task performance (NPS) requires runnable model, i..., SR 0. If the model cannot run, the NPS is zero by default as it cannot make any predictions.
Comprehensive Score (CS) To evaluate both the success rate and the downstream task performance of the generated
AutoML pipelines simultancously, we calculate CS as a weighted sum of SR and NPS, as follows: CS = 0.5 -R-
C. Prompts for AutoML-Agent
C.1. Agent Specifications
This subsection provides the system prompt design for agent specifications in AutoML-Agent, including Agent Manger (C.1.1), Prompt Agent (C.1.2), Data Agent (C.1.3), Model Agent (C.1.4), and Operation Agent (C.1.5).
C.1.1. AGENT MANAGER
System Message for Agent Manager Specification
are experienced senioz project manager
machine leacning jroject
main responsibilities as follows.
Keceive requirements and/or inquiries users through well-structured JSON object. 2. Caing recent knowladge and state-of-the-art studies to devisa promising high-qual ity plans for data scientists, machine learning research engineers, and MiOps engineers in your team to execute subsequent proceases based the user rementennnts have receivad.
C.1.2. PROMPT AGENT
System Message for Prompt Agent Specification
You are an assistant projec: nanager in the AutoML development Lear. the user's veeui vement. into valid J5ON format reference, Your sesponae must exactly follow the given 380N schema and be based only on the user's instruction.
taak
spect fi oatttio schera
Make sure that your answer contains only the uSON response without comment or explanation because it can cause persing errors.
#JSON SPECIFICNTIN SCUEMA4 jjsonn sisos_specifccctiono)
21

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
response begin with "yooo"""" ": and end with .... ")]", respectively.
C.1.3. DATA AGENT
System Message for Data Agent Specification
You are the world's s beat dasa scientist of an automated machine learning project (AutoMM.) that can find the moat preprocessing, perform suitable data augmentation, visulaization to cemprehensively understand the data based on the user requirements. You have the following responsibilities to complete.
datasels,
meaningful
1. Retrieve detazet from the uses search for the datases based the 2.Perform data preprecessing based on the user instruction or best practice based on the given tasks. 3. P.rrform data augmentation neccesary.
instruction.
E.Extract usetul information and uncerlying characteriatics of the dataset.
C.1.4. MODEL AGENT
System Message for Model Agent Specification
automated machine learning project (AutoML) that can find the optinal candidate machine learning models and artificial intelligence algorichra for the given dataset run hypesparemeter tuning to opimize the nedels, and perform metadata extraction and profiling to comprehensively understand the candidate medels algorithms based the requirements. You have the following main respansibilitian to complete.
machine learaing research engineer
well-pertorming candidate models and algorithma the dataset based user's zequizement and instruction.
hyperparamster aptimizetion
those candidate models algorithms.
Extract useful information and underlying characteristice of the candidate models algerithms using metadate extraction and profiling techniques.
4. select the top-k (*k' will given) well-performing models algorithms based the hyperparamater optimi astion and profi ing results.
C.1.5. OPERATION AGENT
System Message for Operation Agent Specification
are world's MLOps enginee:
automated machine learning project (AuLoML) that can implement Lhe optinal solution for production-less deployment, given datasets and models. You have the following main responsibillities te complete.
Pythen codes to retrieve/load the given datase: from the cerresponding 2. Write offective Python codes to preprocess the retrieved dataset. Writa precise Python hyperparameters.
1. Write
vetrieve/load the giver madel and optimize with the suggestad
Rrite efficient Python codes ttain/finetune
retrieved model.
5. write suitable Python codes to prepare the trained model for deployment. this step may include model compzession and conversion according
the Larget inference platform.
6. write Python codes build the web application demo using the Gradic library.
7. Run the madal evaluation using the given Python functions and summariza the results for validation againta the requi
C.2. Variations of the System Prompt
This subsection outlines the variations in system prompt design used in the experiment described in $4.3. These include: generic prompts (C.2.1), generic prompts with instructions (C.2.2), entry-level role descriptions (C.2.3), experience-level role descriptions (C.2.4), and world's-best-lews role descriptions (C.2.5).
22

---

AutoMI-Ageet: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
C.2.1. GENERIC PROMPT
Generie System Message
You are holpful assiatent.
C.2.2. GENERIC PROMPT WITH INSTRUCTIONS
Generic System Message with Instructions
You are heipful assistent. You have main responsibilities as follows. (ppenf-epacific steps)
C.2.3. ENTRY-LEVEL ROLE DESCRIPTIONS
Entry-Level Role System Message
are fjob position)
learning project (AutoML).. responsibilities
folloMs. (pgent-epaci steps)
C.2.4. EXPERIENCE-LEVEL ROLE DESCRIPTIONS
Experience-LLvel Role System Message
are experienced ijob position) automated learning project (AstcHL). responsibi itiea it (agent-apeciiic steps)
as fol lows.
C.2.5. WORLD'S BEST ROLE DESCRIPTIONS
World's Best System Message
are world's bast (job position) of aut.omated learning project (AutioM))) have maiim responaibllities follows. (agent-opecific steps)
C.3. Prompts for Retrieval-Augmeente Planning
This subsection presents prompts for planning-relatee processes (Figure 2(a)), including knowledge retrieval and summary prompts (C.3.1), planning prompt (C.3.2), and plan revision prompt (C.3.3).
C.3.1. KNOWLEDGE RETRIEVAL PROMPT
Prompt for Knowledge Retrieval and Summary for Planning
Kaggle Notebook
searched Kaggle Notebooks find state-of-the-art solut using the keywords: [user_task] [user_domal Hiere is the result:
[context]
Please summarize the given pieces of Python notebooks into single pazagraph of useful knowledge and insights. inslude the codes. instead, the insights from the source codes. aim your summazy to address the following user's zequirements.
23

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
User's Requirements raqur summary___ummrr)
Papers With Code
searched papersws thoade website state-of-the-art models vaing keywords: (user_area] uaer_task). Bere is the result:
[context]
Ploase cummarize the given pieces of search content into single paragraph of useful knowledge and insights. We to use your summary to address the following user a rements. User!' Requirements
arXiv
searched arXiv papers using the keywords: (task_kw) and [domain_kw]] Here 1a the result: ccontext
cummarize gaven pieces asxiv papers into single peragraph knewledge insights. aim to use summary to addreas the fol lowing user! requi rements. User's Requirementa juser_eequizemet__emma
Google WebSearch
searched the web using the query: (searcii_ueer).. Here is the result:
[context]
please summarize the given pieces of search content into single paragraph of useful knowledge and insights. We aim use your summary so addcess the following user's zequirements. User's Requirements (user_ ceagpuiremmtt__ummman
Summary
Please extract and summarize the Iollewing group of contenta collected from different online sources into chuak of insightiul knowledge. Flease format your list of suggestions. will use them address the user's 5 requirements in machine learning tasks.
Source: Google keb Search
Isearch_sunmary
Sourse: asxiv Papers (arxiv Jommayy
Source: Kaggle (kagole_eumacc)
Source: PapersMithhode jpwc_summery:
The user's requisements summarized follows. ieser_requivemett_mmm _ummaryy__uumr
24

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
C.3.2. PLANNING PROMPT
Prompt for Retrieval-Augmenned Planning
NoM, Mant you devise an end-to-and actiorable plan according to the user's roquirementa describad in the fol lowing JSGN object.
json
(user__equiremental
Rere is a 1ist of post exparience cases and krowladge written by an human expert for relavant task: known__nowledge]
When devising a plan, Tollow theae inatructiors and do rot rorgez them:
that your plan Ensure that your plan is based on the requirements and objectives described in the above JSON object. agents are capable
the knowledge,
Ensure that your plan designed
AI agents instead conducting ing machine learning and artificlal intel lignce research.
experts.
Ensuze that your plan aelf contained with sufficient instructiona to be executed by the agenta, modalingi that
Ensure that your plan inoludes all key points and instructions (from handling
the Ar agents can auccessfully implement then, Do NOT directly write the code. your plan completely include model development pipeline in detzil (i.e., from daza retrieval to madel training and evaluation) when applicable based tha given remeitenents.
procese machine learning artificisl intelligence
C.3.3. PLAN REVISION PROMPT
Prompt for Plan Revision
Now, you asked revise rethink Inumžworda [n_plans) different end-to-ead actionable according to the user's requizements described in the J80N object below.
"Joonn
following findings insights summarized previously failed plans. Try you can to avoid the same failure again. [fail_rationale)
Finally, when devieing plan, follow these instructions and not forget them:
Ensure that your plan up-to-date with
stste-ef-the--ee knowledge.
knsure that your plan based on the requitements and objectives described in the above JSUN ebject.
Ensure that your plan designed for AT agents Instead of human experts. These agents capable of conducting machine learning and artificial intelligence research.
Ensure that your plan self-contained plan inoludes all the key points and Instructions (tron handling data to modeling! so that the AI agents сan successfully implement then. Do NOT directly write the code.
sufficient instructions to be exeauted by the agents.
Ensure that
Ensure that your plan completely include the end-to-ezd process of mechine learning artificiel intelligence model devalopment pipaline in detcil (i.e., from data retrieval to madal training and evaluation) when given rements.
applicable based
C.4. Prompts for Prompting-Base Based Plan Execution
This subsection presents prompts for grompting-based plan execution processes (Figure 2(b)), including plan decomposition (Data Agent (C.4.1) and Model Agent (C.4.2)), pseudo data analysis (C.4.3), and training-free model search and HPO (C.4.4).
C.4.1. PLAN DECOMPOSITION: DATA AGENT
Prompt for Plan Decomposition: Data Agent
As proficiant data scientist, surmorize following plan given by senior AutooMt project manager according to the user's requirements and your expertise in data science.
User's Requirements json
25

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
liser_requizeme
Project Plan
[g.an]
summary plan should
you fulfill your respensibilities questions by focusing on the data manipulation and analysis.
following
1. retrieve col lect the dataset (s)?
prepresess the retrieved (s)
3. Row to efficiently augment the dataset (a)?
extract underatand
underlying characteristios
dataset
you cenmot it. sure scientist can exectly reproduce the resulta hased on your summary.
that you should not perform visualizatior
C.4.2. PLAN DECOMPOSITION: MODEL AGENT
Prompt for Plan Decomposition: Model Agent
As proficient machine learning zesearch engineer, summarize the following plan given by the senioz ouL project manager ancord! ng the user! requi rements, your expe"tise "n learning, the comeo Iron. data acientist.
>user'' kequirements
json (eurr_eequiemmenta
AxProject Planxx (projest_plan)
**Fxplanat and Results from the Data SciartistA
tdala_reeull;
The summary of the plan should enable you to fulfill your responeibilities as the answers the Iollowing questions by focusing the modeling and optimization tasks.
1. How to retrieve or find the pigh-perfermance model
optimize
hyparparantera of the retrieved modals?
3. How to extract and underatand the underlying characteris:ics of the dataset (a)
top models algerithms on given plans?
C.4.3. PSEUDO DATA ANALYSIS BY DATA AGENT
Prompt for Pseudo Data Analysis
proficient data scientist, your task explain exdetailedva steps for data manipulation and analysis parts by executing the following mechine learning develcpment plan.
Plan
(decomposed haaamoosed_data_pllade data_plan)
Potentiel Source of Uataset lavailable_tnsreea)
sure your explanstion
these instructions:
All of exp innatioo must be f-conta red without using any plachholda Ider to ensure that other data scientists can exactly reproduce all the steps, but not include any code,
Include how and where zetrieve collect the data.
=nclude how to preprocess the data and which tools or librariss are used for the preprocassing.
"nclude how do
augmentation with datai and _nclude how to extract and understand the characteristics of the datz.
"nclude reasons why each step your explanatiors
essential to effectively complete plan.
Note that should not perforn data visualization because cannot it. Make sure to focua only the data part as it is your expertise. Do not conduct er perforn anything regarding modeling oz training. After complete the explanations, explicitly specify the lexpected) ouscomes and results both quantitative and quslitativa of your explanations.
26

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
C.4.4. TRAINING-FREE MODEL SEARCH AND HPO MODEL AGENT
Prompt for Training- Free Model Search and HPO
As proficient maching learning research engineer, your tack is to explain videtailedri stepa for madeling and optimi nation parts by executing the folinwnng machine learninng ng development pl with the goal finding top -(k) candidate modelo/nlgortthmm.
Suggested
(decompoeed_nodel_pplee
Available Xodel Source
[avallable_souree))
Make aure that your explanation for rinding the top-(k) high-peetormance models or algorithms follows these instructione::
All of your explanations mast be self-contained without using placeholder to easure that other mechine the steps, but inalude any code. "nclude how and where to retrieve rind the top=(k) wel 1-performing mode mndels/alloritee
learning researci engineers can exeetly reproduce
Enclude how to optimize the hyperparanters of the candidace models oz algoritams by clearly specitying which detaf".
hyperrpramten
optimi
Coiresponding to each hyperparanter, explicitly include the accual numerical value that you think It is the optimal value for given dataset and machine learning task.
_nclude how to extract and understand the characteriatics of the candidate models algorithms, such their
corputation complexity, menory usage, and inference latency. interpretability.
part
to visualizotinn
Include reasoas why each step in your explanations is essential to effectively complete the plan. your expersise.
modeling part regarding data manipulation or analysia.
only
conduct perform arything
complete explanations, explicitly specify names (expected) quantitative performance using relevant numerical performance and complexity metrics le.g., number of parameters, FLOPS, modil size, training ng time, inference speed, be the optimal sodel below.
so on) the (num2worda cand' cate mode saallls/agoortte potant ly to
Do not use any placeholder for the quantitative performence. If you do not know the exact values, please use the knowledge and expertise you have astimate those performance and complexity valuss.
C.5. Prompts for Multi-Stage Verification
This subsection presents prompts for multi-stage verification (Figure 2(c)), which ensures the correctness of intermediate results between steps in the framework. These stages include request verification (C.5.1), execution verification (C.5.2), and implementation verification (C.5.3).
C.5.1. REQUEST VERIFICATION
Request Verification (Relevancy)
Is the tollowing atatement relevant to machine learning or actificial intelligence? '[uzez instructionn Answer only Yes! or 'No'
Request Verification (Adequacy)
Given fowlowing
ject repcesenting ing user's requirement potential tell me whether we have essential information (e.g., prcblem and dataset) to be used for a AutoMi project? experts, you must only essential requiraments, e.g., problem
jectect, please
note our users are
and brief dataset descriptions.
You do not need check evezy details of the requirements, You must also answer "year even though. it lacks detailed and specific information.
jjson
(parsed usez requirements)
Please answer with this format: "a 'yes' or "no" answer; your reasons for the answer" by using "f" to separate between the and its
If the answer is '2o', you must :ell me the alternative solutions or examples for completing such missing information.
27

---

AutoMI-AAeet: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
C.5.2. EXECUTION VERIFICATION
Execution Verification
Given proposed solution
requiremente, please carefully verify the proposed
solution 'pass' or "fail' the user's 5 requirements.
**Poposed Solution and implementations nata Manipulation and Analysia: [data_agent__ar come Modeling and iptimiatiion: [model_agent_ourcen
Requirementser
Jsoo
[user_reee requt rementa]
only 'Faas' 'Fail'
C.5.3. IMPLEMENTATION VERIFICATION
Implementation Verification
As the project managez, please cazefully verliy whether the given Python code and zesults satisfy the user's xequirements.
Python Code
** "pythor
(implementation_eeeul ('ded"
*.*
Execution
User's Requirements
juser_
Answer only 'Bass' oz "Fail'
D. Examples of Intermediate Results
This section presents examples intermediate results from the main steps of our framework, including prompt parsing results I), retrieval-augee -aggeeend planning results (D.2), plan decomposition results (D.3), and prompting-ber plan execution results (D.4).
D.1. Prompt Parsing
Task Deseription and Parsed JSON User Requirements
Constraint-Fren User Prompt: Build model to classify banana quality as Good or Bad based on their numerical information about bananas of different quality (size, weight, sweetness, softness, harvest time, ripeness, and acidity). We have uploaded the entire dataset for you here in the banana-quality.csv file.
Parsed Output:
"user": ("intent": "build", "expertise": "medium"),
"problen":
"azea": "Labular data analysis", ream task"; "tabular class
aeplicatiion_omaile "agricultuzef
"description": "Build model clessify banans quality oz information about bananas of different quality (size, weight, sweetness, softness, harvest time, ripeness, acidity).
sesrformanoe_mtrioss.. "complexity_metricee" []
"dataset":
28

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
"name"; qpalnt_.
"modality": ("tabular"],
starget_variable ("quality"), Epeatiiicaioon null, descrigtionn: "A dataset containing numerical information about bananas of different quality, including size, weight, swestness, seftness, time, zipeness, acidity.",
"roppooaaciing" [],
augmentat [],
viauelization":
sourde": *user-upload"
"model":
mane"*
*family;
"cype*: *clasaical machine learning",
"specification": null,
"description": "A model to classify banans quality as Good or Rad based their numerical information."
1,
Constraint-Aware User Prompt: Build machine learning model, potentially XGBoost or LightGBM, to classify banana quality as Good or Bad based on their numerical information about bananas of different quality (size, weight, sweetness, softness, harvest time, ripeness, and acidity). We have uploaded the entire dataset for you here in the banana quality.csv file. The model must achieve at least 0.98 accuracy.
Parsed Output:
"user": ("intent": "build", "expertias"; "modium"), "problem":
"acea": "tabular dats analysis", "dowastream_caak:: "tabular classification",
ingplication_ddmainn "agricultuse",
"descriptioa": "Build a machine learning model, potensielly XGBoost oz LightGDM, to classify banena quality Goad Ead based their numerical information about bananas of different quality (size, weight, sweatnesa, softness, harvast time, ripenass, and acidity). The mode must schieve at least 0.98 accuracy.",
("name": "scsuracy", "value": 0.98!
"complexity_metric""" []
"dataset":
"name": quaaite_
"modality"; ["tabular"].
*target_vard saret__varibbles" ("quality".
"specification" null,
deacriptiion: dataset containing numerics). information about bananas of di different quality, including size, weight, aweetneas, softness, harvest tine, ripeness, and acidity.", "reprocessing"" [],
"agmentaticn" [].
"viaullization [],
*aource"; "user-upload"
"model":
"nmme* *xGsoost",
*fanily: *ensemble models",
"yye"" "ensemble",
"apect ficttiion: null,
"description": potencial model claseity quality Good potentially using or LightGDM."
29

---

utilize their Learned AutoMI-Ageet: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
D.2. Retrieval-Augmented Planning
Example Results from Retrieval-Augmentu Planning with
need a very accurate model to classify images the Butterfly Image Classification dataset into their respective categories. The dataset has been uploaded with its label information in the labels.csv file.
User Input:
Plan #1
## Bad-to- End Actionable Plan for Butterfly Image Claasification
*ibjerttiver:: Uevelep highly accurate neural network model to classify images in the Butterfly image ciassification
into respective categeries.
#### Beta Collection Preprocessing
1. **hataset Toading**:
Load the Butterfly Image Classification dataset and the associated labels from "labels.csv'
Verify the integrity of the dateset, ensuring all images have correspending labels.
Preprozessingaw:
r+image desizingar: sesize all images to uniferm size (e.g., 224x224 pixels) ensure consistency. inMormalization Normalize pixel variance.
the range [0, 1] standardiza to have zero mean
3. +*Data Augmentatione::
Apply data augmentation techniques such as cropping, herizontol and vertical flips, rescoling, shear, zoom, rotation to incresse dataset variability. Utilize libraries like TansorFlow's "imogalatatiineratoo"" or PyTorch's 'erensforma" for implementation.
#++ Model Selection and Architectuse
*+I:ansfer Learning:::
Leverage pre-trained convelutional neural networks (CNNs) such vob16, ves19, Mobilatiet, xception, 5eaNat50, and Inceptiont3.
tine-tune
pre-trained models on Butterfly image classification
features.
AxNodel ArchitectureAv:
Start with inceptionV3, known for high accuracy in similar tasks. Explore other models lika Restict, Densalic, ReficientNet, and vision Transformers. Consider combining Cits with Transformer tectures to capture both local and olobal features.
3. Rnnnemble Methods*=:
Implement ensemble nethoda by combining predictions from multiple models to improve ovezall ciaatiiicatioo performence. Use techniques like bagging or stacking to enhance model robustness.
##4+ 3. Mode: Training
splittingee:
Split the dataset into training, validation, and testing aets (e.g., 80% training and 25% valieatior).
*Tlraining Configuratienc::
Use categorical croas-entropy as the loss function. aMSprop.
Select optimizers
Set up callbacka for early stopping and learning rate reductios en plateau to prevent overfitting.
3. *ellpperaarametr Tunings:;
Perform ayperparemeter tuning using libraries like PyCazet or Optuna to sind the optimal configuration. Utilize techniques like grid search random search for hyperperameter optimization.
Evaluation Validation 1. *Reerfommance Netricsve:
Regularly accuracy lonelionelly, track precision, secall, and Fl score to enaure belanced performance across all classes.
primary performance
*ccoss-validdtiione
Emplay k-fold croas-va lddttio to assess the model's robustneas and genersl ab! lty.
3. **landinng Class Imbelancer+: Address performance scross all categories.
through weighted
ovarsampling techniques ensure
### Interpretability and Bebugging
1. Modeel Interpretab: litye:: interpretability methods like Grad LIMB visualize which pacts Images influeace model's predictions.
Ensure the focuses
biolagical features butterflies.
30

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
##+ 6. sepraducibility and Documentation
(Reproductbil1yy-.
reproducibility results. Rocument the entire procass, including data aplita, model. configurations, and training poramsters.
šet randon seeds for all operatiens
2. Nodel Deploymentwv:
Prepare the model deployment by exporting in suitable (e.g., TensorFlow SavedKodel, (8NX)
Set up an inference engine and deployment endpoint to make pradictions on naw data.
*:Bemonstratione:
Provt de a remonstrat lon of tha modal's capabi littes by cresting user-friendly Interface or viaua ization performance.
ay following comprehensive plan, agents davelop for classi fying butterfly images in the given dataset.
securate
tailored
Plan #2
# End to Bnd Actionable Plan for Butterfly Image Classification
#* 1. Data Collection
DaDatazet Namexv: Butterfly Image Classification
Sourcee+:; User-uploaded
-eescrrpptonn=:: A datasa: for classifying images of butterflies into their respective categories, "sabels.cav' file.
v*label Informationwe:
#1 2. Data Preprocessing
v>Image Preprocessinga::
*AResize Images**: Resize all images to a consistent size le.g., 224x224 pixels; to metch the input size required pre-trained models.
++Norma izationes: Normal ize pixel values to the range [0, "] or use mean aubtraction based on the pre- model's zequicements.
Haannding High-Rerolution. and Iilumination Varistionswe: Apply tecaniques to szendardize illumination and handle high-reselution images.
##t 3. Bata Augmentation
veTchhriquas quachriquas+::
*Croppings*: Randonly crop sections of the images.
*#lipping=:: Apply horizontal vertical flips.
*Kescalingsv: Sescele images by factor.
*ShhaariggAs Apply shear transformations.
> Apply random
*Rcaaticnre: Rotate inages by randon angles.
barrreee::: TensorFlos's "TmagphataGenevatoo" PyTorch's 'pranaforma',
Splitting vstraining Setav: 808 of the datase:
-.Validation SetAs: of the dataset
##t 5, Model Selection and Archicecture
v*Transfer Tearningsv;
vAPze trained Modelsas: Utilize models such as VGa16, VGG19, MobileNet, Xception, ResNet50, and InceptioaV3.
=nfine-tuning-::
models
Butterfly image classification datases.
viModal Architecturava:
.APrimary Model Start with InceptionV3 due to highh performance similar tasks.
**Alternative Modelaes: Evaluate Restet, DenscNiet, RéficientNett, and vision Transformers.
+*Gombines ned Architacturage+:: Explore combining CNN with Transformer-bant archt rectures to capture both local global festures,
Handling
vxcechri quaskus
Neeggtte Less Ponotions*s: Assign higher weights undecrepresent claases. +nvarsampllngge:: Increase the number of samplas for uncorsepresentes classes.
Xodel Training
v*Loss Punctionv4: Use cazegerical eross-entrepy.
-Opptmiizesse start SCD RNsprop.
Ryyperparameter Tuningas:
*aLibrariess:: Use PyCaret or Optuna for hyperpa:ameter oatimization.
**Paramatarsve:
learning rate, batch alze, number of epochs,
v*Orose vAGrose-Va_idate iddttiioonns Use k f-fold cross valldtinld to ensure model robustaess and generallzation.
#+ 8. Performenze Evslustion
*Mtriics*:
**coооaсys*:: Primary netric for evaluation.
Precision, Recall, F. Score*e: Monicor these metrics to ensure belanced performance across classes. Evalustion Techniquese+:
vxConfusion Katrixes: Visualize to understand aisclassificatioos..
31

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
vAROC Curverv: Evaluate model parformance acress different thresholds.
9. knsemble Metheds and Ancwledge Distillation
Ensemble Methoda**; Cembine predictions from miltiple models 1e.3., patch-wise and image-wise classi fiers) Knooleede Distillationee: Transfer knowledge larger teacher modela amaller student models compatt yet powerful models.
## 10. interpretability and Debugging Treshriquesa..
=*Grad-UM4+>: Visualize which parts of the images influence model's predictions.
<*R:MM Provide expl aplaarstins for individua predictions.
v*Puzposetv;
model
on
biological
understand model behavior.
sepraducibility
vARandom Seedsav: Sat random seeds for all processes to ansure reproducibflity.
-:Dosumnnattonne Document daza splits, model contigurations, and training parameters.
## 12. Deployment
v*Tazget DevicevA: Specify Lacget device deployment (e.g., cloud, edge device). *>Degoymmnnt EndpointAs: Define the enépoint for model deployment.
vxInferenze Engineex: Choose appropriate inference engine (e.g., TensorFlow serving, TerchServe).
DeDomrst rat Prepare demonst"ation of the model's capabl lit and performance.
By following this comprehenaive plan, agents effectivel vely build, train, and depl oy highly neural network model tallozed for claseitying butterfly images in the apecifled dataset.
Plan #3
#++ End-to-End Actionsble Plan for Butterfly Image Classification using Neuzal Networks
#d+ 1. **Praject inatializations
+bbjertivee.; Build highly accurate image classification model for the Butterfly Image Clessification datase: biology domain.
*usse Expertise Nedium
##t6 2. +aata collection and ereprocessingev
*aaaasee Source User-uploséed Butterfly -mage classification dataset with label information in "labels.cav'. **tops***
Load images and labels from the specified dacaset and 'labels.cov'.
+*att cleaninga+:
Frsure all images are correctly labeled.
any corrupted unusable images.
**atta Pzeprosessings<:
Resize images to consisrent size (e.g., 224x224 pixels).
Norms pixe! values to range [0, 11.
++aata Augmentationes:
Apply augmentation techniques
cropping, hozizontal vertical Clipe, reacaling, shear, zoom, rotation to increase dataset size and variability.
like
"ImagelataGeneratto"" pytorch's 'eransforma".
#346 3. **Mode1 Dewelopment
*oddel Typeva:
Nevral Networks
**StepsvA:
+everagg Transfer hearnings*:
Use pre aze-trained OWNs such as vos16, VEG19, MobileMlet, Xception, Reallet50, and Inception?3. models
Butterfly Image Classification
Salection Architactura+*:
Start with InceptionVS to ita high acouracy in similar tasks.
Experiment with other models like Restitt, DenseNet, EfficientNet, and Visien Transforme:ss Consider cembining WAs with trensfommerbassed architectures for impreved performance.
32

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
3. vénembbl Methods and knowledge Distillatioav::
Implement ensemble methods by combining petch-wise image-wris classt fiera. knowledge distillation transfer knowledge from larger models smaller, efficient models.
relzaining Evaluatione*
eASteps
1. **hataset splittingue:
Split dataset into training (804), validation (108), Lesting (104)
Traininge::
Use categorical. cross-entropy as the loas function.
Utilize opt imizers such aa 3GD RMSprop.
Implement early stopping and learning
reduction plateou prevent overficting.
Monitoringes:
Regularly moniter metrics such accuracy, precision, recall, and Fl-score. validation
fine-tune hyporporamesers.
4. Lnn Clase Inbalanceve:
Addreas alass tmbal ance uaing weighted losa functions overaampling techniques
ilypesparemeter Tuning
Velidationass
Perform syperparameter tuning with libraries like pycaret or Uptuna. croas-val to anaure robustnesa generalization.
#*4 5. x>Model Interpretability and Debugyingsa
**Stopsr::
Interpretabilityy
Use Grad-CAM or LIME to visualize important image regions influencing model predictiens. Ensure the modal focuses relevant biological features.
Webbgging:*::
Analyze misclassified images to understand model limitations.
Iterate on model improvements based on these insighta,
***6 5. .*Reproducibilaty and Cocumensatione
Reproducibilityye:
Set randon seeds for all processes to ensure reproducibility. Rocument steps, including splits, modal configurations and training parameters.
Verrioo Control..:
Use version control systems (e.g., Git) to track changes and collaborate efficiently.
loyment InferencevA
Modell Export<A:
Export the trained model suitable format le.g., Tensorilow SavedRodel, (NNX).
Deployment
Choose deployment platform (e.g., AKS BageMaker, Googla AI Platform).
Set up the dyployment endpoint for real-time inference.
3. #Inference Engines<:
Optimizo the madel for target device le.g., mobile, adge devices) if necessary. Tmplement efficient inference pipelingg ines to handle incoming image data.
#+*+ 3. ADDeoosstratinn and User Feedbacks
*Stopp++:
33

---

by replicating AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Uemonstratioone:
Create demo application to showcase
capabl itiea.
Provide user-friendly interface for uploading images and viewing classificaticn results.
*+fser Feedback*s:
Collect Leedback from users to identify aless for improvement.
implement
updates based user insights.
By following this detai led plan, AT agent car but a robust and acourata neural retwork model for cleast fying butterfly images biology domain, enauring high
sstisfaction.
D.3. Plan Decomposition
Example Results from Plan Decomposition
Data Agent
#* Summary of the Plan Ior Fashion Image Classification Using PyToreh
Dataset Retrieval and Collection
vaRetzieve Detasetex: Lead the image paths and corcesponding labels from the provided Stain_labels.cov and test_labels.cav files.
the dataset includes images
clothing categories.
#346 2. Checking Dataset Types, Shapea, and Properties
v-Data Types**: Verify that the images in the correct format (e.g., JPEG, PNG) and the labels correctly mapped to the images.
v*Shapeses; Check the dimensions
images. images grayscale,
the single chanzel acress three channels. balanced across tha elothing estegories
Proppprccaas-:: Ensure that the misaing or corrupted le.
#+ 3. Breprocessing the Datase: vNNemalizationvi: Scale pixel volues to the range (0, 11.
images to consisteat size (e.g.r 224x224 pixels) suitable model input. *ssplittinge:: Split the training data into training and validation sets (e.g., 808 training, 208 validatien) prevent overfitting,
<Resizingga:
monitor the model's performance
#d4 4. Efficient Dets Augmentation
v<Ieshriquesee: Apply data augmentation sechniques increase the diversity and rebustaess of the training data, Techniqued include:
hcrizontal flip Normalization
croppiing
Random erasing
Rotating
Shifting
Zooming ng
*Pppelines==:
**Pipeline 1++: Random horizontal flip and normalization.
*sPipeline 2+*: Random horizontal flip, normalization, random cropping, and randon erasing.
##t# 5. Extracting and Understanding Underlying Characteristics
vxClass Diatriautionva: Analyze the distribation of classes
balanced dataset.
Tmage Quality.a: Check for variations in image quality and resclution.
Insightse*: Identify any patterns or anomalias
could model performance.
*oPeroormacce Metricass: Use metrics auch accuracy and confusion matrix to laae the model's performance
and understand misclaseification patteras.
By following these steps, another data scientist should able to raprocuce the results and build a robust classi sesificas fatttin madel that performance analysis.
apect fied acouracy rementa providas inaightful
Model Agent
## Summary of the Plan for Fashion Image Classification Using PyToroh
##f User's s Requirements
user needs classi ficazion model that categorizes images into one clathing categories. should achieve at least 85% accuracy the test set and be implemented using PyTorch. Daza augmentation techniques and a cenfusion natrix for evaluation are elso requiced.
#* Project Plen Overview
34

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
F#f Data Freparstion
r-oad Dataset**; Load image paths and labels from "train_labelscom and 'test_ past_labels.csv' Conovert Greyscale to RGB**; convart grayscole images to RGB.
*PPrepoosseenggss Normalize
resize images
pixels.
vASplit D Split :raining data inte training (809) and validation (204) sets.
#### 2. Data Augmentation
TxTechr iqushaiquess4: ques Apply random honizontal fl normalizar ieatior, randon cropping, random erasing, rotating, shifting, and zooming.
**Pipalinea*<: op two proprocessing pelines with different augrentation techniques.
**4+ 3. Mode: Acchitecture
v*Simple Ch Start
simple CNA model.
viAdvarced Modelsxx: Consider ResNet18, Rfficient or Vision Transformers (ViTs).
..Byorid Models+a: Explore hybrid models combining CIN and VLTs.
### 4, Training
veOptimizer +*ppschs and Early Stoppingve: Train for up to 150 spochs with early stopping. viMonitoringus:
Pune-ionse:
optimizer
Entropy
validation accurany
curves.
#d4 5. Evaluation
least 858 seauracy veConfusion Matrixex; Generate a contusion matrix. viTop-1 AccurazyA:;
**Acoursoy=*:
cop accurscy.
#### Implameen ementation
>*PramewozkAs: Use PyTorch and torchvia prrchiision..o sion.trana tranaforma.... Reesooccsss.:: Utilize resources from Kaggie and PapersBiihCode. model: deployment 1f needed,
v*Dep oyment**;
Visualization Reporting
Conffcsioo Matrix.:: Visualize the confusion matrix. Performance Metrica**: Repors acouracy and other metrica. DDcsumentation-: Document the entise process.
petailed for Deta Monipulation and Analysis
Decaset Netrieval Collection
-*Retrieve betaset*.: Load image patha and labels from 'train_labels.cav and 'teat_ covv
#+ Checking Dataset Types, Shapes, and Properties
v*Data Types**: Verify image formats and label mappings.
viShapesux: check and convert image dimensions to 85B if needad, v*Propertiesse: Ensure balanced dataset and data integrity.
### 3. Preprocessing the Datase:
*No:malieationvs Scale pixel values to [0, 1].
Ressszingee: Resize images 224x224 pixels.
.isplittingas: Split data into training and validation sets.
##+# 4. afficient Data augmentation
..Toahriquasse:: Apply various augmentation techniquas.
>*Pipelinessr: Create two augmentation pipelines.
## Extracting and Understanding Underlying Characterissics
villass Distributionv:: Analyze class cistribation.
vaTmage Qualityиa: Assess varlationn
image quality.
Dataa insightsee: identify pasterns or anomalies.
Porfornacce Matrica**: Rua luate model parformarce using accuracy and confisiio matrix.
# Expected Outcomes and Results
Quantitative: salanced dataset, consistent image nermalized pixel values, augmented data. vuQus" vo Bata integrity, improved nodel gonaralizat ization, insightful analysis.
##t Responsibilitiie Fulfillment
#* Now to Retrieve Find High Performance (s)
**iniiill XodelsAN: Start with simple CNN architectures.
**Advaneed Models+e: Experiment with ResNett8, EfficientNet, and Vision Transformers.
..Hyorid Modelsva: Explore combinations of CINs and ViTs.
**ResoLicese4; Utilize
reposito:les
Kaggle PaperaMLLhCode implementation tuning.
##4 2. Now to Optimize the Hyperpacametera the Retrieved Kodels?
*:Optimizerss:: Use SGD with learning rate 0.015 Adam eptimiser.
*Hypesparamster Tuning*+: Experiment with different learning retes, batch sizes, and augmentstion techniques. **Barly stoppinges: implament early stopping based validation loss prevent overfitting. toMonitorings.: Continuuully monitor validation accuracy and loss curves.
35

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
F## Extract Understand the Underlying characteris:ies the nataset (5)?
**Class vistributiene::: Use 'pandas' analyze the distribution classes.
Image Quality==: Agsess image quality and resolution using 'FI1" or 'apenCV',
v*Data Insightsws:
analysis viPe:formance MetricaAs: Use "aklesta.metiicc" to evaluate model performence and generate confusion matrices.
exploratory
identify patterns anomalies.
### 4. Fоw to Select the Top-k Nodels or Algorithma Based on the Civen Fland?
.ARva vation Metrics:A: Uae accuracy
corfusion matrix to ava luate models.
**Top-k selectione:: Rank models based thair performance matrics and select the tep-k models that axcsed the 858 acauracy requirement.
=:DosumentaLionv:: Document the performance relevant metrics.
including accuzacy, contusion matrix, any
By following this comprehenaive plan, you wi'l be abla to retriave hi high-parfermance models, optimize their hyperpazameters, understand the dataset'a characteriatics, and select the top-k models based on their performence.
D.4. Agent-Specific Plan Execution Outcomes
Example Results from Prompting-Basee Based Plan Execution by Each Agent
Data Agent
##4 betailed Stepa for Data Manipulatioo and Analysia
#iti 1. Detaset Retrieval and Collection
**Retriave batsset**:
is available LLeppeeimett//aaaaeeeeeeee
+SStuccuuree? The directory siructure insludes "train', ' 'test', lebelssebellsccrr and
veloading batasr:
image patha corsespanding lsbels ltaiin_abels.cov 'test_labels This be done using libraries like 'pandas" to read the csv files and "os" to nsvigate through the dizectories.
*hyyes: This step ensures chat we have access to the correct dacaset and that the paths and labels are correctlly mapped for further processing.
#### Checking Bataset Typea, shopes, Properties
**Data "ypes=e:
v*Tmage Formster: Varify images in the format (JFGG, PMG). This done Hibrartes Like 'IIL" (Python Imaging Library).
*eLabelsee:
labels correctly mapped images by checking
files.
**Shapaаx::
vmamage vimensionses: Check the dimensions of the images. If images in grayscale, them by replicating the single channel across three channels using libraries Like "FIL" or 'opencv).
APProertiessv::
v*Halanced Datasetre: Ensure
is balanced scross
clothing categories by analyzing
label distribution using 'pandas".
.xTntegrity Checkex: Verify that there are ro misaing or corrupted fil by attempting to open each Image file.
AWhyyss Easuring errors during preprocessing and augmentation.
Lypes, shapes, propectiea ezucial for model training preventa
#### 3. Preprocessing the batase:
Norrms in+t:
v*Scale e Pixel Valueges: Normalize pixel valves to the range [0, 1] using Libraries Like 'PIL" or "torchvision. Lransforms'
vxConslstent Simews: Resize all imagea to consistent size (8.g., 224x224 pixels) suitable for modell inpot using 'PIL" oz "Lorchvision, Lranscrrmm
*4pplitt
vATrain
craining validation le.g., training, velidation) using oww.ooooooeiiooo.tttiiieceeeeeee
Splitsx: Split training
Khyas: Mormalization ensures that the pixe. values during training. Resizing ensures that ell images have the same dimensions, which is requirement for most deep learning models. splitting the date helps in monitering the model's performance and overfitting.
common acale, which helps in faster convergence
36

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
diveraity, ## Efficient Dota Augmentationn
<echnicuusse::
Randoo Rorizontal Fliper: Randomly fllp :magea horizontally
<oosallizationv:: Normalize Images Lo have zero mean and unit variance.
*anddom Cropping**: sandemly ezop images
introduce variability.
RaRddom Erasingav: Randomly erase parts of images to make tha modal robust to ccclusions. Rottatingua: Rotate images randomly to make the model invariant orientation.
Shiffingg:: Shift images randomly
simulate different viespoints.
*oooning*+ Zoom in and out of imaged to "rtrodsce acala vartability:
Aipellieesw:
v*Pipeline lex: Apply
horizontal flip normelization.
vaPipeline 2xx: Apply randon horizontal flip, normalization, randon cropping, and random erasing.
*hh+++: sugmentation increases the diveraity and rebustness of the training which halps in improving the generalization of the nodel.
#### 5. Extracting and Understanding Underlying Characteris:ics
**Class elatributionnss **Analyse Distrabution:e: Use 'pandas' to analyze the distribu.ion of classes Le easuze a balanced dataset.
**Image Qualityx::
Veriationses;
variations image quality resolutien using 'PIL" 10penCV).
++Data Insights**:
vaIdentity Fatternsww: Use exploratory data analysis (EDA) techniques to identity patterns or anomalies in the data that could affect model. performance.
APPeroormance Metricasy:
veEvaluate Modelaw; metrics
accuracy confusion understend misclassification patterns. This can be done using "bleasn.mmtrress.
evaluste model's performance
Why++: Uaderstanding the underlying characteristics of the data helps in identifyiag potential issues that could affect model perfermance. Anelyzing class distribution ensures that the dateset is belanced, which is crucial for training fair and unbiased model.
Expected Outcomes
Quant tvversr::
Balancee Detaseter: A datase: with equal number of images for each clething category. Consistent image Sizeer: All images resized to 224x224 pixels.
*Nomaalizad Pixel Valuessa: pixel values sccled to the range [0, 1].
vuguennted DataAs: Increased dataset diversity through augmenzation techniques.
Quaa litativee:
vaData Integritys:: No miasing or cocrupted files.
Improved Model Generalizatione:: Bahanced model robustness due diverse and augmented training deta.
vxinsight Analysiars: Clear understanding of class distriburion, image quality, and patential date patterns anomalies.
By folloving these
repraduce the results build robust image classitication mode: that meets the specified acouracy requi rementa and provides rul performance analysis.
steps, another scientist
Model Agent
Steps Modaling Optimization
##4 Retrieving Bigh- Performance Models
*Initial Modela:ve
vxSimple
Convolutional Network (CHN) architectuze. model will as beseline to understand the dataset and the task.
44Rdvanced ModelssAn
ResNetL18- A well known deep learning model that has shown excellent performance in image classicication networka.
tacks. Tt residual connect
training
*BBILciien Known for ita efficiency in terma of parameter count and FLOPs while maintaining high accuracy. RFficientNet-B
be good starting peint.
vevision Transformers Ivits) e*: these models have gaired popularity for their performance in image claasi fication tasks.
variant viT-Small
constdered.
llybrid Models:**
V*NNN vice:: Combining the feature extraction capabilities of with the global context understending of ViTs. This hybrid approach can polentially leverage the strengths of both acchilectures.
37

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
### Optimizing Ryporparomsters
w= Stochastic Gradient Descent with learning of
vAAdamAN; Adaptive Noment Estimation with leatning sate of 0.001.
*Hyperparomatee Tening:vx
vallaarring Rateva: Experiment
learning rates (0.001, 0.006, 0.01, 0.015) to the optimal rate
**Ratch 3ixe+*: Test botch atzes (16, 32, 64) to balarce between mamory usage and training apeed,
Augnentation Techniquesra; Apply different augmentation techaiquea validaticn accurecy.
their impact
v*Number apochse::
up zo epocha early stopping based on validation prevent
overficting. **Weight Decayлr: Reqularization parameter for SGD, aet to 0.0005.
+mptimal Va ves:ve
velearning Ratevi:
0.015
v*Batch Sizes:: 32.
wieeight Decвysи: 0.0005.
vaNumber of Rpochsax: Up 150 with early stopping.
#### Extracting Understanding Characteristics
cCemputation Complexity:as
.-Number of Parameterswe: Calculate the total number of trainable paramezers in each model.
..F1OPS (F (Flosting Point operations per secord)**: Maasura the corpotational complexity
**eemozy Usage:**
v*Model Size+:: betermine
of modell memory (MB).
wiTralning Memory Usagers; Konitor GPU memozy usage during training,
**inference Latency:*.
<eTnference Spead**: Measure time taken for single forward pass (1n Hasconds)
**hyy Each Step Essential:ve
v*Modal Selectiones: Cheosing variety of models that explore different architectures and their strengths.
**Hyperporameter runinge*: rine-tuning hyperparameters erucisl achieving optimal performance ansuring the model generalizes wall.
CCCaaatteiistti Analysisrs: Understancing the computational and memory requirements helpa in selecting modela that not accurate but also efficient and deplcyable.
Candidata Medels and Expected Porformancs
Acourscy+:: Expected to achieve around 884 the test aet.
Number of Pasameterswx; Approximately 11.7 million.
PLOP åround 1.3 billion.
*Model sizes:: Approximazely 44 MB.
vxTrairing Timeva: Approximataly hours single GPU.
veiferennee Speedve; Around 10 milliseconds pez image.
*22 3tfieientMet-S0rr
AccurrcyyA: Expected to achieve around 873 on the test set. Number of Parametergve: agproximstely 5.3 million.
#LOP55 Around 190 million.
<Modal
Approximately
Training Time+e: Appreximately hours on single GPU.
Inference Speadee: Around secondss par imaga.
443, Vision Crensformer IVIT Ss
*czurscyy*:: Expected achieve around 86% the test Number of Parometarsvx: Approximately 22 llion.
billion.
**odaa size+*: Approximately 85 MB.
TTrairing Timer*: Approx imately hours on single GPU,
vIIfference Speedvx: Around 15 milliseconds
image.
By folloving these detailed
and focusing the modeling and optimization parts, you will be able !dent fy and select the top-3 high-per modela for the given Image claasi claasification ficttion taak.
38

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
E. Showcase of Generated Pipelines
Generated Pipeline under Constraint-Fres Free Setting by Auto.ML-Agent
User Prompt: Please provide classification model that categorizes images into one of four clothing categories. The image path, along with its label information, can found in the files train labels.csv and test labels.csv.
Output Pipeline:
import
import random
import time
import pandas a8 pd
from import Tmage
[zom meaenn.odee_sseeeeeee
import train_test_epllt
skiearn.mett mettrissetsissssco.... import accer,,, fl_scere, precision_ecore, recell_score Erom torchvision impert datasets, traasforms, models torch.utils.dats. amport bataleades, Dataset import torch
import torch.an as na import
import numpy as
import gradio as gF
Define conatanta SEED = 42 DACASET PATH
"HAIN_LASELS
os.path. join (DATASET *ssoin_labels.esv
TEST TEST_ JATE_AADESS_CCC CSV os.path, join (DATASET_
"Lest_labels.cav")
18AIN_L_R os. pa.path..jpe [LATASE
TEST_DIR oa path.jiin (DATASET__AAC PATH)
NO0ET_SAVR_PATT
Set seeds for repreducibility random. good (SEED)
corch.manuel.. seed (SB2D) ap ap www.raddd.sedd/aeeeem. (SEK0)
Define device for model operations device device ("cuda" if wtt..cu...i...valebeeee...... else "cpu")
dataset class
class ShopeeUataset [
_nnit_(sals (salf, esv_file, rost_dir, cronsformeNone)
self. read_caviissvttilll
self. root_dir self.transform transform
len (s81f): seturn len
Get get_tttt__t_seeee_e_____________ (self, idx):
ing_nene os.path. join join root__.r self.1ebel df. ilee[idx, [idx, 03) image image.cpen limg .convert ("RG"")
labal self, aaf
:]
if tean.erangfom::
Image self.transform (image)
return image, label
naaa prepracessing and feature engtneering
preprocess_dalalll
transform = transforms. Compasee
Transforms.Reseee 128, 28)1,
transforms. ToTensor Tr (),
transforms. (0.5, (0.5,))
train_delaset ShopeeDs.aset (esv (ile-TAAANRRAIN CSV, :oot__dr-TRAAN DIR, Lransform=
cest_darasee = shopeeUataset (ew_ Foot_di=-1EST__LR, Eransformetransfon
train_size int 10.7 len (eattriin_aataret))
valid_size int 10.2 len (tatiinn_datt
len train_aize val sa
39

---

(_true.extend opu .umpy sxrredeexeendprreiette. AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
train_dataree valid_datases test_dateses carch.atill.dat.rronlee.. (trair__ataset, (tzain val s128,, test_aize]]
sraii__loade batatoadar (train_dataact, dataget, batch_sizaase shuffle=Truel vallideataset, batch_eize=52, atuffle=False) lest__ooade Dataloader (Lest_datsed) batch_si shuffle=False)
volde_
returr tradn_loader, valid_ loader, leader
Xodel training def tra' "arar_mmode model (model, train_ train_loader, va 1d id__eader]:
criterion an.CrossEnt ropyyoosst
optimize: - eptim.Ades Adem (medel parameters 1=-0.00001)
mode to m (device) beat_accuracy 0.0
for epoch in range (150): model.train()
surning_less = 0.0
for inputs, labals train_losder: Inputs, labela to cel,
opt
outputs model iinplttn (suttptt labels)
less. bass.backward)) retimizer.saapp
losning_loss 1- loss. itemm) Laputs.size(00 (0)
spoch_ioss zunning_lose [en
model.evcl(1
correct
total with torch.no_gradl))
for inputs, labels in valid_loader:
isputs, labels = outputs (inpurs)
(deviic
predicted = total labels.size
1)
i- (predicted =- labels) sumll
sccuracy corract sotal
if accuracy best_arcuracy: best_accuracy ascuracy
torch.save laaadelsstat__daae (), MOREI_SAVR_PANT
print (f"Epoch (sp00h, Lose: lossch_, Validation Accuracy: Baccuracy:.4EV1
mode' Loadlood__dee state dict Itorch. load (MOOFLLSAVVV
returt model
Model evaluation
def evaiuate (model, test_loader): modei.eva2 y_true = y_pred
with torah.no_graadt:
for inputs, labels in Leat_loade:: inputs, labels inputs.to tdbele. outputs modeliinpstss) predicted torch.max (outputa, 11
())
performance_scoree
'Accuracy'; securacy_score. (y_tree, y_pred),
"Preciaion's praciaion_acorel iyw.eeccorreyyyuuu.. y_pred, avarage=/
'Recal ": recs _so_ssooee truee, y_pred, average= averaggeweeiihhhheedd
'PI': (y_ pred, "eerage="veighted"
mode-_size path. getsize (MODEL_SAVE PATH) (1024 1024) in start_time time. time.time time ()
in range (100):
mode (inputst
40

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
inference_ time
(time. time () start__time) / 100 average inference time
complexity_scae
'Xodel size (MB)"; model model_aize,
(Inference Time inference_
returr porformanco_ acores, complexity_ccores
Prepare model for daployment
def prepare_model_"ae depl lepploymenttttttttmmmmemmmmm oyment (modal):
model.evaz
scripted_nodel torch. script (model)
SWL.ELLLSV..AA....o...... ptth"", "_scriptad.pt")) returr scripted_modal
scripted_model.
Deploy modal using Gradio
def dep_ oy oy_model (nodel1:
prediet timage):
image Saannfffrs...ompaee
transforms- Resize (28, 281),
transforms, ToTennoorrrr. !),
Lrame Now.maiioeeeeeeeeeee (0.5,), (0.5,)
(1mage) unsqueesze (device)
serch.no_grad!!!
outputs medel (image)
predicted torch.man mox (outputs, 1)
zeturn predicted.itemt)
interface - q2. InterIace Conpreddct, inputs-g: Image (Lype="pi"), outpuls="Label") url_endpoint interface. (sua returr uri_endpoint
The main function to orchestrace the data Loading, data preprocessing, Teature engineering, model training, model preparation, model deployment, and model evaluation def main():
Step 1: Retrieve oz load dacaset train_loader, loade,, test_loader preprocess_data.)
Step 2: Define model mode:. models. reseess. (peetrained)True) num ftrs febeel.f.iin_eeeee fn_tuuuess modo:. Liner (turf Assuming classes
step train_mode (moda), train_ loader, val [d__oodderr mode
Step Evaluate model modee__pesformance model_complexit = evaluate_mode (model, test__ozder)
Step 5: #repare model for deployment
doplayabla_mece prepare_modes mondel_fo=_dppllll doploymmn (modal)
Step 6: Deploy model
url_endpoint deploy_model (deployabie_model)
train loader, rodel_complexity
loader, past_loader, model, deployable, s_ndel,, url_enopoint, model _performance,
1f "_main_
train_loader, valid loader, test_losdes model, degloyable_model. usi_endpoiat, model_perrfomarcee nodel_complexity main()
print (*Model Ferformance on Test Set:", model madel_perfarmance)
print (*Model Complexity:*, compplex_t))
print ("Doployment URL:", srl_enndoint))
Generated Pipeline for Constraint-Awwe Setting by AutoML-Ageet
Please provide classification model that categorizes images into one of four clothing categories. The image path, along with its label information, can be found in the files train. and test.labels.csw. The model should achieve at least 85% accuracy on the test set and be implemented using PyTorch. Additionally, please include data augmentation techniques and a confusion matrix in the evaluation.
User Prompt:

---

AutoMI-AAeen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Output Pipeline:
impoxt
import random
import
import pandas pd
import numpy a5 up
import torch import torch.En
import torch.optin optim
from torchvision smport datasets, transforms, models from worch.utiis.....
impor: Datatoades, Dataset
from PIL import Inage
Erom sklearn.modes aklearn.met:ics import accuracy_acore, [l_score, confusion_melriaa import seaborn as ans
besaarn.mdel_peecctiii selection import train_rest__pile
import hatplatlib.pyple as plt
import gradfo as
Set sandom seeds for reproducibility SEED zandom. seed (SEED)
torch,manual seed (SEED)
np ap.raandm.senn (SERD)
Define device for model operations device torch. device ("cuda" if zorch.cuda. iaee.cuce......viaaeeeeeee else "cpu")
path saving loading (a)
Custon Dataset class
class ShopeeDatasel i
def init init__iiteet__ (self, dataframe, transform-Nene):
self.dotaframe dataframe
self.transform transform
def _
return len (attafrrmma
def getiteee_ (self, idx):
ing_path dat.ffaaaaraan....... iil.rrrr (' ('image')
label selr,dataframe [oo[lex]] labe Y]
Inage Image.cpeniin ting_tinnppali if self.transform:
("RGB')
image sell,transfow (image)
seturn image, label
propracessing and feature engineering def preprocess_datad d)
train_lebels pd.read_ test_labe.s readecsssv path, join I0ATASTTTTT "teat_labels.csv")
path PAIUI labeln cev"))
Laain__abels image'] lsbin_labels: image'' .ppply (lambda os.path. join (DATASET_PAAA PAT4, x)
test lest_labeiss" ('imgee') test_labela [ ('Smage' a.ppplyalllll (lambda x: as.path, join (BATASST__ x!)
Split the data train_data, val_data trein__est__piitt (traan labels, test_size=0.2, stretify=train Labb vál_datá, test_data train_test_spiit data, teat_size=d.s, stratify-val_datad [abbll
Define trensformations
tronsforms. Compose
R zedCrop (224),
RandonRes
trans forms. www.do/ms...dooiiiiiiiioi..ooo.a wontoliilrzzzitlalraa
transforms. kandomootssion (10),
transforms, ToTensor ToTenooo (),
transforms.
www.eforrs...ommlii...... Normslize (mean- 10. 485, 0.456, 0.406), 10d-10.223, 0.224, 0.2251)
LransCorms, Compose
transforms, Resize (2561,
transforms. wtw.smmseeettttrrrrroooo (224).
tranaforms. ToTenaor Tonttoono.. (),
transforms. wwr.formm...omsi....... (mean= 485, 0.456, 0.406), std- std=10.223, 0.224, 0.2251)
Irain_datases ShopeeUsrasett (traia_data, transforrwttaiiasttaaffooo.mmm
42

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
dal_sat shopseistaset (val_data, seaarforrvail.ettttrraeeee
test detaaet shopeenataset iseat_data,, transform-vll_eest ttaaa_taafoat
train_losie loader Dataloadar (troin (traiindotaoet, batch_siza==22, shufflesTruel
val__ Lest__ladaee Dataloader (Lest_datssel, batch_s shuffle-false)
dataset, batch_size-ee shuffle-False)
returr troin_loader, val val_leades, test__laader
def train_model (model, srain_leader, vai_loader);
critarion Cn.Grosannt ro.rrssa optinise: eptim. Adam paramelers)), 1c-0.0005, weight__eecay-D.c
nun_opocha
best_val_ 055 I'
berly_stop_oount carly_atop_patieme 10
epoch zange (nun_epochs!: model.trein()
rurning_less 0.0
for inpitts, labels in train_loader:
inguts, labels inputs. tobele.to(device) opt
outputs modellinpate
(suttput
less. loss.backward() ogtimi running_loss +- loas.itemt) inputs.size(0)
epoch_ running_loss (train_ cader.dataset)
Validation
model.ev:l(1
val_
with Lorch.no_gradl)):
for inputs, labels in val_er;:er:
imputs, labels inputs.to/deviie)) labels.to (devica) outputs model (inpuzs)
criserion/outputs, val_lass +s losa.item() inputa.size
val_loss val_loss lea [eatvel_oodderddtttseee
print (F'E [spoch+1] (anchell//nummmpohhhle train (epoch_losa: Vel Losa: [val_lass: )
Early stopping
If val_loss beat_val_losa, best_val_lose val_
saveh. snodel.scate.ddict
h/tpgettwwwkkeeeee/e/oe//ee///// pth)
eariy_stop_com
ear-y_stop_con +=
early_atop_coure parly_atop_pationee
print ("Early stopping")
break
reture model
def evai model (model, test_loader):
mode lead__tate_dd itorch, laai"h.oonn"..oontt.. aageenttorrkppccttrraaaaaao ts/ettoddssssssttttt_toooooa laaddll__tttmoce pthh")) mode:.eva..)
all_preds
abeli ()
with tordh.no_gradt::
for inputs, labals in test_ loadero
inputs, labela toputs.. ta
outputs modellinpata
preds torch. (cuzputs, cll_press.extend (preds cpu .numpy
all [ll_labelss s.e.teeeseeeeedd labela. ap () nummp ())
accuracy accuracy labels, predss
f1 score (al labels, preda, averagee weighted")
performance szores 'AC!! accuracy, 'F1: 11
43

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Neasure model cemplexicy start_ time.time()
torch.no_grad(1:
for inputs, in Lest_
inputs inputs.to (device))
model (inputs)
inference_tion (time.t () start_time) len (tesst__llade [entteet__oddeerddeesses))
mode si28 08 pataigetaazee wttge/tt m 186 in MR
wwnglexity_score =1
i_nference inference_teee time,
'Xodel size (MB)": model_sip
Generate confusioo matrix conf_matrix pan.uusioo.mmtciilllllaaaae all_preds)
Eiguue 7)
ans-heettmpp (cons_matniiieeeeeee anmot=Irue, fm cmope' Bluas!) pla.xlabe ('Predictadr)
plt.ylabei ('uu'"
plt.titlee Conffuation Matrix') plt.show()
returr performance_ccorss complexity_ccores
def @reppare_oode__ie _tr___oorrepplyyoooe depplomeent (mocel):
Assuming the model is alresdy saved in the best__odel..pt file
warpett_orrrapceetraaaeeeee hodels/best. pthh")
modei.evai
reture model
dep' oy my_ma (model):
def predict limage):
transform transforms Comnssoosscccapeeee [[
transforms. Resiszz (256),
transforns Swa.ffrree.eetiie CenterCrop(221))
transforms.1oiemm
transforms.Noree No..ooorolli..... Imaan= 48., 0.456, C.406], aod= [0.229, 0.224, 0.225])
Inage (crmm www.ueeeeuuuuuuuu.ommm (0) (device)
with torch.no_gradl))
output modal (image)
pred terch. (outtppt, 1)
return pred.item()
interface qe, Inteeeerace (Crpprediot, inpute-g: Image (Lype="pil"), outputs="label1) url_endpoint interfecce.launcle (zses reture us uri_endpoint
the main function to orchestrace the data Loading, data preprocessing, festure engineering, model treining, deployment,
model preporation, der main():
evaluation
Main function
the image classification pipeline.
Step 1. setrieve or laad dataset from hub (if available) usev's local storage (if given) lrade_loader, val_loader test_loade preproceas_datatt
Step 2.
Define zequired medel
mode:. models.resnet18 num num_ftra ftrs model. in features mode.. Ln.Linear (num_"tre, mode". tolavice)
IMAGENET V1)
Step Train the retrieved/loadde mode:. mode:. train_model (model, loader, vol_eeader!
Step 4, Evaluate the trained model
mode-_perTonnee model__omplexity evaiuale_modee (model, Lest_loader)
Step 5, Fregare the model for deployment
deployable_model prepare_modei_. soaae___noee________.eeeooeee (model)
Stap Deploy model url_endpoint deploy_mode (deployabie__ode))
44

---

AutoMI-Ageen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
returr train_iosder, val loader, teat_loader, model, deployable_model.
url_andpoint, mmcel_performence
modal_complexity,
if name _mmin_nn"
trade_ loader, tes:_ioader, model, deployable_model, url_endpoint, model perfl_eerfermance cocel__compeexeee main()
( (*Model Performance on Test Set=", model__ereeteeeeee ppree___erormance
print (*Model Complexity:" model_compll lomplltyy
print (*Gradto URt. Endpoint:", url_endpoint)
F. Detailed Experimental Results
This section presents the detailed experimental results. Tables 6, 7, and 8 show the average scores with standard deviations, computed from five independent runs. Tables and report the results of the ablation and hyperparameter studies, respectively. Table 11 reports the average time and money used to generate the final code in single run. Finally, Table 12 and Table 13 present the search time and performance comparisons between the recent SELA (Chi et al., 2024), Agent K (Grosnit et al., 2024), and our AutoML-Agent, respectively.
Table 6. Performance comparison with the SR metric. Best results are highlighted in bold.
Method
Image Ciuccifleation
Batterlly
Shopen
Classifiation Ecomm
Entail
Banuna
Claciffeation Software
Tabular Regression
Crab
Grop
Smoker
Chastering Student
Weather
Forceasting Electricity
Cora
Classificatinn
Cieseer
Avg.
Constraint Fier
GFT:3.5
0.000
0.000
0.000
0.000
0.300
0.100
0.000
0.000
0.400
0.000
0.000
0.000
0.000
0,000
0.057
GPT-4
(-1.447)
0.200
(40.548)
0,600
(+0.000)
0.000
(+0.548)
0.400
(+1.5548)
0.400
(40.418)
0.400
(+0.548)
0.400
(=0.548)
0.600
(+0.418)
0.000
(+0.000)
0.000
(H1,000)
0.000
(+0.000)
0.000
(+0.548)
0.400
(+0.554))
0,400
(+0.366)
0.314
DS-Aggent
(20.548)
(10.447)
(10.000)
(20.647)
(20.447)
(20.274)
(20.000)
(20.447)
(20.224)
(40.000)
(20.000)
(20.000)
(20.548)
(10.548)
(10.281)
AnnoMI-Aggwd
1,000
1,000
1,000
1,000
1
1,000
1.000
1,000
1.000
1,000
1,000
1,000
1,000
1,000
1,000
(20.000)
(20.000)
(a0.000)
(20000)
(s00000
(20.000)
(60.000)
(e0.000)
(20.900)
(+0.000)
Constratan Aware
GPT-3.5
0.000
0,050
0.000
0,000
0.050
0.150
0.100
0.050
0.150
0.000
0.000
0,000
0,000
0,000
0.039
(40000)
GPT-4
0.150
0.350
0.200
0.200
0.150
0,000
0.650
0.100
0.400
0.000
0.000
0.000
0.400
0.500
0.221
(20.335)
(10.157)
(20.447)
(10.447)
(20.335)
(10.000)
(20.418)
(:0.224)
(20.335)
(10.000)
(10.000)
(20.000)
(20.335)
(10.354)
(10,266)
DS AgeAt
(+0.411)
(+0.487)
(+0.000)
(+0.326)
(30.335)
(+0.487)
(+0.447)
(+0.447)
(+0.274)
(+0.000)
0.000
(+0.000)
(+0.335)
326)
(+0.411)
(+0.306)
0,500
1.000
1,000
1,000
0,750
1,000
1.000
0,750
0.750
0.750
0,900
1,000
0,750
0.750
0,871
(s0.112)
(40.000)
(20.000)
(20.000)
(+0.000))
(40.000)
(200000)
(=00000)
[=0,000)
(=00000)
(60.224)
(a0:900)
(+0.024)
45

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 7. Performance comparison with the NPS metrie. Best results are highlighted in bold.
Method
Image Classffication Batterfly
Shopec
Text Clacsifiation
Ecommm
Entail
Bonana
Tabular Classfication
Software
Tabular Regression
Crab
Crop
Smoker
Tabular Clustering Student
Time-Series Forecasting
Weather
Electricity
Node Chactifcation
Cora
Citeseer
Avg.
Constraiat Free
Human Models
0
0.921
0.935
0.664
0.976
0,669
0.328
0.476
0.513
0.750
0,970
0,916
0.811
0,702
0.754
(=0.002)
(50,0122)
(+0:000)
(#0029)
(400000)
(H0000)
(=0.000)
(=0000))
(=0.000)
(40.000)
(=0,000
(#U0005)
(=0.006)
(#00006)
(#0,005)
AutoCituo
(s0.000)
0.014
(30,000)
0.988
(10.000)
01.887
(10,000)
0.807
(10.000)
0.980
(10,000)
0.534
(s0.000)
0.875
(10.000)
0.479
N/A
N/A
(s0.000)
0.992
(10,002)
0:908
N/A
N/A
(10,000)
0.755
GPT:3.5
0.000
0.000
0,000
0.000
0.588
0.094
0.000
0.000
0.447
0.000
0.000
0.000
0.000
0:000
0.081
(20.000)
(50.000)
(10,0000
(10.000)
(20.535)
(10.209)
(20.000)
(10.000)
(c0.251)
(10,000)
(z0.000)
(10.000)
(20.000)
(40.000)
(10.071)
GPT-4
0.169
0,545
0.196
0.390
0.285
0.220
0.471
0,000
0.000
0.186
0.199
0.217
(+0.379)
(+0,409)
(+8:000)
(+0.295)
(+0.534)
(+0.261)
(+0450)
(+0.247)
(+0.364)
(+0.000)
(=0.000)
(+00000)
(+0.347)
(+0.378)
(+0.257)
DS-Agent
0.305
0.735
0.000
0,500
0.766
0,525
0.000
0:431
0.504
0.000
0.000
0,000
0:474
0.393
0.351
(#0.119)
(20.411)
(20,000)
(30.131)
(20.324)
(=0.00)
(40.000)
(=0.000)
(40,000)
(=0443)
(20,360)
(20.206)
AntoML-Agent
0.945
0.971
0.803
01.87
0.664
0.465
0.521
0.760
0.937
0.592
0.804
(s0.020)
(s0.043)
(10.007)
(s0.006)
(10.019)
(10.174)
(s0.003)
(s0.020)
(=0.038)
(10.021)
(:0.0003
(10.093)
(s0.000)
(10.015)
(10.035)
Constratin- Auvare
GPT-3.5
(20.000)
0.000
(50.386)
0.173
(20.000)
(20.000)
0.000
(20.139)
0.196
(10.476)
0.475
(=0.488)
(+0.181)
(=0.309)
0.338
(10,000)
0.000
(=0.000)
(20,000)
(20.000)
0.000
(40.000)
0:000
(10.163)
0.116
0.157
0.335
0.064
0.155
0.000
0.091
0.463
0.000
0.000
0.637
0.564
0.241
(+0.350)
(40,463)
(+0.44))
(44)
(+0.347)
(+0.000)
(+0.405)
(+0.204)
(00.300)
(+0.000)
(=0.000)
(+0.000)
(+0.356)
(+0.318)
(+0.234)
DS-Agentt
(=04511)
0.330
(#0.485)
0.353
(20.0000)
0.000
0.205
(#0.454)
0.776
(#0.214)
0.383
(=0
0,173
(#0.409)
0.183
(=0000))
0.505
(40.000)
0,000
0.000
(#0,209)
0.093
0.319
(#0.365)
0:403
(#0.264)
0.266
AgooML-Agoot
0.926
0.982
0.796
1..67
0.573
0.473
0.582
0.769
0.978
0.843
I.632
0,810
(20.015)
(30.022)
(10.002)
(L0.027)
(10.002)
(10.142)
(s0.002)
(20.020)
(z0.012)
(10.010)
(z0.028)
(10.001)
(20.034)
(10.037)
(10.027)
Table 8. Performance comparison with the CS metric. Best results are highlighted in bold.
Image Classilication Butterlly
Classilcation
Tabular Classification
Tabular Regression Crab
Tabular Clastering Student
Time-Series Forecasting
Node Classiiiction Cora
Shopee
Ecomm
Entail
Banuna
Software
Crop
Smoker
Weather
Electricity
Citeseer
Avg.
Countaiat-Fes
GPT-3.5
0.000
0,000
0.000
0.000
01:43
0.097
0.000
0.000
0.424
0.000
0,000
0,000
0.000
0,000
0.069
(20,000)
(20,000)
(20.00)
(20.000)
(20.216)
(20.00)
(=0.000)
(=0.237)
(+0.00))
(20.000)
(20.000)
(20.000)
(20.061)
GPT-4
(s0.413)
0.185
(10.523)
0,573
(10.000)
0.000
(10.413)
0.298
(10.541)
(10.322)
(80.499)
0.364
(s0.397)
(c0.325)
0.536
(10.000)
0.000
(10.000)
(s0.000)
(10:417)
0.293
(10.420)
0.299
(10,305)
0.266
DS-Agent
0.352
0.768
0.000
0,600
0.783
0.661
0.000
0.616
0,202
0.000
0,000
0.000
0,537
0.496
0.394
(+0.483)
(40.420)
(+0.00))
(+0.353)
(+0.438)
(40.172)
(=00000)
(-0361)
[=0.111)
(+0000)
(40.000)
(+0.000)
(+0.490)
(+0,253)
(+0.235)
AutoML-Agem
0.962
0,973
0.985
0
0
0,832
0.929
0.732
0.761
0,880
0.998
0.969
0,915
0.79
0,902
(+0,010)
(+0.004)
(s0.00))
(-0.010)
[+0.019)
(+0.010)
(+0.017)
GPT:3.5
0.000
0.111
0.000
0.000
0.123
0,312
0.228
0.066
0.244
0.000
0.000
0.000
0.000
0,000
0.077
(=0,000)
(#0.229)
(80.000)
(20.000)
(+0.227)
(#0.302)
(60.312)
(s0.147)
(+0.00))
(+0.000)
(=0.000)
(40.000)
(20.000)
(+0,108)
GPT-4
(50.343)
0.153
(10.475)
0.543
(10.411)
0.198
(10.296)
0.132
(10.339)
0.151
(10.000)
0.000
(30.394)
0.685
(10.214)
0.096
(20.270)
0.432
(10.000)
0.000
(10.000)
0,000
(10.000)
(10.317)
0,518
(10.319)
0.532
(10,254)
0.231
DS-Aggen
0.315
0.000
0.516
0.186
0.000
0.427
0.267
(10.431)
(10.185)
(10.000)
(10.312)
(20.385)
(10.332)
(20.417)
(20. 128)
(20.137)
(10.000)
(10.00
(20.272)
(20.367)
(10.390)
(10.283)
AutoML
0,863
0.986
0,991
0.898
0,858
0.786
0.930
0.611
0.666
0.760
0.941
0.969
0,796
0.691
0.841
(+0.063)
(+0.011)
(+0.001)
(+0.013)
(+0.001)
(+0.071)
(+0,01))
(-0.021)
(+0.005)
(40.126)
(+0.001)
(+0.318)
(+0.026)
Table 9, Results of ablation study on different variations.
Plan Decomposition
Multi-Step Verification
Image Classification
Text
Tabular Classification
Time-Series Forecasting
Node Classification
RAP
Average
Classification
Success Rate
1.000
0.000
0.000
0.000
1.000
0.400
1.000
1.000
1.000
0.000
1.000
0.800
1.000
1.000
1.000
1.000
1.000
1.000
Normalized Performance Score
0.929
0.000
0.000
0.000
0.734
0.333
0.928
0.982
0.975
0.000
0.748
0.727
0.936
0.971
1.000
0.991
0.812
0.942
Comprehensive Score
0.965
0.000
0.000
0.000
0.867
0.366
0.964
0.991
0.988
0.000
0.874
0.763
/
0.968
0.986
1.000
0.996
0.906
0.971
46

---

AutoMI-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 10. Comparison between the different numbers of plans.
Image Classification
Text Classification
Tabular Classification
Time-Series Forecasting
Node Classification
Number of Plans
Average
Success Rate
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1.000
1.000
1.000
1.000
5
1.000
1.000
1.000
1.000
1.000
1.000
Normalized Performance Score
0.831
0.966
0.958
0.998
0.800
0.911
0.936
0.971
1.000
0.999
0.812
0.944
0.916
0.964
0.973
0.998
0.805
0.931
Comprehensive Score
0.916
0.983
0.979
0.999
0.900
0.955
3
0.968
0.986
1.000
0.999
0.906
0.972
5
0.958
0.982
0.986
0.999
0.903
0.966
Table Time and monetary costs averaged across different tasks and datasets single run under constraint-fres and constraint-
settings.
Cost
Prompt Parsing
Request Verifleation
Retrieval & Planning
Plan Exocution
Execution Verification
Selection and Summarization
Code Generation
Total
Coustrabut-Piee
Money ($)
NIA 10.78
0.00
0.02
0,14
0.00
0.06
0.04 182.60
0.27
Time (s)
1.91
187.71
136.34
1.04
17.88
538.25
Constraint-Awwee
Money (5)
NIA
0.00
0.00 182.38
0.11
0.00
0.15 20.25
0.06
0.32
Time
14.21
3,63
98.62
1.37
191,90
$12.35
Table 12, Comparison of search time and normalized performance score between MCTS brr'S--baedda SELA and our AutoML-Agent,
Search Time (seconds) +
Normalized Performance Score
Dataset
SELA (MCTS)
AutoMI.- (Ours)
SELA (MCTS)
AutoMLL-Aeen (Ours)
Binary Classification
smoker-status
2736.78
206.48
0.785
0.762
click-prediction-ssaal
2227.85
256.24
0.238
0.352
Multi-Class Classification
mfeat-factors
1304.40
219.45
0.957
0.940
wine-quality-white
2372.77
206.41
0.650
0.652
Regression
colleges
674.29
232.03
0.876
0.878
house-prices
2906.95
378.85
0.090
0.090
Average
2037.18
249.91
0.599
0.612
47

---

AutoMI-AAeen: A Multi-Agent LLM Framework for Full-Pipeline AutoMI.
Table 13. Comparison of leaderboard quantile and task-specific performance between Agent K. and our AutoML-Agemt (1 indicates lower spee-ipc performance better)
Learderboard Quantile
Task-Specific Performance
Competition ID
Agent
Agent K (Derive from Rank)
AutoMM--Aeen
AutoML-Agent
(From Figure 8)
No Medal
restaurant-revenue prediction
predcccttnnnnnnnniiiiiiooooo (L)
2279272.777-222862222277 331.167--31.173
1859766.392
playgroun--series-s3llll (1)
88~-89
91
330.141
Bronze
hpp1000-l-challenge
75~-66
0.993~~0,994
0.720
bogsssseaassrddn--eeebeelttiiioo (1)
93~99
82
0.054-00005
0.079
Silver
nipsci
89~900
89
0.809~0.810
0.808
home-data dateedddt-foor for--o-oo---ooooee ml-course (L)
98~-99
91
13187.602--13193.955
14869. 145
Gold
world-championshipp 2023-embro--deeeee classiffication
90~-1
100
0.571~0.551
0,609
99~~100
0.977~00.978
0.971
48


Reviews:
[{"review_id": "TaPt1DaWhp", "reviewer": "Reviewer_mnKA", "summary": "In this paper, the authors propose a novel multi-agent framework tailored for full-pipeline AutoML, including initialization, planning, and execution, incorporating  5 types of agents, such as Agent Manager, Prompt Agent, Data Agent, Model Agent, and Operation Agent. Results on 14 tasks and 5 baselines demonstrate the stronger performance of the proposed models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments on the novelty, motivation, and empirical rigor of our work. We are glad to address your concerns below.\n\n> Method complexity\n\n**R1** Thank you for pointing this out. In the final version, which permits an extra page, we will **move key details currently located in the appendix, such as pseudocode summaries, into the main text**. We hope these changes will make the framework easier to understand for a broader audience.\n\n> Is each agent important? How do these workers contribute to the final score?\n\n**R2** Yes, each agent is essential. Without even one of them, the entire framework would not function properly, considering the current implementation. This is due to a tight coupling between each agent, integrated tool use, and its corresponding step in the pipeline at the code level for stable interoperability. As a result, removing any single agent would cause the system to fail at runtime.\n\nSpecifically, the *Prompt Agent* ensures that user instructions are correctly interpreted and converted into structured input, enabling reliable downstream planning and execution. The *Data Agent* performs crucial data-related tasks that inform model search with domain-aware characteristics, directly enhancing model quality. The *Model Agent* drives performance by executing training-free model search, hyperparameter tuning, and profiling to select optimal candidates, thereby raising the normalized performance score. The *Operation Agent* ensures that high-performing models are correctly implemented and executable, completing the pipeline with valid, deployable code—vital for success rate metrics. Finally, the *Agent Manager* orchestrates the entire process, ensuring that all steps execute in the correct sequence and that each agent’s output is validated before proceeding, thereby enforcing the correctness of the entire pipeline.\n\n\n> What are the errors or detailed analyses of these workers?\n\n**R3** In this submission, we provide intermediate outputs at each stage ($\\S$E) to illustrate what each agent produces—for example, the parsed JSON from the Prompt Agent and outputs from the Data and Model Agents.\n\nBuilding on this, we will expand our analysis to include typical failure modes. For instance, the Data Agent may mishandle uncommon data formats (e.g., failing to impute missing values or referencing incorrect column names), while the Model Agent may select suboptimal models under constrained search spaces. **Our multi-stage verification process detects and corrects most of these issues.** As shown in $\\S4.3$, certain variants fail to produce runnable code without specific modules, whereas the full system identifies and fixes such errors via feedback. The Agent Manager flags plans that violate user constraints (e.g., low accuracy), while implementation verification catches runtime errors (e.g., incomplete code or incorrect references) from the Operation Agent and requests fixes. \n\nNotably, we also encountered consistent challenges with smaller or less capable models. As discussed in $\\S$B.4, models like LLaMA-2-7B, Qwen1.5-14B, and even GPT3.5 often failed at complex planning or executable code generation. Common issues included incomplete code, altered comments without proper continuation, or unchanged code outputs. These patterns align with prior findings in DS-Agent and Data Interpreter, suggesting such limitations are systemic to smaller models rather than specific to our framework. We aim to reassure the reviewer that we have carefully studied “how it makes errors” and that the final results are reliable.\n\n\n> How did you design the prompt for each agent?\n\n**R4** We deliberately design prompts to optimize agent behavior with specific instructional steps based on role:\n- Creative or interpretive agents (Agent Manager, Prompt Agent) benefit from low-pressure, neutral framing to avoid overconfidence. The Prompt Agent adopts a neutral “assistant project manager” persona to support coordination and interpretation.\n- Execution-oriented agents (Data, Model, Operation Agents) respond better to strong, authoritative personas that promote precision and task adherence. The Data Agent is framed as “the world’s best data scientist” to encourage confident, detailed analysis and emphasize responsibilities.\n\nThese specialized prompts guide the LLM toward more accurate outputs compared to using a single, general-purpose prompt. Persona-driven system prompts reliably shape agent behavior. As suggested by `Reviewer L7kx`, we tested *five* prompt variations for each agent, differing in tone and task specificity. Due to space limitations, the experimental results with example outputs can be found at `/example_plans/prompt_sensitivity.md` in our anonymous GitHub repository. Overall, agents are not highly sensitive to exact phrasing **as long as their roles were clearly defined**, which is also reinforced through the user prompts—making the framework robust to variations in system prompts.", "claims_and_evidence": "The introduction of the paper is well-written and the challenges of this topic are well-motivated. The authors aim to propose an Auto-ML multiagent system that can take care of both data and model aspects and try to solve the challenges of planning and accurate implementation.", "methods_and_evaluation_criteria": "Although the method is complex and not easy to understand the full details. The author provides clear figures and an appendix that helps the read er to understand it.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are sound because of sufficient datasets, baselines, and experiments. The biggest concern is that the ablation study is not sufficient enough. For instance, is each agent important? How do these workers contribute to the final score? What are the errors or detailed analyses of these workers? Also, how did you design the prompt for each agent? Why not use others? Since the system is rather complex, a very careful analysis is needed to make it clear how it works and how it makes errors.", "relation_to_broader_scientific_literature": "Auto-ML domain. The full-stack Auto-ML using LLM is the main related work of this paper.", "essential_references_not_discussed": "Generally sufficient.", "other_strengths_and_weaknesses": "The paper is well-written and well-motivated.", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "km7x0dplBd", "reviewer": "Reviewer_Son5", "summary": "The paper introduces AutoML-Agent, a multi-agent LLM framework that automates the full AutoML pipeline from data retrieval to model deployment. Unlike prior approaches that focus on specific pipeline components (e.g., hyperparameter optimization or feature engineering), AutoML-Agent leverages retrieval-augmented planning (RAP) and multi-stage verification to ensure correctness, efficiency, and adaptability.\nThe paper presents a modular, multi-agent architecture where specialized agents handle data processing,model selection, hyperparameter tuning, and deployment. The retrieval agumented planning strategy enhances exploration while multi stage verification ensures the correctness of the generated code. Experiments across seven ML tasks and fourteen datasets show that AutoML-Agent outperforms AutoGluon, DS-Agent, GPT-3.5, GPT-4 and human baselines in terms of success rate, normaluzed performance score, and comprehensive score. Additionally, AutoML-Agent is eight times faster than search-based methods like SELA while maintaining compettive performance.\n\n## update after rebuttal\nI thank the authors for the additional clarifications and stay with my evaluation that I consider this paper to be an accept for the conference.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our contributions and for your thoughtful, detailed feedback. Below, we respond to your specific concerns.\n\n> Scalability concerns\n\n**R1** This is an important point. A key motivation behind AutoML-Agent is to reduce the computational overhead of the search process. Our framework achieves this by relying solely on LLM inferences for agent communication and leveraging retrieval-augmented knowledge for planning, avoiding expensive training-feedback loops.\n\n`Figure 4c` and `Table 11` demonstrate the scalability of our approach. AutoML-Agent has a time usage standard deviation of only **1 minute**, compared to SELA’s **14 minutes**, across datasets ranging **from ~1K to 143K** instances. As a training-free method, AutoML-Agent maintains stable search time, whereas training-based methods like SELA exhibit fluctuations depending on dataset and model size. This suggests our framework scales efficiently without performance degradation. \n\nMore precisely, our computational complexity is approximately $O(1 + m)$, compared to $O(s + m)$ for training-based methods, where $s$ denotes the search-time training feedback and $m$ the model training time before deployment.\n\n\n> Lack of a statistical significance test\n\n**R2** Currently, `Tables 5`, `6`, and `7` report the results with standard deviations over *five* independent runs. While we did not include formal statistical tests, reporting standard deviations is a common practice to convey the reliability of performance differences. Although we believe this is sufficient, please let us know if there is a specific test you would like to see.\n\n> Comparisons with TabPFN\n\n**R3** TabPFN is indeed a highly relevant and efficient method for tabular classification. We would like to clarify that TabPFN is *already included* in our baselines under the **Human Models** category ($\\S$C.3). For your convenience, we summarize the results below.\n| **Models**                    | **Banana** | **Software** |\n| ----------------------------- | ---------- | ------------ |\n| Human Models (**TabPFN**)     | 0.976      | 0.669        |\n| AutoGluon                     | 0.980      | 0.524        |\n| GPT-3.5                       | 0.587      | 0.094        |\n| GPT-4                         | 0.390      | 0.285        |\n| DS-Agent                      | 0.766      | 0.523        |\n| ***AutoML-Agent***            | 0.987      | 0.664        |\n\n\n> Lack of real-world deployment testing\n\n**R4** We acknowledge the importance of real-world deployment testing. To this end, we designed our framework with a deployment stage via a Gradio API, showing a proof-of-concept using benchmark datasets. While full deployment in production environments involves additional considerations (e.g., infrastructure, latency, data privacy, and so on), we view this as a promising direction for future work.\n\n> Extension to RL tasks\n\n**R5** AutoML-Agent’s modularity allows for extension to RL tasks. While our current implementation focuses on (un-)supervised pipelines, the framework is not fundamentally limited to these settings. Extending AutoML-Agent to RL would require incorporating domain-specific modules—for example, agents for environment interaction, action space design, and reward handling (see https://www.automl.org/autorl-survey). As noted in the paper, tasks such as RL or RecSys would benefit from additional agents to manage actor-environment interactions and reward modeling.\n\nThanks to its modular design, one could integrate an “RL agent” that interfaces with an environment (e.g., OpenAI Gym), while the planner generates RL-specific steps such as policy training and evaluation. Although this extension is certainly feasible, it could be substantial enough that an AutoML-Agent for RL—perhaps termed AutoRL-Agent—would merit contributions worthy of a separate paper.\n\n> What are the main failure cases of AutoML-Agent?\n\n**R6** Please kindly refer to `Reviewer mnKA-R3`. \n\n> Edge cases in code generation\n\n**R7** We appreciate the reviewer’s concern regarding LLM hallucinations in code generation. Our framework is specifically designed to mitigate this issue through two core mechanisms: RAP and multi-stage verification.\n- RAP grounds the planning agent’s decisions in real-world information by retrieving external resources, rather than relying solely on the LLM’s internal memory, which can produce hallucinated content. This ensures that generated plan steps are more likely to be valid and based on established approaches.\n- Our multi-stage verification process acts as a safety net. If the LLM produces incorrect or nonsensical code, the implementation stage will flag it, triggering a corrective attempt.\n\nThis dual mechanism—prevention through retrieval and correction through verification—has proven effective (please refer to `Reviewer EaoR-R5` for robustness experiments on noisy information). To further reduce risk, we provide agents with structured pipeline skeletons, which help constrain generation.", "claims_and_evidence": "AutoML-Agent automates the entire ML pipeline frm data retrieval to deployment: The paper provides an end-to-end framework design with clear descriptions of agents handline each stage (data, model, deployment). Experiemnts show successful deployment-ready models.\n\nThe retrievel-augmented planning strategy imprvoes search efficiency: The ablation study shows that RAP + plan decomposition outperforms naive planning, confirming its effectiveness.\n\nThe multi-stage verification improves code correctness: Experimental results indicate that verification prevents failueres, reducing deployment errors.\n\nAutoML-Agent is more efficient than searhc-based AutoML methods like SELA: AutoML-Agent achievs similar performance in the experiments while being eight times faster.", "methods_and_evaluation_criteria": "The proposed evaluation metrics make sense, as they measure both pipelines correctness and downstream model performance. The benchmark datasets are diverse, ensuring sufficient generalizability. However, scalability testing on large datasets is missing, which could impact real-world applicability.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Experiments span seven different ML tasks, making the results generalizable in general.\n\nThe ablation study properly isolates effects of RAP and multi-stage verification.\n\nThe hyperparaameter study analyzes how varying the number of plans affects performance.\n\nHowever, scalability on larlge datasets is not evaluated and the study lacks a statistical significance test in the comparisons with the baselines.", "relation_to_broader_scientific_literature": "The paper covers traditional AutoML techniques such as AutoGluon, AutoSklearn, TPOT. It also discssues LLM-based AutoML methods and it highlights how AutoML-Agent improves upon past work by incorporating multi-agent collaboration and RAP. However, TabPFN as another very recent AutoML tool is missing in the comparisons.", "essential_references_not_discussed": "Hollmann, Noah, et al. \"TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second.\" The Eleventh International Conference on Learning Representations.\n\nHollmann, Noah, et al. \"Accurate predictions on small data with a tabular foundation model.\" Nature 637.8045 (2025): 319-326.", "other_strengths_and_weaknesses": "# Strengths\n- The multi-agent design is a novel approach to AutoML\n- The full-pipeline AutoML capability is impactful for democratizing ML\n- The paper is well-structured and well-written.\n\n# Weaknesses\n- Lack of real-world deployment testing.", "other_comments_or_suggestions": "The authors should test AutoML-Agent on large-scale datasets and compare to TabPFN on the small scale datasets of tabular data.", "questions_for_authors": "- How does AutoML-Agent scale on large datasets? Would it remain efficient?\n- Can AutoML-Agent be extended to reinforcement learning tasks?\n- What are the main failure cases of AutoML-Agent?\n- How does the framework handle edge cases in code generation? Do hallucinations still occur?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QLvgVq5DQR", "reviewer": "Reviewer_EaoR", "summary": "The paper introduces AutoML-Agent, a novel multi-agent framework leveraging large language models (LLMs) to automate the entire AutoML pipeline, including data retrieval, preprocessing, model selection, hyperparameter optimization, and deployment. The proposed framework employs a retrieval-augmented planning strategy, specialized agents for decomposing and executing tasks, and multi-stage verification processes to enhance reliability and efficiency. Extensive experimental evaluation demonstrates that AutoML-Agent consistently outperforms various baselines across multiple datasets and tasks, achieving higher success rates, superior downstream task performance, and better computational efficiency.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and encouraging review. We greatly appreciate your positive assessment of our paper’s novelty, empirical rigor, and technical soundness. Below, we address your thoughtful concerns.\n\n> High computational cost\n\n**R1** We would like to clarify that, apart from the final implementation verification step, AutoML-Agent relies solely on inference throughout the entire search process, including planning. Unlike many prior works, this design ensures efficiency, as runtime does not scale with dataset size or model complexity. Our modular design also enables parallel execution of sub-tasks, further reducing wall-clock time. That said, while computational demands remain low (as discussed in `Reviewer Son5-R1`), using high-performing LLMs for each agent may still incur monetary costs (as noted in `Reviewer mnKA-R3`).\n\n\n> Dependence on external APIs and models\n\n**R2** We agree that relying on a proprietary model like GPT-4o entails cost and reliability implications. However, our framework is model-agnostic—agents can operate with any sufficiently capable LLM. We used GPT-4o to demonstrate state-of-the-art performance, but open-source or local models can be substituted to eliminate external API dependencies, albeit with some performance trade-offs. Given the rapid advancements in LLMs, we expect costs to decrease and become increasingly justified by the resulting performance gains.\n\n> Computational and monetary cost implications in practical deployments\n\n**R3** In practical deployments, the computational and monetary costs of running AutoML-Agent primarily depend on the number of planning iterations (typically 3–5), the complexity of the downstream task (e.g., data modality, task type), and the size and access method of the LLM (e.g., API vs. self-hosted). After pipeline generation, the cost shifts to standard training and inference of the downstream models, which depends on the specific components selected.\n\n\n> Explicit discussion of limitations and potential failure modes\n\n**R4** We have revised the limitations section to include a more explicit discussion of potential failure modes beyond the evaluated tasks. For specific failure cases, please also refer to our response to `Reviewer mnKA-R3`.\n\n\n> Robustness to noisy or outdated information\n\n**R5** Our design does incorporate measures to be robust to bad information. During RAP, the agent cross-verifies information by retrieving from multiple sources against user's requirements before deriving insight knowledge, and the multi-stage verification will catch issues if a retrieved piece of instruction leads to an error (since the code won’t execute or will produce a poor result, prompting a fix). As the inclusion of outdated information is highly unlikely due to retrieval constraints, we focused our evaluation on robustness to noisy information. We simulated two scenarios:\n- Pre-Summary Injection: Injecting unrelated or low-quality examples *before* insight extraction and planning.\n- Post-Summary Injection: Injecting noisy examples *after* insight extraction but before planning, i.e., mixing noisy inputs with the useful insights.\n\nGiven the user requirements, these noises are generated by an extra agent (i.e., adversarial agent) prompted to create 'unhelpful' and 'fake' insights that hinder the planning process.\n| **Dataset**            | **AutoML-Agent** | **w/ Pre-Injection** | **w/ Post-Injection** |\n| ---------------------- | ---------------- | -------------------- | --------------------- |\n| smoker-status          | 0.762            | 0.768                | 0.853                 |\n| click-prediction-small | 0.352            | 0.347                | 0.133                 |\n| mfeat-factors          | 0.940            | 0.930                | 0.915                 |\n| wine-quality-white     | 0.652            | 0.615                | 0.670                 |\n| colleges               | 0.878            | 0.658                | 0.896                 |\n| house-prices           | 0.090            | 0.089                | 0.087                 |\n| **_Average_**          | 0.612            | 0.568                | 0.593                 |\n\nOur results show that AutoML-Agent is robust to such noise. Its built-in error correction and multi-stage verification mechanisms significantly mitigate the impact of noisy inputs, ensuring that the final model performance **remains largely unaffected**. Thanks to this suggestion, we observed that noise injection can even lead to better performance in particular cases. We conjecture that this may be because the Agent Manager is implicitly forced to further distinguish between useful and non-useful information.\n\nDue to the space limit, the generated plans can be found at `/example_plans/plan_with_noises.md` in the anonymous GitHub.\n\n> Adaptability to other ML tasks (e.g., RL or RecSys)\n\n**R6** Please kindly refer to `Reviewer Son5-R5`.", "claims_and_evidence": "The authors' claims regarding the superiority of AutoML-Agent in handling the full AutoML pipeline and achieving higher efficiency and performance are convincingly supported by robust empirical evidence. Results from experiments on seven diverse tasks and fourteen datasets show clear performance improvements over existing state-of-the-art AutoML frameworks and LLM-based agents, with success rates and performance metrics (accuracy, RMSLE, F1-score) explicitly presented. The ablation studies effectively support the claims regarding the importance of the retrieval-augmented planning and multi-stage verification mechanisms.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense and align with current practices in the field of AutoML. The comprehensive experimental setup, including diverse data modalities (image, text, tabular, graph, and time series data), and clearly defined metrics (e.g., accuracy, RMSLE, F1-score, Rand index) offer a fair and thorough assessment of the framework’s capabilities. The multi-agent architecture, retrieval-augmented planning, and multi-stage verification approaches are logically well-structured and suitable for tackling complex, full-pipeline AutoML challenges.", "theoretical_claims": "The paper does not make specific theoretical claims requiring formal proofs.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and rigorous. The comparison against robust baselines (e.g., AutoGluon, DS-Agent, SELA, GPT-3.5, GPT-4) under both constraint-free and constraint-aware scenarios effectively validates the proposed framework. The detailed ablation studies further substantiate the significance of each component, such as retrieval-augmented planning and multi-stage verification.", "relation_to_broader_scientific_literature": "The paper appropriately situates its contributions within the broader literature, effectively referencing previous AutoML frameworks (AutoGluon, AutoML-GPT, SELA), LLM-driven frameworks (DS-Agent, HuggingGPT), and planning methodologies (retrieval-augmented planning, multi-stage verification). Its innovative integration of retrieval-augmented strategies and structured task decomposition clearly addresses existing limitations in the efficiency and generalizability of prior approaches.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Potential high computational cost associated with iterative planning and multi-agent verification processes.\n\nDependence on external APIs and models (GPT-4), which could impact cost.", "other_comments_or_suggestions": "- Clarifying the computational and monetary cost implications more explicitly in practical deployments could strengthen the paper’s discussion.\n\n- A more explicit discussion of limitations and potential failure modes beyond provided tasks could further improve the presentation.", "questions_for_authors": "Have you evaluated the robustness of your system when external knowledge sources or APIs provide noisy or outdated information?\n\nCan you elaborate on your framework’s adaptability to ML tasks significantly differing from those tested (e.g., reinforcement learning or recommendation systems)? How might AutoML-Agent be extended to accommodate these areas?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fAhErZhDT5", "reviewer": "Reviewer_L7kx", "summary": "&nbsp;\n\nThe authors introduce a multi-agent LLM framework for full pipeline AutoML. The authors perform an extensive empirical study against relevant baselines, demonstrating that their AutoML-Agent outperforms both frontier models and partial pipeline language agents. The authors' experiments are informative, showcasing that for simpler, tabular tasks, partial pipeline agents remain competitive. The paper is well-written and the authors release an extensive codebase for their agent. As such I recommend acceptance with the potential to increase my score if the points below are addressed.\n\n&nbsp;\n\n## **Post-Rebuttal Update**\n\n&nbsp;\n\nI have upgraded my score to 5 post-rebuttal follow the authors' extensive additional experiments and clarifications during the author-reviewer discussion phase. I am strongly in favor of accepting the paper and wish to express to the AC my interest in championing the paper for consideration as a spotlight/oral.\n\n&nbsp;", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and constructive review and appreciate your recognition of our paper’s strengths. Below, we address your specific comments.\n\n> Relationships to Magentic-One [7] and Agent K [2]\n\n**R1** Thank you for pointing out these valuable concurrent papers that we had previously missed. Although our Agent Manager functions similarly to [7], it performs constraint-aware plan selection and multi-stage verification, making it more task-specific and grounded.\n\nFor Agent K [2], the current version is not open-sourced, which limits direct comparison. Nevertheless, we provide a qualitative analysis. Although Agent K also leverages LLMs to orchestrate ML pipelines, it is tailored for Kaggle competition settings and relies on a training-based approach with high search overhead. In contrast, AutoML-Agent introduces a platform-agnostic, RAP and multi-stage verification framework designed for broader AutoML applications beyond Kaggle with a *training-free* search method.\n\nWe have clarified these connections and distinctions in Related Work, including those in [1–8].\n \n> Sensitivity to prompts and rationale for system prompts\n\n**R2** While agent-specific prompt design is not the primary focus of this paper, for **user prompts**, our results suggest that AutoML-Agent is minimally sensitive to prompt specificity. In both *constraint-free (**somewhat vague**)* and *constraint-aware (**precise**)* settings, the system achieves high success rates with comparable average NPS (0.804 vs. 0.810), indicating robustness to prompt variation. Clearer prompts generally lead to better outcomes, but the system is designed to handle vague instructions gracefully: the combination of the verification mechanism and RAP allows the system to tolerate a range of prompt qualities. Even when the Prompt Agent interprets vague inputs broadly, execution verification ensures that only valid and high-performing solutions are accepted.\n\nFor **system prompts**, please refer to `Reviewer mnKA-R4`. \n\n> Restrictive skeleton Python scripts\n\n**R3** The provided skeleton serves as a default scaffold for typical ML tasks, offering a high-level structure without enforcing task- or data-specific code. It enables agents to complete the template using *theoretically any* specialized modules by following the TODO list. We believe this approach is significantly more flexible than the data-specific templates used in the DS-Agent framework. Typically, the only component that requires adjustment for new tasks is the evaluation metric.\n\n> Instruction fine-tuning in Prompt Agent\n\n**R4** For entirely new problem classes, we could extend the existing model through additional fine-tuning or by providing a few in-context examples. EvoInstruct can be used to generate instruction–response pairs in the new domain to support this adaptation.\n\nTo provide some context, instruction fine-tuning was initially necessary to ensure precise parsing of user inputs for interoperability between agents, as this work began before OpenAI supported accurate structured outputs. With recent updates to GPT models, we can now directly use a JSON schema, eliminating the need for fine-tuning in many cases. The Prompt Agent can accurately parse user queries by referencing the schema explicitly. The schema is available at `/prompt_agent/schema.json`.\n\n> Potential use of PaperQA [9]\n\n**R5** We agree that a tool like [9] could improve the literature retrieval component. Incorporating PaperQA-style scientific querying could enhance citation relevance and support more context-aware prompt generation, ultimately leading to better planning decisions. We will acknowledge this direction in the paper and consider it for future iterations.\n\n> In-context examples for Retrieval-Augmented Planning\n\n**R6** The in-context examples (`plan_knowledge`) are **automatically generated on the fly**. They are *extracted*, *summarized*, and *organized* from raw retrieved documents and texts (e.g., search results, arXiv papers, Kaggle notebooks) before being passed to the Agent Manager for planning. We have updated the GitHub repository to include examples at `/example_plans/plan_knowledge.md`.\n\n> \"if no dataset is found...\".\n\n **R7** When we mention relying on the LLM’s inherent knowledge, we are referring to a fallback mechanism intended to produce the most plausible output given limited context. However, in the absence of actual data, the pipeline will ultimately fail during final implementation verification due to runtime errors. Our approach assumes that a dataset is either provided (e.g., image classification) or retrievable via search (e.g., node classification). If no dataset is found, the agent will prompt the user to supply one when in interactive mode. We will revise the text to clarify this assumption.\n \n> Writing suggestions\n\n**R8** We have addressed all the mentioned issues, including a table with standardized details (see anonymous GitHub). This will be included in the final version.", "claims_and_evidence": "&nbsp;\n\nThe claims made by the authors are empirical in nature. In terms of reproducibility, the authors provide an anonymous GitHub link to their codebase which is well documented.\n\n&nbsp;", "methods_and_evaluation_criteria": "&nbsp;\n\n1. It would be worth discussing the relationship between the current approach and the Orchestrator-Workers paradigm of [7]. It seems as though the Agent Manager functions as an orchestrator.\n\n2. How does AutoML-Agent compare against Agent K [2] on Kaggle-style problems? This seems like it would be a relevant comparison given that the bespoke AutoML baselines in the current paper are chosen based on their performance in Kaggle notebooks.\n\n&nbsp;", "theoretical_claims": "&nbsp;\n\nNot applicable.\n\n&nbsp;", "experimental_designs_or_analyses": "&nbsp;\n\n1. What is the sensitivity of AutoML-Agent's performance to the user’s prompt with task description, requirements, and/or constraints? It may be worth designing a sensitivity analysis by having e.g. 3 levels of clarity in the user prompt ranging from vague to precise instructions assuming they pass the request verification stage.\n\n2. What is the sensitivity to the system prompts used by the various agents?\n\n&nbsp;", "relation_to_broader_scientific_literature": "&nbsp;\n\nIt may be worth referencing [4] in relation to the use of LLMs for HPO. It may also be worth discussing the relationship. Between AutoML-Agent and the works of [1,3,5,6,7,8]. Of particular note is [2] which tackles Kaggle competitions and by its nature will need to perform aspects of AutoML.\n\n&nbsp;", "essential_references_not_discussed": "&nbsp;\n\nBelow I list the references I have cited elsewhere in the review.\n\n&nbsp;\n\n**__REFERENCES__**\n\n&nbsp;\n\n[1] Narayanan, S., Braza, J.D., Griffiths, R.R., Ponnapati, M., Bou, A., Laurent, J., Kabeli, O., Wellawatte, G., Cox, S., Rodriques, S.G. and White, A.D., 2024. [Aviary: training language agents on challenging scientific tasks.](https://arxiv.org/abs/2412.21154) arXiv preprint arXiv:2412.21154.\n\n[2] Grosnit, A., Maraval, A., Doran, J., Paolo, G., Thomas, A., Beevi, R.S.H.N., Gonzalez, J., Khandelwal, K., Iacobacci, I., Benechehab, A. and Cherkaoui, H., 2024. [Large language models orchestrating structured reasoning achieve Kaggle grandmaster level.](https://arxiv.org/abs/2411.03562) arXiv preprint arXiv:2411.03562.\n\n[3] Tang, X., Liu, Y., Cai, Z., Shao, Y., Lu, J., Zhang, Y., Deng, Z., Hu, H., An, K., Huang, R. and Si, S., 2023. [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code.](https://arxiv.org/abs/2311.09835) arXiv preprint arXiv:2311.09835.\n\n[4] Liu, T., Astorga, N., Seedat, N. and van der Schaar, M., [Large Language Models to Enhance Bayesian Optimization.](https://openreview.net/forum?id=OOxotBmGol) In The Twelfth International Conference on Learning Representations 2024.\n\n[5] Tang, J., Fan, T. and Huang, C., 2025. [AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agents.](https://arxiv.org/abs/2502.05957) arXiv e-prints, pp.arXiv-2502.\n\n[6] Kon, P.T.J., Liu, J., Ding, Q., Qiu, Y., Yang, Z., Huang, Y., Srinivasa, J., Lee, M., Chowdhury, M. and Chen, A., 2025. [Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents.](https://arxiv.org/abs/2502.16069) arXiv preprint arXiv:2502.16069.\n\n[7] Fourney, A., Bansal, G., Mozannar, H., Tan, C., Salinas, E., Niedtner, F., Proebsting, G., Bassman, G., Gerrits, J., Alber, J. and Chang, P., 2024. [Magentic-One: A generalist multi-agent system for solving complex tasks.](https://arxiv.org/abs/2411.04468) arXiv preprint arXiv:2411.04468.\n\n[8] Hu, X., Zhao, Z., Wei, S., Chai, Z., Ma, Q., Wang, G., Wang, X., Su, J., Xu, J., Zhu, M. and Cheng, Y., 2024, July. [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks.](https://proceedings.mlr.press/v235/hu24s.html) In International Conference on Machine Learning (pp. 19544-19572). PMLR.\n\n[9] Lála, J., O'Donoghue, O., Shtedritski, A., Cox, S., Rodriques, S.G. and White, A.D., 2023. [PaperQA: Retrieval-augmented generative agent for scientific research.](https://arxiv.org/abs/2312.07559) arXiv preprint arXiv:2312.07559.\n\n&nbsp;", "other_strengths_and_weaknesses": "&nbsp;\n\nOverall, I think the paper could be strengthened by performing a sensitivity analysis on the various agent prompts as well as comparing against Agent K [2] on external Kaggle datasets. Otherwise I believe the contribution is still solid.\n\n&nbsp;", "other_comments_or_suggestions": "&nbsp;\n\n1. There are some missing capitalizations in the references e.g. \"GPT-4\", \"LLM\", \"AutoML\", \"AI\".\n\n2. In the introduction, no parentheses around \"e.g. natural language and computer vision)\n\n3. The link to the codebase was quite difficult to find (I believe it is only given in Section C of the appendix?). It may be worth moving this to the abstract or introduction.\n\n4. It would be worth running GPT or Claude over the codebase code to write documentation.\n\n5. It would be worth spelling out the acronym RMSLE as root mean square log error when it is first introduced in the main paper.\n\n6. Line 183, typo, there is a space that should be inserted. Same on line 370.\n\n&nbsp;", "questions_for_authors": "&nbsp;\n\n1. The prompt agent requires instruction fine-tuning. How is this instruction fine-tuning expected to generalize to new problem classes not considered in the paper? Will the prompt agent LLM need to be instruction-tuned from scratch? Is EvoInstruct expected to be effective in generating instruction, response pairs for all AutoML problems?\n\n2. What is the reason for the gulf in the level of encouragement for the various agents in the system prompt e.g. prompt agent receives the underwhelming, \"You are an assistant project manager in the AutoML development team.\" Whereas data agent receives the hyperbolic, \"You are the world’s best data scientist of an automated machine learning project\".\n\n3. For the arXiv paper search component in Appendix D, what do the authors think of the potential for a tool such as PaperQA [9] for improving performance in this step?\n\n4. The prompt for Retrieval-Augmented Planning contains in-context examples. How were these generated? Are they contained in the GitHub repo? If so, it is difficult to find them.\n\n5. In Section 3.5, the authors state, \"if no dataset is found, we rely on the inherent knowledge of the LLM\". Presumably this does not result in a successful outcome if there is no data present?\n\n&nbsp;", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "&nbsp;\n\nNot applicable.\n\n&nbsp;"}}]

Author rebuttal:

Meta-review: In this submission, the authors present their multi-agent system tailored for AutoML, AutoML-Agent, which can deliver deployment-ready models according to user's task descriptions. Extensive experiments on several downstream tasks also confirm its benefits and great potentials.

During the discussion, all the reviewers appreciate this work. And please incorporate the new experiments and analysis (such as the comparison against other approaches and the analysis of prompt sensitivity) into the final version. 

Hope the authors find the discussions with reviewers useful and make this submission a better one.
Decision: Accept (poster)