MLZero: A Multi-Agent System for End-to-end Machine Learning Automation
Authors: Haoyang Fang
Boran Han
Nick Erickson
Xiyuan Zhang
Su Zhou
Anirudh Dagar
Jiani Zhang
Ali Caner Turkmen
Cuixiong Hu
Huzefa Rangwala
Ying Nian Wu
Bernie Wang
George Karypis
Venue: NeurIPS 2025 poster ()
Keywords: AutoML
Large Language Models
Multi-Agent Systems
End-to-End Machine Learning
Multimodal Learning
Code Generation
Primary area: deep_learning
TL;DR: MLZero automates end-to-end multimodal machine learning with high success rate and solution quality without human intervention.

Abstract: Existing AutoML systems have advanced the automation of machine learning (ML); however, they still require substantial manual configuration and expert input, particularly when handling multimodal data. We introduce MLZero, a novel multi-agent framework powered by Large Language Models (LLMs) that enables end-to-end ML automation across diverse data modalities with minimal human intervention. A cognitive perception module is first employed, transforming raw multimodal inputs into perceptual context that effectively guides the subsequent workflow. To address key limitations of LLMs, such as hallucinated code generation and outdated API knowledge, we enhance the iterative code generation process with semantic and episodic memory. MLZero demonstrates superior performance on MLE-Bench Lite, outperforming all competitors in both success rate and solution quality, securing six gold medals. Furthermore, when evaluated on our Multimodal AutoML Agent Benchmark, which includes 25 more challenging tasks spanning diverse data modalities, MLZero outperforms the competing methods by a large margin with a success rate of 0.92 (+263.6\%) and an average rank of 2.28. Our approach maintains its robust effectiveness even with a compact 8B LLM, outperforming full-size systems from existing solutions.

MLZero: A Multi-Agent System for End-to-end Machine Learning Automation
Haoyang Fang, Boran Han, Nick Erickson, Xiyuan Zhang, Su Zhou, Anirudh Dagar, Jiani Zhang, Ali Caner Turkmen, Cuixiong Hu, Huzefa Rangwala, Ying Nian Wu, Bernie Wang, George Karypis
Amazon Web Services
(haoyfang, boranhan, nww.ice,, suzhou, ani com {zhajiani, atturkm, tonyhu, rhuzefa, wunyin, yuyawang, gkarypis)
Website: http:://///pooolttaaaaoooooooo www.errrr..ittuu...... 10/
GitHub: http://ggtcabb...oommttttoouoo// autogluon
-osagggsstnnnnnnann assistant
Abstract
Existing AutoML systems have advanced the automation of machine learning (ML); however, they still require substantial manual configuration and expert put, particularly when handling multimodal data. We introduce MI.Zero, novel multi-agent framework powered by Large Language Models (LLMs) that enables end-to-end ML automation across diverse data modalities with minimal human intervention. A cognitive perception module is first employed, transforming raw multimodal inputs into perceptual context that effectively guides the subsequent workflow. To address key limitations of LLMs, such as hallucinated code gen- eration and outdated API knowledge, enhance the iterative code generation process with semantic and episodic memory. ML.Zer demonstrates superior per- formance MLE-Bench Lite, outperforming all competitors in both success rate and solution quality, securing six gold medals. Additionally, when evaluated on our Multimodal AutoMI Agent Benchmark, which includes 25 more challenging tasks spanning diverse data modalities, MI.Zero outperforms the competing methods by large margin with success rate of 0.92 (+263.6%) and an average rank of 2.28. Our approach maintains its robust effectiveness even with a compact 8B LLM, outperforming full-size systems from existing solutions.
1 Introduction
The quest to democratize machine learning has long been championed by AutoML, a field dedicated automating the intricate process of building ML solutions 21 20 19
65 28 However, conventional AutoML systems often struggle to manage the heterogeneity of data formats and modalities, requiring predefined workflows extensive manual tuning by domain experts. Despite advances in automating specific components, such as feature engineering [13] 53), neural architecture
67
search (39 [8], and hyperparameter optimization (75), such systems still fall short of providing an end-to-end solution that spans the entire machine learning lifecycle, from data preprocessing model building. These challenges leave a critical gap for truly autonomous and versatile systems.
The emergence of LLMs brings new opportunities to enhance the flexibility of AutoML. Early attempts leverage LLMs for AutoML targeted specific pain points, such LLM-guided feature engineering 22 82]. hyperparameter optimization 47 neural architecture search [35 55)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Polter with Dhttasrts
Percsption
Minurry
Reade-to-use Model
Fight
Labrutading Librrry Sclenke
Agel
Hisurry Krror ML Librasics Contenniion
Aexmriebs Aput
File sniter
Upticoal Eicc Instruction
N
Fli Perspris
Hieration Geoeralions
Feodbacks
Prediction
Use training data fonnat for outputs
type
Lodoo
MI. Cade Execution
Fireer
Refinn Loge & metric
Query
MLZero (Zom Itreas Baservationn
Output
Figure I: MLZero: An end-to-end multi-agent system that integrates specialized perception agents with dual memory modules (semantic and episodic) to power iterative coding cycles, transforming raw data into ready-to-use to- models and prediction outputs with zero human intervention.
and data preprocessing (88.. Despite advances, these approaches remain limited by their focus on automating individual components rather than providing truly end-to-end multimodal automation.
With the advent of agentic AI, recent research has explored the broader potential of the LLM agent system for comprehensive AutoML solutions [10,
45 70 24), leveraging their demonstrated strengths in planning, code generation 36 31, and verification processes [8 37 However, the
38
success rate of completing complex ML tasks remains hindered by two major challenges: (1) Current systems exhibit a lack of automated data perception, forcing reliance on hard-coded preprocessing logic or rigid data formats (24] 33 45 (2) LLMs, when relying solely on their internal parametric knowledge, tend to produce simplistic solutions that fall short for intricate MI. tasks [8). Even with access to advanced MI. libraries and domain knowledge, their performance deteriorates when managing long, convoluted knowledge due to the absence of efficient memory mechanisms.
To address these challenges, we introduce MLZero, novel multi-agent system designed specifically for end-to-end automation of multimodal ML tasks. As illustrated in Figure our approach begins with specialized agents performing comprehensive perception actions to analyze data structures, inter- pret task requirements, and select the proper ML library, thus establishing a rich perception context for subsequent operations. Building upon the perception context, the system leverages the semantic memory with condensed, curated knowledge about the library to guide iterative code development, while simultaneously utilizing episodic memory for efficient error detection and correction. The dual memory modules enable powerful learning capabilities S3 allowing the system to retain domain knowledge persistently while adaptively learning from past errors, thereby continuously refining its solution through iterations. By seamlessly integrating the reasoning capabilities of LLMs with automated perception and structured memories, MLZero not only overcomes the inherent limitations of previous (LM-based approaches, but also represents a truly end-to-end system that consistently demonstrates superior performance across diverse machine learning tasks. Our main contributions are as follows:
We introduce MI.Zero, novel multi-agent system that delivers high quality end-to-end multi- modal machine learning solutions with minimal human intervention. Our system automates the entire workflow through the perception module, dual memory modules (semantic and episodic), and iterative code generation, coordinated across nine specialized agents. Our primary contri- bution lies in the synergistic integration of these components into hierarchical multi-agent
system specifically designed for end-to-end AutoML on raw, multimodal data.
MLZero demonstrates superior performance on the existing benchmark, MLE-Bench Lite, with
more medal counts (6 gold medal) and higher success rate.
We benchmark MLZero with comprehensive benchmark suite, the Multimodal AutoML Agent Benchmark, comprising diverse datasets that incorporate challenging scenarios including multilingual, multitable, multilahel, zero-shot inference tasks with large unprocessed data. Our comprehensive empirical evidence from the constructed multimodal benchmark showing that MI.Zero outperforms existing ML agents across all evaluated metrics, such success rate (+263.6% %, ). average rank, and time complexity.
Through detailed ablation studies, we identify key components driving these performance gains and analyze failure cases to demonstrate both the common failures MLZero overcomes and opportunities for future research directions.

---

2 Related Work
2.1 LLM Agents for ML
Beyond component-specifc applications, several frameworks have emerged for end-to-ent LLM- based ML automation [30, 33 23
45 38). MLAgenBBench [33] introduces research agent that conducts ML experiments by accessing files and executing code. DS-Agent [24] implements case-based reasoning to leverage human expertise, but requires extensive dataset-code pairs during development and struggles with unfamiliar domains. Both approaches lack effective perception capabilities and require code skeleton samples, limiting their end-to-end functionality. AutoKag- gle 45 offers a multi-agent system with iterative development and testing, but is restricted to tabular data with hard-coded input formats. AIDE
24
code optimization problem with tree search in the solution space 14 46) but underperforms without sufficient external knowledge about ML Libraries. Despite these advances, challenges remain in developing truly end-to-end ML automations that operate on raw data with output specifications and minimal human intervention. Key limitations include maintaining consistency across complex workflows, utilizing memory efficiently, adapting to evolving ML libraries, etc. For more comprehensive evaluation, we by prompting them with selected outputs
338 frames ML engineering
deliberately strengthen the existing approaches 38 from MLZero (Appendix C).
Meanwhile, MLE-bench 110 offers valuable but methodologically limited agent-human comparisons, constrained by its reliance on preprocessed rather than raw data and outdated human baselines collected under different computational and algorithmic contexts. These limitations motivate new benchmark's 's focus on direct agent-to-an agent-to-agent tt-geent comparisons for end-to-end ML automation.
2.2 Tool Use
Recent advances in LLM-based tool-use agents have significantly expanded Al capabilities. Tool- former 633
pioneered an approach where LLMs learn API usage through self-supervision with minimal demonstrations. TooILLM [60] scaled this concept by developing comprehensive frame- work supporting over 16,000 real-world APIs. Chameleon 44 introduced compositional reasoning with plug-and-play approach for multimodal QA tasks. OctoTooos [48) further introduced training- free agentic framework with standardized tool cards, hierarchical planning, and efficient execution components. Other notable contributions include ToolDoc [32), which demonstrated that API docu- mentation can effectively replace demonstrations, and ToolkenGPT (26) with tool embeddings to avoid fine-tuning, MM-REACT [76] extended these capabilities to visual reasoning tasks, while HuggingGPT (66] demonstrated effective orchestration of multiple models.
While these systems have made significant progress, they mainly focus on tasks with short context lengths. Considering the autoML tasks often require perceiving a large amount of data, our work extends beyond this paradigm by developing a multi-agent approach specifically designed for the more complex and interconnected tasks inherent in ML automations.
2.3
Code Generation with LLMs
Large language models have demonstrated increasingly impressive capabilities in code generation and debugging. Models such AlphaCode [4]], Codex [11], and StarCoder 443 have shown competitive
programming performance through effective solution generation and self-verification across multiple programming languages. State-of-the-ae LLMs B 59
22 [6] have also achieved remarkable results on code benchmarks [12]]. Codex CLI enables terminal-based Al coding assistance with secure sandboxed execution. Despite these advances, significant challenges remain producing code that is semantically aligned with complex user intent, particularly for MI. challenges that require specialized library knowledge and intricate data perception and handling. Our work leverages these fundamental code generation capabilities while addressing their limitations through specialized agents and modular designs with ML library integration specifically tailored for ML tasks.
6
3 Methodology
We present MI.Zero (Figure 2. multi-agent system F that automates end-to-end solutions for multimodal ML tasks. The system processes input data and optional user inputs Upgt to produce
3

---

Outrtes FE filed
Data Contect
Frner Contest
Pereepual Contest
Row Data
Grouping
Detth filen
Perception
Sclanal ML. Librar
Retrieval
VNlLAW Coetaroy avettetint
Taak Cusical
Percepussl Contest
Documenss
car
User lepui
Task Perception
Summarization
Coriest
Comtino
Condensation
X ML Library Selection
Selected MI Library
(Offiin))
Initalization
Ganerstion
Stap
Sclcoual MI Library
Sup til
Skyp tri
Episndie Menory
Quipuz
39 Percegiion
Perceptisl Content
Stop
Step #til
Semantic
Poceptial
Stept
Resh Serjht
Pereeptzal
Erver Analyzer
Costest
Perceptani Canieel
Colling Pronup
Pythou
Valled
Cool
Frme Content
Selected
Blah Serpt
ML-Lenyy
Coder
Lag Emes
Suggented Fix
Ocher Phrmpt L0O
Step 1+3
Condenurd
Tunotall
Figure 2: MLZero: A multi-agent system for end-to-end multimodal ML automation with zero human interventions. During the initialization phase, the perception module selects the appropriate ML library and generates perceptual context to initialize semantic and episodic memory. In the subsequent generation phase, the system performs code generation, execution, and debugging iteratively with the assistance of perceptual context, semantic memory, and episodic memory until successful output achieved, The surrounding panels detail the four key modules: (1) Perception (upper left) in Section 3.1. [1: (2) Semantic Memory (upper right) in Section 3.2 (3) Episodic Memory (lower right) in
Section 3.3 and (4) Iterative Coding (lower left) in Section 3.4
solutions including predicted outputs y, code artifacts C, and execution logs L:
UTP) = (y, C, L).
Note that our ML model building process for various tasks is achieved by generating code employing different ML. libraries and executing it. For supervised learning tasks, x typically includes labeled training data, unlabeled test data, and a brief task description or instruction. For zero-shot tasks, would simply consist unlabeled test data and the task description. For example, in a supervised semantic segmentation task as illustrated in Figure xr includes training images and the corresponding masks, a tabular file indicating the image-mask pairs, testing images, a tabular file listing the test images, and simple one-sentence task description. The output y would be the predicted masks for the test images, together with a tabular file that stores the test image-mast pairs. During the evaluation, calculate the task-specific objective C. prediction y and ground truth ÿ. that is, (v.
Our system comprises four modules, where each module is a subsystem with one or more agents, and each agent specialized LLM augmented with utility functions (details of each agent in Appendix B): (1) Perception that interprets arbitrary data inputs and transforms them into structured context; (2) Semantic Memory that enriches the system with knowledge of the MI Library; (3) Episodic Memory that maintains chronological execution records for targeted debugging; and (4) Iterative Coding that implements a refinement process with feedback loops and augmented memory.
3.1 Perception
The Perception module acts as the cognitive lens the system (Eq. 2 orchestrating the transfor- mation of various data inputs into actionable ML workflow specifications. Input is the raw data folder

---

and optional user input. The output is the perceptual context P and the selected ML library M: P(x, U = (P, M).
This module consists of three agents. File grouping and file perception agent (Appendix B.I.I performs structural analysis of raw data x, grouping similar files and interpreting file contents. Task perception agent (Appendix B.1.2) extracts semantic information from raw data, derived context, and user input Upps to identify objectives, constraints, and evaluation criteria, generating task context in natural language. MI Library selection agent (Appendix B.1.3) employs context-aware reasoning to match problem characteristics with the appropriate MI. Library M by analyzing the constructed task context against the library capabilities. This integrated approach efficiently translates user requirements and dataset properties into perceptual context P that guides subsequent iterative processes.
3.2 Semantic Memory
The Semantic Memory Module S, enhances the LLM's parametric knowledge with domain-specift information from external knowledge bases at each iteration t. These knowledge bases are constructed offline by two agents. The summarization agent (Appendix [B2.2) compresses relevant knowledge into concise paragraphs serving as queryable indices, while the condensation agent (Appendix [B.2. transforms this knowledge into precise and streamlined guidance. This agent- agent-based knowledge processing enables the Semantic Memory to incorporate documentation, API references, code examples, and other relevant resources.
At each iteration f, given the error context RG (both Error Summary and Suggested Fix), the Semantic Memory Module processes this information through its retrieval agent (Appendix B.2.3 to query the knowledge base of the selected ML library M, extracting condensed information Gy::
(,, R&) Gt-
In our case, we use tutorial documents to demonstrate the effectiveness of minimal tool initialization. By retrieving contextually relevant tutorials according the current task and errors encountered, the system provides targeted guidance for iterative coding, substantially improving code quality while reducing LLM hallucinations (Section 4.3.1
3.3 Episodic Memory
The Episodic Memory module, &,, enhances the success rate of MLZero in ML model building by providing error context RG at each iteration & leveraging its chronological record of the system execution history.
= R.
(4)
This component is initialized with the perception context P and progressively stores the interaction data each iteration. The stored information includes code artifacts Ct-1, code execution logs Lę-1, retrieved knowledge G.-1, and previous error context R₃-1- When invoked during code generation, the error analyzer agent (Appendix B distills encountered issues and contexts into concise error summaries paired with fix suggestions. This focused approach enables subsequent coding agents to ef- ficiently address specific problems without processing excessive contextual information, significantly reducing token consumption while maintaining problem-solving effectiveness (Section 4.3.2
3.4 Iterative Coding
With the support of components above, our system enters an iterative coding process 9,, where at each iteration refines the solution based on execution feedback:
(P..U Rr, G.) = (yr, Ct, Lt).
(5)
For each iteration f, the system first combines the perceptual context P, optional user input U error context RG, and the retrieved knowledge G2 to guide the coder agent (Appendix [B.4 producing executable code Cr, leveraging parametric, historical, and external knowledge simultaneously. The
PPeriiteration user input is disabled in all experiments show the performance of zero human intervention.
5

---

system then executes the generated code in a configured environment, which the coder can further customize or recreate as needed. During execution, the system captures logs L; including standard output, standard error, log messages, etc., and stores the model output yt- The executer agent (Appendix (.4.2) analyzes these results and logs to determine the next steps: finalizing output y, C, upon success identifying errors and initiating the next coding iteration + 1.
This iterative approach continues until either successful execution or a maximum iteration limit is reached. Notably, the system supports optional per-iteration user input, allowing for human guidance when required while maintaining high degree of automation. Through this comprehensive system, MLZero effectively bridges the gap between noisy raw data inputs and sophisticated ML solutions, providing truly end-to-end automated ML framework adaptive to any modalities.
4
Experiments
To evaluate the effectiveness of MLZero against state-of-the-art ML and coding agents, we conducted extensive experiments across multiple benchmarks and datasets. We first assess performance on MLE- bench Lite [10] with 21 diverse Kaggle competitions, then proceeded to the Multimodal AutoML Agent Benchmark for end-to-end evaluation across 25 diverse datasets spanning various modalities and ML. tasks. We evaluate the performance using multiple metries including success rate, average rank, relative time consumption, and solution quality. Additionally, we performed ablation studies to quantify the contribution of individual components within proposed system. Finally, we conducted a detailed error analysis to identify and categorize failure cases across sesh-perforance methods, providing insights into the robustness and limitations of each approach, followed by efficiency and robustness analysis examining token consumption, cost effectiivnness, and robustness across different LLM backbones and under various noise conditions.
4.1 Implementation Details
Each agent was assigned a 3-hour time limit per dataset to produce results. By default, MLZero uses Claude 3.7 Sonnet as its underlying LLM. It is important note that only MLZero and Codex CLI operate truly end-to-end, while other agents require varying degrees of preprocessing or postprocessing to function on our benchmark. For example, DS-Ageet requires manual code execution, while results from AIDE and AutoKaggle needed manual extraction from working directories. Complete implementation specifications, including model configurations, the prompts used, and other details for each agent, are provided in the Appendix C
4.2
Main Results
4.2.1
MLEbench
We first evaluate MLZero on MLE-bench Lite [10]. consisting of 21 diverse challenges including classification, regression, and generation, spanning various data modalities including image, text, tabular, and audio (more details: Appendix A.2 The results are shown in Figure MI.Zro
demonstrates superior performance on MLEbench-Liit with an average rank of 1.43, significantly outperforming competing approaches including AIDE [8] (2.36), ResearchAgent (MLAB) [33] (3.29), and CodeActAgent (OpenHands) (72 (2.93). Furthermore, MLZero achieves the highest success rate of 86%, compared to AIDE (81 %), MLAB (62%), and OpenHands (71%). This performance advantage extends across multiple evaluation metrics, where our approach achieves six gold and two silver medals, while also outperforming competitors in medal counts. Many tasks in MLEBench Lite cannot be handled by the current integrated ML libraries, thus registering more specific ML libraries, e.g. diffusion or audio models, to MLZero can further push the performance.
While MLE-Bende 10 provides valuable comparisons between ML agents and human performance in Kaggle competitions, relies on dedicated Python scripts to preprocess the raw data, presenting structured rather than raw inputs. This dataset-specific preprocessing significantly simplifies the ML workflow and may overestimate agent capabilities in real-world scenarios. Therefore, to enable more accurate agent-to-agent comparisons in more rigorous end-to-end MI automation, we further tested MI.Zero on our Multimodal AutoMI. Agent Benchmark, which specifically challenges agents to handle completely raw, unprocessed data without any task-specific preprocessing assistance.

---

4.2.2 Multimodal Multimodal C AutoML Agent Benchmark
Table 1: Performance comparioon across ML agents and "ablation configurations on Multimodal AutoML Agent Benchmark, Colorful icons indicate data modality Tx
:
text.
&tabular, image, Classithen Ilassffication
document) and problem type BC binary classification,
RG re-
La RT retrieval, e Multi Label Classification))
gression. PC forecasting.
T(L) indicates higher(lower)) is better. 'Dagger columns are not agents, including MLZene ss optimal Righerrtien
S semantic segmentation,
perforr
A perffommnce
peenttiee open -
performance with current ML libraries to compare with human performance repo in recent sourced literature. Bold numbers indicates best ML. agent performance with comparable default configurations. For each dataset, lataset, results reported as mean,.. format across three independel runs. r A coomon
Configuration desccriptioon def: default settings of each agent, 8B": using LLama 3.1 8B [22], -ext": configeuuati
without external knowledge, -epi": without episodic memory, +rea": : with reasoning LLM (58), Performante
: with external knowledge. Performance e
metrics (Appendix A. 1.13) are computed across three runs: Avg. Rank: average position when ranking agents by their mean valid performance on each dataset, Rel. Time: relative execution time compared to MLZero (def), Success: percentage of
successiul runs. Detailed results for each run and each agent are shown in Appendix E.
Agent
End-to-End
End-to-Ead
End-to-End
MLZero (ours)
Codex CLI
AIDE
DSe-Agent
AK
importted
Human
Dataset
def
Sn
ext
epl'
24hrs
erea"
def
*ext"
zero- shot
def Reported
abalone
o1 on
2.09
2.19.oz
2.13
2.08
2:23
2:27 col
2.16 or
2.18 01
2.24 co 2.36 oo
x
H
1.42 0z
1.75.00
o:
1.30
X
on
0.76 0o
1.14.s:
0.76.00
0.79
x
X
4.68
X
0.76
solar(10m)
1-19 P8
x
1:29
0.18
1.29 c0
1.05 (0)
yolanda
8.53 8.54
8.93.o3
8.53.00
8.18
9.43
X
8.79
airbob
0.43 00
D. 12
0.24.
0.42
0.45
0.39
0.39
0.39
0.31
0.32
sirlines
0.66 00
0.63.o:
0.66.00
0.66
0.63 c2
0.63 01
0.61.co
bio
0.81
0.80.m
0.83
0.80
0.82
0.79
0.85
0.87
0.84
0.79
0.79
camosee
0.84 00
0.16.00
0.81.0
0.85
x
X
X
X
0.89
ed18
0.46 00.21
0.15
-0.94
ss-44.s-1.44.11
-0.05
11.94
no.0.64.
-1.84...
elimate
0.48
X 0.24.o:
0.48 no
0.48
X 0.20 cm.
X
x
X
covertype
0.98 0o
p0 0.88.o:
0.98.00
0.98
0.96.oc
0.92.co
0.88 op
0.86
os
0.96
co 0.95.co
0,94.00
flood
0.69 0.09
0.60
0.68
0.80
0.44
0.71
0.68
0.58
0.79
liqa G
w
0.50 01
x
0.22.00
0.46..s
0.48
0,20 cc
X
x
X
0.78
gead10
0.86 04
0.88.
0.83
0.89
0.85.co
0.90
0.58
0.80
0.11.00 oo
ham10000
0.63 0.57
0.67,00
0.67.
0.56
0.18
0.47
0.81
0.81
X
0.61
hateful
0.59 01 0.57 01
0.35.at
0.50 m
0.61
0.48.co c
0.51 os
0.49 as
0.36 mz
0.60
1sic2017
0.75
0.76
0.11
X
0.78
funding
0.45 0z 0.41.os 0.36.ss
0.44.00
0.51
0.34 o4
X
0.44
0,24.no
0.61
memotion
0.50
0.83.
0.51
0.56
0.53 on 0.76
0.47
midnn
0.95 no 0.96 oo
0.94
0.95 on
0.97
0.32 3a
0.82 cm
0.96
0.94 01
0.96
x
petfinder
0.39 0.40
0.38.o2
0.39
0.41
0.40
0.34 nn
0.38 03
0.36
0.27
0.39
0.41
rondseg
0.47 0o
0.31
0.
0.47
x
X
x
x
0.62
rvledip
0.87 00
0.89.00
0.87
0.92
X
clothing
0.75 0.61.
0.66.
0.72
0.77
0.35
x 0.75
0.35
0.74
Avg. Rank]
2.42
5.14
4.94
2.86
N/A
8.04
5.76
6.16
6.02
8.26
8.12
8.28
N/A
Rel. Time.
1.0
3,17
2.32
1.03
N/A
0.15
0.23
2.83
2.48
N/A
N/A
4.82
N/A
Successt
92.0%
45.3%
69.3%
86.7%
N/A
14.7%
69.3%
25.3%
45.3%
13.3%
20.0%
14.7%
N/A
Table Dprropo providdes detailed performance comparison between MLZeero and several state-of-the oh-the-art machine learning and coding agents on our Multimodal AutoML Agent Benchmark (further details in Appendices ] and A.T). Evaluated across 25 diverse datasets, MLZero demonstrates a significant Aevantddan For each dataset, we conduct three independent runs and report the mean performance and standard deviation of valid submissions. We further evaluate the agents based on average rank, relative time consumption, and success rate (see Appendix A..1.3 for definition details).
MLZero with default configuration achieves a markedly higher success rate (92.0%) compared to all competing agents. Furthermore, MLZero consistently delivers solutions of superior quality, as e evidenced by its substantially lower average rank (2.42). In addition the primary evaluation. we test MLZero with a smaller LLM. The 8B configuration, which uses LLama 3.1 8B (2]. maintains maaemaaiiiin
better performance (45.3% success rate, 5. 5.14 rank average) than other agents despite a significantly reduced model size. This demonstrates the robustness of our multi-agent system design. In terms of r time complexity, MI.Zero is significantly faster than AIDE and AutoKaggle due to the more efficient
unnecessar, design reducing unnecessary Meeeessary iterations, fithough slower than Codex CLI, which excels in instruction following and success rate, but tends to give simple solutions that can executed in a few minutes with degraded performaneee performance. Additionally in the hour extended run-time experiment 24hrs, we
7

---

manually extend the time constraints and improve the quality preset [9 67 65 to show that MLZero can approach expert-level performance (Human Reported) given sufficient computational resources.
While Codex CLI shows limited performance with its default setting using a general-purpose LLM [56]. its performance improves substantially with the +rea configuration where it is equipped with state-of-the-art reasoning models [58]. Without step limitations and with terminal access, Codex CLI produces simple solutions that execute quickly and iterate efficiently, enabling easier error correction. This approach yields a high success rate and, consequently. a good ranking. However, its performance remains low on challenging datasets, revealing limitations in complex problem-sol
Considering that each baseline method has different designs regarding the use of external knowledge, conduct different augmentations ensure fair comparison and evaluate methods equal footing. To assess our method without the advantage of its integrated external knowledge, the -ext configuration removes all access to external MI. libraries. This configuration yields 69.3% success rate and a 4.94 average rank, still outperforming all competitors and highlighting the efficiency of MLZero's other components. We also investigate the contribution of episodic memory through the -epi configuration, which removes episodic memory but retains the LLM's s conversation history within the coder agent. This setup achieves an 86.7% success rate with a 2.86 average rank, demonstrating that while episodic memory provides significant benefits, maintaining a coherent conversational context still yields reasonable performance. In contrast, we explore modifying AIDE to access external knowledge, indicated as +ext. While it shows improvement, MLZero continued to outperform it under these comparable conditions. This result underscores that the superior performance of MLZero stems from its overall system design, not merely from the inclusion of episodic memory (epi) or external knowledge (ext) in isolation.
MLEBench-Lite Results
Perfermance Mettrie
Performance Matrics Across Different Offline Indexing Settings
#Silver+
#Gold
2 to
8.08
pP 9-4
Defestt
Summertcction Betting
No Conderastion
Performance Metrics Across Different Retrieval Counts
Bronze+
(+-64)
2414) Dower subari Rank
Performance Meerics
1.4
1.31
baseline tses nntroedd
Time Offccecy
1.2
1.
1.93 3.0
0.8
0.4
2.62
6:44
5
AIDE
Number Retrieved Documents
#Median+
Success
OpanHands
Figure 4: Ablation study for semantic memory: Im- pact on system performance and efficiency of dif- ferent offline indexing settings (top) and retrieval size (0, 3, 5, 10 documents) (bottom).
Figure 3: Comparing our agent with baselines on MLE-bench. Detailed results for each run and each agent shown in Appendix H.2
4.3 More Ablation Studies
To further investigate the effectiveness of an individual component in the overall system, we conduct ablation studies in eight diverse datasets that span tabular data, multimodal data, multilingual and multitable data, document classification, image segmentation, time series forecasting and text retrieval (detailed information available in Appendix A.3). A.3). For each ablation experiment, we compare modified configurations with our default system using three primary metrics: (1) success rate in generating valid solutions, (2) token efficiency, and (3) time efficiency. Both token efficiency and time efficiency metrics are calculated as the reciprocal of relative token usage and exccution time, respectively, measured only for valid solutions ensure fair comparisons.
4.3.1 More Ablation: Semantic Memory
To further assess our semantic memory module, we evaluated several configurations with varying degrees of functionality. The default configuration represents our complete system with semantic memory using condensation, summarization, and retrieved documents. w/o Condensation uses full tutorials without the condensation process, while w/o Summary performs retrieval based only
8

---

on tutorial titles, without summary-based indexing. We also evaluated Retrieval Size variants by varying the number of documents retrieved (0, 1, 1. 10).
Figure demonstrates that condensation effectively reduces token consumption while preserving performance. Additionally, summary- -ased retrieval surpasses title-based retrieval, highlighting the importance of content-aware indexing for efficient knowledge access. Figure 4 (bottom) reveals how varying retrieval quantities affects performance. While increasing the number of retrievals generally enhances results, benefits level off after five documents. The token and time usage increase nearly linearly with retrieval quantity, creating distinct trade-off between performance and computational efficiency. In the zero-retrieval configuration, only the library name is provided to subsequent agents, which then rely solely on the LLM's parametric knowledge of the library.
These findings validate our design choices for the semantic memory module, demonstrating that the combination of condensed tutorials, summary-basett retrieval, and appropriate retrieval size (5 documents) achieves an optimal balance between performance and resource utilization.
Table 2: Ablation study of episodic memory components. Results show to- ken efficiency and success rate across three configurations. The def'ault con- figuration with both eтror summaries and suggested fixes achieves optimal success rate while maintaining reason- able token efficiency.
Table Comparative error analysis of ML agents across different failure categories. Error types include perception, preprocessing, API hallucination, algorithm implementa- tion, and postprocessing phases.
Error Type
MI.Zero Codex CLI
AIDE
DS
+rea
Perception
0.0%
7.7%
2.1%
24.0%
Preprocessing
2.1%
0.0%
0.0%.
20.0%
Configuration
Token Eff.
Success
API Hallucination
0.0%
3.8%
28.2%
4.0%
Algorithm Impl.
4.3%
3.8%
4.3%
28.0%
def w/o Fix w/o Sum & Fix
1.0
1.0
Postprocessing
0.0%
11.5%
8.7%
0.0%
0.81
0.75
Unreported
N/A
N/A
8.7%
N/A
1.39
0.83
Overall
6.5%
26.9%
47.8%
76.0%
4.3.2 More Ablation: Episodic Memory
To evaluate the contribution of episodic memory, compared three configurations as shown in Table 2 The default configuration with both error summaries and suggested fixes achieves the best overall success rate (1.0). The w/o Sum & Fix configuration removes both components, prompting the subsequent coder with truncated error messages and executer's s analysis instead. shows token efficiency advantages the cost of decreased success rate (0.83). Interestingly, the w/o Fix configuration, retaining error summaries but removing suggested fixes, has the lowest success rate (0.75) with the worst token efficiency due to the increased number of iterations used figure out the solution, suggesting that providing over-condensed error summaries without actionable fix suggestions may further confuse the coder. These results indicate that complete episodic memory produces better outcomes, particularly for complex problems requiring multiple debugging cycles.
4.4 Failure Cases and Analysis
Among the cases that fail, we conduct an error analysis across MLZero, Codex CLI +rea, AIDE, and DS-Agent (DS), categorizing failures as shown in Table MLZero demonstrates remarkable robustness with minimal errors (6.5% overall in 46 datasets), while competing methods exhibit higher failure rates: Codex CLI (26.9%), AIDE (47.8%), and DSA (76.0%). Notably, MLZero completely eliminates several error categories, including perception, postprocessing, and API Hallucination. is important to note that error frequencies may be influenced by system progression, i.c. systems that never encounter later-stage errors that more capable
fail earlier (e.g., at rrreeption/prepooeesstoe systems might face when attempting more complex solutions. A detailed discussion of error patterns and their implications for future MI. agent design is provided in Appendix E.3
4.5 Efffciency and Robustness Analysis
To assess computational efficiency, we compared token consumption and cost across systems on the benchmark. As shown in Table 4 (left top), MLZero achieves superior performance (92% success

---

rate) while using 4.3x fewer tokens than AIDE (63k vs. 273k per dataset on average) and incurring only 23% of its cost (S0.19 vs. $0.82 per dataset). The w/o Fix configuration demonstrates further token reduction (51k) but at the expense of success rate.
We further evaluated robustness across different L.M backbones (Claude Sonnet 3.7, GPT-4.1, and Llama 3.1 8B) on the eight challenging datasets spanning all modalities (Table left bottom). Results show consistent performance across all models, with the compact 8B model still outperforming all baseline systems from Table Note that X indicates execution timeout or failure, and scores with indicate lower better for regression tasks.
To test resilience against noisy inputs, we conducted stress tests with deliberately misleading dataset descriptions the Abalone tabular regression task (Table right). The table shows different noise conditions: misleading information in the dataset description (Noise Des.), incorrect instructions from the user (Noise Ins.), and removal of the correct library from available options (-Lib). The results columns indicate whether the perception module correctly identified the task (Perc.) and whether the overall end-to-end task was completed successfully (Succ.). Even when provided incorrect task type hints or library suggestions, the perception module correctly identified task characteristics in all cases. When explicitly directed use incorrect tools via user instruction, the system successfully recovered through restart mechanisms, demonstrating robust error detection and correction capabilities.
Table 4: Efficiency, robustness, and stress test results. Left top: token efficiency and cost. Left bottom:
robustness across LLM backbones. Right: stress tests with misleading inputs.
System
Succ.
Tokens
Cost
Noise
Noise
-Lib Perc.
Suce.
Des.
Ins.
MI.Zero (def)
92%
63k
$0.19
"This is timeseries
MIZerr (w/o Fix)
75%
51k
$0.15
AIDE (+ext)
45.3%
273k
$0.82
'YOU SHOULD this as timeseries task'
"IMPORIANT: Use
x
Dataset
Sonnet 3.7
GPT-4.1
Llama 8B
TimeSeries MI. Library"
camoseg
0.84
0.82
X
"Try to use Flagndmeedding"
flood
0.69
0.64
0.69
liqa
0.50
0,50
0.50
"YOU SHOULD this as a timeseries task"
midoc
0.95
0.96
0.95
pettinder
0,39
0.40
"Try to use HugEmmeediing"
rvicdip
0.87
0.87
X
solar (10m))
1.49
X
None
yolanda).
8.53
8.53
8.54
Conclusion
In this paper, introduced MI.eero, novel hierarchical multi-agent system that reimagines multimodal ML. automation through the integration of multiple LLM agents with perception, semantic and episodic memory, and iterative coding mechanisms. Our comprehensive evaluation MLE- bench Lite demonstrates superior performance across 21 diverse Kaggle competitions. Further rigorous end-to-end evaluation datasets confirms that MI.eer significantly outperforms existing approaches with superior solution quality and +263% improvement success rate, while our perception module enables truly end-to- end automation without human intervention. This addresses the limitations of existing benchmarks and provides more accurate assessment of agent capabilities on raw, unprocessed data.
Multimodal AutoML Agent Benchmark across 25 diverse
We believe that this advancement facilitates the democratization sophisticated MI. methodologies, enabling individuals with limited domain expertise to address complex ML challenges effectively. The environmental impact of running LLMs warrants consideration, which presents the need for training smaller LLMs to achieve comparable performance. More limitations and future work can be found in Appendix
Broader Impact:
10

---

References
[1] Melbourne Airbnb Open Data.
belbourne-arrbbboopee--e Accessed: 2024-02-02.
[2]
PetFinder.my Adoption Prediction.
https://www.kagge/t www.kk...o..o/./................
petf
Accessed: 2024-02-02.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint w:Xiv:2303.08774. 2023.
[4] Abien Fred Agarap. Statistical analysis on e-commerce reviews, with sentiment classification using bidirectional recurrent neural network (rnn). arXiv preprint arXiv: 110..800500..... 2018.
[5] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language time series. arXiv preprint arXiv:2403.07815, 2024.
[6] Anthropic. Claude 3.7 sonnet.
http://www.trrop........... com/ claude 2025.
[7] Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Doc- former: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993-1003, 2021.
[8] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0777: 2021.
[9] Jock Blackard. Covertype. doi.org/10.24433//C5
UCI Machine Learning Repository, 1998.
DOI: https://-
[10] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint ar 1xXi:22007.03374, 2021.
[13] Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Mare Aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems, 35:32053-32063 2022.
[14] Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang. Jacky Kwok, Ceyao Zhang, et al. Sela: Tree-search enhanced Ilm agents for automated machine learning. arXiv preprint arXiv:2410. 20..2410.777533 17238, 2024.
[15] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo. Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168-172. IEEE, 2018.
[16] DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning, 2025.
11

---

[17] Thomas Diggelmann, Jordan Boyd- Goyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate-fever. A dataset for verification of real-world climate claims. arXiv preprint UrXiv:20/2.000 2020.
[18] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Leurning Research, 20(55):1-21, 2019.
[19] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabuse ttablarr: Robust and accurate automl for structured data, arXiv preprint arXiv:2003.06500 2020.
[20] Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning, Journal of Machine Learning Research, 23(261):1-61, 2022.
[21] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. Advances in neural information processing systems, 28, 2015.
[22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21720 2024.
[23] Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio Iacobacci, Abdelhakim Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level. arXiv preprint arXiv:2411.033555 2024.
[24] Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. arXiv preprint arXiv:2402. 17453, 2024.
[25] Isabelle Guyon, Lisheng Sun-Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag, et al. Analysis of the automl challenge series. Automated Machine Learning, 177:177-201 219, 2019.
[26] Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 01145570-45894 2023.
[27] Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference on Document Analysis and Recognition (ICDAR).
[28] Xin Hc, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Knowledge- based systems, 212: 106622, 2021.
[29] Noah Hollmann, Samuel Müller, and Frank Hutter. Caafe: Combining large language models with tabular predietors for semi-automated data science. In /st Workshop on the Synergy of Scientific and Machine Learning Modeling @ ICML.2023, 2023.
[30] Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An mmm agent for data science. arXiv preprint 2402 18679, 2024.
[31] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo. John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review. ACM Transactions Software Engineering and Methodology, 1378 2024.
[32] Cheng- Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen- Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308.0061 00675, 2023.
12

---

[33] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2023.
[34] Jan Huizinga, Hans De Moel, and Wojciech Szewezyk. Global flood depth-damage functions: Methodology and the database with guidelines. Technical report, Joint Research Centre (Seville site), 2017.
[35] Ganesh Jawahar, Muhammad Abdul-Mageed, Laks VS Lakshmanan, and Dujian Ding. Llm performance predictors are good initializers for architecture search. arXiv preprint arXiv:2310. 16712, 2023.
[36] Juyong Jiang. Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.0055: 2024.
[37] Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology, 33(7):1-30, 2024.
[38] Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Ja- cenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code. arXiv preprint WxXiv22502.13138 2025.
[39] Haifeng Jin, François Chollet, Qingquan Song, and Xia Hu. Autokeras: An automl library for deep learning. Journal of machine Learning research, 24(6))1-6, 2023.
[40] Israel Campero Jurado, Picter Gijsbers, and Joaquin Vanschoren. Automl benchmark with shorter time constraints and early stopping. arXiv preprint arXiv:2504.01222, 2025.
[41] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes, 2021.
[42] Trung-Nghia Le, Tam Nguyen, Zhongliang Nie, Minh-Triet Tran, and Akihiro Sugimoto. Anabranch network for camouflaged object segmentation. Computer vision and image under- standing, 184:45-56, 2019.
[43] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.0611 2023.
[44] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eecles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-Ievel code generation with alphacode. Science, 378(6624):1092-10009
2022.
[45] Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang. Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint 24X12:2410.20424, 2024.
[46] Zujie Liang, Feng Wei, Wujiang Xu, Lin Chen, Yuxi Qian, and Xinhui Wu. I-mets: Enhancing agentic automl via introspective monte carlo tree search. arXiv preprint 166,, 2025.
[47] Siyi Liu, Chen Gao, and Yong Li. Large language model agent for arXiv preprinl arXiv:2402.01888, 2024.
optimization.
[48] Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octo- tools: An agentic framework with extensible tools for complex reasoning. arXiv preprint arXiv:2502.1 1271, 2025.
[49] Pan Lu, Baolin Peng. Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: aud-annd compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36:43447-43477 2023.
13

---

[50] Shreyash Mishra, S Suryavardan, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, Asif Ekbal, and Srijan Kumar. Memotion 3: Dataset on sentiment and emotion analysis of codemixed hindi-english memes, 2023.
[51] Volodymyr Mnih. Machine learning for aerial image labeling. University of Toronto (Canada), 2013.
[52] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nits Reimers. Mteb: Massive text embedding benchmark. arXiv preprint 1xXiv:22100077:: 2022.
[53] Alhassan Mumuni and Fuseini Mumuni. Automated data processing and feature engineering for deep learning and big data applications: a survey. Journal of Information and Intelligence, 2024.
[54] Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1994. DOI:
[55] Muhammad Umair Nasir, Sam Farle, Julian Togelius, Steven James, and Christopher Cleghorn. Llmatic: neural architecture search via large language models and quality diversity optimization. In proceedings of the Genetic and Evolutionary Computation Conference, pages 1110-1118, 2024.
[56] OpenAl. Introducing gpt-4.1 in the api, April 2025. Accessed: 2025-05-02.
[57] OpenAl. Openai codex cli: Lightweight coding agent that runs in your terminal. https 7/github. com/ opt/a//coderrye/e/ April 2025, Command-line tool for local, chat-driven devel- opment with OpenAl models.
[58] OpenAl. Openai o3 and 04-mini system card, April 2025, Accessed: 2025-05-02.
[59] OpenAl. Openai 03-mini. htt.pe//ppppasss.ccc//////////pp 2025, Ac- cessed: 2025-03-25.
[60] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toollim: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 2023.
[61] David Salinas and Nick Erickson. Tabrepo: large scale repository of tabular model evaluations and its automl applications. arXiv preprint arXiv:231 1.02971, 2023.
[62] Dietmar Schabus, Marcin Skowron, and Martin Trapp. One million posts: A data set of german online discussions. In Proceedings the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1241-1244, Tokyo, Japan, August 2017.
[63] Timo Schick, Jane Dwivedi-Yuu Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language mod- els can teach themselves to use tools. Advances Neural Information Processing Systems, 36:68539-68551, 2023.
[64] Holger Schwenk and Xian Li. A corpus for multilingual document classification in eight languages. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France, may 2018. European Language Resources Association (ELRA).
[65] Oleksandr Shehur, Ali Caner Turkmen, Nick Erickson, Huibin Shen, Alexander Shirkov, Tony Hu, and Bernie Wang. Autogluon-timcseric timeseries: Automl for probabilistic time series forecasting. In International Conference on Automated Machine Learning, pages 1 PMLR, 2023.
[66] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends hugging face. Advances Neural Information Processing Systems, 36:38154-3811 154-381800 2023.
14

---

[67] Zhiqiang Tang, Haoyang Fang, Su Zhou, Taojiannan Yang. Zihan Zhong, Tony Hu, Katrin Kirchhoff, and George Karypis. Autogluon mooggluonmmutimodall (automm): Supercharging multimodal automl with foundation models. arXiv preprint arXiv:2404. /6233, 2024.
[68]
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
[69] James Thome, Andreas Vlachos, Christos Christodoulopoolls, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification. arXiv preprint arXiv: 10Xiv:8803.0533 1803.0535,, 2018.
[70] Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent Ilm framework for pill-pipeline automl. arXiv preprint arXiv:2410.02958, 2024.
[71] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[72] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference Learning Representations, 2024.
[73] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with seff-verification. arXiv preprint arXiv:2212.09561, 2022.
[74] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[75] Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputiing 415:295-316, 2020.
[76] Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.1138) 2023.
[77] Aidin Chenaghlu, and Sacideh Nabipour. Multimodal price prediction. Annals of Data Science, 10(3):619-6355 635,63552222 2023.
Ali-Reza Feizi-Derakhshi, Narjes Kharsmkki, Meysam Asgari-
[78] Haochen Zhang, Yuyang Dong, Chuan Xiao, and Masafumi Oyamada. Large language models as data preprocessors. arXiv preprint arXiv:2308. 16361, 2023.
[79] Michael R Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. arXiv preprint arXiv:2312.04528, 2023.
[80] Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. Retrieve anything to augment large language models, 2023.
[81] Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510, 2023.
[82] Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. In International Conference on Machine Learning, pages 880-41 PMLR, 2023.
[83] Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. Lifelong learning of large language model based agents: A roadmap. arXiv preprint arXiv:2501.07278 2025.
[84] Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finetuning for segment anything model. arXiv preprint arXiv:2401. 17868, 2024.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions: (I) introducing MLZero, a hierarchical multi-agent system with specialized perception agents and dtua-ymeorry system for end-to-end MI. automation; (2) establishing comprehensive benchmark suite for evaluation; and (3) demonstrating superior empirical performance with 92.0% success rate versus 25.3% for competing approaches. These claims align with the paper's scope addressing limitations in current multimodal ML automation systems through novel system that handles diverse data types with minimal human intervention.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper includes dedicated "Limitations and Future Work" section (Section G) that explicitly discusses several limitations of MLZero, including its reduced effectiveness with minimally documented MI. libraries, inability to address undocumented bugs in MIL libraries without explicit prompting, and performance gaps between smaller (8B) and larger language models. The section also outlines future research directions to address these limitations.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ted system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
16

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
We propose a hierarchical multi-agent system for multimodal ML automation with no theoretical results requiring proofs. We provides equations describing the system's components, but these are defining the system rather than presenting theorems that need formal proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please check Appendix for the implementation details, and Appendix for the design and prompts for each agent. Our source codes are also attached in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
17

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Source codes and datasets, well as instructions run them, are included in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
hhboli//guuddess/ddeesummm.mmm sssi for more details.
While we encourage the release code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
https/ccccpuliiguuddeeeeeeeeee CodeeSuuee wwt.sSuummssiiioPPa. ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Appendix C for complete implementation details and Ap- pendix A for comprehensive dataset information. Our method does not require hyperpa- rameter tuning, with the only adjustment being maximum token size constraints to ensure compatibility with smaller language models.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
18

---

7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and standard deviation across three independent runs for all experiments in Table [ providing a clear measure of the statistical variability in our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computer resources reproduce the experiments are included in Ap- pendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm our research fully adheres to all its principles and guidelines.
Guidelines:
19

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impact is included in Section S
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., dep.oymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
20

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: They properly cited and credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpsss..pppsssteeep.eo thodee www.coo///aaoa/ datasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide comprehensive documentation alongside our code implementation, including detailed instructions for agent system are thoroughly documented in Appendix B which describes each agent's role, capabilities, and interactions.
usage and reproducibility. All components of our MI.Zero
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
21

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM //tp:e/e///eeppppoccc/oooooorrer for what should or should not be described.
22

---

Contents
Introduction
Related Work
2.1
LLM Agents for ML
2.2
Tool Use
2.3
Code Generation with L.M
3
Methodology
3.1
Perception
3.2
Semantic Memory
3.3
Episodic Memory
3.4
Iterative Coding
Experiments
4.1
Implementation Details
Main Results
4.2.1
ELEnhech
4.2.2
Multimodal AutoML Agent Benchmark
More Ablation Studies
4.3.1
More Ablation: Semantic Memory
4.3.2
More Ablation: Episodic Memory
4.4
Failure Cases and Analysis
4.5
Efficiency and Robustness Analysis
5
Conclusion
10
Dataset Details
25
A.I
Details of Multimodal AutoMI. Agent Benchmark
................................
25
Construction
................................
25
A.1.2
Preprocessing
26
A.1.3
Tabular Classification and Regression
................................
26
1.1
Multimodal Classification and Regression
27 ................................
A.1.5
Document Classification
28
A.1.6
Multilingual and Multitable Classitication
................................
30
A.1.7
Multimodal Classification (Multilabel)
31
A.1.8
Semantic Segmentation
31
A.1.9
Retrieval
32
A.
1.110
Timeseries Forecasting
32
A.
Evaluation
33
23

---

A.1.12
Human Reported Results
33
A.1.13
Average Rank, Relative Time, and Success
34
2
Details of MLE-bench Lite
34
A.2.1
Implementation Details
34
A.2.2
Performance Thresholds for bench Datasets
................................
35
A.2.3
Limitations of ML.E-bench Evaluation
3
Details of Datasets used in Section for Ablation Studies
35
Agents in MI.Zero
36
B.1
Perception 9
36
B.1.1
File Grouping
and File Perception
36
B.1.2
Task Perception
38
B.1.3
ML Library Selection
39
B.2
Semantic Memory
39
B.2.1
Condensation
39
B.2.2
Summarization
40
B.2.3
Retrieval Q
40
B.3
Episodic Memory
B.3.1
Error Analyzer
B.4
Iterative Coding
42
B.4.1
Coder
42
B.4.2
Executer
45
C
More Implementation and Competitors Details
46
C.I
Implementation Details
46
MLZZro
46
Codex CLJ
47
4 AIDE
48
DS-Agent
49
C.6 AutoKaggle
49
ML Libraries in MLZero
50
D.I Tabular
50
D.2 Multimodal
51
Timeseries
Retrieval
52
D.5 Others
52
D.6 Library Integration Process
53
Detailed Results
53
24

---

E.1
Detailed Results on Multimodal AutoML Agent Benchmark
53
Detailed Results ML.F-bench Lite
59
Detailed Error Analysis
59
E.4
Continual Improvement Beyond First Success
60
E.5
Examples of Generated Code
61
E.5.1
Code generated by MLZZro def on Hateful Meme Datasel
E.5.2
Code generated by Codex +rea on the Hateful Meme Datasel
64
E.5.3
Code generated by AIDE def on the Hateful Meme Dataset
65
More Related Work: AutoML with LLMS
70
Limitations and Future Work
70
Dataset Details
A.1
Details of Multimodal AutoML Agent Benchmark
Table 5: Dataset information with modality, problem types, evaluation metrics, and data splits
Dataset
Modality
Problem Type
Metric
Higher
Data Samples
Semantic Seg
Document
Regression
Multiclass
Forecasting
Retrieval
Multi-label
Tabular
Image
Binary
Truin
Text
Test
better
abalone
RMSE FI(weighted)
3.759
418
airbnb
18,316
4,579
airlines
AUROC
485.444
53,939
bio
AUROC
3,375
376
camoseg
S., R² Recall 10 F1(weighted) MASE
3,636
404
250
ed18
2,533
632
climate
1,535
covertype
522,910
58,102
electric(H)
321
flond
Fl Recall@ 10 (lwcighttd)
3,153
557
fiqa
14,166
1,238
648
gnad10
8,228
1,017
1,028
ham10000
Fl(weighted)
10,015
1,512
hateful
F1
7.134
1,794
isic2017
loU
2,000
600
funding
Fl
89,879
22,451
memotion
Average Ace
5,593
1,399
midoc
Fl(weighted)
5,000
4,000
nn5(D)
MASE
petfinder
FI(weighted)
11,994
2,999
roadseg
IoU
1.107
48
rviedip
(wweighte))
320,000
40,000
40,000
solar(10m)
MASE
137
clothing
R
18.789
2,349
2,348
yolanda
RMSE
360,000
40,000
The midoc dataset has varying training sizes (IK, 2K, 5K, 10K) across multiple languages (German, English, Spanish, French, Italian).
A.1.1 Construction
We constructed the Multimodal AutoMI. Agent Benchmark (MAAB) to address critical gap in existing benchmarks: the ability to evaluate agents raw, unprocessed multimodal data. To ensure
25

---

fairness and diversity, all 25 datasets are sourced from reputable public repositories including Kaggle competitions. UCI Machine Learning Repository, and the BEIR benchmark suite. Datasets were selected from published machine learning research, with most lacking direct open-source solutions to prevent contamination. The benchmark spans distinct problem types (binary classification, multiclass classification, multi-label classification, regression, segmentation, retrieval, and time series forecasting) across multiple data modalities including tabular, text, image, document, and multimodal combinations. We open-source the complete benchmark including all datasets, evaluation scripts, and preprocessing specifications. Our evaluation follows standard practices: the knowledge base contains only general library documentation with information about evaluation datasets their solutions, and test sets remain completely held out during all execution stages.
A.1.2 Preprocessing
To maintain the realistic challenge of processing raw data while ensuring fair comparison across meth- ods, we implemented minimal preprocessing interventions. Specifically, we: (1) augmented datasets lacking descriptions with brief task specifications to accommodate baseline agent requirements, (2) removed target columns from test files to prevent data leakage to agent.
A.1.3 Tabular Classification and Regression
We select five diverse datasets (with details below) for tabular classification and regression tasks from TabRepo [6]. To accommodate baseline agent requirements, we augmented each dataset with brief task descriptions. These descriptions provide minimal but sufficient context for evaluation across all comparative systems while ensuring consistency in task. For instance, the abalone dataset includes the description:
Description for Abalone (Tabular Classification and Regression)
Regression on Class_number number of srrinssss.... Eval metric is RMSE.
The Abalone Dataset (abalone) [54] 61 consists of 3, 759 training instances and 418 testing instances, totaling 4, 177 samples. Each observation contains 8 input features: sex (categorical: Male, Female, or Infant), and 7 physical measurements including length, diameter, height, whole weight, shucked weight, viscera weight, and shell weight. The regression task involves predicting the number of rings in the abalone shell, which serves as proxy for the abalone's age. The evaluation metric for this regression task is Root Mean Square Error (RMSE). The dataset presents challenging regression problem due to the non-linear relationship between the physical measurements and the target variable, making it valuable benchmark for assessing regression algorithms.
The Airlines Dataset (airlines) 61 consists of 485,444 training instances and 53,939 test instances for flight delay prediction. Each instance contains 8 features: airline carrier code, flight number, origin airport, destination airport, day of week (1-7), time of departure (in minutes), flight duration (in minutes), and the binary target variable indicating delay status (0 for on-time, 1 for delayed). The dataset captures commercial flight information across various U.S. airports, carriers, and time periods. This binary classification problem represents a practical application in transportation logistics, where accurate delay predictions can significantly impact resource allocation, scheduling decisions, and customer satisfaction in the aviation industry.
The Bioresponse Dataset (bio) 61 contains 3,751 samples (3,375 training, 376 testing) for binary
classification of molecular biological responses. Each row represents molecule with a binary target indicating response presence (I) or absence (0). The feature space consists of 1,776 normalized molecular descriptors (D1-D1776) representing chemical properties such as size, shape, and elemental constitution.
The Covertype Dataset (covertype) 9 61 consists of 522, 522,910 training samples and 58, 102 testing samples, totaling 581, 012 instances. represents a multiclass classification problem aimed at predicting forest cover types from cartographic variables without remotely sensed data. The dataset encompasses four wilderness areas within the Roosevelt National Forest in northern Colorado, characterized by minimal human disturbance, thus reflecting primarily ecological processes rather than forest management influences. The target variable classifies areas into seven cover types, including Spruce/Fir, Lodgepole Pine, Ponderosa Pine, Cottonwood/Willlw, Aspen, Douglas-fic fir, and
26

---

Krummholz. Predictor variables include elevation, aspect, slope, and various soil characteristics represented as binary features. This dataset particularly valuable for evaluating algorithmic performance on high-dimensional. imbalanced, multiclass problems with geographic dependencies.
The Yolanda Dataset (yolanda) 25 61
consists of 400, 000 total samples, divided into 360, 000 training instances and 40, 000 testing instances. This dataset was utilized for a regression task, where the objective is to predict the values in column "101" " based on the preceding features.
A.1.4 Multimodal Classification and Regression
We evaluated our system on nine diverse multimodal classification and regression datasets) [67|
airbnb (airbnb melbourne) [.] cd18 (cd18 cellphone) [77], flood (european flood depth) [4]
gnad10 62], ham10000 711, hateful (hateful meme) 41, funding (kick start funding), petfinder [2
and clothing (women clothing review) 4 These datasets incorporate various combinations of tabular, text, and image data, presenting distinct challenges for automated machine learning systems.
Similarly, we supplemented datasets lacking descriptions with concise task specifications to ensure consistent evaluation across all compared methods.
Description for CD18 (Multimodal Classification and Regression)
The goal of this research is to achieve an arrangement to predict the price of a cellphone based on its specifications.
The Airbnb Melbourne Dataset (airbnb) D 12 67 consists of 22,895 listings, partitioned into 18,316 training samples and 4,579 testing samples. The dataset captures various features of Airbmb accommodations in Melbourne, Australia. The target variable is price label. which represents categorized price ranges grouped into distinct bins. To prevent data leakage, the raw price feature has been removed from the dataset. This classification task requires models predict the price category of listing based on its characteristics, such as location, property type, amenities, and host attributes, making it suitable for evaluating price prediction algorithms in the short-term housing market.
price_label,
Cellphone Dataset (cd18) 77 contains 2,533 training samples and 632 testing samples, en- compassing diverse attributes such as model information, physical characteristics (weight), technical specifications (processor, RAM, storage, display dimensions, resolution), battery information, camera capabilities, operating system, and market metrics (hit count). The dataset spans multiple manufactur- ers, device generations, and price points, providing the information for developing and evaluating predictive models for cellphone pricing based on technical specifications.
European Flood Depth Dataset (flood)
67
comprises 3,710 images for binary classification,
with 3,153 samples in the training set and 557 samples in the test set. Each image is labeled as either "Not useful for determining depth" (0) or "Useful for determining depth" (1), representing the image's utility in flood depth estimation. The dataset includes metadata such image identifiers, confidence scores, and bounding box coordinates. This dataset serves as a foundation for developing models that can automatically identify imagery suitable for flood depth analysis, potentially enhancing rapid damage assessment capabilities during flooding events.
consists of 10, 273 German news articles collected from an Austrian online newspaper, categorized into distinct classes: Web, Panorama, International, Wirtschaft, Sport, Inland, Etat, Wissenschaft, and Kultur. The dataset is split into 8, 228 training samples, 1, 017 validation samples, and 1, 028 testing samples. Article titles and text are concatenated, with author information delibcrately removed to prevent classification based on author-specific keywords. serves as a benchmark for German topic classification tasks, addressing the unique challenges posed by the German language's 's higher inflection rate and longer compound words compared to English.
The 10kGNAD Dataset (gnad10) 622 67
The HAM10000 Dataset (ham10000) 7 67 consists of 10,015 training and 1,512 testing der- matoscopic images collected from diverse populations using various acquisition modalities. We removed the dx_ t=type feature to prevent potential data leakage in our experimental setup. The dataset encompasses seven diagnostic categories of pigmented skin lesions: actinic keratoses and Bowen's disease (akiec), basal cell carcinoma (bcc), benign keratosis-like lesions (bkl), dermatofibroma
27

---

(df), melanoma (mel). melanocytic nevi (nv), and vascular lesions (vasc), Ground truth diagnoses were established through histopathology (>50% of cases), follow-up examination, expert consensus, or in-vivo confocal microscopy. The dataset includes metadata on patient demographies (age, sex), lesion localization, and unique lesion identifiers that allow tracking multiple images of the same
lesion.
The Hateful Memes Dataset (hateful) 41 67 is multimodal benchmark consisting of 7, 134 training and 1, 794 testing examples designed to evaluate hate detection systems. Each sample contains an image paired with overlaid text, requiring models to understand both visual and linguistic content and their potentially harmful interactions. The dataset is labeled binary, where 1 indicates hateful content (targeting protected categories through harmful stereotypes, derogatory language, or harmful contexts) and 0 represents benign content. The challenging nature of this dataset stems from the necessity for models to comprehend cross-modal relationships, as the hateful nature often emerges from the interaction between text and imagery rather than from either modality independently.
The Kickstarter Funding Dataset (funding) [67] consists of 112, 330 crowdfunding campaign records (89, 879 for training and 22, 451 for testing) extracted from the Kickstarter platform. Each record contains 10 features including campaign name, description, funding goal amount, keyword identifiers, communication preferences, country of origin, currency type, deadline timestamps, creation timestamps, and the target variable (final_ status) indicating campaign success or failure. The binary classification task involves predicting whether a crowdfunding campaign will successfully reach funding goal based on the provided features. This dataset captures the multifaceted nature of crowdfunding dynamics across various project categories, geographical regions, and funding scales.
The PetFinder.my Adoption Prediction Dataset (petfinder) 2 67 consists of 14, 993 samples (11, 994 training, 2, 999 testing) designed for predicting pet adoption speed. Each entry contains pet characteristics including type, age, breed, gender, color, size, health status, and vaccination records. Additional features encompass textual descriptions, rescuer information, quantity of photos and videos, adoption fees, and geographical location. The target variable, AdoptionSpeed, is categorized on scale indicating how quickly pets were adopted. The dataset also includes image paths for visual analysis, making suitable for multimodal machine learning approaches to predict adoption outcomes.
comprises 23, 486 cus- tomer reviews women's 's clothing products from an e-commerce retailer. Each observation contains 10 features: a unique clothing identifier, reviewer's age, review title, review text, product rating (I-5 scale), binary recommendation indicator, positive feedback count (number of customers finding the review helpful), and three hierarchical product categorization variables (division, department, and class names). For our experiments, we employed an 8-1-1 train-validattio-tees split of the data. The dataset enables analysis of customer sentiment, recommendation patterns, and demographic preferences across different clothing categories, making it suitable for natural language processing, sentiment analysis, and recommendation system research.
The Women's E-Commerce Clothing Reviews Dataset (clothing) 4 67
A.1.5 Document Classification
Document classification entails the categorization of visual documents, which can be approached as image classification task, an OCR-based text classification task, as a multimodal problem integrating both methodologies. Our experiments utilize the RVL-CDIP (Ryerson Vision Lab Complex Document Information Processing) [27] dataset, comprising 400,000 grayscale images distributed across 16 distinct document classes with 25,000 images per class. The dataset is partitioned into 320,000 training images, 40,000 validation images, and 40,000 test images. This dataset presents particular challenges due to its substantial size and intricate file structure, highlighting the necessity of effective perception rather than naïve loading of all file structures into the LLM, which would exceed context limitations.
Data Structure and Description for RVL-CDIP (Document Classification)
Data Structure
rvl_ wwL.ccii/rraaninn/nffeeeoo..o
28

---

Content:
File Size: 1.55 MB
First few lines of the file:
document
rvl wwtp/ttainiin//vlllccc
Content:
File Size: 1.64 MB
First few lines (up to 1024 characters):
document
label
Anngecs/ssVVAaghasee/00002222223 11
hmaggs///||y//yyyss/00///2226621 jpg 0
imnger////11//r222226////1110000 jpg
rvl wwL_.cddp/zeainnnngrraai....r
Content: File Size: 13.09 MB
First few lines:
document label
/nnges////ooc/occ5cc/////5555555 jpg 15
imngese/o//cccooc22dd//5/122228 jpg 1
rvl wwl_cdap/rrrannnng/reeeeee.tt
Group pattern: rvl_
(total 400000 files)
Example file:
rvl_ h_t.:////haaannnnnnee//eeea////2 tiff
Content:
File Size: 0.03 MB
Image Format: TIFF
Image Mode:
Image Size: (754, 1000)
Array Shape: (1000, 754)
Data Type: uint8
Min Pixel Value: 0
Max Pixel Value: 255
Mean Pixel Value: 76.94
Standard Deviation: 116.63
Description (readme.txt)
RVL-CDIP Dataset
The RVL-CDIP (Ryerson Vision Lab Complex Document Information Processing) dataset consists of 400,000 grayscale images in 16 classes, with 25,000 imagas per class. There are 320, 000 training images, 40,000 validation images, and 40,000 test images. The images are sized so their largest dimension does not exceed 1000 pixels.

---

DETAILS
The label files list the images and their categories in the following format:
path/to/thheiingg.crr tif category
where the categories numbered 0 to 15, in the following order:
letter form
email
3 handwritten
advertisement
acientific report
scientific publication
7 specification
file folder
9 news article budget
11 invoice
presentation
13 questionnaire
14 resume
15 memo
Prediction should be saved in dataframe with column names "document" and "label".
The RVL-CDIP Dataset (rvledip) [27] comprises 400, 000 grayscale document images scanned at approximately 100 dpi, distributed across 16 distinct semantic categories including letters, forms, emails, resumes, and memos. The dataset is partitioned into 320, 000 training images, 40, 000 validation images, and 40, 000 test images, with each class being represented equally. The corpus is characterized by significant quality variations typical of real-world document collections, including scanning artifacts, noise, skew, and low resolution, making it challenging benchmark for document classification tasks.
A.1.6 Multilingual and Multitable Classification
We employed the midoc MLDoc-11109 664 dataset for evaluating multilingual classification capa- bilities, extending beyond the monolingual English usage in previous work (67). Our configuration incorporates multiple languages (German, English, Spanish, French, and Italian) simultaneously, Specifically, we concatenated test files from all five languages form a multilingual while preserving the original structure of the remaining data files.
Description for MLDoc-1 11000 (Multilingual and Multitable Classification)
This a multi lingual task. Train multilingual model and give the predictions.
The Multilingual Document Classification Corpus (mldoc) (44) cross-lingual text classification benchmark. It encompasses eight languages: English, German, French, Spanish, Italian, Russian, Japanese, and Chinese. ML.oc features balanced class distribution across languages and follows standardized train/dev/test split. For most languages, training sets of varying sizes (1K, 2K, 5K, and 10K documents) are provided, with Spanish and Russian having 9,458 and 5,216 training documents respectively. Each language includes IK development set and 4K test set. In our experiments,
30

---

utilize the German, English, Spanish, French, and Italian subsets, treating this as both multi-table and multilingual classification task, with separate training tables for each language and combined test files.
A.1.7 Multimodal Classification (Multilabel)
We adopted the memotion [50] dataset for multilabel classification. We incorporates all five classi- fication dimensions: humor (Not humorous, funny, very funny, hilarious), sarcasm (Not Sarcastic, general, twisted meaning, very twisted), offensive (Not offensive. slight, very offensive, hateful offensive), motivational (Not Motivational, Motivational), and overall sentiment (Negative, Very Negative, Positive, Very Positive, Neutral). We employ average accuracy as the evaluation metric to assess performance across all dimensions.
Description for Memotion (Multimodal Multilabel Classification)
Train the model on image and corrected text to predict "humour", "sarcasm", "offensive", "motivational", "overall_sentimeet
the meme.
humour: Not humorous, funny, very funny, hilarious
sarcasm: Not Sarcastic, general, twisted meaning, very twisted offensive: Not offensive, slight, very offensive, hateful offensive motivational: Not Motivational, Motivational overall_sentiment sentiment: Negative, Very Negative, Positive, Very Positive, Neutral
The Memotion dataset comprises 5, 593 training and 1, 1,399 testing samples, each consisting of meme images paired with corrected text annotations. This dataset is designed for multi-label classification across five dimensions: humor (four classes: humorous, funny, very funny, hilarious), sarcasm (four classes: not sarcastic, general, twisted meaning, very twisted). offensiveness (four classes: not offensive, slight, very offensive, hateful offensive), motivation (binary: not motivational, motivational), and overall sentiment (five classes: negative, very negative, positive, very positive, neutral). The dataset enables research automated understanding emotional and contextual elements internet memes, addressing the complex interplay between visual and textual components in multimodal sentiment analysis.
The Memotion Dataset (memotion) 50 67
A.1.8 Semantic Segmentation
We include three semantic segmentation datasets: camoseg (camouflaged segmentation) [2). roadseg (road segmentation) [51] and isic2017 (15). Each dataset presents unique visual analysis challenges requiring pixel-level prediction capabilities. To standardize evaluation, we provided descriptive task specifications; for instance, the camoseg dataset includes the description:
Description for Camouflaged Segmentation (Semantic Segmentation)
Perform semantic segmentation to identify and delineate camouflaged objects that visually blend with their surroundings, outputting pixel-level binary masks.
The Camouflage Segmentation (camoseg) 67 84 dataset consists 4, 290 images with corresponding pixel-level binary masks, designed for identifying and segmenting camouflaged objects that visually blend with their surroundings. The dataset is partitioned into training (3, 636 images), validation (404 images), and testing (250 images) subsets. Each sample comprises input image paired with its ground truth segmentation mask, where camouflaged objects are delineated at the pixel level. The dataset encompasses diverse range of naturally camouflaged subjects, demonstrating various camouflage mechanisms observed natural environments.
The ISIC 2017 Dataset (isic2017) 15
consists of dermoscopic images used for automated melanoma diagnosis, specifically focused on semantic segmentation skin lesions. The dataset com- prises 2, 000 training images and 600 testing images, each paired with expert-annotater ground truth
31

---

segmentation masks, aimed at advancing computer vision techniques for dermatological applications and improving early detection of melanoma and other skin cancers.
The Massachusetts Roads Dataset (roadseg) [5 67 consists satellite imagery for road segmentation tasks, comprising 1, 107 training samples, 13 validation samples, and 48 testing samples. Each sample consists of an input satellite image paired with its corresponding ground truth road segmentation mask. The dataset is structured facilitate the development and evaluation of machine learning models for automated road network extraction from aerial imagery. The input images are stored as PNG files in designated directories, with corresponding output masks containing pixel-wise road annotations.
A.1.9 Retrieval
We incorporated two retrieval datasets from the BelR benchmark [68]: fiqa (FiQA-2018) and climate (Climate Fever). The Climate Fever dataset serves zero-shot evaluation as it lacks training set, testing systems' ability to perform retrieval without task-specific training data. To optimize computational resources during benchmarking, reduced the Climate Fever corpus size one-tenth of its original scale while maintaining the positive documents. In the description file we specify the output format for easier evaluation:
Description for FiQA-2018 (Retrieval)
Finetune the model on training data. And then retrieve the top 10 results for the queries. The output should have three columns: query-id, corpus-id, score.
Each row is retrieved pair.
e.g.
query-id
corpus-id 566392
score
8
0.6
65404
0.4
15
325273
0.7
The Climate-FEVER Dataset (climate) [17 68
is a zero-shot fact verification dataset consisting of 1, 535 testing query query-document pairs without training data. The original corpus contains 418, 128 documents; however, for computational efficiency, we utilize a 10% subset of the corpus while ensuring all ground truth documents are preserved. The dataset follows the FEVER [69] framework but applies it to real-world climate change-related claims collected from the Internet, offering unique challenges in natural language understanding. For our experiments, require retrieval output using the standard format of query-id, corpus-id, and relevance score, evaluating the recall for top-10 retrieved documents for each query.
The FiQA-2018 Dataset (fiqa) in the BEIR benchmark (68] consists of financial question-answering data with 14, 166 training samples, 1, 238 validation samples, and 648 test samples. The correspond- ing document corpus contains 57, 638 documents. We evaluated performance by retrieving the top-10 most relevant documents for each query in the test set and compute the recall.
A.1.10 Timeseries Forecasting
We incorporated three timeseries forecasting datasets (65 5 with varying temporal resolutions: nn5(D) (nn5 _aily), solar( 10m) (solar_ 10_110_mmmm [0_minutes), and electric(H) (electricit__hourff The training data is provided in compressed ISONL format with json.gz extension. To mitigate potential confusion regarding data format, we supplemented each dataset with detailed format specifications. For example, the mn5 daily dataset includes the following description:
Description for nn5 daily (Timeseries Forecasting)
File Structure:
jsonl, each line is formatted as below: "target": [18. 13, 25. 46, 25. 11,......
32

---

"start": -1996-03-18 00:00:00", "T1", Feat_static_catt [o]
"item_io ii" :
You should train models that forecast the target up to 56 days into the future. Evaluation metric is MASE.
Note that the frequency is 1 day.
The output should a jsonl file.
The Electricity Hourly Dataset (electric(H)) consists of 321 distinct time series representing electricity consumption measurements collected at hourly frequency. Each series contains 428, 176 time steps, spanning multiple years of observations. The forecasting task involves predicting 24 steps (hours) into the future. The dataset is structured in JSONL. format, where each record contains target vector consumption values, a timestamp indicating the starting point, a unique identifier for each series, and categorical features. Models trained this dataset evaluated using the Mean Absolute Scaled Error (MASE) metric.
The NN5 Daily Dataset (nn5(D)) 65 comprises 111 time series from the banking domain, specifically recording daily cash withdrawal amounts from automated teller machines (ATMs) across the United Kingdom. The dataset daily ATM withdrawal patterns. To address data quality concerns, missing values in the original dataset were imputed using a day-of- day-of-week median replacement strategy, whereby each missing observation was substituted with the median value calculated from the same weekday across the entire corresponding time series. The forecasting task involves predicting 56 steps (days) into the future.
designed to evaluate forecasting methods' efficacy in predicting
The Solar Dataset (solar(10m)) 665 comprises 137 time series representing 10-minute obser- vations of solar power production recorded per every 10 minutes in Alabama state in 2006. The forecasting task involves predicting 60 steps (10-minutes) into the future.
A.1.11 Evaluation
We implemented a rigorous evaluation protocol to ensure fair and consistent assessment across all systems. As previously noted, we removed target columns from test files prevent data leakage to agents. The complete test data remained accessible only within the evaluation environment, where performed strict line-by-line ordered comparisons between agent outputs and ground truth. This protocol enforces format compliance, i.c. deviations in output structure, naming conventions, or completeness result in evaluation failure. To facilitate consistent evaluation while maintaining the challenge of raw data processing, we maintained a metadata file for evaluator reference (not exposed to agents):
metadata.json for Abalone dataset
"dataset name": "abalone",
imetric_name" "rmse",
"problem_type type:: "regression",
number_ot "label_colle colemn"; "Class_ (Classsnumber___ of of rings", "modality": ["tabular"]
This structured evaluation framework ensures that success metrics reflect not only predictive per- formance but also practical usability in real-world scenarios where output format requirements are equally critical.
A.1.12 Human Reported Results
The human reported baseline results are retrieved from established open-source benchmarks and liter- ature: camoseg (67], electric(H) (65], flood 677, fiqa [52], gnad10 [67], ham1000 [67], hateful (77
isic2017 [67]. funding [67], nn5(D) [65], petfinder [67] roadseg [67]. rvledip 7 and clothing [67]
33

---

For the remaining datasets, comparable metrics from open-source implementations are unavailable, either due to the absence of published performance metries, differences evaluation metries, or because our modifications to these datasets preclude direct comparison.
A.1.13 Average Rank, Relative Time, and Success
The average relative time usage (Rel. Time) quantifies computational efficiency across models by normalizing execution times against the reference model MI (def). For each dataset d in the collection D, we compute the ratio of execution time m,,d for model relative the reference model's s time fre.d. The average relative time usage T m for model m is then defined as: lmnd
T m
Σ
[Dm.res
lrefad
dEDm.re
where [Dm.refl represents the number of datasets with valid measurements for both the evaluated model and the reference model. Both beefed id and m,d d are averaged across valid results in three runs for each model-dataset combination.
The average rank (Arg. Rank) measures the relative performance of each model across all datasets. For each dataset d, models are ranked in descending order of performance, with the best-performing model receiving rank 1. Invalid results assigned the worst rank. The average rank Rm, for model m is computed as:
Rm
Tmnd
D
dED
where Tm.d is the rank of model m on dataset d. and [D] the number datasets. When multiple models have the same performance score dataset, the average mode is used, where tied models receive the same rank calculated as the average of the ranks they would have received had they not been tied.
The success rate (Suecess) indicates the percentage of dataset-run combinations where model successfully completes the task. For model the success rate Sm, is defined as:
DD
Sm
x 100%
D x 3
where [Dml is the number of dataset-run combinations with valid results for model m, and [D] is the total number of datasets in the evaluation. Each dataset is evaluated across 3 independent runs, resulting in [D] x 3 total evaluation instances.
A.2
Details of MLE-bench Lite
A.2.1
Implementation Details
For the MLE-bench Lite [10] evaluation, we modified our hyperparameter configuration from those used in the Multimodal AutoML Agent Benchmark. Specifically, we configured the agent to use best quality preset parameters when available [19 65| 67 with 4-hour time limit per iteration to thoroughly evaluate performance. We established 24-hour overall limit for each dataset. All experiments were conducted on an AWS EC2 24x.24xl instance equipped with 8 NVIDIA A100 (40GB) GPUs and 96 vCPUs.
Since training requires substantial computational resources and time, we utilize the performance metrics reported in the original ML.E-Bench publication [10). Although the specific LLM model is not explicitly stated these reports, we select the first reported results for each agent to maintain consistency. We excluded the canzor-clip-catheter caap--cttttee--neellaassee cinecllasiification dataset due to preprocessing inconsistencies, ensuring fair comparison across all systems.
Many competitions in the benchmark closed several years ago, making the leaderboard rankings potentially outdated compared to current human capabilities. Therefore, beyond medal counts, we compare the relative performance ranking of each agent across all datasets and compute average rankings. Our evaluation framework incorporates six comprehensive metrics: (1) number of gold medals, (2) number of gold and silver medals combined, (3) number of gold, silver, and bronze medals combined, (4) number of performances above median threshold, (5) success rate (proportion of non-trivee non-trivial submissions with non-zero performance), and (6) average rank across all datasets.
34

---

A.2.2
Performance Thresholds for MLE-bench Datasets
the performance thresholds established for each dataset's medal and above-median
categorization.
Table 6: Dataset Indices and MLF-Benchh Information
ID
Dataset Name
Gold
Silver
Bronze
Median
D1
Aerial Cactus Identification (1)
1.0000
1.0000
1.0000
0.9991
D2
APTOS 2019 Blindness Detection (t)
0.9305
0.9197
0.9145
0.8889
D3
Denoising Dirty Documents (1)
0.0179
0.0261
0.0452
0.0733
D4
Detecting Insults in Social Commentary (1)
0.8332
0.8231
0.7911
0.7784
D5
Dog Breed Identification (J)
0.0005
0.0054
0,0460
0.4721
D6
Dogs vs Cats Redux (L)
0.0388
0.0504
0.0613
0.1222
D7
Histopathologic Cancer Detection (t)
0.9835
0.9798
0.9738
0.9477
D8
Jigsaw Toxic Comment Classification (f)
0.9874
0.9867
0.9864
0,9808
D9
Leaf Classification (J)
0.0000
0.0079
0,0153
0.1083
D10
MLSP 2013 Birds (?)
0.9353
0.9004
0.8737
0.8657
D11
NYC Taxi Fare Prediction (1)
2.8338
2.8819
2.9237
3.5974
D12
NOMAD 2018 Conductors (L)
0.0559
0.0623
0.0658
0.0699
D13
Plant Pathology 2020 FGVC7 (1)
0.9784
0.9747
0.9736
0.9485
D14
Random Acts of Pizza (f)
0.9791
0.7648
0.6921
0.5996
D15
SUMMIIIC Melanoma Classification (t)
0.9455
0.9401
0.9370
0.9128
D16
Spooky Author Identification (L)
0.1651
0.2700
0.2938
0.4188
D17
Tabular Playground Dec 2021 (1)
0.9566
0.9566
0.9566
0.9534
D18
Tabular Playground May 2022 (1)
0,9982
0.9982
0.9982
0.9727
D19
Text Normalization English (f)
0.9972
0.9914
0.9904
0.9904
D20
Text Normalization Russian (1)
0.9901
0,9823
0.9759
0,9759
D21
ICML 2013 Whale Challenge (t)
0.9896
0.9502
0.9052
0.8652
A.2.3
Limitations of MLE--bench Evaluation
MLE-bench [10] provides valuable comparisons between ML agents and human performance on Kaggle competitions, but has notable limitations. The benchmark uses preprocessed data through dedicated Python scripts, presenting structured rather than raw inputs, which may overestimate agent capabilities. This contrasts with our Multimodal AutoML Agent Benchmark that specifically tests end-to-end processing of raw data. Additionally, ML.E-bench's 's agent-human comparisons face methodological issues: different data splits, evaluation conditions, and human leaderboards that often reflect earlier computational environments with limited hardware and algorithms. These factors introduce systematic biases that complicate performance comparisons and may not accurately represent contemporary ML automation capabilities among human competitors. While MLE-bench demonstrates our system's effectiveness within established workflows, our benchmark evaluates the end-to-end capabilities that distinguish MLZero from other ML agent approaches.
A.3 Details of Datasets used in Section 4.3 for Ablation Studies
To evaluate the contribution of individual components to our system's overall performance, we conducted comprehensive ablation studies across diverse subset of datasets. We selected eight representative datasets spanning various machine learning tasks: yolanda [6]), midoc [64], flood [34], petfinder (2). camoseg [42]. rvledip (7). solar( I 0m) [65], and fiqa (68]. We select this subset to
include broad spectrum of domains including tabular data analysis, multimodal data analysis, multi- lingual and multi-table tasks, document classification, image segmentation, time series forecasting, and text retrieval.
35

---

B Agents in MLZero
B.1 Perception
B.1.1 File Grouping and File Perception
Our file grouping mechanism (Algorithm D employs hierarchical approach that analyzes folder structures and file extensions to identify meaningful patterns within raw datasets. We first define max group size 8 (6 by default in all our experiments). For each directory level, the algorithm dynamically determines whether to preserve specific folder names (when I 6 unique names exist at that level) or abstract them using wildcards (when ó names exist), creating balanced representation that captures essential structural information while avoiding over-specification.
Upon establishing these file groups, our perception system selects representative examples for detailed analysis. For small groups I 6, all members undergo comprehensive content inspection, while larger collections are represented by carefully selected exemplars. Each file is then processed through File Perception Agent, which dynamically generates format-appropriate loading and printing code based on file characteristics, thereby providing structured insights into raw data content regardless of format or organization. This two-phase perception approach forms the foundation of our system's 's ability to understand arbitrary data structures without manual preprocessing.
Algorithm 1 File Grouping
1: procedure FILE GROUPING(fles)
2:
FopthFolders +- Map from depth to set of folders
D. First pass: analyze folder structure
for all file files do 4: paths +- SplicPath(/ile.ppall
3:
5: for depth t 0 to |paths| do depthFolders ddpth)] addddddd((pppp.. paths/dsdp))))
6:
7:
end for end for
8:
Second pass: group files
9:
groups <- Map from pattern to file list
10: for all file E files do
11:
paths + (afiiiPPat((fle.pllit pallern -
12:
Build pattern using folder structure
13:
for depth + 0 to |paths| do
| Folders (ddeph]] S. then
15:
pattern. pattern.appendi paths(depthh)
b Use actual folder name
else
17:
pattern. pattern.append(" * *)
b Use wildcard
end if end for www.rnaaapeennGeeeeeeiiiiii (ilenname)) groups proups/pallern) pattern) appendd end for
19:
20:
21:
22:
23:
return groups
24:
end procedure
File Perception Agent
The agent utilizes LLM capabilities analyze file content across various formats, enabling raw data understanding in AutoML workflows. This addresses critical limitation previous systems that required pre-structured data. Prompt:
Context:
f"""Generate
Python code to read and analyze the file: "(file_path)
36

---

Your should:
1. Import all modules used (e.g. import os)
2. Use appropriate libraries based on file type (pandas tabular data, etc.)
3. For tabular files (csv, excel, parquet, etc.):
Display column names. If there are more than 20 columns, only display the first and last 10.
Show first 2-3
with truncated cell content
Do not show additional index column if it's not in the original table
If failed to open the file, treat it as text file
["- Count total
and provide basic statistics" if self details else " No additional info needed. "}
4. For text files:
Display first few lines (up to (max_chars) characters)
5. For compreased tabular or text files, show its decompressed content as described.
("6. For other files, provide appropriate summary" if self details else
"6. For binary or other files, provide only file size. 7. Keep the total output under fmax_charry characters
Return ONLY the Python code, no explanations or markdown. The code should be self-contained and executable on its own. """
Generated Code Example:
import pandas as pd import os
# Get absolute path and file size
file abs path aeeed//aaneraaaaaaatte///aaaaala abspathhht path)
file size os est..ah.oo getssigeeteeleeepattee / (1024 1024)
print Ff"File Size: (file_size: size: 2f) MB")
Read and analyze CSV file try:
df
# Display column names
print Cooluum names: (list cofftddf.olll colffcollmms)))
Show first rows print ("First rows:") print (df. (3)
except Exception as e:
print Errrr reading file CSV: )e)")
# Try as text file if CSV reading fails try:
with open(file_ppteeeeeeeeeeeeeeeleep 'r') f: text r.rd (1000) print ("File content (first 1000 chars):
print (text)
except Exception e2:
37

---

print (f"Error reading text: {e2)")
Example Output:
Absolute path: //tea//ggg//aaaaaattee/////////r
Content:
File Size: 0.02 MB
Column names: ['Sex', 'Length', 'Diameter', 'Height',
Whole_weight/ "Shucked_ "Shucked_weight', 'Viscera "escera_weight', SShell_weight; Class balass_number, of [aabberrof_rrng__
First rows:
Sex
Length
Diameter
Height
Whole, weight
Whole_weight
Shucked Shucked_weight...
0
I
0.620
0.485
0.18
1.1785
0.4675...
F
0.645
0.525
0.17
1.3700
0.6135...
2
F
0.620
0.480
0.17
1.1045
0.5350
B.1.2 Task Perception
The task perception in MLZero occurs in two main phases. First, the system uses an LLM to identify potential description files from the data prompt, looking specifically for files like READMES, documentation, or task definitions. The I.LM's response is parsed extract filenames and returns both the list of files and the analysis explanation. Second, the system generates a comprehensive task description by reading the content of the identified files and using another L.LM call extract key information. It processes the file contents along with the original data prompt and previous analysis produce a structured description that includes the core data science objective, requirements, constraints, data sources, and success metrics. These steps create foundation for understanding what the task requires before proceeding to ML library selection.
Task Perception Agent
Prompt to Find Description File f"""Given this data prompt:
(data_prompt)
Please identify any files that appear contain project descriptions, requirements, or task definitions. Look for files like README, documentation files, or task description files.
Format your response as follows:
Description Files: [list ONLY the absolute path, one per line] Explanation: [explain why these files were identified as description files]"nn "nn
Prompt to Generate Task Descriptions
f"""Based on this data prompt and description files:
Data Prompt:
(IMPORTANT: The metadata example files in Data Prompt may not be representative do not make assumptions about data statistics based on examples.)
(data_prompt)
Description File Analysis: {description_analyee)
Description File Contents:
38

---

{description_conte
Based ONLY the information explicitly stated in the provided data prompt, description files, and analysis, provide a condensed
description of the data science task. Include only details that are directly mentioned in the source materials. Do not add assumptions or infer unstated information. unu
B.1.3 ML Library Selection X
The ML library selection process follows task perception steps and uses an LLM identify most suitable ML library for the data science task at hand. It takes the data prompt, task description, and available libraries from the registry as inputs. The prompt includes the task details and formatted information about each available library (name, version, description, and special features). The LLM's response includes both the selected library name and the reasoning behind the selection.
MI. Library Selection Agent
Prompt to Select ML Libraries
f"""Given the following data science task:
Data Description: (data_prompt)
Task Analysis: (description}
Available tools and their capabilities:
{_format_toolae info L ]nfo))
Please select the most appropriate tool for this task. Consider:
1. The nature of the data (tabular, time series, multimodal, etc.)
2. The specific requirements of the task
3. Any limitations or special features of each tool
Format your response as follows: Selected Tool: [tool name ONLY]
Explanation: [detailed explanation of why this tool is the best choice, including specific features that match the task requirements] "n"
B.2
Semantic Memory
B.2.1
Condensation
The condensation agent processes tutorial content by breaking it into manageable chunks when necessary, then uses LLM to condense each chunk while preserving essential implementation details, code samples, key concepts, critical configurations, and important warnings. It works on chunk-by-chunt basis handle large tutorials, maintains the overall document structure, and ensures the resulting condensed content stays within a specified maximum length by truncating at section boundaries when needed.
39

---

Condensation Agent
Prompt to Condense Document Chunk
context
"This is a continuation of the previous chunk. "
if 0 else
f""" "omncontetttCoddeese this portion of the tutorial while preserving essential implementation details, code samples, and key concepts. Focus on:
1. Implementation details and techniques
2. Code snippets with necessary context
3. Critical configurations and parameters
4. Important warnings and best practices
Chunk
{chunk}
Provide the condensed content in markdown format. ..*
B.2.2 Summarization
The summarization agent generates concise summaries of the documents that help coder agent understand implementation knowledge, coding tasks, and key features covered the tutorial. processes the already condensed content through an LLM create single paragraph summary of less than 100 words that starts with "Summary: " and highlights the most important aspects of the tutorial for practical implementation.
Summarization Agent
Prompt to Summarize the Document
g"""Generate concise summary (within 100 words) this tutorial that helps a code generation LLM understand:
1. What specific implementation knowledge or techniques can find in this tutorial
2. What coding tasks this tutorial can help with
3. Key features or functionalities covered
Tutorial content:
{condensed_ content)
Provide the summary in a single paragraph starting with "Summary:
B.2.3 Retrieval Q
The retrieval agent scans documents of the selected ML libraries, extracts their titles and summaries, and uses an LLM to select the most relevant documents based on the user's task, data description, question, and any previous errors. The selected documents are then provided coder agent to correctly integrate the selected ML library.
Retrieval Agent
Prompt to Retrieve the Condensed Document context
f""nTask: (task_traooooo (task_prompt)
40

---

Data: phata_promppt User Question:
Previous Error: (error_prompt)"
f"""Given the following context and list of tutorials with their summaries, select the {max_num_ num_ nuu__tuorials) most relevant tutorials for helping with this task. Consider how well each tutorial's 's title and summary match the task, data, user question, and any errors.
Context:
{context)
Available Tutorials:
(tutorials_info)
IMPORTANT: Respond ONLY with the numbers the selected tutorials (up to {max_ (max_num__tt sas__nuu__tttrre (uttoriaollss separated by commas.
For example: "1,3,4" or "2,5" or just "1" if only one is relevant. DO NOT include any other text, explanation, formatting in your response. """
B.3
Episodic Memory
B.3.1
Error Analyzer
The error analyzer processes error messages from failed code executions. It takes the original task description, data, user prompt, previous code, and error message as input, and uses an LLM to generate a concise error summary identify root causes and suggested debugging steps provide tactical guidance. The agent is asked not to include actual code fixes, but focusing on clear, actionable insights within strict length constraints.
Retrieval Agent
Prompt to Analyze the Error
"""[task_ (tttss_prompp))
(data_prompt)
{user_prompt)
Previous Python Code:
(python_code] code]
Previous Bash Script to Execute the Python Code:
{bash_
(bash_script}
dnstrieved_tutorine
Error Message:
Cerror_message
Analyze the error message and context provided. Your response MUST contain exactly two short paragraphs as follows:
ERROR SUMMARY: Provide brief, technical description of the error in 1-3 sentences. Focus only on identifying the root cause and affected component without background explanations.
SUGGESTED FIX: Offer specific debugging directions in 1-3 sentences. Do not include actual code or commands, only tactical debugging guidance.
41

---

Each paragraph must be concise (maximum 3 sentences). Do not include general advice, explanations beyond the direct debugging strategy, or any additional paragraphs. nen
B.4
Iterative Coding
B.4.1
Coder
The Coder agent creates programming solutions based and multi-turn interactions. In our experiments, we use single-turn interaction by default as episodic memory can provide information about past errors. In the ablation study of removing episodic memory, we use multi-turn interactions where coders in all iterations run in a common session. The Code agent also ensures code is properly formatted within language-specins code blocks (python or bash in our experiments), extracting clean scripts from LLM responses.
prompts. supports both single-turn
For Python code generation, the prompt includes system prompt, library-specific instructions, task perception, data perception, optional user input, error analysis from episodic memory, and retrieved documents from semantic memory.
Similarly, the bash script generation prompt assembles contextual components including environ- ment configuration preferences (tailored to whether environment setup is required), code execution instructions, the generated Python code to run, and (when applicable) previous bash scripts and messages.
Coder Agent
Prompt to Generate Python Code
f"n"
As an AutoML Agent, you will be given folder containing data and description files. Please generate Python code using {tool_ name> to train predictor and make predictions on test data. Follow these specifications:
ONLY save files to the working directory: {output_folder)
1. Data preprocessing:
Remove training data samples without valid labels (unless told not to do so).
Remove the unneccesary index column (if applicable)
2. Model training:
Use {tool [tool_name) with appropriate parameters for the task If a model is trained, save it in a folder with random timestamp within foutput_foldes
3. Prediction:
Make predictions on the test data Save the predicted results to forput_folder), result name should be "results", the format and extension should be same as the test data file
Output column names must exactly match those in the training sample submission files without adding "predicted_" prefixes or creating any new columns.
4. Documentation:
Add brief docstring at the beginning of the script explaining its purpose and usage
42

---

Also include additional installation steps with comments at the beginning of the script
Include comments explaining any complex operations or design decisions
5. Others:
To avoid DDP errors, wrap the code in: if name == ___main__"" ": Ensure errors are propagated up and not silently caught not use try/except blocks unless you explicitly reraise the exception.
(tool_prompt}
Please provide the complete Python script that accomplishes these tasks, ensuring it's ready to run given the appropriate data inputs.
Task Description:
(task_description
{data perception)
{user input (optional)}
ferror analysis (from episodic memory)
(retrieved documents (from semantic memory)
uan
Code To Generate Prompt for Bash Script Generation
# Truncate error message if needed
if len (error_meeee (error_message) max_ mex__error_aae
error message error_message
error message max_ eeeeeerrrresssee arsgg__ // 2]
" ln... (truncated) \n" error peror_eessage [mmaeeeemaoxrraaaaaa mtweerrrrrossse_eenneeoe. // 2 :]
# Build the core instructions instructions = []
if create_ create_venty venv:
instructions. extend
[
"CCreate and configure conda environment in (outputffoldde folder):",
" Python version: 3. 3.11",
Activate the environment",
" Install required packages",
]
elif mnstall_packagee
( instructions. (
"The environment may not be fully configured. Install any packages required in the python code."
else:
instructions. (
"The environment is already configured. Do not install or update any package."
43

---

instructions. instructions.append "Execute the Python script: {python_ fertton_file___e file
file_path)"" path)")
# Build the prompt with optional context
prompt_parts
"Generate minimal bash script that will:", "An" (ff" {instr}" for i, instr in enumerate enumerate (instructions)),
if current_python:
.aartt..pppndd dedent(
f"*"
Current Python code:
<'' "python
fcurrent_pythonn
) s)r
if error message:
prompt_parts append append (f"Previous error:
if previous_ rrasiou_bbash and error arrormeessage:
prompt_parts.appe
dedent(
f"n"
Previous failed bash script:
< bash
{previous_bast bash)
strrip()
if grevious_python and error peror_message
prompt_parts.co appedd dedent(
f"n"
Previous Python code:
<' "python
fprevious_python?
<cr
s.strip()
# Add final instructions prompt_parter ( (
"""
Notes:
Generate minimal, executable bash script
Focus essential commands only
Handle common environment and package only if there were
44

---

a*. errora
.strip()
return Anlnn.joiie (prompt_pprttttl
B.4.2 Executer
The Executer agent executes generated code and captures real-time outputs with timeout protection. Success determination is two-fold: first checking the script execution completes with a zero return code, then passing execution logs to LLM that evaluates deeper success criteria beyond just exit codes. The Executer agent makes critical "FINISH" or "FIX" decisions with explainations based on log analysis, even overriding apparent successes when detects logical emrors poor performance in the outputs, ensuring both technical completion and task fulfillment before concluding the execution eycle.
Executer Agent
Prompt to Determine Execution is Successful
"""You are expert code evaluator. Analyze the execution results of the following Python code and determine if the execution was successful if issues need to be fixed.
pdask__pooppt/ttta,, prompt)
## Python Code
<< "python
{python_code)
## Execution Results
### Standard Output (stdout)
{stdout or "No standard output")
### Standard Error (stderr)
<c.
fstderr "No standard error")
<cr
Evaluate the execution results and decide on one of the following
actions:
1. FINISH If the execution was completely successful and met all requirements.
2. FIX If there were errors, issues, or performance problems that need addressed.
Provide your decision the following format: DECISION: [FINISH or FIX] ANALYSIS: [Brief analysis of errors if any, or "None" no errors]
The error analysis should be brief but informative enough for another agent to understand what needs to be fixed.
Even if the code executed without throwing errors, it might still
45

---

have issues with logic or not meet all requirements. """
C More Implementation and Competitors Details
C.1 Implementation Details
For MLZero, the code execution environment operates with a 3-hour timeout and a maximum of 5 coding iterations. We enforce strict prompt size constraints: 1.024 characters for files, 8,192 characters for tutorials, and 2,048 characters for user inputs. We leverage Claude 3.7 Sonnet [6] as our LLM backbone, using temperature for planning and file reading tasks, while employing temperature 0.5 for coding. All agents utilize a 65536-token context window with task-specific configurations. For the 8B configuration of MLZero, we adjusted parameters accommodate the smaller context window. Specifically, we set the context window to 8, 8,192 tokens across all agents, reduced retrieval size 3, and limited maximum tutorial length to 4,069 characters.
To ensure fair comparison with AIDE's default setting (def), we standardized its configuration by disabling automatic file copying and utilizing identical LLM backbones [6] for code generation and feedback processes. We maintained a 3-hour timeout and 5-step reasoning process to match our method's 5 coding iterations. Additionally, we also conducted evaluations (AIDE +ext) that incorporates six comprehensive tutorials covering fundamental knowledge for all tasks to facilitate coding with external ML libraries. the enhanced configuration, we also provide the system prompt of MLZero to optimize AIDE's performance.
enhanced version
3-hour timeout but no step limit. For other comparative systems that typically require manual data pre- processing 33 24 [5], we enhanced their capabilities by augmenting them with outputs from data perception module, providing additional data insights that enable end-to-end execution on our benchmark. Since DS-Agent [24] lacks native code execution support, manually executed their generated code to obtain results; thus relative execution times are unavailable as they would not represent true end-to-end agent performance. As both AIDE (38 and AutoKaggle [45] (AK) lack robust mechanisms for saving results through user instructions, we manually extracted results from their working directories for evaluation. We also standardized all description file names accommo- date baseline agents. For agents (Codex CLI, DS-Agent, AutoKaggle) that lack compatibility with Claude 3.7 Sonnet 66 due implementation constraints or outdated interfaces, we substitute with GPT-4o [3, general-purpose purpose model with comparable benchmark performance.
Codex CLI [57] performs end-to-end with minimal human intervention. We similarly
As MLE-bench offers longer time (24 hours) for each agent each dataset, we prompt the MI. libraries 19 655 67 to use best quality preset with a 3-hours time limit per iteration for each dataset. Other implementations in MI MerZero remain the same. For the implementation of AIDE, ML.AgenBBench 333, and OpenHands (72), please refer to MI.E-bench [0]
All experiments were conducted on an AWS EC2 p4d.24xlarge instance equipped with 8 NVIDIA A100 40G GPUs. Computation time varied based on iterations required to achieve success. For the Multimodal AutoML Agent Benchmark, the maximum computational cost for running one agent is 3 hours x 25 datasets x GPUs, totaling 600 GPU hours. The total computational upper bound for Table is 600 GPU hours X 11 agents x runs, totaling 19,800 GPU hours. For MLE-bench, the maximum computational cost for running MLZero once is 24 hours x 21 datasets x GPUs, totaling 4,032 GPU hours.
The operational cost of using Claude 3.7 Sonnet varies with the number of iterations required for success. Approximately, MLZero incurs cost of S0.25 per dataset, while AIDE averages $0.5 per dataset. However, these figures may fluctuate significantly depending on the iterations needed to achieve successful completion.
C.2 MLZero
The default configuration of ML.Zero is attached below. Please check Appendix B for the detailed prompts and implementation of each agent in MLZero, and Appendix D for the ML libraries used.
46

---

MI.Zero Default (def) Configuration
stream output: True per per_execution, tttettmimtuttttt 10800
max max_chars_per_ 1024
max_num_tutoon tututiatoi:ls::
ww__user_input__ae
2048
max_error serroormessage_eeee 1ength: 2048
tew._eessagellenthhh:
max_tuttorial_ennthe 8192
create_ venv: false
condense_tutonn tutorials: True
Default LLM Configuration
For each agent (coder, etc.) you can use a different one 11m: &default_1in,
Note: bedrock only supported in limited AWS regions #
and requires AWS credentials provider: bedrock
model: "us www.aaaiichacc.... claude- --som-et------------------------ 20250219-v1:0" 10 max_
max_tokens: tokens: 65536
proxy url: mull
temperature:
verbose: True multi_ture turn: False
coder: <<: *default_Il temperature: 0.5 top_p:
planner: <<: rdefault_llx 11m
max_stdout_ dedduu_leenghh 8192
max_ max_stderr__enntha 2048
file_reader: file_ reader: <<: *default_ 11m
details: False
C.3 Codex CLI
OpenAl's Codex CLI represents practical implementation of LLM-bbsed tool-using agents for software development. When equipped with strong reasoning models, it demonstrates effeective cetructtoo-followwiii capabilities and can reliably execute software engineering tasks. However, tends to generate simplistic solutions that underperform on complex machine learning workflows. Moreover, its performance degrades substantially when deployed with general purpose language mod- els that lack robust reasoning capabilities, underscoring the limitations of current tool tool-using agents in efficiently handling the sophisticated requirements of and-to-end machine learning automation.
caneral-purpose
In our experiments (Section 4.2). we evaluate Codex CLI in two configurations. The default configu- ration (Codex CLJ) uses OpenAl's GPT-4.1 [56). general-purpose model, ensure fair comparison with other agents that employ non-reasoning models. We also include an enhanced configuration (Codex CLI +rea) utilizing o4-mini (58), reasoning model, demonstrate Codex CLI's upper performance bound. Both configurations employ the following standardized prompt:
47

---

Prompt for Codex CLI
Solve the ml task described folder SCTRAINING_PATHH Do modify any files in $(TRAINING PATH}. All temp or saved files should be located Bomewhere under $COUTPUT _IR). Save the predicted results in the same format as training data to $(OUTPUT_DIN) Name the result file 'results. xxx'' where extension should be same the test data file.
C.4
AIDE
AIDE 38 conceptualizes machine learning engineering code optimization problem, employing tree search methodologies within the solution space. Similar approaches are found in SELA [14] and I-MCTS [46], though these frameworks focus primarily on tabular data and do not provide end-to-end solutions.
For MLE-bench 10 comparisons, utilize the officially reported AIDE results. However, for Multimodal AutoML Agent Benchmark, we implemented several configuration modifications to ensure fair and efficient comparison:
1. We disabled automatic data copying functionality, several datasets in benchmark extremely large, making this process prohibitively time-consuming.
2. Archive unpacking functionality was disabled to preserve the original data structure integrity.
3. Report generation was disabled as increases token and time usage without affecting performance metries.
4. Both code generation and feedback mechanisms changed Claude 3.7 Sonnet, consistent with our system's 's configuration.
To facilitate proper evaluation within our benchmark framework, we implemented additional hard- coded configurations: (1) Description files are automatically fed into AIDE, the system cannot independently locate these files; (2) Output files are renamed from AIDE's default "submission esv" to each dataset's is required filename, AIDE uses hardcoded output naming conventions that would otherwise impede proper evaluation.
For the default AIDE configuration (AIDE def), we incorporated the following additional prompt to
ensure output format consistency and proper file referencing:
Additional Prompt for AIDE (def)
Output column names must exactly match those in the training or sample submission files without adding "predicted_" prefixes or creating any new columns.
Always use absolute file paths when your model prediction outputs reference files.
For the enhanced AIDE configuration (AIDE +cxt), we further augmented the system with our MI.Zero coder's prompt, MI. library prompts (detailed in Appendix B and Appendix D and several critical condensed tutorials from Mi.Zro's knowledge base. This enhanced configuration represents effort provide AIDE with comparable knowledge resources for equitable performance comparison:
Additional Prompt for AIDE (+ext)
[MLZero coder prompt]
[MLZero ML library prompts]
[Condensed Tutorials]
48

---

C.5 DS-Agent
DS-Agent [24] leverages LLMs with case-based reasoning (CBR) solve machine learning prob- lems, particularly excelling in comprehending task requirements and constructing machine learning pipelines. Unlike ResearchAgent [33 focused primarily on generating reasonable plans, DS-Agent implements a two-stage approach:
1. A comprehensive development stage utilizing the full CBR framework to capitalize on expert knowledge from Kaggle competitions and iteratively improve performance through feedback mechanisms.
2. A resource-efficient deployment stage with streamlined CBR to adapt previously successful solutions for direct code generation.
For benchmarking DS-Agent, we utilized official records of the development stage. As DS- Dg-Agent lacks end-to-end execution system, we modified the Multimodal AutoML Agent Benchmark to align with the data requirements [3], then incorporated MLZeroos perception results for each dataset commented paragraphs as the provided skeleton code.
For both default (def) and zero-shot evaluation settings, followed the original implementation without modifications. And we copied and executed the generated code within the Multimodal AutoML Agent Benchmark environment. To ensure consistent evaluation criteria across all systems, we also supplemented the description files with additional standardized prompts to enforce consistent output formatting and behavior.
Additional Prompt for DS-Agent
ONLY save files to: "./".
Make predictions on the test data. Save the predicted results to "./", result file name should be "results", the format and extension should be same as the test data file.
Output column names must exactly match those in the training or sample submission files without adding "predicted_" prefixes creating any new columns.
Tensorflow is not installed. But you pytorch when needed.
C.6 AutoKaggle
AutoKaggle [45] presents a collaborative multi-ugent framework for automated machine learning solutions, comprising five specialized agents (Reader, Planner, Developer, Reviewer, and Summarizer) working in concert throughout the solution development lifecycle. The framework implements an iterative development methodology with comprehensive testing procedures and leverages predefined machine learning tools library. However, a significant limitation is its rigid input data format requirements, it exclusively supports evaluation of tabular datasets from Kaggle competitions within a predefined directory structure. For AutoKaggle to process data correctly, the input files must adhere to the following structural convention:
Required Data Format for AutoKaggle
competition/
train. csv
cav test, csv
sample_submission submission, CSV overview.txt
To accommodate these constraints while evaluating AutoKaggle within our benchmark framework, we implemented several adaptations to preprocess data: renaming descriptions txt files to overview. txt; augmenting training and test datasets with sequential index columns; converting diverse data formats to the mandatory CSV format with standardized train. cav and test.
49

---

filenames; and generating appropriate sample_submission submission csv files that conform to the expected index and target column structure. These adaptations ensure fair evaluation while maintaining the integrity of the original datasets within our comparative benchmark.
D ML Libraries in MLZero
Our system aims to minimize required human effort by leveraging ML frameworks with com- prehensive capabilities rather than specialized libraries. Each integrated framework may contain undocumented issues (mentioned in GitHub issues rather than official documentation), for which we provide supplementary prompts to enhance LLM performance. These prompts were also provided to AIDE (+exi) to ensure fair comparison. The MLZero (-ext) configuration include only general descriptions of machine learning algorithms D.3
MLZero incorporates only few well-maintained ML frameworks that demonstrate state-of-the performance across various modalities, keeping the ML libraries used minimal to avoid "overfitting" the tasks. Our selection criteria prioritize frameworks that provide comprehensive automation capabilities, robust implementation quality, and consistent maintenance to ensure reliable performance. This approach enables our system to handle wide spectrum machine learning tasks with minimal human intervention while maintaining competitive performance. Adding more ML libraries with better performance could further improve the agent's 's performance, but we intentionally limited our selection to prevent overfitting to our evaluation benchmarks.
D.1 Tabular
For tabular data processing, we integrate frameworks with strong performance structured data tasks [19, 40). The time limit and presets parameters are specified for benchmarking purposes but can be adjusted through user input in practical applications.
Tool Registry for Tabular Tasks
"name": "autogluon. tabiular,,,b
"version": "1.2.0",
"description": "AutoGluon Tabular is open- source AutoML framework
that automates the training and tuning of machine learning models for tabular data, handling tasks from preprocessing to model ensembling with minimal code required.".
"features": [
"Works best when there only tabular data (categorical and numerical).",
"Does not work very well on nlp tasks.", "Does not work with image data."
1,
"requirements": 0
"Use Autogluon Tabular with the following parameters:",
" time limit: 1800 seconds",
presets: \\\ "\mmmeduum "idium__ualityyll".
" tuning_data: only use validation if there is a validation dataset.
" ppooleemttppe type:: binary, multiclass, or regression."
}
50

---

D.2 Multimodal
For multimodal tasks, selected AutoMM 667 due its performance in multimodal AutoMI. tasks and comprehensive support across diverse problem types. This framework excels at integrating multiple data modalities into unified representations, providing good performance with minimal configuration requirements.
Tool Registry for Multimodal Tasks
"name": "autogluon.mmltimddall mum.:moddal
"version": "1.2.0",
"description": "AutoGluon Multimodal is an open-source AutoML framework that simplifies the training models across multiple data types including text, images, and tabular data, automating tasks from preprocessing model ensembling with minimal code required.", "features";
"Support multimodal classification or regression, document classification semantic segmentation",
"Does not work the with pure tabular data (categorical and numerical).",
"Does not support image or text generation tasks." "
},
"requirements": []
pprompt_templatee
"Use Autogluon Multimodal with the following parameters:",
"- time limit: 1800 seconds",
" presets:: "medimm "alnniiimuuaaatyyl//l qualityyl\!"",
"- tuninn_dataa data: only use validation if there is validation dataset.",
"The usage of document prediction is different from image prediction."
"Check data path carefully when encounter ValueError: No model is available for this dataset.",
"For semantic segmentation, use single GPU by setting CUDA_ CUDA_VISIBIE VISIBLE DESTBLE_DEVICESO01
"For semantic segmentation, save mask as greyscale JPG image (squeeze then cv2. im2iite)t in [I."predictd_maski output folder and save absolute path label column." "No need to specify model names, and do not increase default per gpu batch size avoid 00M errors.".
folder under
The selection of AutoMM aligns with approach of leveraging well-maintained frameworks with comprehensive capabilities. Its performance across diverse problem types (image-text classification, document understanding, semantic segmentation, etc.) enables MLZero handle wide spectrum of multimodal tasks without requiring specialized implementations for each modality combination. In future work, advanced multimodal ML libraries should be included support additional problem types and to further improve performance.
D.3 Timeseries
For timeseries forecasting, incorporate the MI. library with state-of-the-art performance [65] Note that [19 67 65) share a similar API but internal logics are completely different and thus used three libraries.
51

---

Tool Registry for TimeSeries Tasks
"name": "autogluuo.timeerriees"
"version": "1.2.0",
"description": "AutoGluon Timeseries is an open-source AutoML framework that automates the training and tuning forecasting models for time series data, handling tasks from preprocessing to model ensembling with built-in support for both univariate and multivariate forecasting.", ".
"features":
"timeseries forecasting"
],
"requirements": 0
[
"DO NOT drop any data samples (to make the frequency isregular).",
"Use Autogluon Timeseries with the following parameters:", "- time limit: 1800 seconds",
time_limit:
" presets: \\ "medimu "al""mddumqqualttiii ""meedum_qualityyl""" quality\l\"",,
soming_dataa only use validation if there is valida validationdataset..
"Note that the prediction is given in column named \mmea\" ", You need rename the column in the result." "frrom data_ faaaaffrmeee_ method of TimeSeriesDDttaFrans does not accept 'target' parameter.",
"If there are known covariates, they should be specified in both TimeSeriesPredictor initialization AND predict."
D.4 Retrieval
Our system incorporates FlagEmbedding 774 80 to support tasks requiring semantic search, docu- ment ranking, and retrieval-augmented generation capabilities.
Tool Registry for Retrieval Tasks
"name": "FlagEmbedding",
"version"; "1.3.4",
"description": "Retrieval and Retrieval-augmentar LLMs", "features": [ "retrieval", "reranking"
],
"requirements": 0
"prompt_template""
template": [
"DO NOT SAVE THE MODEL.
}
D.5
Others
When specialized frameworks are not suitable, e.g. image-to-image generation, audio tasks, sequence- to-sequence generation, etc., MI.Zero can fall back general machine learning algorithms, providing greater flexibility for diverse tasks.
52

---

Tool Registry for General MI Algorithm
"name": "machine learning",
"version": "0.1.0",
"description": "You should select this as general reference of machine learning or deep learning algorithms case other tools are not helpful.
"features"; 0
"requirements". 0
"prompt_template"
"In the bash script, install all necessary packages."
}
D.6 Library Integration Process
One of the key design principles of MLZero is the ease of integrating new ML libraries into the system's knowledge base. Adding a new library requires approximately one minute of human effort using provided registration Python script. Users provide only four basic pieces of information: the library name (e.g., "FlagEmbedding"), version number (e.g.. "FlagEmbedding=0xxxx" sentence description, and the path to the directory containing the library's ' official documentation. Users may optionally provide additional prompts or usage guidelines, though this is not required. Importantly, coding skills are needed for this registration process.
one-
After this brief human setup, the system operates completely autonomously. The Summarization and Condensation agents automatically process the provided documentation into structured, condensed formats suitable for the Semantic Memory module. This automated processing includes extracting relevant sections from official documentation. using LLM agents condense lengthy tutorials into essential information, organizing information by topics and APIs, and building retrieval indices for efficient semantic search. While this processing requires computational time (typically 10-30 minutes depending on documentation size), it requires zero human intervention or supervision.
Once a library is successfully integrated into Semantic Memory, MLZero operates from raw data to final predictions without further human input. This stands stark contrast to many existing AutoML systems that require manual preprocessing, hard-coded logic tailored to individual datasets, expert configuration of hyperparameters, or post-processing extract results. Our experiments demonstrate that this design achieves true end-to-end automation while maintaining extensibility across diverse modalities and problem types.
E
Detailed Results
E.1 Detailed Results on Multimodal AutoML Agent Benchmark
This appendix provides detailed breakdown of all experimental runs conducted with different agents
and configurations on the Multimodal AutoML Agent Benchmark in Section 4.2 Each table presents
the complete results across three independent runs for each dataset in the benchmark suite: MLZero def in Table MLZero 8B Table MLZero -ext in Table MLZero -epi in Table I0 Codex
CLI def' in Table IT Codex CLI +rea in Table 12] AIDE def in Table 13 AIDE +ext in Table 14
DS-Agent def in Table 15] DS-Agent zeroshot in Table 16 and Auto def in Table 17
For each rum, we report the performance metric (see Appendix A.I) and the computational time seconds required to complete the task. The symbol indicates runs where the model failed to complete or with invalid outputs. These detailed results complement the aggregated performance metrics discussed in the main paper and demonstrate the consistency and robustness of our approach across multiple executions.
53

---

Table 7: Three runs of ML.Zer (def) Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Datasct
Result
Time (s)
Result
Time (8)
Result
Time (8)
abalone
2.135
142
2.120
111
2.135
112
airbnb melbourne
0,426
1202
0,428
998
0,426
1041
airlines
0.656
1285
0.658
442
0.656
1262
bioresponse
0.801
3050
0.814
497
0.801
167
camo sem seg
0.833
4089
0.839
2192
0.843
6358
cd18
0,445
215
0.495
501
0.445
261
climate lever
0,476
649
0.471
342
0.479
3719
covertype
0.976
1963
0.976
2634
0.976
4556
electricity hourly
-1,436
875
11.27
1848
-.393
2779
europeanfoooddepth
0.680
524
0.690
615
0.690
509
fiqabeir
0.482
0.507
0,505
889
gnad10
0.837
717
0.911
725
0.829
722
ham 10000
0.555
815
0.555
771
0.779
1049
hateful meme
0.571
682
0.587
663
0.600
628
isic2017
0.751
4139
0.758
8440
kick_starter_funnt
0.438
1330
0.471
0.435
1027
memotion
0.503
3852
0.483
0.514
3807
mldoc
0,956
3699
0.949
3588
0,948
2836
nn. ddally_iithuut_mmseeee
-0.765
251
-0.765
485
X
X
petfinder
0.387
701
0.387
747
0.397
791
road segmentation
0.466
2069
rvl rvl_edip
0.871
2979
0.872
2868
solar 10 minutes
-2,273
-0.704
X
X
women, clothing review
0.748
958
0.748
1068
0.751
935
yolanda
8.533
2066
8,533
2007
8,533
1902
Table 8: Three runs of MI.Zero (8B) on Multimodal AutoMI. Agent Benchmark.
Run
Run 2
Run 3
Dataset
Result
Time (s)
Result
Time (s)
Result
Time (s)
abalone
2.087
385
2.087
353
2.087
374
0.428
3857
0,414
3054
X
X
airlines
0.731
3939
x
0.656
6638
bioresponse
0.797
550
0.797
2113
camo. sem sem_seg seg cd18
X
-08557
1061
0.445
506
X
x
climate fever covertype
X
X
X
X
0.976
7634
X
0.974
8769
clectricity_ hourly
curopeanflooddeethh fiqabeir
0.695
738
X
X
X 3451
gnad10
0.850
3453
0.843
2574
0.843
ham 10000
0.569
1703
0.569
1716
X
X
hateful meme
0.607
2060
X
0.531
5122
isic2017
X 3382
fick_starter_fundee funding
0.327
3309
0.438
4400
0.466
memotion
X
X
X 7009
midoc
0.947
4236
0.948
5422
0,949
nn5 _n5_daily without wiisiou
X
X
X
X
petfinder
0.405
3355
0.389
3224
road seggmmmtmtttiion
rvl. rvl_edip
solar 10 minutes women caaeee_clothingnneee review
X
X
X
0.724
5132
X
0.503
2337
yolanda
8.533
7263
8.540
8669
8,533
4141
54

---

Table 9: Three runs of MLZero (-ext) on Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (s)
Result
Time (s)
Result
Time (s)
abalone
2.218
165
2.196
352
2.158
290
airbnb melbourne
0.398
329
0,087
797
X
X
airlines
0.642
422
0.618
234
0.644
160
bioresponse
0.877
659
0.800
333
0.801
145
camo sem sem_seg seg
0.464
X
X
cd18
-0.634
223
4,145
2934
0.078
277
climatefevee
X
X
0.231
849
0.253
2808
covertype
0.887
688
0,876
120
X
X
electricity hourly
1.753
848
1.755
10422
europeanfoooddepth
0.583
335
0.707
347
0.496
843
fiqabeir
X
0.225
4573
X
X
gnad10
0.883
0.872
219
0.873
184
ham ham10000 10000
0.671
6045
X
X
X
hateful meme
0.382
3521
0.310
638
isic2017
X
fik__starter_fuddiinn
0.442
0.335
0,303
438
memotion
0.655
4394
0.999
mldoe
0.961
1524
0,928
872
X
X
nn5
X
1,459
8994
0.823
165
petfinder
0.357
338
0.385
1803
0.393
3340
road segmentation
0.444
10253
0.179
3866
rvl rvl_edip
0.886
4829
solar 10 minutes
X
X
X
women, women_clothing review
0.617
225
0.659
390
0.697
809
yolanda
8.892
290
8,931
307
8.974
338
Table 10: Three runs of MI.Zero (-epi) on Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (s)
Result
Time (s)
Result
Time (s)
abalone
2.135
115
2.120
107
2.135
265
airbnb_meei menobuure
0.423
3217
0,415
901
0.428
1086
airlines
0.656
1232
0.658
406
0.656
1215
bioresponse
0.797
178
0,801
174
0.797
197
camo sem seg cd18
0.841
2239
0.835
4304
0.840
6320
0.445
166
0.495
494
0.586
526
climate fever covertype clectricity hourly
X
0.476
3001
0.475
6657
0.976
2652
0.976
1924
0.976
1948
-1.406
848
-1399
1012
-1.393
781
europeanfRooddetth
0.680
504
0.680
541
0.690
475
fiqabeir
0,376
0.494
0,515
192
gnad10
0.837
709
0.837
0.829
690
ham 10000 hateful meme
X
X
0.555
752
0.779
669
0.571
648
0.587
653
0.600
608
isic2017
X
kick_stard sis_sttattr_funding
0.438
2437
0.438
742
0.438
1226
memotion
X
0.507
2696
0.514
2462
midoe
0.948
1117
0.948
2970
0,946
4681
nn5 Aaily_withoutt missiin
-0.765
219
-0.764
201
-0.765
211
petfinder
0.387
714
0.387
743
0.397
757
road_ seddssmemetttion
0.597
2047
0.601
4903
rvl__ct wvl_edip
0.871
2461
0.871
2570
solar 10 minutes women women_clothing review
-1.286
X
X
0.748
1073
0.657
590
0.751
957
yolanda
8.533
1908
8,533
1983
8.533
2010
55

---

Table 11: Three runs of Codex CLI (def) on Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (8)
Result
Time (s)
Result
Time (s)
abalone
x
x
2.230
47
x
X
airbnb melbourne
X
X
X
airlines
X
bioresponse
0.792
30
camo sem sem_seg
X
X
cd18
-0.412
85
-1.477
173
X
X
climate_evve
X
X
X
X
covertype
X
0.955
48
X
X
electricity hourly
X
europcanfoooddepth
fiqabeir
X
X
gnad10
X 0.790
0.847
159
ham 10000
0.484
36
X
X
hateful meme
X
isic2017
fis_ssttter__unddinn
X
X
X
memotion
X
0.532
92
mldoe
X
X
0.094
114
0,553
85
nn5 mod__daly_withhut_miiiiil missing
X
X
X
X
X
petfinder
X
road segmentation
X
rvl rvl_edip
X
X
X
X
solar 10 minutes
X
X
X
X
X
women, women_ccottine clothing review
X
X
X
yolanda
X
X
X
X
Table 12: Three runs of Codex CLI (trea) on Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (s)
Result
Time (s)
Result
Time (s)
abalone
2,232
201
2.232
93
2.351
299
harbnb_melbourgy
0.384
85
0.388
108
0.384
86
airlines
0.653
175
0.612
177
0.628
81
bioresponse
0.882
163
0,800
90
0.880
117
camo sem_ seg cd18
81
X
162
-1.288
-1.685
-1,362
125
climate fever covertype clectricity hourly
0.123
1029
0.233
165
0.231
155
0.957
84
0.955
90
0.841
75
75
101
europeanflooddepth fiqabeir
0.435
219
190
100
0.199
204
0.199
120
gnad10
0.848
116
0.845
0.845
131
ham 10000 hateful meme
0,511
471
0.451
80
0.451
75
0.572
119
0,454
110
0.403
61
isic2017
0.111
473
440
391
kick_starte sis_sttartr_funding
0.362
109
0.362
72
0.285
76
memotion
0.702
95
0.819
152
midoe
0.933
221
0.788
0.739
172
nn5 Aaily_withoutt missiin
X
X
X
166
X
149
petfinder
0.396
115
0.396
108
0.396
67
road_ seddssmemettiion
131
154
rvi__ct evl_edip
128
147
solar 10 minutes women women_clothing review
X
-1.88
221
0.419
73
0.029
157
0.589
127
yolanda
9.091
135
9.598
86
9,598
75
56

---

Table 13: Three runs of AIDE (def) Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (8)
Result
Time (8)
Result
Time (s)
abalone
2.182
483
2.095
234
2.188
445
airbnb melbourne
0.394
174
X
airlines
X
bioresponse
0.868
434
0.877
933
X
camo. seg
X
X
cd18
X
X
X
X
climate_fever
X
X
X
X
X
covertype
X
X
0.880
197
electricity hourly
X 4365
curopeanflooddepth
0.722
2170
0.693
liqabeir
X
X 2912
gnad10
X
0.908
10812
0.897
ham 10000
0.809
1492
X 0.467
0.812
3839
hateful meme
0.557
2672
1907
X
X
isic2017
kick_starter_funet
X
X
memotion
0.471
1050
X
X
mldoc
X
0.965
10801
X
X
nn5 _awald__wihhhu__miii missing
X
X
X
X
petfinder
0.336
652
road segmentation
rvl_edip
X
solar 10 minutes
-1.054
2698
X
X
X
women, women_clothinne clwthing review
X
X
X
X
yolanda
X
X
X
X
Table 14: Three runs of AIDE (+ext) on Multimodal AutoMI. Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (s)
Result
Time (s)
Result
Time (s)
abalone
2.189
385
2.135
651
2.230
469
airbnb_meelae men_uueelorree
0.415
2689
X
0.363
329
airlines
0.655
3771
0.643
155
bioresponse
0.876
7400
0.801
309
camo. sem sem_seg seg
X
X
cd18
0.239
248
X
-0.335
113
climate fever covertype
X
X
X
X
X
0.744
7856
0.974
7415
0.860
182
cectricity_hourfe
X
curopeanflooddeethh fiqabeir
0.697
730
0.680
3784
0.659
1654
X
X
X
gnad10
X
X
0.585
309
ham 10000
0.785
813
0.796
5001
0.847
8672
hateful meme
0.518
2045
0.501
10802
0.442
637
isic2017
X 0.438
fick_starter_fundea funding
0.440
7496
1970
memotion
X
X
midoc
0.948
1082
0,925
195
nn5 daily_wiihhel missoin
X
X
X
X
X
petfinder road seggmmmtmtttiinn
0.382
2359
0.367
679
0.390
171
X
rvl. rvl_edip
solar 10 minutes women caaeee__lothingneeee review
X
X
X
0.747
1134
X
X
X
X
yolanda
8.540
1953
X
X
9.032
172
57

---

Table 15: Three runs of DS-Agent (def) on Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (8)
Result
Time (8)
Result
Time (s)
abalone
2.238
19
x
x
X
airbnb melbourne
X
X
X
X
airlines
X
bioresponse
0.790
4
0.793
4
camo sem sem_seg
X
X
cd18
-1998
137
X
X
X
climatefevee
X
X
X
X
covertype
0.957
110
X
X
X
electricity hourly
X
europeanfnooddepth
fiqabeir
X
X
X
gnad10
X
X
X
X
ham ham10000 10000
X
hateful meme
X
isic2017
X
kick_starter_funne
X
X
X
memotion
X
X
mldoe
X
0.949
1265
X
X
nn5
missing
X
X
-4662
6
X
X
petfinder
0.355
9
0.350
0.369
road segmentation
X
X
rvl rvl_edip
X
solar 10 minutes
X
X
X
X
women, women_ccothiinee clwthinggtteeeeeeeee review yolanda
X
X
X
X
X
X
X
X
Table 16: Three runs of DS-Agent (zero-shot) on Multimodal AutoMI. Agent Benchmark.
Run 1
Run
Run
Dataset
Result
Time (s)
Result
Time (s)
Result
Time (s)
abalone
2,362
2
X
X
X
X
X
X
X
0.314
16
airlines
0.612
61
X
X
X
bioresponse
0.793
4
0.793
4
camo sem__ssse seg ed18
X
X
X
0.338
2
X
X
-1.615
climate fever
X 0.952
X
X
X
X
covertype
99
X
X
cestricitt_hourty
X
11.662
10
europeanfloooddepth fiqabeir
X
X
X
gnad10 ham 10000 hateful meme
0.691
0.903
304
X
X
X
X
X
X
X
X
X
isic2017
X
memotion
X
X
midoc
X
X
X
X
X
nn5 3_5_ddil__ma without wiissoottttmiiinngggggggo
X
X
X
X
X
petfinder
0.389
4
0.214
0.207
12
road_segee seggmenttnnaiinn
X
rvl. rvl_edip
X
solar minutes
X
X
X
X
women ciaieen_oothinnnnee review
X
X
X
0.353
18
yolanda
X
X
X
X
X
58

---

Table 17: Three runs of AutoKaggle (def) on Multimodal AutoML Agent Benchmark.
Run
Run 2
Run
Dataset
Result
Time (s)
Result
Time (s)
Result
Time (s)
abalone
X
Burbnb_melbonn
0.253
2886
0.314
2409
0.383
1165
airlines
X
X
bioresponse
X
X
X
X
camo_sem_sez
X
X
X
X
cd18
-1.840
1972
X
X
dlimatefeverr
X
covertype
0.941
9284
X
electricity_ hourly
X
X
X
X
europeanffoooddethh
X
X
X
0.583
2017
fiqabeir
X
X
gnad10
0.105
11281
ham10000
hateful_meme
0.337
1732
0.382
1945
X
isic2017
X
X
X
sttrttttttrrrree funding
X
0.237
2935
memotion
X
X
midoc
X
X
nn5 baill__ithout_mmstteee
petfinder
0.388
1881
X
X
X
gond_ssgmentation rvl edip
X
X
X
X
X
X
X
X
X
solar 10 minutes ciannn_coohhiggreviee
X
X
X
X
X
X
yolanda
X
E.2 Detailed Results on MLE-bench Lite
This appendix presents the comprehensive experimental results that supplement the analysis on MLE-bench Lite provided in the (4.tion[4.2lor main paper. Table [8 offers detailed comparison of our method against three state-of-the-an baseline approaches across all 21 datasets in the MLE-Bench Lite benchmark. Note that for metrics where lower values originally indicated better performance, we applied a negative sign to convert them, ensuring that higher values consistently represent better performance throughout the table. See Appendix A.2 for details about MLE-bench Lite. Gold, silver, and bronze highlighting denote the type of medal gain for each dataset, while underlined values indicate performance above the median. The symbol 'X' represents cases where methods failed to produce valid solutions within the allocated time constraints or encountered critical errors. As multiple datasets
demonstrated in both subtables, our approach achieves superior performance across various machine learning tasks, confirming the findings summarized in the main paper. These detailed results provide additional evidence of our method's robustness, versatility, and consistent performance advantages over existing techniques in automated machine learning systems.
E.3 Detailed Error Analysis
To better understand the limitations of current AutoML systems, we conducted a comprehensive error analysis across MLZero, Codex CLI. AIDE, and DS-Agemi (DS). Table presents the frequency of distinct error types observed in the final iterations each system's execution.
MLZero exhibits exceptional robustness, encountering minimal errors (6.5% overall) limited to algorithm implementation (4.3%) and preprocessing stages (2.1%). The algorithm implementation errors occurred specifically had insufficient knowledge of relevant MI. libraries. The preprocessing errors manifested exclusively on image-to-iman tasks with heterogeneous input resolutions. In contrast, competing methods demonstrated substantially higher failure rates: DS DS-Agent (76.0%), AIDE (47.8%), and Codex CLI with reasoning (26.9%).
sequence-to-squeees and audio classification tasks where the system
59

---

Table 18: Methods Performance on MLE-bench Lite. Gold/silver/bronze highlight the gold/sil- ver/bronze medal, underline denotes above-median Note that for metrics where lower values originally indicated better performance, we applied negative sign convert them, ensuring that higher values consistently represent better performance throughout the table. See table 6 for the
dataset details.
Method
D1
D2
D3
D4
D5
D6
D7
D8
D9
DI0
DI1
Ours
1.000
0.904
X
0.936
-0.442
-0.008
0.998
0.985
-0.242
X
-5.111
AIDE
1.000
0.855
X
X
-0.694
-0.817
0.996
0.903
-0.801
X
-5.463
MLAB
0.943
0.712
X
0.852
-4,800
-12.759
X
0.953
X
X
-10,022
OD
0.495
X
-0.220
0.884
X
-0.426
0.853
0.971
-0934
X
-1053.080
(a) Performance on Datasets from DI to D11.
Method
D12
D13
D14
D15
D16
D17
D18
D19
D20
D21
Ours
-0.059
0.990
0.787
0.673
-0.384
0.963
0.960
X
0.958
0.625
AIDE
-0.069
0.962
0.642
0.859
-0.426
0.958
0.899
0.991
X
0.869
MLAB
-0.063
0.817
0.500
0.421
-0.555
0.943
X
X
X
X
OD
-0.542
0.494
0.684
0.635
-0.563
0.958
X
X
X
0.914
(b) Performance Datasets from D12 to D22.
DS-Agent encounters difficulties at foundational stages, including perception (24.0%) and preprocess- ing (20.0%), indicating systemic failure in data comprehension and requirement translation, despite being provided with data perception context from MLZero. Its high algorithm implementation error rate (28.0%) likely stems from the substantial domain gap between deployment and development datasets.
AIDE's predominant failure mode is API hallucination (28.2%), wherein the system attempts to utilize non-existent or incorrectly implemented library functions. This underscores the necessity for external knowledge of ML. libraries rather than exclusive reliance parametric knowledge embedded LLMs.
Notably, MLZero completely eliminates several error categories, including perception (0.0%), API hallucination (0.0%), and postprocessing (0.0%). This achievement can be attributed our dual- memory architecture, which effectively anchors the LLM's generations in factually accurate external knowledge while maintaining execution consistency through structured episodic memory. The preprocessing errors observed in MLZero result from the system utilizing example file metadata to construct solutions for datasets with variable input resolutions across images, representing conscious trade-off between context length compression and preservation of critical information in file grouping and perception.
is important to note that this analysis focuses on final iteration errors and does not necessarily indi- cate error propensities intermediate stages. For instance, AIDE's higher apparent API hallucination errors compared to DS-Agent may reflect AIDE's attempts at implementing more sophisticated solu- tions after successfully navigating earlier stages that DS-Agent fails to complete. Additionally. AIDE exhibits unreported errors (8.7%) in the assessments.
evaluation, potentially influencing comparative
These findings highlight the significant advantages our integrated approach, which systematically addresses error-prone areas of the ML workflow through specialized perception agents, structured knowledge retrieval, and iterative refinement with targeted error correction mechanisms.
E.4 Continual Improvement Beyond First Success
While the default MI.Zero configuration employs an early stopping strategy (terminating upon first successful execution), we investigated the system's capability for iterative refinement beyond initial success. We evaluated two configurations across eight challenging datasets covering all modalities: the default configuration with early stopping upon first success (def), and an extended
60

---

configuration allowing up to 5 iterations regardless of initial success (def + nostop). We also evaluated both configurations the challenging setting without specialized external libraries (-ext) assess improvement potential when the first solution is typically suboptimal.
Table T9 presents detailed results showing that the extended configuration demonstrates clear capa- bility to refine successful solutions. Notable improvements include camoseg accuracy improving from 0.84 to 0.86 (+2.4%), flood from 0.69 0.74 (+7.2%), ), and solar RMSE dramatically reducing from 1.49 to 0.38 (-74.5%). The improvement pattern is more pronounced in the -ext setting where initial solutions are weaker due to lack of specialized libraries, with camoseg improving from 0.46 to 0.55 (+19.6%) and petfinder from 0.38 to 0.44 (+15,8%). These results validate that optimizing an existing working solution is generally easier than achieving the first success.
Table 19:
Continual improvement results. Allowing iterations beyond first success improves perfor- but significantly increases computational cost. Metrics: higher is better for classification tasks; lower is better for regression tasks (marked with L). X indicates failure.
Dataset
def
def nostop
-ext
-ext + nostop
camoseg
0.84
0.86
0,46
0.55
flood fiqa
0.69
0.74
0.60
0.62
0.50
0.51
0.22
0.23
mldoc
0.95
0.96
0.94
0.96
petfinder
0.39
0.38
0.38
0.44
rvledip
0.87
0.87
0.89
0.88
solar (10m)
1.49
0.38
X
X
yolanda..
8.53
8.54
8.93
8.75
Avg Tokens|
63k
186k
51k
146k
Avg Time (s)
1904
5388
1731
4520
Despite performance gains, continual improvement incurs substantial computational overhead. Token usage increases 3 from 63k 186k tokens per dataset in the default setting, computation time nearly triples from 1904s to 5388s per dataset, and costs increase proportionally due to additional LLM calls. This overhead occurs because datasets that succeed in the first iteration still complete all 5 iterations rather than stopping early, continuing to explore alternative implementations even when initial performance is already strong.
Given these findings, we chose the early stopping strategy for the main paper to prioritize efficiency while maintaining high performance (92% success rate). The modest improvements from continual iteration (typically 2-7% gain) do not justify tripling computational costs for most practical appli- cations. However, for scenarios where computational resources are not a constraint and optimal performance is paramount, the continual improvement capability remains valuable. As demonstrated Table I extending the time limit 24 hours with the best best_quality preset further improves performance, approaching expert-level human results on several datasets.
E.5
Examples of Generated Code
E.5.1
Code generated by MI.Zero def on the Hateful Meme Dataset
Evaluation Score (fl): 0.5868
Time Used: 663 seconds
Hateful Meme Classification using AutoGluon MultiModal
This script uses AutoGluon's MultiModalPredictor to classify memes as hateful (1) not hateful (0)
based both image content and text. The model processes image pairs make binary predictions.
Installation requirements: pip install autogluon multimode
61

---

pip install pandas
pip install numpy ann
11
12
import o8
13
import pandas pd
import numpy as np
import datetime
16
from autogluon multimodal import MultiModalPredictor
17
18
def
(df, base_path);
e** Expands relative image paths to absolute paths
20
21
22
Arga:
df: DataFrame containing image paths base_path: Base directory for images
25
26
Returns:
27
*** DataFrame with expanded image paths
29
df copy
30
Convert image paths to absolute paths if they are not already if 'ing' in columss: df ['img' ('ig' apply (lambda os. path join (base Latasee_ahh___ if not os path. isabs (x) else x)
31
df
return df
34
35
def preprocess aate datainiaittatad test_df):
37
Preprocesses training and test data
38
39
Arga:
train_dff df: Training DataFrame test_df: Test DataFrame
42
43
Returns:
44
Preprocessed training and teat DataFrames
***
Remove unnecessary index column present
47
if 'Unnamed: 0' in train_de coluens:
48
train_de df train df df dropd Unnamed: o'])
if Unnamed: 0' in test_df columns:
test_df test_df. drw..ddddp.....o.. wwwpppooluuss-a....oooooo UUnnameed O'])
51
52
# Drop rows with NA values in training data
53
train_df = train_df dropna (subset=["labe)) '])
return frain_df, test_df
56
57
def train_model (train_datar nodel_save_pathh
58
Trains MultiModalPredictoo model
60
61
Args:
60
train_ data: Training data DataFrame
model_save_path: Path to the trained model
65
Returns:
66
Trained MultiModalPredictor
67
Initialize and train the model
predictor MultiModalPredictorr label='label', path-model_save save_path____
70
71
72
62

---

problem_typer binary', "typer'biiary', Binary classification task
75
Train the model with medium quality preset and time limit predictor fit(
76
77
train_ beain_data-train.de data, time time_ 1im0t==1000,, 30 minutes presets="aedium_que quality"
79
80
81
82
return predictor
84
def make_predictions (predictor, test_data, output_path):
85
86
Makes predictions test data and saves results
Args:
89
predictor: Trained Mult:ModalPredictor test_data: Test data DataFrame
90
91
output_path: Path to save predictions
***
93
Generate predictions
94
predictions predictor predict predicct
_atet
95
Create results DataFrame with the format test data results test_ data copy results ('label')))) = predictions
98
99
100
Save predictions to CSV
results_path 08 path. join (output_path) "results.cav" results. to_ cav (results_path, inder=Palse)
join
103
104
print Prediiictio saved to (results_path)")
__name_.. "___main_...
Define paths
108
output_dir "/media/ agentsomeet3__hatee hateful meme_ " makedid (output_dir, exist_ox=True)
109
111
Create a timestamped folder for the model timestamp = datetime datetime.non model_ os.path. p.t.aah..ol. (output_dir, ("model__ttieestampp>> os makedirs (model_dir, exist_ok=True)
112
116
# Load data crain_data__. training/train.c cay
= //aaa///gee///////tattttttttt///
test_ best_data_path
= //aat///eer/aaaaaaaaatttteeeeeee 119
iinference.cs"
training"
train_df df pd. read cav (awanitnnnndattaappaaaaa test_df pd. read_ tettte_ddtt__ptttt
122
Preprocess data train_df, test_df greprocess_dataateee
test_df)
126
127
Expand image paths
train_dt df expand image aanage_pathet (train df, image base base__athh test_df expand_imaan ieagiiiaaaee_ile (teat_df, image_base_
image_base_ppe
131
print (I"Training data shape (train_dt df, shape}") print Te"t data shape {test df shape}")
132
63

---

Train model
predictor train_ (train_dff df, model_ dir)
136
137
# Make predictions
pake_predictions (predictor, test_df, output_dir)
print ("Hateful meme classification completed successfully!!
141
E.5.2 Code generated by Codex +rea on the Hateful Meme Dataset
Evaluation Score (f1): 0.4538
Time Used: 110 seconds
!//ur/ h/t/us//bi//eeee// env python3
Baseline script for Hateful Meme classification using TF-IDF and Logistic Regression.
Loads training data, trains text-only model, and predicts inference data.
Saves predictions to treaults.cav in the current directory.
import
import pandas as pd
from sklearn wwlearn.feature_e... feature_ text import TfidfVectorizer from sklearn linear_ linear_model model import LogisticRegression
10
def main():
13
# Paths to data
14
train_path * " h/cct///geernaa/eattteeeettttttd
15
train.cav
test_path
16
inference.css cav"
# Load datasets
17
train_df df = pd read_ (traai_pathh
test_df rdad (test_pata)
20
Text vectorization
21
vectorizer = TfidfVectorizer (max_feature==5000, ngram_range=(1, 2)
22
stop_words= )
X_train vectorizer, fit wtt.::fftttttffrrreeeeeee cannffforttrainddee (train_ ttex astype aat ['lbbll')
y y_train
24
25
# Train classifier
clf (aa.aaicccaareeeoooooooooooa fit(XXtrain, y_train)
nax_iter=1000)
28
29
Predict on test data
30
x test vectorizer transform (test_de [' '] asty)e
preds clf pprdict(X_ttes)
33
Prepare and save results ext patp. output_path os.hath join (os (), f"results(ext)"" result_df DataFaarae ({ 'id': test_df("id'] 'label': preds
34
path splitext (test_path)
37
38
39
})
result df to_cav (output_pppp paatp index=False) print (f Saved results to
42
43
if __mame_.. == '___ain__" main ()
44
64

---

E.5.3 Code generated by AIDE def on the Hateful Meme Dataset
Evaluation Score (fI): 0.5573
Time Used: 2672 seconds
import
import pandas as pd import numpy as np
import torch
import torch.nn
import wtrch.utils.dde data import Dataset, DataLoader
from torch.
from torchvision import transforms, models
from transformers import DistilBertTokenzeer, DistilBertNodel
from PIL import Image
from sklearn Anlearn.model_eee mwdel_ import StratifiedKFold
11
from eetearn.metrice. import accuracy_score fi_score
12
import warnings
13
warnings filterwarnings ("ignore"
15
16
Set device
17
device torch. device ("cuda" if torch cuda. as else "cpu") print (f (f"Using device: (device)")
18
20
Load data arain_df = pd. read_cev(" (aaaee...i/in///aaaeeeaaaa wttppt/ttttattncnettetee
21
22
test_df = pd read_ (ttp://ppppt/nffferccceceo
Check label distribution
25
print Lf"Label distribution: (train_de [aaii_dd[['aal ['label '] value, vallleeccouuttteeeeeeeeeeeaeeaee counts ())")
26
27
Define image transforms transfors transforms Compose
29
30
transforms Resize ( (224, 224)),
31
transforms ToTensor oonensor (),
transforms Normalize (mean [0. 485, 0. 456, 0, 0.406], std [00=0[o.00 0 229 0.224, 0.225]),
34
35
Initialize tokenizer tokenizer DistilBertTokenier: fron_pretrained ("distilbert basee- uncased")
37
38
Define dataset class class KultimodalDataset (Dataset):
41
42
def __ (self, dataframe, tranaform=MMone, is 1s_teat=False):
43
self dataframe dataframe
self. transform tranaform
self.is_tes test
46
42
def __len__ (self): return len (self dataframe)
48
50
def _..getttmm__et.... (self, idx):
51
row aelf dataframe iloc [idx]
52
# Load and transform image
img_path os os. join ('//inuut", row cow["ing"] try:
55
56
image Image open (img-path) convert ("RGB")
57
if self transform:
65

---

image self. transform (image) except Exception
print Errroor loading image {img_pathh; {e}")
61
Create a blank image if there's an error image torch. zeros 224, 224)
62
63
# Tokenize text text str (row (row encoding tokenizer
66
67
text,
68
return_ tersors=sess""""
padding="max_les max_ truncation=True, max_length-128,
70
71
72
74
# Get input_ids and attention, caantinnmmmak mask
75
inppt__iiin ids ancoding["ing" ids"] squeeze (0)
_ds
76
attention_mast encoding aatatteetioonnatt"" squeeze (0)
77
if self isstest: return {
79
80
"id": rov["id"], "image": image,
81
"input_is"" input_ide
"attention mask": insention_mskk
84
}
85
else:
86
return {
"id": [owd"] "image": image, "input_ids" input_ide,
89
90
"attention_man : attention actention_mask,
"label": torch. tensor (row ["label"), dtype-
93
94
95
Define model
class MultimodalModel (nn. Module):
def __init___ __init__ (self): super (KultimodalModel,, self) init__()
98
99
# Image encoder (ResNet50)
self. amage_encodee for param in self image_ encoder encoder..prameterss]] param. requires_gresr
models reenttt
parameters ():
103
False
104
self. www.iinnneeeddeeee in features, 512)
nn Linear (self imaiemmgeee__ee encoder fc.
106
# Text encoder (DistilBERT) self _oxt_enccder = DistilBertModel from_pretrained ("
107
for param self talfftext_eene tex___node_ wtw.enede..prrammteee...e.eee
parameters ()
param. requires_
False
110
self telf.text_pooler = nn. Linear (768, 512)
# Classifier
self classifier nn Sequentiai nn Linear (1024, 512),
115
ReLU(),
Dropout (0.3).
Linear (512, 256), nn. ReLU(,
119
nn mn. Dropout (0.3)
120
nn Linear (256, 2)
66

---

def forward image, input_ids attention_mashl
124
# Image features imagu_features self image_encoded (image)
125
# Text features
text output self text_encoders
129
input_ide=inpt__de;
130
text featurea text eext_output. last last_hidden,statell. 0, :1 Use
[CLS] token text_features self text_ (teoterrteetfffttuuuee
133
134
# Concatenate features
torch. cat cat.:aaattaammamaaa ((wmage__eeatuueeeeeeeeeeeeeeeee text text_features).
dim=1)
137
# Classification
output self classifiers (cmmorredddeeeeee return output
141
142
Training function def train_modes (model, crain_loader, optimizer, criterion, device) model. train total_lose loss
145
146
147
for batch in train_loader:
image batch("image (dvvic)))
input_ide ids batch [" (mippuu___ ["input_ids"" ids to (device)
attention maakk batch latteetionnnnnnntoooomoo mask"] ".. to (device)
151
labels = batch ["label"] to (device)
optimizer zero.grad
outputs model input_ide attention,maske mask)
155
loss criterion (outputs, labels)
155
loss backward
optimizer (tep
total_ loss += Loss.iter ()
160
return total_loss len(train_loader)
164
Evaluation function
165
def ialluate_zodell (model, val loader, criterion, device):
model eval
total loss =
all_preds = []
169
all_labels []
with torch. no_grad():
for batch in val_loader
173
image batch(" (t (device)
174
input_ide ids batch (" inint "]. to attention mask batch("attention mask to (device)
" (device)
labels batch ["label") to (devicc)
177
178
outputs model (image, input_ ids, attention_mask)
loss cnttrioonouuputta labels)
total loss loss. item()
182
183
-' preds torch, max (outputs, 1)
67

---

all _i__preds.. extend (preds cpu () numpy ())
all labela extend (labels. cpu numpy ())
187
accuracy accuracy_score fall_la all all_preds)
188
f1 fi_score (all_labels, all_preds,
189
total loss len sen(val_ooade) accuracy, f1
192
193
Prediction function
def predict (model, test__oader, device): model. evall)
196
predictions []
197
ids = []
198
with torch. (o)gra for batch in test test_ loader:
201
image batch ("image" ] to (device)
202
input_ids batch ("input_idss to (device)
203
attention_mask batch (ratthctiinnnnnncoo maskiiionmmss__]]___ to (device)
batch_ids batch ("id"]
205
206
outputs = model (image, input_ ids, attention_mase preds torch max (outputs, 1)
207
predictions extend cpu () numpy )))
210
ids extend bbatch_ids numpy ())
211
212
return ids, predictions
215
Cross-validatin setup splits = skf 42)
216
shuffle-True, random state
219
Hyperparameters
220
batchhssiee size 16
num aum_epochs learning_rate rate 1e-4
223
224
Lists to store metrics accurrcasieees =
cv_fi_scores
227
228
For storing test predictions from each fold test_predictions []
229
Create test dataset and loader
232
gost_datases MultimodalDataset (test_df, df, saansoor=iingg_ttrnnttt is _ test=True)
233
test loader DataLoader (test (tasstdatttttdds batch www.:siieeebchseeee size, shuffle- False)
235
Cross-validatie loop
236
for fold, (train idxx val_idx) in Coumerate(skf split (ariitttrinndd
crain_df["labelt ffold 1t/fn_splits)"")
238
239
# Split data
240
train_data data train df iloc [train, idx]. reset reset_index (drop= val_data train_df iloc _voc_val_iea reset_iieeeeee (drop=True)
243
Create datasets and dataloaders
244
68

---

dataset hultimodalDataset (train_ _atai transforms image_ transforms)
val val_dataset MultimodalDataset (val_ data, transform= maage_transforss)
246
247
train_loader DataLoader (train_datassa batch_sizz-batchhsiia shuffle=True)
val_loades DataLoader wral_dataset, batch_ wiw.-batceebcceeeee size, shuffle=False)
240
250
Initialize model, optimizer, and loss function model MultimodalModel (devicc) optimizer torch optim Adam (model parameters Ir=learning_rate) critersos
252
253
criterion nn. CrossEntropyLoss((
254
Training loop epoch in range (nun_epochs):
257
train_ loss = train_moder (model, train_loader, optimizer, criterion, device)
258
val_loss, accuracy, f1 evaluate_model (model, val_loader, criterion, device)
260
print(
261
EpEpoch {epoch+1)/<num heu__eem__s,,,__ Train Loss: {train_lose losannlosss____
f}. Val Loss: {val_loss: 4f), Accuracy: {accuracy: 44),, F1: {f1: f}"
263
264
Final evaluation
accuracy, evaluate_model (model, val_loader, criterion, device)
cv cv_accuracies append (accuraac)) (f1)
267
print (fold 1} Final Accuracy: (accuracy: :44), F1: {f1 ".4f))
270
271
Make predictions test set
ids, preds predict (model, test_loaded device) test.predictions (ppend(pred)
214
275
Calculate average metrics
meanennaeee accuracy np.meaa mean (cv_ (cc_accraace) mean_fi np mean
278
std_accuracy up op.sta(ov std_fi ap.std(cv_ff_scorrs)
279
280
print (f" wrWNaaCros--aaddattoii
Results:")
print Meee Accuracy: amean_accurach .4f {etd gatd_accuracyy :4f))
283
print Meee F1 Score: (xean_f1:.4f) +- (std_f1:.4f)")
284
285
Average test predictions from all folds ttes__sseeeeee rest_predictions np. ap.array (test_predictions)) pinal_predcction np round (np mean (teat_prediction) axis=0)) astype (int)
288
Create submission file submission pd. p DataFrame (("id"; test_df df [f "1abel"; final_predictions))
291
292
Save submission file
submission to_ tt ww.//orrkin//aummsssoonnnccm cav", index=False)
print Subbuissio file aaved to ./vorking/ heteiinn//ummisiionnoo ")
205
69

---

More Related Work: AutoML with LLMs
Recent advances have demonstrated the potential of integrating LLMs into AutoML systems across various components of the ML workflow. For feature engineering, researchers [29 82 have leveraged LLMs generate meaningful features with explanations and efficiently evaluate candidate features. In hyperparameter optimization, LLM-based approaches [79] 47 generate configurations through dataset analysis and iterative refinement processes. For neural architecture search, researchers have combined LLMs with constructed performance predictors for estimating neural network efficiency [35]] LLMSs have also shown effectiveness for data preprocessing tasks such error detection and imputation 178). While these approaches successfully apply LLMs to specific components, MLZerr differs by providing an end-to-end multi-agent solution that fully automates the entire ML workflow.
algorithms to generate diverse network architectures [55] and
G Limitations and Future Work
Despite MI.Zero's strong performance across diverse machine learning tasks, several limitations warrant consideration. First, while system leverage both documentation and code examples, its effectiveness may be limited when working with ML libraries that have minimal description in either text code format. Second, machine learning libraries contain unavoidable undocumented bugs that are difficult to address through in-context learning, and LLMs cannot anticipate these issues unless explicitly prompted. Additionally, our experiments reveal that smaller LLMs (8B parameters) still demonstrate a significant performance gap compared to larger models despite showing competitive results.
Future research directions should address these challenges through several avenues. First, enhancing performance with smaller language models through improved alignment with external knowledge and curated code examples represents a promising direction, especially considering our 8B parameter model already demonstrates reasonable performance. Second, developing finetuning process that allows LLMs to actively experiment with and learn ML libraries during training could help address both undocumented bugs and improve smaller model performance, while also expanding MLZeeoo's applicability to emerging libraries with less structured documentation. This approach would enable models acquire practical knowledge about the ML library behavior beyond what is explicitly documented. Finally, conducting longitudinal studies on real-world ML workflows would provide insights for better supporting human-Al collaboration in data science processes, particularly for iterative development scenarios where expert guidance can further improve automated approaches.
70


Reviews:
[{"review_id": "FRRTHI5cjH", "reviewer": "Reviewer_dDLh", "summary": "This paper introduces MLZero, a novel multi-agent system that automates end-to-end machine learning workflows with minimal human intervention, specifically designed to handle raw, multimodal data. The system employs a perception module to interpret diverse inputs and uses dual semantic and episodic memory to improve iterative code generation while mitigating LLM hallucinations. When evaluated on the newly created Multimodal AutoML Agent Benchmark, MLZero achieved a 92% success rate, a 263.6% improvement over competitors. It also demonstrated superior performance on MLE-Bench Lite, securing six gold medals.", "questions": "please refer to the weakness part.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tNovel System Architecture: A well-designed multi-agent system featuring perception and dual-memory modules that automates the end-to-end ML workflow using complex, raw multimodal data.\n2.\tRigorous and Extensive Evaluation: The paper introduces a new, challenging benchmark for raw data and provides comprehensive ablation studies and failure analyses to validate the system's effectiveness.\n3.\tState-of-the-Art Results: The system demonstrates impressive performance, achieving a 92% success rate (+263.6% improvement) and showing robust results even when using a much smaller 8B model.\n4.\tClarity and Reproducibility: The work is clearly presented and highly reproducible, supported by a detailed appendix containing agent prompts, configurations, and comprehensive experimental results for verification.\nWeaknesses & Question\n1.\tNovelty of Individual Components: While the system's integration is novel, its core memory modules (Semantic Memory akin to RAG , Episodic Memory to self-correction ) build on existing concepts. The paper should better contextualize this, clarifying the primary novelty lies in the successful application and synergy for the specific, complex domain of end-to-end AutoML rather than the invention of the components themselves.\n2.\tScalability and Human Intervention: Performance relies on an offline, human-supervised process to create \"condensed tutorials\" for the knowledge base. This challenges the system's scalability and its \"zero human intervention\" claim. How much manual effort is required to integrate a new ML library, and how automated is this crucial knowledge preparation process?\n3.\tSystem Rigidity and Error Propagation: The system's linear, multi-agent pipeline creates a risk of cumulative error, while its effectiveness depends on a few high-level libraries. How does MLZero handle tasks outside its libraries' scope? More importantly, can a downstream coding agent detect and trigger a correction for an early-stage error, such as a flawed library selection made by the Perception module?", "was_revised": true, "final_justification": "I'd like to thank the authors for their response. Some of my concerns have been addressed in the rebuttal. Therefore, I will keep my positve rating.", "rebuttal": "We appreciate your positive assessment of our work, especially recognizing the novelty of the system architecture and the rigor of our evaluation. We address your specific concerns below.\n\n---\n\n## Q1: Novelty of Individual Components\n\nWe agree that the foundational concepts behind our memory modules are established in the LLM literature. For semantic memory, we enable true lifelong learning capability. For episodic memory, we efficiently condense information to 400-500 characters per failed iteration (~150 tokens), which is sufficient for the limited number of iterations typically needed per task, allowing effective error correction without exceeding context limitations.\n\n**However, as you mentioned, the \"successful application and synergy\" is key. Our innovation lies in how we've adapted and implemented these concepts for the AutoML domain.** We believe the system innovation can be essential as other alternative approach greatly deteriorate the performance (Ablation study). Similarly, prompting semantic memory to other methods still can't outperform our framework (Table 1), which shows that synergy can be essential for AutoML agent. Furthermore, The effectiveness of this specific synergy is demonstrated by our complete elimination of API hallucination and perception errors (Table 3), which are primary failure modes for strong baselines like AIDE. \n\n**The novelty of MLZero, as a complete framework, has been mentioned in Line 55.** We will update it in the introduction as follows: \n\n> \"Our primary contribution is not the invention of these components in isolation, but their novel, synergistic integration into a hierarchical multi-agent system specifically architected for the complex domain of end-to-end AutoML on raw, multimodal data\".\n\n---\n\n## Q2: How much manual effort is required to integrate a new ML library, and how automated is this crucial knowledge preparation process?\n\nAdding a new ML library requires only about **one minute of human effort using our provided registration python script.** The user simply inputs four basic pieces of information: the library name (e.g., \"FlagEmbedding\"), version number (e.g., \"FlagEmbedding==0.x.x\"), a brief one-sentence description, and the path to the directory containing the library's official documentation. Users can optionally provide additional prompts for the library, but this is not required. Importantly, **no coding skills are needed** for this knowledge preparation process.\n\nAfter this brief human setup, our system takes over completely. The Summarization and Condensation agents (Appendix B.2) automatically process the documentation into structured, condensed formats to construct the Semantic Memory. While this processing takes time, it requires no further human intervention or supervision - the LLM agents handle all summarization, structuring, and knowledge extraction tasks autonomously.\n\nOnce a library is integrated, MLZero operates from raw data to final prediction with no human input, as demonstrated in our experiments. This is a stark contrast to many existing systems that require manual preprocessing or hard-coded logic per dataset.\n\n---\n\n## Q3: How does MLZero handle tasks outside its libraries' scope?\n\nOur system is able to handle out-of-scope tasks effectively using a general fallback mechanism while selecting ML Libraries (outperforming all competitors even without specialized libraries). MLZero can fallback for tasks that fall outside the scope of its specialized, high-level libraries. \n\n**As detailed in Appendix D.5**, the system can select \"machine learning\" as a general tool. In this mode, the Coder agent is prompted to generate a solution from scratch using fundamental libraries (e.g., PyTorch, scikit-learn), leveraging the LLM's parametric knowledge. This provides flexibility and extends the system's applicability beyond the pre-registered libraries. Importantly, **our results show that even when all specialized ML libraries are removed (denoted as \"-ext\" in the main table), MLZero still achieves a rank of 4.94 and success rate of 69.3%, outperforming all competitors**. This demonstrates that our system remains effective even for tasks outside its specialized knowledge due to the effective system design.\n\n\n---\n\n## Q4: More importantly, can a downstream coding agent detect and trigger a correction for an early-stage error, such as a flawed library selection made by the Perception module?\n\nOur system is robust to late-stage errors through iterative correction loops. For early stages, experiments with deliberately misleading dataset descriptions proved our perception module's strong reasoning capabilities against noisy inputs. Even for deliberately created early-stage errors (which never occurred in our experiments), our system can recover by re-trigger perception:\n- **Notably, across all our experiments, we have not encountered failures due to early-stage perception errors, highlighting the robustness of our system.** However, to further test the perception module's resilience, we conducted additional stress tests with the Abalone dataset (Tabular Regression) as shown in the table below. Even with deliberately misleading descriptions and limited library availability (Tests 1-6), our perception module consistently correctly identified it as a tabular task and chose either the Tabular library or general machine learning (if the tabular library is removed).\n- If explicitly directed to use an incorrect tool in the user instruction (Tests 7), our system will choose the incorrect tool due to the respect to user instruction, but it can then utilize a restart option to re-trigger the perception module after detecting the failure.\n- **The system is also robust to subsequent errors after the Perception module.** The iterative coding loop (Sec 3.4), powered by the Episodic Memory and Error Analyzer agent (Sec 3.3, Appendix B.3.1), allows the system to recover from a wide range of execution failures.\n\n| Test Case | Noise in Description | Noise in User Instruction | Remove Correct Libraries | Perception Module | Overall System |\n|-----------|---------------------|---------------------------|-------------------------|-------------------|----------------|\n| **Test 1** | \"This is a timeseries task,\" | None | No | ✅ **Success** | ✅ **Success** |\n| **Test 2** | \"YOU SHOULD treat this as a timeseries task,\" | None | No | ✅ **Success** | ✅ **Success** |\n| **Test 3** | \"IMPORTANT: Use a TimeSeries ML Library,\" | None | No | ✅ **Success** | ✅ **Success** |\n| **Test 4** | \"Try to use FlagEmbedding.\" | None | No | ✅ **Success** | ✅ **Success** |\n| **Test 5** | \"YOU SHOULD treat this as a timeseries task,\" | None | Yes | ✅ **Success** | ✅ **Success** |\n| **Test 6** | \"Try to use FlagEmbedding.\" | None | Yes | ✅ **Success** | ✅ **Success** |\n| **Test 7** | None | \"Use FlagEmbedding\" | Yes | ❌ Fail | ✅ **Success** |"}, {"review_id": "VCA96u5JQy", "reviewer": "Reviewer_7xrS", "summary": "The paper introduces MLZero, a novel multi-agent system that achieves fully automated machine learning (ML) across diverse data modalities with zero human intervention. MLZero integrates Large Language Models (LLMs) with a perception module and dual memory systems—semantic and episodic—to intelligently process raw multimodal data, iteratively generate and refine ML code, and deliver ready-to-use solutions. The system coordinates nine specialized agents to perform data understanding, library selection, code generation, execution, and error correction. MLZero demonstrates superior performance on both the MLE-Bench Lite and a newly proposed Multimodal AutoML Agent Benchmark, significantly outperforming existing approaches in success rate, efficiency, and robustness. Through ablation studies, the paper shows the crucial roles played by memory components in reducing hallucinations and improving code quality. This work presents a promising step toward general-purpose, autonomous ML agents.", "questions": "1. It would be better if authors are able to compare the token costs of your method.\n\n2. Is the system robust to different models?", "limitations": "See questions and weaknesses", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Advantages:\n1. Fully Automated Workflow: Handles the entire ML pipeline from raw data to results without human input.\n2. Memory-Augmented Reasoning: Uses semantic and episodic memory to reduce hallucinations and improve reliability.\n3. Strong Performance: Outperforms baselines with higher success rates and better solution quality.\n\nWeakness:\n1. Still lacks several experiments to validate the effectiveness of their method", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive review and for highlighting the strengths of our fully automated workflow and memory-augmented reasoning.\n\n---\n\n## Q1: Comparison of Token Costs\n\nMLZero achieves superior performance (92% success rate) while using **4.3x fewer tokens than AIDE (63k vs. 273k per dataset) and costing only 23% as much (0.19 v.s. 0.82 \\$/dataset), as shown in the table below.** Since most competitor systems don't report token usage, we estimated AIDE's consumption based on the cost of 25 benchmark runs. Codex CLI likely uses fewer tokens but at the significant cost of solution quality. This demonstrates MLZero's significant efficiency advantage while delivering better results.\n\n| System | Success Rate | Average Token Per Dataset | Average $Cost Per Dataset |\n|:-------------------------:|:------------:|:-------------------------:|:-------------------------:|\n| MLZero (def) | 92% | 63k | $0.19 |\n| MLZero (no suggested fix) | 75% | 51k | $0.15 |\n| AIDE (+ext) | 45.30% | ~273k | $0.82 |\n\n**Note:** MLZero (no suggested fix) was evaluated on 8 challenging datasets selected for ablation studies covering all modalities, explaining its lower success rate compared to the full 25-dataset benchmark used for MLZero (def) and AIDE (+ext). Despite using significantly fewer tokens (4.3x less than AIDE), MLZero achieves substantially better performance at a fraction of the cost.\n\n---\n\n## Q2: Is the system robust to different models?\n\nMLZero performs effectively across different LLM backbones. In Table 1 of the paper, we present results for **MLZero (def)**, which uses the large Claude 3.7 Sonnet model, and **MLZero (Llama 3.1 8B)**, which uses a much smaller model. Crucially, **MLZero (Llama 3.1 8B) still outperforms all other baseline agents**, including those using full-sized Claude 3.7 or GPT-4 class models, demonstrating **MLZero is agnostic to the choice of LLM backbones.**\n\nWe also append here the performance of Claude Sonnet 3.7, **GPT 4.1**, and LLama 8B on 8 challenging datasets selected for ablation studies covering all modalities. Our results demonstrate that the MLZero framework is indeed robust to the choice of the underlying LLM.\n\n| Dataset | MLZero - Claude Sonnet 3.7 | MLZero - GPT 4.1 | MLZero - LLama 8B |\n|:------------------:|:--------------------------:|:----------------:|:-----------------:|\n| camoseg | 0.84 | 0.82 | X |\n| flood | 0.69 | 0.64 | 0.69 |\n| fiqa | 0.5 | 0.5 | 0.5 |\n| mldoc | 0.95 | 0.96 | 0.95 |\n| petfinder | 0.39 | 0.4 | 0.4 |\n| rvlcdip | 0.87 | 0.87 | X |\n| solar(10m)↓ | 1.49 | X | X |\n| yolanda↓ | 8.53 | 8.53 | 8.54 |"}, {"review_id": "Ld5PQmDKIV", "reviewer": "Reviewer_aYBU", "summary": "The paper proposes a multi-agent system titled “MLZero” which achieves impressive results on MLE Bench and their own Multimodal AutoML benchmark while being fully automated. The paper discussed their system setup and agent definitions along with the benchmarks.", "questions": "1. There is no mention of which LLM is used in the “def” setting of MLZero. Could this be added to the paper? The config present in the appendix section seems to indicate this is Claude 3.7 Sonnet, but it would be great if the authors could confirm this.\n2. The agent seems to “finish execution” once it successfully executes the code (or reaches the context limit). It would’ve been great to see the agent continually create better solutions and maintain a track of the best solution. Ablations to see the improvements from this setup versus the current would’ve provided valuable insights into the self improvement capabilities of LLMs (something like AlphaEvolve). I’m not sure if these ablations were done. Could you add details about those if they were done?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is very detailed and well written.\n2. They achieve impressive results on the MLE Bench and Multimodal AutoML benchmarks.\n3. The paper contains extensive ablations and detailed task-level results on the various ML benchmarks.\n\nWeaknesses:\n1. While the results are impressive, the system itself is simple and doesn't represent a significant contribution to the field. The crux of the paper lies in the setup of the system and the way the different agents interact with each other, and while that is discussed in fair detail, the system isn’t complex or novel enough to meaningfully move the field forward.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your feedback and are glad you found the paper well-written and the results impressive.\n\n---\n\n## Q1: System Simplicity and Contribution\n\nOur innovation lies in how we invent **a new synergetic multi-agent system for the AutoML domain, which is challenging given the total number of the agents (9 in total) in MLZero**. While the individual components are inspired by previous literature under different contexts. The system innovation can be essential **while other alternative approach greatly deteriorate the performance** (see ablation study in **Section 4.2.2 and 4.3**). Similarly, adding memory module to other methods (e. g. AIDE) still can't outperform MLZero (Table 1). Those evidence show that the synergy can be essential for AutoML applications. Furthermore, The effectiveness of this specific synergy is demonstrated by our complete elimination of API hallucination and perception errors (Table 3), which are primary failure modes for strong baselines like AIDE and DS-Agent. Its significance lies in achieving effectiveness (+263.6% improvement) and state-of-the-art performance in end-to-end ML automation. **The novelty has been mentioned in L55.**\n\nBecause using LLM agent to propose high quality ML solution are considered to be very challenging and requires **higher entry bar [MLAgentBench (ICML'24)]**, hence MLZero can be potentially adaptable to broader data science related tasks with minimal modifications, which can inspire future work in this community.\n\n---\n\n## Q2: Which LLM is used in the \"def\" setting?\n\nThe LLM used in the def configuration is **Claude 3.7 Sonnet**. This is specified in the implementation details in **Appendix C.1** and the configuration file snippet in **Appendix C.2**.\n\n---\n\n## Q3: Continual Improvement of Solutions\n\n**Without the early stopping, MLZero can continue to improve the performance after first success** (up to +0.09 accuracy), we have conducted experiments exploring continuous improvement across 8 ablation datasets as shown in the table below:\n\n| Dataset | def | def + nostop | -ext | -ext + nostop |\n|:------------------:|:----:|:------------:|:----:|:-------------:|\n| camoseg | 0.84 | **0.86** | 0.46 | **0.55** |\n| flood| 0.69 | **0.74** | 0.6 | **0.62** |\n| fiqa | 0.5 | **0.51** | 0.22 | **0.23** |\n| mldoc | 0.95 | **0.96** | 0.94 | **0.96** |\n| petfinder | **0.39** | 0.38 | 0.38 | **0.44** |\n| rvlcdip | 0.87 | 0.87 | **0.89** | 0.88 |\n| solar(10m)↓ | 1.49 | **0.38** | X | X |\n| yolanda↓ | **8.53** | 8.54 | 8.93 | **8.75** |\n| **Avg Token↓** | **63k** | 186k | **51k** | 146k |\n| **Avg Time↓** | **1904** | 5388 | **1731** | 4520 |\n\n**Key findings from our experiments:** Performance improvements are observed but are relatively modest, as the default setting already achieves strong results. For example, on the `camoseg` dataset, accuracy improves from 0.84 to 0.86, and on `flood` from 0.69 to 0.74. The improvement pattern is more pronounced in the challenging \"-ext\" setting (without external libraries), showing larger gains (e.g., `camoseg` from 0.46 → 0.55, `petfinder` from 0.38 → 0.44). **MLZero demonstrates clear capability to refine successful solutions, as optimizing an existing working solution is generally easier than achieving the first success.**\n\nHowever, our experiments reveal a trade-off: continuing iterations after the first success leads to performance improvements but comes with larger computational costs. Even when limiting to 5 iterations maximum, the token usage triples from 63k to 186k, and computation time increases from 1904s to 5388s in the default setting. This occurs because datasets that succeed in the first run will still complete all 5 iterations instead of stopping early. **Therefore, in the main paper, we chose our current \"stop-at-first-success\" design to prioritize efficiency while maintaining high performance.**\n\nWe agree that an explicit iterative improvement is a promising direction for future work, especially in scenarios where computational resources are not a constraint. This would involve carefully balancing the trade-off between optimality and efficiency. **In the main paper, we also explored the system's potential in the 24hrs experiment (Table 1)**. By extending the time limit and using the best_quality preset for the generated code, MLZero's performance improves significantly, approaching expert-level human results on several datasets. This further validates our findings about the system's capability to produce higher-quality solutions when given more computational resources, while highlighting the practical value of our efficient default setting for most real-world applications."}, {"review_id": "bySLpbmJge", "reviewer": "Reviewer_b88G", "summary": "This paper presents MLZero, a hierarchical multi-agent system for fully automated machine learning (AutoML) across multimodal data. The framework integrates perception agents, semantic memory (knowledge retrieval), episodic memory (error-driven refinement), and iterative code generation based on LLMs, aiming for zero human intervention. MLZero is benchmarked on both MLE-Bench Lite and a new Multimodal AutoML Agent Benchmark introduced by the authors, showing competitive results in success rate and performance against various baselines. Extensive ablation and error analyses are also provided to justify architectural design choices.", "questions": "1. **Benchmark Diversity**: How were the 25 datasets in your new benchmark chosen? Is there any risk that the benchmark favors MLZero due to design alignment?\n2. **Failure Analysis Detail**: For the minimal 6.5% error rate in MLZero, what are the qualitative patterns of failure? Are these reproducible errors or stochastic?\n3. **Knowledge Contamination Risk**: Given that some agents access condensed tutorials and summaries, how do you ensure no contamination from evaluation sets?\n4. **Memory Scalability**: How does episodic memory scale with longer runs or larger datasets? Is there a cost trade-off in performance vs. token consumption that might make it infeasible in practice?\n5. **Security and Robustness**: Does MLZero ever produce unsafe or incorrect solutions that *look* correct (e.g., hallucinated accuracy but failed logic)? Any defenses?\n\n*A more favorable evaluation could be reached if the authors:*\n\n* Provide clearer guarantees or at least empirical error bounds for LLM planning and execution cycles.\n* Include human validation of at least a subset of generated pipelines.\n* Address benchmark neutrality through external evaluation or open access submission.", "limitations": "The authors acknowledge runtime cost and large-model reliance, but do not adequately address:\n\n* **Generalization to new ML libraries**: It is unclear how easily MLZero adapts to unseen or rapidly changing APIs.\n* **Dependence on LLM correctness**: The system heavily assumes the base LLM will not hallucinate or misinterpret errors in a critical way.\n* **Memory curation and freshness**: Episodic and semantic memories appear stateless across datasets—how would they behave in a real, long-term deployed setting?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "#### **Strengths**\n\n* **Well-structured System Design**: The paper presents a clear and modular architecture, combining LLM planning and execution with memory augmentation and agent specialization.\n* **Multimodal Generality**: MLZero operates on diverse tasks spanning tabular, text, image, and document classification, which showcases practical versatility.\n* **Comprehensive Evaluation**: Experiments span two benchmarks, including a new 25-task suite with raw data processing requirements.\n* **Ablation and Failure Analysis**: The inclusion of ablation studies and error breakdown by category (e.g., hallucinations, preprocessing) is commendable and unusually thorough.\n* **Competitive Results**: MLZero outperforms existing baselines in success rate and average rank, even when stripped of certain components like external knowledge or episodic memory.\n\n#### **Weaknesses**\n\n1. **Excessive Engineering Emphasis, Weak Novelty in Learning**: While the architecture is elaborate, most techniques (e.g., retrieval, summarization, error tracing) are standard LLM engineering practices. There is limited novelty from a learning or algorithmic standpoint.\n2. **Limited Insight into LLM Limitations**: The reliance on LLMs is central, but the paper does not examine when/how LLMs fail during ML pipeline construction, or what qualitative reasoning errors occur.\n3. **Benchmark Bias**: The newly introduced benchmark is entirely authored by the same authors. While it addresses limitations of MLE-Bench, the fairness and diversity of its design should be externally validated.\n4. **Lack of Human Evaluation or Expert Annotation**: The paper focuses solely on automatic metrics, without any human judgment of code quality, correctness, or failure modes—this makes the empirical superiority somewhat opaque.\n5. **Minimal Theoretical or Formal Analysis**: There is no formal grounding in agent planning, LLM reliability, or bounded correctness. A system this complex could benefit from guarantees or at least probabilistic behavior analysis.\n6. **Missing Comparison with Planning-based LLM Approaches**: Other recent planning-agent frameworks (e.g., ReAct, Reflexion, CAMEL) are not directly compared or discussed in depth. This weakens claims of generality and superiority.", "was_revised": true, "final_justification": "I have read the author response and appreciate the detailed clarifications and additional experiments. My main concerns — including benchmark fairness, system novelty, and error analysis — have been addressed satisfactorily. The paper is a solid empirical systems contribution with thoughtful design and extensive evaluation.\n\nI am therefore increasing my score to 4.", "rebuttal": "To address 17 bulletpoints (weakness: 6 bulletpoints, questions: 5+3=8 bulletpoints, and limitations sections: 3 bulletpoints) comprehensively and concisely, we've organized our response into three categories: Methodology, Benchmark Design, and Evaluation.\n\n---\n\n# Methodology (Q1, 10, 16, 17)\n\n## Q1: Excessive Engineering Emphasis, Weak Novelty in Learning\n\nWe respectfully disagree with the characterization of our work as \"standard LLM engineering.\" While we use established techniques, our key contribution is the new and cohesive framework with 9 agents coordinating synergetically for end-to-end AutoML on raw, multimodal data. The synergistic combination of four modules and nine specialized agents creates a system that empirically eliminates entire classes of errors and achieves significant performance gains over existing approaches. Our ablation studies demonstrate that **simply adding semantic memory modules to other methods or using alternative approaches results in substantially degraded performance, highlighting that the specific integration and coordination of these components is essential.** This is fundamentally an empirical systems paper where the novel architecture and its rigorous validation constitute the core scientific contribution **[e.g. MLAgentBench (ICML'24), DS-Agent (ICML'24), MDAgents (Neurips'24 Oral), etc.]**.\n\n## Q10: Memory Scalability\n\nFirst, episodic memory consumption is independent of dataset size - it only depends on the number of failed steps. Our system adds approximately 150 tokens per failed step to the context through our Error Analyzer agent's efficient condensation, and can theoretically support more than 100 steps while staying within context limits. As for the cost trade-off question, while continuing iterations beyond first success can improve performance (up to +0.09 accuracy gain, as detailed in our response to Q3 to reviewer aYBU), the initial successful solution already achieves strong results. The memory overhead remains manageable since most tasks succeed within 3 steps, making the system practical and cost-effective. Users can always trade additional computational resources for incremental improvements and our system is scalable to support that.\n\n## Q16: Dependence on LLM correctness\n\nOn the contrary, we have specifically engineered our system to **address LLM limitations through our multi-agent architectural design**. Our system is deliberately structured to detect and recover from LLM failures through multi-agent synergy and iterative refinement. When one agent makes an error, others can detect and correct it, preventing fatal failures. Our experimental results confirm this robustness—across all experiments, we encountered no failures due to early-stage perception errors, and our stress tests with deliberately misleading descriptions ( as detailed in our response to Q4 to reviewer dDLh) still resulted in correct task perception. Furthermore, Tab 1 shows that even without external knowledge or episodic memory components, or using a very small 8B LLM, performance remains superior to competitors.\n\n## Q17: Memory curation and freshness\n\n**For fair benchmarking, our system runs independently on each task**, with memories reset between tasks to prevent information leakage. In a real-world deployment, both memory systems would provide additional benefits: the semantic memory can be continuously updated with new library documentation, best practices, and community knowledge; the episodic memory would accumulate valuable insights across similar tasks, improving efficiency through transfer learning. The modular design allows for easy updates to both memory systems, ensuring continued relevance and performance as ML tools and techniques evolve.\n\n---\n\n# Benchmark Design (Q3, 7, 9, 11, 14)\n\n## Q3, Q7, Q14: Benchmark Bias, Diversity, and Neutrality\n\n**We clarify that MLZero achieves superior results on the existing MLE-Bench Lite benchmark before introducing our own benchmark, which was designed independently without bias towards MLZero.** We created the Multimodal AutoML Agent Benchmark (MAAB) out of necessity, as existing benchmarks don't support evaluation on raw, unprocessed data—a critical gap that masks real-world automation challenges.\n\nTo ensure fairness and diversity, MAAB is built on 25 diverse public datasets from reputable sources (Kaggle, UCI, BEIR) selected from machine learning publications, **most without direct open-source solutions**. It also spans 8 distinct problem types (classification, segmentation, retrieval, etc.) across multiple modalities (tabular, text, image, document data).\n\nWe will open-source the entire benchmark, including all datasets and evaluation code, for external validation and community use (included in supplemental materials).\n\n## Q9: Knowledge Contamination Risk\n\nOur knowledge base contains only public, official tutorial documentation for ML libraries themselves—no information about evaluation datasets or their solutions. The dataset does not appear in our knowledge base, and processing raw data is fundamentally different from using pre-processed tutorial examples. More importantly, our evaluation datasets and their test sets are completely held out. This follows standard RAG practices [Retrieval-Augmented Generation for Large Language Models: A Survey] where external knowledge remains general and task-agnostic.\n\n## Q11: Security and Robustness\n\n**During benchmarking, test data is completely held out from all agents, eliminating any possibility of hallucinated accuracy claims**. Our system itself also includes robust safeguards against producing incorrect solutions that appear correct. The Executer agent doesn't simply check for zero exit codes but uses an LLM to analyze execution logs and outputs to determine logical success. It can trigger a \"FIX\" decision even when code runs without crashing if it detects logical errors or poor performance (e.g., accuracy of 0.01), ensuring higher solution reliability.\n\n---\n\n# Evaluations (Q2, 4, 5, 6, 8, 12, 13, 15)\n\n## Q2: Limited Insight into LLM Limitations\n\nWe respectfully disagree with this point. Our paper provides a very detailed analysis of LLM limitations in this context **in Section 4.4 (Failure Cases and Analysis) and Appendix E.3 (Detailed Error Analysis)**. Due to context limitation, we'll not repeat here.\n\n## Q4, Q13: Lack of Human Evaluation/Validation for code quality, correctness, or failure modes\n\nNote that we prioritized automated metrics for task success (**correctness**) because they are objective, scalable, and directly measure the system's ability to solve the problem. Given that **code quality** is important, we have included full, verbatim examples of generated code from MLZero, Codex CLI, and AIDE in **Appendix E.4**. These examples allow for a direct qualitative comparison. For instance, the MLZero-generated code for the Hateful Meme task (Appendix E.4.1) is modular, well-commented, and uses a sophisticated multimodal approach, contrasting with the simpler, text-only logistic regression from Codex CLI (E.4.2) and the more complex but less efficient boilerplate from AIDE (E.4.3). **As stated above in Q2, we have included expert annotation of failure modes in main paper and appendix.**\n\n## Q5, Q6: Minimal Theoretical Analysis & Comparison with Planning-based LLM Approaches\n\nOur work is an empirical contribution focused on building and validating a complex system. Formal guarantees for LLM-based agentic systems are a nascent and very challenging open research problem beyond the scope of this paper.\n\nRegarding planning agents like ReAct, Reflexion, or CAMEL, these are general reasoning frameworks, not end-to-end ML Agent systems, therefore can't be compared directly. Furthermore, we do include **Codex CLI—a state-of-the-art planning agent approach**—as a baseline in Table 1. However, per the reviewer's request, we tested CAMEL on our benchmark because it is still actively maintained among those three methods for the updated API calls. The results below show CAMEL struggled significantly, with most attempts failing (marked with \"X\") across three runs (12.5% overall success rate):\n\n| CAMEL setting: GPT-4o, with CodeExecutionToolkit and FileWriteToolkit | Run 1 | Run 2 | Run 3 |\n|:---------------------------------------------------------------------:|:-----:|:-----:|:-------:|\n| camoseg | X | X | X |\n| flood | X | X | X |\n| fiqa | X | X | X |\n| mldoc | X | X | 0.63|\n| petfinder | X | X | 0.34 |\n| rvlcdip | X | X | X |\n| solar(10m)↓ | X | X | X |\n| yolanda↓ | X | 9.62 | X |\n\n## Q8, Q12: Failure Analysis Detail & Empirical Error Bounds\n\nOur analysis of MLZero's 6.5% error rate shows non-deterministic (Table 6) failures that vary between LLM runs, stemming from two main sources: algorithm implementation errors (\\~4.3%) on tasks lacking specialized libraries (particularly audio processing) and preprocessing errors (\\~2.1%) on image-to-image tasks with heterogeneous input sizes.\n\nDespite this variability in specific implementations, our manual annotation of 46 datasets provides statistical confidence in the reported **6.5% error rate (standard deviation \\~3.6%)**, while establishing empirical reliability bounds with a **92% success rate (from 75 runs, standard deviation \\~3.1%)** across diverse tasks. Although formal guarantees for LLM systems remain an open research challenge, our systematic error categorization and ablation studies offer valuable insights into failure modes and component contributions to reliability. This comprehensive empirical framework demonstrates that while individual errors may differ between runs due to the probabilistic nature of LLM outputs, the overall error patterns remain consistent enough to inform targeted improvements in future work.\n\n## Q15: Generalization to new ML libraries\n\n**One of our key contributions is a system designed for easy integration of new ML libraries.** Due to context limitation, please refer to the answer to Q2 of reviewer dDLh."}]

Author rebuttal:

Meta-review: (a) Summary of scientific claims and findings
This paper presents MLZero, a hierarchical multi-agent framework for end-to-end ML automation across multimodal data. The system integrates a perception module with semantic and episodic memory to reduce hallucinations and improve iterative code generation. MLZero is evaluated on MLE-Bench Lite and a newly introduced Multimodal AutoML Agent Benchmark (25 diverse tasks), achieving a high success rate and outperforming strong baselines. The framework also demonstrates efficiency (fewer tokens, lower cost) and robustness across different LLM backbones, including smaller models.

(b) Strengths
- Well-structured system design with nine coordinated agents.
- Strong empirical results: consistent improvements over competitive baselines on two benchmarks.
- Comprehensive evaluation
- Clear and reproducible presentation

(c) Weaknesses
- Limited novelty in individual components; many techniques are adaptations of known methods.
- Heavy emphasis on engineering, lacks deeper insights.
- The newly introduced benchmark is authored by the same team and has not been externally validated.
- Minimal exploration of LLM limitations and failure cases beyond quantitative error rates

(d) Reasons for decision
The paper makes a solid systems contribution: while the novelty lies more in integration than in new algorithms, the framework is thoughtfully designed, rigorously evaluated, and demonstrates clear performance gains. The authors' rebuttal convincingly addressed concerns about scalability, benchmark fairness, and robustness. The work is impactful for the AutoML and agent communities, but the lack of fundamental theoretical advances or stronger novelty makes it better suited as a poster rather than spotlight/oral.

(e) Discussion and rebuttal period
Reviewers raised questions on novelty, benchmark design, scalability, and system robustness. The authors clarified that adding new libraries requires minimal human effort, provided additional results on robustness across LLMs, offered token/cost comparisons, and presented stress tests and error analyses. While concerns on originality and benchmark neutrality remain partly valid, the rebuttal addressed most technical doubts satisfactorily. Overall, the reviewers converged on a positive but cautious stance, leaning toward borderline accept. I weigh the strong empirical contribution and practical relevance over the novelty concerns, leading to my recommendation for poster acceptance.
Decision: Accept (poster)