Title: MLZero: A Multi-Agent System for End-to-end Machine Learning Automation
Venue: NeurIPS 2025
Accepted: yes
Keywords: AutoML

Large Language Models

Multi-Agent Systems

End-to-End Machine Learning

Multimodal Learning

Code Generation
Abstract: Existing AutoML systems have advanced the automation of machine learning (ML); however, they still require substantial manual configuration and expert input, particularly when handling multimodal data. We introduce MLZero, a novel multi-agent framework powered by Large Language Models (LLMs) that enables end-to-end ML automation across diverse data modalities with minimal human intervention. A cognitive perception module is first employed, transforming raw multimodal inputs into perceptual context that effectively guides the subsequent workflow. To address key limitations of LLMs, such as hallucinated code generation and outdated API knowledge, we enhance the iterative code generation process with semantic and episodic memory. MLZero demonstrates superior performance on MLE-Bench Lite, outperforming all competitors in both success rate and solution quality, securing six gold medals. Furthermore, when evaluated on our Multimodal AutoML Agent Benchmark, which includes 25 more challenging tasks spanning diverse data modalities, MLZero outperforms the competing methods by a large margin with a success rate of 0.92 (+263.6\%) and an average rank of 2.28. Our approach maintains its robust effectiveness even with a compact 8B LLM, outperforming full-size systems from existing solutions.

MLZero: A Multi-Agent System for End-to-end Machine Learning Automation

Abstract

Existing AutoML systems have advanced the automation of machine learning (ML); however, they still require substantial manual configuration and expert input, particularly when handling multimodal data. We introduce MLZero, a novel multi-agent framework powered by Large Language Models (LLMs) that enables end-to-end ML automation across diverse data modalities with minimal human intervention. A cognitive perception module is first employed, transforming raw multimodal inputs into perceptual context that effectively guides the subsequent workflow. To address key limitations of LLMs, such as hallucinated code generation and outdated API knowledge, we enhance the iterative code generation process with semantic and episodic memory. MLZero demonstrates superior performance on MLE-Bench Lite, outperforming all competitors in both success rate and solution quality, securing six gold medals. Additionally, when evaluated on our Multimodal AutoML Agent Benchmark, which includes 25 more challenging tasks spanning diverse data modalities, MLZero outperforms the competing methods by a large margin with a success rate of 0.92 (+263.6%) and an average rank of 2.28. Our approach maintains its robust effectiveness even with a compact 8B LLM, outperforming full-size systems from existing solutions.

1 Introduction

The quest to democratize machine learning has long been championed by AutoML, a field dedicated to automating the intricate process of building ML solutions [21, 20, 19, 67, 65, 28]. However, conventional AutoML systems often struggle to manage the heterogeneity of data formats and modalities, requiring predefined workflows or extensive manual tuning by domain experts. Despite advances in automating specific components, such as feature engineering [13, 53], neural architecture search [39, 18], and hyperparameter optimization [75], such systems still fall short of providing an end-to-end solution that spans the entire machine learning lifecycle, from data preprocessing to model building. These challenges leave a critical gap for truly autonomous and versatile systems.

The emergence of LLMs brings new opportunities to enhance the flexibility of AutoML. Early attempts to leverage LLMs for AutoML targeted specific pain points, such as LLM-guided feature engineering [29, 82], hyperparameter optimization [47, 79], neural architecture search [35, 55],

Query

Folder with Datasets

Optional user

instruction

Use training data

format for outputs

MLZero (Zero Human Intervention)

Perception

Complex

File structure

Data type

Digital

Environment

Iterative Generations

Memory

Feedbacks

History Error

Error

ML Libraries

Logs & metric

ML Code

Output

Ready-to-use

Model

Prediction

File Perception

Agent

Task Understanding

Agent

Agent

Summarization Agent

Condensation Agent

Coder

Agent

Executer

Agent

Agent

Knowledge

Retrieval Agent

Execution

DDDD

and data preprocessing [78]. Despite advances, these approaches remain limited by their focus on automating individual components rather than providing truly end-to-end multimodal automation.

With the advent of agentic AI, recent research has explored the broader potential of the LLM agent system for comprehensive AutoML solutions [10, 38, 45, 70, 24], leveraging their demonstrated strengths in planning, code generation [36, 31], and verification processes [81, 37, 73]. However, the success rate of completing complex ML tasks remains hindered by two major challenges: (1) Current systems exhibit a lack of automated data perception, forcing reliance on hard-coded preprocessing logic or rigid data formats [24, 33, 45]. (2) LLMs, when relying solely on their internal parametric knowledge, tend to produce simplistic solutions that fall short for intricate ML tasks [38]. Even with access to advanced ML libraries and domain knowledge, their performance deteriorates when managing long, convoluted knowledge due to the absence of efficient memory mechanisms.

To address these challenges, we introduce MLZero, a novel multi-agent system designed specifically for end-to-end automation of multimodal ML tasks. As illustrated in Figure 1, our approach begins with specialized agents performing comprehensive perception actions to analyze data structures, interpret task requirements, and select the proper ML library, thus establishing a rich perception context for subsequent operations. Building upon the perception context, the system leverages the semantic memory with condensed, curated knowledge about the library to guide iterative code development, while simultaneously utilizing episodic memory for efficient error detection and correction. The dual memory modules enable powerful learning capabilities [83], allowing the system to retain domain knowledge persistently while adaptively learning from past errors, thereby continuously refining its solution through iterations. By seamlessly integrating the reasoning capabilities of LLMs with automated perception and structured memories, MLZero not only overcomes the inherent limitations of previous LLM-based approaches, but also represents a truly end-to-end system that consistently demonstrates superior performance across diverse machine learning tasks. Our main contributions are as follows:

We introduce MLZero, a novel multi-agent system that delivers high quality end-to-end multimodal machine learning solutions with minimal human intervention. Our system automates the entire workflow through the perception module, dual memory modules (semantic and episodic), and iterative code generation, coordinated across nine specialized agents. Our primary contribution lies in the synergistic integration of these components into a hierarchical multi-agent system specifically designed for end-to-end AutoML on raw, multimodal data.

MLZero demonstrates superior performance on the existing benchmark, MLE-Bench Lite, with more medal counts (6 gold medal) and higher success rate.

We benchmark MLZero with a comprehensive benchmark suite, the Multimodal AutoML Agent Benchmark, comprising diverse datasets that incorporate challenging scenarios including multilingual, multitable, multilabel, zero-shot inference tasks with large unprocessed data. Our comprehensive empirical evidence from the constructed multimodal benchmark showing that MLZero outperforms existing ML agents across all evaluated metrics, such as success rate ( +263.6% ), average rank, and time complexity,

Through detailed ablation studies, we identify key components driving these performance gains and analyze failure cases to demonstrate both the common failures MLZero overcomes and opportunities for future research directions.

2 Related Work

2.1 LLMAgents for ML

Beyond component-specific applications, several frameworks have emerged for end-to-end LLMbased ML automation [30, 33, 23, 24, 45, 38]. MLAgentBench [33] introduces a research agent that conducts ML experiments by accessing files and executing code. DS-Agent [24] implements case-based reasoning to leverage human expertise, but requires extensive dataset-code pairs during development and struggles with unfamiliar domains. Both approaches lack effective perception capabilities and require code skeleton samples, limiting their end-to-end functionality. AutoKaggle [45] offers a multi-agent system with iterative development and testing, but is restricted to tabular data with hard-coded input formats. AIDE [38] frames ML engineering as a code optimization problem with tree search in the solution space [14, 46] but underperforms without sufficient external knowledge about ML Libraries. Despite these advances, challenges remain in developing truly end-to-end ML automations that operate on raw data with output specifications and minimal human intervention. Key limitations include maintaining consistency across complex workflows, utilizing memory efficiently, adapting to evolving ML libraries, etc. For a more comprehensive evaluation, we deliberately strengthen the existing approaches [38, 24, 45] by prompting them with selected outputs from MLZero (Appendix C).

Meanwhile, MLE-bench [10] offers valuable but methodologically limited agent-human comparisons, constrained by its reliance on preprocessed rather than raw data and outdated human baselines collected under different computational and algorithmic contexts. These limitations motivate our new benchmark's focus on direct agent-to-agent comparisons for end-to-end ML automation.

2.2 Tool Use

Recent advances in LLM-based tool-use agents have significantly expanded AI capabilities. Toolformer [63] pioneered an approach where LLMs learn API usage through self-supervision with minimal demonstrations. ToolLLM [60] scaled this concept by developing a comprehensive framework supporting over 16,000 real-world APIs. Chameleon [49] introduced compositional reasoning with a plug-and-play approach for multimodal QA tasks. OctoTools [48] further introduced a trainingfree agentic framework with standardized tool cards, hierarchical planning, and efficient execution components. Other notable contributions include ToolDoc [32], which demonstrated that API documentation can effectively replace demonstrations, and ToolkenGPT [26] with tool embeddings to avoid fine-tuning. MM-REACT [76] extended these capabilities to visual reasoning tasks, while HuggingGPT [66] demonstrated effective orchestration of multiple models.

While these systems have made significant progress, they mainly focus on tasks with short context lengths. Considering the autoML tasks often require perceiving a large amount of data, our work extends beyond this paradigm by developing a multi-agent approach specifically designed for the more complex and interconnected tasks inherent in ML automations.

2.3 Code Generation with LLMs

Large language models have demonstrated increasingly impressive capabilities in code generation and debugging. Models such as AlphaCode [44], Codex [11], and StarCoder [43] have shown competitive programming performance through effective solution generation and self-verification across multiple programming languages. State-of-the-art LLMs [3, 59, 6, 22, 16] have also achieved remarkable results on code benchmarks [12, 8]. Codex CLI [11] enables terminal-based AI coding assistance with secure sandboxed execution. Despite these advances, significant challenges remain in producing code that is semantically aligned with complex user intent, particularly for ML challenges that require specialized library knowledge and intricate data perception and handling. Our work leverages these fundamental code generation capabilities while addressing their limitations through specialized agents and modular designs with ML library integration specifically tailored for ML tasks.

3 Methodology

We present MLZero (Figure 2), a multi-agent system F that automates end-to-end solutions for multimodal ML tasks. The system processes input data x and optional user inputs U opt to produce

Raw Data

👀

Perception

Selected ML Library

Perceptual Context

🕜

Episodic

Memory

📚

Semantic

Memory

Initialization

…

⚙

…

⚙

Iterative

Coding

Outputs

🕜

📚

📚

🕜

Perceptual

Context

📚

🕜

…

⚙

…

Generation

User

Input

Raw

Data

📁

File

Grouping

💡

Task

Perception

🛠

ML Library Selection

📖

File

Perception

Data Context

Task Context

Selected ML

Library

Perceptual

Context

👀

Selected ML

Library

Perceptual

Context

Documents

⭐

Summarization

✏

Condensation

Indexing (Offline)

🕜

Error

Context

🔍

Retrieval

Query

Retrieve

Condensed Doc

Perceptual

Context

Selected

ML Library

User

Input

📚

Coding Prompt

👩💻

Coder

Python Code

🚀

Executer

Outputs

Log & Errors

If Failed

Other Prompts and I/O

Step t

Step t+1

Perceptual

Context

👀

⚙

📚

Condensed

Tutorials

❗

Error Summary

Suggested Fix

Log & Errors

Python Code

Error Context

If Success

Step t+1

👀

📚

⚙

🕜

Step 0 … t-1

Step 0 … t-1

Step t+1 …

Step t

Step t+1 …

Step 0

Step t

🕜

🕜

📚

TrainDataset/Imgs/

- *.png

(matches all PNG files)

口

TestDataset/CAMO/Imgs/

—·.png

(motches all PNG files)

train.csv

—(training data annotations)

visuelly blend with their surrondings

- Quick Start

AutoMMfor Sen

2. Specified Requirements and Constreints:

sxsu Kaulq tana-axd aq pinys ndo

GT

Output

trein,csv

predicted_masks

lapoupaos

results.csv

Complete the task within 10

minutes.

Python code using outogluon.

multinodol to troin α predictor

ond ngke predictions on test doto.

ning doto ond

solutions including predicted outputs y , code artifacts C , and execution logs L :

Note that our ML model building process for various tasks is achieved by generating code employing different ML libraries and executing it. For supervised learning tasks, x typically includes labeled training data, unlabeled test data, and a brief task description or instruction. For zero-shot tasks, x would simply consist of unlabeled test data and the task description. For example, in a supervised semantic segmentation task as illustrated in Figure 2, x includes training images and the corresponding masks, a tabular file indicating the image-mask pairs, testing images, a tabular file listing the test images, and a simple one-sentence task description. The output y would be the predicted masks for the test images, together with a tabular file that stores the test image-mask pairs. During the evaluation, we calculate the task-specific objective L x on prediction y and ground truth ˆ y , that is, L x ( y, ˆ y ) .

Our system comprises four modules, where each module is a subsystem with one or more agents, and each agent is a specialized LLM augmented with utility functions (details of each agent in Appendix B): (1) Perception that interprets arbitrary data inputs and transforms them into structured context; (2) Semantic Memory that enriches the system with knowledge of the ML Library; (3) Episodic Memory that maintains chronological execution records for targeted debugging; and (4) Iterative Coding that implements a refinement process with feedback loops and augmented memory.

3.1 Perception

The Perception module P acts as the cognitive lens of the system (Eq. 2), orchestrating the transformation of various data inputs into actionable ML workflow specifications. Input is the raw data folder

and optional user input. The output is the perceptual context P and the selected ML library M :

This module consists of three agents. File grouping and file perception agent (Appendix B.1.1) performs structural analysis of raw data x , grouping similar files and interpreting file contents. Task perception agent (Appendix B.1.2) extracts semantic information from raw data, derived context, and user input U opt to identify objectives, constraints, and evaluation criteria, generating task context in natural language. MLLibrary selection agent (Appendix B.1.3) employs context-aware reasoning to match problem characteristics with the appropriate ML Library M by analyzing the constructed task context against the library capabilities. This integrated approach efficiently translates user requirements and dataset properties into perceptual context P that guides subsequent iterative processes.

3.2 Semantic Memory

The Semantic Memory Module S t enhances the LLM's parametric knowledge with domain-specific information from external knowledge bases at each iteration t . These knowledge bases are constructed offline by two agents. The summarization agent (Appendix B.2.2) compresses relevant knowledge into concise paragraphs serving as queryable indices, while the condensation agent (Appendix B.2.1) transforms this knowledge into precise and streamlined guidance. This agent-based knowledge processing enables the Semantic Memory to incorporate documentation, API references, code examples, and other relevant resources.

At each iteration t , given the error context R t (both Error Summary and Suggested Fix), the Semantic Memory Module processes this information through its retrieval agent (Appendix B.2.3) to query the knowledge base of the selected ML library M , extracting condensed information G t :

In our case, we use tutorial documents to demonstrate the effectiveness of minimal tool initialization. By retrieving contextually relevant tutorials according to the current task and errors encountered, the system provides targeted guidance for iterative coding, substantially improving code quality while reducing LLM hallucinations (Section 4.3.1).

3.3 Episodic Memory

The Episodic Memory module, E t , enhances the success rate of MLZero in ML model building by providing error context R t at each iteration t leveraging its chronological record of the system execution history.

This component is initialized with the perception context P and progressively stores the interaction data at each iteration. The stored information includes code artifacts C t -1 , code execution logs L t -1 , retrieved knowledge G t -1 , and previous error context R t -1 . When invoked during code generation, the error analyzer agent (Appendix B.3.1) distills encountered issues and contexts into concise error summaries paired with fix suggestions. This focused approach enables subsequent coding agents to efficiently address specific problems without processing excessive contextual information, significantly reducing token consumption while maintaining problem-solving effectiveness (Section 4.3.2).

3.4 Iterative Coding

With the support of components above, our system enters an iterative coding process G t , where at each iteration t it refines the solution based on execution feedback:

For each iteration t , the system first combines the perceptual context P , optional user input U opt t 1 , error context R t , and the retrieved knowledge G t to guide the coder agent (Appendix B.4.1) in producing executable code C t , leveraging parametric, historical, and external knowledge simultaneously. The

system then executes the generated code in a configured environment, which the coder can further customize or recreate as needed. During execution, the system captures logs L t including standard output, standard error, log messages, etc., and stores the model output y t . The executer agent (Appendix B.4.2) analyzes these results and logs to determine the next steps: finalizing output y, C, L upon success or identifying errors and initiating the next coding iteration t +1 .

This iterative approach continues until either successful execution or a maximum iteration limit is reached. Notably, the system supports optional per-iteration user input, allowing for human guidance when required while maintaining a high degree of automation. Through this comprehensive system, MLZero effectively bridges the gap between noisy raw data inputs and sophisticated ML solutions, providing a truly end-to-end automated ML framework adaptive to any modalities.

4 Experiments

To evaluate the effectiveness of MLZero against state-of-the-art ML and coding agents, we conducted extensive experiments across multiple benchmarks and datasets. We first assess performance on MLEbench Lite [10] with 21 diverse Kaggle competitions, then proceeded to the Multimodal AutoML Agent Benchmark for end-to-end evaluation across 25 diverse datasets spanning various modalities and ML tasks. We evaluate the performance using multiple metrics including success rate, average rank, relative time consumption, and solution quality. Additionally, we performed ablation studies to quantify the contribution of individual components within our proposed system. Finally, we conducted a detailed error analysis to identify and categorize failure cases across high-performance methods, providing insights into the robustness and limitations of each approach, followed by efficiency and robustness analysis examining token consumption, cost effectiveness, and robustness across different LLM backbones and under various noise conditions.

4.1 Implementation Details

Each agent was assigned a 3-hour time limit per dataset to produce results. By default, MLZero uses Claude 3.7 Sonnet as its underlying LLM. It is important to note that only MLZero and Codex CLI operate truly end-to-end, while other agents require varying degrees of preprocessing or postprocessing to function on our benchmark. For example, DS-Agent requires manual code execution, while results from AIDE and AutoKaggle needed manual extraction from working directories. Complete implementation specifications, including model configurations, the prompts used, and other details for each agent, are provided in the Appendix C.

4.2 Main Results

4.2.1 MLEbench

We first evaluate MLZero on MLE-bench Lite [10], consisting of 21 diverse challenges including classification, regression, and generation, spanning various data modalities including image, text, tabular, and audio (more details: Appendix A.2). The results are shown in Figure 3. MLZero demonstrates superior performance on MLEbench-Lite with an average rank of 1.43, significantly outperforming competing approaches including AIDE [38] (2.36), ResearchAgent (MLAB) [33] (3.29), and CodeActAgent (OpenHands) [72] (2.93). Furthermore, MLZero achieves the highest success rate of 86%, compared to AIDE (81%), MLAB (62%), and OpenHands (71%). This performance advantage extends across multiple evaluation metrics, where our approach achieves six gold and two silver medals, while also outperforming competitors in medal counts. Many tasks in MLEBench Lite cannot be handled by the current integrated ML libraries, thus registering more specific ML libraries, e.g. diffusion or audio models, to MLZero can further push the performance.

While MLE-Bench [10] provides valuable comparisons between ML agents and human performance in Kaggle competitions, it relies on dedicated Python scripts to preprocess the raw data, presenting structured rather than raw inputs. This dataset-specific preprocessing significantly simplifies the ML workflow and may overestimate agent capabilities in real-world scenarios. Therefore, to enable more accurate agent-to-agent comparisons in more rigorous end-to-end ML automation, we further tested MLZero on our Multimodal AutoML Agent Benchmark, which specifically challenges agents to handle completely raw, unprocessed data without any task-specific preprocessing assistance.

4.2.2 Multimodal AutoML Agent Benchmark

Table 1 provides a detailed performance comparison between MLZero and several state-of-the-art machine learning and coding agents on our Multimodal AutoML Agent Benchmark (further details in Appendices C, and A.1). Evaluated across 25 diverse datasets, MLZero demonstrates a significant advantage. For each dataset, we conduct three independent runs and report the mean performance and standard deviation of valid submissions. We further evaluate the agents based on average rank, relative time consumption, and success rate (see Appendix A.1.13 for definition details).

MLZero with def ault configuration achieves a markedly higher success rate (92.0%) compared to all competing agents. Furthermore, MLZero consistently delivers solutions of superior quality, as evidenced by its substantially lower average rank (2.42). In addition to the primary evaluation, we test MLZero with a smaller LLM. The 8B configuration, which uses LLama 3.1 8B [22], maintains better performance (45.3% success rate, 5.14 rank average) than other agents despite a significantly reduced model size. This demonstrates the robustness of our multi-agent system design. In terms of time complexity, MLZero is significantly faster than AIDE and AutoKaggle due to the more efficient design reducing unnecessary iterations, although slower than Codex CLI, which excels in instruction following and success rate, but tends to give simple solutions that can be executed in a few minutes with degraded performance. Additionally, in the 24-hour extended run-time experiment 24hrs , we

manually extend the time constraints and improve the quality preset [19, 67, 65] to show that MLZero can approach expert-level performance (Human Reported) given sufficient computational resources.

While Codex CLI shows limited performance with its def ault setting using a general-purpose LLM [56], its performance improves substantially with the +rea configuration where it is equipped with state-of-the-art reasoning models [58]. Without step limitations and with terminal access, Codex CLI produces simple solutions that execute quickly and iterate efficiently, enabling easier error correction. This approach yields a high success rate and, consequently, a good ranking. However, its performance remains low on challenging datasets, revealing limitations in complex problem-solving.

Considering that each baseline method has different designs regarding the use of external knowledge, we conduct different augmentations to ensure a fair comparison and evaluate methods on equal footing. To assess our method without the advantage of its integrated external knowledge, the -ext configuration removes all access to external ML libraries. This configuration yields a 69.3% success rate and a 4.94 average rank, still outperforming all competitors and highlighting the efficiency of MLZero's other components. We also investigate the contribution of episodic memory through the -epi configuration, which removes episodic memory but retains the LLM's conversation history within the coder agent. This setup achieves an 86.7% success rate with a 2.86 average rank, demonstrating that while episodic memory provides significant benefits, maintaining a coherent conversational context still yields reasonable performance. In contrast, we explore modifying AIDE to access external knowledge, indicated as +ext . While it shows improvement, MLZero continued to outperform it under these comparable conditions. This result underscores that the superior performance of MLZero stems from its overall system design, not merely from the inclusion of episodic memory ( epi ) or external knowledge ( ext ) in isolation.

Rank

#Gold

#Silver+

#Bronze+

#Median+

Success

[4-1.4] (lower is better)

[0-6.0]

[0-8.0]

[0-8.0]

[0-12.0]

[0.5-0.9]

MLEBench-Lite Results

Ours

AIDE

MLAB

OpenHands

4.3 More Ablation Studies

To further investigate the effectiveness of an individual component in the overall system, we conduct ablation studies in eight diverse datasets that span tabular data, multimodal data, multilingual and multitable data, document classification, image segmentation, time series forecasting and text retrieval (detailed information available in Appendix A.3). For each ablation experiment, we compare modified configurations with our default system using three primary metrics: (1) success rate in generating valid solutions, (2) token efficiency, and (3) time efficiency. Both token efficiency and time efficiency metrics are calculated as the reciprocal of relative token usage and execution time, respectively, measured only for valid solutions to ensure fair comparisons.

4.3.1 More Ablation: Semantic Memory

To further assess our semantic memory module, we evaluated several configurations with varying degrees of functionality. The def ault configuration represents our complete system with semantic memory using condensation, summarization, and 5 retrieved documents. w/o Condensation uses full tutorials without the condensation process, while w/o Summary performs retrieval based only

Default

No Summarization

No Condensation

Experimental Setting

0.6

0.8

1.0

1.2

1.4

Performance Metrics

1.00

1.00

0.88

1.00

0.92

1.14

1.00

0.79

0.93

Performance Metrics Across Different Offline Indexing Settings

Baseline (Default)

Success Rate

Time Efficiency

Token Efficiency

0

1

3

5

10

Number of Retrieved Documents

0.6

0.8

1.0

1.2

1.4

Performance Metrics

0.62

0.62

0.88

1.00

0.88

0.67

1.16

0.90

1.00

1.12

0.68

1.35

0.77

1.00

0.80

Performance Metrics Across Different Retrieval Counts

Baseline (5 Docs Retrieved)

Success Rate

Time Efficiency

Token Efficiency

on tutorial titles, without summary-based indexing. We also evaluated Retrieval Size variants by varying the number of documents retrieved (0, 1, 3, 5, 10).

Figure 4 (top) demonstrates that condensation effectively reduces token consumption while preserving performance. Additionally, summary-based retrieval surpasses title-based retrieval, highlighting the importance of content-aware indexing for efficient knowledge access. Figure 4 (bottom) reveals how varying retrieval quantities affects performance. While increasing the number of retrievals generally enhances results, benefits level off after five documents. The token and time usage increase nearly linearly with retrieval quantity, creating a distinct trade-off between performance and computational efficiency. In the zero-retrieval configuration, only the library name is provided to subsequent agents, which then rely solely on the LLM's parametric knowledge of the library.

These findings validate our design choices for the semantic memory module, demonstrating that the combination of condensed tutorials, summary-based retrieval, and an appropriate retrieval size (5 documents) achieves an optimal balance between performance and resource utilization.

4.3.2 More Ablation: Episodic Memory

To evaluate the contribution of episodic memory, we compared three configurations as shown in Table 2. The def ault configuration with both error summaries and suggested fixes achieves the best overall success rate (1.0). The w/o Sum & Fix configuration removes both components, prompting the subsequent coder with truncated error messages and executer's analysis instead. It shows token efficiency advantages at the cost of decreased success rate (0.83). Interestingly, the w/o Fix configuration, retaining error summaries but removing suggested fixes, has the lowest success rate (0.75) with the worst token efficiency due to the increased number of iterations used to figure out the solution, suggesting that providing over-condensed error summaries without actionable fix suggestions may further confuse the coder. These results indicate that complete episodic memory produces better outcomes, particularly for complex problems requiring multiple debugging cycles.

4.4 Failure Cases and Analysis

Among the cases that fail, we conduct an error analysis across MLZero, Codex CLI +rea, AIDE, and DS-Agent (DS), categorizing failures as shown in Table 3. MLZero demonstrates remarkable robustness with minimal errors (6.5% overall in 46 datasets), while competing methods exhibit higher failure rates: Codex CLI (26.9%), AIDE (47.8%), and DSA (76.0%). Notably, MLZero completely eliminates several error categories, including perception, postprocessing, and API Hallucination. It is important to note that error frequencies may be influenced by system progression, i.e. systems that fail earlier (e.g., at perception/preprocessing) never encounter later-stage errors that more capable systems might face when attempting more complex solutions. A detailed discussion of error patterns and their implications for future ML agent design is provided in Appendix E.3.

4.5 Efficiency and Robustness Analysis

To assess computational efficiency, we compared token consumption and cost across systems on the benchmark. As shown in Table 4 (left top), MLZero achieves superior performance (92% success

rate) while using 4.3 × fewer tokens than AIDE (63k vs. 273k per dataset on average) and incurring only 23% of its cost ($0.19 vs. $0.82 per dataset). The w/o Fix configuration demonstrates further token reduction (51k) but at the expense of success rate.

We further evaluated robustness across different LLM backbones (Claude Sonnet 3.7, GPT-4.1, and Llama 3.1 8B) on the eight challenging datasets spanning all modalities (Table 4, left bottom). Results show consistent performance across all models, with the compact 8B model still outperforming all baseline systems from Table 1. Note that X indicates execution timeout or failure, and scores with ↓ indicate lower is better for regression tasks.

To test resilience against noisy inputs, we conducted stress tests with deliberately misleading dataset descriptions on the Abalone tabular regression task (Table 4, right). The table shows different noise conditions: misleading information in the dataset description (Noise Des.), incorrect instructions from the user (Noise Ins.), and removal of the correct library from available options (-Lib). The results columns indicate whether the perception module correctly identified the task (Perc.) and whether the overall end-to-end task was completed successfully (Succ.). Even when provided incorrect task type hints or library suggestions, the perception module correctly identified task characteristics in all cases. When explicitly directed to use incorrect tools via user instruction, the system successfully recovered through restart mechanisms, demonstrating robust error detection and correction capabilities.

5 Conclusion

In this paper, we introduced MLZero, a novel hierarchical multi-agent system that reimagines multimodal ML automation through the integration of multiple LLM agents with perception, semantic and episodic memory, and iterative coding mechanisms. Our comprehensive evaluation on MLEbench Lite demonstrates superior performance across 21 diverse Kaggle competitions. Further rigorous end-to-end evaluation on our Multimodal AutoML Agent Benchmark across 25 diverse datasets confirms that MLZero significantly outperforms existing approaches with superior solution quality and a +263% improvement in success rate, while our perception module enables truly end-toend automation without human intervention. This addresses the limitations of existing benchmarks and provides a more accurate assessment of agent capabilities on raw, unprocessed data.

Broader Impact : We believe that this advancement facilitates the democratization of sophisticated ML methodologies, enabling individuals with limited domain expertise to address complex ML challenges effectively. The environmental impact of running LLMs warrants consideration, which presents the need for training smaller LLMs to achieve comparable performance. More limitations and future work can be found in Appendix G.

References

Melbourne Airbnb Open Data. https://www.kaggle.com/datasets/tylerx/ melbourne-airbnb-open-data . Accessed: 2024-02-02.

PetFinder.my Adoption Prediction. https://www.kaggle.com/c/ petfinder-adoption-prediction/data/ . Accessed: 2024-02-02.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Abien Fred Agarap. Statistical analysis on e-commerce reviews, with sentiment classification using bidirectional recurrent neural network (rnn). arXiv preprint arXiv:1805.03687 , 2018.

Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 , 2024.

Anthropic. Claude 3.7 sonnet. https://www.anthropic.com/claude , 2025.

Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision , pages 993-1003, 2021.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021.

Jock Blackard. Covertype. UCI Machine Learning Repository, 1998. DOI: https://doi.org/10.24432/C50K5N.

Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095 , 2024.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc'Aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems , 35:32053-32068, 2022.

Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, et al. Sela: Tree-search enhanced llm agents for automated machine learning. arXiv preprint arXiv:2410.17238 , 2024.

Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018) , pages 168-172. IEEE, 2018.

DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.

Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate-fever: A dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614 , 2020.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research , 20(55):1-21, 2019.

Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505 , 2020.

Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning. Journal of Machine Learning Research , 23(261):1-61, 2022.

Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. Advances in neural information processing systems , 28, 2015.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio Iacobacci, Abdelhakim Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level. arXiv preprint arXiv:2411.03562 , 2024.

Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning. arXiv preprint arXiv:2402.17453 , 2024.

Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag, et al. Analysis of the automl challenge series. Automated Machine Learning , 177:177-219, 2019.

Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems , 36:45870-45894, 2023.

Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference on Document Analysis and Recognition (ICDAR) .

Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Knowledgebased systems , 212:106622, 2021.

Noah Hollmann, Samuel Müller, and Frank Hutter. Caafe: Combining large language models with tabular predictors for semi-automated data science. In 1st Workshop on the Synergy of Scientific and Machine Learning Modeling@ ICML2023 , 2023.

Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An llm agent for data science. arXiv preprint arXiv:2402.18679 , 2024.

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology , 33(8):1-79, 2024.

Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint arXiv:2308.00675 , 2023.

Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302 , 2023.

Jan Huizinga, Hans De Moel, and Wojciech Szewczyk. Global flood depth-damage functions: Methodology and the database with guidelines. Technical report, Joint Research Centre (Seville site), 2017.

Ganesh Jawahar, Muhammad Abdul-Mageed, Laks VS Lakshmanan, and Dujian Ding. Llm performance predictors are good initializers for architecture search. arXiv preprint arXiv:2310.16712 , 2023.

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 , 2024.

Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology , 33(7):1-30, 2024.

Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138 , 2025.

Haifeng Jin, François Chollet, Qingquan Song, and Xia Hu. Autokeras: An automl library for deep learning. Journal of machine Learning research , 24(6):1-6, 2023.

Israel Campero Jurado, Pieter Gijsbers, and Joaquin Vanschoren. Automl benchmark with shorter time constraints and early stopping. arXiv preprint arXiv:2504.01222 , 2025.

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes, 2021.

Trung-Nghia Le, Tam V Nguyen, Zhongliang Nie, Minh-Triet Tran, and Akihiro Sugimoto. Anabranch network for camouflaged object segmentation. Computer vision and image understanding , 184:45-56, 2019.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 , 2023.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022.

Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424 , 2024.

Zujie Liang, Feng Wei, Wujiang Xu, Lin Chen, Yuxi Qian, and Xinhui Wu. I-mcts: Enhancing agentic automl via introspective monte carlo tree search. arXiv preprint arXiv:2502.14693 , 2025.

Siyi Liu, Chen Gao, and Yong Li. Large language model agent for hyper-parameter optimization. arXiv preprint arXiv:2402.01881 , 2024.

Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic framework with extensible tools for complex reasoning. arXiv preprint arXiv:2502.11271 , 2025.

Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems , 36:43447-43478, 2023.

Shreyash Mishra, S Suryavardan, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, Asif Ekbal, and Srijan Kumar. Memotion 3: Dataset on sentiment and emotion analysis of codemixed hindi-english memes, 2023.

Volodymyr Mnih. Machine learning for aerial image labeling . University of Toronto (Canada), 2013.

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 , 2022.

Alhassan Mumuni and Fuseini Mumuni. Automated data processing and feature engineering for deep learning and big data applications: a survey. Journal of Information and Intelligence , 2024.

Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C55C7W.

Muhammad Umair Nasir, Sam Earle, Julian Togelius, Steven James, and Christopher Cleghorn. Llmatic: neural architecture search via large language models and quality diversity optimization. In proceedings of the Genetic and Evolutionary Computation Conference , pages 1110-1118, 2024.

OpenAI. Introducing gpt-4.1 in the api, April 2025. Accessed: 2025-05-02.

OpenAI. Openai codex cli: Lightweight coding agent that runs in your terminal. https: //github.com/openai/codex , April 2025. Command-line tool for local, chat-driven development with OpenAI models.

OpenAI. Openai o3 and o4-mini system card, April 2025. Accessed: 2025-05-02.

OpenAI. Openai o3-mini. https://openai.com/index/openai-o3-mini/ , 2025. Accessed: 2025-03-25.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 , 2023.

David Salinas and Nick Erickson. Tabrepo: A large scale repository of tabular model evaluations and its automl applications. arXiv preprint arXiv:2311.02971 , 2023.

Dietmar Schabus, Marcin Skowron, and Martin Trapp. One million posts: A data set of german online discussions. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , pages 1241-1244, Tokyo, Japan, August 2017.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems , 36:68539-68551, 2023.

Holger Schwenk and Xian Li. A corpus for multilingual document classification in eight languages. In Nicoletta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) , Paris, France, may 2018. European Language Resources Association (ELRA).

Oleksandr Shchur, Ali Caner Turkmen, Nick Erickson, Huibin Shen, Alexander Shirkov, Tony Hu, and Bernie Wang. Autogluon-timeseries: Automl for probabilistic time series forecasting. In International Conference on Automated Machine Learning , pages 9-1. PMLR, 2023.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems , 36:38154-38180, 2023.

Zhiqiang Tang, Haoyang Fang, Su Zhou, Taojiannan Yang, Zihan Zhong, Tony Hu, Katrin Kirchhoff, and George Karypis. Autogluon-multimodal (automm): Supercharging multimodal automl with foundation models. arXiv preprint arXiv:2404.16233 , 2024.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355 , 2018.

Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent llm framework for full-pipeline automl. arXiv preprint arXiv:2410.02958 , 2024.

Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data , 5(1):1-9, 2018.

Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations , 2024.

Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561 , 2022.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.

Li Yang and Abdallah Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing , 415:295-316, 2020.

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 , 2023.

Aidin Zehtab-Salmasi, Ali-Reza Feizi-Derakhshi, Narjes Nikzad-Khasmakhi, Meysam AsgariChenaghlu, and Saeideh Nabipour. Multimodal price prediction. Annals of Data Science , 10(3):619-635, 2023.

Haochen Zhang, Yuyang Dong, Chuan Xiao, and Masafumi Oyamada. Large language models as data preprocessors. arXiv preprint arXiv:2308.16361 , 2023.

Michael R Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. arXiv preprint arXiv:2312.04528 , 2023.

Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. Retrieve anything to augment large language models, 2023.

Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B Tenenbaum, and Chuang Gan. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510 , 2023.

Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. In International Conference on Machine Learning , pages 41880-41901. PMLR, 2023.

Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, and Qianli Ma. Lifelong learning of large language model based agents: A roadmap. arXiv preprint arXiv:2501.07278 , 2025.

Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finetuning for segment anything model. arXiv preprint arXiv:2401.17868 , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the paper's contributions: (1) introducing MLZero, a hierarchical multi-agent system with specialized perception agents and dual-memory system for end-to-end ML automation; (2) establishing a comprehensive benchmark suite for evaluation; and (3) demonstrating superior empirical performance with a 92.0% success rate versus 25.3% for competing approaches. These claims align with the paper's scope of addressing limitations in current multimodal ML automation systems through a novel system that handles diverse data types with minimal human intervention.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper includes a dedicated "Limitations and Future Work" section (Section G) that explicitly discusses several limitations of MLZero, including its reduced effectiveness with minimally documented ML libraries, inability to address undocumented bugs in ML libraries without explicit prompting, and performance gaps between smaller (8B) and larger language models. The section also outlines future research directions to address these limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We propose a hierarchical multi-agent system for multimodal ML automation with no theoretical results requiring proofs. We provides equations describing the system's components, but these are defining the system rather than presenting theorems that need formal proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please check Appendix C for the implementation details, and Appendix B for the design and prompts for each agent. Our source codes are also attached in supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Source codes and datasets, as well as instructions to run them, are included in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Appendix C for complete implementation details and Appendix A for comprehensive dataset information. Our method does not require hyperparameter tuning, with the only adjustment being maximum token size constraints to ensure compatibility with smaller language models.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the mean and standard deviation across three independent runs for all experiments in Table 1, providing a clear measure of the statistical variability in our results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources to reproduce the experiments are included in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm our research fully adheres to all its principles and guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impact is included in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: They are properly cited and credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide comprehensive documentation alongside our code implementation, including detailed instructions for usage and reproducibility. All components of our MLZero agent system are thoroughly documented in Appendix B, which describes each agent's role, capabilities, and interactions.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Contents

A Dataset Details

A.1 Details of Multimodal AutoML Agent Benchmark

The mldoc dataset has varying training set sizes (1K, 2K, 5K, 10K) across multiple languages (German, English, Spanish, French, Italian).

A.1.1 Construction

We constructed the Multimodal AutoML Agent Benchmark (MAAB) to address a critical gap in existing benchmarks: the ability to evaluate agents on raw, unprocessed multimodal data. To ensure

fairness and diversity, all 25 datasets are sourced from reputable public repositories including Kaggle competitions, UCI Machine Learning Repository, and the BEIR benchmark suite. Datasets were selected from published machine learning research, with most lacking direct open-source solutions to prevent contamination. The benchmark spans 7 distinct problem types (binary classification, multiclass classification, multi-label classification, regression, segmentation, retrieval, and time series forecasting) across multiple data modalities including tabular, text, image, document, and multimodal combinations. We open-source the complete benchmark including all datasets, evaluation scripts, and preprocessing specifications. Our evaluation follows standard practices: the knowledge base contains only general library documentation with no information about evaluation datasets or their solutions, and test sets remain completely held out during all execution stages.

A.1.2 Preprocessing

To maintain the realistic challenge of processing raw data while ensuring fair comparison across methods, we implemented minimal preprocessing interventions. Specifically, we: (1) augmented datasets lacking descriptions with brief task specifications to accommodate baseline agent requirements, (2) removed target columns from test files to prevent data leakage to agent.

A.1.3 Tabular Classification and Regression

We select five diverse datasets (with details below) for tabular classification and regression tasks from TabRepo [61]. To accommodate baseline agent requirements, we augmented each dataset with brief task descriptions. These descriptions provide minimal but sufficient context for evaluation across all comparative systems while ensuring consistency in task. For instance, the abalone dataset includes the description:

Description for Abalone (Tabular Classification and Regression)

Regression on Class_number_of_rings. Eval metric is RMSE.

The Abalone Dataset (abalone) [54, 61] consists of 3 , 759 training instances and 418 testing instances, totaling 4 , 177 samples. Each observation contains 8 input features: sex (categorical: Male, Female, or Infant), and 7 physical measurements including length, diameter, height, whole weight, shucked weight, viscera weight, and shell weight. The regression task involves predicting the number of rings in the abalone shell, which serves as a proxy for the abalone's age. The evaluation metric for this regression task is Root Mean Square Error (RMSE). The dataset presents a challenging regression problem due to the non-linear relationship between the physical measurements and the target variable, making it a valuable benchmark for assessing regression algorithms.

The Airlines Dataset (airlines) [61] consists of 485,444 training instances and 53,939 test instances for flight delay prediction. Each instance contains 8 features: airline carrier code, flight number, origin airport, destination airport, day of week (1-7), time of departure (in minutes), flight duration (in minutes), and the binary target variable indicating delay status (0 for on-time, 1 for delayed). The dataset captures commercial flight information across various U.S. airports, carriers, and time periods. This binary classification problem represents a practical application in transportation logistics, where accurate delay predictions can significantly impact resource allocation, scheduling decisions, and customer satisfaction in the aviation industry.

The Bioresponse Dataset (bio) [61] contains 3,751 samples (3,375 training, 376 testing) for binary classification of molecular biological responses. Each row represents a molecule with a binary target indicating response presence (1) or absence (0). The feature space consists of 1,776 normalized molecular descriptors (D1-D1776) representing chemical properties such as size, shape, and elemental constitution.

The Covertype Dataset (covertype) [9, 61] consists of 522 , 910 training samples and 58 , 102 testing samples, totaling 581 , 012 instances. It represents a multiclass classification problem aimed at predicting forest cover types from cartographic variables without remotely sensed data. The dataset encompasses four wilderness areas within the Roosevelt National Forest in northern Colorado, characterized by minimal human disturbance, thus reflecting primarily ecological processes rather than forest management influences. The target variable classifies areas into seven cover types, including Spruce/Fir, Lodgepole Pine, Ponderosa Pine, Cottonwood/Willow, Aspen, Douglas-fir, and

Krummholz. Predictor variables include elevation, aspect, slope, and various soil characteristics represented as binary features. This dataset is particularly valuable for evaluating algorithmic performance on high-dimensional, imbalanced, multiclass problems with geographic dependencies.

The Yolanda Dataset (yolanda) [25, 61] consists of 400 , 000 total samples, divided into 360 , 000 training instances and 40 , 000 testing instances. This dataset was utilized for a regression task, where the objective is to predict the values in column "101" based on the preceding features.

A.1.4 Multimodal Classification and Regression

We evaluated our system on nine diverse multimodal classification and regression datasets) [67]: airbnb (airbnb melbourne) [1], cd18 (cd18 cellphone) [77], flood (european flood depth) [34], gnad10 [62], ham10000 [71], hateful (hateful meme) [41], funding (kick start funding), petfinder [2], and clothing (women clothing review) [4]. These datasets incorporate various combinations of tabular, text, and image data, presenting distinct challenges for automated machine learning systems. Similarly, we supplemented datasets lacking descriptions with concise task specifications to ensure consistent evaluation across all compared methods.

Description for CD18 (Multimodal Classification and Regression)

The goal of this research is to achieve an arrangement to predict the price of a cellphone based on its specifications.

The Airbnb Melbourne Dataset (airbnb) [1, 24, 67] consists of 22,895 listings, partitioned into 18,316 training samples and 4,579 testing samples. The dataset captures various features of Airbnb accommodations in Melbourne, Australia. The target variable is price_label , which represents categorized price ranges grouped into distinct bins. To prevent data leakage, the raw price feature has been removed from the dataset. This classification task requires models to predict the price category of a listing based on its characteristics, such as location, property type, amenities, and host attributes, making it suitable for evaluating price prediction algorithms in the short-term housing market.

Cellphone Dataset (cd18) [77, 67] contains 2,533 training samples and 632 testing samples, encompassing diverse attributes such as model information, physical characteristics (weight), technical specifications (processor, RAM, storage, display dimensions, resolution), battery information, camera capabilities, operating system, and market metrics (hit count). The dataset spans multiple manufacturers, device generations, and price points, providing the information for developing and evaluating predictive models for cellphone pricing based on technical specifications.

European Flood Depth Dataset (flood) [34, 67] comprises 3,710 images for binary classification, with 3,153 samples in the training set and 557 samples in the test set. Each image is labeled as either 'Not useful for determining depth' (0) or 'Useful for determining depth' (1), representing the image's utility in flood depth estimation. The dataset includes metadata such as image identifiers, confidence scores, and bounding box coordinates. This dataset serves as a foundation for developing models that can automatically identify imagery suitable for flood depth analysis, potentially enhancing rapid damage assessment capabilities during flooding events.

The 10kGNAD Dataset (gnad10) [62, 67] consists of 10 , 273 German news articles collected from an Austrian online newspaper, categorized into 9 distinct classes: Web, Panorama, International, Wirtschaft, Sport, Inland, Etat, Wissenschaft, and Kultur. The dataset is split into 8 , 228 training samples, 1 , 017 validation samples, and 1 , 028 testing samples. Article titles and text are concatenated, with author information deliberately removed to prevent classification based on author-specific keywords. It serves as a benchmark for German topic classification tasks, addressing the unique challenges posed by the German language's higher inflection rate and longer compound words compared to English.

The HAM10000 Dataset (ham10000) [71, 67] consists of 10,015 training and 1,512 testing dermatoscopic images collected from diverse populations using various acquisition modalities. We removed the dx_type feature to prevent potential data leakage in our experimental setup. The dataset encompasses seven diagnostic categories of pigmented skin lesions: actinic keratoses and Bowen's disease ( akiec ), basal cell carcinoma ( bcc ), benign keratosis-like lesions ( bkl ), dermatofibroma

( df ), melanoma ( mel ), melanocytic nevi ( nv ), and vascular lesions ( vasc ). Ground truth diagnoses were established through histopathology (>50% of cases), follow-up examination, expert consensus, or in-vivo confocal microscopy. The dataset includes metadata on patient demographics (age, sex), lesion localization, and unique lesion identifiers that allow tracking multiple images of the same lesion.

The Hateful Memes Dataset (hateful) [41, 67] is a multimodal benchmark consisting of 7 , 134 training and 1 , 794 testing examples designed to evaluate hate detection systems. Each sample contains an image paired with overlaid text, requiring models to understand both visual and linguistic content and their potentially harmful interactions. The dataset is labeled binary, where 1 indicates hateful content (targeting protected categories through harmful stereotypes, derogatory language, or harmful contexts) and 0 represents benign content. The challenging nature of this dataset stems from the necessity for models to comprehend cross-modal relationships, as the hateful nature often emerges from the interaction between text and imagery rather than from either modality independently.

The Kickstarter Funding Dataset (funding) [67] consists of 112 , 330 crowdfunding campaign records ( 89 , 879 for training and 22 , 451 for testing) extracted from the Kickstarter platform. Each record contains 10 features including campaign name, description, funding goal amount, keyword identifiers, communication preferences, country of origin, currency type, deadline timestamps, creation timestamps, and the target variable (final_status) indicating campaign success or failure. The binary classification task involves predicting whether a crowdfunding campaign will successfully reach its funding goal based on the provided features. This dataset captures the multifaceted nature of crowdfunding dynamics across various project categories, geographical regions, and funding scales.

The PetFinder.my Adoption Prediction Dataset (petfinder) [2, 67] consists of 14 , 993 samples ( 11 , 994 training, 2 , 999 testing) designed for predicting pet adoption speed. Each entry contains pet characteristics including type, age, breed, gender, color, size, health status, and vaccination records. Additional features encompass textual descriptions, rescuer information, quantity of photos and videos, adoption fees, and geographical location. The target variable, AdoptionSpeed, is categorized on a scale indicating how quickly pets were adopted. The dataset also includes image paths for visual analysis, making it suitable for multimodal machine learning approaches to predict adoption outcomes.

The Women's E-Commerce Clothing Reviews Dataset (clothing) [4, 67] comprises 23 , 486 customer reviews of women's clothing products from an e-commerce retailer. Each observation contains 10 features: a unique clothing identifier, reviewer's age, review title, review text, product rating (1-5 scale), binary recommendation indicator, positive feedback count (number of customers finding the review helpful), and three hierarchical product categorization variables (division, department, and class names). For our experiments, we employed an 8-1-1 train-validation-test split of the data. The dataset enables analysis of customer sentiment, recommendation patterns, and demographic preferences across different clothing categories, making it suitable for natural language processing, sentiment analysis, and recommendation system research.

A.1.5 Document Classification

Document classification entails the categorization of visual documents, which can be approached as an image classification task, an OCR-based text classification task, or as a multimodal problem integrating both methodologies. Our experiments utilize the RVL-CDIP (Ryerson Vision Lab Complex Document Information Processing) [27] dataset, comprising 400,000 grayscale images distributed across 16 distinct document classes with 25,000 images per class. The dataset is partitioned into 320,000 training images, 40,000 validation images, and 40,000 test images. This dataset presents particular challenges due to its substantial size and intricate file structure, highlighting the necessity of effective perception rather than naïve loading of all file structures into the LLM, which would exceed context limitations.

Data Structure and Description for RVL-CDIP (Document Classification)

Data Structure

---------- rvl_cdip/training/inference.txt

The RVL-CDIP Dataset (rvlcdip) [27] comprises 400 , 000 grayscale document images scanned at approximately 100 dpi, distributed across 16 distinct semantic categories including letters, forms, emails, resumes, and memos. The dataset is partitioned into 320 , 000 training images, 40 , 000 validation images, and 40 , 000 test images, with each class being represented equally. The corpus is characterized by significant quality variations typical of real-world document collections, including scanning artifacts, noise, skew, and low resolution, making it a challenging benchmark for document classification tasks.

A.1.6 Multilingual and Multitable Classification

We employed the mldoc (MLDoc-11000) [64] dataset for evaluating multilingual classification capabilities, extending beyond the monolingual English usage in previous work [67]. Our configuration incorporates multiple languages (German, English, Spanish, French, and Italian) simultaneously. Specifically, we concatenated test files from all five languages to form a multilingual test set while preserving the original structure of the remaining data files.

Description for MLDoc-11000 (Multilingual and Multitable Classification)

This is a multi lingual task. Train a multilingual model and give the predictions.

The Multilingual Document Classification Corpus (mldoc) [64] is a cross-lingual text classification benchmark. It encompasses eight languages: English, German, French, Spanish, Italian, Russian, Japanese, and Chinese. MLDoc features balanced class distribution across all languages and follows a standardized train/dev/test split. For most languages, training sets of varying sizes (1K, 2K, 5K, and 10K documents) are provided, with Spanish and Russian having 9,458 and 5,216 training documents respectively. Each language includes a 1K development set and a 4K test set. In our experiments, we

utilize the German, English, Spanish, French, and Italian subsets, treating this as both a multi-table and multilingual classification task, with separate training tables for each language and combined test files.

A.1.7 Multimodal Classification (Multilabel)

We adopted the memotion [50] dataset for multilabel classification. We incorporates all five classification dimensions: humor (Not humorous, funny, very funny, hilarious), sarcasm (Not Sarcastic, general, twisted meaning, very twisted), offensive (Not offensive, slight, very offensive, hateful offensive), motivational (Not Motivational, Motivational), and overall sentiment (Negative, Very Negative, Positive, Very Positive, Neutral). We employ average accuracy as the evaluation metric to assess performance across all dimensions.

Description for Memotion (Multimodal Multilabel Classification)

Train the model on image and corrected text to predict "humour", "sarcasm", "offensive", "motivational", "overall_sentiment" of the meme.

humour: Not

humorous,

funny,

very

funny,

hilarious

sarcasm: Not

Sarcastic,

general,

twisted

meaning,

very

twisted

offensive: Not

offensive,

slight,

very

offensive,

hateful

offensive

motivational: Not

Motivational,

Motivational

overall_sentiment: Negative,

Very

Negative,

Positive,

Very

Positive,

Neutral

The Memotion Dataset (memotion) [50, 67] The Memotion dataset comprises 5 , 593 training and 1 , 399 testing samples, each consisting of meme images paired with corrected text annotations. This dataset is designed for multi-label classification across five dimensions: humor (four classes: not humorous, funny, very funny, hilarious), sarcasm (four classes: not sarcastic, general, twisted meaning, very twisted), offensiveness (four classes: not offensive, slight, very offensive, hateful offensive), motivation (binary: not motivational, motivational), and overall sentiment (five classes: negative, very negative, positive, very positive, neutral). The dataset enables research on automated understanding of emotional and contextual elements in internet memes, addressing the complex interplay between visual and textual components in multimodal sentiment analysis.

A.1.8 Semantic Segmentation

We include three semantic segmentation datasets: camoseg (camouflaged segmentation) [42], roadseg (road segmentation) [51], and isic2017 [15]. Each dataset presents unique visual analysis challenges requiring pixel-level prediction capabilities. To standardize evaluation, we provided descriptive task specifications; for instance, the camoseg dataset includes the description:

Description for Camouflaged Segmentation (Semantic Segmentation)

Perform semantic segmentation to identify and delineate camouflaged objects that visually blend with their surroundings, outputting pixel-level binary masks.

The Camouflage Segmentation (camoseg) [67, 42, 84] dataset consists of 4 , 290 images with corresponding pixel-level binary masks, designed for identifying and segmenting camouflaged objects that visually blend with their surroundings. The dataset is partitioned into training ( 3 , 636 images), validation ( 404 images), and testing ( 250 images) subsets. Each sample comprises an input image paired with its ground truth segmentation mask, where camouflaged objects are delineated at the pixel level. The dataset encompasses a diverse range of naturally camouflaged subjects, demonstrating various camouflage mechanisms observed in natural environments.

The ISIC 2017 Dataset (isic2017) [15, 67, 84] consists of dermoscopic images used for automated melanoma diagnosis, specifically focused on semantic segmentation of skin lesions. The dataset comprises 2 , 000 training images and 600 testing images, each paired with expert-annotated ground truth

segmentation masks, aimed at advancing computer vision techniques for dermatological applications and improving early detection of melanoma and other skin cancers.

The Massachusetts Roads Dataset (roadseg) [51, 67, 84] consists of satellite imagery for road segmentation tasks, comprising 1 , 107 training samples, 13 validation samples, and 48 testing samples. Each sample consists of an input satellite image paired with its corresponding ground truth road segmentation mask. The dataset is structured to facilitate the development and evaluation of machine learning models for automated road network extraction from aerial imagery. The input images are stored as PNG files in designated directories, with corresponding output masks containing pixel-wise road annotations.

A.1.9 Retrieval

We incorporated two retrieval datasets from the BeIR benchmark [68]: fiqa (FiQA-2018) and climate (Climate Fever). The Climate Fever dataset serves as a zero-shot evaluation as it lacks a training set, testing systems' ability to perform retrieval without task-specific training data. To optimize computational resources during benchmarking, we reduced the Climate Fever corpus size to one-tenth of its original scale while maintaining the positive documents. In the description file we specify the output format for easier evaluation:

The Climate-FEVER Dataset (climate) [17, 68] is a zero-shot fact verification dataset consisting of 1 , 535 testing query-document pairs without training data. The original corpus contains 5 , 418 , 128 documents; however, for computational efficiency, we utilize a 10% subset of the corpus while ensuring all ground truth documents are preserved. The dataset follows the FEVER [69] framework but applies it to real-world climate change-related claims collected from the Internet, offering unique challenges in natural language understanding. For our experiments, we require retrieval output using the standard format of query-id, corpus-id, and relevance score, evaluating the recall for top-10 retrieved documents for each query.

The FiQA-2018 Dataset (fiqa) in the BEIR benchmark [68] consists of financial question-answering data with 14 , 166 training samples, 1 , 238 validation samples, and 648 test samples. The corresponding document corpus contains 57 , 638 documents. We evaluated performance by retrieving the top10 most relevant documents for each query in the test set and compute the recall.

A.1.10 Timeseries Forecasting

We incorporated three timeseries forecasting datasets [65, 5] with varying temporal resolutions: nn5(D) (nn5_daily), solar(10m) (solar_10_minutes), and electric(H) (electricity_hourly). The training data is provided in compressed JSONL format with json.gz extension. To mitigate potential confusion regarding data format, we supplemented each dataset with detailed format specifications. For example, the nn5_daily dataset includes the following description:

The Electricity Hourly Dataset (electric(H)) consists of 321 distinct time series representing electricity consumption measurements collected at hourly frequency. Each series contains 8 , 428 , 176 time steps, spanning multiple years of observations. The forecasting task involves predicting 24 steps (hours) into the future. The dataset is structured in JSONL format, where each record contains a target vector of consumption values, a timestamp indicating the starting point, a unique identifier for each series, and categorical features. Models trained on this dataset are evaluated using the Mean Absolute Scaled Error (MASE) metric.

The NN5 Daily Dataset (nn5(D)) [65, 5] comprises 111 time series from the banking domain, specifically recording daily cash withdrawal amounts from automated teller machines (ATMs) across the United Kingdom. The dataset was designed to evaluate forecasting methods' efficacy in predicting daily ATM withdrawal patterns. To address data quality concerns, missing values in the original dataset were imputed using a day-of-week median replacement strategy, whereby each missing observation was substituted with the median value calculated from the same weekday across the entire corresponding time series. The forecasting task involves predicting 56 steps (days) into the future.

The Solar Dataset (solar(10m)) [65, 5] comprises 137 time series representing 10-minute observations of solar power production recorded per every 10 minutes in Alabama state in 2006. The forecasting task involves predicting 60 steps (10-minutes) into the future.

A.1.11 Evaluation

We implemented a rigorous evaluation protocol to ensure fair and consistent assessment across all systems. As previously noted, we removed target columns from test files to prevent data leakage to agents. The complete test data remained accessible only within the evaluation environment, where we performed strict line-by-line ordered comparisons between agent outputs and ground truth. This protocol enforces format compliance, i.e. deviations in output structure, naming conventions, or completeness result in evaluation failure. To facilitate consistent evaluation while maintaining the challenge of raw data processing, we maintained a metadata file for evaluator reference (not exposed to agents):

This structured evaluation framework ensures that success metrics reflect not only predictive performance but also practical usability in real-world scenarios where output format requirements are equally critical.

A.1.12 Human Reported Results

The human reported baseline results are retrieved from established open-source benchmarks and literature: camoseg [67], electric(H) [65], flood [67], fiqa [52], gnad10 [67], ham10000 [67], hateful [67], isic2017 [67], funding [67], nn5(D) [65], petfinder [67], roadseg [67], rvlcdip [7], and clothing [67].

For the remaining datasets, comparable metrics from open-source implementations are unavailable, either due to the absence of published performance metrics, differences in evaluation metrics, or because our modifications to these datasets preclude direct comparison.

A.1.13 Average Rank, Relative Time, and Success

The average relative time usage (Rel. Time) quantifies computational efficiency across models by normalizing execution times against the reference model MLZero (def). For each dataset d in the collection D , we compute the ratio of execution time t m,d for model m relative to the reference model's time t ref,d . The average relative time usage T m for model m is then defined as:

where |D m,ref | represents the number of datasets with valid measurements for both the evaluated model and the reference model. Both t ref,d and t m,d are averaged across valid results in three runs for each model-dataset combination.

The average rank (Avg. Rank) measures the relative performance of each model across all datasets. For each dataset d , models are ranked in descending order of performance, with the best-performing model receiving rank 1. Invalid results are assigned the worst rank. The average rank R m for model m is computed as:

where r m,d is the rank of model m on dataset d , and |D| is the number of datasets. When multiple models have the same performance score on a dataset, the average mode is used, where tied models receive the same rank calculated as the average of the ranks they would have received had they not been tied.

The success rate (Success) indicates the percentage of dataset-run combinations where a model successfully completes the task. For model m , the success rate S m is defined as:

where |D m | is the number of dataset-run combinations with valid results for model m , and |D| is the total number of datasets in the evaluation. Each dataset is evaluated across 3 independent runs, resulting in |D| × 3 total evaluation instances.

A.2 Details of MLE-bench Lite

A.2.1 Implementation Details

For the MLE-bench Lite [10] evaluation, we modified our hyperparameter configuration from those used in the Multimodal AutoML Agent Benchmark. Specifically, we configured the agent to use best quality preset parameters when available [19, 65, 67] with a 4-hour time limit per iteration to thoroughly evaluate performance. We established a 24-hour overall limit for each dataset. All experiments were conducted on an AWS EC2 p4d.24xlarge instance equipped with 8 NVIDIA A100 (40GB) GPUs and 96 vCPUs.

Since training requires substantial computational resources and time, we utilize the performance metrics reported in the original MLE-Bench publication [10]. Although the specific LLM model is not explicitly stated in these reports, we select the first reported results for each agent to maintain consistency. We excluded the ranzcr-clip-catheter-line-classification dataset due to preprocessing inconsistencies, ensuring fair comparison across all systems.

Many competitions in the benchmark closed several years ago, making the leaderboard rankings potentially outdated compared to current human capabilities. Therefore, beyond medal counts, we compare the relative performance ranking of each agent across all datasets and compute average rankings. Our evaluation framework incorporates six comprehensive metrics: (1) number of gold medals, (2) number of gold and silver medals combined, (3) number of gold, silver, and bronze medals combined, (4) number of performances above median threshold, (5) success rate (proportion of non-trivial submissions with non-zero performance), and (6) average rank across all datasets.

A.2.2 Performance Thresholds for MLE-bench Datasets

Table 6 presents the performance thresholds established for each dataset's medal and above-median categorization.

A.2.3 Limitations of MLE-bench Evaluation

MLE-bench [10] provides valuable comparisons between ML agents and human performance on Kaggle competitions, but has notable limitations. The benchmark uses preprocessed data through dedicated Python scripts, presenting structured rather than raw inputs, which may overestimate agent capabilities. This contrasts with our Multimodal AutoML Agent Benchmark that specifically tests end-to-end processing of raw data. Additionally, MLE-bench's agent-human comparisons face methodological issues: different data splits, evaluation conditions, and human leaderboards that often reflect earlier computational environments with limited hardware and algorithms. These factors introduce systematic biases that complicate performance comparisons and may not accurately represent contemporary ML automation capabilities among human competitors. While MLE-bench demonstrates our system's effectiveness within established workflows, our benchmark evaluates the end-to-end capabilities that distinguish MLZero from other ML agent approaches.

A.3 Details of Datasets used in Section 4.3 for Ablation Studies

To evaluate the contribution of individual components to our system's overall performance, we conducted comprehensive ablation studies across a diverse subset of datasets. We selected eight representative datasets spanning various machine learning tasks: yolanda [61], mldoc [64], flood [34], petfinder [2], camoseg [42], rvlcdip [27], solar(10m) [65], and fiqa [68]. We select this subset to include a broad spectrum of domains including tabular data analysis, multimodal data analysis, multilingual and multi-table tasks, document classification, image segmentation, time series forecasting, and text retrieval.

B Agents in MLZero

B.1 Perception

B.1.1 File Grouping and File Perception

Our file grouping mechanism (Algorithm 1) employs a hierarchical approach that analyzes folder structures and file extensions to identify meaningful patterns within raw datasets. We first define max group size δ ( δ = 5 by default in all our experiments). For each directory level, the algorithm dynamically determines whether to preserve specific folder names (when ≤ δ unique names exist at that level) or abstract them using wildcards (when > δ names exist), creating a balanced representation that captures essential structural information while avoiding over-specification.

Upon establishing these file groups, our perception system selects representative examples for detailed analysis. For small groups ≤ δ , all members undergo comprehensive content inspection, while larger collections are represented by carefully selected exemplars. Each file is then processed through File Perception Agent, which dynamically generates format-appropriate loading and printing code based on file characteristics, thereby providing structured insights into raw data content regardless of format or organization. This two-phase perception approach forms the foundation of our system's ability to understand arbitrary data structures without manual preprocessing.

Algorithm 1 File Grouping

File Perception Agent

Context: The agent utilizes LLM capabilities to analyze file content across various formats, enabling raw data understanding in AutoML workflows. This addresses a critical limitation in previous systems that required pre-structured data.

Prompt:

f"""Generate Python code to read and analyze the file: "{file_path}"

B.1.2 Task Perception

The task perception in MLZero occurs in two main phases. First, the system uses an LLM to identify potential description files from the data prompt, looking specifically for files like READMEs, documentation, or task definitions. The LLM's response is parsed to extract filenames and returns both the list of files and the analysis explanation. Second, the system generates a comprehensive task description by reading the content of the identified files and using another LLM call to extract key information. It processes the file contents along with the original data prompt and previous analysis to produce a structured description that includes the core data science objective, requirements, constraints, data sources, and success metrics. These steps create a foundation for understanding what the task requires before proceeding to ML library selection.

{description_context}

Based ONLY on the information explicitly stated in the provided data prompt, description files, and analysis, provide a condensed description of the data science task. Include only details that are directly mentioned in the source materials. Do not add assumptions or infer unstated information. """

B.1.3 MLLibrary Selection

The ML library selection process follows the task perception steps and uses an LLM to identify the most suitable ML library for the data science task at hand. It takes the data prompt, task description, and available libraries from the registry as inputs. The prompt includes the task details and formatted information about each available library (name, version, description, and special features). The LLM's response includes both the selected library name and the reasoning behind the selection.

MLLibrary Selection Agent

Prompt to Select ML Libraries

B.2 Semantic Memory

B.2.1 Condensation

The condensation agent processes tutorial content by breaking it into manageable chunks when necessary, then uses an LLM to condense each chunk while preserving essential implementation details, code samples, key concepts, critical configurations, and important warnings. It works on a chunk-by-chunk basis to handle large tutorials, maintains the overall document structure, and ensures the resulting condensed content stays within a specified maximum length by truncating at section boundaries when needed.

B.2.2 Summarization

The summarization agent generates concise summaries of the documents that help coder agent understand the implementation knowledge, coding tasks, and key features covered in the tutorial. It processes the already condensed content through an LLM to create a single paragraph summary of less than 100 words that starts with "Summary: " and highlights the most important aspects of the tutorial for practical implementation.

Summarization Agent

Prompt to Summarize the Document

f"""Generate a concise summary (within 100 words) of this tutorial that helps a code generation LLM understand:

What specific implementation knowledge or techniques it can find in this tutorial

What coding tasks this tutorial can help with

Key features or functionalities covered

B.2.3 Retrieval

The retrieval agent scans documents of the selected ML libraries, extracts their titles and summaries, and uses an LLM to select the most relevant documents based on the user's task, data description, question, and any previous errors. The selected documents are then provided to coder agent to correctly integrate the selected ML library.

Retrieval Agent

B.3 Episodic Memory

B.3.1 Error Analyzer

The error analyzer processes error messages from failed code executions. It takes the original task description, data, user prompt, previous code, and error message as input, and uses an LLM to generate a concise error summary to identify root causes and suggested debugging steps to provide tactical guidance. The agent is asked not to include actual code fixes, but focusing on clear, actionable insights within strict length constraints.

Retrieval Agent

Prompt to Analyze the Error

"""{task_prompt}

{data_prompt}

{user_prompt}

Previous Python

Code:

{python_code}

Previous Bash

Script

to

Execute

the

Python

Code:

{bash_script}

{retrieved_tutorials}

Error Message:

{error_message}

Analyze the

error

message

and

context

provided.

Your

response

MUST contain

exactly

two

short

paragraphs

as

follows:

ERROR SUMMARY: Provide a brief, technical description of the error in 1-3 sentences. Focus only on identifying the root cause and affected component without background explanations.

SUGGESTED FIX: Offer specific debugging directions in 1-3 sentences. Do not include actual code or commands, only tactical debugging guidance.

Each paragraph must be concise (maximum 3 sentences). Do not include general advice, explanations beyond the direct debugging strategy, or any additional paragraphs."""

B.4 Iterative Coding

B.4.1 Coder

The Coder agent creates programming solutions based on user prompts. It supports both single-turn and multi-turn interactions. In our experiments, we use single-turn interaction by default as episodic memory can provide information about past errors. In the ablation study of removing episodic memory, we use multi-turn interactions where coders in all iterations run in a common session. The Code agent also ensures code is properly formatted within language-specific code blocks (python or bash in our experiments), extracting clean scripts from LLM responses.

For Python code generation, the prompt includes system prompt, library-specific instructions, task perception, data perception, optional user input, error analysis from episodic memory, and retrieved documents from semantic memory.

Similarly, the bash script generation prompt assembles contextual components including environment configuration preferences (tailored to whether environment setup is required), code execution instructions, the generated Python code to be run, and (when applicable) previous bash scripts and error messages.

Coder Agent

Prompt to Generate Python Code f"""

As an AutoML Agent, you will be given a folder containing data and description files. Please generate Python code using {tool_name} to train a predictor and make predictions on test data. Follow these specifications:

ONLY save files to the working directory: {output_folder}.

Data preprocessing:

-Remove training data samples without valid labels (unless told not to do so).

-Remove the unneccesary index column (if applicable)

Model training:

-Use {tool_name} with appropriate parameters for the task

-If a model is trained, save it in a folder with random timestamp within {output_folder}

Prediction:

-Make predictions on the test data

-Save the predicted results to {output_folder}, result file name should be "results", the format and extension should be same as the test data file

-Output column names must exactly match those in the training or sample submission files without adding "predicted_" prefixes or creating any new columns.

Documentation:

-Add a brief docstring at the beginning of the script explaining its purpose and usage

B.4.2 Executer

The Executer agent executes generated code and captures real-time outputs with timeout protection. Success determination is two-fold: first checking if the script execution completes with a zero return code, then passing execution logs to LLM that evaluates deeper success criteria beyond just exit codes. The Executer agent makes critical "FINISH" or "FIX" decisions with explainations based on log analysis, even overriding apparent successes when it detects logical errors or poor performance in the outputs, ensuring both technical completion and task fulfillment before concluding the execution cycle.

have issues with logic or not meet all requirements."""

C More Implementation and Competitors Details

C.1 Implementation Details

For MLZero, the code execution environment operates with a 3-hour timeout and a maximum of 5 coding iterations. We enforce strict prompt size constraints: 1,024 characters for files, 8,192 characters for tutorials, and 2,048 characters for user inputs. We leverage Claude 3.7 Sonnet [6] as our LLMbackbone, using temperature 0 for planning and file reading tasks, while employing temperature 0.5 for coding. All agents utilize a 65536-token context window with task-specific configurations. For the 8B configuration of MLZero, we adjusted parameters to accommodate the smaller context window. Specifically, we set the context window to 8,192 tokens across all agents, reduced retrieval size to 3, and limited maximum tutorial length to 4,069 characters.

To ensure fair comparison with AIDE's default setting (def), we standardized its configuration by disabling automatic file copying and utilizing identical LLM backbones [6] for code generation and feedback processes. We maintained a 3-hour timeout and 5-step reasoning process to match our method's 5 coding iterations. Additionally, we also conducted evaluations on an enhanced version (AIDE +ext) that incorporates six comprehensive tutorials covering fundamental knowledge for all tasks to facilitate coding with external ML libraries. In the enhanced configuration, we also provide the system prompt of MLZero to optimize AIDE's performance.

Codex CLI [57] performs end-to-end with minimal human intervention. We similarly use a 3-hour timeout but no step limit . For other comparative systems that typically require manual data preprocessing [33, 24, 45], we enhanced their capabilities by augmenting them with outputs from our data perception module, providing additional data insights that enable end-to-end execution on our benchmark. Since DS-Agent [24] lacks native code execution support, we manually executed their generated code to obtain results; thus relative execution times are unavailable as they would not represent true end-to-end agent performance. As both AIDE [38] and AutoKaggle [45] (AK) lack robust mechanisms for saving results through user instructions, we manually extracted results from their working directories for evaluation. We also standardized all description file names to accommodate baseline agents. For agents (Codex CLI, DS-Agent, AutoKaggle) that lack compatibility with Claude 3.7 Sonnet [6] due to implementation constraints or outdated interfaces, we substitute with GPT-4o [3], a general-purpose model with comparable benchmark performance.

As MLE-bench offers a longer time (24 hours) for each agent on each dataset, we prompt the ML libraries [19, 65, 67] to use best quality preset with a 3-hours time limit per iteration for each dataset. Other implementations in MLZero remain the same. For the implementation of AIDE, MLAgentBench [33], and OpenHands [72], please refer to MLE-bench [10].

All experiments were conducted on an AWS EC2 p4d.24xlarge instance equipped with 8 NVIDIA A100 40G GPUs. Computation time varied based on iterations required to achieve success. For the Multimodal AutoML Agent Benchmark, the maximum computational cost for running one agent is 3 hours × 25 datasets × 8 GPUs, totaling 600 GPU hours. The total computational upper bound for Table 1 is 600 GPU hours × 11 agents × 3 runs, totaling 19,800 GPU hours. For MLE-bench, the maximum computational cost for running MLZero once is 24 hours × 21 datasets × 8 GPUs, totaling 4,032 GPU hours.

The operational cost of using Claude 3.7 Sonnet varies with the number of iterations required for success. Approximately, MLZero incurs a cost of $0.25 per dataset, while AIDE averages $0.5 per dataset. However, these figures may fluctuate significantly depending on the iterations needed to achieve successful completion.

C.2 MLZero

The default configuration of MLZero is attached below. Please check Appendix B for the detailed prompts and implementation of each agent in MLZero, and Appendix D for the ML libraries used.

C.3 Codex CLI

OpenAI's Codex CLI represents a practical implementation of LLM-based tool-using agents for software development. When equipped with strong reasoning models, it demonstrates effective instruction-following capabilities and can reliably execute software engineering tasks. However, it tends to generate simplistic solutions that underperform on complex machine learning workflows. Moreover, its performance degrades substantially when deployed with general-purpose language models that lack robust reasoning capabilities, underscoring the limitations of current tool-using agents in efficiently handling the sophisticated requirements of end-to-end machine learning automation.

In our experiments (Section 4.2), we evaluate Codex CLI in two configurations. The default configuration (Codex CLI) uses OpenAI's GPT-4.1 [56], a general-purpose model, to ensure fair comparison with other agents that employ non-reasoning models. We also include an enhanced configuration (Codex CLI +rea) utilizing o4-mini [58], a reasoning model, to demonstrate Codex CLI's upper performance bound. Both configurations employ the following standardized prompt:

Prompt for Codex CLI

Solve the ml task described in folder ${TRAINING_PATH}. Do not modify any files in ${TRAINING_PATH}. All temp or saved files should be located somewhere under ${OUTPUT_DIR}. Save the predicted results in the same format as training data to ${OUTPUT_DIR}. Name the result file 'results.xxx', where the extension should be same as the test data file.

C.4 AIDE

AIDE [38] conceptualizes machine learning engineering as a code optimization problem, employing tree search methodologies within the solution space. Similar approaches are found in SELA [14] and I-MCTS [46], though these frameworks focus primarily on tabular data and do not provide end-to-end solutions.

For MLE-bench [10] comparisons, we utilize the officially reported AIDE results. However, for our Multimodal AutoML Agent Benchmark, we implemented several configuration modifications to ensure fair and efficient comparison:

We disabled automatic data copying functionality, as several datasets in our benchmark are extremely large, making this process prohibitively time-consuming.

Archive unpacking functionality was disabled to preserve the original data structure integrity.

Report generation was disabled as it increases token and time usage without affecting performance metrics.

Both code generation and feedback mechanisms were changed to use Claude 3.7 Sonnet, consistent with our system's configuration.

To facilitate proper evaluation within our benchmark framework, we implemented additional hardcoded configurations: (1) Description files are automatically fed into AIDE, as the system cannot independently locate these files; (2) Output files are renamed from AIDE's default 'submission.csv' to each dataset's required filename, as AIDE uses hardcoded output naming conventions that would otherwise impede proper evaluation.

For the default AIDE configuration (AIDE def), we incorporated the following additional prompt to ensure output format consistency and proper file referencing:

Additional Prompt for AIDE (def)

Output column names must exactly match those in the training or sample submission files without adding "predicted_" prefixes or creating any new columns.

Always use absolute file paths when your model prediction outputs reference files.

For the enhanced AIDE configuration (AIDE +ext), we further augmented the system with our MLZero coder's prompt, ML library prompts (detailed in Appendix B and Appendix D), and several critical condensed tutorials from MLZero's knowledge base. This enhanced configuration represents our effort to provide AIDE with comparable knowledge resources for a more equitable performance comparison:

Additional Prompt for AIDE (+ext)

[MLZero coder prompt] [MLZero ML library prompts] [Condensed Tutorials]

C.5 DS-Agent

DS-Agent [24] leverages LLMs with case-based reasoning (CBR) to solve machine learning problems, particularly excelling in comprehending task requirements and constructing machine learning pipelines. Unlike ResearchAgent [33] focused primarily on generating reasonable plans, DS-Agent implements a two-stage approach:

A comprehensive development stage utilizing the full CBR framework to capitalize on expert knowledge from Kaggle competitions and iteratively improve performance through feedback mechanisms.

A resource-efficient deployment stage with streamlined CBR to adapt previously successful solutions for direct code generation.

For benchmarking DS-Agent, we utilized the official records of the development stage. As DS-Agent lacks an end-to-end execution system, we modified the Multimodal AutoML Agent Benchmark to align with the data requirements [33], then incorporated MLZero's perception results for each dataset as commented paragraphs as the provided skeleton code.

For both default (def) and zero-shot evaluation settings, we followed the original implementation without modifications. And we copied and executed the generated code within the Multimodal AutoML Agent Benchmark environment. To ensure consistent evaluation criteria across all systems, we also supplemented the description files with additional standardized prompts to enforce consistent output formatting and behavior.

Additional Prompt for DS-Agent

ONLY save

files

to:

"./".

Make predictions on the test data.

Save the predicted results to "./", result file name should be "results", the format and extension should be same as the test data file.

Output column names must exactly match those in the training or sample submission files without adding "predicted_" prefixes or creating any new columns.

Tensorflow is not installed. But you can use pytorch when needed.

C.6 AutoKaggle

AutoKaggle [45] presents a collaborative multi-agent framework for automated machine learning solutions, comprising five specialized agents (Reader, Planner, Developer, Reviewer, and Summarizer) working in concert throughout the solution development lifecycle. The framework implements an iterative development methodology with comprehensive testing procedures and leverages a predefined machine learning tools library. However, a significant limitation is its rigid input data format requirements, as it exclusively supports evaluation of tabular datasets from Kaggle competitions within a predefined directory structure. For AutoKaggle to process data correctly, the input files must adhere to the following structural convention:

Required Data Format for AutoKaggle

competition/

-

train.csv

-

test.csv

-

sample_submission.csv

-

overview.txt

To accommodate these constraints while evaluating AutoKaggle within our benchmark framework, we implemented several adaptations to preprocess data: renaming descriptions.txt files to overview.txt ; augmenting training and test datasets with sequential index columns; converting diverse data formats to the mandatory CSV format with standardized train.csv and test.csv

filenames; and generating appropriate sample_submission.csv files that conform to the expected index and target column structure. These adaptations ensure fair evaluation while maintaining the integrity of the original datasets within our comparative benchmark.

D MLLibraries in MLZero

Our system aims to minimize required human effort by leveraging ML frameworks with comprehensive capabilities rather than specialized libraries. Each integrated framework may contain undocumented issues (mentioned in GitHub issues rather than official documentation), for which we provide supplementary prompts to enhance LLM performance. These prompts were also provided to AIDE (+ext) to ensure fair comparison. The MLZero (-ext) configuration include only general descriptions of machine learning algorithms D.5.

MLZero incorporates only a few well-maintained ML frameworks that demonstrate state-of-the-art performance across various modalities, keeping the ML libraries used minimal to avoid "overfitting" the tasks. Our selection criteria prioritize frameworks that provide comprehensive automation capabilities, robust implementation quality, and consistent maintenance to ensure reliable performance. This approach enables our system to handle a wide spectrum of machine learning tasks with minimal human intervention while maintaining competitive performance. Adding more ML libraries with better performance could further improve the agent's performance, but we intentionally limited our selection to prevent overfitting to our evaluation benchmarks.

D.1 Tabular

For tabular data processing, we integrate frameworks with strong performance on structured data tasks [19, 40]. The time_limit and presets parameters are specified for benchmarking purposes but can be adjusted through user input in practical applications.

D.2 Multimodal

For multimodal tasks, we selected AutoMM [67] due to its performance in multimodal AutoML tasks and comprehensive support across diverse problem types. This framework excels at integrating multiple data modalities into unified representations, providing good performance with minimal configuration requirements.

The selection of AutoMM aligns with our approach of leveraging well-maintained frameworks with comprehensive capabilities. Its performance across diverse problem types (image-text classification, document understanding, semantic segmentation, etc.) enables MLZero to handle a wide spectrum of multimodal tasks without requiring specialized implementations for each modality combination. In future work, more advanced multimodal ML libraries should be included to support additional problem types and to further improve performance.

D.3 Timeseries

For timeseries forecasting, we incorporate the ML library with state-of-the-art performance [65]. Note that [19, 67, 65] share a similar API but internal logics are completely different and thus used as three libraries.

D.4 Retrieval

Our system incorporates FlagEmbedding [74, 80] to support tasks requiring semantic search, document ranking, and retrieval-augmented generation capabilities.

D.5 Others

When specialized frameworks are not suitable, e.g. image-to-image generation, audio tasks, sequenceto-sequence generation, etc., MLZero can fall back to general machine learning algorithms, providing greater flexibility for diverse tasks.

D.6 Library Integration Process

One of the key design principles of MLZero is the ease of integrating new ML libraries into the system's knowledge base. Adding a new library requires approximately one minute of human effort using our provided registration Python script. Users provide only four basic pieces of information: the library name (e.g., "FlagEmbedding"), version number (e.g., "FlagEmbedding==0.x.x"), a onesentence description, and the path to the directory containing the library's official documentation. Users may optionally provide additional prompts or usage guidelines, though this is not required. Importantly, no coding skills are needed for this registration process.

After this brief human setup, the system operates completely autonomously. The Summarization and Condensation agents automatically process the provided documentation into structured, condensed formats suitable for the Semantic Memory module. This automated processing includes extracting relevant sections from official documentation, using LLM agents to condense lengthy tutorials into essential information, organizing information by topics and APIs, and building retrieval indices for efficient semantic search. While this processing requires computational time (typically 10-30 minutes depending on documentation size), it requires zero human intervention or supervision.

Once a library is successfully integrated into Semantic Memory, MLZero operates from raw data to final predictions without further human input. This stands in stark contrast to many existing AutoML systems that require manual preprocessing, hard-coded logic tailored to individual datasets, expert configuration of hyperparameters, or post-processing to extract results. Our experiments demonstrate that this design achieves true end-to-end automation while maintaining extensibility across diverse modalities and problem types.

E Detailed Results

E.1 Detailed Results on Multimodal AutoML Agent Benchmark

This appendix provides a detailed breakdown of all experimental runs conducted with different agents and configurations on the Multimodal AutoML Agent Benchmark in Section 4.2. Each table presents the complete results across three independent runs for each dataset in the benchmark suite: MLZero def in Table 7, MLZero 8B in Table 8, MLZero -ext in Table 9, MLZero -epi in Table 10, Codex CLI def in Table 11, Codex CLI +rea in Table 12, AIDE def in Table 13, AIDE +ext in Table 14, DS-Agent def in Table 15, DS-Agent zeroshot in Table 16, and AutoKaggle def in Table 17.

For each run, we report the performance metric (see Appendix A.1) and the computational time in seconds required to complete the task. The symbol × indicates runs where the model failed to complete or with invalid outputs. These detailed results complement the aggregated performance metrics discussed in the main paper and demonstrate the consistency and robustness of our approach across multiple executions.

E.2 Detailed Results on MLE-bench Lite

This appendix presents the comprehensive experimental results that supplement the analysis on MLE-bench Lite provided in the Section 4.2 of main paper. Table 18 offers a detailed comparison of our method against three state-of-the-art baseline approaches across all 21 datasets in the MLE-Bench Lite benchmark. Note that for metrics where lower values originally indicated better performance, we applied a negative sign to convert them, ensuring that higher values consistently represent better performance throughout the table. See Appendix A.2 for details about MLE-bench Lite. Gold, silver, and bronze highlighting denote the type of medal gain for each dataset, while underlined values indicate performance above the median. The symbol 'X' represents cases where methods failed to produce valid solutions within the allocated time constraints or encountered critical errors. As demonstrated in both subtables, our approach achieves superior performance on multiple datasets across various machine learning tasks, confirming the findings summarized in the main paper. These detailed results provide additional evidence of our method's robustness, versatility, and consistent performance advantages over existing techniques in automated machine learning systems.

E.3 Detailed Error Analysis

To better understand the limitations of current AutoML systems, we conducted a comprehensive error analysis across MLZero, Codex CLI, AIDE, and DS-Agent (DS). Table 3 presents the frequency of distinct error types observed in the final iterations of each system's execution.

MLZero exhibits exceptional robustness, encountering minimal errors (6.5% overall) limited to algorithm implementation (4.3%) and preprocessing stages (2.1%). The algorithm implementation errors occurred specifically on sequence-to-sequence and audio classification tasks where the system had insufficient knowledge of relevant ML libraries. The preprocessing errors manifested exclusively on image-to-image tasks with heterogeneous input resolutions. In contrast, competing methods demonstrated substantially higher failure rates: DS-Agent (76.0%), AIDE (47.8%), and Codex CLI with reasoning (26.9%).

(a) Performance on Datasets from D1 to D11.

(b) Performance on Datasets from D12 to D22.

DS-Agent encounters difficulties at foundational stages, including perception (24.0%) and preprocessing (20.0%), indicating systemic failure in data comprehension and requirement translation, despite being provided with data perception context from MLZero. Its high algorithm implementation error rate (28.0%) likely stems from the substantial domain gap between deployment and development datasets.

AIDE's predominant failure mode is API hallucination (28.2%), wherein the system attempts to utilize non-existent or incorrectly implemented library functions. This underscores the necessity for external knowledge of ML libraries rather than exclusive reliance on parametric knowledge embedded in LLMs.

Notably, MLZero completely eliminates several error categories, including perception (0.0%), API hallucination (0.0%), and postprocessing (0.0%). This achievement can be attributed to our dualmemory architecture, which effectively anchors the LLM's generations in factually accurate external knowledge while maintaining execution consistency through structured episodic memory. The preprocessing errors observed in MLZero result from the system utilizing example file metadata to construct solutions for datasets with variable input resolutions across images, representing a conscious trade-off between context length compression and preservation of critical information in file grouping and perception.

It is important to note that this analysis focuses on final iteration errors and does not necessarily indicate error propensities at intermediate stages. For instance, AIDE's higher apparent API hallucination errors compared to DS-Agent may reflect AIDE's attempts at implementing more sophisticated solutions after successfully navigating earlier stages that DS-Agent fails to complete. Additionally, AIDE exhibits unreported errors (8.7%) in the MLE-Bench evaluation, potentially influencing comparative assessments.

These findings highlight the significant advantages of our integrated approach, which systematically addresses error-prone areas of the ML workflow through specialized perception agents, structured knowledge retrieval, and iterative refinement with targeted error correction mechanisms.

E.4 Continual Improvement Beyond First Success

While the default MLZero configuration employs an early stopping strategy (terminating upon first successful execution), we investigated the system's capability for iterative refinement beyond initial success. We evaluated two configurations across eight challenging datasets covering all modalities: the default configuration with early stopping upon first success (def), and an extended

configuration allowing up to 5 iterations regardless of initial success (def + nostop). We also evaluated both configurations in the challenging setting without specialized external libraries (-ext) to assess improvement potential when the first solution is typically suboptimal.

Table 19 presents detailed results showing that the extended configuration demonstrates clear capability to refine successful solutions. Notable improvements include camoseg accuracy improving from 0.84 to 0.86 (+2.4%), flood from 0.69 to 0.74 (+7.2%), and solar RMSE dramatically reducing from 1.49 to 0.38 (-74.5%). The improvement pattern is more pronounced in the -ext setting where initial solutions are weaker due to lack of specialized libraries, with camoseg improving from 0.46 to 0.55 (+19.6%) and petfinder from 0.38 to 0.44 (+15.8%). These results validate that optimizing an existing working solution is generally easier than achieving the first success.

Despite performance gains, continual improvement incurs substantial computational overhead. Token usage increases 3 × from 63k to 186k tokens per dataset in the default setting, computation time nearly triples from 1904s to 5388s per dataset, and costs increase proportionally due to additional LLM calls. This overhead occurs because datasets that succeed in the first iteration still complete all 5 iterations rather than stopping early, continuing to explore alternative implementations even when initial performance is already strong.

Given these findings, we chose the early stopping strategy for the main paper to prioritize efficiency while maintaining high performance (92% success rate). The modest improvements from continual iteration (typically 2-7% gain) do not justify tripling computational costs for most practical applications. However, for scenarios where computational resources are not a constraint and optimal performance is paramount, the continual improvement capability remains valuable. As demonstrated in Table 1, extending the time limit to 24 hours with the best_quality preset further improves performance, approaching expert-level human results on several datasets.

E.5 Examples of Generated Code

E.5.1 Code generated by MLZero def on the Hateful Meme Dataset

Evaluation Score (f1): 0.5868

Time Used: 663 seconds

E.5.2 Code generated by Codex +rea on the Hateful Meme Dataset

Evaluation Score (f1): 0.4538

Time Used: 110 seconds

E.5.3 Code generated by AIDE def on the Hateful Meme Dataset

Evaluation Score (f1): 0.5573

F More Related Work: AutoML with LLMs

Recent advances have demonstrated the potential of integrating LLMs into AutoML systems across various components of the ML workflow. For feature engineering, researchers [29, 82] have leveraged LLMs to generate meaningful features with explanations and efficiently evaluate candidate features. In hyperparameter optimization, LLM-based approaches [79, 47] generate configurations through dataset analysis and iterative refinement processes. For neural architecture search, researchers have combined LLMs with Quality-Diversity algorithms to generate diverse network architectures [55] and constructed performance predictors for estimating neural network efficiency [35]. LLMs have also shown effectiveness for data preprocessing tasks such as error detection and imputation [78]. While these approaches successfully apply LLMs to specific components, MLZero differs by providing an end-to-end multi-agent solution that fully automates the entire ML workflow.

G Limitations and Future Work

Despite MLZero's strong performance across diverse machine learning tasks, several limitations warrant consideration. First, while our system can leverage both documentation and code examples, its effectiveness may be limited when working with ML libraries that have minimal description in either text or code format. Second, machine learning libraries contain unavoidable undocumented bugs that are difficult to address through in-context learning, and LLMs cannot anticipate these issues unless explicitly prompted. Additionally, our experiments reveal that smaller LLMs (8B parameters) still demonstrate a significant performance gap compared to larger models despite showing competitive results.

Future research directions should address these challenges through several avenues. First, enhancing performance with smaller language models through improved alignment with external knowledge and curated code examples represents a promising direction, especially considering our 8B parameter model already demonstrates reasonable performance. Second, developing a finetuning process that allows LLMs to actively experiment with and learn ML libraries during training could help address both undocumented bugs and improve smaller model performance, while also expanding MLZero's applicability to emerging libraries with less structured documentation. This approach would enable models to acquire practical knowledge about the ML library behavior beyond what is explicitly documented. Finally, conducting longitudinal studies on real-world ML workflows would provide insights for better supporting human-AI collaboration in data science processes, particularly for iterative development scenarios where expert guidance can further improve automated approaches.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MLZero, a hierarchical multi-agent system for fully automated machine learning (AutoML) across multimodal data. The framework integrates perception agents, semantic memory (knowledge retrieval), episodic memory (error-driven refinement), and iterative code generation based on LLMs, aiming for zero human intervention. MLZero is benchmarked on both MLE-Bench Lite and a new Multimodal AutoML Agent Benchmark introduced by the authors, showing competitive results in success rate and performance against various baselines. Extensive ablation and error analyses are also provided to justify architectural design choices.\n\nSTRENGTHS AND WEAKNESSES:\n#### **Strengths**\n\n* **Well-structured System Design**: The paper presents a clear and modular architecture, combining LLM planning and execution with memory augmentation and agent specialization.\n* **Multimodal Generality**: MLZero operates on diverse tasks spanning tabular, text, image, and document classification, which showcases practical versatility.\n* **Comprehensive Evaluation**: Experiments span two benchmarks, including a new 25-task suite with raw data processing requirements.\n* **Ablation and Failure Analysis**: The inclusion of ablation studies and error breakdown by category (e.g., hallucinations, preprocessing) is commendable and unusually thorough.\n* **Competitive Results**: MLZero outperforms existing baselines in success rate and average rank, even when stripped of certain components like external knowledge or episodic memory.\n\n#### **Weaknesses**\n\n1. **Excessive Engineering Emphasis, Weak Novelty in Learning**: While the architecture is elaborate, most techniques (e.g., retrieval, summarization, error tracing) are standard LLM engineering practices. There is limited novelty from a learning or algorithmic standpoint.\n2. **Limited Insight into LLM Limitations**: The reliance on LLMs is central, but the paper does not examine when/how LLMs fail during ML pipeline construction, or what qualitative reasoning errors occur.\n3. **Benchmark Bias**: The newly introduced benchmark is entirely authored by the same authors. While it addresses limitations of MLE-Bench, the fairness and diversity of its design should be externally validated.\n4. **Lack of Human Evaluation or Expert Annotation**: The paper focuses solely on automatic metrics, without any human judgment of code quality, correctness, or failure modes—this makes the empirical superiority somewhat opaque.\n5. **Minimal Theoretical or Formal Analysis**: There is no formal grounding in agent planning, LLM reliability, or bounded correctness. A system this complex could benefit from guarantees or at least probabilistic behavior analysis.\n6. **Missing Comparison with Planning-based LLM Approaches**: Other recent planning-agent frameworks (e.g., ReAct, Reflexion, CAMEL) are not directly compared or discussed in depth. This weakens claims of generality and superiority.\n\nQUESTIONS:\n1. **Benchmark Diversity**: How were the 25 datasets in your new benchmark chosen? Is there any risk that the benchmark favors MLZero due to design alignment?\n2. **Failure Analysis Detail**: For the minimal 6.5% error rate in MLZero, what are the qualitative patterns of failure? Are these reproducible errors or stochastic?\n3. **Knowledge Contamination Risk**: Given that some agents access condensed tutorials and summaries, how do you ensure no contamination from evaluation sets?\n4. **Memory Scalability**: How does episodic memory scale with longer runs or larger datasets? Is there a cost trade-off in performance vs. token consumption that might make it infeasible in practice?\n5. **Security and Robustness**: Does MLZero ever produce unsafe or incorrect solutions that *look* correct (e.g., hallucinated accuracy but failed logic)? Any defenses?\n\n*A more favorable evaluation could be reached if the authors:*\n\n* Provide clearer guarantees or at least empirical error bounds for LLM planning and execution cycles.\n* Include human validation of at least a subset of generated pipelines.\n* Address benchmark neutrality through external evaluation or open access submission.\n\nLIMITATIONS:\nThe authors acknowledge runtime cost and large-model reliance, but do not adequately address:\n\n* **Generalization to new ML libraries**: It is unclear how easily MLZero adapts to unseen or rapidly changing APIs.\n* **Dependence on LLM correctness**: The system heavily assumes the base LLM will not hallucinate or misinterpret errors in a critical way.\n* **Memory curation and freshness**: Episodic and semantic memories appear stateless across datasets—how would they behave in a real, long-term deployed setting?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-04"}

comments: SUMMARY:
The paper proposes a multi-agent system titled “MLZero” which achieves impressive results on MLE Bench and their own Multimodal AutoML benchmark while being fully automated. The paper discussed their system setup and agent definitions along with the benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is very detailed and well written.
2. They achieve impressive results on the MLE Bench and Multimodal AutoML benchmarks.
3. The paper contains extensive ablations and detailed task-level results on the various ML benchmarks.

Weaknesses:
1. While the results are impressive, the system itself is simple and doesn't represent a significant contribution to the field. The crux of the paper lies in the setup of the system and the way the different agents interact with each other, and while that is discussed in fair detail, the system isn’t complex or novel enough to meaningfully move the field forward.

QUESTIONS:
1. There is no mention of which LLM is used in the “def” setting of MLZero. Could this be added to the paper? The config present in the appendix section seems to indicate this is Claude 3.7 Sonnet, but it would be great if the authors could confirm this.
2. The agent seems to “finish execution” once it successfully executes the code (or reaches the context limit). It would’ve been great to see the agent continually create better solutions and maintain a track of the best solution. Ablations to see the improvements from this setup versus the current would’ve provided valuable insights into the self improvement capabilities of LLMs (something like AlphaEvolve). I’m not sure if these ablations were done. Could you add details about those if they were done?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-24

comments: SUMMARY:
The paper introduces MLZero, a novel multi-agent system that achieves fully automated machine learning (ML) across diverse data modalities with zero human intervention. MLZero integrates Large Language Models (LLMs) with a perception module and dual memory systems—semantic and episodic—to intelligently process raw multimodal data, iteratively generate and refine ML code, and deliver ready-to-use solutions. The system coordinates nine specialized agents to perform data understanding, library selection, code generation, execution, and error correction. MLZero demonstrates superior performance on both the MLE-Bench Lite and a newly proposed Multimodal AutoML Agent Benchmark, significantly outperforming existing approaches in success rate, efficiency, and robustness. Through ablation studies, the paper shows the crucial roles played by memory components in reducing hallucinations and improving code quality. This work presents a promising step toward general-purpose, autonomous ML agents.

STRENGTHS AND WEAKNESSES:
Advantages:
1. Fully Automated Workflow: Handles the entire ML pipeline from raw data to results without human input.
2. Memory-Augmented Reasoning: Uses semantic and episodic memory to reduce hallucinations and improve reliability.
3. Strong Performance: Outperforms baselines with higher success rates and better solution quality.

Weakness:
1. Still lacks several experiments to validate the effectiveness of their method

QUESTIONS:
1. It would be better if authors are able to compare the token costs of your method.

2. Is the system robust to different models?

LIMITATIONS:
See questions and weaknesses
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MLZero, a novel multi-agent system that automates end-to-end machine learning workflows with minimal human intervention, specifically designed to handle raw, multimodal data. The system employs a perception module to interpret diverse inputs and uses dual semantic and episodic memory to improve iterative code generation while mitigating LLM hallucinations. When evaluated on the newly created Multimodal AutoML Agent Benchmark, MLZero achieved a 92% success rate, a 263.6% improvement over competitors. It also demonstrated superior performance on MLE-Bench Lite, securing six gold medals.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tNovel System Architecture: A well-designed multi-agent system featuring perception and dual-memory modules that automates the end-to-end ML workflow using complex, raw multimodal data.\n2.\tRigorous and Extensive Evaluation: The paper introduces a new, challenging benchmark for raw data and provides comprehensive ablation studies and failure analyses to validate the system's effectiveness.\n3.\tState-of-the-Art Results: The system demonstrates impressive performance, achieving a 92% success rate (+263.6% improvement) and showing robust results even when using a much smaller 8B model.\n4.\tClarity and Reproducibility: The work is clearly presented and highly reproducible, supported by a detailed appendix containing agent prompts, configurations, and comprehensive experimental results for verification.\nWeaknesses & Question\n1.\tNovelty of Individual Components: While the system's integration is novel, its core memory modules (Semantic Memory akin to RAG , Episodic Memory to self-correction ) build on existing concepts. The paper should better contextualize this, clarifying the primary novelty lies in the successful application and synergy for the specific, complex domain of end-to-end AutoML rather than the invention of the components themselves.\n2.\tScalability and Human Intervention: Performance relies on an offline, human-supervised process to create \"condensed tutorials\" for the knowledge base. This challenges the system's scalability and its \"zero human intervention\" claim. How much manual effort is required to integrate a new ML library, and how automated is this crucial knowledge preparation process?\n3.\tSystem Rigidity and Error Propagation: The system's linear, multi-agent pipeline creates a risk of cumulative error, while its effectiveness depends on a few high-level libraries. How does MLZero handle tasks outside its libraries' scope? More importantly, can a downstream coding agent detect and trigger a correction for an early-stage error, such as a flawed library selection made by the Perception module?\n\nQUESTIONS:\nplease refer to the weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}