AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 19:24:00

Abstract: Automated Machine Learning (AutoML) offers a promising approach to streamline the training of machine learning models. However, existing AutoML frameworks are often limited to unimodal scenarios and require extensive manual configuration. Recent advancements in Large Language Models (LLMs) have showcased their exceptional abilities in reasoning, interaction, and code generation, presenting an opportunity to develop a more automated and user-friendly framework. To this end, we introduce AutoM3L, an innovative Automated Multimodal Machine Learning framework that leverages LLMs as controllers to automatically construct multimodal training pipelines. AutoM3L comprehends data modalities and selects appropriate models based on user requirements, providing automation and interactivity. By eliminating the need for manual feature engineering and hyperparameter optimization, our framework simplifies user engagement and enables customization through directives, addressing the limitations of previous rule-based AutoML approaches. We evaluate the performance of AutoM3L on six diverse multimodal datasets spanning classification, regression, and retrieval tasks, as well as a comprehensive set of unimodal datasets. The results demonstrate that AutoM3L achieves competitive or superior performance compared to traditional rule-based AutoML methods. Furthermore, a user study highlights the user-friendliness and usability of our framework,
compared to the rule-based AutoML methods. Code is available at:  https://anonymous.4open.science/r/anonymization_code.

Main paper: 
 introduction: 
 Multimodal data is crucial in various machine learning (ML) tasks as it provides the ability to capture more comprehensive feature representations. Real-world data often combines heterogeneous sources, such as integrating table product information with associated images and textual descriptions. Similarly, in the financial sector, user photos, text, transactions, and other data types are frequently consolidated in tabular formats for analysis and management. However, the inherent diversity of these modalities introduces complexities, particularly in selecting optimal machine learning or deep learning model architectures and seamlessly synchronizing features across modalities Consequently, there is often a heavy reliance on manual involvement in the ML pipeline.Automated Machine Learning (AutoML) has emerged as a promising approach to reduce the need for manual intervention in the ML pipeline [6, 10, 16, 30, 31, 33] . However, a significant gap exists for multimodal data, as the majority of AutoML solutions primarily focus on unimodal data. AutoGluon 1 made an initial attempt at multimodal AutoML but suffers from several limitations. Firstly, it lacks comprehensive automation of feature engineering, which is crucial for effectively handling multimodal data. Secondly, it presents a steep learning curve for users to become familiar with its configurations and settings, contradicting the user-friendly automation principles that AutoML aims to embody. Moreover, Auto-Gluon's adaptability is constrained by pre-set settings such as the search space, model selection, and hyperparameters, necessitating significant manual intervention. Lastly, extending AutoGluon's capabilities by integrating new techniques or models often requires complex manual code modifications, hindering its agility and potential for growth.The scientific community has been captivated by the rapid rise of large language models (LLMs), particularly due to their transformative potential in task automation [2, 4, 29, 34] . LLMs have evolved beyond their initial purpose as text generators and have now become highly autonomous entities capable of self-initiated planning and execution [14, 27, 32, 35, 36] . This evolution presents a compelling opportunity to enhance the performance and adaptability of multimodal AutoML systems. Leveraging this potential, we introduce AutoM 3 L, an innovative LLM framework for Automated Multimodal Machine Learning. Unlike platforms such as AutoGluon, which are constrained by predefined pipelines, AutoM 3 L distinguishes itself through its dynamic user interactivity. Specifically, it seamlessly integrates ML pipelines tailored to user instructions, enabling unparalleled scalability and adaptability throughout the entire process, from data pre-processing to model selection and optimization.The major contributions are four-fold, summarized as follows. (1) We introduce AutoM 3 L, a novel framework that automates the development of machine learning pipelines for multimodal data. AutoM 3 L enables users to derive accurate models for each modality from a diverse pool of models and generates an executable script for cross-modality feature fusion, all with minimal natural language instructions. This approach simplifies the process of building multimodal ML pipelines and makes it more accessible to a wider range of users. (2) We advance the automation of feature engineering by leveraging a LLM to intelligently filter out attributes that could hinder model performance while simultaneously imputing missing data. This automated feature engineering process reduces the need for manual intervention and improves the overall quality of the input data. (3) We automate hyperparameter optimization by combining the LLM's self-generated suggestions with external API calls. This approach eliminates the need for labor-intensive manual explorations and enables more efficient and effective hyperparameter tuning. (4) We conduct comprehensive evaluations, comparing AutoM 3 L with conventional rule-based AutoML on a diverse set of ○ Automate feature engineering for feature filtering and data imputation. 3 ○ Select optimal models for each modality.○ Generates executable scripts for model fusion and data processing to assemble the training pipeline. 5 ○ Search optimal hyperparameters. The detailed system prompts for LLMs in each stage can be found in Appendix A.multimodal and unimodal datasets. Additionally, a user study further highlighted the distinct advantages of our framework in terms of user-friendliness and a significantly reduced learning curve. 
methods: 
 In this paper, we propose an Automated Multi-Modal Machine Learning (AutoM 3 L) framework that utilizes Large Language Models (LLMs) to automate the machine learning pipeline for multimodal scenarios. This section begins by introducing the organization of the multimodal dataset in Sec.2.1. In Sec.2.2 to 2.6, we elaborate on the five functional components enhanced by LLMs in AutoM 3 L: (1) modality inference, (2) automated feature engineering, (3) model selection, (4) pipeline assembly, and (5) hyperparameter optimization, as illustrated in Fig. 1 .Most existing studies utilize the JavaScript Object Notation (JSON) to represent multimodal data. However, JSON cannot capture the interplay between different modalities, making it unsuitable for analysis by language models. To address this limitation, we follow [3, 12, 15, 28] and employ the structured tables to represent multimodal data. Structured tables offer a clear representation that captures the interaction between different modalities and effectively aggregates information from various formats into a unified structure. Additionally, these tables encompass a diverse range of data modalities, including images, text, tabular data, and more. AutoM 3 L begins with the Modality Inference-LLM (MI-LLM) component, which identifies the associated modality for each column in the structured table. To simplify its operation and minimize additional training costs, MI-LLM leverages in-context learning. As illustrated in Fig. 2 (a), the guiding prompt for MI-LLM consists of three essential parts: (1) An ensemble of curated examples is utilized for in-context learning, assisting MI-LLM in establishing strong correlations between column names and their associated modalities, thereby generating the desired format responses. These examples Given user directives, the system initiates a query to identify the top 5 models that align with each input modality. From this refined subset, MS-LLM then determines and selects the most suitable model. serve as a foundation for MI-LLM to learn from and adapt to the specific dataset at hand, enabling it to accurately infer the modality of each column based on the provided examples. (2) A subset of the input structured table, consisting of randomly sampled data items paired with their respective column names, is included. The semantic richness of this subset acts as a guiding force, directing MI-LLM towards accurate identification of modalities. By providing a representative sample of the dataset, MI-LLM can better understand the context and characteristics of each column, allowing it to make more informed modality inferences. (3) User directives go beyond mere instructions, enriching the process with deeper contextual information. These directives leverage the LLM's exceptional interactivity to enhance the refinement of modality inference. For instance, a directive such as "This dataset delves into the diverse factors influencing animal adoption rates" provides MI-LLM with valuable contextual information, enabling a more insightful interpretation of column descriptors. This additional context helps MI-LLM to make more accurate and relevant modality inferences by considering the overall theme and purpose of the dataset.Feature engineering is a critical pre-processing phase to address common data challenges, such as handling missing values. While conventional AutoML solutions rely heavily on rule-based feature engineering, our AutoM 3 L framework leverages the exceptional capabilities of LLMs to enhance this process. Specifically, we introduce the Automatic Feature Engineering-LLM (AFE-LLM), as illustrated in Fig. 2(b ). This module utilizes two distinct prompts, resulting in two core components: AFE-LLM filter and AFE-LLM imputed . The AFE-LLM filter component effectively sifts through the data to eliminate irrelevant or redundant attributes. On the other hand, the AFE-LLM imputed component is dedicated to data imputation, ensuring the completeness and reliability of essential data. Importantly, these components work together in synergy. After AFE-LLM filter refines the features, AFE-LLM imputed then addresses relevant data gaps in the dataset.To enhance feature filtering, AFE-LLM filter incorporates the following prompts: (1) An ensemble of examples for in-context learning, which includes introducing attributes from diverse datasets and intentionally incorporating irrelevant attributes. The objective of AFE-LLM filter is to effectively distinguish and eliminate irrelevant attributes (2) Column names in the structured table, containing abundant semantic information about each feature component, thereby enhancing the LLM's capability to distinguish between crucial and dispensable attributes. (3) Modality inference results derived from MI-LLM, guiding the LLM to remove attributes of limited informational significance. For instance, when comparing a binary attribute that indicates whether someone is over 50 with a continuous attribute such as age, it becomes apparent that the binary attribute may be somewhat redundant. In this case, the binary attribute can be identified and removed. (4) User instructions or task descriptions can be embedded when available, aiming to establish a connection between column names and the corresponding task.On the other hand, the AFE-LLM imputed component is dedicated to data imputation, ensuring the completeness and reliability of essential data. Regarding data imputation, AFE-LLM imputed exploits its inferential capabilities to effectively identify and fill missing data. The prompt for this aspect includes the following: (1) Data points with missing values, enabling AFE-LLM imputed to fill these gaps by discerning patterns and inter-attribute relationships. (2) A selected subset of data instances from the training set that involves deliberately masking individual attributes and presenting them in Q&A pairs, laying down an inferential groundwork. (3) Where available, user instructions or task descriptions are incorporated, offering a richer context and further refining the data imputation process.Importantly, these components work together in synergy. After AFE-LLM filter refines the features, AFE-LLM imputed then addresses relevant data gaps in the dataset. By combining feature filtering and data imputation, this module ensures that the dataset is optimized for the subsequent steps in the AutoM 3 L pipeline.Upon successfully performing the modality inference and automated feature engineering modules, AutoM 3 L proceeds to determine the optimal model architecture for each data modality. The candidate models are cataloged within a model zoo, with each model stored as a model card. The model card captures a wide range of details, including the model's name, type, applicable data modalities, empirical performance metrics, hardware requirements, and other relevant information. To streamline the generation of these cards, we leverage LLM-enhanced tools, such as ChatPaper [37] , to eliminate the need for laborious manual writing processes. We generate embeddings for these model cards using a text encoder, thereby allowing users to retrieve relevant model cards and seamlessly expand the model zoo by appending new cards, as illustrated in Fig. 3(a) .Following the model card generation, we propose the Model Selection-LLM (MS-LLM) to effectively match each modality with the appropriate model. We view this task as a single-choice dilemma, where the context provides a range of models for selection. However, due to limitations on the context length of LLM, it is not feasible to present a complete array of model cards. Hence, we initially filter the model cards based on their applicable modality type and keep only those that are aligned with the specified data modality. Next, a subset of the top 5 models is identified using text-based similarity metrics to compare the user's requirements with the model cards' descriptions. These high-ranking model cards are then incorporated into the prompt of MS-LLM, along with user instructions and data descriptions. This combination guides MS-LLM in making its final decision, ultimately identifying the most suitable model for the given modality, as illustrated in Fig. 3(b) .The MS-LLM prompt fuses the following components: (1) A selected subset of five model cards, providing insight into potential model candidates. (2) An input context that intertwines data descriptions and user instructions. The data descriptions clarify important aspects such as data type, label type, and evaluation standards. Meanwhile, user instructions can provide clarification on specific model requirements. For example, a user instruction such as "deploy the model on the CPU device" would guide MS-LLM to models optimized for lightweight deployments. This enhances the user-friendliness and intelligence of the framework by enabling interactive execution.After retrieving the unimodal models, a crucial step involves fusing them. We employ a late fusion strategy for integration, which can be mathematically expressed as:EQUATIONEQUATIONEQUATIONwhere concat denotes concatenation, x 𝑖 represents the input data of modality 𝑖 (𝑖 = 1, • • • , 𝑛), and feature_adapter 𝑖 adapts the output of model 𝑖 to a consistent dimension. The fusion_head and fusion_model are the target models that need to be built. Determining the architectures for fusion_head and fusion_model using rule-based methods that require manual scripting is impractical, as the architectures depend on the number of input modalities. Instead, we reframe this process as a code generation challenge, where the Pipeline Assembly-LLM (PA-LLM) is responsible for generating the fusion model architecture. PA-LLM leverages the code generation capabilities of LLMs to produce executable code for both model fusion and data processors, as depicted in Fig. 4 (a). This is achieved by providing the module with relevant model configuration files within the prompt. The data processors are generated based on the specified data preprocessing parameters in the configuration file. We prioritize the integration of pre-trained models from various modalities, sourced from well-known libraries such as HuggingFace and Timm. By establishing ties with the wider ML community, we have significantly enhanced the versatility and applicability of our model zoo. In conventional ML pipelines, hyperparameters such as learning rate, batch size, hidden layer size, and loss weight are commonly adjusted manually, which is labor-intensive and time-consuming.Although external tools like ray.tune allow users to conduct optimization by specifying hyperparameters and their search intervals, there is still room for further automation. To bridge this gap, we propose the HyperParameter Optimization-LLM (HPO-LLM), which extends the foundational capabilities of ray.tune. The core functionality of HPO-LLM lies in its ability to determine optimal hyperparameters and their corresponding search intervals through careful analysis of a provided training configuration file, as depicted in Fig. 4(b) . Leveraging the extensive knowledge base of LLMs in ML training, we first utilize LLM to generate comprehensive descriptions for each hyperparameter specified in the configuration file. The descriptions, combined with the original configuration file, constitute the prompt for HPO-LLM, which then provides recommendations on the most suitable hyperparameters for optimization. The input prompt provided to HPO-LLM encompasses the following components: (1) The training configuration file, containing a comprehensive set of hyperparameters, assists HPO-LLM in selecting the most suitable hyperparameters for optimization.(2) LLM-generated text descriptions for each hyperparameter, enabling HPO-LLM to gain a comprehensive understanding of the significance of each hyperparameter. (3) Optional user directives provide a personalized touch, allowing users to incorporate additional instructions that guide HPO-LLM's decision-making process. These directives can include emphasizing specific hyperparameters based on unique requirements, resulting in a tailored optimization approach. By integrating the capabilities of ray.tune with our HPO-LLM, we have pioneered an approach that enhances hyperparameter optimization by combining automation with advanced decision-making. For the retrieval-based tasks, we employed the Shopee-Price Match Guarantee dataset (SPMG), which aims to determine if two products are identical, relying on data from image and text modalities. Our performance metrics include accuracy for multiclass classification tasks, the area under the ROC curve (AUC) for binary classification tasks and retrieval tasks, and the root mean square error (RMSE) for regression tasks. We also evaluated AutoM3 L on a large number of unimodal datasets from the AutoML Benchmark [10] available from OpenML 2 , which cover regression and binary/multiclass classification tasks. The metrics include logarithmic loss (Log-Loss) for multiclass classification tasks, the root mean square error (RMSE)2 https://www.openml.org/ for regression tasks, and the area under the ROC curve (AUC) for binary classification tasks. 
baseline.: 
 Given the scarcity of specialized multimodal Au-toML frameworks, our experimental evaluations were exclusively performed using the AutoKeras 3 and AutoGluon framework. Au-toKeras is dedicated to neural architecture search (NAS) and hyperparameter optimization for a given dataset. Setting up training pipelines in AutoGluon required meticulous manual configurations. This involved specifying which models to train and conducting an extensive pre-exploration to determine the suitable parameters for hyperparameter optimization, including their respective search ranges. It's crucial to highlight that the automation and intelligence levels of AutoGluon remain challenging to quantify, and in this research, we innovatively measure them through the user study from the human perspective. See Appendix E for detailed experimental settings.The user study conducted in this research has received full approval from the Institutional Review Board (IRB). All methodologies, protocols, and procedures pertaining to human participants were carefully reviewed to ensure they align with ethical standards.We first carried out quantitative evaluations, drawing direct comparisons with AutoKeras and AutoGluon, focusing on the modality inference, automated feature engineering, and the automated hyperparameter optimization modules. For modality inference evaluation, apart from the modality inference component, all other aspects of the frameworks are kept consistent. For feature engineering and hyperparameter optimization, we aligned the modality inference from AutoKeras and AutoGluon with the results of AutoM 3 L to analyze their respective impacts on performance. To enhance the robustness of our results, we performed 10-fold cross-validation experiments on all datasets. The accuracy is reported as the mean value with its corresponding standard deviation. Afterwards, we evaluate the pipeline assembly module in terms of intelligence and usability through user study in the next section, due to its inherent difficulty in quantitative evaluation.Inference. Table 2 depicts the comparative performance analysis between AutoGluon's modality inference module and our LLM-based modality inference approach across various multimodal datasets. Since AutoKeras utilizes manually predefined data modality for each column, we excluded it from the comparisons in this experiment. Within AutoGluon, modality inference operates based on a set of manually defined rules. For instance, an attribute might be classified as a categorical modality if the count of its unique elements is below a certain threshold. Upon observing the results, it's evident that AutoM 3 L offers accuracy on par with AutoGluon for most datasets. This similarity in performance can be primarily attributed to the congruence in their modality inference outcomes. However, a notable divergence is observed with the CH-SIMS dataset. Due to the manually defined rules in AutoGluon being unable to infer video modality and misclassified the "text" attribute as "categorical", the assembly of the training pipeline was hindered, resulting in the failure of the training task. We manually corrected the misinference of the "text" attribute in AutoGluon, achieving the accuracy of 0.543(0.032). In comparison, AutoM 3 L demonstrated a significantly superior accuracy, with a notable 3.2% improvement. Such a result highlights the robustness of our LLM-based modality inference approach, which effectively infers modality details from column names and their associated data through in-context learning with only a few examples, making it significantly more efficient than cumbersome manually designed rules.for Feature Engineering. Table 3 illustrates the comparisons of data preprocessing modules using AutoGluon and AutoKeras with our LLM-based automated feature engineering module on multimodal datasets. Given the completeness of these datasets, we randomly masked portions of the tabular data and manually introduced noisy features from unrelated datasets to assess the effectiveness of automated feature engineering. For datasets without tabular modality, only noise features are introduced. Note that, AutoGluon lacks a dedicated feature engineering module for multimodal data, making this experiment a direct assessment of our automated feature engineering. We observed that automated feature engineering, which implements feature filtering and data imputation, effectively mitigates the impact of noisy data. Across all test datasets, automated feature engineering showed improvements, while AutoGluon and AutoKeras suffered from performance degradation as they struggled to handle noisy data. Since retrieval tasks and video modality inputs are not supported, we did not test AutoKeras on relevant datasets.Optimization. We also conduct experiments to evaluate the automated hyperparameter optimization module within AutoM 3 L. Contrasting with AutoKeras and AutoGluon, which often require users to manually define the hyperparameter search space, AutoM 3 L simplifies this process.From Table 4 , it's evident that the integration of hyperparameter optimization during the training phase contributes positively to model performance. Impressively, AutoM 3 L matches AutoGluon's accuracy on most datasets and, due to its effective utilization of video information, it has realized a 2.7% improvement on the CH-SIMS dataset. However, the standout advantage of AutoM 3 L lies in its automation. While AutoGluon requires a manual setup, which can often be tedious, AutoM 3 L significantly reduces the need for human intervention, providing a more seamless and automated experience. Another finding is that AutoKeras achieves lower accuracy on all datasets. In our analysis, we attribute it to the network structures obtained within its limited network search space, which lacks pretraining on large-scale datasets. In contrast, our approach leverages the strength of pretrained models by linking with open-source communities such as HuggingFace and Timm. This integration allows us to access more powerful pretrained models, contributing to the improved performance demonstrated in our work.Given that most AutoML frameworks currently focus on single-modality AutoML, to demonstrate the scalability of AutoM 3 L, we also evaluated AutoM 3 L on a large-scale single-tubular modality AutoML Benchmark [10] from OpenML. We compared it with a plethora of popular single-modal AutoML frameworks [7, 9, 11, 20, 22, 30, 31] on large and representative datasets from AutoML Benchmark covering binary classification, multiclassification, and regression tasks. The metrics include logarithmic loss (LogLoss) for multiclass classification tasks, the root mean squared error (RMSE) for regression tasks, and the area under the ROC curve (AUC) for binary classification tasks. We reported the mean and standard deviation based on 10-fold crossvalidation. In the experiment, we employed the user instruction: "the model with the best performance on tabular data" to drive MS-LLM for model selection, and FT-Transformer was chosen and integrated with other components to form a training pipeline for subsequent training. The results in Table 5 and Table 6 demonstrate AutoM 3 L's strong performance even in single-modality settings. Notably, in most frameworks compared, model ensemble techniques are employed to produce final predictions. However, in this experiment, AutoM 3 L solely utilized a single model for evaluation and achieved competitive results, even outperforming others on most experimental tasks. We will showcase more comparative test results in the upcoming open-source projects.To assess AutoM 3 L's effectiveness, we conducted a user study focused on whether the LLM controller can enhance the degree of automation within the multimodal AutoML framework. We formulated null hypotheses: We performed single-sided t-tests to evaluate statistical significance. Specifically, we compared AutoM 3 L and AutoGluon on the following variables: task execution time, the number of attempts, system usability, and perceived workload.As depicted in Fig. 6 , our user study's workflow unfolds in structured phases. Note that the user study has been reviewed by IRB and granted full approval. The study begins with the orientation phase where voluntary participants are acquainted with the objectives, underlying motivations, and procedural details of the user study. This phase is followed by a user background survey, which gleans insights into participants' professional roles, their prior exposure to technologies such as LLM and AutoML, and other pertinent details. The core segment of the study involves hands-on tasks that participants undertake in two distinct conditions: perform multimodal task AutoML with AutoGluon and with AutoM 3 L. These tasks center around exploring the automation capabilities of the AutoML frameworks, as well as gauging the user-friendliness of their features such as hyperparameter optimization. Participants, guided by clear instructions, are tasked with constructing multimodal training pipelines employing certain models and defining specific hyperparameter optimization domains.To ensure a balanced perspective, participants are randomly split into two groups: the first interacts with AutoGluon, while the second delves into AutoM 3 L. Upon task completion, the groups swap platforms. For a holistic understanding of user interactions, we meticulously track both the time taken by each participant for task execution and the number of attempts before the successful execution. The study culminates with a feedback session, where participants articulate their impressions regarding the usability and Testing. The data we gathered spanned four variables, visualized in Fig. 5 . To validate our hypotheses, we performed paired two-sample t-tests (essentially one-sample, one-sided t-tests on differences) for the aforementioned variables across two experimental conditions: AutoGluon and AutoM 3 L. These tests were conducted at a significance level of 5%. The outcomes in Table 7 empower us to reject all the null hypotheses, underscoring the superior efficacy and user-friendliness of AutoM 3 L. The success of AutoM 3 L can be largely attributed to the interactive capabilities endowed by LLMs, which significantly reduce the learning curve and usage costs. Since most researchers were familiar with LLMs but had limited AutoML experience, increasing their learning curve on AutoGluon. Whereas the majority of engineers and students were novices in both these spheres, facing steeper challenges in grasping Auto-Gluon. Interestingly, even researchers acquainted with AutoML felt that AutoM 3 L demonstrated superior ease of use comparatively. Collectively across backgrounds, AutoM 3 L attained higher user ratings, lower task completion times, and fewer failed attempts, which quantitatively validates its improved user-friendliness. 
conclusion: 
 In this work, we introduce AutoM 3 L, an LLM-powered Automated Multimodal Machine Learning framework. AutoM 3 L explores automated pipeline construction, automated feature engineering, and automated hyperparameter optimization. This enables the realization of an end-to-end multimodal AutoML framework. Leveraging the exceptional capabilities of LLMs, AutoM 3 L provides adaptable and accessible solutions for multimodal data tasks. It offers automation, interactivity, and user customization. Through extensive experiments and user studies, we demonstrate AutoM 3 L's generality, effectiveness, and user-friendliness. This highlights its potential to transform multimodal AutoML. AutoM 3 L marks a significant advance, offering enhanced multimodal machine learning across domains. Our future direction is to encompass a diverse range of data modalities, spanning graph, audio, and point clouds, among others. 
 
 Appendix: https://github.com/keras-team/autokeras 


Decision: This paper introduces AutoM3L, an Automated Multimodal Machine Learning framework leveraging LLMs as controllers to automatically construct multimodal training pipelines, which simplifies user engagement and shows competitive or superior performance on diverse datasets and in user-friendliness compared to traditional methods. However, the reviewers raised concerns but not fully addressed in the rebuttal including:
1) This work is based on FT-Transformer in AutoGluon. It enabled the hyperparameter optimization of FT-Transformer and made some trivial optimizations.  This has not made a significant contribution to any of the fields of AutoML.
2) The experimental scope is restricted to a limited range of datasets with relatively simple features, and the study lacks ablation tests to evaluate the impact of various components. 
3) The performance improvement in multimodal tasks is not significant compared to AutoGluon. The experimental comparison is confined to a single baseline, AutoGluon, and does not adequately address or contrast with other relevant works in multimodal AutoML or Neural Architecture Search (NAS). 

The AC has throughly checked the paper and the reviews, and agree to these concerns. In conclusion, although the paper covers an interesting topic, the lack of depth in research insights and comprehensive evaluation reduces its overall contribution to the field. Considering the intense competition at ACM MM, the AC suggests that the authors revise the paper based on the reviewers' feedback and consider submitting it to the next conference with best wishes.

*** TPC Meta-Review Addendum ***
This paper is very much on the borderline for acceptance. Besides the official reviews, there was significant discussion about this paper between the AC, SAC, and the TPC. The paper has limitations in terms of its enhancements over AutuGluon and the limited performance gains in different tasks. At the same time, multiple reviewers have recognized its strengths as an end to end framework, and its value as a tool that enables multimedia research by a broader set of researchers. The TPC is suggesting that the paper be accepted as a poster to allow for more conversation on this topic. The tool, if useful for a small percentage of others, will already be a positive step forward.

Reviews: review: paper_summary: paper_summary: - This paper introduces AutoM3L, a framework that automates the development of machine learning pipelines for multimodal data by leveraging LLMs.
- The framework enhances the process of feature engineering, model selection, and hyperparameter optimization, reducing the need for manual intervention.
- Comprehensive experiments on diverse multimodal and unimodal datasets demonstrate that AutoM3L achieves competitive or superior performance compared to conventional rule-based AutoML frameworks like AutoGluon.
- Additionally, a user study highlights the user-friendliness and reduced learning curve of AutoM3L compared to rule-based AutoML methods, enabled by the interactive capabilities of LLMs.
strength_weakness: strength_weakness: - The paper introduces a novel approach to multimodal AutoML by harnessing the capabilities of large language models (LLMs) as controllers for automating various stages of the machine learning pipeline. 
- The paper demonstrates the effectiveness of AutoM3L through extensive experiments on diverse multimodal and unimodal datasets.
- The paper is well-structured and clearly written, providing a detailed description of each component of the framework and its role within the overall system.
limitations: limitations: - The paper does not fully address the computational costs associated with using LLMs, especially when considering large-scale applications or real-time processing needs.
- While AutoM3L automates the machine learning pipeline construction process, the paper does not extensively discuss the interpretability and explainability of the generated pipelines. Providing insights into how LLMs make decisions during various stages, such as model selection and hyperparameter optimization, would enhance the transparency and trustworthiness of the framework.
- The user study, though insightful, appears limited in scope with only 20 participants. A larger and more diverse participant pool might provide more robust evidence of usability and learning curve improvements.
- Moreover, the metrics used to evaluate user interaction could be expanded beyond time and accuracy to include qualitative aspects of user satisfaction and cognitive load, which would provide a deeper understanding of the user experience.
paper_summary: paper_summary: This paper introduces an automated multimodal machine learning framework called AutoM3L, which leverages large language models (LLMs) as controllers to automatically construct multimodal training pipelines. The authors point out that AutoML reduces the need for manual intervention in the machine learning process, but most AutoML solutions primarily focus on unimodal data. AutoM3L automates five functional components of the machine learning pipeline using LLMs: modality inference, automated feature engineering, model selection, pipeline assembly, and hyperparameter optimization. The authors conducted experiments on six multimodal datasets, including classification, regression, and retrieval tasks. Compared with the AutoKeras and AutoGluon frameworks, AutoM3L demonstrated competitive or superior performance on most datasets. A user study was conducted to evaluate the usability and effectiveness of AutoM3L. The results of the user study supported AutoM3L's advantages in reducing the learning curve and enhancing user-friendliness.
strength_weakness: strength_weakness: Automated multimodal machine learning (AutoM3L) is indeed an important research area. Automating this process can significantly reduce the workload of data scientists and machine learning engineers in model selection, feature engineering, and hyperparameter tuning. As far as I know, there is currently no widely recognized open-source platform specifically designed for automating multimodal machine learning (although there are some tools and libraries available for automating multimodal machine learning tasks, they may not be specifically designed for multimodal data).

The authors conducted experiments on six multimodal datasets, including classification, regression, and retrieval tasks. Compared with the well-known frameworks AutoKeras and AutoGluon, AutoM3L demonstrated competitive or superior performance on most datasets.
limitations: limitations: The fusion of multimodal data is a crucial step in multimodal data processing. In this paper, the AutoM3L framework employs a component called Pipeline Assembly-LLM (PA-LLM) which generates executable code for model fusion and data processing based on the provided model configuration files. However, the authors did not specifically validate the performance of this module in their experiments. Providing dedicated case studies and performance evaluations of this module would be beneficial in understanding its effectiveness and potential impact on the overall framework's performance.
paper_summary: paper_summary: The paper proposes an automated multimodal machine learning framework by using large language models as the procedure controller, including dataset organization, modality inference module, feature engineering module, model selection module, pipeline assembly module, and hyperparameter optimization module. Experiments on various multimodal and single-modal datasets are conducted to show the effectiveness of the framework. Further user study demonstrates the automation of the framework.
strength_weakness: strength_weakness: 1. The paper proposes an automated multimodal machine learning framework by using large language models as the procedure controller.
2. The paper is well-written and the experiments are comprehensive.
limitations: limitations: 1. The method lacks some novelty because some previous works [1][2] have done something similar to this paper.

    [1] HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

    [2] Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

2. The evaluation metrics in this paper are all end-to-end. Are there any modular evaluation metrics or ablation experiments to demonstrate the effectiveness of each design module?

3. The efficiency issue of large language models. Is there any advantage in efficiency of this paper compared to traditional methods, as invoking large language models requires high costs and time?
paper_summary: paper_summary: The manuscript presents AutoM3L, a framework for automated multimodal machine learning. This framework leverages large language models (LLMs) to enhance the processes of modality inference, feature engineering, model selection, pipeline assembly, and hyperparameter tuning. Comparative analysis with frameworks such as AutoGluon suggests that the proposed method outperforms competitive baselines. Additionally, the study introduces a user study to evaluate the framework.
strength_weakness: strength_weakness: 1. The paper explores the integration of LLMs within an AutoML framework, which is a significant contribution to the field.
2. The authors have conducted a user study to assess the usability of the proposed method, adding depth to the evaluation.
3. The framework demonstrates efficacy advantages on single-modal tasks, showcasing its versatility.
limitations: limitations: 1. The paper does not clearly articulate the specific problem it aims to address, instead listing a series of optimizations targeting AutoGluon, which are trivial.
2. There is a lack of ablation studies to evaluate the impact of individual modules.
3. The improvements in performance on multimodal data compared to AutoGluon are marginal.
4. The linked code repository contains empty code folders, which is not conducive to reproducibility. The authors should populate the repository with the actual source code.
5. Given the significant financial cost associated with LLMs, there is a need for a discussion on cost, which is currently absent.
6. If the primary contributions are attributed to the LLMs, a comparison of different LLMs should be discussed in terms of their effectiveness.
7. The quality of figures is suboptimal; for example, the figure in the upper left corner of Figure 1, and Figure 4(a) are not clearly rendered.
8. There are typographical errors in Figure 1; "Genger" should be corrected to "Gender", and "Hyperparamter" should be corrected to "Hyperparameter".
9. Figure 3 requires additional specifics for coherence: What is the precision of the mentioned retrieval? How many models are included in the model zoo, and what is the distribution of model usage within the zoo?