Large Language Models Synergize with Automated Machine Learning
Authors: Jinglue Xu
Jialong Li
Zhen Liu
NAV Suryanarayanan
Guoyuan Zhou
JIA GUO
Hitoshi Iba
Kenji Tei
Venue: Accepted by TMLR ()

Abstract: Recently, program synthesis driven by large language models (LLMs) has become increasingly popular. However, program synthesis for machine learning (ML) tasks still poses significant challenges. This paper explores a novel form of program synthesis, targeting ML programs, by combining LLMs and automated machine learning (autoML). Specifically, our goal is to fully automate the generation and optimization of the code of the entire ML workflow, from data preparation to modeling and post-processing, utilizing only textual descriptions of the ML tasks. To manage the length and diversity of ML programs, we propose to break each ML program into smaller, manageable parts. Each part is generated separately by the LLM, with careful consideration of their compatibilities. To ensure compatibilities, we design a testing technique for ML programs. Unlike traditional program synthesis, which typically relies on binary evaluations (i.e., correct or incorrect), evaluating ML programs necessitates more than just binary judgments. Our approach automates the numerical evaluation and optimization of these programs, selecting the best candidates through autoML techniques. In experiments across various ML tasks, our method outperforms existing methods in 10 out of 12 tasks for generating ML programs. In addition, autoML significantly improves the performance of the generated ML programs. In experiments, given the textual task description, our method, Text-to-ML, generates the complete and optimized ML program in a fully autonomous process. The implementation of our method is available at https://github.com/JLX0/llm-automl.

Jia Guo Published in Transactions on Machine Learning Research (09/2024)
Large Language Models Synergize with Automated Machine Learning
Jinglue Xu University of Tokyo
jingluen@@maillcom com com
Jialong Li Tokyo Institute of Technology
Ejjialong@ffjj.uuusccdoo
Zhen Liu University of Tokyo
Nagar Anthel Venkatesh Suryanarayanan University of Tokyo
woo-ssuryadgg.cc..uttooo/
Guoyuan Zhou
ahouguoyaannuwcmmaiilaaa.cccc
Hosei University Institute of Integrated Science and Technology
guojia311 (atto3111@@mmll.com com
Hosei University Institude of Integrated Science and Technology
Hitoshi Iba University of Tokyo
ibo Mibaaib..t.oo f. t.-kyyyo.tooooco ac.jp
Kenji Tei
rei@c.titecl ac.jp
Tokyo Institute of Technology
Reviewed on OpenReview: https: // opwneevie.. id- RDEaIfOsJM
Abstract
Recently, program synthesis driven by large language models (LLMs) has become increas- ingly popular. However, progrum synthesis for machine learning (ML) lasks still poses significant challenges. This paper explores novel form of program synthesis, targeting ML programs, by combining LLMs and automated machine learning (autoML). Specifically, goal is to fully automate the gencration and optimization of the code of the entire ML workflow, from data preparation to modeling and post-processing, utilizing only textual descriptions of the ML tasks. To manage the length and diversity of ML programs, we propose to break each ML program into smaller, manageable parts. Each part is gener- ated separately by the LLM, with careful consideration their compatibilities. To ensure compatibilities, we design a testing technique for MI. programs. Unlike traditional program syuthesis, which typically relies on binary evaluations (i.e., correct incorrect), evaluat- ing ML programs necessitates more than just. binary judgments. Our approach automates the numerical evaluation and optimization of these programs, selecting the best candidates through autoML techniques. In experiments across various ML tasks, our method outper- forms existing methods 10 out of 12 tasks for generating ML programs. In addition, autoML significantly improves the performance of the generated ML programs. In experi- ments, given the textual task description, our method, Teat-to-Mo ML, generates the complete and optimized ML program in a fully autonomous process. The implementation of our method is available at https://github. pectthbb.cmm/ILX/111maaaaaa

---

Published in Transactions on Machine Learning Research (09/2024)
1 Introduction
1.1
Background, challenges, and motivation
Textual description of the task
Optimized data preparation module
The description of the input data is: Argumentative essays written by 8th-12th grade English Language Learners..
isport
Data leading
trein.dr = pd.reed.co covd www.:/wwwkkweeeeeeeooee worksspcce/rreen.coomm cov')
The description of the output data is: Scores according to six analytic
Text cleening
measures: cohesion, syntax, vocabulary, phraseology, grammar, and conventions.
trein_df] fradn_rrrlulltttee text train.dfi text
apply (lanbda x: re.sibl" ["a-zA-21", ",
objective the machine learning task is: For each essay, predict of measures.
Tast-specarze feature engineering
train_afl of[ ava_sentencs 1ongth" trwin_ofl ofl "esssyylengthh" |rain_df[[ www.peraanueeeennee perpperagraphs"
The evaluation metric MCRMSE.
The following files are available: (workspace//rain.cev "A table of columns. The first column named 'text_id' describes the unique ID for each essay. The second column named 'full. text' contains the essays. The third to the eighth columns are named 'cchesion,, 'syntax', vocabulary, 'phraseology!, grammar, and 'conventions' respectively, with each column containing scores ranging from 5.0 increments of O.5."
attention. www.s. apsks.a inpou ngl attention, mask"" roal ccohesion. "syntax' vocabulary"
plr wowoloyy
"grammerr" "conventione tolixt()
Optimized modeling module
import ... class BERTLSTNWodel(mm Module): :
Automatic generation and optimization
LLM
AutoML
self bert DertModel, Fron.pretraineed
( ('bert-bese-une) self lsst nS.LSTM input_sice=768,
Generate each candidate program modularly
Select the best program via optimization
hidcen.eize--25. nun. nww. leyersn2, farth_Far self.fe fo rn.Lineer
def forvard(selr, input__dd,, attention_naak):
Optimized post processing module
outputs self. bertfbberr(tnput... ids-idduttttttt ids, Sacnttion_nsk--atttniioo Istn_output, salf. wtttts/sssuuusseeeeeo ouuppt))
Optimized hyperparameters
Learning rate, batch size. weight decay, momentum...
Figure 1: One-click program synthesis for machine learning. This figure presents example of our qualitative results based on the dataset Learning. Our method, Text-to-MT,, transforms textual task description into an optimized program, encompassing data preparation, modeling, post processing, and hyperparameters. All modules are scamlessly integrated into a single executable program. The entire process, from the user's input of a lextual description to the creation of an end-to-end, optimized, and executable program, fully antomated. The code examples the figure are folded and abbreviatod for readability.
Program synthesis is the automatic creation of computer programs that satisfy high-level specifications (Manna & Waldinger, 1971). In recent years, autoregressive large language models (LLMs) (Achiam et al., 2023; Touvron et al., 2023; Anil et al., 2023) have demonstrated impressive performance in program synthesis tasks (Austin et al., 2021; Li et al., 2022; Shinn et al., 2023). However, existing LLM-based program synthesis methods primarily focus on traditional coding problems, such as interview questions for software engineering positions (Austin et al., 2021; Chen et al., 2023b; Madaan et al., 2023), while the synthcsis of machine learning (ML. programs remains largely unexplored (Figure 2). The challenges associated with program synthesis for ML stem from their length, diversity, and complexity in the testing phase:
Length: The first step in our study generate an entire ML program, which consists of three task-specific components: (1) data preparation, such data loading, cleaning, encoding, and feature engineering; (2) modeling, which implements ML model, such as BERT (Devlin et al,, 2018); and (3) post-processing, such as performance evaluation and decoding the results (Figure 1). These components collectively demand a lengthy output from LLMs. For instance, the validly generated code for the ML task depicted in Figure 1 consists of an average of 114#32 lines, in stark contrast to the average 6.3 + 5.0 lines of the reference solutions in the HumanEval benchmark, which addresses traditional coding problems (Chen et al., 2021). For LLMs, the output length strongly correlates with the difficulty of the generation process (Anil et al., 2022; Newman et al., 2020).
2

---

Published in Transactions on Machine Learning Research (09/2024)
Diversity: To achieve the automatic generation of the program for the entire ML program, many lask-specific components of the ML. program should be generated during the process, rather than relying on an inventory of pre-written modules (Surís et al., 2023; Gupta & Kembhavi, 2023; Lu et al., 2023). Depending on pro-written modules for a wide range of tasks is often impractical (Hollmann et al., 2023); especially given the vast combinations of requirements and characteristics across different ML tasks, such as the structure of dataset files, dataset features, and desired output format.
Complexity in testing phase: Evaluation of prograins for traditional coding problems often relies on manually written white-boc tests (Chen et al., 2023b; Li et al., 2022), such as unit tests (Myers et al., 2004). Unit tests verify the correctness of individual code components, such as functions or classes, by comparing their actual behavior with the expected behavior based on specific inputs and outputs. For ML programs, unit tests are insufficient and costly to create (Tian et al., 2018; Wang et al., 2021b). From the perspective of both software testing and ML, syntactically valid ML programs should be further examined based on numerical performance metrics, such as Top-1 accuracy or mean absolute error, and a consistent evaluation procedure for semantic accuracy (Kim et al., 2019; Braiek Khomh, 2020; Saha et al., 2022; Cambronero Rinard, 2019).
Traditional coding problems
Given a string s, find length of the longest substring without repeating characters.
1. Given student-written essays, the objective is to predict the scores of the essays
O LLM
generates
Code
Tests
Machine learning programs
graded by experts.
2. The evaluation metrics is mean columnwise root mean squared error (MCRMSE)
provide feedback
instructs
3. The available files include...
Figure 2: Program synthesis driven by autoregressive LLMs. The desired functions are provided as instruc- tions to the LLM, which generates the corresponding code. An optional testing phase can be included, where the code is tested for errors, and the test results provide feedback for the LLM to debug and refine the code.
Notably, the complexity in testing phase entails a novel form of program synthesis, sctting it apart from traditional program synthesis. In traditional conlexts, program synthesis often concludes with code genera- tion and perhaps testing through binary evaluation procedures, such as unit tests. However, MT. programs necessitate numerical evaluations, extending the synthesis process beyond just code generation (Kim et al., 2019; Braick & Khomh, 2020; Saha et al., 2022: Cambronero & Rinard, 2019).
How to ensure that each generated program is compatible with the code of consistent. and fair training, testing, and evaluation procedure? More importantly, how to effectively utilize numerical feedback involving performance scores? A numerical optimization algorithm efficiently addresses the challenge. Consequently,
for MT. programs, program synthesis often necessitales optimization. Real et al. (2020); Saha et al. (2022); Cambronero & Rinard (2019) also explore this form of program synthesis (scc Scction 2.3). Similar these studies, we also utilize automated machine learning (autoML) as the numerical optimization step. However, our approach is distinct in that the generated programs are inherently compatible with optimization algorithms, requiring no further modification.
ML heavily impacts modern society, with applications spanning healthcare, finance, manufacturing, and beyond. Implementing ML typically demands substantial expertise and effort, which limits ML's accessibility for many potential users. Many studies work on automating MI. utilizing LLMS or autoML, however, the full automation of ML tasks at the code level remains an unresolved challenge (see Sections 2.2 and 2.3 and Appendix A.2). In this paper, we explore the limits of program synthesis for ML. by combining LLMSs
3

---

Published in Transactions on Machine Learning Research (09/2024)
User Input
breaks down and generates the program modularly
generated modules
pre-written modules
Textual description of an
instructs
Data preparation
Loss function
ML task:
Data
Objective
LLM
data flow
verifies
data flow
Optimizer
Eva Evaluation metrics
Local files
Modeling
suggests
Training procedure
Optimization Phase
data ftow
verifies
Serve
verilies
AutoML
options
Candidate hyperparameter values
Post processing
Testing procedure
Generation Phase
selects
automaticaly Integrates
Repeat the generation and testing phases generate multiple candidate ML programs
The best ML program and best associated hyperparameters based the metrics
enobles
Candidate ML programs
A complete ML program
Unit tests
Testing Phase
Figure 3: Text-to-ML: three-phase process of generation, lesting, and optimization. Initially, with the task description, the LLM generates the modules. The modules are verified by unit tests to ensure inter- compatibility. With the compatibility, they are automatically integrated. Then, iterate to produce multiple candidate programs. Finally, autoML selects the best program and associated hyperparameters.
and autoML. During this exploration, aim to automate the generation and optimization of the entire ML program, reducing the human workload to mere textual descriptions of the tasks. Our study focuses on synthesizing the programs involving supervised ML tasks, encompassing traditional algorithmss such as random forests and SVM, as well as deep learning domains including computer vision (CV) and natural language processing (NLP).
1.2 Automatic code generation
Recently, many studies have leveraged autoregressive LLMs for sequence generation lasks. For example, Chain-of-Thhuugh (Wei al., 2022) enhances reasoning performance by incorporating input-output examples within the imput sequence, while Reflexion (Shinn al., 2023) utilizes reinforcement learning strategy to boost generation performance by reflecting feedback. Despite these advancements, these frameworks still struggle with generating MI. programs (see Table 1). To manage the length and diversity in ML programs, we introduce a novel sequence generation framework for autoregressive LLMs, Contertual Modular Generation (Algorithm 1). As a form compositional reasoning (Andreas et al., 2016), this framework breaks down the programs into multiple smaller modules, with each module being generated in separate process and representing a distinct code component of the overall program (Figures and 3).
Unlike traditional compositional approaches (Andreas et al., 2016; Surís et al., 2023: Gupta & Kembhavi, 2023; Lu et al., 2023), Contextual Modular Generation is distinctive for two features: (1) instead of hinging on a fixed inventory of pre-written modules (c.g., selecting and reusing code from repository of pro-written codes for available feature engineering methods), the framework dynamically generates the essential modules tailored to each specific task. In our study, the generated modules include data preparation, modeling, and post processing. (2) the generated modules are readily compatible without requiring additional sequence generation steps from LLMs for combination. This feature contributes scalability by ensuring linear growth in generation complexity program length (Theorem 3.8).
4

---

Published in Transactions on Machine Learning Research (09/2024)
1.3 Automatic unit testing
With each module being generated in a separate process, Contextual Modular Generation requires that the generated modules be readily compatible with each other to reach the linear complexity in the generation process (Theorem 3.8). To ensure this compatibility, we devise a novel testing technique for ML programs. This technique verifies compatibility by examining that the modules exhibit consistency in data structures (such tensor shapes) and contents (such as data features in cach tensor dimension). The verification process utilizes aulomatically generaled unit tests and synthelic data constructed from LLMs (see Section 4).
This technique not only ensures compatibility among newly generated modules but also between these mod- ules and pro-written, less-variant modulcs. Specifically, the technique maintains uniformity in certain mod- ules that should be identical across different program instances, such as the loss function, optimizer, training and testing strategies, evaluation procedure of prediction accuracy, and optimization strategy (Figure 3).
Contextual Modular Generation with the testing technique effectively generates syntactically valid ML pro- grams (see Table 1). However, our findings align with studies such as Gu et al. (2022) and Kim et al. (2019), which suggest that unit tests are inadequate for ML applications (sce Table 3). Most notably, the semantic accuracy (i.e,, the prediction performance) is often for from optimal, which underlines the limited practical utility of ML programs generated solely by LLMs and unit tests. To address this issue, we implement autoML in pre-written modules to further evaluate and optimize the programs, capitalizing on the compatibility of generated modules with pre-written modules.
1.4 Automatic code optimization
Candidate solutions
Optimization
Evaluation
choice
Data
Post
Training
preparation
processing
Modeling
Hyper-
Decision variables
Objective function
Testing
parameters
feedback
Figure 4: The autoML step in our method. Each solution is program that implements a candidate combination of data preparation, modeling, and post-processing algorithms and associated hyperparameters. First, the optimization algorithm chooses a solution. This chosen solution is then trained and evaluated, which provides feedback to improve selections in subsequent iterations.
The last step of our method is autoML (Figure 3). AutoML is the process of automatically selecting algorithms and hyperparameters for MT. tasks (Thornton et al., 2013). For a task, an autoML. process begins by designating a solution and then proceeds to data processing and model training based on the solution. Subsequently, the solution evaluated, providing feodback for future iterations to designate more promising solutions (Figure 4). Typically, autoMI. utilizes optimization algorithms, such as evolutionary computation or reinforcement lcarning, to designate the solutions. In our study, the docision variables the optimization (i.e., the elements of the solution) include choices and hyperparameters regarding data preparation methods, ML models, and post-processing methods implemented by the ML program. The objective function (i.e., the goal of the solution) is the prediction performance of the ML program, measured by the numerical metrics.
Due to the aforermentioned complexity involved in lesting ML programs, existing works in program synthesis for ML often employ autoML methods (see Section 2.3). The autoML step in our method adapts the strategies in Öztürk et al. (2022) by incorporating zero-cost (ZC) proxics (Mellor et al., 2021) (soe Appendix C.5). These proxies predict the performance of the trained neural networks using at most a single forward/ forward/backward backward propagation pass, which significantly reduces the cost from evaluating (training and testing) the models.
5

---

Published in Transactions on Machine Learning Research (09/2024)
1.5 Summary
With LLMs, divide the code into smaller modules and generate each separately
How to ensure the modules are compatible with each other?
Program synthesis of the complete ML workflow
How to generate lengthy code like ML programs?
How to numerically evaluate the generated programs and utilize evaluation feedback?
Without numerical evaluations, the generated programs are far from optimal
Numerical optimization from autoML
The generation framework with the testing technique
Native compatibility between generated modules from LLMs and pre-written modules
Figure 5: Program synthesis for ML. (1) black rectangle: the goal of this study. (2) red rectangles: challenges. (3) green rectangles: methods or features of methods. (4) solid rectangles: novel goals, methods, or features introduced in our study. (5) dashed rectangles: existing challenges or methods.
Figure 5 illustrates the relation among the goals, challenges, and methods in our study, Our contributions are summarized follows:
We explore the limits of program synthesis for ML. Qualitatively, for 12 supervised ML tasks over traditional data, CV, and NLP, Teat-to-ML. generates the code of the end-to-end, optimized program in a fully automatic process, given textual descriptions of the tasks (Figure and Section 6.2). In the exploration, we offer a new perspective for combining LLMs and autoML. We identify synergy in this combination that enhances program synthesis for ML (see Section 7.1).
We propose sequence generation framework Contertual Modular Generation for autoregressive LLMs, with thcoretical analysis of scalability over sequence length (Theorem 3.5 and Theorem 3.8). To ensure compatibility among the modules, as demanded by the framework, we devise novel testing technique based on progenitor testing protocols (see Section 4) and LLMs. Our approach generates codes that are natively compatible with optimization algorithms,, removing the need for extra adaptation and facilitating the immediate deployment of code optimization.
For 10 out of 12 ML. tasks across modalities, Contextual Modular Generation outperforms existing methods (Section 6.2). Furthermore, autoML substantially improves the performance of the gener- ated programs (Section 6.4). In addition, we find that ZC proxies speed up the search with both pretrained CV and NLP models (Section 6.4).
2
Related Works
2.1 Large language models
Let the input and output sequences be (11, Ø2...., FMM and Y (1..,,,...,,u .N,,,, where each Im or Yn represents token in the respective sequence. Let the current state of the LLM (e.g., choice of LLM, parame- ters in the LLM, and choice of sampling method) be X. The conditional probability distribution of generating Y given X is: p( = N Px (x(ym Men X). Throughout the paper, we denote (aj, a2....,,aa.. ax-1) as azN: where each an is a token (and a<N is a sequence) or each on is a sequence (and asN is concatenated sequence).
6

---

Published in Transactions on Machine Learning Research (09/2024)
Compositional reasoning decomposes reasoning process into multiple intermediate steps (Andreas et al., 2016). Compared to traditional compositional reasoning (Andreas et al., 2016; Surís et al., 2023; Gupta & Kembhavi, 2023; Lu et al., 2023), our method generates modules during the reasoning process. In this aspect, a similar method is Le et al. (2023). Both methods decompose the generation of the output sequence into multiple generations of smaller modules: YI,Y?,... YY,, where each YJ = (vi. v....... v.)) For each Y), let the corresponding input sequence be X then p(Y)|X1) II p Wk X
A major challenge in generating the modules is whether V "2....,,?? Y form a valid solution. Le et al. (2023) addressos the issuc by incorporating the modules as the input sequence in an additional generation step to adapt the modules for the final output. However, Le el al. (2023) mainly targets traditional coding problems. For ML programs, this additional generation step is less effective (Table 1), due the length of the programs. By comparison,
method automatically ensures the compatibility of generated modules and requires no extra generation step for combination.
Besides compositional reasoning, sequence generation methods for autoregressive LLMs also include iterative and prompting approaches. See Appendix A.l for more related works of our method regarding LLMs.
2.2 Automated machine learning
AutoML methods typically involve optimization algorithm and scarch space (i.e., set of candidate solutions) (Zoph& & Le, 2016; Liu et. al., 2018). Öztürk et al. (2022) explores the search space of pretrained deep learning models. Our autoML method adapts the strategies in Öztürk et al. (2022). In addition, we incorporate ZC proxics from Mellor ct al. (2021) and Tanaka et al. (2020) in our method, which accolcrates the optimization for deep learning lasks by predicting the performance of the models.
Very recently, several works have explored the application of LLMs for autoML, however, none of the works achieve automatic generation of the entire ML program. For a qualitative comparison, see Appendix A.2.
2.3 Software engineering
Program synthesis is a longstanding objective within the arctificial intelligence research community (Manna Waldinger, 1971). Recently, there has been a surge of interest in MT. for program synthesis, particularly with LLMs (Austin et al., 2021; Li et al., 2022). Conversely, program synthesis for ML is also an emerging topic and often involves autoML. Real et al. (2020) evolves programs for ML models. Saha et al. (2022); Cambronero & Rinard (2019) construct the programs for the modeling and part of data preparation for tasks about tabular data. Leveraging LLMs, our research expands the scope of program synthesis for ML by targeting the entire ML program across task modalities.
Software testing techniques (Myers et al., 2004) can assist program synthesis evaluating the programs. Software testing for ML programs is challenging topic (Braick & Khomh, 2020). Pei et al. (2017) and Tian et al. (2018) address the challenge for computer vision lasks. Recently, Chen et al. (2022); Shinn et al. (2023) employ LLMs to automatically generate tests for traditional coding problems. Based on LLMs, the testing technique in our study is designed for implementing the Contextual Modular Generation franework. Compared to existing works, our technique is uniquely applicable for testing ML programs at a modular level and across task modalities (traditional algorithms, CV, and NLP).
3 Generation of Lengthy and Complex Sequences
3.1 Decomposition and compatibility
Compared to typical sequence generation tasks, code generation demands a complex output sequence. For example, generaling 10 lines of code to meet a certain requirement is often considerably more challenging than an open-ended short writing of similar length. In addition, for an ML program, certain components suchh as data preparation, modcling, and post-processing -ecessiinn-uuuurree.. task-spocific solutions. The components constitute lengthy concatenated sequence, which can be many times longer than those found in traditional coding problems.

---

Published in Transactions on Machine Learning Research (09/2024)
In this section, introduce sequence generation framework, Contertual Modular Generation, designed to scale effectively with the length of highly complex sequences (Figure 6 and Algorithm 1). The idea is straightforward: the required program is decomposed into distinct components modules, which are then generated one at a time. As cach module is generated, it undergoes an evaluation procodure (c.g., unit tests) ensure integrales seamlessly with the other modules considering the context. If a module fails this evaluation, the generation process is iterated, using feedback from the evaluation procedure refine the output.
This framework is form of compositional reasoning (Andreas et al., 2016), which solves challenging problems by breaking them down into smaller, manageable components modules. The framework also mirrors the intuitive practices of many human programmers who break down a large program inlo manageable chunks, verifying the functionality and compatibility of each segment during its creation. While the framework is generic for any sequence generation tasks under the availability of sequence evaluation methods defined Definition 3.1, the framework is particularly effective in generating ML programs.
provide feedback
be verified
Textual task description
Co>
A complete ML program
generates
Module
Module 2
Madule J
LLM
Evaluations
integrate into
Figure 6: Contextual Modular Generation for ML tasks. First, the task description is given to the LLM. Then the LLM generates each module (i.e., the code for a step in ML) sequentially. Each module generated in an iterative process, which includes the generation of code by the LLM, followed by verification via the evaluation methods (unit tests in Section 4), and incorporating feedback from these evaluations back into the LLM. In the end, the generated modules are automatically integrated into the ML program.
3.2 Formal definition
Consider the scenario where binary evaluation method, denoted E, is available to test whether generated sequence meets specified requirement (e.g., absence of detectable errors and presence of desired functionality). For a given task, there exists a set of sequences that fulfill this requirement, represented as = (wi, WC). Let E(Y) denote that the sequence Y has been assessed by E and meets the specification. To establish fair comparison framework applicable to sequence generation methods (see Section 2.1 and Appendix A.1), we focus on the task repeatedly generating the target sequence until a satisfactory sequence produced. This task involves LLM with state X and set of input sequences X,. In the in iteration, the LLM generates LLM produces a sequence Y, such that Y can be any sequence or any concatenation of sequences generated in the current or previous iterations that meet the user's requirements as verified by E.
output sequence Y, using X;. The process continues until the
Algorithum 1 defines our sequence generation framework (Figure 6). As a form of compositional reasoning, A Algorithm 1 first divides the entire program into J modules. Then for each module j, the algorithm repeat- edly generates the code until success (lines 4 to 11). In the ith iteration, the LLM fx first generates the code YJ using the instruction X² (line 8). Then, the evaluation method for the module E examines Y and produce the feedback R (line 9) Tn line 6, previous instructions x Zi generated code y and feedback Ret are assembled as the instruction x² for the current iteration.
A key premise of the framework is the availability set evaluation methods as defined in Definition 3.1. These methods verify each module's 's individual functionality and compatibility with other modules.
8

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm Contextual Modular Generation
Input: An autoregressive LLM fA, number of modules J, initial instructions for generating each module (Xg. X ...... ..',x,'h, Xo', contextual modular ovaluations for cach module {E E² p.... E and instruction constructors for each module {Φt $2 ......41)
2: for =1 to J do
3: i
4: repeat
if i then
X 1 (X Y Rea)
end if
Y fx(X))
R2 EY?)
10:
i=i+1
11:
until R, True
12:
Yi yi=Y²
13:
end for
14: Output: YY,Y?...., YJ
Definition 3.1. For task with desired sequences Ω suppose the output sequences in one decomposition are [Y-, y?..., Y.). For a set of binary evaluation methods {E*, ,E,...,,BBB E'}. We call the set of binary evaluation methods as contentual modular cvaluations for the task, if that every in {Y vr,y2,.. y Y. is verified as valid by the corresponding E in (E), E² E EA} guarantees that { y2,..,, yV3 form valid prgram (if ((vj, (Y")) (")) 22)).
In other words, if the generated modules are verified by the contextual modular evaluations, then the modules are compatible with each other to form desired program without any modification. Contextual modular evaluations eliminate the need for extra computation for adapting the generated modules inlo sequence Y such that Y E Ω. In our study, the contextual modular evaluations verify individual components in the ML program to ensure consistency in data flow based on a novel testing technique (Section 4).
3.3 Theoretical analysis
Next, we theoretically analyze the scalability of Algorithm 1 over the length of complex output sequences. First, we define several metrics for comparing sequence generation methods. There are multiple ways to define the output length of a sequence generation problem (Anil et al., 2022). In our study, we define the output length of the task as the length of the longest valid output sequence: r maz([Y]] S2. Under the comparison scheme defined in Section 3.2, suppose the process of repeated generation terminates and the total number of iterations is G(T) (We count G(T) as oa if the iteration never terminates). Then the expectation of generations per success Ez~p[G(T)) is measurement of the generation efficiency.
For a sequence and an integer n, let (n(a, denote the set of sequences {s|]s, A (3m (1 S m S T), 3) A (a, 8, 8,8))] ]] m (a, 8, 8x) E [), where 8, is an arbitrary sequence (including the empty sequence) and |a|+ + n < P. Intuitively, in a sequence generation process, @g(a, n2) represents all valid sub-sequences from the (lal + 1),, position the (Jal + л)Tл position in the output sequence, given that the previously generated part in the output sequence is a. nf = 1, then On(a, n) represents all valid tokens the (la) 1),,, position. For example, in Figure 7, let t; denotes the ieh candidate token, then On((tz, ta te), 1) {41, ts).
Definition 3.2. For a task with desirod sequences Ω, LLM X, and input sequence X, let Px (*) denote:
Px (0 px (v. Onlven, ()U((i) | (wer E Oa(0,i 1al--11uunf 1) 1))
where () represents the empty sequence, 1 - 1) represents the event that the generation process terminates before the (i- 1)th position in the output sequence, ((i) represents the union of 17(1) and that the generated sequence Ω yı () if 17()) and yei () if n 1).
9

---

Published in Transactions on Machine Learning Research (09/2024)
Candidate tokens, from the to the 8th
Px (1)
Q2
1 position
Px 0.
P.A(2) (2)
2nd position
fdd (3)
}
Px
Sequential transitions
3 position
Px (4)
4th position
a.d Px (5)
(5)
5thh position
0
Figure 7: Sequential transitions in generating a sequence. The diagram provides simplified illustration of lask in which the put sequence length, r is 5, and the vocabulary size (i.e., number of choices for each token) is 8. Each solid dot represents a candidate token that could be included in desired sequence at the specified position. Each red dot signifies a valid end-of-sequence (EOS) token, marking the conclusion of a sequence. The yellow and green lines each represent a tree structure, spanning all possible paths from a valid starting token to any EOS token, to construct desired sequence. The cumulative probability of adhering to any path in any of the tree structures could decrease exponentially with each additional token generated.
We define token-level complexity at (0 < e 1) for the task as:
1SSISTARR - (p 1 <<
Remark 3.3. Px (i) represents the probability that the generation step is correct for the its position of the output sequence or a correct sequence with a length shorter than i is generated. With this formulation, the generation process can be modeled as a Markov process with sequential transitions (successful token-level generation steps) and an absorbing state (a sequence with an incorrectly generated token) (Figure 7). <x (e) represents the number tokens that are difficult to generate correctly (with a success rate less than e) and (R.((e) (e) < I' for any e.
<2-(())
0 0 < Assumption 3.4. For the repeated generation task defined above, in every iteration, the token-level com- plexity at is r.
Theorem 3.5. For any sequence generation process, if there exists a step in the process that generates the entire output sequence and Assumption 3.4 holds for the step, then the best case complerity of Eg-p(G(T) the process is ((erp(T))
Remark 3.6. Assumption 3.4 posits that errors may occur for any token of the output sequence, and the validity of this assumption is contingent upon Ω X, and X. Theorem 3.5 suggests that for traditional sequence generation approaches based on autoregressive LLMs, if the task exceeds a certain complexity level, the generation difficulty escalates exponentially over the length of the problem. If the value of &x ( (c)
is ascertainable or estimable for a given e, regardless the applicability of Assumption 3.4, deriving a more specific lower bound of (GT)) might be possible (Appendix B).
Assumption 3.7. For the repeated generation task defined above, in every iteration, for any e 0, the token-level complexity at is r.
Theorem 3.8. Iʃ Assumplion 3.7 holds, then for Contextual Modular Generation defined in Algorithm 1, the best case complerity of [[_r](((T) is O(T).
Remark 3.9. Assumption 3.7 imposes a more adverse condition compared to Assumption 3.4. Theorem 3.8 suggests that for Contextual Modular Generation, the best case, the generation difficulty increases only linearly over the problem length, even for highly challenging tasks.
4 Unit Tests for Machine Learning
Contextual Modular Generation conceptually intuitive, but its implementation demands meticulous design and non-trivial efforts. In this study, utilize unit tests contextual modular evaluations (as defined in
10

---

Published in Transactions on Machine Learning Research (09/2024)
Definition 3.1) to ensure compatibility among the modules. The desiderata for the unit tests in study are:
As a basic role of unit tests, for each module, the tests should ensure the module is free from errors detectable by the code execution environment (e.g., Python interpreter) and confirm the presence of required functionalities, such as properties of the data processed by the module.
Due to the high diversity in the code of the generated modules, the tests should be automatically generated. For instance, the required tensor shapes for the modules vary significantly depending the ML tasks involved.
Internal compatibility: In our method, for each module in the ML program, multiple code imple- mentations of different candidate methods are generated to construct multiple candidate programs. (Figure 8 and Figure 3). Verified candidate codes any module should be compatible with the candidate codes of other modules. This compatibility crucial for the unit tests' role as contextual modular evaluations (as defined in Definition 3.1). Furthermore, for an ML program with N mod- ules, if M candidate codes for each module generated, this compatibility increases the ratio of generated program combinations over generation cost by factor of up to M (Figure 8).
External compatibility: Verified modules should also be compatible with pre-written modules, in- cluding those implementing a consistent evaluation procedure, training procedure, testing procedure, and autoML methods (Figure 8).
These critoria require a balance between variation and constraint: each test targets task-specific module while adhering the principles of both internal and external compatibility. To address this unique challenge, develop the testing technique Constrained Generative Unit Testing, illustrated in Figure 8. Given that both the inputs and outputs of all modules in ML programs data-centrie, the technique predominantly ensures consistency in the structure and content of the data flow. A detailed formulation of the technique is available in Algorithm 3 of Appendix C.1.
verification
Specific unit tests with additional requirements
Protocol: minimum
preparation
Data
Modeling processing
Post
Pre-written
requirements
Content of each dimension
modules
1. Image height
2. Size each dimension
1. Loss
2.
Image width
constraint
4.
3. Data type in each dimension
2. Optimizer
Training
verification
4. Testing
variation
Data flow
Synthetic data
LLM
Multiple ML programs
Figure 8: Constrained Generative Unit Testing for CV tasks. The yellow arrow represents the data flow between codes. The unit tests focus on verifying the structure and content of the data exchanged among the codes. For each module, codes for different candidate methods are generated. With the pre- written modules, each pathway through the network forms an ML program.
In the technique, the progenitor testing protocol is suite of unit tests that serves the foundational basis for the more specific umit tests generated by LLMs (Appendix C.1). This protocol may encompass one several subsets of tests, each addressing a particular domain of tasks, such as NLP or CV. Each subset. imposes minimum requirements that modules must meet for any task within the targeted domains. For instance, for typical CV tasks, the data processed by any modules should at least include one feature dimension for image height and another for image width. Besides addressing some of the requirements for internal compatibility, the primary purpose of the protocol to ensure external compatibility -ny generated modules compatible with the set. of pre-written modules for the domain.
11

---

Published in Transactions on Machine Learning Research (09/2024)
The LLM specifies additional, more precise requirements for the given ML task. For example, besides image height and image width, the LLM specifies which other feature dimensions should be included and their appropriate sizes. These requirements extend the foundational constraints outlined in the protocol, ensuring that all specific instructions remain compliant with the protocal. The core mechanism of the technique lies in the interplay between the progenitor lesting protocol and the LLM. The progenitor lesting protocol establishes the minimal constraints shared across any task within the domain. Mcanwhile, the LLM addresses the task-specific features required for the modules.
After the specific unit tests constructed, the LLM generates multiple programs for producing synthetic data (Appendix C.1). This synthetic data serves as simulated input for the codes during unit testing. Every program for synthetic data is verified to adhere to the protocol. Hence, code is considered valid upon successfully processing syuthetic data from at least one program. Compared to actual data from the data preparation module, synthetic data can be deployed before the data preparation modules are generated and is independent of the specific data processing procedures.
5 Optimization of Machine Learning Programs
Algorithm Text-to-ML
1: Input: An autoregressive LLM fx and textual description of an MT. task
2: = and S = Search Space Generation (T, fx)
3: for in S do
Cº = Module Generation (M,0, // M represents the module be generated is for modeling end for
6: if based on S, the task is deep learning task then
7:
S ZC Proxies Evaluation (8, (C
8: end if
9: repeat
10: 0 Search Strategy (s, HI)
11: for module j in 0(j + M) do
12: if no prior attempt exists for generating Cý then
13:
C = Module Generation (j, fx
14:
end if end for
15:
16: if All syntactically valid modules for 0 exist then
17: Evaluate the ML program based on 0, with evaluation result as Rs, then append 0 and Rs to H
18: end if
19:
until resources are exhausted
20:
Output: The best ML program based on H
As Figure 8 illustrates, network of codes is generated to form multiple candidate MI. programs. AutoMI. then selects the high-performance program (i.e., pathway within the network), along with associated hyper- parameter values. Leveraging the external compatibility (Section 4), autoML is implemented as pre-written modules.
Many autoML methods architoct high-perfon reah-performacce deep learning models from scratch- as determining the number of convolutional layers and initiating training from zero. Such process is prohibitively expensive for regular users under low resource constraints (Öztürk et al., 2022). By comparison, the search space in study leverages the ability of LLMs to directly deploy pretrained models as candidate methods for modeling.
As a similar search space is explored in Özüürk et al. (2022), we employ a comparable set of hyperparameters as in Öztürk et al. (2022) (Appendix C.7). By contrast, the search space in our study includes more candidate pretrained models and both CV and NLP models. To manage the expanded scope, our autoML method incorporates four ZC proxies applicable with synthetic data for both CV and NLP (Appendix C.5), which significantly mitigates the cost associated with training and testing MT. programs (Figure 4).
12

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 2 outlines comprehensive methodology, from code generation to code optimization, illus- trated in Figure 3. In Algorithm 2, the I.M fx initially creates the search space & with Search Space. Generation (Appendix C.2), suggesting candidate algorithms for each module, such as candidate feature engincering techniques and ML modcls, and candidate hyperparameter values (line 2). Then for the ML model in each candidate solution of the search space 0, the LLM generates the corresponding code for the model C with Module Generation (Appendix C.4) (lines 3 to 5). Module Generation generates the code with the LLM based on Algorithm and Algorithm 3.
Notably, with the synthetic data from Algorithm 3, for deep learning tasks, the modeling modules are eval- uated with ZC Prozies Evaluation (Appendix C.6) before generating other modules, saving both generation and optimization costs (lines 6 to 8). ZC Prozies Evaluation removes a fraction of lower-performing candidate solutions from the search space with minimal cost.
Subsequently, Algorithm 2 itcratively examines the candidate solutions in the search space through trial- and-error. In each ileration, with the previous optimization history HI and search space 8 Search Stralegy (Appendix C.3), which is an optimization algorithm, selects promising solution for the ML task (line 10). Then, for the selected solution, Module Generation generates all modules except the modeling module (lines 11 to 15). Once all valid modules for the solution are created, Algorithm 2 assembles the complete ML program, evaluates it using the designated numerical metrics in T, and records the solution and its performance in the optimization history.
Ultimately, Algorithm 2 delivers the code for the ML task that exhibits the highest performance among all evaluated solutions. The code is executable program that seamlessly integrates the modules.
6 Experiments
6.1 Setup
To explore the limits of program synthesis for ML. in experiments, we consider the synthesis of the entire ML program, given textual descriptions of the ML task. For an ML task, let the set of valid programs (as defined in Section 6.3) for the entire ML program of the task be the desired sequences Ω. Given the textual description T, testing data Dies, and evaluation metrics E, then the objective to automatically generate the program Co that maximizes the cvaluation score basod on E and D
arg max E (Ce, Dies) s.t. C. E E 8
In other words, at first, precondition is that the generated programs are free of major syntax errors. Then among these programs, select the best program, by maximizing the performance score evaluated by the metrics. For other sequence generation methods, which are incompatible with autoML due to the lack of external compatibility (sce Section 4), we only evaluate their efficioncy in code generation (Section 6.2).
For our method, we further perform code optimization (Section 6.4). We generate each candidate program Co in an iterative approach through LLMs and autoML. Given an LLM fx, generation method V fx x T C₁ autoML method O, training data D and training procedure H Cj D x 0 Co. in the in iteration, our method generates C₁;
= C== (C,,, 0(i),, where C = V (AA,T,0())
Setup of datasets For the novel problem described above, curate suite of datasets in our experiments by carcfully addressing the relevant factors. For each of the traditional algorithms, CV and NLP, the (acliess(Appr (Appendix D.1) and datasets from the Kaggle competitions (Appendix D.2). The widely recognized datasets can be less biased; however, the training corpus of the L.LMs may encompass knowledge about these datascts. For the Kaggle datascts, we conduct a thorough review of all medal-awarding competitions held between October 2021 and July 2023 (Appendix D.2), considering the knowledge cutoff dates of the LLMs.
datasets include 2 widely-recognizedd datasets for traditional studies
13

---

Published in Transactions on Machine Learning Research (09/2024)
Setup of methods We use GPT-4-0613, GPT-3.5-turbo-061. (Achiam et al., 2023), and PaLM 2 (Anil et al., 2023) as LLMs fx- In Algorithm 1, we incorporate the self-reflection mechanism in Shinn et al. (2023)
in { (see Appendix A.1). We employ both random search and the hyperparameter optimization method BOHB (Falkner et al., 2018) as the algorithms for code optimization (see Appendix C.3). For ZC proxics, we utilize flops, parums, naswol, and synfow (Appendix C.5). flops and paramis are two simple yet powerful 7C proxies (Krishnakumar
al., 2022). While most ZC proxies are designed for CV tasks, Zhou al. (2022b) and Javaheripi et al. (2022) suggest the suitability of naswol and synfow for NLP tasks. Throughout the experiments, for the same dataset, provide the same task description T for each generation method (Appendix E.).
6.2 Generation of valid programs
As outlined in the task of repeated generations (Section 3), compare each sequence generation method across the datasets by repeatedly attempting to generate valid ML programs. This process is capped at a maximum of 100 attempts. If a method generates a valid program within these 100 attempts, the method is reset and tasked with generating another valid program until all 100 attempts are exhausted. For all methods, particularly Contextual Modular Generation, each sequence generation process (each time line 8 in Algorithm is executed) counts as an allempl.
In Table among the techniques, Reflexion and Contextual Modular Generation incorporate self-reflection mechanism, as outlined by Shinn (2023). This mechanism retains memories of previous trials, including leedback from the testing results of earlier ilerations (see Figure 3 and Algorithm 1). Such feedback is used as an imput sequence to provide insights for the generation process. However, we observe that these two methods often fall into repetitive cycles involving the same code-errors-feedbeee loop. To address this issuc, we force a memory reset if the method fails to produce a valid program after 10 altempts, thus mitigating the effects of poor initiations. For CoT, we set the number of input-output. examples as for GPT-4 and GPT-3.5 (Table 6) and 2 for PaLM (Table 7).
Table Mean number attempts per valid program with GPT-4. (1) CoT, Zero-shot, and Reflexion refer to Wei et al. (2022), Kojima et al. (2022), and Shinn al. (2023) respectively. (2) Self-Revision: the generation based the replacement of the contextual modular evaluations in Algorithm with the self-revision mechanism in Le et al. (2023). (3) for Contextual Modular Generation, w/o tests: Algorithm I by removing the unit tests (Figure 8 and Algorithm 3), w/o reflection: Algorithm 1 by removing the self- reflection mechanism in (4)), standard: Algorithm 1. (4) the datasets are divided into three groups in the order tasks for traditional algorithms, CV, and NLP. Details of cach dataset are available in Appondix D. (5)- -: the method is unable to generate any valid program for the dataset within 100 attempts.
Contextual Modular Generation (ours)
Datasets
CoT
Zero-shot
Reflexion
Self-Revision
w/o tests
w/o reflection
standard
Boston
10.2 + 2.2
3.7 + 0.8
4.0 + 0.6
4.1 + 1.0
3.9 = 0.7
3.4 + 0.6
Iris
2.1 + 0.5
3.5 + 0.6
6.1 + 0.8
3.7 + 1.1
3.6 + 0.7
3.4 - 0.6
3.2 : 0.4
Age
21.6 7.6
13.0 4.0
8.8 + 2.9
9.1 = 1.0
7.1
Default
34.3 9.0
14.7 + 5.6
13.1 + 1.6
10.6 + 3.2
11.6 + 2.1
CIFAR-10
4.5 : 1.1
6.3 + 1.8
4.3 + 0.5
13.6 + 1.6
6.9 = 1.6
6.7 + L.5
CIFAR-100
7.2 7 1.4
6.6 1.2
13.3 + 3.9
20.4 + 3.9
8.3 + 3.6
5.6 + 0.9
Whale
30.5 13.4
9.7 = 0.8
11.0 + 1.2
Strip
12.5 + 2.4
9.9 + 1.1
IMDb Reviews
6.0 1.5
44 1.1
6.2 + 1.4
10.5 + 1.6
11.1 1.1110 1.5
3.5 + 0.5
AG News
12.0 + 1.1
15.1 3.2
9.6 1.4
17.3 2.9
7.4 - 1.5
4.2 + 1.2
Eawm
12.1
10.7 + 2.6
Learning
20.0 + 2.8
12.6 + 1.6
12.8 + 1.6
In Table 1, we report the average number of attempts required to generate one valid ML program. Overall, our method outperforms other methods in 10 out of 12 of the datasets. For ML tasks with longer program lengths,
14

---

Published in Transactions on Machine Learning Research (09/2024)
especially the deep learning tasks and those from Kaggle, Contextual Modular Generation significantly outperforms other methods. In particular, prompting methods (CoT and Zero-Shot) are generally unable to address these tasks.
The performance gap between Contextual Modular Generation w/o reflection and Contextual Modular Gen- eration standard is narrow, and in some instances, the former even shows superior results. This could be attributed to the possibility that the lengthy memory involving previously generated ML programs is over- whelming and counterproductive. When we remove unit tests (Algorithm 3) from the process (Contextual Modular Generation w/o tests), there is a noticeable increase in the generation difficulty across datasets (Contextual Modular Generation w/o tests in Table 1), which suggests that the testing technique (Figure 8 and Algorithm 3) is necessary for Contextual Modular Generation in most situations.
We also conduct experiments with GPT-3.5 and PaLM 2 (Appendix F). For both LLMs, Text-to-ML out- performs other methods in 11 out of the 12 datasots. Overall, GPT-4 demonstrates the highest success rate among the LLMs
For the code generation process, analyze the mean number of iterations required to correct the errors (Table 8 in Appendix F). In the experiments, for code that fails unit tests, we calculate the average number of iterations needed to pass the tests again. We only count corrections before the forced memory reset. Overall, (1) data preparation modules are generally the most difficult to correct, averaging 5.43 iterations; (2) for GPT-4, with its larger imput token limit, the self-reflection mechanism shows the greatest benefit; (3) for LLMs with smaller input token limits (GPT-3.5 and PaLM 2), the memory from self-reflection can become unmanageable when single previous iteration exceeds the input token limit, resulting in automatic memory truncation.
6.3 Unit tests and false positives
In strict sense, errors in ML programs are nearly impossible to eliminate compared to traditional coding problems. Traditionally, software systems are constructed deductively by explicitly programming the rules that govern system behavior. In contrast, ML. rules are inferred inductively from the data (Braiek & Khomh, 2020). For example, the program may be error-free for one set of data points but buggy for another (Kim et al., 2019; Braiek & Khomh, 2020; Wang et al., 2021b). Moreover, many ML programs rely on high-level libraries like PyTorch and FensorFlow, and third-party libraries such as the Intel Math Kernel Library, which often lack comprehensive specifications or even detailed source code documentation (Braiek & Khomh, 2020).
Nevertheless, as in the practices of many human programmers, ML programs enrors and function effectively in most daily scenarios. In our study, we define valid programs as those that satisfy two criteria: (1) the program executes over the datasets in the task without interruptions from errors detectable by the execution environment (the Python interpreter), and (2) for programs that meet the first requirement, we manually ensure that the programs form a complete program for the specified MI, task. Specifically, we manually check whether the generated program includes complete ML pipeline, including data preparation, modcling, and post processing, as specified by the LLM in Search Space Generation in Algorithm 3 (see Appendix C.4).
still be free of major
Similar to traditional coding problems (Li et al., 2022), an ML program can be a false positive, appearing valid in unit tests but actually failing to meet the criteria mentioned above. In our study, each program is divided into modules, each verified by unit tests before assembly. However, modules verified as valid individually may not necessarily remain valid when combined. Therefore, the entire assembled program undergoes an additional simple test by the execution method automatically verify the first criterion.
In the experiment in Section 6.2, for our method (Contextual Modular Generation standard Table 1), we examine the false positive (FP) rates of the generated programs both before and after this additional testing step.
As Table 2 shows, the false positive rates for deep learning tasks are significantly higher than those of traditional algorithms, but they are still within a reasonable range. The false positive rates of the programs after meeting the first criterion are noticeably low, which suggests the reliability of ML programs that can be created automatically (by employing the execution step as additional testing step).
15

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Unit tests efficiency and false positive rates. (1) Traditional: tasks based on traditional MT. algorithms such as random forest and SVM. (2) FP rate w/o exccution: false positive rates of the programs post-verification but prior to lesting by the execution environment (3) FP rale w/o execution: false positive rates of the programs following both verification and testing by the execution environment.
Modality
ML programs verified by unit tests FP rate w/o execution
FP rate w execution
Traditional CV
86
7%
0%
79
34%
2%
NLP
99
29%
0%
Notably, for the experiments in Section 6.2, we do not insist on the compatibility of generated modules with any pre-written module, which is more lenient towards the other methods, particularly those without testing, such as CoT and Zero-Shot. Achieving such external compatibility (Section 4), is infeasible for methods that do not include testing rely traditional testing techniques. However, in method, the generated modules are also lested for compatibilily, which is crucial for performing consistent training and testing procedures and for uniform evaluation of these modules.
6.4 Improvements from optimization
Our method ensures that generated programs are compatible with a consistent evaluation procedure. Lever- aging this unique advantage, Text-to-ML automatically searches the programs generated by Contextual Modular Generation against metrics specified in the textual task descriptions. For each dataset, with GPT- 4, we generate codes up to 20 candidate algorithms, each for data preparation and modeling and at least 1 code for post processing, forming up 400 module combinations due the internal connectivity (Soction 4) provided by the testing technique (the network in Figure 8).
From these modules, we first randomly sample and evaluate 200 configurations of module combinations (pathways in the network in Figure 8) and their hyperparameters. We employ two optimization strategies: random search and BQHB (Falkner et al., 2018) (Appendix C.3). For both strategies, we examine the identified program at the cost of 25 and 100 full evaluations (i.e., the cost equivalent to the cost of fully training and testing 25 and 100 models). In the search, ZC proxies and BOHB prematurely terminate the evaluations for less promising configurations (Appendix C.3). Then, we rank the best program identified during the search among the 200 sampled configurations.
For deep learning lasks, we train each model on of 4 NVIDIA A100 GPUs. For sampling the configurations and random search, we train each model until it meets the early stopping criteria with palience of 3 and minimum delta of 0.0 (Prechelt, 2002).
The programs generated by traditional sequence generation methods are incompatible with autoML; however, these methods do allow for the program to be specified by L.MSs. As a simple baseline, we also evaluate the one ML program chosen by the LLM without any further optimization (No Search in Table 3). We utilize the prompt select the best configuration for the machine learning task.
Table 3 shows random search already yields significant improvements over the baseline, particularly with @25 In addition, ZC proxies effectively increase the search efficiency for both CV and NLP, improving the ranks of programs from the search with cos by 31 + 12% and the search with cost 100 by 9 + 3%.
7 Discussion
7.1 Synergy between LLMs and autoML
A major goal of autoML is to democratize ML by reducing the amount effort required by human experts (Thornton al., 2013; Wang et al., 2021a; Xin et al., 2021; Van der Blom al., 2021). AutoML has achieved substantial progress in recent years on an algorithm level (Zoph & Le, 2016; Liu et al., 2018). However, in practice, autoML. methods are gradually becoming tool for experts to potentially boost the performance
16

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Mean rank of the identified ML program 10 trials. (1) Traditional: tasks based on traditional ML algorithms such as random forest and SVM. (2)Rows with white and grey backgrounds represent scarch results with cost@25 and cost@100 respectively.
Search only
Search + ZC proxies
Modality
No Search
Random search
BOHB
Random search
BÓHB
10.8 + 7.2
10.8 + 9.3
N/A A
N/A
Traditional
97.4 + 41.0
5.0 + 2.8
4.2 = 1.9
N/A A
N/A
10.2 + 4.7
8.9 = 5.8
6.2 + 4.2
7.5 + 5.4
CV
75.2 + 34.9
4.3 + 1.7
4.0 +1.1
3.8 + 0.8
3.6 + 0.7
8.5 + 6.6
10.7 + 5.4
6.6 + 4.7
5.8 + 2.4
NLP
93.8 31.4
4.7 + 2.1
4.0 #1.5
4.2 + 0.7
3.8 + 1.0
of the workflow (Tornede et al., 2023; Van der Blom et al., 2021), due the following three issues: (1) most autoMT methods only select the solutions al the algorithm level. However, the programming required to implement the algorithms can demand substantial human effort and expertise. (2) for deep learning, many autoML methods select model architectures, such as the number of convolutional layers, from scratch and train the models anew, which is prohibitively expensive for regular users in practical scenarios under low resource constraints (Özüük et al., 2022). (3) many autoMT. methods do not offer a comprehensive and unified solution applicable across task modalities (e.g., traditional algorithms,, CV and NLP).
In our study, autoML numerically evaluates and selects ML programs generated by LLMs. Conversely, LLMs offer solution to all three issues of autoML. As demonstrated in Figures and 9 and discussed in Section 6, LLMs generate the programs for the entire ML workflow across task modalities, while directly utilizing pretrained models for deep learning. Without the code generation from LLMs, there would be no code for autoML to select and optimize in the first place. Our approach not only generates this code but also ensures that the code is natively compatible with optimization algorithms.
The objectives autoML, and prograin synthesis for MI. are closely aligned, with significant overlap in their purposes and methods (see Section 2.3). Nonetheless, strictly speaking, the goal of our study on program synthesis for ML, rather than autoML per se. This distinction ariscs because most autoML methods do not address ML tasks at the code level. In this context, autoML is employed step in our method (Figure 3).
For the gool of synthesizing ML programs, LLMs synergize with autoML: LLMs bridge the gap between algeredhamecenteeed auloML and auloML in practice, and auloML addresses the compler numerical evaluations of the programs generated by LLMs. In this synergy, LLMs contribute extensive coding knowledge embedded in their weights, facilitating code generation, while autoMI. brings robust mathematical rigor from numerical optimization, enabling code optimization.
7.2 Sequence generation and autonomous agent
From sequence generation perspective, our work explores the resolution of ML problems as a multi- step sequence to sequence (seq2seq) task (Sutskever et al., 2014). The input sequence is the textual task description. The output sequence is the corresponding optimized program for the entire ML program. Intermediary steps include sub-seq2seq tasks based on LLMs and optimization via autoML.
Our method, Te,t-to-ML, seamlessly integrates all intermediary steps into a single fully autonomous process (Figure and Algorithm 2). Future research may explore such multi-step seq2seq tasks as autonomous agents. Another interesting future exploration involves incorporating text-to-text or image-to-text methods to enhance task description generation, further automating the process.
17

---

Published in Transactions on Machine Learning Research (09/2024)
7.3 Limitations
While the complexity of Contextual Modular Generation could increase only linearly over the problem length (Theorem 3.8), implementing this framework demands considerable effort (Appendix C.I). The practical significance of the thcoretical linear complexity should be more extensively tested for other types of sequence generation lasks. Furthermore, our approach still struggles with highly challenging ML. lasks, particularly those that involve highly complex forms of input and output data (Appendix D). Whether addressing these tasks requires more efficient implementation of the framework or a transition to a different framework remains an open problem.
In particular, challenging tasks on the Kaggle platform are typically addressed through the collective efforts of thousands of participants. The participants can freely exchange ideas and solutions, greatly enhancing the efficiency of developing sigh-performance solutions. Additionally, the fighest-perfomiing publicly shared solution often becomes the default baseline, as many participants replicate it in their solutions. These practices make a mcaningful comparison between the solutions from human participants and automated solutions infeasible. In practice, most ML tasks are lackled by only one or a few human programmers..
Regardless, our work is not intended to outperform the collaborative efforts of thousands of competition participants with automatic solutions. Instead, our focus is reducing the amount of effort and knowledge required for writing ML programs in everyday scenarios. Qualitatively, our method achieves an umprece- dented degree of automation for ML tasks at the code level. Additionally, our work demonstrates significant performance improvements in the generated ML programs from the automatic optimization process.
To target performance enhancement in a competitive environment, a future direction could involve signif- icantly expanding the number of candidate solutions. For example, in our method, a modeling module might consist of a pretrained model and custom architecture designated by the LLMs (Appendix C.2, Figure 1 and Figure 9). Future research could extend Text-to-ML with additional autoML techniques, such as predictor-based methods (Dudziak et al., 2020) and weight-sharing methods (Pham et al., 2018), to search for the custom architecture. Whether and to what extent such a pursuit can coexist with applications for regular users under low resource constraints is a crucial consideration. Additionally, establishing a systematic benchmark to compare automated solutions with human- -ennanneeee ones is another significant challenge in this direction.
8 Conclusion
This study extends the scope of program synthesis for ML. By combining LLMs and autoML, we achieve the automatic generation and optimization of the complete ML programs algorithms, CV, and NLP. Experimentally, over 10 out of the 12 tasks, our generation framework Contextual Modular Generation outperforms existing methods, particularly for deep learning tasks and tasks outside the training corpus of the LLMs. Thcorctically, Contextual Modular Generation offers a lincar best-case complexity class for generation difficulty over sequence length.
12 tasks across traditional
The testing technique Constrained Generative Unit Testing is instrumental for the implementation of the framework. A notable advantage of our approach is the seamless compatibility between generated and pre-written modules, allowing for consistent numerical evaluation and native integration with optimization algorithms. This compatibility enables autoMM--driven numerical optimization to substantially improve program performance. In addition, we demonstrate the effectiveness of ZC proxies for pretrained models in both CV and NLP.
Broader Impact Statement
From the perspective of autoML, our approach aims to enhance the automation of the ML workflow, espe- cially for everyday scenarios with limited resources. Text-to-ML is a potential tool for democratizing ML by significantly reducing the required efforts, expertise, and computational resources. Users simply provide textual description of the task, and from there, the process is fully automatic, making ML in practice accessible to broader audience.
18

---

Published in Transactions on Machine Learning Research (09/2024)
However, the broader impacts of Text-to-ML must be carefully considered to ensure its responsible and ethical use. Our commitment to responsible AT is paramount, and we strongly urge the responsible use of automated tools for ML, including Text-to-ML.
Accessible Machine Learning: Benefits and Risks
By enabling a broader range individuals and organizations to leverage ML, the Text-to-ML framework can foster innovation, accelerate scientific discovery, and enhance decision-making across various fields. However, this increased accessibility also introduces potential risks, such as misuse by individuals with malicious intent or those lacking proper understanding of MT. principles. We list the guidelines for responsible use of Text- to-ML as below:
Ethical Education: Users should complete a general ethics course ML/ before accessing Text-to- ML. This course would cover topics including fairness, bias, and risks in ML, and provide examples of both responsible and irresponsible uses of ML technology.
Bias and Toxicity Data: Before applying Text-to-MI, to dataset, we recommend thoroughly checking for common biases such selection bias, label bias, and sampling bias to ensure the dataset represents diverse populations, avoids subjective influencos, and uses appropriate sampling methods (Mehrabi et al., 2021). Additionally, it is crucial to filter out toxicity in the data, including hate speech, offensive content, and misinformation.
Data Privacy: Users of Text-to-ML must adhere relevant data privacy laws and regulations, such as the General Data Protection Regulation (GDPR) (Voigt & Von dem Bussche, 2017) in the European Union and similar laws in other jurisdictions. For instance, using Texl-to-ML. on personal data without consent or scraping data from unauthorized sources would be strictly prohibited.
Application Restrictions: The of Texl-to-ML for irresponsible harmful applications should be explicitly prohibited. Examples include generating misleading or harmful content and developing surveillance systems that infringe on individual privacy rights.
Impact Assessment: Users should conduct an impact assessment for their ML programs using Text- to-ML, outlining the potential socictal and ethical implications. For example, project aimed predicting criminal behavior would need to consider the risks of reinforcing biases and stigmalizing individuals or communities.
Transparency and Accountability in the Automation of Machine Learning
Automating the synthesis of ML programs streamlines the creation of ML models, but it can also obscure the underlying decision-making process. This lack of transparency is particularly concerning in critical fields such as healthcare, finance, criminal justice, job hiring, and college admissions. For instance, in finance, transparency is necessary to ensure fair lending practices and avoid biases in credit scoring. In college admissions, understanding the rationale behind acceptance or rejection decisions is vital for maintaining fairness and avoiding biases that could disadvantage certain groups of applicants. We outline our measures and guidelines increase transparency and accountability for Text-toMML as follows:
Logging the Automatic Process: The implementation of TeX in our study features a com- prehensive logger that records the entire process of synthesizing the ML program. This includes decisions regarding the search space construction (Appendix C.2), rationales behind the sugges- tions of candidate solutions (Figure 9), successfully generated modules, and the performance of each module combination being evaluated. We strongly recommend that other implementations of the Text-to-ML framework also incorporate such a logger. This detailed logging information helps users trace outcomes back to specific decisions and increases accountability.
Tracing the Prompts and Responses of LLMs: Text-to-ML, involves a set of dynamically constructed prompts (Appendix E) in intermediate steps through the instruction constructors (($', : $ $2....,00 &'} in Algorithm 1). The implementation of Text-to-MT. in our study transparently displays the input
19

---

Published in Transactions on Machine Learning Research (09/2024)
and output sequences for each inquiry involving LLMs. This history of prompts and responses enables users to understand the decisions made by the LLMs in each step of the process, enhancing transparency and traceability.
Interpretable Models: We advocate for the use of interpretable models and techniques that allow users understand how specific outcomes are derived. This includes the incorporation of explainable AT (XAT) methods (Hoffman et al., 2018) within the Text-to-MT, framework to provide insights into the decision-making process.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXie:2303.08714 2023.
Altavish.
Boston housing dataset.
boston-housing dataset, basson-ast, 2023.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Kloin. Neural module networks. In Proceedings of the IEEE conference on compuler vision and poltern recognilion, pp. 39 48, 2016.
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Newral Information Processing Systems, 35:38546-38556, 2022.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. orXiv preprinl arXiv:2305. 10403, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. ar preprint XrXi::2108.00 07732, 2021.
James Bergstra and Yoshua Bengio. Random search for ppperpparreete optimization. Journal of machine learning research, 13(,), 2012.
Houssem Ben Braiek and Foutse Khomh. On testing machine learning programs. Journal of Systems and Software, 164:110542, 2020.
José P Cambroncro and Martin C Rinard. Al: autogenorating supervised learning programs. Proceedings of the ACM on Programming Languages, ((OOPSLA):1-25 2019.
Angelica Chen, David M Dohan, and David R So. Evoprompting: Language models for code-level neural architecture search. arXiv preprint ar Xiv:23300 14838, 2023a.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. ar preprint 0rXiv:2207.1039 2022.
Mark Chen, Jerry Tworek, Hcewoo Jun, Qiming Yuan, Henrique Ponde de Olivcira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiu preprint ar "xXix:2107.03374 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self- debug. arXiv preprint arXiv:2304.05122 2023b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichrr Nakano, al. Training verifiers solve math word problems. arXiv preprint or Xin:21 10. 14168, 2021.
20

---

Published in Transactions on Machine Learning Research (09/2024)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl ar N810:111004440 2018.
Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction-basod nas using gens. Advances in Newral Information Processing Systems, 33:10480-10490, 2020.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In International conference on machine learning, pp- 1437-1446. PMLR, 2018.
Jiazhen Gu, Xuchuan Luo, Yangfan Zhou, and Xin Wang. Muffin: Testing deep learning libraries via neural architecture fuzzing. In Proceedings of the 44th International Conference on Software Engineering, pp. 1418 1418-1430, 2022.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953-14962. 2023.
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metries for explainable ai: Challenges and prospects. arXiv preprint arXi::1812.00 04608, 2018.
Noah Hollmann, Samuel Müller, and Frank Hutter. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. In Thirty-seventh Conference on Newral Information Processing Systems, 2023.
Mojan Javaheripi, Gustavo de Rosa, Subhabrata Mukherjee, Shital Shah, Tomasz Religa, Caio Cesar Teodoro Mendes, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. Litetransformerrearch: Training-free neural architecture search for efficient language modcls. Advances in Neural Information Processing Systems, 35:24254-24266 2022.
Kaggle. Kaggle- 1lm science exam competition data. https://ww..kaggleeeu www..//ggg....mmooomttttiiiooooe
kaggle- 1ew--lllmmmmmmeee crrienneeeeeamddeee 2023.
Kaggle and Mayo Clinic. Mayo clinic strip ai competition data. attps://wwz www.ww.......................... com/competiitions mayo--lliii-ssriippaaatata 2023.
Kaggle and American Express. American express default prediction competition data. https://www. 2023.
Kaggle and Happywhale. Happywhale whale and dolphin identification competition data. https://www. 2023.
kaggle. com/ httprrtoooooppaaaalalaaaaaeeeeea
Kaggle and LLC InVitro Cell Research. ler - identifying age-related conditions competition data. https:
//www.kagsa kaggle. wwt.cccccccc/ccccceeeo
hrtp::/aooaaeettttoooooooooooaaa 2023.
Kaggle and The Learning Agency Lab. Feedback prize - english language learning competition data. https: 2023.
//www. kaggle
Jinhan Kim, Robert Feldt, and Shin Yoo. Guiding deep learning system testing using surprise adequacy. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 1039 1049. IEEE, 2019.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22211 2022.
Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, and Frank Hutter. Nas-bench- suite-zero: Accelerating research on zero cost proxies. Advances in Neural Information Processing Systems, 35:28037-2800 2022.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Alex Krizhevsky et al. Cifar-10 10 and cifar-100 datasets. http:://www..c..ooootooodedd//// html, 2023.
Lakshmi25npathi. Indb dataset of 50k movie reviews. lakshmi Af -eve-eeeeee 2023.
Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. Codechain: Towards modular code generation through chain of self-revisions with representative sub-modules, arXiv preprint arXiv:2310.08002 2023.
UCI Machine Learning. Iris species. attps://wwy https://ww..kaggle wtt.eee.ccmmaaaaaaaeeeeeeeeeeeee haaag//ee.oooaaatattsssaccc////a
2023.
Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei- Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-angmentedd code emulator. arXiv preprint ar Xiv:2311 2..91...00 044744 2023.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit- approach to hyperparameter optimization. Journal of Machine Learning Research, 18 (185):1-52, 2018.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624) 1097, 2022.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.00 09055, 2018.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song- Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXio:2304. 09842, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. arXiv preprint ar Xrv:23303. 7651, 2023.
Zohar Manna and Richard Waldinger. Toward automatic program synthesis. Communicctiinss of the ACM, 14(3):151-16) 165, 1971.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprinl arXiv:2202. 00..2220211877, 12837, 2022.
Glenford J Myers, Tom Badgett, Todd M. Thomas, and Corey Sandler. The art of software testing, volume 2. Wiley Online Library, 2004.
Benjamin Newman, John Hewitt, Percy Liang, and Christopher D Manning. The eos decision and length extrapolation. arXiv preprint arXi::2010.07174 2020.
Ekrem Öztürk, Fabio Ferreira, Hadi Jomaa, Lars Schmidt-Thieme, Josif Grabocka, and Frank Hutter. Zero- shot automl with pretrained models. In International Conference on Machine Learning, pp. 17138 17155. PMLR, 2022.
Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In proceedings of the 26th Symposium on Operating Systems Principles, pp. 1 1-18, 2017.
22

---

Published in Transactions on Machine Learning Research (09/2024)
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference machine learning, pp. 4095-4104. PMLR, 2018.
Lutz Prechelt. Early stopping-bin but when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer, 2002.
Aman Anand Rai. Ag news classification dataset. httpp://ww...gggg...caa/aattttts www.ess-lassifiaatttaeeeeae
2023.
Esteban Real, Chen Liang, David So, and Quoe Le. Automl-zero: Evolving machine learning algorithms from scratch. In International conference on machine learning, pp. 8007-8019. PMLR, 2020.
Ripon K Saha, Akira Ura, Sonal Mahajan, Chenguang Zhu, Linyi Li, Yang Hu, Hiroaki Yoshida, Sarfraz Khurshid, and Mukul R Prasad. Sapientml: synthesizing machine learning pipelines by learning from human-wrilen solutions. In Proceedings of the 44th International Conference on Software Engineering, pp. 1932-1944 2022.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpl: Solving ai tasks with chatgpt and its friends in huggingface, arXiv preprint arXiv:2303 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seoenth Conference Newral Information Processing Systems, 2023.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint XrXi::2303.0 08128, 2023.
llya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33: 6377-6389, 2020.
Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Browe Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery und data mining, pp. 847 855, 2013.
Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. Decptest: Automated testing of decp-neural- network-driven autonomous cars. In Proceedings of the 40th international conference on sofbware engi- neering, pp. 303 314, 2018
Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne T'heodorakopoulos, Tanja Tornede, Henning Wachsmuth, et al. Automl in the age of large language models: Current challenges, future opportunities and risks. arXio preprint arXiv:2306. 08107, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anma Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXi::200 13971, 2023.
Koen Van der Blom, Alex Serban, Holger Hoos, and Joost Visser. Automl adoption in ml software. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021.
Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10/3152666) 2017.
Dakuo Wang. Q Vera Liao, Yunfong Zhang, Udayan Khurana, Horst Samulowitz, Soya Park, Michael Muller, and Lisa Amini. How much automation does a data scientist want? arXiv preprint 201.02200. 2021a.
23

---

Published in Transactions on Machine Learning Research (09/2024)
Song Wang, Nishtha Shrestha, Abarna Kucheri Subburaman, Junjic Wang, Moshi Wei, and Nachiappan Nagappan. Automatic unit test generation for machine learning libraries: How far are we? In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 1548 1560. IEEE, 2021b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisenncy improves chain of thought reasoning in language models. arXiv preprint MXXi:2203.1117:: 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural In Information Processing Systems, 1124822-248877 2022.
Doris Xin, Eva Yiwei Wu, Doris Jung-Lin Lee, Niloufar Salchi, and Aditya Parameswaran. Whither autom?? understanding the role of automation in machine learning workflows. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-16, 2021.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint orXic:2210003666 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan Tree of thoughts: Deliberate problem solving with large language models. ai preprint ar 2rXiv:2305.1000 2023.
Caiyang Yu, Xianggen Liu, Chenwei Tang, Wentao Feng, and Jiancheng Lv. Gpt-nas: Neural architecture search with the generative pre-trained model. arXiv preprinl arXim:2305.05355 2023.
Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. Micopilot: Unleashing the power of large language models in solving machine learning tasks. arXiv preprint arXi:2204.14977 2023a.
Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. Automl-gpt: Automatic machine learning with gpt. arXiv preprint ar Xix::235500224444 2023b.
Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanic. Can gpt-4 perform neural architecture search? arXiv preprint Xrv::22004. 10970, 2023.
Denny Zhou, Nathanael Schärli, Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le, et al. Least-to-most prompting enables complex reasoning in large language models. orXiv preprint arxi::2205.0 10625, 2022a.
Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free transformer architecture search. In Proceedings of the IEEE/CV /CVF Conference on Computer Vision and Pattern Recognition, pp. 10894 10903, 2022b.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 2016.
A More Related Works
A.1 Large Language Models
Iterative methods repeatedly generate the output sequences in order to improve the output sequence based on a sequence evaluation procedure. Let the initial input sequence be Xo, the sequence genera- tion process be fx, and the evaluation procedure be E. Then the output sequence in the inh. ileration is Y = (2, (.Y).), itcration by integrating the output sequences, their evaluations, and the initial input sequence from preced- ing iterations. Recent works along this line include Yao al. (2022); Madaan et al. (2023); Cobbe et al. (2021); Shinn et al. (2023); Chen et al. (2023b). In reinforcement learning context, each output sequence
where 4, represents a mechanism that assembles the input sequence for the
24

---

Published in Transactions on Machine Learning Research (09/2024)
represents action (Yao et al., 2022; Shinn et al., 2023). The implementation of our method incorporates the self-reflection mechanism in Shinn et al. (2023). More specifically, Φ; in our implementation assembles input sequence from code execution results and an explanation of the results.
Prompting methods improves the output sequence Y by transforming the input sequence X. Let y represents the prompt engineering process, then p(]
(x))). Notable prompting methods include in-context learning (Wei et al., 2022; Zhou et al., 2022a; Wang al., 2022; Min et al., 2022; Yao et al., 2023; Li et al., 2023) and zero-shot prompting (Kojima et al., 2022). Notably, although prompting methods break the problem into intermediate steps, such Wei et al. (2022) and Li et al. (2023), these methods perform the steps in one sequence generation process, differing from our definition of compositional reasoning (see Section 2.1).
A.2 Automated Machine learning
Table compares our method with existing methods that employ LLMs for autoML. A detailed analysis of recent advancements in this domain can be found in Tornede et al. (2023). To the best of our knowledge, Text-to-MI is the first method that automatically generates the optimized program of the entire ML program across task modalities.
While recent autoML techniques also include predictor-based methods (Dudziak et al., 2020) and weight- sharing methods (Pham et al., 2018), these techniques are less suitable for the desiderata our study: applications in low resource constraints scenarios and search space with diverse pretrained deep learning models.
Table 4: Comparison with existing methods in terms of qualitative features. (1) Code generation: whether the mothod automatically generates programs (2) Deep lcarning: whether the method automatically gener- ales (part of) programs for deep learning tasks. (3) Data preparation: whether the method automatically chooses the strategy for data preparation (4) Modeling: whether the method automatically chooses the strat- egy for modeling (5) Optimization: whether the method optimizes the programs (components). * Although the method can automatically choose the strategy for modeling, the method does not automatically generate the code for modeling. ** The method uses the LLM as pseudo-optinnzation model instead of a dedicated optimization algorithm (e.g., evolutionary computation and reinforcement learning.)
Code generation
Deep learning
Data
Methods
Modeling
Optimization
preparation
GPT-NAS (Yu et al., 2023)
MT.Copilot (Zhang et al., 2023a)
*
GENIUS (Zheng et al., 2023)
*
**
Evoprompting (Chen et al., 2023a)
V
x*
CAAFE (Hollmann al., 2023)
V
ML (Saha et al., 2022)
d
AutoMI.-GP GPT (Zhang al., 2023b)
/
HuggingGPT (Shen et al., 2023)
V
Text-to-ML
B Proofs of Theorems
We start by connecting the concept of token-level complexity (Definition 3.2) with the success rate of gen- cration.
In Definition 3.2,
25

---

Published in Transactions on Machine Learning Research (09/2024)
Px (i)
=mx(u. E Oalven 1) ((() & -1(()))))))nm px((1 E On(ves, 1)
PalWUe (n((), -)(0)--)nnn--))
Since (ys E On(ve: 1)) and ((t) are mutually exclusive and (y<s Oz(0, 1) and my - 1) are mutually exclusive,
Px (1)
1
Pa((Ue E
E 1)) (otp::/tliiiloooeeeeoaaaa0o0---0 + PA ((v E Oalve, 1)) n (nii - 1) n x
px((()) n (lver E (0(() ((((-)-- +x (
By summing over Oa(ves; 1) and On(() i 1),
Px (1)
Ev EU Vel nX) - - x
Σ
Pa(i== y Nyei = U nX)
Σ
Pa(u= of (
Σ
v2. (0((),))))
For each i, there are three mutually exclusive events in the sequence generation. If the sequence generation is ongoing (and might or might not terminate at the &s token),
1x
Σ
p = -)(-))
Σ (0((()))))
pa(C(i) = U2 0X)
3f (vcu.))
= =(x((()nm(-i))xx) =0
If the sequence generation terminates at the (i 1)xr token,
1)) Σ Σ
Px(Ui Vi Vei = Ves nX)
E0,(), 1-1) ()
Σ
px(u- - vi nn X) = (x((() n nnli x)
If the sequence generation terminates before the (i 1)tk. token,
26

---

Published in Transactions on Machine Learning Research (09/2024)
Σ
Pxly<r Ver nx)=
E
E
Pa(u - y Nyci yez n
M2 (0,().)))))
12, V2,, vi (Qn(ve.l)
E
Pa(M vi nn(i - 1)nx)=
Σ
pa(S(i) y XX =0
y (
Thus,
Px (i)
20n = V/ Nyci = V2, nX)
(A( = V2, nx)
1) (yki v22 nx) + Px(S(t) nn 1 1) (i nx)
Ey (x)
For simplicity of notation,
E (u
Ex = nx) + n n))
==
px(n(i-1)na - 1X)
x(1) On(()ii1),, v11) On(vei; 1), T(i)= = Σ =
22()x(1
Then we have
II Px
(A(
1=1
Σ
Σ
Σ
px(ni = vi Nike= = (x) (2)) +
Ex(i) v7 Ex()
Σ Σ
pA(i1= = N = (x) + x
Vea1 Ex(1) (FE
Σ
Σ
P2(y2 = 1/ Nc2 V22 0 X)) +
X...X
Enc))(2) (2)
Σ
Σ
Px(UI of Nyer = nx
En() ()EN())
Let each y2, in x(i) be Uein (1 5 No) and each of in vii) be vi. (1 E m S M2), then
27

---

Published in Transactions on Machine Learning Research (09/2024)
Í IIxx() ==1
(Px(1n = vi, Oye1 = V22,, nXx +... +.....pp.e Px -V Aye1 = VES1, (
+...
pa(ui = vi.1 Mvvl= = VELN 0X) ......r66. = Vi.M OvKl VEINN (x)
+ 1)
(- (pxxex = V2,1 Nye2 = V22,11 nX) +... + px(22 V2,M3 Ny02 = V22,11 nx)
+...
pa(u2 v2,1 Ny²22 V222N, 0X) Pa(u2 = V2.M2 OyE2 V22,N2 (x)) +1,
PA(U1 (- (Px((11= vi.1 ( - = nx) + +p(y1 = Vi,M, Ny<1= ver,a nx) +...
palus vt. nyel VEr,,, nx) + + -Vi,M, NyK1 VEI,N, nx)) +1) Pa(ii of UC(1 1 (o2) (n( nx) x ppu = f\ uc(1) (0 un(( 11 PA(U1 vř U(()) (22 Un) nx) x... Px(II o7 UC() (m( +
Pa(y1 = v Uc(1) (v 0 (()) n X) x.. x px(yi -//a UC(T) (32 Un(I - 1)) nx))
[0
(x vt Uc(1) (022 unii nx)
1=
[0]
Σ
(Y = X(
1=1
P(Y EΩ)
(1)
Theorem. (Theorem 3.5) For any sequence generation process, if there exists a step in the process that generates the entire output sequence and Assumption 3.4 holds for the step, then the best case complexity of [E=P))(()] of the process is O(erp(T)).
Proof. If Assumption 3.4, then for every i, there exists an Eaaz < 1 such that px (x) S emar iisis I and I=.).
In the best case, px (1) emaz (Isis) 1). Then by equation (1), P(Y x) e max and E(G) Emaz" Therefore, the best case complexity of (EmP[G((T) is O(exp(T)).
In addition, if for a given e, ex cl-((e) is known, then
R2 BA (1) x (2) x.. X Px (1) < (2) I Px P (1) P
Then, in the best case, P(Y E Q) = 22 and Er-r xx))
28

---

Published in Transactions on Machine Learning Research (09/2024)
Theorem. (Theorem 3.8) If Assumption 3.7 holds, then for Contextual Modular Generation defined in Algorithm 1, the best case complexity of Ez-r(G(T) is O(T).
Proof. For contextual modular generation, for each sequence generation of yı, let the token-level complexity of the generation be
(t)) (c) - - 11SISTAAA RA (1) {)
XX.Y
If Assumption 3.7 holds, then in the best case, for each YJ, for each i, exists cmin such that cmnn S I and I D). Let Z represent the number of tokens in module j (Z) S M, where M is the maximum number of tokens across all modules).
Px,y (f) (1 S i J,1 Sis
R.A
Then by equation (1), in the best case, for each Y2, P(Y E = Modular Generation, each module is iteratively generated until being evaluated as valid. Therefore, we have
= min 2 (1أ In Contextual
++] Σ
E PGG()] < E (T)
mmm
M
j=1
Thus, the best case complexity of Ez- is O(T).
C Method Details
C.1 Unit Testing Details
Regardless of the specific task and modules, the generated modules should be compatible with pre-written set of modules for evaluation, training, and testing. Progenitor testing protocols first ensure the external connectivity by testing the form of the input and output data in the ML task, since all modules, including data preparation, modeling, post processing, optimizer, and loss functions, center around the input and output data in the ML task.
In this context, distinguishing between the input/output in the ML task and for the modules is crucial. For instance, the input to the data preparation module consists of raw data, while its oulput comprises the processed input data (features) and output data (targets) in the MI. task. The unit tests described in Algorithm 3 focus on the ML task's input and output data by scrutinizing the output produced by each module when provided with its respective input. For all modules except data preparation, synthetic input/output. data are used for testing in the MT, task.
Besides the requirements for external connectivity, internal connectivity demands additional tests for each specific task. Therefore, the LLM in Algorithm 3 first generates a plan specifying the suitable form and content of the input and output data for the task. The specified form of the input and output data should be a subset of the form in the progenitor testing protocols. The progenitor testing protocols also ensure that the synthetic data meet the form specification in the plan.
In our experiments, utilize the Python libraries Pytorch, PyTorch Lightning, and Transformers for deep learning tasks and Scikit-learn, XGBoost, CatBoost, and LightGBM for tasks involving tabular data. For deep learning, the specifications in the plan include (1) the tensor lypes for each tensor. (2) the dimensionality of each tensor. (3) the meaning of each dimension. (4) the size of each dimension. (5) isomorphic dimensions:
dimensions that should share the same size over different settings of the data preparation process and modeling configuration. (6) suitable ranges of the size for each dimension. For tabular data tasks, the specifications limited to the dimensionality of arrays for each of input and output in the MT. task.
29

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 3 Constrained Generative Unit Testing
Input: a progenitor testing protocol Φ, an autoregressive LLM fx, the textual description of an ML
task T, specification of the targeted module j, and the number of versions of synthetic data N
R False
repeat
4:
Given T and Q, fa devises a plan P that details the specific features and data flow of the modules tailored for the task.
Verify P with & until P allowed by B
With P and P, construct the specific unit tests for the ML task {E*}
8: for = 1 to N do
9:
repeat
10:
Given P, fa generates program Da, which produces synthetic input and output data for the ML
task
11:
Verify D, with the test for synthetic data in {E'}
12: until Dn passes the test in {E*}
13: end for
14: Given T and j, fx generates the program Cи for
15: for n = 1 to N do
16: Verify Cý with D, and the test for the module in {E))
17: if Cj passes the test then
18:
R=True
19: break
20: end if
21: end for
22: Output: R
C.2 Search Space Generation
We leverage the LLM to generate the search space given the textual task description. Initially, the LLM cat- egorizes the lask among tabular data lasks, CV lasks, and NLP tasks. further distinguishes among various task categories among the options: binary classification, multi-closs classification, multi-label classification, single-output regression, multi-output regression, scqueenceto--sequene and others. For classification tasks, the LLM also determines the most appropriate output format, choosing between integer representation of class labels and a probability representation of class labels.
Following the classifications, the LLM recommends candidate methods for each module and identifies suitable hyperparameter ranges. terms of model recommendations, a single suggestion may encompass multiple models. Figure 9. shows an example of an I.MM-generated model recommendation for NLP tasks:
Search Space Generation in Algorithm 2 is intertwined with the generation of the plan for high-level require- ments of the modules in Algorithm 3. With the classification of the task and the textual task description, the LLM also decides the suitable form and content of the input and output data in the task. This decision not only guides the search space definition but also informs the planning for unit tests, as constraints on the form of input and output data inherently limit the program space.
C.3 Search Strategy
In our study, we employ two scarch strategies: random scarch and BOHB (Falkner et al., 2018). Random search is a powerful baseline in hyperparameter optimization due to its wide applicability and high search efficiency for tasks with low effective dimensionality (Bergstra & Bengio, 2012). BOHB combines Bayesian Optimization (BO) and Hyperband (HB) (Li al., 2018) to optimize hyperparameters efficiently. BOHB inherits HB's robustness and flexibility, and BO's ability to ensure strong performance. BOHB starts by rapidly identifying promising configurations like HB and refines them using BO to achieve optimal solutions
30

---

Published in Transactions on Machine Learning Research (09/2024)
I{
"BERT": "Pretrained BERT model can be used for this NLP task as it is specifically designed for natural language processing tasks."
"GPT-?": "GPT-2 model can also he used for this NI task as it is known for generating coherent text and can handle long-range dependencies."
),
"CLMo": "CLMo model can useful this task as captures contextuelized word representaticns, which can be valuable for understanding the ease of reading."
"RFRT+I inear Regression": {
"reason": "The BERT model can extract useful features from the input data, which can then be used as input to linear regression model for predicting the ease of reading.", "BERT": "backbone",
"Linear Regression"; "head"
b.
"ROBERTA+MLP": "Using RoBERTa as feature extractor with an NLP can an effective way predict the ease of reading."
}.
"Transformer Encoder+<Transformer Decoder+Linear Regression':
"reason": "The Transformer Encoder can extract features from the input data, which can then be fed into the Transformer Decoder to generate predictions, The predictions then be refined using linear regression model.".
"Transformer Encoder": "encoder". "Transformer Decoder": "decoder", "Linear Ragression": "head"
}.
"BERT+GPT-2+Linea" Regression": {
"reason": "The combination of BERT and GPT-2 can leverage strengths of both models extracting features and generating text, respectively. These features and generated text can then be used as input to linear regression model for predicting the ease of reading.", "BERT": "backbone",
"GPT-2" "framework", "Linear Regression": "head"
}]
Figure I.L.-generated model suggestions.
faster. It's designed to be simple, computationally efficient, and adaptable to various tasks. For BOHB, use the default hyperparameters in Falkner et al. (2018) and set 30 epochs as the maximum budget for deep learning tasks.
C.4 Module Generation
In our study, for each ML task, we generate three modules: data preparation, modeling, and post processing. Fach module is generated in step (j) of Algorithm 1. First, the choice specifications (e.g., choice of feature
31

---

Published in Transactions on Machine Learning Research (09/2024)
engincering method choice of models) from Search Space Generation are given input sequence X Subsequently, the LLM fA generates the module yJ. Then the generated module YJ is assessed with the contextual modular evaluation EE,, which is the unit test for the module generated from Algorithm 3. After the tests, Module Generation employs the self-reflection strategy from Shinn et al. (2023) to improve the generalion efficiency. The input sequence assembler Φ instructs the LLM JA to reflect on the Teedback from the tests and then assemble previously generated codes, the tests feedback, and the reflection into the following form for the next iteration:
(Choice specification for the module)
(Previously generated code) (Unit test feedback)
(Reflection results)
(Instruction for generating modified code)
C.5 ZC proxies details
ZC proxics are an autoML technique that allows for the prediction of fully trained neural network's performance with small computational expenses. In this study, we utilize four ZC proxies: flops, params, naswot, and synflow. The term flops stands for Floating Point Operations Per Second, indicating the computational power of the network. Params denotes the total number of parameters within the network, reflecting its complexity. Although flops and params are fundamental concepts within deep learning, they have been validated as powerful baseline 7C proxies (Krishnakumar et al., 2022).
naswot (Mellor et al., 2021) leverages the overlap of activations between data points in untrained networks to derive a measure indicative of the network's future trained performance. This approach allows for rapid and efficient searching of powerful networks without the extensive computational costs typically associated with training each candidato network during the search process. Originally designod for network pruning, synflow identifies sparse trainable subnetworks within untrained neural works The key principle of synflow is to maintain a balance of synaptic strengths through the network, thus preserving the network's ability to be trained post-pruning.
All ZC proxies, except for params, require data to be passed to the models for evaluation. To investigale the 3 ZC proxies' applicability with synthetic data, we first generate the program for producing synthetic data, then sample 50 versions of synthotic data. based on different random seeds with a fixed shape of the data. We observe that performance variations reported by each proxy are minimal. Notably, synflow exhibits the most significant variation, with a Mean Relative Deviation (MRD) of 0.76%, indicating the suitability of the ZC proxies for synthetic data.
C.6 ZC Proxies Evaluation
For deep learning tasks, after the modeling modules specified by Search Stralegy are generated, the modules are evaluated with ZC proxies. ZC Prozies Evaluation ranks each module based on the average relative ranks over the 7C proxies. Subsequently, the modules that fall below a certain threshold, defined by a fraction denoted as μ are removed from the search space. In our experiments, we set μ 0.5.
C.7 Hyperparameters for finetuning
For deep learning, our research focuses on an unusual search space with pretrained models al the architect ural level. Öztürk et al. (2022) also explored a similar search space. Consequently, we utilize a comparable set of hyperparameters for fine-tuning the models (Table 5). To tailor this approach our specific requirements, we have reduced the minimum batch size from 16 to accommodate the varying GPU memory demands across different tasks.
32

---

Published in Transactions on Machine Learning Research (09/2024)
Table 5: Finetuning strategy
Name
Type, Scale
Range
Batch size
int, log
12 64
Learning rate
float, log
10-6,10 10
Weight decay
float, log
[0 1000,10
Momentum
float
00.01, 0.99
Optimizer
cat
(SGD, Adam, AdamW)
Scheduler cat
(plateau, cosine}
D Datasets Details
D.1 Widele-recoonizeed datasets
Boston The Boston dataset is a widely used regression dataset that contains information about housing in the suburbs of Boston. It consists of 506 samples, each representing a suburb, with 13 features describing various aspects of the housing environment, such as crime rate, average number of rooms, and accessibility to highways. The target variable is the median value of owner-occupied homes in thousands of dollars. This datasot is often usod for regression tasks to predict housing prices basod on the given fcatures. We obtain the files of the dataset from Altavish (2023).
Iris The Iris dataset is a classic classification dataset used to demonstrate various machine learning algorithms and techniques. It contains 150 samples of iris flowers from three different species: sctosa, versicolor, and virginica. There are four features for each flower: sepal length, sepal width, petal length, and petal width. The goal is to classify the flowers into their respective species based on these features, making it a popular dataset for teaching and practicing classification algorithms. We obtain the files of the dataset from Learning (2023).
CIFAR-10 The CIFAR-10 dataset is a well-known benchmark for image classification tasks. It consists of 60,000 32x32 color images across 10 different classes, with 6,000 images per class. The classes include objects like airplanes, cars, birds, cats, and more. The dataset is divided into a training set of 50,000 images and a test set of 10,000 images, making it suitable for evaluating the performance of various image classification algorithms. We obtain the files of the dataset from Krizhevsky et al. (2023).
CIFAR-100 Similar to CTFAR-10, the CIFAR-100 dataset also used for image classification tasks. How- ever, CIFAR-100 is more challenging further grouped into 20 superclasses, making it more fine-grained classification task. The dataset is de- signed to test. the model's ability to handle larger number of classes and fine distinctions between them. We obtain the files of the dataset from Krizhevsky et al. (2023).
contains 100 classes, each with 600 images. These classes are
IMDb Reviews The IMDB dataset is often used for sentiment analysis and text classification tasks. It consists of movie reviews labeled with sentiment labels (positive or negative). The reviews are represented as text and the goal to predict the sentiment of a given review. This datasct is useful for exploring natural language processing techniques and building models that can understand and classify textual data based on sentiment. We obtain the files of the dataset from Lakshmi25npathi (2023).
AG News The AG News datasct is commonly used for text classification tasks, particularly for news calegorization. It contains news articles from four different categories: World, Sports, Business, and Sci- Tace/Technolog. The dataset is often used to develop models that can classify news articles into their respective catogorics based on their content. It's relatively simple text classification task that serves as a good starting point for experimenting with various natural language processing algorithms. We obtain the files of the dataset from Rai (2023).
33

---

Published in Transactions on Machine Learning Research (09/2024)
D.2 Kaggle datasets
To choose the Kaggle datasets for our study, survey all Kaggle datasets that satisfy the following require- ments:
The datasct is from competitions that award medals, representing the highest level of difficulty available on the Kaggle platform.
The dataset is about supervised MT. tasks for tabular data, CV, or NLP.
The dataset is publicly available (upon accepting the rules of the competition) and the dataset contains target labels.
The dataset is from competitions held between October 2021 and July 2023. The knowledge cutoff dates of the LLMs are September 2021 for GPT-4 and GPT-3.5 and mid-2021 for PaLM 2.
We have identified a total of datasets for tabular data tasks, datasets for NLP tasks, and 13 datasets for CV tasks. From these, we chose the 2 simplest datasets for each category, based on the diversity of file types present in the datasets and the task types. For CV, tasks that involve image segmentation, object detection, or 3D reconstruction are significantly more demanding than image classification tasks. For NLP, tasks that require named entity recognition and sequence matching present greater challenges compared to sequence regression and sequence classification tasks. We find that all the methods we tested in our experiments, including Text-to-ML, are unable to successfully generate even a single program for the most challenging ones the NLP and CV datasets. However, as datasets from medal-awarding competitions, even the simplest ones among the datasets still pose considerable challenges compared to the widely-recognized datasets. The selected Kaggle datasets are listed as follows:
Age The ICR Identifying Age-Related Conditions competition (Kaggle & InVitro Cell Rescarch, 2023) involves the development of machine learning models capable of detecting various age-related conditions from medical imaging data. Participants are provided with a dataset containing images labeled with different age-related conditions, and the task is to create models that can accurately classify these conditions. The competition aims to advance research in medical imaging by leveraging AI improve the diagnosis and understanding of age-related diseases.
Default The American Express Default Prediction competition (Kaggle & Express, 2023) challenges par- ticipants to predict the likelihood of credit default by American Express cardholders. The dataset provided includes a wide range of anonymized features spanning transactional data, account information, and user behavior over time, designed to simulate real-world conditions. The goal is to enhance credit risk models and improve financial inclusion by accurately identifying al-risk customers.
Whale The Happywhale- Whale and Dolphin Identification competition (Kaggle & Happywhale, 2023) focuses the identification of individual whales and dolphins from images. Participants are tasked with developing algorithms that can recognize and match cetaceans based unique physical features captured in photographs, contributing to marine biology research and conservation efforts. The dataset comprises images labeled with species and individual IDs, making it a challenge in both classification and individual identification.
Strip The Mayo Clinic STRIP AI competition (Kaggle & Clinic, 2023) is aimed at predicting stroke and other thrombotic events by analyzing electroencephaalgran (EEG) data. This competition provides a platform for the development of AI models that can assist clinicians in early detection and intervention for stroke patients, leveraging a dataset of EEG recordings with associated clinical outcomes. This challenge stands at the intersection of Al and healthcare, with the potential to significantly impact patient care and outcomes in neurology.
Exam The Kaggle- LLM Science Ezam competition (Kaggle, 2023) challenges participants to create lan- guage models that can solve science exam questions. The focus on evaluating the ability of these models to understand and apply scientifie knowledge accurately. This competition is part of Kaggle's efforts to
34

---

Published in Transactions on Machine Learning Research (09/2024)
benchmark the performance of language models in educational contexts, providing a unique opportunity to advance AT in the field of automated question answering.
Learning The Feedback Prize English Language Learning competition (Kaggle & Lab, 2023) aims to develop models that can provide feedback on written English tasks by students learning English as a foreign language. The challenge involves analyzing turtenttwrittenn essays to identify areas for improvement, thus aiding in language learning and teaching. This competition seeks leverage Al enhance English language education by providing scalable, personalized feedback.
E
Prompts
For every dataset, a common instruction prompt is provided the beginning (Figure 10)
The user is trying to create a Python program for a supervised machine learning task. Your goal is to assist the user in creating the program. The user might provide the following information for the task:
1. A description of the input data. The input data contains the features that are the characteristics or attributes of the dataset. These features provide information that the model uses to make predictions.
A description of the output data, also known as 'labels' or 'targets', represents the desired prediction outcome associated with each input example.
3. The task objective, which describes the goal of the machine learning task.
4. The evaluation metrics, which numerical measure of how well the model's predictions match the desired outcome.
5. A description of the avaitable files.
Figure 10: Common instruction prompt.
Then, for each dataset, task-specific prompt follows. Below, we detail the prompts utilized in our experi- ments across the 12 datasets. In each prompt, workspace can be replaced with the path to the directory of the files for the task.
35

---

Published in Transactions on Machine Learning Research (09/2024)
The description of the input data is: Various features about various locations in Boston, such as per capita crime rate and proportion of residential land zoned for lots over 25,000 sq. ft.
The description of the output data is: The housing price of each location.
The objective of the machine learning task is: For each location in Boston, predict the housing price of the location.
The evaluation metric is: RMSE.
The following files are available: { prices across different locations in Boston. The table has fourteen columns, The first thirteen columns of the table are features used for prediction. The last column named 'MEDV' is the output/target of the prediction (housing price). "}
('workspace/data.com "A table of various features and housing
Figure 11: Prompt for Boston.
The description of the input data is: Properties of flowers.
The description of the output data is: Three species of flowers.
The objective of the machine learning task is: For each flower, classify the species of the flower.
The evaluation metric is: Top Accuracy.
The following files are available: ("workspacee/rrssccs// "A table of six columns. The first column named 'Id' contains the IDs of the flowers, for example, 1. The second to fifth columns each contain a property of the flower. The sixth column contains the species name, for example, 'Iris-setosa"")
Figure 12:
Prompt for Iris.
The description of the input data is: Fifty anonymized health characteristics.
The description of the output data is: Age-related conditions.
The objective the machine learning task is: For each subject, based on the anonymized health characteristic, predict whether a subject has or has not been diagnosed with age- related conditions.
The evaluation metric is: Balanced logarithmic loss.
The following files are available: (wworkspace/train.csy::: "A table of 58 columns. The first column named 'Id' contains IDs of the subjects, for example, 000ff2bfdfe9. The 2nd to the 57th columns contain the anonymized health characteristics. The 58th column named 'Class' contains the classification of the subject's conditions, with indicating the subject has been diagnosed with the three conditions, indicating they have not.")
Figure 13: Prompt for Age.
36

---

Published in Transactions on Machine Learning Research (09/2024)
The description of the input data is: Various features about customers of bank.
The description of the output data is: The probability that a customer does not pay back their credit card balance amount in the future.
The objective of the machine learning task is: For each customer, based on the features of the customer, predict the probability future default.
The evaluation metric is: The mean of the normalized Gini Coefficient and default rate captured at 4%.
The following files are available: /'workspacc//rrain data.csv': "A table of 190 columns. The first column named 'customer_ID' contains the IDs of each customer, such htt.://6000.00c0000a/1/777770000
The rest of the columns contain various features of the customers." wwokkspace//rainn latp:rrrriiiilabbss....s... "A table of 2 columns. The first column is similar to that of workspace/train data.csv' The second column named 'target' contains the labels for each customer, with representing future payment default and representing default.")
Figure 14: Prompt for Delault.
The description of the input data is: Colour images of various types of objects.
The description of the output data is: The type the object.
The objective of the machine learning task is: For each image of an object, predict an integer that represents the type of the object in the image.
The evaluation metric is: Top-1 Accuracy.
The following files are available: ("workspace/CIFAR 10': "a folder of six files, including 'data batch_1h 1, 'data batch 2', 2'a...'data batch 5' and 'validation batch'. In this folder, each one of @tata_batch___
aitta_bbtch____ 2'. and 'data_ 'data_batch_5 is a part of the training data. The file dealidation_batche is the validation data. For each of the six files: 1. it is dictionary that can be loaded using wwccke..oad(eecoodiiiibbooeee 'labels'. 3. the value of the key 'data' is 10000x3072 numpy array of uint8s. Each row of the array stores a colour image of size 32x32. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image stored in row-major order, so that the first 32 entries the array the value of the key 'labels' is: a list of 10000 numbers in the range 0-9. The number at index indicates the label of the ith image in the array data.")
2. the dictionary contains two keys, 'data' and
the red channel values of the first row of the image. 4.
Figure 15: Prompt for CIFAR-10. The loading code is copied from the official description of the datasets (Krizhevsky et al., 2023)
37

---

Published in Transactions on Machine Learning Research (09/2024)
The description of the input data is: Colour images of various types of objects.
The description of the output data is: The type of the object.
The objective of the machine learning task is: For each image of object, predict an integer that represents the type of the object in the image.
The evaluation metric is: Top-1 Accuracy.
The following files are available: ("WOrkspace/CCIARR 100/train': It file that contains the training data. The file has the following properties: 1. is a dictionary that can be loaded using wor.ee.oaad/eccodngg/bbeee
2. the dictionary contains two keys, b'fine_labe labells' and b' b'data'. 3. the value of the key b b'data' is 50000x3072 numpy array of uintês. Each row of the array stores a colour image of size 32x32. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored row-majjo order, so that the first 32 entries of the array are the red channel values of the first row of the image. 4. the value of the key b b'fine labels' is: list of 50000 numbers in the range 0-99. The number at index indicates the label of the ith image in the array data.", workspace/ CIFAR_100/test 100/test': "It is a file that contains the testing data. The file format of CIFAR_100/tesst is similar to 'CIFAR_ The differences are that the shape of the numpy array corresponding to the key b b'data' is 10000X3072 and the length of the list corresponding to the key
is 10000.")
Figure 16: Prompt for CIFAR-100. The loading code is copied from the official description of the datasets (Krizhevsky et al., 2023)
The description of the input data is: Images of the flukes of humpback whales.
The description the output data is: Identification of the humpback whales out 15587 unique IDs.
The objective of the machine learning task is: From each image of the flukes of the humpback whales, predict the identity of the whales.
The evaluation metric is: MAP@5.
The following files are available: { (workspacc/ttain.ccs// "A table of three columns. The first column named 'image' contains the names of the image files, for example, a0021adfb725ed..ppp
The second column named 'species' describes the species of the whales in the images. The third column named 'individual Ld' contains the identification of the whales in the image, for example, sadddb1636b9"" 'wrkspacc//maaee:: aa folder of images. For example, 00021 adfb725ed.jjgg.'
Figure 17: Prompt for Whales.
38

---

Published in Transactions on Machine Learning Research (09/2024)
The description of the input data is: High-resolution whole-slide digital pathology images depicting blood clots from patients.
The description of the output data is: Classification of the blood clots for the etiology either Cardioembolic (CE) or LAA (Large Artery Atherosclerosis).
The objective of the machine learning task is: From each image of a blood clot, classify the etiology of the blood clot either CE or LAA.
The evaluation metric is: Weighted multi-class logarithmic loss.
The following files are available: (wworkspace/train.cs/// "A table of five columns. The first column named 'image id' contains the ids of the images, for example, '006388 O'. The second column named "center_ id' contains the IDs of the medical center where the slides were obtained, for example, 11. The third column named 'patient_ie id' contains the IDs of the patients, for example, '006388'. The fourth column named 'image_me sumaee contains the enumerations of images for the same patient. for example, 0. The fifth column named 'label' contains the etiology the clot, either 'CE' or 'LAA", wordspace//mages"; folder of images. For example, 006388_0.tt
Figure 18: Prompt for Strip.
The description of the input data is: Reviews of movies.
The description of the output data is: Whether the review is positive or negative.
The objective of the machine learning task is: For each review, predict an integer that represents the sentiment of the review. A value of 1 represents positive sentiment. value of 0 represents negative sentiment.
The evaluation metric is: Error rate (the percentage of incorrect predictions).
The following files are available: wookspacc//MMDB/dttt..s "A table of two columns. The name of the first column is 'review'. Each value of the first column is string that represents review. The name of the second column is 'sentiment'. Each value of the second column is string that represents sentiment. A value of "positive' represents positive sentiment. A value of 'negative' represents negative sentiment."]
Figure 19: Prompt for IMDb Reviews.
39

---

Published in Transactions on Machine Learning Research (09/2024)
The description of the input data is: The titles and descriptions of some news articles.
The description of the output data is: The type of the news articles.
The objective of the machine learning task is: For each news article, predict an integer that represents the type of the news article.
The evaluation metric is: Error rate (the percentage of incorrect predictions).
The following files are available: (woorkspace/A//tainncmm
"A table of three columns. The file 'AG/train.cs' contains the training data. The name of the first column is 'Class Index'. Each entry of the first column is an integer that represents the type of a piece of news article. The total number of types is five. The name of the second index is 'Title'. Each entry of the second column is a string that represents the title of the news article. The name of the third column 'Description'. Each entry of the third column is string that represents the description of the news article.")
Figure 20: Prompt for AG News.
The description of the input data is: Multiple-choice questions with options A, B, C, D, and E.
The description of the output data is: The answers to the questions.
The objective of the machine learning task is: For each question, predict up to 3 options as answers.
The evaluation metric is: MAP@3.
The following files are available: ('workspace/train.com :ttp:////tiiiccc::::.. "A table of 8 columns. The first column named 'id' describes the unique ID for each question. The second column named 'prompt' contains the question. The third to the seventh columns are named 'A', 'B', 'C; 'D', and 'E' respectively, with each column containing the respective option. The eighth column contains the correct option for the question."]
Figure 21: Prompt for Exam.
40

---

Published in Transactions on Machine Learning Research (09/2024)
The description of the input data is: Argumentative essays written by 8th-12th grade English Language Learners.
The description of the output data is: Scores according to six analytic measures: cohesion, syntax, vocabulary, phraseology, grammar, and conventions.
The objective of the machine learning task is: For each essay, predict the score of each of the six measures.
The evaluation metric is: MCRMSE.
The following files are available: (woorkspace/triin.css) "A table columns. The first column named 'text jd' describes the unique ID for each essay. The second column named 'full_tee text' contains the essays. The third to the eighth columns are named 'cohesion', 'syntax', 'vocabulary' "phraseology', 'grammar' and 'conventions' respectively, with each column containing scores ranging from 1.0 5.0 increments of 0.5.")
Figure 22: Prompt for Learning.
41

---

Published in Transactions on Machine Learning Research (09/2024)
Next, present the prompts for the intermediate steps in Text-to-ML For Figures 23 to 29, texts high- lighted in yellow represent prompis based on the common instruction prompt (Figure 10), prompts based on the user input (Figures 11 to 22), or prompts dynamically assembled from responses in previous steps via in Algorithm Texts highlighted in green represent prompts that depend on the specific implermentation of the pre-written modules (e.g., loss functions, optimizers, train- ing procodure, and testing procodure) and are subject to the constraints in the progenitor testing protocols (Section 4 and Appendix C.1). Texts in red represent prompts configured in the background settings.
the instruction constructors ({ " db..... ......... .b)
Your goal is to analyze a machine learning task by classifying the machine learning task and determining suitable format of input and output data in the machine learning task. Both the input and output data refer to the data after all data processing steps. The model takes the input data, and the model generates the predicted output data.
The common instruction prompt without the first two sentences
The textual task description from the user
Formatting instructions for the analysis
An example for this part prompt as below:
Your answer must consist the following parts:
Part classification 'computer vision'). The clussification should be based which lypes machine learning models suitable for the task. Traditional algorithms refer to ML lasks that are processed with traditional machine learning models (instead of deep learning models). Natural language processing refers deep learning tasks involving text processing. Computer vision refers deep learning tasks involving image processing
machine learning task among of l'traditional algorithms', 'natural language processing,
Part A classitication of the machine learning task among of ('binary classitication", 'mutticlass classitication', 'multi-label classification', 'multi-output regression', 'single-output regression', "squence-to-sequenee 'sequence generation', 'image generation', 'other').
Part 3: A classification of the format of the ground truth output of the machine learning task among one of ["probability representation of class labels', 'integer representation of class labels', 'regression output').
Part 4: A classificatiinn of the specitic type of machine learning task. the machine learning task about 'computer vision', then examples include image classification, object detection, image segmentation, facial recognition, optical cheracter recognition, edge detection, motion analysis and object tracking, depth cstimation, imago generation, pose estimation, and image super- resolution. the machine learning task about 'natural language processing', then examples include text generation, text classification, machine translation,, question answering, sentiment analysis, topic modeling, g, text summarization, and named entity recognition. the task about 'traditional algorithms', then you do not need to choose and simply say 'N/A' for Part Part Determine suitable format input and output data the machine learning task. the machine learning task about traditional algorithms (in part 1), your answer for part is simply 'N/A', your answer ends here and you should not answer Part 1, 5-2, 5-3, 5-4, 5-5. machine learning task about deep learning (in part 1), your answer Part should consider following both input and output data refer the data after data processing steps. The model takes input data, torch.Tense objects. The
the model generates predicted output torch.utils.dd wat.aaaaaaacooooaa dataloader.rr... LataLoader object manage the input and output data. And, the machine learning is about for part should Part 5-1; Determine suitable number of types of tensors in input data. the machine learning task is natural language processing task (in part the types tensors must include type of tensor about attention, mask for each type sequence.
input output
deep learning (in part your
following five parts:
Part Determine suitable number of types of tensors in output data. the task multi- pnutti-output regression classification task, combine all outputs in one type of tensor and the suitable number of types of tensor 1.
Part 5-3: For each type of tensor, determine suitable shape for the type of tensor. Name each dimension.
Part 5-4: For each dimension of each type of tensor, explain the meaning of the dimension.
Part 5-5: For each dimension of each type of tensor, determine whether the size of the dimension changes over the settings and parameters of the data preparation process.
Let's think step by step:
Figure 23: Prompt for task analysis
42

---

Published in Transactions on Machine Learning Research (09/2024)
There is an analysis of machine learning task.
Your goal is to convert the analysis into:
Formatting instructions for the answer
An example for this part of the prompt is as below:
A Python list of five elements for deep learning tasks or Python list of three elements for tasks about traditional algorithms.
Below is the analysis:
Response from the task analysis step
Example format for the answer
An example for this part of the prompt is as below:
Your answer must be a Python dictionary
Below is example Python dictionary for image classification task:
{'input': [('name': 'images', 'shape': ['batch_ Tbatch_size', 'channel', 'height', 'width'], 'batch_size':
('meaning': 'the size of the batch', fixed_or_variabll or__aabbb"b 'variable"), 'channel': ("meaning:: 'tho number of channels the image', 'fixed_o @ied_Lor__te _rrl_____va__ variable::' 'fixed'), 'height': ('meaning: 'the height of the image', 'fixed or variable': 'variable),, 'width': ("meaning': 'the width of the image', "fieed_ or variable': 'variable')]], 'output': [('name': "predictions', 'shape': "batch_size"" bbatch_size"s ('meaning:: 'the size of the batch', 'fixed_or or variable': variable')))
Intro for the answer
An example for this part of the prompt is below: Here is the Python dictionary:
Figure 24: Prompt for task classification
43

---

Published in Transactions on Machine Learning Research (09/2024)
The common instruction prompt
The textual task description from the user
Classification of the ML task extracted from the response from the task classification step
Your task is to suggest some data preparation steps for the machine learning task.
[Steps to be planned for each domain in the settings]
Formatting instructions for the answer
example for this part of the prompt below:
Your answer should be Python dictionary.
The keys in the dictionary are I'data loading', 'text cleaning', 'data augmentation", "task-specific feature engineering').
The value of each key is string that describes the specific method for the key. The value key can be 'none' if your suggestion is to skip the step. An example dictionary can be: ('data loading': 'load the raw data using Pandas', 'text cleaning': 'romove unnecessary characters, such as HTML tags, URLs and special characters', 'deta augmentation': 'use back translation to generate more data', "tas-sppecific feature engineering:: 'none").
Intro for the answer
An example for this part of the prompt is as below: Here is the Python dictionary:
Figure 25: Prompt for data preparation module planning
44

---

Published in Transactions on Machine Learning Research (09/2024)
The common instruction prompt
The textual task description from the user
Classification of the ML task extracted from the response from the task classification step
Your task is to suggest some pretrained models for the machine learning task. Some machine learning tasks only need one model, some tasks need multiple models (e.g., one for backbone and one for head, one for backbone and one for the framework, one for encoder and one for decoder, or one for generator and one for discriminator), and some tasks can be solved by either one model or multiple models.
Formatting instructions for the answer
An example for this part the prompt is as follows:
Your answer should be Python list and only a python list. Each entry of the list is dictionary that describes a choice of single model For a choice of single model, the dictionary should be dictionary of single key with the format of (name_ of model: explanation of the peiitabbbility of mf__molee..... For choice of a combination of models, the dictionary should be a dictionary multiple keys with format ('name_od of model in the combination-name of model in the ttt__tte_ooreeeeeo ('reason:' 'explanation_of hwtpiaaattiooot_h__e___e____ee wtt_suttbbllt__oootte_ooooo nation': 'role_of _f__odee___in the combinaatii,iin 'name_of of @irame__f_mmde_222 ____________ ___n__te____________________e_ tet__t___ooooee 'role_op offoff__oooo modee_ 2 th_ www_tte_commee combination' ...)). Examples of role of modeLL tt_i___t_______ the_combination include 'backbone', 'head', 'encoder', 'decoder', 'generator', 'discriminator! and 'rramework.. The list can consist of only choices of a single model, only choices of combinations of models, or
combination of models.
combinstiont...!
(nmmmmmo_nooooo _a______m___ of_modee___in____
choices of both single model and combinations of models.
You should suggest [an integer from the settings] choices and the Python list should have [an integer from the settings] keys.
Intro for the answer
An example for this part of the prompt is below: Here is the Python list:
Figure 26: Prompt for modeling module planning
45

---

Published in Transactions on Machine Learning Research (09/2024)
The common instruction prompt
The textual task description from the user
Classification of the ML task extracted from the response from the task classification step
Your task is to generate the Python code for the data preparation module of the machine learning task.
Requirements of the code
An example for this part of the prompt is as below:
The generated Python code should meet the following requirements:
(1): There is function named 'generate dataloader' in the code.
(2): The data preparation module should load the files. All avaitable files are listed in the user's description of the available files.
(3): The data preparation module should contain the following steps in suitable order:
Plans for the data preparation module extracted from the response from the data preparation module planning step
(4): The returned objects of the function must be
vatader,vaLloaaeee
leader is
torch.utils.data. ood object that generates the training data and val_loader a torch. proch.utll.ddta.. data..ataaoade object that generates the validation data. Both
torch.utils daw.sddaaa.aa.... LotaLoder objects should be an iterable that generates a batch of data in the format a tuple of tensors. Each tensor in the tuple represents one type of tensor. (5): The input and output data from the torch.utils.data. Dataloader objects must satisfy the following requirements:
Format the data flow extracted from the response from the task classification step
Intro for the answer
An examplc for this part of the prompt is as bolow:
Here is the Python code:
Figure 27: Prompt for data preparation module code generation
46

---

Published in Transactions on Machine Learning Research (09/2024)
The common instruction prompt
The textual task description from the user
Classification of the ML task extracted from the response from the task classification step
Your task is to generate the Python code for the modeling module of the machine learning task.
Requirements of the code
An example for this part of the prompt is as below:
The chosen model for the machine learning task is:
Plans for the modeling module extracted from the response from the modeling module planning step
The generated Python code shoutd meet the following requirements:
(1): There a function named 'generate mameertt_mmoel'' the code.
(2): The returned values the function must a forch.nn.Module Module object.
(3): The torch.nn. Module object should have forward method that takes all types of tensor in input data as arguments. The value of each argument is single-type tensor for a batch of data points. The forward method should return all types of tensors in output data. The value of each returned object single-type tensor batch of data points. (4): The input data and (ground-truth) output data are from torch. utils.ddta. datalaaddaadddrr..mm DataLoader object. The input and output data from the wrrch.utils.daaeee tlatt.dataloadee..e. waw.allsdder.DaaLaaddeeede DataLoader object are already processed by data preparation steps (including tokenization for natural language processing tasks). The content of the processed input and output data is follows:
Format of the data flow extracted from the response from the task classification step
Intro for the answer
An example for this part of the prompt is as below:
Here is the Python code:
Figure 28: Prompt for modeling module code generation
47

---

Published in Transactions on Machine Learning Research (09/2024)
There is a machine learning task, your goal is to generate the code for the evaluation metrics and for processing the output data for the evaluation metrics.
The common instruction prompt without the first two sentences
The textual task description from the user
Classification of the ML task extracted from the response from the task classification step
Format of the data flow extracted from the response from the task classification step
Requirements of the code
An example for this part of the prompt as below:
The generated Python code should meet the following requirements:
(1): There is function named 'generate evvllutionn in the code.
(2): The function 'eeeerae e should always include and only include the following arguments: An argument named 'predicted. scores'. The value of the argument is the torch. Tensor predicted output about scores. argument named aetual_scores The value of the argument is the torch. torch.Tensor of actual (ground truth) output about scores.
(3): The function 'generate eerauuatioaaaaiinnnne should convert the raw output into the format acceptable by the evaluation metrics.
(4): The function generate_evaluation should compare the predicted output(s) and actual output(s) based the specified evaluation metrics in this machine learning task.
(5): The function o should return one object, which is the numerical score based on the evaluation metrics in the format of Python built-in float.
Intro for the answer
An example for this part of the prompt is as below:
Here is the Python code:
Figure 29: Prompt for post processing module code generation
48

---

Published in Transactions on Machine Learning Research (09/2024)
F
Additional results
Table Mean number of attempts per success with GPT-3.5.
Contextual Modular Generation (ours)
Datasets
CoT
Zero-shot
Reflexion
Self-Revision
w/o tests
w/o reflection
standard
Boston
9.0 + 0.7
5.5
2.1
3.7 0.5
3.6
+
Iris
3.0 + 0.2
3.4 = 0.6
5,0 = 1.5
5.0 + 1.0
3.6 + 0.6
3.2 + 0,4
3.3 + 0.5
Age
16.1 2.0
20.3 + 6.8
8.7 1.4
8.3 = 1.9
Eapress
30.2 + 4.0
17.3 + 3.2
13.2 + 2.1
11.0 + 2.6
CIFAR-10
5.7 + 0.9
6.7 = 1.3
6.4 = 0.5
20.2 + 5.4
5.9 + 1.6
5.6 = 0.6
CIFAR-100
7.4 1.9
7.6 = 1.3
13.5 3.7
7.5 + 1.3
7.2 = 0.7
Whale
16.8 1.9
17.8 1.6
Strip
13.0 + 2.8
14.7 + 2.9
IMDb Reviews
6.2 + 0.7
5.3 = 1.5
6,4 = 1.3
11.6 + 1.9
3,6 = 0.7
4.0 + 0.6
AG News
11.8 + 2.2
12.3 + 2.9
10.4 + 1.5
4.0 + 0.8
4.2 + 0.5
Exam
15.0 + 5.8
13.8 + 4.8
Learning
15.6 5.3
17.0 = 2.8
Table 7: Mean number of attempts per success with PaLM 2.
Contextual Modular Generation (ours)
Datasets
CoT
Zero-shot
Reflexion
Self-Revision
w/o tests
w/o reflection
standard
Boston
15.6 + 3.5 3.9 +
8.3 + 1.8
10.4 + 2.7
4.4 + 0.7
5.2 + 1.6
3.4 + 0.5
Iris
0.7
5.3
3.8
0.5
3.6 + 0.7
Age
7.2 + 0.6
7.9 + 1.5
Express
20.2 5.2
1.7
CIFAR-10
8.7 + 0.8
9.8 4.0
7.4 0.7
7.3 + 1.0
CIFAR-100
7.4 + 1.3
20.2 + 5.1
16.8 + 3.3
8.8 + 1.4
7,4 + 1.8
Whale
18.3 + 7.3
18.0 + 3.0
Strip
33.3 + 7.6
IMDb Reviews
5.8 + 1.2
1.0
10.5 2.1
12.2 1.1
5.0
3.9 -
AG News
12.2 + 1.0
18.2 + 3.4
6.0 + 1.1
4.9 + 0.7
Exam
17.7 2.5
14.6 + 4.9
Learning
19.8
24
16.7 + 4.3
(1) tests-only: feedback is provided solely through unit tests (Figure and Algorithm 3). (2) reflection: feodback provided through both unit tests and the self- reflection mechanism.
Table 8: Mean iterations to correct errors.
Modules
GPT-4
GPT-3.5
PaLM
lests-only
reflection
lests-only
reflection
Lests-only
reflection
Data preparation
4.1 + 1.8
4.2 1.9
5.6
6.0 1.7
5.8 2.4
6.1 2.2
Modeling
2.7 + 1.1
2.4 + 1.1
2.9 + 1.4
3.3 + 1.9
5.4 + 2.2
5.4 + 3.6
Post processing
3.0 + 1.0
2.8 1.4
3.1 1.9
2.7 1.2
4.5 1.5
3.9 1.2
49


Reviews:
[{"review_id": "H4mtDFCZiS", "reviewer": "Reviewer_TZVm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\n1. The proposed method is novel and integrates software engineering practices in a relatively neat way, specifically tailored for autoML.  \n2. The text-to-ML task is practically useful and challenging while the role of LLMs for it is not extensively studied yet.  \n3. This paper is comprehensive on addressing the details and includes theoretical analysis for the complexity of the proposed approach.  \n4. The empirical results are good on multiple benchmarks.\n\n### Weaknesses\n\n1. I feel some details are difficult to follow and it is not very clear how the LLMs are instructed exactly at each stage. In the appendix, I suggest the authors give examples for the exact prompt used for LLMs for each stage during the process. Currently the appendix only shows the general prompt during evaluation.  \n2. Several of the popular datasets such as CIFAR and IMDB review classification are widely used over the Internet. Thus the LLMs are very likely to have seen many programs for these datasets already.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on program synthesis from LLMs, and particularly for an autoML setting. Given text descriptions of the task, the authors propose a new prompting framework, contextual modular generation, to prompt the LLMs to generate the machine learning programs. The prompting framework is novel to integrate some design heuristics from software engineering, such as decomposing the program synthesis as generating several smaller modules respectively (mainly classified as data preparation, modeling, and postprocessing). The program of each module is verified through unit tests from LLMs to ensure different modules are compatible with each other and the entire program is free of syntax errors. This paper also presents a theoretical analysis on the scalability of the method over the length of complex output sequences, which indicates that the generation difficulty of contextual modular generation scales linearly over the problem length in the best case. Experimental results on 12 datasets demonstrate that the proposed method outperforms baselines such as COT and reflexion, identifying valid programs with the least number of attempts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I suggest to include specific prompts used to instruct the LLMs at each stage in the appendix, so that the readers can better understand what is going on exactly from the specific examples.  \n2. I suggest clarifying the GPT-4 versions that the paper use in the main text of the paper, given that there are so many versions of GPT-4 and the authors talk about the knowledge cutoff date.", "broader_impact_concerns": "The paper focuses on program synthesis for an autoML purpose, which is possible to produce ML programs that can automatically train a model to learn from toxic data for certain purposes. This aspect may warrant some discussion which the current version of the paper lacks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H1FgHUmo7O", "reviewer": "Reviewer_br1n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n   - The paper introduces an approach by integrating Large Language Models (LLMs) with AutoML systems. This synergy addresses the limitations of current AutoML methods and showcases the potential for enhanced machine learning automation.\n   - The paper provides extensive experimental results, comparing different sequence generation methods across various datasets. This robust evaluation demonstrates the effectiveness of the proposed methods, particularly those incorporating self-reflection.\n\nWeaknesses\n\n   - While the framework shows promise for various tasks, it may struggle with highly complex ML tasks. The paper acknowledges this limitation, indicating a need for further research to address such challenges. The paper notes the infeasibility of comparing automated solutions with human-generated ones in competitive environments like Kaggle. \n   - The scalability and generalization of the proposed methods to different domains and datasets may require further validation. Ensuring that the framework can handle diverse scenarios is crucial for broader applicability.\n   - The integration of LLMs with AutoML systems might be resource-intensive, potentially limiting its use in resource-constrained environments. Addressing this issue would be important for wider adoption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Integration of LLMs with AutoML**:  The study explores the synergistic relationship between LLMs and AutoML, proposing a framework where LLMs generate machine learning (ML) programs that AutoML systems can then optimize. This integration addresses the limitations of current AutoML methods, which often require substantial human expertise for programming and are resource-intensive.\n\n**Text-to-ML Framework**: The paper introduces a novel Text-to-ML framework that automates the entire ML program synthesis process. This framework converts textual task descriptions into optimized ML programs through a multi-step sequence-to-sequence (seq2seq) approach. Each step involves generating and optimizing program components using LLMs and AutoML.\n\n**Enhanced Sequence Generation**: By incorporating a self-reflection mechanism, the proposed methods improve the generation of valid ML programs. The self-reflection mechanism allows the system to retain feedback from previous trials, enhancing the accuracy and efficiency of subsequent generations.\n\n**Robust Evaluation**: The paper conducts extensive experiments using various datasets and compares different sequence generation methods. The results demonstrate that the proposed methods, particularly those incorporating self-reflection, outperform others in generating valid ML programs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The current framework may not be fully capable of addressing highly complex ML tasks. Enhancing the framework to better handle these tasks through advanced optimization techniques and more sophisticated models is crucial.\n   \nThe implementation's efficiency, particularly in practical applications, needs improvement. Optimizing the sequence generation process and reducing computational requirements are necessary steps.\n   \nDeveloping methods to better compare automated solutions with human-generated ones would provide valuable insights into the performance gap and areas for improvement. This comparison is essential to demonstrate the framework's effectiveness in real-world scenarios.", "broader_impact_concerns": "The democratization of machine learning (ML) through the proposed Text-to-ML framework increases accessibility, which can be both a benefit and a risk. While it empowers non-experts to leverage ML, it also raises the potential for misuse by individuals with malicious intent or without adequate understanding of ML principles. A Broader Impact Statement should address the measures in place to prevent misuse, such as guidelines for responsible use and potential monitoring mechanisms.\n\nThe automation of ML program synthesis can obscure the decision-making process, making it difficult to understand how certain outcomes are derived. This lack of transparency can hinder accountability, especially in critical applications like healthcare or criminal justice. The Broader Impact Statement should emphasize the importance of transparency and accountability, suggesting ways to ensure that automated processes are interpretable and that outcomes can be traced back to specific decisions.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PDsNQGe9eR", "reviewer": "Reviewer_gNyJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The method is interesting and the authors provide a detailed explanation of the approach and the rationale behind components of the method.\n- A theoretical analysis of their approach is provided, which is quite welcome as it is often missing in similar works.\n\n## Weaknesses\n\n- This method includes a large number of components which, despite being justified in text, are not evaluated in isolation.\n  This makes it difficult to understand the contribution of each component to the overall performance of the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new method to synthesize machine learning programs using\nlarge language models and automated machine learning. The authors derive a program\ndecomposition method that breaks down the task into smaller modules, and then utilize unit tests\nto guide the search for the best combination of modules. This method\nautomates the entire ML workflow, breaking it into three steps: data preprocessing, modelling, and post-processing.\nThe authors demonstrate the effectiveness of their method on a small set of tasks, showing that it\noutperforms common baselines in neural program synthesis such as chain-of-thought and and Reflexion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors should provide a more detailed evaluation of the individual components of their method. This would help to understand the contribution of each component to the overall performance of the method.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers all recommended accepting this paper. There were initial concerns about lack of clarity that were resolved during the rebuttal, so that the current draft should be clear and readable. Reviewers did note that the proposed method was simple and that it could primarily target very simple ML applications, but level of novelty is not a criterion for acceptance at TMLR and many applciations of ML involve simple problems.
Decision: Accept (poster)