Real-Time AutoML
Authors: Iddo Drori
Brandon Kates
Anant Kharkar
Lu Liu
Qiang Ma
Jonah Deykin
Nihar Sidhu
Madeleine Udell
Venue:  (2021)
Keywords: Automated machine learning
zero-shot learning
graph neural networks
transformers

Abstract: We present a new zero-shot approach to automated machine learning (AutoML) that predicts a high-quality model for a supervised learning task and dataset in real-time without fitting a single model. In contrast, most AutoML systems require tens or hundreds of model evaluations. Hence our approach accelerates AutoML by orders of magnitude. Our method uses a transformer-based language embedding to represent datasets and algorithms using their free-text descriptions and a meta-feature extractor to represent the data. We train a graph neural network in which each node represents a dataset to predict the best machine learning pipeline for a new test dataset. The graph neural network generalizes to new datasets and new sets of datasets. Our approach leverages the progress of unsupervised representation learning in natural language processing to provide a significant boost to AutoML. Performance is competitive with state-of-the-art AutoML systems while reducing running time from minutes to seconds and prediction time from minutes to milliseconds, providing AutoML in real-time.

Under review as a conference paper at ICLR 2021
REAL-TIME AUTOML
Anonymous authors Paper under double-blind review
ABSTRACT
We present new zero-shot approach automated machine learning (AutoML) that predicts a high-quality model for a supervised learning task and dataset in real-time without fitting a single model. contrast, most AutoML systems require tens or hundreds of model evaluations. Hence our approach accelerates AutoML by orders of magnitude. Our method uses transformer-basee ding to represent datasets and algorithms using their free-text descriptions and a meta-feature extractor to represent the data. We train a graph neural network which each node represents a dataset to predict the best machine learning pipeline for a new test dataset. The graph neural network generalizes to new datasets and new sets of datasets. Our approach leverages the progress of unsupervised repre- sentation learning in natural language processing to provide a significant boost to AutoMI.. Performance is competitive with state-of-the-ar AutoMI. systems while reducing running time from minutes to seconds and prediction time from minutes milliseconds, providing AutoML in real-time.
language embed-
INTRODUCTION
data scientist facing a challenging new supervised learning task does not generally invent a new al- gorithm. Instead, they consider what they know about the dataset and which algorithms have worked well for similar datasets in past experience. Automated machine learning (AutoML) seeks to auto- mate these tasks to enable widespread use of machine learning by non-experts. major challenge is to develop fast, efficient algorithms to accelerate applications of machine learning (Kokiopoulou al., 2019). This work develops automated solutions that exploit human expertise to learn which datasets are similar and what algorithms perform best. We use wransformer-hased language model (Devlin et al. 2018) allowing our AutoMI. system to process text descriptions of datasets and algo- rithms, and a feature extractor (BYU-DML (019) to represent the data itself. Using such models for our representation brings in large-scale data. We allow to train our model on other existing AutoML system solutions, specifically AutoSklearn (Feurer et al., 2015), AlphaD3M (Drori et al., 2018), OBOE (Yang et al. 2019). and TPOT (Olson & Moore, 2019). tapping into their diverse set of solutions. Our approach fuses these representations (dataset description, data, AutoML pipeline descriptions) and represents datasets as nodes in a graph of datasets.
Generally, graph neural networks are used for three main tasks: (i) node prediction, (ii) link pre- diction, and (iii) sub-graph or entire graph classification. In this work we use a GNN for node prediction, which predicts the machine learning pipeline for an unseen dataset. Specifically, we use graph attention network (GAT) (Veličković et al.) 20188) with neighborhood aggregation, in which an attention function adaptively controls the contribution of neighbors. An advantage of us- ing a GNN for AutoML is boosting AutoML performance by sharing information between datasets (graph nodes): including description and algorithm, by message passing between the nodes in the graph. In addition, GNNs generalize well to a new unknown dataset using the aggregated weights learnt over the training datasets. GNN weights are shared with the test dataset for prediction. GNNs generalize to entire new sets of datasets. Finally, prediction is in real-time, within milliseconds.
A simple idea is to use machine learning pipelines that performed well (for the same task) on similar datasets. What constitutes similar dataset? The success of an AutoML system often hinges this question, and different frameworks have different answers: for example, AutoSkleam (Feurer et al. 2013) computes set of meta-features, which dataset, while OBOE (Yang et al.] (019)) uses the performance few fast, informative models to compute latent features. More generally, for any supervised learning task, one view the list of
features describing the data features. for each

---

Under review as a conference paper at ICLR 2021
recommended algorithms generated by any AutoML system vector describing that task. This work is the first to use the information that a human would check first: a summary description of the dataset and algorithms, written in free text. These dataset features induce metric structure the space of datasets. Under an ideal metric, a model that performs well on one dataset would also perform well on nearby datasets. The methods metric using the recommendations of an AutoML framework together with the dataset description. We provide a new zero-shot AutoML method that predicts accurate machine learning pipelines for an unseen dataset and classification task in real-time and runs the pipeline in a few seconds.
develop in this work show how learn such
We use a transformer- based language model embed the description of the dataset and pipelines and feature extractor to compute meta-features from the data. Based on the description embedding and meta-features, we build a graph as the input to a graph neural network (GNN). Each dataset represents a node in the graph, together with its corresponding feature vector. The GNN is trained to predict a machine learning pipeline for new node (dataset). Therefore, given new dataset, our real-time AutoML method predicts pipeline with good performance within milliseconds. The running time our predicted pipeline is a few seconds and the accuracy of the predicted pipeline is competitive with state-of-the aht- AutoML methods that are given minute. This work makes several contributions by using language embeddings and GNNs for AutoML for the first time, and leveraging existing AutoML systems. The result real-time high-quality AutoML. system.
Real-time. Our system predicts machine learning pipeline for a new dataset in milliseconds and then runs the pipeline and tunes its hyper-parameters within three seconds. This reduces computa- tion time by orders of magnitude compared with state-of-the-art AutoML systems, while improving performance.
GNN architecture. Our work achieves real-time AutoML by introducing several architectural components that new AutoML. These include embeddings for datasel descriptions and algo- rithm descriptions using a state-of-the-art transformer-based language model in addition to (stan- dard) embeddings for data; model employing GNN on the graph of datasets. Importantly, the GNN recommends a pipeline for a new dataset by adding a node to the graph of datasets and sharing the GNN weights with the new node. Using the information and relationships between all datasets boosts AutoML performance.
-on-Euclidean embedding of datasets as graph: and a predictive
Embeddings. Bringing techniques from NLP to AutoMI, specifically using a large-scale transformer-based language model to embed the description of the dataset and algorithms, brings in information from large corpra of text. This allows our zero-shot AutoML to train small set of datasets with state-of-the-art test set performance.
Leveraging existing AutoML systems. Our flexible architecture can use pipeline recommenda- tions from any number of other AutoML systems to improve performance.
RELATED WORK
AutoML is an emerging field of machine learning with the potential to transform the practice of Data Science by automatically choosing model to best fit the data. Several comprehensive surveys of the field are available (He et al., 2019 Zöller & Huber 2019).
Processing each dataset in isolation. The most straightforward approach to AutoMI. considers settings for given algo-
each dataset in isolation and asks how to choose the best rithm. While the most popular method is still grid search, other more efficient approaches include Bayesian optimization (Snock et al. 2012) or random search (Solis & Wets. 1981.
Recommender systems. These methods learn (often, exhaustively) what algorithms and hyper- parameter settings performed best for training set-of-datasets and use this information to select better algorithms on a test set without exhaustive search. This approach reduces the time required to find a good model. An example is OBOE (Yang et al. 2019: 2020), which fits low rank model to learn the low -ow-dimeniional representations for the models (or pipelines) and datasets that best predict the cross- cross-valideee validated errors, among all bilinear models. To find promising models for new dataset,
2

---

Under review as a conference paper at ICLR 2021
OBOE rums set of fast but informative algorithms on the dataset and uses their validated emrors to infer the feature vector for the new dataset. A related approach (Fusi et al. 2018 using probabilisticc matrix factorization powers Microsoft Azure's AutoML service (Mukunthu 2019).
Search trees. Auto-Tuned Models (Swearingen et al. 2017 represent the search space tree with nodes being algorithms or hyper-parameters and searches for the best branch using a multi- armed bandit.
Model-based reinforcement learning. AlphaD3M (Drori toML as a single player game. The system uses reinforcement learning with self-play and pre- trained model which generalizes from many different datasets and similar tasks.
2018 2019a) formulated Au-
Genetic programming. TPOT (Olson & Moore 2019) and Autostacker (Chen al. 2018 use genetic programming to choose both hyper-parameter settings and a topology of machine learning pipeline. TPOT represents pipelines as trees, whereas Autostacker represents them as layers.
Bayesian optimization. AutoSklearn (Feurer et al. 2015) chooses model for new dataset by first computing (ad hoc) data meta-features to find nearest-neighbor datasets. The best-performing methods on the neighbors are refined via Bayesian optimization and used to form an ensemble.
Differentiable programming. End-to-end learning of machine learning pipelines is performed using differentiable primitives (Milutinovic et al. 2017) forming directed acyclic graph.
Algorithmic primitives. One major factor in the performance of an AutoML system is the base set of algorithms it can use to compose more complex pipelines. For a fair comparison, our numerical experiments compare our proposed methods only other AutoML systems that Scikit-learn (Pedregosa et al., 2011 primitives.
Embeddings. Language has a common unstructured representation as sequence of words, sen- tences. or paragraphs. The most significant recent progress in NLP large-scale transformer-based models and embeddings (Devlin et al.) 20188] Shoeybi et al. 2019 Raffel et al.) 2019) based on attention mechanisms (Vaswani al. 2017). An unsupervised corpus of text transformed into supervised dataset by defnning content-targe pairs along the entire text: for example, target words that appear each sentence, or target sentences which appear each paragraph. A language model is first trained to learn a low dimensional embedding of words sentences followed by map from low dimensional content to target (Mikolov et al. 2013). This embedding is then used on a new, unseen and small dataset in the same low- low-dimensional space. Our work uses such embeddings for automatic machine learning. a similar fashion to recent work (Drori et al.) (2019b) we use an em- bedding for the dataset and algorithm descriptions. In this work model the non-linear interactions between these embedding using a neural network as well.
METHODS
Our zero-shot AutoML predicts a machine learning pipeline for a classification task on a dataset based on the dataset description and data, and based on other datasets, their relationships, and their recommended pipelines by AutoML systems. We embed the dataset description and extract data meta-features to construct a graph of datasets where each node represents a dataset. The graph is processed using a graph neural network (GNN). Each node of the graph contains feature vector which the fusion of the description embedding and data meta-features, and the GNN node rep- resentations includes other AutoML solutions. The machine learning pipeline for new dataset is predicted by the GNN. detailed architecture is illustrated in Figure and described by Algorithms The notation used in this work given in Table I
2 and
3.1 PRE-PROCESSING
Our pre-processing consists of (i) dataset description embedding: (ii) dataset meta-feature extrac- tion; and (ii) pipeline computation and description embeddings, described next and summarized in Algorithm
3

---

Under review as a conference paper at ICLR 2021
Notation
Description
D
Dataset
M(D)
Dataset deseription
P
Machine learning pipeline
M(P)
Machine learning pipeline description
C.O.S,A.T
OBOE, AutoSkleamn, AlphaDSM, TPOT
Pc(D)
Pipeline recommended C dataser D
P.(D)
Best pipeline dataset D
P(D)
Predicted pipeline datasel
R(P,D)
Performance of running pipeline P on dataset D
meta-lealuress
FM ELM(D)
Emhedding of dataset description
F [FD, FM]
Concatenation
Fp BLM(P))
Embedding of pipeline description
Datasets graph
j JENO
Neighbors of node
F Fo(FD; FP,
Fusion network outpet on graph node
Features of node 9
gv(e))
Fusion network, features of node in GNN
{U))ENT (U. besr)))
Features node neighbors in GNN GNN with parameters W,
Table shoo AutoML notation and description.
Fratur Vecter
Grapk
Fastee F(Ps)
Fratue Natur K(P))
Fenur Verte F(Po)
Vester F(P)
Fiakee Nevoek
Draddees
Tranderner
Trandanser
Trandone
AppatsM
TPOTT
Autchlors
Data D
Dastiptin 3t
(D.M)
Zero-shot AutoMI. architecture: Dataset descriptions are embedded using language model. The data itself passed through feature extractor. Other AutoML. system algorithms are embedded using a language model. Fully connected neural networks fuse together the encoded fea- ture vectors. A graph captures the relationships between the embedded representations. At training time GNN learns the aggregation of each node in the graph and its neighbors. The GNN predicts pipeline for a new node (dataset). At test time a dataset added as a new node in the graph and the GNN predicts the best machine learning pipeline without running any AutoML system or evaluating any pipeline. Inputs are colored green, neural networks blue, intermediate outputs in red, and predicted output in yellow.
Figure 1:
Dataset Description Embedding. We create feature vector by embeding the description M(D) of each dataset 1024-dimensional vector FM = E(M(D)) E 1024 using BERT (Devlin al. 2018). The supplementary material shows examples of dataset descriptions embedded using our approach.
Data meta-features. We compute meta-features Fo TR 141 for the dataset D using feature extractor (BYU-DML_ 2019), restricting to meta-features that can be computed in one second on any of the datasets used in our experiments. Meta-features include statistics of the datasets and results of simple algorithms.

---

Under review as a conference paper at ICLR 2021
Pipelines and pipeline embedding. For each dataset, compute the recommended pipeline returned by AutoML systems OBOE (O), AutoSklearn (S), AlphaD3M (A), and TPOT (T). We create feature vectors for recommended pipelines by embedding the Scikit-learn documentations for pre-processor or feature selector and estimator (which is unique within each pipeline). Again, we the BERT embedding, to form 1024-dimensional emhedding ((M(()(D))) E IR. IR:104 for each pipeline, where C' ranges over the AutoML methods O, S. A, and T. The best-performing pipeline pp* returned by any AutoML system serves as our training label: we train our system recommend this pipeline.
Fused dataset representations. The combined representation of dataset D; with description M(D)) fuses together the dataset description embedding and data meta-features using a neural network (whose weights are learned);
F. = (IF., E R2512
(1)
We also represent the dataset and its best pipeline by fusing this representation with the pipeline embedding using a second neural network:
u go (e.. 12
(2)
Experimentally, these fused representations improve performance compared concatenation.
Algorithm Zero-shot AutoMI. pre-processing
Input:
training datasets Output: features (FMs. Fr. (P..))iie. for i to do compute embedding of description FMs E(Mi)
compute data meta- ceatures FD: for all C O. S, A. T do compute recommended pipeline Pc((D))
compute performance dataset R(Pc. D:) end for
select best performing pipeline P.))) embed pipeline Fp. (P,) = E(M(P. end for
3.2 GRAPH REPRESENTATION
We build a graph G = E) where each node e V represents the dataset Dj and has feature vector V,.
The feature vector v; = FPrDDl E R156 for node representing dataset D; with description M(D,) concatenates the fused dataset representation (described above) Fj R512 and the pipeline embedding Fp E(P,) I 1024 for the pipeline P. that performed best on the dataset. During training, we mask the pipeline embedding from the feature vector and learn to predict a node using the GNN.
Nodes.
Edges. To compute the edges of the graph we compute the distance d between each pair of datasets i, as d = ||F; FFfl= where F; and F, are the fused dataset representations (described above) for the datasets. Two datasets are connected by an edge if dataset neighbors of dataset or vice versa. In our experiments, we chose k 20: found that our method reasonably robust to the choice k; that Euclidean distance outperforms cosine similarity; and that a k-NN graph outperforms threshold-basee graph.
one of the nearest
At training time, we build this graph on the training datasets. At test time, given a new test dataset, dynamically connect the new node to the graph using its fused feature representation M[e FP...) to choose edges. Notice that the edges for the new dataset are chosen quickly, without fitting single machine learning model.
3.3 NEURAL NETWORK ARCHITECTURE
The neural networks we train for zero-shot AutoML consist of two fusion networks and a graph attention network (a type of GNN). The fusion networks are used to capture the non-linear inter-
5

---

Under review as a conference paper at ICLR 2021
actions between the features corresponding the dataset description, the data meta-features, and the pipeline embedding. The GNN predicts the best pipeline for a new dataset based the weights optimized during training described next.
Graph Attention Network. graph attention network (GAT) (Veličković et al. 2018) is used to predict the best pipeline for new dataset. Each layer / = 1,.., L of the GNN updates the feature vector at the i-th node as:
u au::Wu!-
aiy aeyWus JEN(()
E
where W is learnable weight matrix, (i) are the neighbors of the i-th node, and aiy the attention coefficients, defined as:
exp(o(2 [Wu, Wuyl)
ai EREN(O) exp(o(=" [Wu,, Wus)))
where = a learnable vector, and (.) is the leaky ReLU activation function.
Our GNN consists of GAT layers. The last layer of GNN is softmax which computes vector of probabilities over pipelines. Hence the output of the GAT probability distribution over pipelines for each node. The network recommends the pipeline that maximizes this probability. Alternatively, we may sample from this probability distribution to obtain several pipelines that can be combined into an ensemble.
3.4 TRAINING AND TESTING
Training. Our training process is illustrated in Figures 1 and 2 and described in Algorithm At each training iteration, we randomly select a node i. We mask the pipeline embedding of the the pipeline with best performance
i-th node u, go (IF 0))- The true label is defined among the four AutoML systems P. (D;) on the i-th dataset. The resulting problem is multi-class classification problem with many classes there are distinct algorithms.
The loss function is defined by cross-entropy between the probability P of predicted algorithm P(D.) and one-hot encoding of the best algorithm P. (D,):
m L(P(D.), Σ yı log(p:).
(5)
1=1
Fo Fi F..c)))
Fr falF D.mma
d(F,,,))
Fasts
Tradiesur
D
Bouttpiie
FP.(D))
Figure 2: Illustration of zero-shot AutoML dataset graph construction and prediction.

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Zero-shot AutoML training
Input:
training datasets, descriptions M(D.))e....
Output: datasets graph G. GNN hw,s, fusion networks fo and go.
pre-process: compute [FMs; FD., Fr.chhee
initialize fusion networks weights Φb, 0. initialize GNN weights W. for each backprop iteration do generate updated datasets graph G: for i = to 72 do
compute fused representation F, fo(FDuM)) end for
compute pairwise distances ((FF: for i 1 to n do connect node to k-NN nodes (1) end for
select random node compute u; = g 0)
for all * do compute u go (F,; Fr. Fr. (12))
end for predict best pipeline PDD:) = hw;= (u: (usbeN()
compute loss L(P(DD,) (D.))
update weights end for
Testing. Our testing process is illustrated the right path of Figure and Algorithm Given a new dataset 'D and description M, we compute the description embedding FM and data meta- features FD and the fused dataset representation F. We use this representation edges of this new node in the graph of all datasets. Next, we add the new node, with features == ([F, 0)), the current graph, replacing the embedding of the pipeline with the zero vector. Finally, we use the graph neural network to recommend pipeline for the test dataset.
compute the
Algorithm Zero-shot AutoML testing
Input: dataset D., description (M.)), datasets graph G. GNN, S.L. é V (disjoint train and test).
Output: predict hest pipeline ((D..) for task on dataset. generate node in G: compute embedding of description FM E(M(D.)) compute data meta-features Fo compute fused representation F = (F FM) connect node i to k-NN nodes ((),,, V = (t). compute u: (F 0) predict best pipeline P(D.) (u,, {u, }SN())
Notice our method does not need to complete even single model fit to recommend a model with hyper-parammters. On the other hand, we must fit the model (to learn the parameters) on the dataset predict output values for new input data. Our method can always recommend model in seconds, but training is still needed for prediction.
RESULTS
Table shows our results for representative set of test datasets, comparing our approach with state-of-the-art AutoML. systems and baselines. For each dataset (row), Table [ reports the mean evaluation accuracy of different AutoML methods. Digure@ccompant
the of accuracy on the test set between our zero-shot approach given 3 seconds of computation, and other AutoML systems and random forest baseline given minute of computation. Our new zero-shot AutoML approach the only AutoML system that provides predictions within 3 seconds. OBOE requires at least 20 seconds to perform predictions, and then only on a few of the datasets. AlphaD3M reaches performance slightly better than our approach, however given minute of computation. See the supplementary

---

Under review as a conference paper at ICLR 2021
3 seconds
minute
Datasel
Zero Shot
Linear
Random Forest
OBOE
AutoSkleam
TPOT
AIphaD3M
Random Forest
Lymph
0.867
0.670
0.800
0.800
0.867
0,933
0.867
Heaut-C C
0.774
0.710
0.806
0.742
0.806
0.742
0.871
0.839
Vehicle
Hayes- Roth
0.769
0.404
0.731
0.692
0.712
0.654
0.750
0.769
Colloges
0.838
0.726
0.786
0.803
0.812
0.812
0.838
0.812
KCI
0.872
0.848
0.829
0.251
0.872
0.877
0.886
0.872
Banana
0.745
0.551
0.881
0.808
0.898
0.911
0.911
0.900
Cardi
1.000
1,000
Chae 9
0.935
0.954
0.954
0.935
0.954
0.889
0.972
0.916
0.952
1.000
L.000
0.905
Wall-Robot
1,000
0.907
1,000
1,000
1,000
1,000
1,000
0.998
Cardi-Multi
0.995
0.869
0.986
0,995
0.986
0,995
0,995
0.995
BachChoral
0.776
0.580
0.786
0.212
0.817
0.774
0.797
0.787
Cjs
0.982
0.846
0.971
0.978
1,000
1,000
1,000
0.925
LED-Display
Wine Quality
0.686
0.451
0.678
0.484
0.982
0.686
0.714
0.692
SpeedDating
0.851
0.870
0.847
0.837
0.865
0.835
0.870
0.862
Mofa
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
Table 2: Comparison of testing performance and time between AutoML systems and baselines: zero-shot approach given 3 seconds and AutoSklearn, OBOE, TPOT, and AlphaD3M given minute. Testing time for predicting the machine learning algorithm is milliseconds. Testing time for running the predicted machine learning algorithm and computing performance is 3 seconds. Our new zero-shot AutoML approach the only AutoML system that provides predictions within seconds.
material for additional results which validate the performance of our method. First, our zero-shot method generally outperforms other simple baselines given the same amount of computation time. Second, our zero-shot method, in 3 seconds, gives results comparable to state-of-the-art AutoML systems given I minute.
Zero Shat Aucon a3 socore))
Zero Shot RuaML seconcs)
Shet AatoMi seconds)
0.8
3.8
0.8
0.6
0.8
0.4
0.
6.0
8
Zeru-Shul
0.2 6.4 06 0'3 1.3
022 044 06 0' 1.3
0.2 0.4 0.5 6.8 1.0
022 044 0.6 C.8 1.0
6.2 0.1 0.6 C.U 1.0 0
AutoSileam (1 minuto)
OBDE (1 minute)
TPOT (1 minuce;
AlchaDaR (1 minutel
Random Forest (1 minute)
Figure 3: Comparison of accuracy on test set between our zero-shot approach given 3 seconds of computation and other AutoML systems and Random Forest baseline given 1 minute of computa- tion. Our zero-shot approach matches the performance of baselines while running 20 times faster.
5 CONCLUSIONS AND FUTURE WORK
We introduce a new zero-shot approach to AutoML that is able to recommend a good pipeline to use for given dataset in real-time. Our system builds graph from both NLP text embedding of the dataset and pipeline descriptions as well as data meta-features and uses a graph neural network to predict the best pipeline for a given dataset. Our approach matches the performance of other state-of-the--rt AutoML systems and is significantly faster, reducing running time from minutes to seconds and prediction time from minutes to milliseconds. Future work will extend our approach to handle different types of data, including audio and images. In addition, we envision extension to superviee AutoML by using a GNN to embed a large unsupervised set of datasets without pipelines, such the 25 Million datasets available on Google dataset search (Brickley et al. 2019), together with a small supervised set of datasets with AutoML pipelines. Finally, we will make our data, models, and code public upon publication.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dan Brickley, Matthew Burgess, and Natasha Noy. Google Dataset Search: Building search engine for datasets in an open web ecosystem. In The World Wide Web Conference, pp. 1365-1377 2019.
BYU-DML. BYU's python library of useable tools for metalearning, 2019. URL https://
github coombyu-- dml http:///w...aaaaeeee//e///a//a/a
Boyuan Chen, Harvey Wu, Warren Mo, Ishanu Chattopadhyay, and Hod Lipson. Autostacker: A compositional evolutionary learning system. The Genetic and Evolutionary Computation Confer- ence, 2018
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Iddo Drori, Yamuna Krishnamurthy, Remi Rampin, Raoni Lourenco, Jorge One, Kyunghyun Cho, Claudio Silva, and Juliana Freire. AlphaD3M: Machine learning pipeline synthesis. In ICMI,
International Workshop Automated Machine Learning, 2018
Iddo Drori, Yamuna Krishnamurthy, Remi Rampin, Raoni Lourenco, Kyunghyun Cho, Claudio Silva, and Juliana Freire. Automatic machine learning by pipeline synthesis using model-based reinforcement learning and a grammar. In ICML International Workshop on Automated Machine Learning. 2019a.
Iddo Drori, Lu Liu, Sharath Koorathota, Nian Yi, Jie Li, Antonio Khalil Moretti, Juliana Freire, and Madeleine Udell. Automl using metadata language embeddings. In NeurlPS Workshop on Meta-Learning, 2019b.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, pp. 2962-2970, 2015.
Nicolo Fusi, Rishit Sheth, and Melih Elibol. Probabilistic matrix factorization for automated ma- chine learning. In Advances in Neural Information Processing Systems, pp. 3348-3357, 2018.
Xin He, Kaiyong Zhao, and Xiaowen Chu. AutoML: A survey of the state-of-the-arr arXiv preprint arXiv: 1908. 00709, 2019.
Efi Kokiopoulou, Anja Hauth, Luciano Sbaiz, Andrea Gesmundo, Gabor Bartok, and Jesse Berent. Fast task-aware architecture inference. arXiv preprint arXiv:1902.0578) 2019.
Tomas Mikolov, G.s Corrado, Kai Chen, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In International Conference Learning Representations, pp. 1-12, 01 2013.
Mitar Milutinovic, Atilim Gunes Baydi, Robert Zinkov, William Harvey, Dawn Song, and Frank Wood. End-to-end training of differentiable pipelines across machine learning frameworks. In NIPS Workshop Autodiff, 2017.
Deepak Mukunthu. Announcing automated ML capability Azure Machine Learning, 2019. URL
https://azure. mi www.iirross.. crosoft. com
Randal S Olson and Jason H Moore. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Automated Machine Learning, pp. 151-160. Springer, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2019.

---

Under review as a conference paper at ICLR 2021
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM.. Training multi-billion parameter language models using GPU model parallelism. arXiv preprint arXiv:/909.000
2019.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pp. 2951-2959, 2012.
Francisco Solis and Roger J-B Wets. Minimization by random search techniques. Mathematics of Operations Research, 6(1):19-30, 1981.
Thomas Swearingen, Will Drevo, Bennett Cyphers, Alfredo Cuesta-Infante, Arun Ross, and Kalyan Veeramachaneni. ATM: distributed, collaborative, scalable system for automated machine learning. In IEEE International Conference Big Data, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998 60086 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations, 2018.
Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell. OBOE: Collaborative filtering for AutoMI. model selection. In International Conference on Knowledge Discovery & Data Mining, pp. 1173-1183 1833, 2019.
Chengrun Yang, Ziyang Wu, Jicong Fan, and Madeleine Udell. AutoML pipeline selection: Effi ciently navigating the combinatorial space. In ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining (KDD), 2020.
Marc-André Zöller and Marco Huber. Benchmark and survey on automated machine learning. arXiv preprint ar 1rX1v:1904. 2019.
10


Reviews:
[{"review_id": "egPsFurA02", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary\nThe authors propose an AutoML by combining graph neural network (GNN) and transformer-based embeddings, the recommended pipelines from existing four AutoML systems (alphaD3M, TPOT, OBOE, autoSklearn) into its design. Specifically, the proposed framework is novel because\n- it creates the dataset description embedding by BERT, extract meta features of the dataset, and generate data embedding from description embedding and the meta features.\n- it will select the best pipeline predicted from the four AutoML frameworks and then convert the pipeline into pipeline embedding.  The best pipeline is used as the ground truth label for training the GNN.\n- it creates a graph with datasets as nodes where edges denote similarity between dataset embeddings.\n- the dataset graph is used as input to GNN (specifically, Graph Attention Network) to learn the GNN weights\n- for a new test dataset, the trained GNN simply predicts a ML pipeline by dynamically adding it as node in the dataset graph via K-nearest neighbors\n\n##### Clarity\nThe paper is mostly well written and easily understandable even for readers who are new to AutoML. The framework is clearly explained in Section 3. Result section in Section 4 is limited under the page constraint. However, I strongly feel the authors should bring **TABLE 3** and corresponding discussions in **Appendix C: Computation Time** from Supplementary into the main paper especially when the proposed model pitches real-time performance.\n\n\n##### Main Questions\n1. In Introduction,  \"....develops automated ***solution**  that ~~**exploit**~~  ***leverage** human expertise....\"  The only human component I found the system is using is the textual descriptions provided for the dataset and estimators. The authors should rephrase the statement not to imply human-in-loop system.\n2. In Methods,  Figure 1: Please use notations in figure consistent with Table 1.\n3. Algorithm 3 (Testing): Is the proposed system scalable since for every new test dataset? Because the proposed method needs to compute K-NN to create new edges.\n4. Figure 3: What does each point in the graph denote? \n5. What was the performance of other AutoML systems in Table 2 at 3 seconds ? Also, what is the reason for having a 1 minute time limit for these baselines? I understood limit of 3 seconds for zero-shot AutoML since *Cardi* test dataset took ~3 seconds as per Table 3 in Supplementary.\n6. What were the ML pipelines predicted by the proposed zero-shot AutoML system and other AutoML systems for the test datasets in Table 2 ?\n7. What are the precise meaning of running time and prediction time in the paper? In Abstract, authors stated the proposed framework can reduce running time to second and prediction time to milliseconds. In Section 3.4, authors said the method always recommend models in 3 seconds.  Is the running time the amount of Time the system take to determine the best pipeline? Authors highlight training time is still needed for the prediction. How much time authors give to the best pipeline for training?\n8. No real-time applications are demonstrated in the experiments. When the framework only take  3 seconds to return the best pipeline for a dataset, it just means that the search time is short enough. The real-time is misleading readers that authors tackle the real-time applications with AutoML. Their experiments are not strong enough to support the \"real-time\".\n9. The time measurement is not fair for baselines. Authors  ignore the pre-processing time. The running time for predicting a ML pipeline is short from the framework. However, if Auto-Sklearn does not count the time to build surrogate model for Bayesian optimization, Auto-Sklearn can also predict a pipeline in a short time. Authors should clarify how they measure time for predicting a pipeline.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bbjTf5SLlqM", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a very interesting idea of utilizing the documentation for the data and the operators in the pipeline to generate meta-features for meta-learning. This is a very novel application of graph neural networks GNNs and language models for AutoML meta-learning. This view of meta-learning takes a very intuitive on a very high level. The use of the outputs of existing AutoML systems (such as auto-sklearn, TPOT, etc) is also very intuitive and well motivated. All these intuitive ideas are put together into a novel AutoML recommendation architecture making use of modern deep learning components.\n\n\nHowever, in its current form, I am somewhat unsatisfied with the paper. Firstly, even with all this sophisticated meta-learning, it appears that a RandomForest is fairly competitive at 3 seconds (5 wins, 4 ties, 9 losses to zero-shot) and at 1 minutes (5 wins, 8 ties, 5 losses to zero-shot), which is somewhat disappointing. Given that XGBoost and LightGBM are usually more efficient and accurate, chances are, they would improve upon RandomForests. It seems that the gain from the proposed sophisticated meta-learning is not fully realized with the empirical evaluation.\n\n\nBeyond this, there are various choices made in the empirical evaluation which is not very well justified, and various technical details that are missing in the paper:\n\n- Why are we focusing on 3 seconds, why not 1 second or 1 minute? Also, does 3 seconds constitute just the recommendation of pipelines or the recommendation + training. Different parts of the text imply different things -- End of section 3 says \"Our method can always recommend a model in 3 seconds, but training is still needed for prediction\". Then in the beginning of section 4, the authors say \"Our new zero-shot AutoML approach is the only AutoML system that provides predictions within 3 seconds\".\n- In the introduction, the authors say \"... runs the pipeline and tunes its hyper-parameters within three seconds\", but nowhere in the text could I precisely find what happens once the system recommends a pipeline -- is it just trained as it is or is hyper-parameter optimization or CASH run on it?\n- Is the generation of the meta-data (document parsing, statistics generation), which can take significant amount of time for decent sized data sets, part of the 3 seconds?\n- For each node during meta-learning (training of the GNN and the fusion networks), the potential pipelines  $\\mathcal{P}_{\\mathcal{C}} ( \\mathcal{D} )$ are the ones generated by auto-sklearn, TPOT, OBOE, etc, on $\\mathcal{D}$ and \"the output of the GAT is a probability distribution over pipelines for each node\". \n  - What is the set of potential pipelines over which the output probability distribution is defined for a new test data set $\\mathcal{D}{\\mbox{new}}$ for which we have no potential pipelines $\\mathcal{P}_{\\mathcal{C}} ( \\mathcal{D}{\\mbox{new}} )$. This is not clear from the description and is a very important part of the \"zero-shot\" learning.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LyBF1WL702t", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an efficient way to automatically choose the best or most suitable pipeline for different datasets. The proposed method can accelerate the AutoML using a pre-trained meta module. In particular, the AutoML job of a new supervised learning task can be accomplished without model evaluations, namely zero-shot / real-time AutoML. The meta module is constructed as a graph structure in which each node represents a dataset used for meta-training. \n \nPros:\n \n1. The target problem in this paper is practical, and the proposed model, GAT is simple and seems effective. \n\n2. The model is built on top of existing AutoML systems and only needs to predict the most suitable pipeline chosen among the existing AutoML systems. \n \n3. The authors claim that this procedure can be done in 3 seconds, i.e., real-time while the other pipelines require 1 minute to train the comparative performance. \n\nCons:\n \n1. The feature extractor method is based on a Github project, i.e., BYU. However, the Github project doesn’t provide any text description of the technique. To be a self-contained paper, the authors should clarify the details of the method, especially in this paper where the feature is a key input to the GAT. Thus, a more detailed elaboration on how to extract the feature is recommended.\n \n2. The proposed work lacks theoretical or empirical support on why the current feature of the datasets can indicate the choice of the pipeline. For example, given two supervised learning tasks with high dimension features and a limited number of instances, the similarity on two datasets cannot directly indicate using the same pipeline. \n \n3. The proposed method is a systematic solution for AutoML rather than an AutoML method. The paper’s solution leverages multiple existing AutoML pipelines, including OBOD, AutoSklearn, AlphaD3M, and TPOT, and then recommend a pipeline for a new coming supervised learning task by measuring the similarity between the new task’s dataset and all seen dataset. To be specific and avoid misguiding, I’d suggest to name it “Selecting AutoML pipeline”.\n \n4. Compare to other AutoML methods, the proposed dataset embedding-based AutoML solution highly relies on how the new task’s dataset is linked to the existing meta-training datasets. This assumption is difficult to be satisfied in real-world applications.\n \n5. The experimental analysis of this paper is weak. The dataset graph should be visualized with colored points for each pipeline.\n \n6. The description of the dataset is also a key feature to calculate the similarity of two supervised learning tasks. In real work applications, the dataset description is not always well-written in natural language. The use of a dataset description does not always benefit; sometimes, it will be harmful to measuring the similarity of datasets.\n \n \nBelow are some questions:\n \n1. In section 3.1, the authors claim the pipeline embedding is derived by a transformer and form a 1024-dimensional embedding. Can the authors elaborate more details on the inputs of the transformer? Is the representation semantically meaningful?\n \n2. Why the proposed AutoML system can infer faster than only using either one of the pipelines as in Table 2? Do the 3 seconds\nonly account for the procedure of choosing the best pipeline options? \n \n3. The authors claim this is a real-time autoML system. I am curious about if the system can handle the query in parallel? For example, there are multiple nodes added into the graph at the same time, can the proposed handle this case and how to build the graph for multiple new nodes, and how is the performance?\n \n4. In the current system, the authors assume a fixed number of AutoML systems, i.e., OBOE, AutoSklearn, TPOT, AlphaD3M, Random Forest. Why these five systems selected? What if the user wants to add more systems? Are there any selection criteria for the base systems?\n \n5. I am curious to see some analysis / visualizations on how the current system chooses the best pipeline and which pipeline is typically suitable for datasets of what kinds of types? Maybe a few examples are sufficient to give an intuitive sense.\n \n6. To support the reproducibility of the paper, do the authors have any plan to open source or release the experimental source codes?", "extra_scores": {}, "extra_text": {}}, {"review_id": "C58D6vPrEGA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem that the authors attempt to solve is to determine what ML pipeline will perform best on any new dataset, without incurring in the extra cost of actually running a large number of such pipelines, as is typically done in AutoML algorithms. The way this paper tackles the problem is to train a neural network that given a new dataset as input, will output a pipeline that is predicted to perform well on that dataset. This neural network is trained on other datasets, for which high performing pipelines are already known. Predicting a pipeline for a new dataset thus only require a forward pass through their NN.\n\nThe core details of the solution describes are the following:\n* To get labels (in the form of well performing pipelines) for the training set, each dataset in the training set is run through 4 existing AutoML solutions, and the best performing pipeline among those generated this way is selected and used as a ground truth label\n* They generate an embedding for each dataset, that combines metafeatures and a textual describtion of the dataset\n* As an additional step, the embedding describing the dataset is transformed into a graph in the following way: all existing datasets (during prediction including the new dataset) are represented as a graph, where each node corresponds to a dataset and is connected to the closest k graphs according to the distance of the embeddings\n* The above graph is then passed as input to a GNN, to predict the final pipeline\n* In addition, each node has a feature describing the selected pipeline, with the value set to 0 for the current active node that is being passed as the input dataset, as far as I can tell this is the only way in which the NN is aware what the current active node is.\n* The problem is framed as a multiclass classification problem with a discrete set of possible pipelines to choose from, thus the final softmax layer of the network simply outputs a probability distribution over this discrete set of pipelines\n\n\nThe idea is interesting, although it is not really clear why the exact proposed architecture would be the best way to approach the issue. From a theoretical point of view it's not clear why a simpler architecture would not work better, or as at least give comparable performance. Even more importantly, the experimental results leave some doubts about the effectiveness of the method.\n\nI found the exposition also hard to follow in places. More precisely, a lot of the discussion of the use of GNNs in the introduction is hard to follow without the context of how the GNN is used exactly, which is explained only later in the text. The second paragraph of the introduction, about the use of GNNs, is very hard to understand without first knowing what exactly is passed in as input and output of the NN. Same for the forth paragraph. I would suggest to leave a lot of these discussions out or put them later in the paper. More in general the proposed architecture is not really clear until the reader reads all the details of the paper, a better and more self-contained introduction would help the comprehension of the paper.\n\nThe biggest concern for me however are the results, in particular how the method, although relatively complex, seems to perform no better (or only very slightly better) than random forests with default parameters, as seen in the results of Table 2, and Figure 2 of the Supplemental material. In addition the authors seem to have tested many different versions of the algorithm, as per the supplemental material. This in itself is positive and I appreciate the ablation studies. The issue is that even if all variations would all perform no better than the random forest baseline, some architectures would appear to perform better than it just due to noise. Making the result even more clear cut. \n\nOverall I believe the approach of the authors is interesting, but further work seems necessary to reach a publishable paper, both from an exposition point of view, and from an experimental one. Unfortunately the paper misses strong and compelling evidence that the method really gives results than simple baselines.\n\n#### Additional comments / Questions\n\nWhen describing embeddings: the following sentence is not very clear unless one already is familiar with the relevant literature, in which case it does not add anything: \". An unsupervised corpus of text is transformed into a\nsupervised dataset by defining content-target pairs along the entire text: for example, target words\nthat appear in each sentence, or target sentences which appear in each paragraph\"\n\nIt would be good to have the code available, as quite a few low level details are left out of the paper.\n\nHow big is the variety of pipelines selected? This would be interesting to know.\n\nIn \"Embeddings\": corpra -> corpora\n\nIn \"Methods\": their relationships -> it is not really clear what this means in this contest\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5McocLMV5b_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n--Summary--\nThis paper introduces a method for matching an ML pipeline to a dataset using the text features of the pipeline of and the dataset along with some additional metadata about the features of the data. They evaluate and train their model on ~20 datasets and ~5 pipelines.\n\n--Strengths--\nThe authors introduce an approach to pick a training pipeline for a dataset. I like the fact that the authors provided useful illustrations and clear notation.\n\n--Weaknesses--\nThis paper suffers from several issues:\n\n**Low impact**\nThe claim of real-time AutoML is exaggerated:\na) The proposed approach does not work for general AutoML tasks and search spaces such as architecture search, swapping pipeline pieces in and out. Instead it matches datasets to a handful of pre-existing pipelines (~5).\n\nb) Training the model can severely dominate the suggestion time for the proposed algorithm.  This approach focuses on the speed of the first suggestion (zero-shot setting). \n\nc) How do you recover if you make a bad decision about a dataset/algo? The proposed algorithm is not iterative.\n\nd) Are text features sufficient? What about datasets and pipelines which sound similar but are totally different.\n\n**Low novelty**\na) This is a method for matching a dataset to a training pipeline largely using text features from their descriptions/documentation. This is a common information retrieval task.\n\nb) It’s unclear why a GNN and a transformer is needed for a simple feature matching task. Standard retrieval and ranking algorithms such as vector dot product, TF-IDF or LSA could have sufficed.\n\nc) Edges in the GNN are chosen using a kNN threshold applied to the node representations. No structure learning is performed.\n\n**Weak results**\na )The number of datasets (around 20) and pipelines (around 5)  that this approach is evaluated on is woefully inadequate. The results could easily suffer from overfitting.\n\nb) For some datasets test performance is already at 100% e.g. Cardi, Wallrobot, Mofn. These seem like easy tasks. More complex and difficult datasets are needed.\n\nc) No comparison with other methods. e.g. simple information retreival method such as vector dot product, tf-idf or LSA.\n\n**Technical issues:**\na) Description of training setup is lacking. Is it training on the same set of datasets? If so there’s leak of training -> test data.\n\nb) What is the total training time? If it dominates the suggestion. Then this method is not useful since compute has already been spent on fully training the datasets.\n\n---What can make this paper better?---\n- The core approach fundamentally needs improvement. A simple pipeline to dataset matching is unlikely to have a significant contribution to science or application.\n\n-  The authors need much higher quality results\na) Many more datasets\n\nb) Difficult datasets: Not ones that achieve close to 100% accuracy.\n\nc) Many more pipelines or complexity of pipeline components.\n\n- To claim Real-time AutoML the authors need to demonstrate that the method works\n\na) For general AutoML tasks and search spaces such as architecture search, swapping pipeline pieces in and out.\n\nb) The method is iterative in nature. Otherwise it is impossible to recover from poor choices of pipelines.\n\nc) The real-time claim implies an “online” setting, wherein results are fed back and suggestions are offered in real time. The proposed approach does not have the ability to learn on the fly. The current GNN and Transformer representations will be costly to re-train.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an algorithm for real-time auto-ML based on zero-shot learning, which matches an ML pipeline to a dataset via the meta-features of the pipeline and the dataset.  It aims to address an important problem, and the idea of the proposed solution seems interesting. However, there are several issues with the current draft: 

(1) A central question is how to justify the complexity of the proposed solution, given that simple alternatives, such as Random Forest, can achieve similar or better results than the proposed method. 

(2) The proposed solution lacks in novelty contribution of methodology and theoretical justification 

(3) Various technical details are missing in the current draft.

The authors' feedback did not fully address the issues above. We hope that reviews can help the authors improve the draft for a strong publication in the future. 
Decision: Reject