Real-Time AutoML

1 INTRODUCTION . A data scientist facing a challenging new supervised learning task does not generally invent a new algorithm . Instead , they consider what they know about the dataset and which algorithms have worked well for similar datasets in past experience . Automated machine learning ( AutoML ) seeks to automate these tasks to enable widespread use of machine learning by non-experts . A major challenge is to develop fast , efficient algorithms to accelerate applications of machine learning ( Kokiopoulou et al. , 2019 ) . This work develops automated solutions that exploit human expertise to learn which datasets are similar and what algorithms perform best . We use a transformer-based language model ( Devlin et al. , 2018 ) allowing our AutoML system to process text descriptions of datasets and algorithms , and a feature extractor ( BYU-DML , 2019 ) to represent the data itself . Using such models for our representation brings in large-scale data . We allow to train our model on other existing AutoML system solutions , specifically AutoSklearn ( Feurer et al. , 2015 ) , AlphaD3M ( Drori et al. , 2018 ) , OBOE ( Yang et al. , 2019 ) , and TPOT ( Olson & Moore , 2019 ) , tapping into their diverse set of solutions . Our approach fuses these representations ( dataset description , data , AutoML pipeline descriptions ) and represents datasets as nodes in a graph of datasets . Generally , graph neural networks are used for three main tasks : ( i ) node prediction , ( ii ) link prediction , and ( iii ) sub-graph or entire graph classification . In this work we use a GNN for node prediction , which predicts the machine learning pipeline for an unseen dataset . Specifically , we use a graph attention network ( GAT ) ( Veličković et al. , 2018 ) with neighborhood aggregation , in which an attention function adaptively controls the contribution of neighbors . An advantage of using a GNN for AutoML is boosting AutoML performance by sharing information between datasets ( graph nodes ) : including description and algorithm , by message passing between the nodes in the graph . In addition , GNNs generalize well to a new unknown dataset using the aggregated weights learnt over the training datasets . GNN weights are shared with the test dataset for prediction . GNNs generalize to entire new sets of datasets . Finally , prediction is in real-time , within milliseconds . A simple idea is to use machine learning pipelines that performed well ( for the same task ) on similar datasets . What constitutes a similar dataset ? The success of an AutoML system often hinges on this question , and different frameworks have different answers : for example , AutoSklearn ( Feurer et al. , 2015 ) computes a set of meta-features , which are features describing the data features , for each dataset , while OBOE ( Yang et al. , 2019 ) uses the performance of a few fast , informative models to compute latent features . More generally , for any supervised learning task , one can view the list of recommended algorithms generated by any AutoML system as a vector describing that task . This work is the first to use the information that a human would check first : a summary description of the dataset and algorithms , written in free text . These dataset features induce a metric structure on the space of datasets . Under an ideal metric , a model that performs well on one dataset would also perform well on nearby datasets . The methods we develop in this work show how to learn such a metric using the recommendations of an AutoML framework together with the dataset description . We provide a new zero-shot AutoML method that predicts accurate machine learning pipelines for an unseen dataset and classification task in real-time and runs the pipeline in a few seconds . We use a transformer-based language model to embed the description of the dataset and pipelines and a feature extractor to compute meta-features from the data . Based on the description embedding and meta-features , we build a graph as the input to a graph neural network ( GNN ) . Each dataset represents a node in the graph , together with its corresponding feature vector . The GNN is trained to predict a machine learning pipeline for a new node ( dataset ) . Therefore , given a new dataset , our real-time AutoML method predicts a pipeline with good performance within milliseconds . The running time of our predicted pipeline is a few seconds and the accuracy of the predicted pipeline is competitive with state-of-the-art AutoML methods that are given one minute . This work makes several contributions by using language embeddings and GNNs for AutoML for the first time , and leveraging existing AutoML systems . The result is a real-time high-quality AutoML system . Real-time . Our system predicts a machine learning pipeline for a new dataset in milliseconds and then runs the pipeline and tunes its hyper-parameters within three seconds . This reduces computation time by orders of magnitude compared with state-of-the-art AutoML systems , while improving performance . GNN architecture . Our work achieves real-time AutoML by introducing several architectural components that are new to AutoML . These include embeddings for dataset descriptions and algorithm descriptions using a state-of-the-art transformer-based language model in addition to ( standard ) embeddings for data ; a non-Euclidean embedding of datasets as a graph ; and a predictive model employing a GNN on the graph of datasets . Importantly , the GNN recommends a pipeline for a new dataset by adding a node to the graph of datasets and sharing the GNN weights with the new node . Using the information and relationships between all datasets boosts AutoML performance . Embeddings . Bringing techniques from NLP to AutoML , specifically using a large-scale transformer-based language model to embed the description of the dataset and algorithms , brings in information from a large corpra of text . This allows our zero-shot AutoML to train on a small set of datasets with state-of-the-art test set performance . Leveraging existing AutoML systems . Our flexible architecture can use pipeline recommendations from any number of other AutoML systems to improve performance . 2 RELATED WORK . AutoML is an emerging field of machine learning with the potential to transform the practice of Data Science by automatically choosing a model to best fit the data . Several comprehensive surveys of the field are available ( He et al. , 2019 ; Zöller & Huber , 2019 ) . Processing each dataset in isolation . The most straightforward approach to AutoML considers each dataset in isolation and asks how to choose the best hyper-parameter settings for a given algorithm . While the most popular method is still grid search , other more efficient approaches include Bayesian optimization ( Snoek et al. , 2012 ) or random search ( Solis & Wets , 1981 ) . Recommender systems . These methods learn ( often , exhaustively ) what algorithms and hyperparameter settings performed best for a training set-of-datasets and use this information to select better algorithms on a test set without exhaustive search . This approach reduces the time required to find a good model . An example is OBOE ( Yang et al. , 2019 ; 2020 ) , which fits a low rank model to learn the low-dimensional representations for the models ( or pipelines ) and datasets that best predict the cross-validated errors , among all bilinear models . To find promising models for a new dataset , OBOE runs a set of fast but informative algorithms on the new dataset and uses their cross-validated errors to infer the feature vector for the new dataset . A related approach ( Fusi et al. , 2018 ) using probabilistic matrix factorization powers Microsoft Azure ’ s AutoML service ( Mukunthu , 2019 ) . Search trees . Auto-Tuned Models ( Swearingen et al. , 2017 ) represent the search space as a tree with nodes being algorithms or hyper-parameters and searches for the best branch using a multiarmed bandit . Model-based reinforcement learning . AlphaD3M ( Drori et al. , 2018 ; 2019a ) formulated AutoML as a single player game . The system uses reinforcement learning with self-play and a pretrained model which generalizes from many different datasets and similar tasks . Genetic programming . TPOT ( Olson & Moore , 2019 ) and Autostacker ( Chen et al. , 2018 ) use genetic programming to choose both hyper-parameter settings and a topology of a machine learning pipeline . TPOT represents pipelines as trees , whereas Autostacker represents them as layers . Bayesian optimization . AutoSklearn ( Feurer et al. , 2015 ) chooses a model for a new dataset by first computing ( ad hoc ) data meta-features to find nearest-neighbor datasets . The best-performing methods on the neighbors are refined via Bayesian optimization and used to form an ensemble . Differentiable programming . End-to-end learning of machine learning pipelines is performed using differentiable primitives ( Milutinovic et al. , 2017 ) forming a directed acyclic graph . Algorithmic primitives . One major factor in the performance of an AutoML system is the base set of algorithms it can use to compose more complex pipelines . For a fair comparison , in our numerical experiments we compare our proposed methods only to other AutoML systems that use Scikit-learn ( Pedregosa et al. , 2011 ) primitives . Embeddings . Language has a common unstructured representation as a sequence of words , sentences , or paragraphs . The most significant recent progress in NLP is large-scale transformer-based models and embeddings ( Devlin et al. , 2018 ; Shoeybi et al. , 2019 ; Raffel et al. , 2019 ) based on attention mechanisms ( Vaswani et al. , 2017 ) . An unsupervised corpus of text is transformed into a supervised dataset by defining content-target pairs along the entire text : for example , target words that appear in each sentence , or target sentences which appear in each paragraph . A language model is first trained to learn a low dimensional embedding of words or sentences followed by a map from low dimensional content to target ( Mikolov et al. , 2013 ) . This embedding is then used on a new , unseen and small dataset in the same low-dimensional space . Our work uses such embeddings for automatic machine learning . In a similar fashion to recent work ( Drori et al. , 2019b ) we use an embedding for the dataset and algorithm descriptions . In this work we model the non-linear interactions between these embedding using a neural network as well . 3 METHODS . Our zero-shot AutoML predicts a machine learning pipeline for a classification task on a dataset based on the dataset description and data , and based on other datasets , their relationships , and their recommended pipelines by AutoML systems . We embed the dataset description and extract data meta-features to construct a graph of datasets where each node represents a dataset . The graph is processed using a graph neural network ( GNN ) . Each node of the graph contains a feature vector which is the fusion of the description embedding and data meta-features , and the GNN node representations includes other AutoML solutions . The machine learning pipeline for a new dataset is predicted by the GNN . A detailed architecture is illustrated in Figure 1 and described by Algorithms 2 and 3 . The notation used in this work are given in Table 1 . 3.1 PRE-PROCESSING . Our pre-processing consists of ( i ) dataset description embedding ; ( ii ) dataset meta-feature extraction ; and ( iii ) pipeline computation and description embeddings , as described next and summarized in Algorithm 1 . Dataset Description Embedding . We create a feature vector by embeding the descriptionM ( D ) of each dataset as a 1024-dimensional vector FM = E ( M ( D ) ) ∈ R1024 using BERT ( Devlin et al. , 2018 ) . The supplementary material shows examples of dataset descriptions embedded using our approach . Data meta-features . We compute meta-features FD ∈ R148 for the dataset D using a feature extractor ( BYU-DML , 2019 ) , restricting to meta-features that can be computed in one second on any of the datasets used in our experiments . Meta-features include statistics of the datasets and results of simple algorithms . Pipelines and pipeline embedding . For each dataset , we compute the recommended pipeline returned by AutoML systems OBOE ( O ) , AutoSklearn ( S ) , AlphaD3M ( A ) , and TPOT ( T ) . We create feature vectors for recommended pipelines by embedding the Scikit-learn documentations for pre-processor or feature selector and estimator ( which is unique within each pipeline ) . Again , we use the BERT embedding , to form a 1024-dimensional embeddingE ( M ( PC ( D ) ) ) ∈ R1024 for each pipeline , where C ranges over the AutoML methods O , S , A , and T. The best-performing pipeline P ? returned by any AutoML system serves as our training label : we train our system to recommend this pipeline . Fused dataset representations . The combined representation of dataset Di with description M ( Di ) fuses together the dataset description embedding and data meta-features using a neural network ( whose weights are learned ) : Fi = fφ ( [ FDi , FMi ] ) ∈ R512 . ( 1 ) We also represent the dataset and its best pipeline by fusing this representation with the pipeline embedding using a second neural network : ui = gθ ( [ Fi , FP ? ( Di ) ] ) ∈ R 512 ( 2 ) Experimentally , these fused representations improve performance compared to concatenation . Algorithm 1 Zero-shot AutoML pre-processing Input : training datasets { ( Di , Mi ) } i∈V . Output : features { FMi , FDi , FP ? ( Di ) } i∈V . for i = 1 to n do compute embedding of description FMi = E ( Mi ) compute data meta-features FDi for all C ∈ O , S , A , T do compute recommended pipeline PC ( Di ) compute performance on dataset R ( PC , Di ) end for select best performing pipeline P ? ( Di ) embed pipeline FP ? ( Di ) = E ( M ( P ? ( Di ) ) ) end for

Summary: The problem that the authors attempt to solve is to determine what ML pipeline will perform best on any new dataset, without incurring in the extra cost of actually running a large number of such pipelines, as is typically done in AutoML algorithms. The way this paper tackles the problem is to train a neural network that given a new dataset as input, will output a pipeline that is predicted to perform well on that dataset. This neural network is trained on other datasets, for which high performing pipelines are already known. Predicting a pipeline for a new dataset thus only require a forward pass through their NN.