Learning meta-features for AutoML

1 INTRODUCTION . Getting the peak performance of an algorithm portfolio on a particular problem instance is acknowledged a main bottleneck in domains ranging from Constraint Programming and Satisfiability to Machine Learning ( Rice , 1976 ; Hutter et al. , 2009 ; Stern et al. , 2010 ; Kotthoff , 2014 ; Bergstra et al. , 2011 ; Feurer et al. , 2015 ; Hazan et al. , 2018 ; Fusi et al. , 2018 ; Yang et al. , 2019 ) . Early approaches have been investigating the use of general performance models ( Rice , 1976 ) , estimating a priori the performance of any algorithm on any problem instance , where each problem instance is described by a vector of so-called meta-features , and the performance model is learned in this meta-feature space . In the context of supervised Machine Learning , many meta-features have been manually designed to describe datasets ( Caliński & Harabasz , 1974 ; Vilalta , 1999 ; Bensusan & Giraud-Carrier , 2000 ; Pfahringer et al. , 2000 ; Peng et al. , 2002 ; Ali & Smith , 2006 ; Song et al. , 2012 ; Bardenet et al. , 2013 ; Feurer et al. , 2014 ; 2015 ; Pimentel & de Carvalho , 2019 ; Lorena et al. , 2019 ) . After a series of international AutoML challenges , aimed to automating the selection and tuning of ML pipelines1 ( Hutter et al. , 2019 ; Guyon et al. , 2019 ) , it seems that a general accurate performance model can hardly be based on these meta-features ( Misir & Sebag , 2017 ) ( Section 2 ) : for instance the challenge-winner AutoSkLearn ( Feurer et al. , 2015 ) relies on Bayesian optimization and iteratively learns and exploits one performance model specific to the dataset at hand ; PMF ( Fusi et al. , 2018 ) uses a probabilistic collaborative filtering approach , where the cold-start problem is handled as in AutoSkLearn ; OBOE ( Yang et al. , 2019 ) likewise uses a collaborative filtering approach , combined with active learning . Nevertheless , the definition of good meta-features remains desirable for two reasons . The first motivation remains to achieve AutoML with a decent performance vs cost trade-off . Relevant meta-features are expected to define a reliable topology on the dataset space , such that two datasets are close iff the best hyper-parameter configurations for these datasets are close . Such a topology would support an inexpensive and efficient AutoML strategy : selecting the best hyper-parameter configurations of the nearest neighbor ( s ) of the current dataset . The second motivation is to better understand the dataset space w.r.t . a given ML algorithm , to estimate its intrinsic dimension and to appreciate the distribution of the ML benchmark suites thereon . 1An ML pipeline consists of a data preparation stage followed by the model learning stage . Each stage involves a number of options and a varying number of hyper-parameters , depending on the former selected options . Terms ML pipeline and ML algorithm will be used interchangeably in the remainder of the paper . This paper presents the Meta-learning for Tabular Data ( METABU ) approach , formalizing and tackling the construction of good meta-features relatively to an ML algorithm A as an Optimal Transport ( OT ) problem ( Cuturi , 2013 ; Peyré & Cuturi , 2019 ) . Formally , METABU considers two representations of the datasets : the basic one consists of 135 manually designed meta-features ( Appendix E ) . The target one , out-of-reach except for the datasets in the benchmark suite , represents a dataset as the distribution of the hyper-parameter configurations of A yielding the top performances for this dataset . Optimal Transport is used to find a linear transformation of the basic meta-features , such that the resulting Euclidean distance emulates the Wasserstein-Gromov distance ( Mémoli , 2011 ) on the target representation ( Section 3 ) . Overall , Metabu learns once for all new meta-features , aimed to capture the topology and neighborhoods corresponding to the target representation . These meta-features can be computed from scratch for each new dataset . This approach contrasts with Yang et al . ( 2019 ) and Fusi et al . ( 2018 ) that both require a cold-start phase , launching configurations for each new dataset and using their performance to find the representation of the new dataset . The contribution of METABU is threefold . Firstly , the METABU meta-features define an efficient topology , that can be used to sample the most promising hyper-parameter region for new datasets . Secondly , the relevance of these meta-features is demonstrated as they can be used as representation space to initialize AutoSkLearn ( Feurer et al. , 2014 ) and PMF ( Fusi et al. , 2018 ) : the hybrid approaches AutoSkLearn+METABU and PMF+METABU , are shown to significantly outperform AutoSkLearn and PMF on the OpenML CC-18 ( Bischl et al. , 2019 ) benchmark . Lastly , the approach provides some hints into the AutoML problem , enabling to estimate the intrinsic dimensionality ( Facco et al. , 2017 ) of the dataset space w.r.t . an ML algorithm : the higher the dimensionality , the more complex the algorithm . It is interesting to compare the intrinsic dimensions of the OpenML CC18 w.r.t . AutoSkLearn ( Feurer et al. , 2015 ) , SVM ( Boser et al. , 1992 ) , or Random Forest ( Breiman , 2001 ) . Furthermore , the METABU meta-features can be inspected and confirm some `` tricks of the trade '' about when an algorithm does well . The paper is organized as follows . Section 2 briefly discusses related work and introduces OT formal background for the sake of self-containedness . Section 3 describes the METABU algorithm . In Section 4 , the merits of the METABU meta-features are empirically demonstrated on configuration selection and optimization tasks , comparatively to the state of the art . Lastly , we discuss how the METABU meta-features provide an interpretable description of the niche of the considered ML algorithms . 2 RELATED WORK AND FORMAL BACKGROUND . AutoML & meta-features Most ML meta-features ( Caliński & Harabasz , 1974 ; Vilalta , 1999 ; Bensusan & Giraud-Carrier , 2000 ; Pfahringer et al. , 2000 ; Peng et al. , 2002 ; Ali & Smith , 2006 ; Song et al. , 2012 ; Bardenet et al. , 2013 ; Feurer et al. , 2015 ; 2014 ; Pimentel & de Carvalho , 2019 ; Lorena et al. , 2019 ) have been manually designed to describe supervised datasets based on descriptive statistics , information theory ( quantifying relationships among features/labels ) , geometrical structure of the dataset , and landmarking ( performance of cheap classifiers such as linear discriminant and decision trees ) . In the neighbor fields of Satisfiability or Constraint Programming , circa one hundred meta-features have also been manually designed ( Nudelman et al. , 2004 ; Xu et al. , 2008 ) . In contrast to the efficiency of SAT or CP meta-features however ( Kotthoff , 2014 ) , the AutoML search can hardly rely on the only metric defined from the ML meta-features after ( Misir & Sebag , 2017 ; Muñoz et al. , 2018 ) ; in practice , they are often used to initialize the optimization search ( Feurer et al. , 2015 ) . Another approach is to learn meta-features , e.g . by making strong assumptions on the performance model ( Hazan et al. , 2018 ) or by leveraging distributional neural networks ( de Bie et al. , 2019 ; Maron et al. , 2020 ) . In the latter case , these meta-features are functions of the dataset distribution and consist of the last layer of a distributional NN trained in view of a particular task . Dataset2Vec ( Jomaa et al. , 2021 ) learns meta-features to detect whether two data patches ( subset of samples described by a subset of features ) are extracted from the same whole dataset . OTDD ( Alvarez-Melis & Fusi , 2020 ) uses OT to learn a mapping over the joint feature and label spaces . A significant drawback of distributional neural network approaches , limiting their ability to handle general tabular datasets ( with widely varying number of features , missing values , heterogeneous variables ) is due to the shortage of training ( meta ) -samples . Neural networks notoriously need large amounts of samples to be efficiently trained , while AutoML benchmarks include less than a hundred datasets . For this reason , the proposed METABU approach proceeds by building upon existing meta-features . Optimal Transport Let ( Ωx , dx ) and ( Ωy , dy ) denote compact metric spaces , and x and y distributions2 respectively defined on Ωx and Ωy . The search space Γ ( x , y ) is the space of all distributions on Ωx × Ωy with marginals x and y . Let the transport cost function c : Ωx × Ωy 7→ IR+ be a scalar function on Ωx × Ωy3 . The OT problem consists in finding a distribution in Γ ( x , y ) yielding a minimal transport cost expectation ( Peyré & Cuturi , 2019 ) ; this minimal transport cost expectation defines the Wasserstein distance of x and y : dqW ( x , y ) = min γ∈Γ ( x , y ) IE ( x , y ) ∼γ [ c q ( x , y ) ] 1/q , with q a positive real number , set to 2 in the following . Another OT-based distance is the Gromov-Wasserstein distance ( GW ) ( Mémoli , 2011 ) , measuring how well a distribution in Γ ( x , y ) preserves the distances on both Ωx and Ωy , akin a rigid transport between both domains : dqGW ( x , y ) = min γ∈Γ ( x , y ) IE ( x , y ) ∼γ , ( x′y′ ) ∼γ [ |dx ( x , x′ ) − dy ( y , y′ ) |q ] 1/q . The Fused Gromov-Wasserstein ( FGW ) distance ( Titouan et al. , 2019 ) combines both these distances . Definition 1 The Fused q-Gromov-Wasserstein distance is defined on Ωx × Ωy as follows : dqFGW ; α ( x , y ) = min γ∈Γ ( x , y ) ( 1− α ) ∫ Ωx×Ωy cq ( x , y ) dγ ( x , y ) 1 q ︸ ︷︷ ︸ Wasserstein Loss + α ∫ Ωx×Ωy ∫ Ωx×Ωy |dx ( x , x′ ) − dy ( y , y′ ) |qdγ ( x , y ) dγ ( x′ , y′ ) 1 q ︸ ︷︷ ︸ Gromov-Wasserstein Loss ( 1 ) α ∈ [ 0 , 1 ] is a trade-off parameter : For α = 0 ( resp . α = 1 ) , the fused q-Gromov-Wasserstein distance is exactly the q-Wasserstein distance dqW ( resp . the q-Gromov-Wasserstein distance d q GW ) . The Wasserstein distance and variants thereof have been successfully used to evaluate the `` alignment '' among datasets , e.g . between the source and the target datasets in the context of domain adaptation ( Courty et al. , 2017 ) or transfer learning ( Alvarez-Melis & Fusi , 2020 ) . FGW distance has been used to enforce the consistency of the latent space when jointly training several Variational Auto-Encoders ( Xu et al. , 2020 ; Nguyen et al. , 2020 ) . METABU will likewise take inspiration from OT to create a bridge between two representations of the datasets : the basic one , and the target one , critically using both GW and FGW distances .

Summary: In this paper, the authors address the AutoML problem, which aims to automatically select the best ML algorithm and its hyperparameter configuration for a dataset, and propose an approach to this problem that learns meta-features of the dataset. The proposed method, MetaBu, learns new meta-features by optimal transport according to the space of distributions of hyperparameter configurations. Meta-features in MetaBu is known only once and induce a topology in a set of data sets. Experiments on the OpenML CC-18 benchmark have shown that MetaBu meta-features can improve the performance of the state-of-the-art AutoML systems AutoSklearn and Probabilistic Matrix Factorization. Furthermore, the examination of MetaBu meta-features provides hints on when an ML algorithm will work. Finally, a topology based on MetaBu meta-features can estimate the intrinsic dimension of the OpenML benchmark for a given ML algorithm or pipeline.