Learning meta-features for AutoML

1 INTRODUCTION . Getting the peak performance of an algorithm portfolio on a particular problem instance is acknowledged a main bottleneck in domains ranging from Constraint Programming and Satisfiability to Machine Learning ( Rice , 1976 ; Hutter et al. , 2009 ; Stern et al. , 2010 ; Kotthoff , 2014 ; Bergstra et al. , 2011 ; Feurer et al. , 2015 ; Hazan et al. , 2018 ; Fusi et al. , 2018 ; Yang et al. , 2019 ) . Early approaches have been investigating the use of general performance models ( Rice , 1976 ) , estimating a priori the performance of any algorithm on any problem instance , where each problem instance is described by a vector of so-called meta-features , and the performance model is learned in this meta-feature space . In the context of supervised Machine Learning , many meta-features have been manually designed to describe datasets ( Caliński & Harabasz , 1974 ; Vilalta , 1999 ; Bensusan & Giraud-Carrier , 2000 ; Pfahringer et al. , 2000 ; Peng et al. , 2002 ; Ali & Smith , 2006 ; Song et al. , 2012 ; Bardenet et al. , 2013 ; Feurer et al. , 2014 ; 2015 ; Pimentel & de Carvalho , 2019 ; Lorena et al. , 2019 ) . After a series of international AutoML challenges , aimed to automating the selection and tuning of ML pipelines1 ( Hutter et al. , 2019 ; Guyon et al. , 2019 ) , it seems that a general accurate performance model can hardly be based on these meta-features ( Misir & Sebag , 2017 ) ( Section 2 ) : for instance the challenge-winner AutoSkLearn ( Feurer et al. , 2015 ) relies on Bayesian optimization and iteratively learns and exploits one performance model specific to the dataset at hand ; PMF ( Fusi et al. , 2018 ) uses a probabilistic collaborative filtering approach , where the cold-start problem is handled as in AutoSkLearn ; OBOE ( Yang et al. , 2019 ) likewise uses a collaborative filtering approach , combined with active learning . Nevertheless , the definition of good meta-features remains desirable for two reasons . The first motivation remains to achieve AutoML with a decent performance vs cost trade-off . Relevant meta-features are expected to define a reliable topology on the dataset space , such that two datasets are close iff the best hyper-parameter configurations for these datasets are close . Such a topology would support an inexpensive and efficient AutoML strategy : selecting the best hyper-parameter configurations of the nearest neighbor ( s ) of the current dataset . The second motivation is to better understand the dataset space w.r.t . a given ML algorithm , to estimate its intrinsic dimension and to appreciate the distribution of the ML benchmark suites thereon . 1An ML pipeline consists of a data preparation stage followed by the model learning stage . Each stage involves a number of options and a varying number of hyper-parameters , depending on the former selected options . Terms ML pipeline and ML algorithm will be used interchangeably in the remainder of the paper . This paper presents the Meta-learning for Tabular Data ( METABU ) approach , formalizing and tackling the construction of good meta-features relatively to an ML algorithm A as an Optimal Transport ( OT ) problem ( Cuturi , 2013 ; Peyré & Cuturi , 2019 ) . Formally , METABU considers two representations of the datasets : the basic one consists of 135 manually designed meta-features ( Appendix E ) . The target one , out-of-reach except for the datasets in the benchmark suite , represents a dataset as the distribution of the hyper-parameter configurations of A yielding the top performances for this dataset . Optimal Transport is used to find a linear transformation of the basic meta-features , such that the resulting Euclidean distance emulates the Wasserstein-Gromov distance ( Mémoli , 2011 ) on the target representation ( Section 3 ) . Overall , Metabu learns once for all new meta-features , aimed to capture the topology and neighborhoods corresponding to the target representation . These meta-features can be computed from scratch for each new dataset . This approach contrasts with Yang et al . ( 2019 ) and Fusi et al . ( 2018 ) that both require a cold-start phase , launching configurations for each new dataset and using their performance to find the representation of the new dataset . The contribution of METABU is threefold . Firstly , the METABU meta-features define an efficient topology , that can be used to sample the most promising hyper-parameter region for new datasets . Secondly , the relevance of these meta-features is demonstrated as they can be used as representation space to initialize AutoSkLearn ( Feurer et al. , 2014 ) and PMF ( Fusi et al. , 2018 ) : the hybrid approaches AutoSkLearn+METABU and PMF+METABU , are shown to significantly outperform AutoSkLearn and PMF on the OpenML CC-18 ( Bischl et al. , 2019 ) benchmark . Lastly , the approach provides some hints into the AutoML problem , enabling to estimate the intrinsic dimensionality ( Facco et al. , 2017 ) of the dataset space w.r.t . an ML algorithm : the higher the dimensionality , the more complex the algorithm . It is interesting to compare the intrinsic dimensions of the OpenML CC18 w.r.t . AutoSkLearn ( Feurer et al. , 2015 ) , SVM ( Boser et al. , 1992 ) , or Random Forest ( Breiman , 2001 ) . Furthermore , the METABU meta-features can be inspected and confirm some `` tricks of the trade '' about when an algorithm does well . The paper is organized as follows . Section 2 briefly discusses related work and introduces OT formal background for the sake of self-containedness . Section 3 describes the METABU algorithm . In Section 4 , the merits of the METABU meta-features are empirically demonstrated on configuration selection and optimization tasks , comparatively to the state of the art . Lastly , we discuss how the METABU meta-features provide an interpretable description of the niche of the considered ML algorithms . 2 RELATED WORK AND FORMAL BACKGROUND . AutoML & meta-features Most ML meta-features ( Caliński & Harabasz , 1974 ; Vilalta , 1999 ; Bensusan & Giraud-Carrier , 2000 ; Pfahringer et al. , 2000 ; Peng et al. , 2002 ; Ali & Smith , 2006 ; Song et al. , 2012 ; Bardenet et al. , 2013 ; Feurer et al. , 2015 ; 2014 ; Pimentel & de Carvalho , 2019 ; Lorena et al. , 2019 ) have been manually designed to describe supervised datasets based on descriptive statistics , information theory ( quantifying relationships among features/labels ) , geometrical structure of the dataset , and landmarking ( performance of cheap classifiers such as linear discriminant and decision trees ) . In the neighbor fields of Satisfiability or Constraint Programming , circa one hundred meta-features have also been manually designed ( Nudelman et al. , 2004 ; Xu et al. , 2008 ) . In contrast to the efficiency of SAT or CP meta-features however ( Kotthoff , 2014 ) , the AutoML search can hardly rely on the only metric defined from the ML meta-features after ( Misir & Sebag , 2017 ; Muñoz et al. , 2018 ) ; in practice , they are often used to initialize the optimization search ( Feurer et al. , 2015 ) . Another approach is to learn meta-features , e.g . by making strong assumptions on the performance model ( Hazan et al. , 2018 ) or by leveraging distributional neural networks ( de Bie et al. , 2019 ; Maron et al. , 2020 ) . In the latter case , these meta-features are functions of the dataset distribution and consist of the last layer of a distributional NN trained in view of a particular task . Dataset2Vec ( Jomaa et al. , 2021 ) learns meta-features to detect whether two data patches ( subset of samples described by a subset of features ) are extracted from the same whole dataset . OTDD ( Alvarez-Melis & Fusi , 2020 ) uses OT to learn a mapping over the joint feature and label spaces . A significant drawback of distributional neural network approaches , limiting their ability to handle general tabular datasets ( with widely varying number of features , missing values , heterogeneous variables ) is due to the shortage of training ( meta ) -samples . Neural networks notoriously need large amounts of samples to be efficiently trained , while AutoML benchmarks include less than a hundred datasets . For this reason , the proposed METABU approach proceeds by building upon existing meta-features . Optimal Transport Let ( Ωx , dx ) and ( Ωy , dy ) denote compact metric spaces , and x and y distributions2 respectively defined on Ωx and Ωy . The search space Γ ( x , y ) is the space of all distributions on Ωx × Ωy with marginals x and y . Let the transport cost function c : Ωx × Ωy 7→ IR+ be a scalar function on Ωx × Ωy3 . The OT problem consists in finding a distribution in Γ ( x , y ) yielding a minimal transport cost expectation ( Peyré & Cuturi , 2019 ) ; this minimal transport cost expectation defines the Wasserstein distance of x and y : dqW ( x , y ) = min γ∈Γ ( x , y ) IE ( x , y ) ∼γ [ c q ( x , y ) ] 1/q , with q a positive real number , set to 2 in the following . Another OT-based distance is the Gromov-Wasserstein distance ( GW ) ( Mémoli , 2011 ) , measuring how well a distribution in Γ ( x , y ) preserves the distances on both Ωx and Ωy , akin a rigid transport between both domains : dqGW ( x , y ) = min γ∈Γ ( x , y ) IE ( x , y ) ∼γ , ( x′y′ ) ∼γ [ |dx ( x , x′ ) − dy ( y , y′ ) |q ] 1/q . The Fused Gromov-Wasserstein ( FGW ) distance ( Titouan et al. , 2019 ) combines both these distances . Definition 1 The Fused q-Gromov-Wasserstein distance is defined on Ωx × Ωy as follows : dqFGW ; α ( x , y ) = min γ∈Γ ( x , y ) ( 1− α ) ∫ Ωx×Ωy cq ( x , y ) dγ ( x , y ) 1 q ︸ ︷︷ ︸ Wasserstein Loss + α ∫ Ωx×Ωy ∫ Ωx×Ωy |dx ( x , x′ ) − dy ( y , y′ ) |qdγ ( x , y ) dγ ( x′ , y′ ) 1 q ︸ ︷︷ ︸ Gromov-Wasserstein Loss ( 1 ) α ∈ [ 0 , 1 ] is a trade-off parameter : For α = 0 ( resp . α = 1 ) , the fused q-Gromov-Wasserstein distance is exactly the q-Wasserstein distance dqW ( resp . the q-Gromov-Wasserstein distance d q GW ) . The Wasserstein distance and variants thereof have been successfully used to evaluate the `` alignment '' among datasets , e.g . between the source and the target datasets in the context of domain adaptation ( Courty et al. , 2017 ) or transfer learning ( Alvarez-Melis & Fusi , 2020 ) . FGW distance has been used to enforce the consistency of the latent space when jointly training several Variational Auto-Encoders ( Xu et al. , 2020 ; Nguyen et al. , 2020 ) . METABU will likewise take inspiration from OT to create a bridge between two representations of the datasets : the basic one , and the target one , critically using both GW and FGW distances .

Summary: This paper focuses on the AutoML problem for tabular data and proposes a meta-learning based novel solution. They consider the optimal transport to define distances between two datasets, utilizing the Wasserstein-Gromov distance between the distribution of the top performing hyperparameters for the respective datasets. Given this distance, they propose learning a linear transformation of existing dataset meta-features such that the Euclidean distance between a pair of datasets in this transformed space is proportional to their Wasserstein-Gromov distance. This method is termed Metabu.  The empirical evaluation compares Metabu to existing meta-learning schemes on (i) their ability to capture the desired Wasserstein-Gromov distance, (ii) their ability to find better hyperparameters via sampling without an underlying optimizer, and (iii) their ability to find better seed hyperparameters for hyperparameter optimizers. The results on the OpenML CC-18 suite with 3 machine learning models indicate that Metabu significantly improves upon existing meta-learning schemes. The paper also demonstrates how the learned linear transformation of existing dataset meta-features allow us understand the importance of different existing dataset meta-features and how these vary between machine learning models.