Learning meta-features for AutoML
Authors: Herilalaina Rakotoarison
Louisot Milijaona
Andry RASOANAIVO
Michele Sebag
Marc Schoenauer
Venue: ICLR 2022 Spotlight (2022)
Keywords: AutoML
Meta-features
Hyper-parameter Optimization
Optimal Transport

Abstract: This paper tackles the AutoML problem, aimed to automatically select an ML algorithm and its hyper-parameter configuration most appropriate to the dataset at hand. The proposed approach, MetaBu, learns new meta-features via an Optimal Transport procedure, aligning the manually designed \mf s with the space of distributions on the hyper-parameter configurations. MetaBu meta-features, learned once and for all, induce a topology on the set of datasets that is exploited to define a distribution of promising hyper-parameter configurations amenable to AutoML. Experiments on the OpenML CC-18 benchmark demonstrate that using MetaBu meta-features boosts the performance of state of the art AutoML systems, AutoSklearn (Feurer et al. 2015) and Probabilistic Matrix Factorization (Fusi et al. 2018). Furthermore, the inspection of MetaBu meta-features gives some hints into when an ML algorithm does well. Finally, the topology based on MetaBu meta-features enables to estimate the intrinsic dimensionality of the OpenML benchmark w.r.t. a given ML algorithm or pipeline. The source code is available at https://github.com/luxusg1/metabu.

Published as a conference paper at ICLR 2022
LEARNING META-FEATURES FOR AUTOML
Herilalaina Rakotoarison
1+
Louisot Milijaona" 2*
Andry Rasoanaivo
Michèle Sebag*
Mare Schoenauer'
1 TAU, LISN-CNRS-INRIA, Université Paris-Saclay, Orsay, France 2 MISA, LMI, Université d' Antananarivo, Ankatso, Madagascar
ABSTRACT
This paper tackles the AutoML problem, aimed to automatically select an ML algorithm and its hyper-parameter configuration most appropriate to the dataset hand. The proposed approach, MetaBu, learns new meta-features via an Optimal Transport procedure, aligning the manually designed meta-features with the space of distributions on the hyper-paramette configurations. MetaBu meta-features, learned once and for all, induce topology on the set of datasets that is exploited to define distribution of promising hyper-parameter configurations amenable AutoMI. Experiments on the OpenML CC-18 benchmark demonstrate that using MetaBu meta-feature features boosts the performance of state of the art AutoML systems, AutoSkLearn (Feurer et al. 2015) and Probabilistic Matrix Factorization (Fusi et al. 2018). Furthermore, the inspection of MetaBu meta-features gives some hints into when an ML algorithm does well. Finally, the topology based on MetaBu meta- features enables to estimate the intrinsic dimensionality of the OpenML. benchmark w.r.t. a given MI. algorithm or pipeline. The source code is available at htttp://ggihub.com//uuuuu// 1/metabu
INTRODUCTION
Getting the peak performance of an algorithm portfolio on a particular problem instance is acknowl- edged a main bottleneck in domains ranging from Constraint Programming and Satisfiability to
Machine Learning RRice 1976 Hutter et al. 2009; Stern et al.
2014 Bergstra et al.
2011 Feurer et al. 2015 Hazan et al. 2018 Fusi al.) 2018
Yang et al. 2019). Early approaches have been investigating the use of general performance models (Rice: 1976), estimating a priori the
performance of any algorithm on any problem instance, where each problem instance described by a vector of so-called meta-features, and the performance model is learned in this meta-feature space.
In the context of supervised Machine Learning, many meta-featuree have been manually designed
to describe datasets (Caliński & Harabasz 1974; Vilalta 1999 Bensusan & Giraud-Carrier 2000
Pfahringer et al. 2000;; Peng et 20.2|2002|AA Ali & Smith 22066] [Song et al.] 2012 Bardenet et al. 2013
Feurer et al.) 2014, 2015]
2015) Pimentel & de Carvalho 2019; Lorena et al. 2019). After series of
international AutoML challenges, aimed to automating the selection and tuning of ML pipeline
(Hutter et al. 2019 Guyon et al.) 2019). it seems that general accurate performance model can hardly be based on these meta-features Misir & Sebag 2017) (Section :: for instance the challenge-winnn AutoSkI (Feurer et al. 2015) relies on Bayesian optimization and iteratively learns and exploits one performance model specific to the dataset at hand; PMF (Fusi et al., 2018) uses a probabilistic collaborative filtering approach, where the cold-start problem is handled as in AutoSkLearn; OBOE (Yang et al., 2019) likewise uses a collaborative filtering approach, combined with active learning.
Nevertheless, the definition of good meta-features remains desirable for two reasons. The first motivation remains to achieve AutoML with a decent performance vs cost trade-off. Relevant
* Equul contribution whwrilalaina.rakottoaaoooo
@iniiii@@iiiaia @inria fr miliaonalluuisstt@@@ gmmailcommm
An ML pipeline consists of a data preparation stage followed by the model learning stage. Each stage involves number of options and varying number of hyper-parameter depending on the former selected options. Terms MI. pipeline and ML. algorithum will be used interchangeably in the remainder of the paper.

---

Published as a conference paper at ICLR 2022
meta-features are expected to define reliable topology the dataset space, such that two datasets are close iff the best hyper-parameter configurations for these datasets are close. Such a topology would support an inexpensive and efficient AutoML strategy: selecting the best hyper-parameter configurations of the nearest neighbor(s) of the current dataset.
The second motivation is to better understand the dataset space w.r.t. given MI. algorithm, to estimate its intrinsic dimension and to appreciate the distribution of the ML benchmark suites thereon.
This paper presents the Meta-learning for Tabular Data (METABU) approach, formalizing and tackling the construction of good meta-features relatively to an ML algorithm A as an Optimal Transport (OT) problem (Cuturi 2013; Peyré & Cuturi 2019). Formally, METABU considers two representations of the datasets: the basic one consists of 135 manually designed meta-features (Appendix ED. The target one, out-of-reach except for the datasets the benchmark suite, represents a dataset as the distribution of the hyper-parametee configurations of A yielding the top performances for this dataset. Optimal Transport is used to find linear transformation of the basic meta-features. such that the resulting Euclidean distance emulates the Pesserstein-Gromon distance (Mémoli 2011 on the target representation (Section 3 Overall, Metabu learns once for all new meta-features, aimed to capture the topology and neighborhoods corresponding to the target representation. These meta-features can be computed from scratch for each new dataset. This approach contrasts with Yang al. (2019) and Fusi et al. (2018) that both require cold-start phase, launching configurations for each new dataset and using their performance to find the representation of the new dataset.
The contribution of METABU is threefold. Firstly, the METABU meta-features define efficient topology, that can be used to sample the most promising hyper hypre-parameter region for new datasets. Secondly, the relevance of these meta-features demonstrated as they can be used as representation space to initialize AutoSkLearn (Feurer et al.) 2014) and PMF (Fusi et al. 2018): the hybrid approaches AutoSkLeam+METABUU and PMF+METABU, are shown to significantly outperform AutoSkLearn and PMF on the OpenML CC-18 (Bischl et al. 2019) benchmark. Lastly, the approach provides some hints into the AutoMI. problem, enabling to estimate the intrinsic dimensionality (Facco et al., (017) of the dataset space w.r.t. an MI. algorithm: the higher the dimensionality, the more complex the algorithm. It is interesting to compare the intrinsic dimensions of the OpenML CC- 18 w.r.t. AutoSkLeam (Feurer et al.) 2015), SVM (Boser et al. [192), or Random Forest (Breiman 2001). Furthermore, the METABU feat-reatures can be inspected and confirm some "tricks of the trade about when an algorithm does well.
The paper is organized as follows. Section2briicth discusses related work and introduces OT formal background for the sake of seff-containedness. Section describes describeesss the METABU algorithm. In Section 4 the merits of the METABU meta-features are empirically demonstrated on configuration selection and optimization tasks, comparatively the state of the art. Lastly, we discuss how the METABU meta-features provide an interpretable description of the niche of the considered ML algorithms.
RELATED WORK AND FORMAL BACKGROUND
AutoML & meta-features
Most ML meta- -eatures (Caliñski & Harabasz 1974| Vilalta
1999
Bensusan & Giraud-Carrier 2000 Pfahringer et al.) 2000 Peng al. 2002 Ali & Smith 2006
Song et al.] 2012] Bardenet et af.) (013) et al. 2015 2014 Pimentel & de Carvalho 2019
Lorena al. (219) have been manually designed to describe supervised datasets based on descriptive statistics, information theory (quantifying relationships among features/labels), geometrical structure of the dataset, and landmarking (performance of cheap classifiers such as linear discriminant and decision trees). In the neighbor fields of Satisfiability or Constraint Programming, circa one hundred meta-features have also been manually designed (Nudelman al. [2004] [u et al. 2008). In contrast to the efficiency of SAT or CP meta-features however (Kotthofi) 2014), the A search can hardly rely on the only metric defined from the MI. meta-features after (Misir Sebag. 2017, Muñoz et al.) 2018); in practice, they are often used to initialize the optimization search (Feurer et al. 2015).
Another approach is to learn meta-features, e.g. by making strong assumptions on the performance model (Hazan et al. 2018) or by leveraging distributional neural networks (de Bie al. 2019 Maron et al. 2020). In the latter case, these meta-features are functions of the dataset distribution and consist of the last layer of distributional NN trained in view of particular task. Dataset2 Vec (Jomaa ctt al., (22T)) learns meta- featurfs to detect whether two data patches (subset of samples described by subset of features) are extracted from the same whole dataset. OTDD (Alvarez-Melis & Fusi
2

---

Published as a conference paper at ICLR 2022
2020) uses OT to learn mapping over the joint feature and label spaces. A significant drawback of distributional neural network approaches, limiting their ability to handle general tabular datasets (with widely varying number of features, missing values, heterogeneous variables) is due to the shortage of training (meta)-samples. Neural networks notoriously need large amounts of samples to be efficiently trained, while AutoMI. benchmarks include less than hundred datasets. For this reason, the proposed METABU approach proceeds by building upon existing meta-fcatures.
Optimal Transport Let (Ry. dz) and (Ny, dyy) denote compact metric spaces, and and y distri- butions respectively defined on f2 and Qy. The search space I'(x, y) is the space of all distributions on 22 By with marginals x and y. Let the transport cost function e: 22 S2y + IR" be scalar function on S2 x 2
The OT problem consists in finding a distribution in ('(x yielding a minimal transport cost expectation (Peyré & Cuturi 2019); this minimal transport expectation defines the Wasserstein min ((xx))>>( w)]2/9, with q positive real number, to yEr(X.y)
distance of x and y: div (x.y) y)
1 in the following.
Another OT-based distance the Gromov- Wasserstenn distance (GW) (Mémoli 2011). measuring how well distribution in I ['(x, y) preserves the distances both Stp and S2y. akin rigid transport dy (y. y/ ///
between both domains: dow y) min ((x')) [ (2, x²)
yET((xy)
The Fused Coomov-Wasserstein (FGW) distance (Titouan et al. 2019) combines both these distances.
Definition The Fused q- Wasserteenss distance is defined on S2y x Gy as follows:
/
d (x.))
min (1 a) (xT(X.y)
cx, ()).)))
P2 Kyy
Wasserstein Loss
(I)
7
+
De Doxly
Wusserstein
E [0, is a trade-off off parameter: For a (resp. 1), the fused q- g-Gromov- Wasserstein distance is exactly the q- g-Wasserslein distance dy (resp. the g-Gromov-
-.Gromov-Wassersteen Wasserstein distance d'ow).
The Wasserstein distance and variants thereof have been successfully used to evaluate the "alignment" among datasets, e.g. between the source and the target datasets in the context of domain adaptation (Courty et al.) 2017) or transfer learning (Alvarez-Melis enforce the consistency the latent space when jointly training several Variational Auto-Encoders (Xu et al. 2020 Nguyen et al.) (2020). METABU will likewise take inspiration from OT to create bridge between two representations of the datasets: the basic one, and the target one, critically using both GW and FGW distances.
Fusi 2020). FGW distance has been used
3 OVERVIEW OF METABU
Let A and O respectively denote an ML pipeline and its hyper-pann parameter configuration space; subseript omitted when clear from the context. Space θ embedded into the a-dimensional real-valued space R" using a one-hot encoding of Boolean and categorical hyper-parameters. After describing the principle of the approach, some key issues are detailed: the augmentation of the AutoML benchmark to avoid overfitting, and the setting of the number d of the METABU meta- features, estimated from the intrinsic dimensionality of the AutoML benchmark suite.
Distribbutions will be denoted in holdface
When Ω2 Q2 2 unless otherwise stated, the transport cost c(a,, is the Fuclidean distance d(z, y)-
3

---

Published as a conference paper at ICLR 2022
Principle. Intuitively, two representations can be associated with a dataset: The basic represen- kation Z E IR' of a dataset reports the values of the D manually designed meta-features for this dataset. By construction, it can be cheaply computed for any dataset. The target representation z of a dataset is the distribution on the space @ supported by the configurations yielding the best performances on this dataset. This precious target representation is unreachable in practice, but can be approached after the performances of the models learned with number of configurations (aka configuration performances) have been assessed. In practice, the configuration performances are only available for small number of datasets (more below). The difference between the basic and the target topologies is depicted on Fig. in θ space (projected on first two PCA eigenvectors).
In order to build a bridge between both representations, let us consider an intermediate representation derived from the target representation, mapping each (zi) (ggi on some u, E R²Z using distance-preservingg projection, e.g. Multi-Dimensional Scaling (MDS) (Cox & Cox 2001). to learn mapping vp : R + Rd from the basic representa- tion on the projected target representation space such that the (xxz))iss are aligned with the u,s in the sense of the q-Fused Gromov-Wassersee Wasserstein distance (Section 2 In brief, mapping d sends the basic meta-feature space on RA, such that the Euclidean metric on the (xx;) reflects the Euclidean metric on the u4s, itself reflecting the metric on the target 2,S. The descriptive features of the (axa),
Delaset
METABU tackles an Optimal Transport problem
Figure 1: Top configurations of datasets
A, B, and C, where B, in orange (resp. C, in green) is the nearest neighbor of A w.r.t. target (resp. basic) representation.
referred to as METABU meta-features, are meant to both be cheaply computable from the basic meta-features, and define Euclidean distance conducive to the AutoML task.
Augmenting the AutoML benchmark. The OpenML CC-18 18 (Bischl et al., 2019), to our knowl- edge the largest curated tabular dataset benchmark (that will be used in the experiments), contains n = 72 classification datasets; the target representation is available for 64 of them. The shortage of such datasets yields risk of overfitting the learned meta-features. This challenge is tackled by aug- menting the OpenMI. CC-18 benchmark suite, using a bootstrap procedure (Efron) 1979 The goal pave the meta-feature space more densely and more accurately than through c.g., perturbing the basic representation with Gaussian noise (the visualization of the augmented benchmark is displayed on Fig. Appendix
The algorithm The algorithm is provided the p = 1, 1,000 x n training datasets of the benchmark suite, augmented as described above (pseudo-code in Appendix B). The METABU meta-features are constructed in a 3-step procedure, illustrated on Fig. 2
Step 1: Target representation and Wasserstein distance. Considering the i-th training dataset, let θ C T denote the set of hyper-parameter configurations with performance in the top-L. known configuration performances (I. 20 in the experiments)
The target representation z; of the i-th dataset is the discrete distribution with support +,, The distance dl (z., 2;) is the W-Wasserstein distance among distributions (Section (
Step 2: Projecting the larget representation on IR The second step consists in projecting the z,s on Rd where d is identified using an intrinsic dimensionality procedure (details below), using Multi-DDmensional Scaling (Cox & Cox 2001), such that the distance d(u;, uz) approximates the 1- Wasserstein distance dly (z, 2,) (Fig. 2 leftmost and second subplots). Note that by construction, the u.s are defined up to an isometry on Rd
*For each É-size dataset E' in the benchmark suite, K = 1, 000 new datasets Fi.... Fx are generated, where Fi includes E examples selected in E uniformly with replacement. The basic representation of F, is computed, and its target representation is set to that of E.
$ 'Early attempts to define O in a more sophisticated way, e.g. using t-test to distinguish the "good" configurations from the others. led to an uninformative target representation. A tentative interpretation for this fact is that quite few OpenMI. datasets are very easy, leading to retain all configurations for these datasets and blurring the target representation.
4

---

Published as a conference paper at ICLR 2022
Target Representation
STEP
Metabu Representation
Basic Representation
Embedding
STEP 1
STEP
FGW loss lEq.
Figure 2: From basic to METABU meta-features using Fused Gromov- Wasserstein. Basic (respectively METABU) representations are depicted by circles (resp. squares). Target representations are depicted in the leftmost subplot. Neighbor datasets in the target space have same color in all subplots.
Step 3: Learning the METABU meta-features. Let Ei=10 8z; denote the uniform discrete distribution on IR" whose support is the set of p datasets using their basic representations.
Let = _/2 Ei= du, denote the uniform discrete distribution on IR whose support is the set of u,s defined above. The METABU meta-feature space is built by finding a mapping & from Rº Id that pushes the representation metric R that is, such that the image of via & is close as possible to and reflects its topology in the FGW sense (Fig, 2 rightmost and third subplots).
Formally, let 4/yx def overall optimization problem to find mapping 26" that minimizes the FGW distance between the distribution and the push distribution v
E (as)) be the push-forward distribution of x on Rd for a given db. The
arg min drow;a (12;x, u) A||bb|l
(2)
NEW
with À the regularization weight and ||भا|| the norm of the yi function. Note that, as and v are distributions on the same space R the transport cost c the Euclidean distance on Rdd
In the following, only linear mappings t are considered for the sake of avoiding overfitting and facilitating the interpretation of the METABU meta-features w.r.t. the manually designed meta-feature The norm of & is set to the L1 norm of its weight vector.
Taking inspiration from Xu et al, (2020). the efficient optimization of Eq. iis achieved using a bilevel optimization formulation. For given th, the inner optimization problem consists of minimizing drow,a (V/x, u) (Eq. [ This problem is solved using a proximal gradient method (Xu et al. 2019), along an iterative approach: given an estimation of the transport map 7(0), a sub-problem is defined to refine y, it solved using the Sinkhorn algorithm (Cuturi 2013). and its solution used compute ((++1) (the number of iterations is set 10 in the experiments).
The outer opúmization problem consists of optimizing ub: The transport matrix y is treated as constant, and the outer objective function (Eq, ( solved with ADAM optimizer (Kingma & Ba
2015) with learning rate 0.01, = 0.5 and X 0.001.
Intrinsic dimension of the space datasets The main hyper- parameter of METABU the number d of meta-features needed to approximate the target representation. Indeed, d depends on the considered algorithm A: the more diverse the target representations associated with datasets, the harder the AutoML problem, the higher d needs to be. In the other extreme case (all datasets have similar target representations), the AutoML problem becomes trivial.
To our best knowledge, measuring the intrinsic dimension of the dataset space w.r.t. learning algorithm has not been tackled in the literature. The approach proposed to do so builds on Levina & Bickel 2005) and (Facco et al. 2017). exploiting the fact that the number of points in hypersphere of radius r in dimension d increases like nd Formally, to each sample x is associated its first and
second nearest neighbors, respectively noted z(1) and (2) and let μ(x) =(=,=())
(2) be the ratio of their distances to r. With no loss generality, the samples are ordered by increasing value of μ (i.e.,
uxx:) S p(z,) for all j). Let d be the slope of the linear approximation of the 2D curve defined
by ((oog(u(z))) Iog(1 - < m). Then d provides guaranteed approximation of the
5

---

Published as a conference paper at ICLR 2022
intrinsic dimensionality of the manifold where the 2,8 family lives (Facco al. 2017 is commonplace to say that the good distance between any two items depends on the considered task. The original approach used in METABU in order estimate the intrinsic dimensionality of the dataset space, is to set the distance of two datasets to the 1- 1-Wasserstein distance among their target representations.
EXPERIMENTS
All materials (code, data, and instructions) are made available at httpp////ggtuub.ooo/uuuug/g///// Runtimes are measured on an Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz.
4.1 EXPERIMENTAL SETTINGS
Goals of experiment. The first goal is to assess the dataset neighborhoods induced by the METABU meta-feature (constructed on the top of the manually designed 135 meta-features from the literature) and the relevance of these dataset neighborhoods w.r.t. the AutoML problem. The performances are assessed against three baselines: AutoSkLearn meta-feature set (Feurer et al. 2014), Landmark (Pfahringer al. 2000) and SCOT (Bardenet al. 2013) meta-feature sets. All meta-feature detailed in Appendix E For Tasks 2 and 3 (see below), an additional baseline is based on the uniform sampling the hyper-parameter configuration space, for sanity check.
The second goal of experiments is to assess the sensitivity of METABU w.r.t its own two hyper- parameters, the weight a used balance the importance of the Wasserstein and Gromov- Wasserstein distances in FGW (Eq. [ and the regularization weight À involved in the optimization of & (Eq. 2 The third goal to gain some understanding of the dataset landscape, and see whether the METABU meta-features give some hints into when a given ML algorithm or pipeline does well (its niche).
Performance indicators. Three tasks are considered to investigate the relevance of the METABU meta-features. The performance indicators are measured using Leave- One-One---nee Out process (detailed in Appendix C
Task 1: Capturing the target topology. For each test dataset, one considers its nearest neighbors w.r.t. the target topology (the 1-Wasserstein metric on the target representation), and nearest neighbors w.r.t. the Euclidean distance on the METABU and meta-feature sets. The alignment between both ordered lists is measured using the normalized discounted cumulative gain over the first & neighbors (NDCG@) @k) (Burges et al. 2005), with 5 < k < 35. The performance indicator is the NDCG @k averaged on test datasets.
Task 2:
AutoML with no performance model (Initializationn. For each test dataset and each meta- feature set mff, let 2 be the distribution on the considered
configuration space:
1 Z
imf
crp(-e) z
where z, is the target representation of the é-th neighbor of the dataset w.r.t. Euclidean distance on the mff space, and 2 normalization constant. This distribution 2m, is used to iteratively and independently sample the hyper-parameter configurations, and the performances of the learned models are measured. Letting r(t, mff) denote the rank of the performance associated with meta-feature set mff after iterations, the performance curves report r(t, mf) for the METABU and baseline meta-feature sets (plus uniform hyper-paramete configuration sampler for sanity check), averaged over the test datasets.
Task 3: AutoML with performance model (Optimization). AutoML systems based performance models cannot be directly compared with METABU as they acquire additional information along the AutoML search: they iteratively use a performance model to select hyper-parameter configu- ration, and update the performance model using the performance of the selected configuration. In Task 3, the relevance of meta-feature sets is investigated in that they govern the initialization for AutoSkLeam and PMF performance models. The performance indicator is the rank of the perfor- mance obtained by AutoSkI Lerrn using METABU meta-features initialize its performance model,

---

Published as a conference paper at ICLR 2022
noted METABU+AutoSKLerrt (respectively, the rank of the performance of PMF using METABU meta-features to initialize its performance model, noted METABU+PMF).
The difference between Tasks 2 and can be viewed in terms of Exploration vs Exploitation: getting a good performance on Task 2 requires to identify a sweet configuration spot for each dataset Task 3 requires to identify
(Exploitation). Quite the contrary, getting good performance sufficiently good and diverse configuration region, such that the search initialized this region, gathering additional information about the performance of new configurations on the current dataset along time, eventually yields an even better configuration (Exploration).
Benchmarks. The considered AutoML benchmark is the OpenML Curated Classification suite 2018 (Bischl et al., 2019), including 72 binary or multi-class datasets out of which 64 have enough learning performance data to give a good approximation of their target representation. The perfor- mance indicators are measured using Leave-Ori Lave-One-Out (details in Appendix CC). The basic meta-features are computed for each dataset using the open source library PyMFE (Alcobaça et al. 2020).
METABU is validated in the context of three ML algorithms: Adaboost (Freund & Schapire 1997). RandomForest (Breiman 200T) and SVM (Boser et al. 1992), using their scikit-learn implementation
PPedregosa et al. 2011 ); and two AutoML pipelines, AutoSkLearn (Feurer et al., 2015) and PMF configuration spaces are detailed in Appendix D
(uus et al.
2018 The associated hyper--pramnann
For Adaboost, RandomForest and SVM, the target representation of cach training dataset is based on the top-20 configurations in OpenML (out of 37,289 for Adaboost, 81,336 for RandomFores, and 37,075 for SVM), initially generated by van Rijn & Hutter (2018). For AutoSkLeam, the target representation is generated from scratch, running 500 con configurations per training dataset and retaining the top-20. For PMF, the top-20 configurations are extracted from the collaborative filtering matrix for each training dataset (Fusi et al. 2018).
4.2 COMPARATIVE EMPIRICAL VALIDATION OF METABU
The performances of METABU and the baselines on the three tasks are displayed on Fig. 3 The overall CPU cost on Task (resp. Task is circa 1,900 (resp. 2,300) hours (full runtimes Fig. 7
Appendix H reports the detailed results in Tables 6 and [ indicating the confidence level of the results after a Wilcoxon rank-sum sum test for performances and Mann Whitny Wilcoxon test for ranks.
Task 1: Capturing the target topology, Fig. 3a The results show that the metric based on the METABU meta-features better matches the target topology than the metric based on the baseline meta-feature sets, all the more so as the number & of nearest neighbors increases. The higher variance of NDCG@k for METABU is explained as the metric depends on the meta-feature training, while the metrics based the baselines are deterministic. As could be expected, this variance decreases with k. Despite this variance, METABU significantly outperforms all baselines for all and all hyper-parameter configuration spaces.
Task 2: AutoML with no performance model (Initialization), Fig. 3b All rank curves start at 3, as five hyper-parameter configuration samplers are considered. For RandomForest, the sampler based on the SCOT meta-feature set dominates in the first 5 iterations, and remains good at all time; METABU dominates after the beginning; all other approaches but the uniform sampler yield similar performances. For Adaboost, the sampler based on the AutoSkLearn meta-feature set dominates in the first 3 iterations, and METABU statistically significantly better than all other approaches thereafter. For SVM, METABU very significantly dominates all other approaches.
Task 3: AutoML with performance model (Optimization), Fig. 3c In first time steps (left of the dashed bars), the performance models of AutoSkLearn or PMF are initialized using the performances configurations sampled as in Task 2; in the following time steps, the hyper- of the configurations are sampled using the performance model. The most striking result is parameter that the METABU+ BAAABUAAAtSSLLAAA AutoSkLearn rank improves on that of AutoSkLearn (Fig. 3c left) although they only differ in the initialization of the performance model, and the AutoSkLearn meta-feature set is optimized to Task 3. Likewise, the rank of METABU+PMF improves on that of PMF (Fig, 33 right). The comparison also involves Random2x and Random4x uniform samplers, respectively returning the best performance out of 2 or 4 uniformly sampled configurations (Fusi et al.) 2018); METABU+PME significantly improves Random4x after the 10th iteration. This suggests that on
7

---

Published as a conference paper at ICLR 2022
Metabu
AutoSklearn
Landmark
Random Forest
Adaboost
NDCG@k
0.8
NDCG@k
0.8
NDCG@k
0.8
0.6
0.6
0.4
0.4
0.4
35
(a) Task Capturing target topology; the higher NDCG@K, the better.
Metabu AutoSklearn
Landmark
SCOT
Random1x
Random Forest
Adaboost
SVM
Rank
Rank
Rank
tteration
iteration
iteratiio
(h) Task 2: Sampling the hyper-parameter configuration space; the lower the rank, the better.
Metabu + AutoSkleam
Random1x
Metabu+ PMF
Random4x
Random1x
AutoSKlearn Metabu AutoSKleam
Metabu+PMF
2.5
Rank
2.0
Rank
1.5
10
40
10
20
40
50
teeraiion
Iteratioon
(c) Task 3: Initializing a performance model to sample the hyper-parameter configuration space.
Figure 3: Empirical assessment of METABU meta-fcatures comparatively to the baselines meta-fcature sets and uniform hyper- paramerre sampling (better seen in color).
the OpenML benchmark, the METABU meta-feature efficiently enable both to passively sample the syper-parameter configuration space, and to retrieve the configurations best appropriate to update the performance model and explore good regions of the space.
4.3 SENSITIVITY ANALYSIS
The own two syper-parameters of METABU are the a trade-off parameter between Wasserstein and Gromov- Wasserstein distance (Eq. [ and the regularization weight (Eq. 2 The sensitivity of METABU w.r.t. both param- eters is investigated on Task 1, by inspecting the differ- ence NDCG@ 10(METABU) NDCG@ (0(AutoSLLearm) for ranging in 10.1, 0.3, 0.5, 0.7, 0.99} and X in {10-1... _.o-I..... ..... 10-41 $3. The result, displayed in Fig. shows that the difference is positive in the whole considered domain, with NDCG@ @ 10(METABU) statistically signifi- cantly better than NDCG@ (O(uutSSLeaar)) according to Student ttest with p-value 0.05.
03
0.05
0.071
0.11
0.11
05
0.11
0.
0.038
0.065
0.11
0.1
0.99
0.033
0.075
1e-1
1e-2
1e-3
1e-4
Figure 4: METABU: Sensitivity of
Interestingly, low sensitivity of METABU is observed w.r.t. the regularisation weight A, provided that is small enough (A 10 10-3). For such small À values, low sensitivity is also observed w.r.t. a in a large range (.3 < a < .7). This result confirms the importance of taking into account both the Wasserstein and Gromov- Wasserstein distances the
NDCG@ @ 10 w.r.t a and X, compara- tively to AutoSkLeamn (darker is better).
8

---

Published as a conference paper at ICLR 2022
target representation space: discarding the former (a I .1) the latter (a .99) significantly degrades the performance, and the performance stable in the [.3, .7] region.
4.4 TOWARD UNDERSTANDING THE DATASET LANDSCAPE
A first original result is provide a principled estimate of the intrinsic dimension of the dataset space w.r.t. the considered ML algorithms. As detailed in Appendix G.I with a stability analysis, the intrinsic dimension d of the OpenMI. benchmark circa 6 for AutoSkl.earn, Adaboost, 9 for RandomForest and 14 for Support Vector Machines. As reflects by construction how diverse the datasets are w.r.t. the ML algorithm, it is interesting to see that the most flexible AutoSkLearn ML pipeline cor- responds to the lowest intrinsic dimension.
2
ttes get
for
01R
310
METABU also delivers some insights into what matters in the dataset landscape, and why given algorithm should behave better than another on a particular dataset, as fol- lows. The images (xx;) of datasets according to the METABU meta- meta-features learned in the context of an algorithm A are processed using PCA, and the importance of a manually designed meta-feature measured from the norm of its projection A (mf)) on the first PCA axis.
Figure 5: Comparative importance of meta-features for RandomForest (x-axis) and Adaboost (y-axis).
Two MI. algorithms or pipelines A and B can thus be visually compared. by plotting each meta- feature as a 2D point with coordinates (im(m)), (mf)).. As shown on Fig. S with respectively A set to RandomForest and B to Adaboost, one sees that actually few features matter for both GandomForest and Adaboost (the features nearest to the upper right corner), mostly the Dunn index (uunn 1973) and the features importance. Some findings reassuringly confirm the practitioner's expertise: the percentage of instances with missing values matters much more for Adaboost than for RandomForest; the class imbalance (Class Pry and TlassPrrbabiliiyMMin matters for Adaboost. Complementary results (Appendix (i2) show that the sparsity of the data matters for Support Vector Machines. Some other findings are less expected, e.g. the importance of the data density, minimal skewness and kurtosis for AutoSkLearn; these findings are tentatively explained from the fact that AutoSkLearn includes classifiers such as Linear Discriminant or Logistic Regression.
5 CONCLUSION AND PERSPECTIVES
METABU provides an algorithm-depender way to achieve AutoML, through learning meta-features as linear combinations of the manually designed meta-features of the literature, optimized to capture both the top configurations for the datasets and their topology, via preserving their Wasserstein and Kromov-Wasserste distances. The efficiency of the approach empirically demonstrated as the METABU meta-features contribute to outperform strong baselines, including AutoSkLeear (Feurer ct al. 2014) and PMF (Fusi et al. 2018).
An interesting side-product of the approach is to shed some light the complexity of the AutoML problem, by estimating the intrinsic dimension of the dataset landscape. Surprisingly, this intrinsic dimension is relatively modest (< 14). While this result is comforting when considering the small number of datasets in the AutoML benchmarks, it should however be taken with a grain of salt: the intrinsic dimension might merely reflect the specifics of the OpenML benchmark, as the datasets might have been selected over the years to provide evidence for the merits of mainstream MI. algorithms while discarding too hard datasets.
A perspective for further research is to assess the validity of the proposed meta-features and the stability of intrinsic dimensions on other AutoML benchmarks: the underlying question is which extent AutoML, too, is prone to overfitting.
Another perspective is exploit METARU to conduct comprehensive empirical assessment of new algorithm A on time budget, by alternatively learning the meta-features relative to A, and selecting the datasets most diverse according to these meta-features, in the spirit of experiment design.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
The approach is not concerned with privacy and confidentiality of the data. The AutoML goal aims to reduce the computational resources needed get the peak performance from an ML portfolio of algorithms or pipelines.
ACKNOWLEDGMENTS
We gratefully thank the anonymous reviewers for their constructive comments and suggestions.
This work, and in particular Herilalaina Rakotoarison, fully funded by ADEME (# (#1782C0034) project NEXT.
This work is also supported by TAILOR, an ICT48 network funded by EU Horizon 2020 programme GA 952215.
REFERENCES
Edesio Alcobaça, Felipe Siqueira, Adriano Rivolli, Luís P. F. Garcia, Jefferson T. Oliva, and André C. P. L. de Carvalho. MFE: Towards reproducible meta-feature extraction. Journal of Machine Learning Research, 21(111):1-5, 2020.
Shawkat Ali and Kate A. Smith. On learning algorithm selection for classification. Applied Soft Computing, 6(2): 19-138, 2006.
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 2 21428-21439, 2020.
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sehag. Collaborative hyperparame- ter tuning. Proceedings of the International Conference on Machine Learning (ICML), pp. 11-199-11-207. JMLR.org, 2013.
Hilan Bensusan and Christophe G. Giraud-Carrier. Discovering task neighbourhoods through landmark learning performances. In Proceedings of the 4th European Conference on Principles of Data Mining and Knowledge Discovery, pp. 325 -300 Springer- Verlag, 2000.
James Bergstra, R. Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for
optimization. In J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems (NIPS), volume 24, 2011.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G. Manto- vani, Jan N. van Rijn, and Joaquin Vanschoren. OpenMI. Benchmarking Suites. arXiv 1708.03731, 2019.
Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory (COLT), pp. 144-152. Association for Computing Machinery, 1992.
Leo Breiman. Random forests. Mach. Learn., 45(1):5-33 2001.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings the International Conference on Machine Learning (ICML), pp. 89-96. Association for Computing Machinery, 2005.
T. Caliński and J. Harabasz. A dendrite method for cluster analysis. Communications in Statistics, 3 (1):1-27, 1974.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell., 39(9):1833-1866 2017.
T.F. Cox and M.A.A. Cox. Multidimensional Scaling. Chapman and Hall, 2001.
10

---

Published as a conference paper at ICLR 2022
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances Neural Information Processing Systems (NIPS), volume 26, 2013.
Gwendoline de Bie, Gabriel Peyré, and Marco Cuturi. Stochastic deep networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the International Conference on Machine Learning (ICML), volume 97, pp. 1556-1565. PMLR, 2019.
Joseph C. Dunn. A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters. Journal of Cybernetics, 3:32-57, 1973.
B. Efron. Bootstrap methods: Another look the jackknife. Ann. Statist., 7:1-26, 1979.
Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimen- sion of datasets by a minimal neighborhood information. Scientific Reports, 7(12140). 2017.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta-learning to initialize bayesian optimization of hyperparameters. In Proceedings of the 2014 International Conference on Meta-Learning and Algorithm Selection Volume 1201, pp. 3-10, 2014.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenherg. Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In C. Cortes, N. D. Lawrence, D. D. Lee, Sugiyama, and Garnett (eds.), Advances Neural Information Processing Systems (NIPS), pp. 2962-2970. 2015.
Yoav Freund and Robert E Schapire. A decision theoretic generalization on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-139) 1997.
-ecision-theoreticc
Nicolo Fusi, Rishit Sheth, and Huseyn Melih Elibol. Probabilistic matrix factorization for automated machine learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Isabelle Guyon, Lisheng Sun-Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michéle Sebag, Alexander Statnikov, WeiWei Tu, and Evelyne Viegas. Analysis of the AutoML challenge series 2015-2018. In AutoML, Springer series on Challenges in Machine Learning, 2019.
Elad Hazan, Adam Klivans, and Yang Yuan. Hyperparameter optimization: A spectral approach. In International Conference Learning Representations (ICLR), 2018.
Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. ParamILS: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research JAIR, 36:267-306. 2009.
Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello (ed.), Learning and Intelligent Optimization, pp. 507-523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. The Springer Series on Challenges Machine Learning. Springer, 2019.
Hadi S Jomaa, Lars Schmidt-Thieme, and Josif Grabocka. Dataser2Vec: learning dataset meta- features. Data Mining and Knowledge Discovery, 35:964-985, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR). 2015.
Lars Kotthoff. Algorithm selection for combinatorial search problems: A survey. Al Magazine, 35 (3):48-60) 60,8866068800 2014.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In L. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems (NIPS), volume 17. MIT Press, 2005.
11

---

Published as a conference paper at ICLR 2022
M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp. J. Marben, P. Müller, and F. Hutter. Boah: A tool suite for multi- fidelity bayesian optimization & analysis of hyperparameters, arXiv:1908.0677 [cs.LG], 2019.
Ana Lorena, Luís Paulo Garcia, Jens Lehmann, Marcilio de Souto, and Tin Ho. How complex is your classification problem?: A survey on measuring classification complexity. ACM Computing Surveys, 52:1-34, 09 2019. doi: 10.1145/3347711.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning of symmetric elements. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the International Conference on Machine Learning, volume 119, pp. 6744- PMLR, 2020.
Mustafa Misir and Michèle Sehag. Alors: An algorithm recommender system. Artificial Intelligence, 244:291-314, 2017.
Mario A. Muñoz, Laura Villanova, Davaatseren Baatar, and Kate Smith-Miles. Instance spaces for machine learning classification. Machine Learning, 107(1):: (0791):09-1147, 2018.
Facundo Mémoli. Grssev-Waseerseinn distances and the metric approach to object matching. Foundations of Computational Mathematics, 11:417-487, 08 2011.
Khai Nguyen, Son Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Improving relational regularized autoencoders with spherical sliced fused Samoo-Wassersttiin arXiv 2010.01787. 2020.
Eugene Nudelman, Kevin Leyton-Brower Holger H. Hoos, Alex Devkar, and Yoav Shoham. Under- standing random SAT: beyond the clauses- toe--to-vvaaaa variables ratio. In Mark Wallace (ed.), Proceedings of the International Conference on Principles and Practice of Constraint Programming (CP), volume 3258 of Lecture Notes in Computer Science, pp. 438-452. Springer, 2004.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 -222025-2830. 2011.
Yonghong Peng, Peter A. Flach, Carlos Soares, and Pavel Brazdil. Improved dataset characterisation for meta-learning. Steffen Lange, Ken Satoh, and Carl H. Smith (eds.), Proceedings of the International Conference on Discovery Science, volume 2534 of Lecture Notes in Computer Science, pp. 141-152. Springer, 2002.
Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends Machine Learning, 1165-)):35--077,220 2019.
Bernhard Pfahringer, Hilan Bensusan, and Christophe G. Giraud-Carrier Meta-learning by land- marking various learning algorithms. Proceedings of the International Conference on Machine Learning (ICML), pp. 743-750, Morgan Kaufmann Publishers Inc., 2000.
Bruno Almeida Pimentel and André C.P.L.F. de Carvalho. A new data characterization for selecting clustering algorithms using meta-learning. Information Sciences, 477:203-219, 2019.
John R. Rice. The algorithm selection problem. Advances in Computers, 1 15:65-188, 1976.
Qinbao Song, Guangtao Wang, and Chao Wang. Automatic recommendation of classification algorithms based on data set characteristics. Puttern Recognition, 45(7):2672-2689, 2012.
David H. Stern, Horst Samulowitz, Ralf Herbrich, Thore Graepel, Luca Pulina, and Armando Tacchella. Collaborative expert portfolio management. In Maria Fox and David Poole (eds.), AAAI Conference Artificial Intelligence. AAAI Press, 2010.
Vayer Titouan, Nicolas Courty, Romain Tavenard, Chapel Laetitia, and Rémi Flamary. Optimal transport for structured data with application graphs. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the International Conference on Machine Learning (ICML), volume 97, pp. 6275-6284 PMLR, 2019.
12

---

Published as a conference paper at ICLR 2022
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(6))2579-2660 2008.
Jan N. van Rijn and Frank Hutter. Hyperparameter importance across datasets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 2367-2376. ACM, 2018.
Ricardo Vilalta. Understanding accuracy performance through concept characterization and algorithm analysis. In Workshop on Recent Advances in Meta-Learning and Future Work, 16th International Conference Machine Learning, pp. 3-9, 1999.
Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wasserstein learning for graph partitioning and matching. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buu, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems (NeurIPS), pp. 3046-3056, 2019.
Hongteng Xu, Dixin Luo, Ricardo Henao, Svati Shah, and Lawrence Carin. Learning autoencoders with relational regularization. In Hal Daumé IIII and Aarti Singh (eds.), Proceedings of the International Conference on Machine Learning (ICML), volume 119, pp. 2020.
PMLR,
L. Xu, Hutter, H.H. Hoos, and K. Leyton-eytoootot Browon SATzilla: portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research (JAIR), 32(1):5655666 6066, 2008.
Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell. OBOE: collaborative filtering for automl model selection. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1173-1183, ACM, 2019.
13

---

Published as a conference paper at ICLR 2022
Learning Meta-Features for AutoML
Supplementary Material
The supplementary material includes additional details
The augmentation of the OpenML benchmark (Appendix A);
The pseudo-code of the algorithm (Appendix B
The experimental setting, performance indicators and validation procedure (Appendix
The hyper-paranee apper-paramete configuration space (Appendix D):
The list of basic meta-features and baseline meta-featuresetss (Appendix D
The details of the computational time (Appendix (F):
The insights into the AutoML. problem provided by the approach: intrinsic dimensionality (Appendix G.1 and visualization of the niches of the considered MI. algorithms (Appendix G.2):
The detailed results with standard deviation on all three tasks (Appendix
Pairwise comparison of METABU with baseline meta-features (Appendix)
Sensitivity analysis of dimension (Appendix 0
Performance curves from Task 2 (Appendix
A THE AUGMENTED OPENML BENCHMARK SUITE
The visualisation of the augmented benchmark (Fig. 6 projected using tSNE (yan der Maaten & Hinton 2008) on the basic representation space), shows that the datasets built by bootstrapping of some initial dataset E form a cluster close E' (as could have been expected as the manually designed meta-features are stable under small stochastic variations), and separated from the clusters generated from other datasets, suggesting that the initial benchmark suite only sparsely paves the basic meta-feature space. Complementary experiments (omitted) with perplexity ranging in [5, 10, 15, 25, 30, 40, 50] show that clusters generated by augmentation of different OpenML datasets keep staying far apart from another.
75
50
25
dabetos
-25
-50
-75
Figure 6: 2-D ISNE Visualisation of the OpenML datasets basic representation space (legended with * +'s) + their boostrapped augmentations. Only few dataset names are present for the sake of readability.
14

---

Published as a conference paper at ICLR 2022
B PSEUDO CODE OF METABU
The learning procedure of METABU is described in Alg. detailing the description presented in Section 3 of the main paper. The density on the happer-parameter space, used sample hyper- parameter configurations for given dataset depending on the considered meta-features (involved in Tasks 2 and presented in Alg.
Algorithm I: Learning METABU meta-features
Data:
Set training datasets, each represented with its basic representation (meta-feature vector) and its target representation (set of top 20 hyper-parameters) Z; for
I...n
Result: Embedding layer 20*
// Build projected target representation
Ci. 4 (2,,2,2) 2,) for = n.j. = 1... + Pairwise Wass Dist.
2 Estimate intrinsic dimension d from matrix C using (Facco et al. 2017); MDS(C,d);
/ Multicimensional Scaling */
// Learn +
4 & + Linear(135,dd d):
/* 135 basic meta-features
3 X < +Ei=1 óg;;
+ FGW as defined in Eq. 7 2/th + ADAM(C, 4/xx, u);
Algorithm 2: Fit die_dinstty
Data: Set of training datasets, each represented with its meta-feature vector and its set of top hyper-parametens O for 1...n. n. Test dataset represented with its meta-feature vector
Result: Distribution of configurations z.
Order training datasets such that: ||x zell < || - ret1ll for é I...n;
7. 2 E exp((-) Exco, 0:
15

---

Published as a conference paper at ICLR 2022
C MEASURING PERFORMANCE INDICATORS
As said, the OpenML benchmark includes 72 datasets, with only 64 of them having a target repre- sentation. The other datasets are too heavy (e.g. ImageNet) to launch the many runs required to estimate their target representation.
For Task the performance indicator measured along a Leave-One-Out procedure, with 64 folds: in each fold, all datasets but one are used to train the METABU meta-features; the NDCG@N @k is measured on the remaining dataset. Eventually, the NDCG@k averaged over all 64 folds.
For Tasks 2 and 3, the performance indicator is likewise measured using Leave-One-Out procedure with 64 folds. The difference is that besides the remaining dataset, the datasets with no target representation at all are also used as test datasets.
In Tasks 2 and 3, the performance associated with a computed after training the model on 1 CPU with time budget of 15 mn, with memory less than 8Gb, using the train/validationneest splits given by OpenML.; the validation score is estimated using a 5-CV strategy.
configuration for dataset is
In Task 2, for each test dataset, and for each meta-feature set mf in (METABU, AutoSkLearn, Landmark, SCOT):
The distribution
f
1
Σ 8=1
10 exp( f)zz
Z
is defined, with Z₁ the target representation of the f-th neighbor of the considered (test) dataset, among the training datasets, according the Euclidean distance based on their mf representations.
For 1 < t < T, hyper- hhper-ppramen parameter configuration is independently drawn from 2mft and model is learned using this configuration;
The performance of this model is measured validation dataset;
The model with best validation performance up to iteration is retained for each meta- meta-fcature mf and its performance the test dataset is computed;
The rank r(t, mff is determined by comparing the performance on the test set, of the models retained for each meta-feature type.
The performance curve reports r(t, mf). averaged over datasets.
In Task 3, the meta-features are used to initialize the performance model:
In AutoSkLearn, the performance model for AutoSkLearn is initialized as follows. The best configurations for the top-10 neighbors of the current dataset are retained and run on the current dataset; their performance is used to initialize the Bayesian Optimisation search using the SMAC BO implementation Hutter et al. (201 l). These top-10 10 neighbors are computed using the Euclidean distance on the meta-feature set. Note that AutoSkLeart meta-features have been crafted to achieve automatic configuration selection the context of the AutoSkLearn pipeline (Pedregosa et al., 2011)) thus constituting most strong bascline on Task 3.
For PMF, the best configurations for the top-5 neighbors of the current dataset are likewise selected; their performance is computed to fill the row of the collaborative matrix associated to the current dataset, and determine the latent representation of the current dataset. The probabilistic model learned from the matrix is used to select further hyper-parameee config- urations; their performances are computed and used to refine the latent representation the dataset.
16

---

Published as a conference paper at ICLR 2022
D THE HYPER-PARAMETER CONFIGURATION SPACES
The hhpee-ppaae parameters used for Adaboost, Random Forest and SVM and their range are detailed in Tables [ and 2 For AutoSkLearn, we only included the list of considered hyper-parameters: their ranges are detailed in AutoSkLearn (Feurer et al. 2013). The hyper-parameter space used in PMF is the same as in AutoSkLeam. The METABU implementation uses the ConfigSpace library (Lindauer et al. 2019) to manage the hyper- prramtearameters
Classifier
Range
imputation
mean, modian, Irequent (50, 500]
_estimator algorithm
Adaboost
SAMME, SAMMERR
max depth imputation
mean, median, most frequent
criterion
gini, entropy
features
0, 0
RF
min sampmsmmmsssppp split
2, 20
sassssmmmssse leaf
20
bootstrup
True. False
imputation
mean, median, most frequent [0.03125, 32768] rbf, poly. sigmoid 1.5
kernal degree gamma coeff)
SVM
8.0517578125 10"2.8)
[-1, 1]
shrinking tol
True. False
[10-5 10-1)
Table 1: Hyper-parameter ranges of Adaboost, Random Forest and SVM
17

---

Published as a conference paper at ICLR 2022
Methods
Parameters
balancing
Strllogy
adahoosi hemonili nh
learning rale, max depill, n. estimatoos
deptn
poiles
decision_tree
min inppunity_dcccreeeeeeeettteeeeee decresse, min samplssmmpesseeee leaf, min samples_piii min weight fraction, leuf
Enietion, max depeh, max Teatures,
extra_trees
leaf nodes,
impurity decrease, min samples, leaf,
min samples split, min, weight fraction
12_ 12_regularizationn learning rale, loss,
max hins, max depth, max leaf nodes, stinnss Jeaf, scoring,
bocwinng
neer no change, validation fraction weights
I. Lecarest neighbers
lda
shrinkage_
intercept, intercept scaling.
liblinear_swe sve
lnss, multi class, penalty,
pammaa kennel, max ifor
lihsvm_stv
strinking, coeff)
dogree
alphn, batch, sine, Dota beta_ sarly_stopping..m hidden, layer sopping.cpailon.
mlp
leaming rate, jnit, n_iter_ co_nchnge,
nom, nodes per layer, shuffie, solver,
tol, validarion Fraction
multinomial mb
Rt pitor
passive Aggressive
average. intercep
qda
JC0, param
eritetion, max depth, max Jeatures, max Jea__nodes, podes, min impurity decrease, min Jeaf,es samples, split, min, weight. fruction leaf
random forest
average, intereept, leaming_ran
sgd
loss, penalty. tol.
epsilon, stuf), ratio
pesslly
features, depph,
Istes_ ettra_rree_pppppoc for clussification
max, leuf nodes, min_ impurity_decrreeee decrease, min samples, leaf, samples split. min weight fraction leaf, estimators
[un, whiten, composees
feature (coture_agglomeet agglomeration
linkage, it, fluslsssrss pooling fune
kemel_pe pea
iLcomponnnts, coef0, degree.
Rccomponents
Gual Bt intervept, micecpp sealing loss, multi class, penalty,
liblinarr svc preproccesoo
n_commpoentt, coeff) degree,
nystroem_samge sampler
pea
whiten
polynomial
include bias, Interaction only
trees_ wwwttreeeeeeeeeeeeeeeeeeeeeeeeee
embedding
depth,,
man man_stnppec_coo
min samples split, min weight fraction estimators score fune
select percentile classi fication select_rates_ee cectt_raee__clese classticciiiiaiinnnn
Func, mude
Table 2: List of paraa--arsmeterss considered in AutoSkLeam pipeline.
18

---

Published as a conference paper at ICLR 2022
E LIST OF META-FEATURES
The list of meta-features used the experiments detailed Tables and Meta-features extracted with PyMFE (Alcobaça al., 2020) except for AutoSkLearn, SCOT and Landmark meta-features which are computed from the AutoSkLearn library.
Mesa femures best cllac discr
Desciption
AutSSiZeum
Laddaek
SCOT
Metalu
Performance of the best single decision ure mode.
Performance of Blie Naaood Naighba
Performence Linear Disemminamt clasiiter.
boyes
Performance Nave clasererr
Performance Neuree Negghou classiber.
Berformanee single
dassinn
inhursl
Compuis ltemset by
Compure two leeneet
Compatc entrapy of class propottius.
Compune the inbolance ratio.
Clusccing coeflisient.
denetty
Avergg dersily of the nztoork
Masimum Facee's dicriminunt niin
flv
Directinal vector masimum lisher's discrimmom ratio.
Volume of the evellaping repion.
Computo festume maximum indivitual efficiency.
Compute collective
Sun
peogmamingg
Compure Linearty linezz clusitter.
svenge canlially.
Computc fiusion beedatlnn paints
Ratio of imm and exim thas ncarest nrighber distance.
Enoc raie the nearest neighbor classifeer.
n4
Compuic the mnnlinzzelly of the &-N Chasilier.
V'accaon of by pepespheress covering ttm.
12
Compate the everage nimber of featues per dimetsion.
Compuse the everge sunber of PCA dinessions points.
Compute the niio of the PCA dimession the origiaal dimension. Cainsti Hlarabasz
the INT
Compute the Compute nimber
entrgy.
Compuic peanans
distonace.
given size.
Compate
Compuic Devics Boridin
Campute
ltores
Compuse the number leaf noces in the DT model.
Javve Seanch
Comput: the size brandies is the DT madi).
freves nomeh
Compane the frawes comobccation the ETT model.
homo
Compuie the DT model Homogenelty
every mode.
leeves pla_class
Compute proporson of leaves class in DT model.
unies
Cumpute the sumber of non- noles DT model.
Compute the
number attributes
model.
nodes_per_
Compute the
model.
Canguic
per
andoourppcttel
Compute number repested mndes model.
Compune depth every
jmbulatce
Compuse inbalance
shape
Compuie lee dipp extrs mule.
var importance can cor
Compute the features imgortance of the UI model
ormbute.
Compute conoical come lntio dats.
car
Compute the absolute value of the conciatio disinct dataes column pains marrmm: didinct dussett stribete pains Compute the eigenvannes covatance manns trom diraset
con
Compate the absolutc value
eigenaatnes a_meu
Cumpute the geanetic mear of each insibuc.
srasity
Conquse the disunce betwee misority and mujusit clases censer of muss.
Compate the harmunic maan
attibat..
Compure the
(URR each atribote.
Compuie karssis atribate.
Compute Lawley-funolling
Compure Mulian
(MAD) sejinoal
Compute maximum valse
anstbute.
mcan
Compute mzan
modun
Compate the modian value each atribute.
min
Compunc the mitimum value from eoch atnbute.
Compuse the number of disenct highly cocreisted par utmibutes.
nr. dise
Compate the nunler of cantical conclain betwean each abtte and clue.
nom
Compate the sumber strbutes romnally divestuted based in given method.
Compuse the number of atribunes with least outler value. Cumpute the Pille's tracz.
Cumpute the mage (mas nin) of attribuic.
Table 3: List of meta-features, 1/2
19

---

Published as a conference paper at ICLR 2022
Meta features
Deserycion
Landmork
800F
Menoho
Conguse the Koy larges rore. Compate the standand dexistion of rash strihan
mali
Compute a coristicll los for humggrnckk of mwvaiimnes
Compunc the skowness for each sttrittte
spanity
Compute (posshll tomalized) spenity meme for each stmtuuee.
mon
Compune the trimed mran of each strihttt
var
Compatr the oeierre cach antieee
J.Janiaaa
Compute the Wilk' Lambla value.
cone
Compute concentration seef. each of disinct amriootes
anr err
Compote Shonton's entroyy for each predictre arnbeene
clas_oone class_oer
Compuse concentrtiio coefiiciin betwezn euch utsibute and class. unsiuuee Shatnon's eumrpy.
Compuse
nim, juint, Lest
Compure the number of atribuges equivilen for peciitivve tisk.
Cumpaic dhe join entrps les wown each altibute and cleo.
Compuic the musal aforarin bewees each snributz and teget.
Compute the tniines unriates.
Compuse the improved versinn the wiightal diconee,
cobesiveness
that cetueee how daner or quane is exmmle compledirrbaiin
Compais cencept
conceptoo
weabillty
exumples.
Compute the implosed concept valation that
impeomeeeear
estimates the vanchiity of class labels smong examples.
Compuic weighted thstaes explunes
wy
deme spaee ctample distilution
Compue
the
Compune Compune frsprncy catgeri
Compute
or
Compute
Compate
bioory atnbutes
Comput
corgenall
Compate the number
Compute
Compune insunces featires
num co
Compute number
catequnical
PCASkevnssdFootee
Slzwnes exampie las priscipal componzes
Kanoss examples
prnccipll componens
Fraction of compencui of
caplainal samiass
Landnask
Peffrmanc acarest sciphber claseiiis
Pesfertance decióos conideting one faaure
DeceinttDeeniniaaa zorner
Perfommane of decision coneieeing all festuros
Landmokk Decsinl
Performance
cheatfier
Landmor Naaeelgge
Perfornance Naive Bayes clastiter
LandmakLiD
Perfomance LIM clasetter
SkrwnessSID
Standani destation temtr shownes
SkewnesMMeas
Meas of feature skevnes
SessnessMax
Maximum of featur skewsess
Skewnesstii
Minimum teanute stewnes
STT
Sunlard devision of feature karkesis ceefiiiints
Kunosiiiicu
Meas Seature kurosis cocfiiiatt
KunossMix
Mix of fecure lutteis coctlicias
KurtosisMis
Mice festure kanosis cwcllscett
SymbhhkSum
Sun of cateariial fasure voudals
SunbokSTD
Sundad devinion of categencaa fratue synbols
SymbeteMran
Mein catggrical fratre symbol
SymbolsMlas
Mox of cateonicl frrum symbols
Sanbeéshin
Min of congurical fcaturc symbol
Clas@hotchiliSSTD
Stadded deviation of class probutiiness
Mean of class probestines
Chaodhrtehiiigyes
Maxinum clas gresshittiin
Mitimam of class probubiities
InveseDanseeRato
Invens of datseet rilo
Dansetkatio
Dataset rato
RatisNomizal ToNumerica
Ratio sumber nominal sunciiaa features
RutsNNamrric
Ratio sumerical nomiaal
Nunber of cutegetical features
NfndicOfNammei Featuncs
Number of sumrric fcaiunes
Ounbar006ssinng Offasiing Values
Namberr of mising valuce
MumberOfFaaneeWWihh Wichiieing Values
Number of featurss with mising values
NumberOiraatne d'dMiscin
Number instnnoe with
valuea
Numbaoffeaattne.
Number of ficuuo
SunberfiClress
Sanber cheacs
Number
datesnt ratee
PesertagetrMiiscug Villues
Percemage vilues
PertemapeOii CoroOOeeaaneeee Withllssingg Villes
Percentage of Percentage instanees with sumber
Percentage Sttesuuneeeeesseeeee Minssuunneeeiiiiiiiiiiiieie
LogNanberOfFeaurs
LuSSuabeOBassueee
Table 4: List of meta-features, 2/2
20

---

Published as a conference paper at ICLR 2022
F COMPUTATIONAL EFFORT
Fig. indicates the runtime for pre-processing (extracting the 135 meta-features, top row), and for training METABU (second row). The training times for learning one model is indicated for comparison (from row 3 to 5: Adaboost, RandomForest and SVM).
Extracting Basic MF
4
Learning Metabu MF
Training on Adaboost
Triningg Random Forest
Training on SVM
10"
10a
10a
10a
Runtime (sec)
Figure 7: METABU computational effort: average runtime of the meta-feature extraction (in blue) and METABU training (in orange). The average training time of one hyper parameter Adaboost (green), Random Forest (red) and SVM (purple) pipelines are shown for comparison.
'On Intel(R) Xeon(R) CPU ES-2660 v2 2.20GHz.
21

---

Published as a conference paper at ICLR 2022
G
TOWARD UNDERSTANDING THE DATASET LANDSCAPE
G.1
THE STABILITY OF THE INTRINSIC DIMENSION
Dataset Ratio
0.1
0.25
0.5
0.75
1
Adaboost
5.65 (2.74)
6.81 (1.81)
7.14 (1.44)
6.59 (1.29)
6.98
Random Forest
5.33 (2.17)
7.44 (1.28)
8.48 (1.56)
8.49
SVM
8.56 (2.54)
(2.71)
12.83 (3.16)
13,99 (2.40)
14.41
AutoSkLearn
5.17 (2.08)
4.47 (1.26)
4.98 (0.95)
5.34 (1.06)
5.51
Table 5:
Intrinsic dimension of the dataset space w.r.t ML algorithms Adaboost, RandomForest, SVM and AutoSkLcarn, depending on the fraction of datasets considered in OpenML
In T we investigate how the intrinsic dimension varies when considering various numbers of datasets in OpenML. It is observed that the intrinsic dimension tends to increase with the number of considered datasets, particularly so for SVM and to lesser extent for RandomForest. This suggests that the hyper- parameter configurations investigated in the OpenML benchmark for these algorithms do sufficiently sample the (good regions of the) configuration spaces.
G.2 INTERPRETATION: IMPACT OF THE HC META-FEATURES ON PERFORMANCE THE LEARNING ALGORITIM
METABU meta-features are built from the initial meta-features using the trained linear mapping 3/, depending on the current learning algorithm A. Accordingly, the importance of the initial, humanly defined and interpretable meta-features w.r.t. .A can be estimated, shedding some light on which specifies of dataset matter in order to give a good/bad performance with A.
The importance of meta-feature w.r.t. estimated as follows. Let = { denote the matrix made of the multi-dimensional scaling representation of the target representation over all datasets (section 3.2). Let u denote the first principal component of U and let j be the index the H column most contributing to (j* argmax, (v, h h.r)) ). Then the importance (k) of the k-th initial meta-feature for the A algorithm is defined as the absolute value of t that is, the weight of the k-th initial meta-feature build the most important METABU meta-feature.
This estimate is used to visually appreciate the feta-feature importance w.r.t. two learning algo- rithms A and B, by depicting each k-th meta-feature in the 2D plane as the point with coordinate (k), (k)). Intuitively, rithms. Meta-features far from the diagonal are much more important for one algorithm than for the other. The visualization of the meta-feature importance w.r.t. AutoSkLeam and Random Forest is InverseDatase- displayed Fig. [ left. Meta-features such as KurtosisMin, CogNumberOflnntaanee iRatio- all retained as AutoSkLearn meta-features are critical for AutoSkLent whereas they have no impact for RandomForest. Inversely, some features like "pb" (average Pearson correlation between class and features) matter significantly more for RandomForest than for AutoSkLear..
on the diagonal have the same importance for both algo-
Likewise, the meta-feature importance w.r.t. Support Vector Machines and Random Forest is displayed in Fig. right. The skewness features (mean and std deviation over all attributes) matter significantly more for Support Vector Machines than for RandomForest. In retrospect, there is little surprise that the meta-features related to the potential difficulties of inverting the Gram matrix matter for SVM.
Overall, the impact of some meta-fcatures for some learning algorithms is intuitive; confirms the practitioner expertise, which is conforting.
22

---

Published as a conference paper at ICLR 2022
KurtosisMin
density
Qparity
0.06
Joy roy_root
SkeWlessSTD
0.05
Teaves SkewnessMean
DatsesRDat
0.04
SVM 32)
mut inf
Salaam
bubs
0.04
sd. ratio
SkewnessMin
0.03
LogDatasetRatio
to num
agcdist
0.02
Lo aogivgraaaatssetaatt
leaves_pee
KartosfMax
aatooumerriaaToNoomie
6
to atl
d
b
attr conc
0.02
RCAKrttsisFiratPC
norm
importance
0.01
jmportance
ydu
attr conc
0.00
000
nly
or. nororm
0.00
16/
vdu n4
gb
0.01
0.02
0.03
0.04
0.05
0.06
0.00
0.01
0.02
0.03
0.04
0.05
0.06
43)
43)
Figure 8: Comparative importance of meta-features for RandomForest Vs AutoSkLearn (left) and SVM (right). The specific Luto meta-feat fettures are indicated as their name begins with capital letter.
23

---

Published as a conference paper at ICLR 2022
H DETAILED RESULTS
Detailed results of Task are presented in Table Random Forest, Table 7 Adaboost and Table
8 for SVM. We consider the Mann Whitney Wilcoxon test to assess the significance of the rankings.
OpenML Task ID
METABU MF
AutuSkLearm MF
Landmark MF
SCOT MIF
Randem ls
Average Kank
1.50
2.77
2.70
2.44
4,17
0.993 20.000"
0,983 0,001
0,983
10.0000
0,993
0.000
0,983 1: 0.901
10.001
0.0022
0.918
0.655
0.657
0.002
0.658
0.001
0.657
12
0.964 20.002
0.961 0.001
0.965 10.001
0.003 0.814
0.963
0.961 4: 0.004
14
0.830 +0.006
0.812 + 0.004
01813
0.005
-0.00"
0.808 + 0.005
15
0.983 -0.004
0.982 0.003
0.983
+0.000
0.581
0.005
0.983 + 0.004
16
0.955 000007
0.951 0,010
0.958 :: 0.004
0.959
000
0,950 0.005
0.675
0.000
0.001
+0.007
0.765
0.001
0.005
0,769
0.746
23
0.535 0.004
0.535 0.004
0,541 :: 0.005
0.542 0.008
0.552
0.006
28
0.083 -0.001
0.982
0.001
01,983 + 0.000
0.9K3
-0.00
0.978 + 0.001
29
0.854 10.006
0.878
1 0.004
0.882 1. 007
0.878 - 0.005
0.882 1 0.004
31
0 7.7 10.003"
0.725 10.013
0.716 1 0.003
0,707 - 0,010
0,713 0.007
32
0.993 20.001
0.992 0,001
=0000"
0.989
0,000
0.811 10.000"
0.006
0.812
0.011
0.808
0913
0.914 0.002
0,17 -0.001"
0.908
45
0.946 0.001
0.946 0.002
0.947 = 0.002
0 = 0.004
0.944 + 0.00L
49
0.962 +0.002
0.965
+0.002
0.966 + 0.000
0.963
0.002
0.957 + 0.005
53
0.780 -0.00""
0.777 0.001
0.786 +0 D06
0 0.012
0.768 + 0.006
219
0.923 10.002"
0.919 0.009
0.918 1 0.004
0,923 008
0,913 1 0.00)
2074
0.880
2079
10.00
0.628 0.010
0.647
3021
0.49 1..003
0.943 0.004
0.949 1.0 003"
0.945 0.005
0.933 0.005
3022
0.962 50.003
0.99 10.010
0.945 :: 0.017
0.927 = 0.014
0. 0.906 + 0.025
3549
0,951 1 0.015
0,984 1 0,002
0.967 1 0.021
0.977 - 0.007
0.957 1 0.023
3560
0.253 10.006
0.253 10.021
0,248 0.017
0.250 0.023
0,241 0.013
3502
0.756
0.754 20.00""
0.020
0,753 0.007
3900
0,551
0.554 0.003
0,008
0,557 -00000
0,549 +
3004
0.002 0.000
0.602 0.0033
0.600 0.002
0
0JN1
0,594 + 0.002
3913
0.619 + 0.006
0.616 + 0.016
0.634 = 0.013
0
0.001
0.609 + 0.021
3917
0.667 10.015"
0.677 10.000
0.669 :: 0.002
0.672
0.005
0.659 + 0.00
3918
0.655 +0.003*
0.661 +0.008
0.651 0.008
0.649 0.006
0.622 0.009
7592
0.778 10.002"
0.781 10.001
0.777
0.000
0.778
0.002
0,777 0.002
10.002-
0
0.747
9046
0,953 -0.006"
0.951 0.007
0,957
0.008
0,963 -0.011
0.941 + 0.010
9952
0,850 10.001
0.891 10.001"
0,882 1. 0.007
0.880 _ 0.006
0.878 T. 0.003
9937
0.366 10.007
0.864 1 0.008
0.858 1. 0.002
0.872 0.000"
0.868 1 0.005
9500
0.994 20.000"
0.993 : 0.000
0.994
5000
0.993 = 0.000
0.993 + 0.000
9961
0.927 10.00"
0.915 10. 0200
0.912 0.014
0,914
0.005
0.891 0.015
0.563 0.018
0.587 0.005
0.566 0.00%
9966
0.845 -00007"
0.846
10,04""
0,841
0,010
= 0.001
0.842 +
9077
0.961 +0.000"
0.960 0.000
0.961
+ +0.000"
0.061
0.960 + 0.001
907X
0.672 *0.007
0.680
+0.006'
0.671 + 0.003
0.677 = 0.009
0.670 + 0.004
9981
0.926 0.002
0.936
1 0.011
0,947 1. 0.019
0.941 - 0,030
0.927 1 0.022
9985
0.475 007
0.478 0.004
0.475 + 0.002
0.479 000'""
0.467 0.00T
10093
10101
0.611 0.005
0.621
0.614
14952
0.965 =000000
0.963 0.002
0.965 1:0 001
0.933
0.001
0.956 1: 0 0.004
14954
0.844 10.022"
0.835 1: 0.023
0.853 1:0 00.009
0.333 - 0.004
0,799 T: 0.013
11965
0.711 10.001*
0.712 10.022
0.710 1 0.001
0.710
0.003
0,709 1 0.002
1490)
0.597 20.005
0
0.007
0.591
= 0.008
0
=0,02""
0,575 + 0.008
125920
0.598 10.010"
0,597 0.009
0.600
-0,00
0,589 0.010
125922
-0.002"
0.002
0.976
001
0.068
146198
0.642 =00000"
0.628
0.004
0.644
0.622
146500
0.971 0.013
0.9801
0.009
0.974
0.006
0.986
0J002
0.962 1: 0.010
146817
0.835 +0.004"
0.817
0.009
0.826 +0 +0.007
0,824
0.012
0.812 + 0.004
146819
0.561 10.014"
0.859
1 0,010
0.861 T 0.015
0,870
00000""
0.869 1 0.005
146820
0.963 10.001'
0.855 0.014
0.851 1: 0.018
0.83 : 0.065
0.855 1..0100
146821
+0.001
0.972 +0.001
0.934
0.005
-0000"
146324
0.68 10.000
0.969 0.001
0.968 0.001
0.969 _0.002"
0.960 1. 0.007
146825
0.254 20.306
0.582 :: 0.504
0.293 =: 0.507
0.872 -0,00"
0.869 +: 0.003
167119
0.767 10.001"
0.764 1: 0.003
0.762 1. 0.003
0.765 -0.00""
0.765 1 0.000
167121
0.275 476
0.000 0.000
0.582 +0 505
0.000 = 0.000
0.274 + 01.475
167125
0.921 20.000"
0.002
0.924 =0
0.001
0.899
167140
0.930 0.004
0.935
0.924
167141
0.834 +0.00"
0.829 0.005
0.833 0.002
0.837 -0.00""
0.832 0.001
Table 6: Comparative learning performances on OpenMI. datasets over sampling 30 configurations of the Random Forest pipeline. Performances that statistically significant compared to the second best are in bold. Statistically comparable performances are indicated with (*). Pairwise comparison and p-value along the iterations are presented in Fig.
24

---

Published as a conference paper at ICLR 2022
OpenMLL Task ID
MITTAO MF
AutoSkL_aaMMP
Landmark MF
SCOT MP
Kandomix
Avenie Kank
2,48
235
2.89
2.84
3.50
0:305 +11.011
0,994
00,000
1399 #00001
0:994 # 0.002
0.99% =: 0.001
0.970 0.001
0.969 0.002
0.967 0.002
0.928 100,083
0.891 1: 000071
0.914 1: 0.069
0.973 1.0.017
0.920 4: 0.093
12
0.977 +110011
0.977 +0.001
0.976 + 0.002
0.977 + 0.001
0.975 + 0.002
14
0.827 #0.007
0.829 =: 0.004
0.824 0.006
02825 + 0.002
0.829
=0.004"
15
0.964 +0.004""
0.962 0.006
0.969 +0.004"
0.967 + 0.005
0.967 + 0.006
0.963 +0.001"
0.966
0.967 +0.003"
0.0044
0.001
0.691 +0.016
0.674 0.003
0.695 #0.01
0.012
0.680
22
0.796 1.0002
0.800 0.004
0880 100006
0.794 1: 0.0144
0.791 0.005
23
0.572 +1.009"
0.579
0.011
0.582 +0.015
0:575 + 0.004
0.581 + 0.009
28
0.988 +0.0011
0
0.001
0.987 0.001
0.989 +0.0011
0.987 + 0.001
29
0.865 0.006
0.866
0.008
0.875 0.009
0,882 1. 0.006
0,845 0.015
0.739 +0.011"
0.007
+0.014"
0.995 +0.001"
0.997
+0.000"
0.996 0.000
+ 0.001
0.994
37
0.790 10.005"
0.794 1.0018"
0.771
0.008
0.782 1. 0.010
0.784 0.002
43
0.936 +-1.011
0.933
+ 0.003
0.934
+0.00*
13932 + 0.004
0.933
0.002
45
0.961 + +0.003
0.951
0.000
0.957 +0.005
1.954 + 0.004
0.951 + 0.003
0.993 1 0,0002
0.985
0.000
0.997
0.003
0.967 1. 0.012
0,998 0.002
0.785
0.007
0.804 -00003
+:
219
10000
0-877
00000
0:231
0.006
0233
0A003
0.974 0.001
2074
0.903
0,902
100017
0.901
0.001
0,901
T 0.002
0,901
0.001
2079
0.657
110111
0,64/
2:1006-
0.648
1 0.010
136441 1 0.007
0.627 : 0.002
3021
0.955
0.986
0
1 0.003
0.955 1 0.002
0.953 1 0.002
3022
0,952
0,020
0,960
0,000
0,966
0.005
0,972
0.00
0,960 0.008
D.9RS
0.90
-000
0,988
0.255 #10
0:241
10000
0:258
0.020
0,262 +11.012"
0.253
3902
0.762 290.1
0,769
0.756 1 0.024
0,755 4. 0.012
0.760 7. 0.014
3903
0.604
0,581
1 0.010
0,575 1 0.016
01574 1 0.015
0,598 1...000000 L0.015
3504
0,615 TUT
0,608
: 0.006
0.612 = 0.003
0.615 +0 002"
0.613 + 0.002
3913
0.665
0.661
-0.015
0,656
0,011
0.649 1 0.006
0,659 0.019
3917
0,080
0,682
0,003
0,682
0.008
0.691 10.015
3918
0.683 #0119"
0.696
=0.008"
0.688
0.019
0.675 0.018
7592
0.798 1.,000
0.797
1: 0.000
0.796 0.000
0.799
1..000
0.798 - 0.001
9910
0.798 110011
0,796
1 0.002
0.797 10.004'
0.797 1 0.003
0.793 1 0.005
9346
0.977 11.110"
0.986
0.003
0.993 007007
0.987 + 0.011
0.990 = 0.005
9952
0.890 1'1
0.899
0.002
0.900 0.002
0,900
0.89
+00000"
0.871
0,010
+
9900
0.997 +00001"
0.097 0.000
0.097
0.001
0:097 +10.001
0.9%
9-64
0.932 0.006
0.943 = 0.003
0.940 0.005
0.937 + 0.008
0.928 = 0.005
9071
0.563 +0.039"
0.573 - 0.002
0.576
+0.010
0.558 + 0.007
0.565 + 0.027
9976
0,846 +0 DIE
0.834 0.004
0.844 +0.008
0 + 0.012
0.833 + 0.012
0.964
0.966 0.002
0.964 0.001
0.001
0,967 0.001
0.690 D026
0.696
9981
08885 +0.011"
0.890 0.004
0.808
0.004
0.802 +-0.002"
0
958S
0,475 +1.000"
0.473 0.005
0.475 #0.001"
0.475 + 0.002
0.473 + 0.007
10093
0.997 200.033*
0.994 100001
0 : 0.003
0.993 +: 0.004
0.994 : 0.005
10101
0.619 +0.011'
0.615 + 0.000
0.621
0.002
0.626 +0.013
0.619 0.0009
14952
0.964 =000022
+0
0,869 110,007
1..018"
14965
0:711 0.004
0.715
0.001
0:714
0.003
0.716 0,002
0.714
14969
0.617 -.0.0.""
0.606
1. 0.015
0.610
0.000
0.610
1..00.'
0.607
0.012
125920
0,572 111110
0,569
1 0.015
0.569
10.015
0,5665 1 0.005
0,555 1. 0.013
125922
0.992 +00001
0.992
+00000
0.991
0.000
0.992 + 0.001
0,989 0.000
146800
0.996 11.002
0.996
0.003
0.998
10.001"
0.997
1 0.001
0,990 0.004
0.817 100001
0.812
0.005
0,810
0.014
0.821
0,805
146819
0.797
0:766
0.039
0.782
0.024
0.817
1:0 025"
0.806 + 0.020
146520
0.859
10.004"
0.859
0.008
0.849 1 0.006
0.855 1 0.015
0,860 00.
14682]
0,960 10.015
0.974 1 0.003
0,979 1 0.009
0
1...011
0.970 1 0.010
146822
0.943 1 0,000
0,942 1 0.003
0.945 0,000
0.943 1: 0.004
0,941 1 0.004
146824
0.977 10.001
0.976
1..002"
0.976
0.002
0.974
0.001
0.972 1 0.003
0.818 10.002
0.816 0.003
0.817 10.001"
167125
0.914 10.002"
0.914 10.000"
0,910 1 0.002
0.911 1. 0.002
0.912 1 0.002
167140
0.954 10.005"
0.954 10.003"
0.953 :: 0.002
0.952 +: 0.003
0.948 =: 0.002
167141
0,824 10.004
0,823 1. 0.002
0.825 1. 0.003
0,826
10.0011
0.822 1. 0.003
Table 7: Comparative learning performances on OpenMI. datasets over sampling 30 configurations of the Adaboost pipeline. Performances that are statistically significant compared to the second best are in bold. Statistically comparable performances are indicated with (*). Pairwise comparisons and the associated p- p-value along the iterations are reported in Fig. 10
25

---

Published as a conference paper at ICLR 2022
OpenML Task ID
METABU MF
AutuSkLeamm MF
Landmark MF
SCOT MIF
Random
Average
2.97
3:27
0.595
10:001"
0.953
0.004
0.994 10.005
0.962
0.015
0.988 1. 0.001
0,827
10:253"
0.969 1 0.008
0.973 10. 10.000
0.967
0.010
0.937 1 0.000
11
0.967 200045"
0.929 a: 0,050
0.897 = 0.104
0.99%
-0000""
0.958
+ 0.015
12
0.975 10.007
0,980
0.001
0.982
0.002
0.031 0,984
0.007
0.936 0.018
-0.029"
0.843 +0.010
0.830
0.018
0.839 0.014
0.980
0.983 #0.000"
0,980 0.004
0.953
0.981 +0.004
0.979 0.002
0.981 +00.005*
0.981
0.976 0.011
18
0.728
+0.013
0.721 0.025
0.736 +0.010
0.731
0.013
0.717 + 0.014
22
0,836 00.016"
0.806 0.001
0.818 = 0.051
0.807 0.009
0.820 =0,02T
23
0.561 0.010
0.601 0.001
0,583 4: 0.028
0.607 = 0.010
0.585 1 0.011
+0.000"
0.987
0.012
0.87%
31
0 736 20.017
0.767 0.011
0.780 1:0 012"
0.730 0.015
0.757 :: 0.004
32
0.994 20.001
0.993
0.005
0 1 0.005
0.996
-0,00L"
0.985 -: 0.001
37
0,838 10.014
0,811
1 0,010
0.853 1 0.008
0.839
_0.018*
0.853 1 0.015
43
0.922 =0.004
0,929
=0.000
0.922 = 0.011
0.919 = 0.013
0.917 + 0,018
45
0.952 -0.00""
0.935 0,004
0,943
0.021
0.010
0.952 10.003"
0,972
0.978 +0.018
0.054
0.940
53
0.7%% 0.059
0.878 0.004
0.846 0.013
219
0.397 0.035
0.843 0.018
0 =: 0.007
0.856 = 0.053
0.871 # 0.049
2014
0.905 +0.011*
0,888 + 0.008
0.905
+0.012
0 0.004
0.887 + 0.017
2079
0.639 -0.002"
0.578 0.051
0.648
+00000*
0.31 = 0.034
0.556 + 0.023
3021
0.962 10.011"
0.939 0.017
0,955 1. 0.008
0 - 0.041
0,958 1
3022
0.876
3549
0.985 10.00""
0.000
0,006
0,992 00,02"
3560
0.235 10.002
0,238 0.012
0,213 0.014
0.243 0"014"
0.191 0.007
3902
0.876 20.033"
0.829 0.028
0.858 = 0.010
0,800 =0104"
0,851 = 0.017
3900
0.754 10.003"
0.738 10.622
0.676 1 0.084
0.737 - 0.026
0.691 1 0.033
3504
0.642 -0.016
0.651 0.017
0.655 + 0.028
0.665 -0.014"
0.664 + 0.008
3913
0,833 =0.018'
0,790 0.041
0,804 0.019
0.739
3018
0.758 +0.029
0.736 0.059
0.761 +0.011"
0.717
7592
0.844 20.002"
0.831 0.004
0.838 =0.015
0,819 = 0.009
0.822 + 0.010
9910
0.708 000003
0,761 1 0.024
0.774 :: 0.044
0.783 - 0.017
0.795 00.003
9346
0:982 +0.012
0.983 0.005
0.992 + 0.021
0.994
-0.010"
0 + 0.002
9952
0.894 10.006
0,885 10.017
0.858 1 0.031
0,83 0.091
0.877 1 0.019
9400
0.995
0.017
0,040
10.011
9961
0,925 10.011
0.939 10.020
0,909 1. 0.043
0,924 0.036
0.922 T. 0.010
9971
0.098 20.053"
0.697 00.0
0.674 + 0.008
0,068 = 0.006
0.666 = 0.054
9976
0,756 20 121"
0,746 + 0.151
0,746
10.1
0.739 0.110
0.670 + 0.054
9077
0.971 +0.001
0.948 0.011
0.967 +0.000
0.9% - 0.009
0.965 + 0.802
0.840 =0.019"
0,865 +0.028
0.816
9981
0.980 =0000""
0.964 =0.014"
0.017
0,953 =
0,888 +
0085
0.484 +0.021
0.475 0.013
0.48% +0.011
0
0.477 0.018
10093
1,000 =0.000"
0.996 0.001
0.993- + 0.005
0.994 - 0.067
0 +0
10101
0.675 10.003
0.685 10.021
0.665 1. 0.049
0.661 - 0.019
0.675 1. 0.047
14952
0.959 =00000"
0.959 + 0.003
0.956 + 0.006
0.949 = 0.022
0.963 + 0.00"
14954
0.854 10.025
0.841 10.027
0,800 0.040
0.814 0.014
0.799 0.027
10.014
0.072
0,856
14909
0.545 0.028
0.641 0.014
0,585
0.546
0.605
125520
0.572 10.000
0.577 10.033"
0,535 + 0.024
0.549 - 0.050
0.566 1 0.032
125922
0.997 =0.001*
0,093 + 0.003
0.996 +0.
0.993 - 0.002
0.987 + 0.017
146195
0.094 20.022*
0.616 + 0.111
0.720
=01 022
0.551 = 0,130
0.670 0.048
146800
0.999 10.001
0.999 0.002
0.979 :: 0.015
0.999 00000"
0.994 0.008
146817
0.795 0.021
0.019
0.056
0.802
1408
0.848 =0.015"
0.039
0.818 0.022
10.016"
146830
0.924 10.0300*
0.91911 0.045
0.963 1:0 005"
0.107 0.875
0.955 : 0.008
146821
0.963 +0.040*
0.969 + 0.021
01.42 + 0.052
0.990
-0.010"
0.9K1 + 0.008
146522
0.927 10.012*
0.939 10.015
0.935 1 0.008
0.916 L 0.021
0.926 1 0.003
146524
0.962 10.000"
0.968 0.012
0.954 1 0.033
0.976 -0000"
0,974 0.008
146825
0.857 -0.014"
0.847 10.020"
0.810 0.056
0.890
0.882 10.006"
167121
0.912 0.025
0.726 0.158
0.180
0.731 0.154
167125
0.897 -0.006
0.891 0.002
0.889 =: 0.003
0.894 -0,2
0.884 :: 0.004
167140
0.944 +0.006
0.942 + 0.006
0.946
+0.004
0.991 - 0.000
0.886 + 0.000
167141
0.801 +0.002"
0.842 0.031
0.838 + 0.027
0 = 0.027
0.846 +0.027
Table 8: Comparative learning performances OpenML datasets over sampling 30 configurations of the SVM pipeline. Performances that are statistically significant compared to the second best are in bold. Statistically comparable performances are indicated with (*). Pairwise comparisons and the associated p-value along the iterations are presented in Fig.
26

---

Published as a conference paper at ICLR 2022
1 PAIRWISE COMPARISONS
Figs. highlight side-by-side comparison of METABU with each baseline set meta-features. These comparisons establish the relative improvement over each baseline, that may be lost in the general comparison, Fig. S
On Random Forest pipeline, METABU meta-features perform par with SCOT meta-features. Whereas its improvement over Landmark MF is only significant between the (approximately) 8th and 23rd iteration, METABU consistently outperforms Random and AutoSkLearm meta-features along the iterations.
Metabu MF
AutoSklearn MF
wood feas on yorr
10-1
1.6
p P_value
Rank
10
000
10
000
000
10
20
30
iteration
iteration
Metabu MF
Landmark
eexx
10-1
Rank
16
p vavluue
10
1.4
10 -3
10
20
30
10
20
30
iteration
iteration
Metabu
--- SCOT MF
10-1
P value
Rank
10 -2
1.4
10
10
20
30
10
20
30
iteration
iteration
Metabu MF
Random1x
10-3
1.75
P valuu
Rank
10 -9
1.50
000e
000
-15
1.25
30
0
iteration
iteration
Figure 9: Pairwise comparison METABU with baseline meta-features on Random Forest pipeline. Left: the average ranks. Right: the v-vulue assessing the statistical significance of the ranks according to the Mann- -hinney Wilcoxon test; the black horizontal line indicates the significance threshold p-value=0.05.
27

---

Published as a conference paper at ICLR 2022
On Adaboost, METABU meta-features perform similarly as Landmark feta-feetures. Interestingly. METABU always has a better average rank than the baselines except for the first two iterations of the AutoSkLearn baseline. It is seen that the p-value is most generally below the threshold .05, establishing the statistical significance of the rank performance.
Metabu MF
AutoSklearn MF
p P_value
10-1
Rank
1.4
10
000
00
10
20
30
10
20
30
iteration
iteration
Metabu MF
Landmark MF
10-1
1.6
p P_value
Rank
00000
10 -2
10
10
20
30
10
iteration
iteration
Metabu
SCOT MF
1.8
10-1
p. valuee
Rank
110-
10
20
30
10
20
30
iteration
iteration
Metabu MF
--- Random1x
10-1
1.8
ael whin waw
Rank
1.6
P_ D_value
10
1.4
10
1.2
10
20
30
0
10
20
30
iteration
iteration
Figure 10: Pairwise comparison of METABU with baseline meta-features on Adaboost pipeline. Left: the average ranks. Right: the p-value assessing the statistical significance of the ranks according to the Mann-Whitney Wilcoxon test; the black horizontal line indicates the significance threshold p-v0lue=0.05.
Lastly, the gaps in performance for SVM are striking. METABU meta-fcatures consistently outperform all the baselines meta-features with high confidence.
28

---

Published as a conference paper at ICLR 2022
Metabu
AutoSklearn MF
10-1
16-
p value
000000
Rank
0000
10 -3
1.4
10
20
30
10
20
30
iteration
iteration
Metabu MF
-
Landmark MF
10-1
1.6
p P_value
Rank
10-3
1.4
10
1.2
10
20
30
10
iteration
iteration
Metabu MF
SCOT MF
1.8
*t wex wisk wer te teece
10-2
Rank
1.6
p vavluue
10 -4
1.4
10 -6
1.-
0000
10
20
30
0
10
20
30
iteration
iteration
Metabu MF
--- Random1x
10
1.8
a01e0 seacl sext
Rank
1.6
p P_value
10 -4
1.4
10
1.-
10
20
30
0
10
20
30
iteration
iteration
Figure 11: Pairwise comparison of METABU with baseline meta-feature on SVM pipeline. Left: the average ranks. Right: the v-vuue assessing the statistical significance of the ranks according to the Mann- Mann-Whitne Whitney Wilcoxon test; the black horizontal line indicates the significance threshold p-5.lue=0.05.
J SENSITIVITY ANALYSIS OF d
Table 9 reports the NDCG@k performance of METABU Task 1 for varying values of d, showing that: 1) the best results are obtained for the intrinsic dimension in the vast majority of cases; ii) the sensitivity w.r.t. is very moderate.
The intrinsic dimension the OpenML benchmark is circa 6 for AutoSkLeam, for Adaboost, for RansomForest and 14 for Support Vector Machines.
29

---

Published as a conference paper at ICLR 2022
Random Forest
Adaboost
SVM
NDCG@
20
25
10
25
20
2
0.37
0.65
0.71
0.76
0.6
0.67
0.73
0.78
0.55
0.62
0.68
0.73
0.58
0.65
0.71
0.75
0.6
0.67
0.73
0.78
0.58
0.65
0.7
0.74
0.57
0.65
0.71
0.76
0.63
0.7
0.75
0.8
0.58
0.65
0.71
0.75
15
0.58
0.66
0.72
0.76
0.62
0.7
0.75
0.79
0.57
0.65
0.71
0.76
20
0.58
0.67
0.73
0.77
0.62
0.69
0.74
0.79
0.58
0.65
0.71
0.76
0.57
0.65
0.71
0.76
0.62
0.69
0.75
0.79
0.58
0.65
0.71
0.76
intrinsic
0.59
0.67
0.73
0.78
0.62
0.69
0.75
0.8
0.59
0.67
0.73
0.78
Table 9: Sensitivity of METABU w.r. the number d of METABU meta-features Task 1. The performance is the NDCG@k score measuring the relevance of the ranking induced by METABU w.r.t. the target representation.
K PERFORMANCE CURVES
These curves, in addition the rank results displayed Fig. 3b display the performance values on 10 representative datasets from OpenML CC-18, in the context of Task for respectively Random Forest (Fig. 12), Adaboost (Fig. 13), and SVM (Fig. [4). each iteration, the curve reports the average performance value with its the standard deviation (on 3 runs).
30

---

Published as a conference paper at ICLR 2022
Metabu MF
AutoSkleam MF
Landmark
SCOT Random1x
task:
task
2885
0.83
0.0
0370
0.79
0.78
039
039
093
030-
0.00
0.9
038
0.555
0.000
8825
0.000
0:365
8.60
2355
2.344
2.500
0.4
0.5
3.7
0.6
0.00
0.5
0.58
0.4
30
052
Reratin
Exnation
Figure 12: Performance curves on Random Forest.
31

---

Published as a conference paper at ICLR 2022
Metabu
AkleSk
Landmark
SCOT
Random1x
0965
8330
8935
0550
0.222
0508
0.2
0.0
0.8
078
0.4
022
8825
0800
8:60
0.0
858
0.5
033
0.90
8:5
task:
task:
0.62
0.5
035
8560
0.2
task: 167119
task: 125920
0.82
8:5
0.90
856
855
854
10
Figure 13: Performance curves on Adaboost.
32

---

Published as a conference paper at ICLR 2022
Metabu
AkleSk
Landmark
SCOT Random1x
0.0
0.5
0.7
0.-
055
0.
0.00
Saureed Accumco)
9.0
0.0
0.45
Dinnocd Accuncc
04-
02
09
0.5
146819
146825
0.3
0.8
Deseced Accuncyy
04
0.0
167121
Oeeecen Accuncy
0.
04-
0.4
02
25
Figure 14: Performance curves on SVM.
33


Reviews:
[{"review_id": "tuhynvvTTeS", "reviewer": "Reviewer_paJ4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written and well-organized. \n- The proposed method has some novelty and will contribute to the AutoML community. Learning meta-features for improving AutoML searching space is an interesting direction.\n- Experimental results prove the effectiveness of learned meta-features.\n\nWeakness:\n- Page 4-5 mentions $d$ is a main hyper-parameter of METABU and is identified using an intrinsic dimensionality procedure. But the actual values of $d$ (estimated using Facco et al., 2017) in the experiments are not reported. Since this is listed as a contribution, in the experiments, it is better to assess the sensitivity of METABU w.r.t. $d$.\n- The method has been only tested on traditional classifiers. It would be better to add experiments to neural networks. \n- The proposed method assumes the relationship between target space and meta-features is linear. Does it make sense? Is it possible to use Kernel methods or non-linear MLP to replace OT?\n- What does \"topology\" mean in the paper? It is not well-defined in the paper.", "summary_of_the_paper": "This paper tackles the AutoML problem. It proposes to learn a linear combination of manually designed meta-features, which aligns meta-features with the space of hyper-parameter configurations via an Optimal Transport procedure. Experiments on OpenML benchmark demonstrate the power of the proposed method on boosting AutoML systems.", "summary_of_the_review": "The topic and claims in this paper may be interesting to AutoML community. It contains novelty and was well written. \nOverall, this is a good paper, which is marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LX3IhbDWMHB", "reviewer": "Reviewer_cRCu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is based on the exciting idea of extracting the features of the target data as meta-features using optimal transport. For example, strengths and Weaknesses in this paper are respectively given as follows.\n\nStrength.\n- AutoML is a hot topic, and the idea of using optimal transport is interesting.\n- Experiments on Open ML CC-18 benchmark data show that the proposed method improves the performance of AutoSklearn etc.\nThe proposed method's speed is acceptable, and the meta-features provide practical hints for using ML algorithms.\n\nWeakness.\nThe proposed method's effectiveness has been shown empirically, mainly through experiments, but not sufficiently theoretically. Therefore, the limitations of the proposed method are not clear.", "summary_of_the_paper": "In this paper, the authors address the AutoML problem, which aims to automatically select the best ML algorithm and its hyperparameter configuration for a dataset, and propose an approach to this problem that learns meta-features of the dataset. The proposed method, MetaBu, learns new meta-features by optimal transport according to the space of distributions of hyperparameter configurations. Meta-features in MetaBu is known only once and induce a topology in a set of data sets. Experiments on the OpenML CC-18 benchmark have shown that MetaBu meta-features can improve the performance of the state-of-the-art AutoML systems AutoSklearn and Probabilistic Matrix Factorization. Furthermore, the examination of MetaBu meta-features provides hints on when an ML algorithm will work. Finally, a topology based on MetaBu meta-features can estimate the intrinsic dimension of the OpenML benchmark for a given ML algorithm or pipeline.\n", "summary_of_the_review": "This paper proposes a method to extract meta-features based on optimal transport to improve the performance of AutoML, which is a hot topic in the field of machine learning. The usefulness of the method is shown through benchmark data, but the theoretical argumentation is not sufficient.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C2V5UnHd_0", "reviewer": "Reviewer_A4JL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the main idea of the paper is quite intriguing. However, due to issues with the presentation, I had a hard time to follow the paper and understand what is actually done. Still, I am not fully convinced I was really able to catch the paper in its entirety.\n\nFurthermore, I could identify some statements or claims, which are either wrong or not adequately supported (either in terms of a reference or some theoretical/empirical proof):\n- AutoML is the problem of automating the entire data engineering process, not only the model selection and HPO stage.\n- \"Neural networks notoriously need large amounts of samples to be efficiently trained, [...]\": I am pretty sure that the efficiency of the training does not directly depend on the number of samples.\n- \"By construction, it can be cheaply computed for any dataset.\" => Meta features are not always cheap to compute. Especially, landmarking meta-features can become quite expensive.\n- OpenML CC-18 is referred to as the largest curated tabular dataset benchmark. However, there is a dataset published with the publication of Fusi et al. providing performance data across roughly 600 data sets.\n- \"the best regions in hyper-parameter space are the same for most datasets\" => The authors do not provide any support for this claim.\n\n\nDetailed coments:\n- The authors refer to two sets of meta features and call them \"the basic one\" and \"the target one\". First of all, it takes some time to notice that these are actually the names. Secondly, the explanation of \"the target one\" is not understandable to me.  How does such a distribution of the hyper-parameter configurations of A look like?\n- \"OpenML CC\" => OpenML CC-18\n- what does it mean \"to achieve AutoML in the context of AutoSklearn\"?\n- The definition of Optimal Transport introduces a variable q which is first not used but then suddenly appears in the expected value.\n- \"a one-hote encoding\" => an one-hot encoding\n- the explanation of what \"\\psi\" does is not understandable. The sentence starts with \"In brief, mapping \\psi sends the naive meta-feature[...]\"\n- How is the bootstrapping done that 1,0000 x n training data sets are given to the learning algorithm?\n- What is \"the AutoML selection problem\"?\n- What is meant by \"learning performance data\"?\n- What is considered to be \"enough learning performance data\"? There is no explanation on that.\n- The significance test for the performances first of all assumes a normal distribution and further that the samples are independent (which they are not, since the samples are paired with respect to the test data set). Therefore, this test cannot be applied here.\n\n# Update after Rebutttal\n\nAfter reading the other reviews, the rebuttal, and the updated version of the paper, my concerns regarding the paper vanished. Furthermore, considering the recommendations of the other reviewers it seems that I am the outlier here, so I updated my score accordingly. Still I would like to note that AutoML is not only about algorithm selection and hyper-parameter optimization. The model selection stage is what current AutoML systems are capable of but this is only a small part of the whole AutoML vision which aims to automate the entire data science process and not only model selection.", "summary_of_the_paper": "The paper \"Learning meta-features for AutoML\" proposes an approach to learn a linear combination of data set meta features. To this end, the problem of learning good meta-features is tackled via an optimal transport problem considering the data set meta features as well as the top performing hyper-parameter configurations of a learning algorithm.\n\n", "summary_of_the_review": "While the results seem to be very promising, due to the issues raised above, I feel that this paper is not ready for publication yet. Therefore, I cautiosly recommend reject.\n\n# Update after Rebuttal\nDue to the strong responses by the authors as well as the updated version of the paper I recommend accepting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5tzSxZ56WR9", "reviewer": "Reviewer_H82u", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Correctness\n\nAlthough I like the presented idea very much, I find the empirical evaluation far from convincing. The main reason for this is the sole usage of ranking based metrics in order to assess how much information the meta-features capture in different tasks. \n\nStarting with task 1, it is unclear how the relevance scores, which are required to compute the NDCG are computed. This is extremely important as NDCG values will naturally vary drastically depending on how the relevance score is defined. Moreover, assessing the quality of the meta-features in terms of a ranking metric gives only limited insight on the quality of the learned embedding. While it seems that the learned meta-features are well suited for capturing proximities to datasets with similar rankings over the algorithm configurations, it is unclear how well the embedding actually captures actual performance similarities. As an illustration, consider an example with three datasets $D_1$, $D_2$ and $D_3$ and two configurations $c_1$ and $c_2$, where $c_1 (c_2)$ yields an accuracy of 0.9 (0.1) on $D_1$, 0.4 (0.1) on $D_2$ and 0.33 (0.1) on $D_3$. Clearly, $c_1$ should always be ranked before $c_2$ as it always yields the better performance. However, when considering $D_2$ as a test dataset, it very much depends on the application if one wants $D_1$ or $D_3$ to be closer to $D_2$ in the learned embedding. If one is ONLY interested in the ranking across configurations, then the choice is irrelevant and both serve the purpose. However, if one is in addition to the ranking also interested in an estimated performance (e.g. for the usage inside an acquisition function in Bayesian optimization such as in AutoSklearn), then $D_3$ should be closer to $D_2$ than $D_1$ as the performance of $c_1$ on $D_3$ is closer to its performance on $D_2$.  Depending on how the relevance score of the NDCG metric is defined, the metric might also indirectly give an insight wrt. to the aforementioned example. However, adding (in addition to the ranking metric) another metric, which quantifies performance similarities in a more direct manner, is crucial to understand the quality of the learned meta-features and should thus be added.\n\nThe main weaknesses of the evaluation can be found in tasks 2 and 3, in my opinion. Showing only rank plots in the main paper is insufficient as they hide the degree of improvement. This becomes particularly clear if one considers the actual performance tables provided in appendix H (for task 2) where one can see that on many datasets the performance differences between METABU MF and  the other methods are negligible. In fact, there seems to be almost no statistically significant differences with only a handful of exceptions. To me this is quite in contrast to the claims made by the authors in the main paper that their approach is statistically significantly better than the baselines. This might be true for the average rank, but it seems to be far from the truth when looking at the exact performances. Furthermore, there are no detailed results for task 3 at all in the appendix such that the rank plots are even less meaningful for task 3. The drastically varying p-levels in Tables 6-8 in the appendix also do not give a good impression. In my opinion, the authors should revise the evaluation to show concrete performance gains instead of rank plots. In particular, instead of showing the ranks of the methods over the time of the optimization, I suggest to plot the concrete performance of the corresponding solutions. However, from the results in the appendix I have doubts that such improvements exist in a non-negligible form. \n\nFurthermore, it is unclear to me why (for task 3) the target distributions for AutoSklearn and PMF are different. In my eyes, the target distribution should be independent of the used AutoML method and only depend on the search space of the corresponding method. If this is meant by the authors, I strongly suggest to use the same search space for AutoSklearn and PMF as the results are hard to compare otherwise. \n\nLastly, I believe that some statements in the paper are formulated in a way making them at least debatable:\n\n- p. 2: \"the ML meta-features have hardly been effective to achieve AutoML (Misir & Sebag, 2017) or even to distinguish among hard and easy datasets w.r.t. a given learning algorithm (Muñoz et al., 2018)\" → I believe this claim to be wrong. Several works have shown that meta-features are well suited for algorithm recommendation even in the context of ML algorithms (including AutoSklearn 2.0 [1]).\n- p. 3: \"By construction, it can be cheaply computed for any dataset.\" → Not true for landmarking features, which are notoriously expensive.\n- p. 6 goal of experiments: \"The first goal is to measure the performance of the METABU meta-features\" → meta-features themselves cannot have a performance.\n- Explanation of results\n    - Task 1: \"[...] the metrics based on the baselines are deterministic\" → the landmarking baselines are certainly not deterministic as they also rely on training the respective ML algorithm whose performance is tracked\n    - Task 2: \"METABU dominates after the beginning; all approaches but the uniform sampler yield similar performances\" → Contradiction. Does it dominate or is performance similar? Both statements cannot be true at the same time.\n- Ethics statement: I do not believe that the goal of AutoML is to reduce the computational resources needed to get peak performance from an ML portfolio. While I agree that AutoML methods are often faster in generating a high quality ML pipeline for a given dataset than experts as some competitions have shown, I highly doubt that they use less computational resources than the corresponding human experts would.\n\nWith all of the above in mind, I believe that one cannot assess the empirical quality of the proposed methodology based on the current state of the evaluation. Accordingly, claims made regarding the quality of the methodology are not well supported, in my opinion.\n\n## References\n\n[1] Feurer, Matthias, et al. \"Auto-sklearn 2.0: The next generation.\" arXiv preprint arXiv:2007.04074 (2020).\n\n# Novelty and Significance of Technical Algorithms, Models, or Theory\n\nWhile the proposed methodology certainly is novel in the specific context of AutoML, it is well known in other contexts as acknowledged by the authors. I believe this to be completely okay for an empirical field such as AutoML. However, I believe that the authors could have focused more on giving explanations on other embedding approaches from the context of AutoML. Although the authors point to PMF and OBOE, the differences of the embedding learned in this work compared to the other embedding approaches should be explored more thoroughly on a methodological level. Thus, while the work is well contextualized wrt. to existing literature on a broad level, it lacks a good contextualization on a more nuanced level.\n\nThe significance of the methodology is hard to assess as the empirical evaluation is not reliable as stated under correctness. However, assuming that the methodology does indeed yield good meta-features, I deem it significant for the AutoML community. \n\n\n# Novelty and Significance of Empirical Advancements, Insights, or Datasets\n\nSince I have doubts regarding the empirical evaluation as a whole in this work, I deem it to be non-significant. However, the results presented in 4.4 are certainly interesting and I have not seen an analysis of the intrinsic dimension of the OpenML CC-18 benchmark before. Thus, at least this part of the evaluation yields novel insights. \n\n\n# Minor Issues\n\n- different writings of AutoSklearn throughout the paper\n- in contrast with → in contrast to\n- \"OT, the Wasserstein distance [...]\" → OT seems to be out of place there.\n- p. 4: difference in notation between step 1 and step 2 - in step 1 \"d\" is the 2-Wasserstein distance whereas in step 2 \"d_W^2\" is the distance.\n\n-----\n# Update after Rebuttal\n\nI would like to thank the authors for their strong rebuttal and excuse myself for misjudging that the learned embedding is performance preserving instead of rank preserving rendering my criticism of the evaluation task 1 infeasible. My remaining questions have been thoroughly answered and the requested changes to the evaluation have been added to the paper where applicable. As my main concern has been with the evaluation, all reasons why I could not suggest acceptance of this paper have been ruled out. Although I do not agree to all opinions presented in the paper and the rebuttal (e.g. computational resources), I do not have any further factual criticism.  As such, I will adjust my rating correspondingly to suggest acceptance of this work. ", "summary_of_the_paper": "The paper presents an approach, called MetaBu, for learning a meta-feature embedding from an existing meta-feature space into a latent space, which is aims at being rank preserving regarding different hyper-parameter configurations. The special kind of embedding and its property of aiming at being performance preserving in the context of AutoML is the main contribution of the paper, in my opinion. The quality of the learned meta-features is assessed through different experiments such as capturing to what degree the embedding is indeed performance preserving and how well AutoML tools perform when initialized with the corresponding meta-features. Moreover, the authors provide a sensitivity analysis of relevant hyper-parameters of MetaBu and demonstrate how to gain insights from the learned embeddings.", "summary_of_the_review": "Overall I like the presented idea very much and I deem the paper to be overall well written, except for some debatable statements (see Correctness). As such, both the methodology and the experiments are well described. Unfortunately, the part of the evaluation assessing the quality of the learned meta-features is insufficient (see Correctness) and needs to be revised in my opinion. Accordingly, I cannot recommend acceptance of this work in its current form.\n\n-----\n# Update after Rebuttal\nAll my concerns have been fixed or ruled out by the authors in the revised version. I recommend to accept the paper in its revised form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3ywz1EoTWeC", "reviewer": "Reviewer_f2By", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- This is a well written paper for the most part, proposing a very interesting meta-learning technique. It is relatively easy to follow from my perspective.\n- The proposed method combines two forms of meta-learning: one that solely leverages the dataset meta-features to develop a notion of similarity between datasets, and use that information to seed the AutoML; and the other that does not focus on the dataset meta-features  but solely relies on the performance of similar hyperparameters on different datasets to implicitly define similarities between pairs of datasets.\n- This method focuses on the general AutoML problem for tabular data where we cannot leverage well-established techniques used for AutoML in deep learning.\n- The proposed scheme utilizes a linear transformation of existing interpretable meta-features, allowing for interpretation of dataset landscapes and differences between the AutoML problems with different ML models.\n\n\n\nWeaknesses:\n\n- While most of the paper was quite easy to follow, the short paragraphs on the \"intrinsic dimension of the space of datasets\" and the corresponding empirical evaluation and discussion (in section 4.4) is a little hard for me to follow. I realize that partly the issue is the novelty of the considered problem -- usually we are thinking about the intrinsic dimension of the dataset, but here we need to focus on the intrinsic dimension of a meta-dataset of datasets with respect to a particular AutoML problem, where a dataset is a \"sample\" in this meta-dataset, and the intrinsic dimension of this meta-dataset somewhat quantifies the difficulty of the AutoML problem for a particular AutoML problem (that is, a hyperparameter optimization for a specific ML model). However, the empirical results and discussion seem counterintuitive to me -- to me the AutoML problem in AutoSklearn (which is solving a Combined Algorithm Selection and Hyperparameter optimization or CASH problem) is a harder AutoML problem than the AutoML problem for SVM. In fact, the AutoML problem of SVM is part of the AutoML problem in Auto-sklearn. The search space of Auto-sklearn contains the search space of SVM (and RandomForest and Adaboost). So such a result confuses my interpretation of the learned meta-features and the meta-dataset of datasets. I thought the most \"flexible\" AutoML problem will also be the hardest, not the easiest.\n", "summary_of_the_paper": "This paper focuses on the AutoML problem for tabular data and proposes a meta-learning based novel solution. They consider the optimal transport to define distances between two datasets, utilizing the Wasserstein-Gromov distance between the distribution of the top performing hyperparameters for the respective datasets. Given this distance, they propose learning a linear transformation of existing dataset meta-features such that the Euclidean distance between a pair of datasets in this transformed space is proportional to their Wasserstein-Gromov distance. This method is termed Metabu.\n\nThe empirical evaluation compares Metabu to existing meta-learning schemes on (i) their ability to capture the desired Wasserstein-Gromov distance, (ii) their ability to find better hyperparameters via sampling without an underlying optimizer, and (iii) their ability to find better seed hyperparameters for hyperparameter optimizers. The results on the OpenML CC-18 suite with 3 machine learning models indicate that Metabu significantly improves upon existing meta-learning schemes. The paper also demonstrates how the learned linear transformation of existing dataset meta-features allow us understand the importance of different existing dataset meta-features and how these vary between machine learning models. \n", "summary_of_the_review": "Overall, I would like to recommend this paper for an accept for the following reasons:\n\n- I think this is a well-written paper that positions itself well relative to existing work, and presents the novel scheme is a clear and easy-to-follow manner.\n- The paper presents a significantly new meta-learning scheme building upon existing meta-learning techniques. Hence the novelty is significant in my opinion.\n- The empirical evaluation is done against valid baselines, and the proposed scheme shows significant margins of improvement.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses an important issue of AutoML systems, specifically their ability to "cold start" on a new problem. Some of the reviewers initially had concerns about the experimental validation and the theoretical foundations of the method, but during the discussion phase the authors addressed concerns extremely well. The authors already included most of the feedback of reviewers, further strengthening the paper.
Decision: Accept (Spotlight)