Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors address the AutoML problem, which aims to automatically select the best ML algorithm and its hyperparameter configuration for a dataset, and propose an approach to this problem that learns meta-features of the dataset. The proposed method, MetaBu, learns new meta-features by optimal transport according to the space of distributions of hyperparameter configurations. Meta-features in MetaBu is known only once and induce a topology in a set of data sets. Experiments on the OpenML CC-18 benchmark have shown that MetaBu meta-features can improve the performance of the state-of-the-art AutoML systems AutoSklearn and Probabilistic Matrix Factorization. Furthermore, the examination of MetaBu meta-features provides hints on when an ML algorithm will work. Finally, a topology based on MetaBu meta-features can estimate the intrinsic dimension of the OpenML benchmark for a given ML algorithm or pipeline.

main_review: The proposed method is based on the exciting idea of extracting the features of the target data as meta-features using optimal transport. For example, strengths and Weaknesses in this paper are respectively given as follows.

Strength.
- AutoML is a hot topic, and the idea of using optimal transport is interesting.
- Experiments on Open ML CC-18 benchmark data show that the proposed method improves the performance of AutoSklearn etc.
The proposed method's speed is acceptable, and the meta-features provide practical hints for using ML algorithms.

Weakness.
The proposed method's effectiveness has been shown empirically, mainly through experiments, but not sufficiently theoretically. Therefore, the limitations of the proposed method are not clear.
summary_of_the_review: This paper proposes a method to extract meta-features based on optimal transport to improve the performance of AutoML, which is a hot topic in the field of machine learning. The usefulness of the method is shown through benchmark data, but the theoretical argumentation is not sufficient.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the AutoML problem. It proposes to learn a linear combination of manually designed meta-features, which aligns meta-features with the space of hyper-parameter configurations via an Optimal Transport procedure. Experiments on OpenML benchmark demonstrate the power of the proposed method on boosting AutoML systems.
main_review: Strengths:
- The paper is well-written and well-organized. 
- The proposed method has some novelty and will contribute to the AutoML community. Learning meta-features for improving AutoML searching space is an interesting direction.
- Experimental results prove the effectiveness of learned meta-features.

Weakness:
- Page 4-5 mentions $d$ is a main hyper-parameter of METABU and is identified using an intrinsic dimensionality procedure. But the actual values of $d$ (estimated using Facco et al., 2017) in the experiments are not reported. Since this is listed as a contribution, in the experiments, it is better to assess the sensitivity of METABU w.r.t. $d$.
- The method has been only tested on traditional classifiers. It would be better to add experiments to neural networks. 
- The proposed method assumes the relationship between target space and meta-features is linear. Does it make sense? Is it possible to use Kernel methods or non-linear MLP to replace OT?
- What does "topology" mean in the paper? It is not well-defined in the paper.
summary_of_the_review: The topic and claims in this paper may be interesting to AutoML community. It contains novelty and was well written. 
Overall, this is a good paper, which is marginally above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on the AutoML problem for tabular data and proposes a meta-learning based novel solution. They consider the optimal transport to define distances between two datasets, utilizing the Wasserstein-Gromov distance between the distribution of the top performing hyperparameters for the respective datasets. Given this distance, they propose learning a linear transformation of existing dataset meta-features such that the Euclidean distance between a pair of datasets in this transformed space is proportional to their Wasserstein-Gromov distance. This method is termed Metabu.

The empirical evaluation compares Metabu to existing meta-learning schemes on (i) their ability to capture the desired Wasserstein-Gromov distance, (ii) their ability to find better hyperparameters via sampling without an underlying optimizer, and (iii) their ability to find better seed hyperparameters for hyperparameter optimizers. The results on the OpenML CC-18 suite with 3 machine learning models indicate that Metabu significantly improves upon existing meta-learning schemes. The paper also demonstrates how the learned linear transformation of existing dataset meta-features allow us understand the importance of different existing dataset meta-features and how these vary between machine learning models. 

main_review: Strengths:

- This is a well written paper for the most part, proposing a very interesting meta-learning technique. It is relatively easy to follow from my perspective.
- The proposed method combines two forms of meta-learning: one that solely leverages the dataset meta-features to develop a notion of similarity between datasets, and use that information to seed the AutoML; and the other that does not focus on the dataset meta-features  but solely relies on the performance of similar hyperparameters on different datasets to implicitly define similarities between pairs of datasets.
- This method focuses on the general AutoML problem for tabular data where we cannot leverage well-established techniques used for AutoML in deep learning.
- The proposed scheme utilizes a linear transformation of existing interpretable meta-features, allowing for interpretation of dataset landscapes and differences between the AutoML problems with different ML models.



Weaknesses:

- While most of the paper was quite easy to follow, the short paragraphs on the "intrinsic dimension of the space of datasets" and the corresponding empirical evaluation and discussion (in section 4.4) is a little hard for me to follow. I realize that partly the issue is the novelty of the considered problem -- usually we are thinking about the intrinsic dimension of the dataset, but here we need to focus on the intrinsic dimension of a meta-dataset of datasets with respect to a particular AutoML problem, where a dataset is a "sample" in this meta-dataset, and the intrinsic dimension of this meta-dataset somewhat quantifies the difficulty of the AutoML problem for a particular AutoML problem (that is, a hyperparameter optimization for a specific ML model). However, the empirical results and discussion seem counterintuitive to me -- to me the AutoML problem in AutoSklearn (which is solving a Combined Algorithm Selection and Hyperparameter optimization or CASH problem) is a harder AutoML problem than the AutoML problem for SVM. In fact, the AutoML problem of SVM is part of the AutoML problem in Auto-sklearn. The search space of Auto-sklearn contains the search space of SVM (and RandomForest and Adaboost). So such a result confuses my interpretation of the learned meta-features and the meta-dataset of datasets. I thought the most "flexible" AutoML problem will also be the hardest, not the easiest.

summary_of_the_review: Overall, I would like to recommend this paper for an accept for the following reasons:

- I think this is a well-written paper that positions itself well relative to existing work, and presents the novel scheme is a clear and easy-to-follow manner.
- The paper presents a significantly new meta-learning scheme building upon existing meta-learning techniques. Hence the novelty is significant in my opinion.
- The empirical evaluation is done against valid baselines, and the proposed scheme shows significant margins of improvement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper "Learning meta-features for AutoML" proposes an approach to learn a linear combination of data set meta features. To this end, the problem of learning good meta-features is tackled via an optimal transport problem considering the data set meta features as well as the top performing hyper-parameter configurations of a learning algorithm.


main_review: Overall, I think the main idea of the paper is quite intriguing. However, due to issues with the presentation, I had a hard time to follow the paper and understand what is actually done. Still, I am not fully convinced I was really able to catch the paper in its entirety.

Furthermore, I could identify some statements or claims, which are either wrong or not adequately supported (either in terms of a reference or some theoretical/empirical proof):
- AutoML is the problem of automating the entire data engineering process, not only the model selection and HPO stage.
- "Neural networks notoriously need large amounts of samples to be efficiently trained, [...]": I am pretty sure that the efficiency of the training does not directly depend on the number of samples.
- "By construction, it can be cheaply computed for any dataset." => Meta features are not always cheap to compute. Especially, landmarking meta-features can become quite expensive.
- OpenML CC-18 is referred to as the largest curated tabular dataset benchmark. However, there is a dataset published with the publication of Fusi et al. providing performance data across roughly 600 data sets.
- "the best regions in hyper-parameter space are the same for most datasets" => The authors do not provide any support for this claim.


Detailed coments:
- The authors refer to two sets of meta features and call them "the basic one" and "the target one". First of all, it takes some time to notice that these are actually the names. Secondly, the explanation of "the target one" is not understandable to me.  How does such a distribution of the hyper-parameter configurations of A look like?
- "OpenML CC" => OpenML CC-18
- what does it mean "to achieve AutoML in the context of AutoSklearn"?
- The definition of Optimal Transport introduces a variable q which is first not used but then suddenly appears in the expected value.
- "a one-hote encoding" => an one-hot encoding
- the explanation of what "\psi" does is not understandable. The sentence starts with "In brief, mapping \psi sends the naive meta-feature[...]"
- How is the bootstrapping done that 1,0000 x n training data sets are given to the learning algorithm?
- What is "the AutoML selection problem"?
- What is meant by "learning performance data"?
- What is considered to be "enough learning performance data"? There is no explanation on that.
- The significance test for the performances first of all assumes a normal distribution and further that the samples are independent (which they are not, since the samples are paired with respect to the test data set). Therefore, this test cannot be applied here.

# Update after Rebutttal

After reading the other reviews, the rebuttal, and the updated version of the paper, my concerns regarding the paper vanished. Furthermore, considering the recommendations of the other reviewers it seems that I am the outlier here, so I updated my score accordingly. Still I would like to note that AutoML is not only about algorithm selection and hyper-parameter optimization. The model selection stage is what current AutoML systems are capable of but this is only a small part of the whole AutoML vision which aims to automate the entire data science process and not only model selection.
summary_of_the_review: While the results seem to be very promising, due to the issues raised above, I feel that this paper is not ready for publication yet. Therefore, I cautiosly recommend reject.

# Update after Rebuttal
Due to the strong responses by the authors as well as the updated version of the paper I recommend accepting the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an approach, called MetaBu, for learning a meta-feature embedding from an existing meta-feature space into a latent space, which is aims at being rank preserving regarding different hyper-parameter configurations. The special kind of embedding and its property of aiming at being performance preserving in the context of AutoML is the main contribution of the paper, in my opinion. The quality of the learned meta-features is assessed through different experiments such as capturing to what degree the embedding is indeed performance preserving and how well AutoML tools perform when initialized with the corresponding meta-features. Moreover, the authors provide a sensitivity analysis of relevant hyper-parameters of MetaBu and demonstrate how to gain insights from the learned embeddings.
main_review: # Correctness

Although I like the presented idea very much, I find the empirical evaluation far from convincing. The main reason for this is the sole usage of ranking based metrics in order to assess how much information the meta-features capture in different tasks. 

Starting with task 1, it is unclear how the relevance scores, which are required to compute the NDCG are computed. This is extremely important as NDCG values will naturally vary drastically depending on how the relevance score is defined. Moreover, assessing the quality of the meta-features in terms of a ranking metric gives only limited insight on the quality of the learned embedding. While it seems that the learned meta-features are well suited for capturing proximities to datasets with similar rankings over the algorithm configurations, it is unclear how well the embedding actually captures actual performance similarities. As an illustration, consider an example with three datasets $D_1$, $D_2$ and $D_3$ and two configurations $c_1$ and $c_2$, where $c_1 (c_2)$ yields an accuracy of 0.9 (0.1) on $D_1$, 0.4 (0.1) on $D_2$ and 0.33 (0.1) on $D_3$. Clearly, $c_1$ should always be ranked before $c_2$ as it always yields the better performance. However, when considering $D_2$ as a test dataset, it very much depends on the application if one wants $D_1$ or $D_3$ to be closer to $D_2$ in the learned embedding. If one is ONLY interested in the ranking across configurations, then the choice is irrelevant and both serve the purpose. However, if one is in addition to the ranking also interested in an estimated performance (e.g. for the usage inside an acquisition function in Bayesian optimization such as in AutoSklearn), then $D_3$ should be closer to $D_2$ than $D_1$ as the performance of $c_1$ on $D_3$ is closer to its performance on $D_2$.  Depending on how the relevance score of the NDCG metric is defined, the metric might also indirectly give an insight wrt. to the aforementioned example. However, adding (in addition to the ranking metric) another metric, which quantifies performance similarities in a more direct manner, is crucial to understand the quality of the learned meta-features and should thus be added.

The main weaknesses of the evaluation can be found in tasks 2 and 3, in my opinion. Showing only rank plots in the main paper is insufficient as they hide the degree of improvement. This becomes particularly clear if one considers the actual performance tables provided in appendix H (for task 2) where one can see that on many datasets the performance differences between METABU MF and  the other methods are negligible. In fact, there seems to be almost no statistically significant differences with only a handful of exceptions. To me this is quite in contrast to the claims made by the authors in the main paper that their approach is statistically significantly better than the baselines. This might be true for the average rank, but it seems to be far from the truth when looking at the exact performances. Furthermore, there are no detailed results for task 3 at all in the appendix such that the rank plots are even less meaningful for task 3. The drastically varying p-levels in Tables 6-8 in the appendix also do not give a good impression. In my opinion, the authors should revise the evaluation to show concrete performance gains instead of rank plots. In particular, instead of showing the ranks of the methods over the time of the optimization, I suggest to plot the concrete performance of the corresponding solutions. However, from the results in the appendix I have doubts that such improvements exist in a non-negligible form. 

Furthermore, it is unclear to me why (for task 3) the target distributions for AutoSklearn and PMF are different. In my eyes, the target distribution should be independent of the used AutoML method and only depend on the search space of the corresponding method. If this is meant by the authors, I strongly suggest to use the same search space for AutoSklearn and PMF as the results are hard to compare otherwise. 

Lastly, I believe that some statements in the paper are formulated in a way making them at least debatable:

- p. 2: "the ML meta-features have hardly been effective to achieve AutoML (Misir & Sebag, 2017) or even to distinguish among hard and easy datasets w.r.t. a given learning algorithm (Muñoz et al., 2018)" → I believe this claim to be wrong. Several works have shown that meta-features are well suited for algorithm recommendation even in the context of ML algorithms (including AutoSklearn 2.0 [1]).
- p. 3: "By construction, it can be cheaply computed for any dataset." → Not true for landmarking features, which are notoriously expensive.
- p. 6 goal of experiments: "The first goal is to measure the performance of the METABU meta-features" → meta-features themselves cannot have a performance.
- Explanation of results
    - Task 1: "[...] the metrics based on the baselines are deterministic" → the landmarking baselines are certainly not deterministic as they also rely on training the respective ML algorithm whose performance is tracked
    - Task 2: "METABU dominates after the beginning; all approaches but the uniform sampler yield similar performances" → Contradiction. Does it dominate or is performance similar? Both statements cannot be true at the same time.
- Ethics statement: I do not believe that the goal of AutoML is to reduce the computational resources needed to get peak performance from an ML portfolio. While I agree that AutoML methods are often faster in generating a high quality ML pipeline for a given dataset than experts as some competitions have shown, I highly doubt that they use less computational resources than the corresponding human experts would.

With all of the above in mind, I believe that one cannot assess the empirical quality of the proposed methodology based on the current state of the evaluation. Accordingly, claims made regarding the quality of the methodology are not well supported, in my opinion.

## References

[1] Feurer, Matthias, et al. "Auto-sklearn 2.0: The next generation." arXiv preprint arXiv:2007.04074 (2020).

# Novelty and Significance of Technical Algorithms, Models, or Theory

While the proposed methodology certainly is novel in the specific context of AutoML, it is well known in other contexts as acknowledged by the authors. I believe this to be completely okay for an empirical field such as AutoML. However, I believe that the authors could have focused more on giving explanations on other embedding approaches from the context of AutoML. Although the authors point to PMF and OBOE, the differences of the embedding learned in this work compared to the other embedding approaches should be explored more thoroughly on a methodological level. Thus, while the work is well contextualized wrt. to existing literature on a broad level, it lacks a good contextualization on a more nuanced level.

The significance of the methodology is hard to assess as the empirical evaluation is not reliable as stated under correctness. However, assuming that the methodology does indeed yield good meta-features, I deem it significant for the AutoML community. 


# Novelty and Significance of Empirical Advancements, Insights, or Datasets

Since I have doubts regarding the empirical evaluation as a whole in this work, I deem it to be non-significant. However, the results presented in 4.4 are certainly interesting and I have not seen an analysis of the intrinsic dimension of the OpenML CC-18 benchmark before. Thus, at least this part of the evaluation yields novel insights. 


# Minor Issues

- different writings of AutoSklearn throughout the paper
- in contrast with → in contrast to
- "OT, the Wasserstein distance [...]" → OT seems to be out of place there.
- p. 4: difference in notation between step 1 and step 2 - in step 1 "d" is the 2-Wasserstein distance whereas in step 2 "d_W^2" is the distance.

-----
# Update after Rebuttal

I would like to thank the authors for their strong rebuttal and excuse myself for misjudging that the learned embedding is performance preserving instead of rank preserving rendering my criticism of the evaluation task 1 infeasible. My remaining questions have been thoroughly answered and the requested changes to the evaluation have been added to the paper where applicable. As my main concern has been with the evaluation, all reasons why I could not suggest acceptance of this paper have been ruled out. Although I do not agree to all opinions presented in the paper and the rebuttal (e.g. computational resources), I do not have any further factual criticism.  As such, I will adjust my rating correspondingly to suggest acceptance of this work. 
summary_of_the_review: Overall I like the presented idea very much and I deem the paper to be overall well written, except for some debatable statements (see Correctness). As such, both the methodology and the experiments are well described. Unfortunately, the part of the evaluation assessing the quality of the learned meta-features is insufficient (see Correctness) and needs to be revised in my opinion. Accordingly, I cannot recommend acceptance of this work in its current form.

-----
# Update after Rebuttal
All my concerns have been fixed or ruled out by the authors in the revised version. I recommend to accept the paper in its revised form.
review_title: null

Meta-review: This paper addresses an important issue of AutoML systems, specifically their ability to "cold start" on a new problem. Some of the reviewers initially had concerns about the experimental validation and the theoretical foundations of the method, but during the discussion phase the authors addressed concerns extremely well. The authors already included most of the feedback of reviewers, further strengthening the paper.
Decision: Accept (Spotlight)