Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new Kernel for Bayesian Optimization to incorporate 'Landmark meta-features' of groups of tasks, so that the Kernel can be used to improve the efficience of the Bayesian Optimization on a new related task. The Kernel is based on 2 nested neural networks that are jointly trained via first-order meta-learning method REPTILE. The authors show that the method is overall better performing on the benchmark HPO-B-v3.
main_review: Strengths:
- The topic is important. Hyperparameter is expensive and transfer learning is a promising solution to reduce the cost of HPO. 
- The solution is relatively complex, involving deep Kernels using neural networks and trained with meta-learning, and yet it is presented clearly and is easy to understand.
- The benchmark is large, including 10 different relevant algorithms on 935 tasks with 16 different search spaces. 
- The experiments cover different aspects of the contributed algorithm; T-SNE projections of the meta-features and the impact of the sample size.

Weaknesses:
- A lot of emphasize in introduction is on the fact that DKLM would address negative transfer. I do not see this discussed explicitly in any of the experiments. If it is important, it should be supported by the experiments.
- The proposed solution is most certainly more computationally expensive than random search, or the fast Bayesian Optimization algorithm ABLR. The experiments should also show the improvement in terms of running time so that we can see how it performs compared to these faster algorithm. 
- The standard deviation on Table 1 is way too large to be able to conclude anything. I could run again the same experiments with different seeds and observe an upward trend instead. 
- The functions f and g introduces many hyperparameters to DKLM. The paper does not discuss how to chose these hyperparameters and whether tuning them is important for the performance of the algorithm. 

Minor comments:
What is a meta-dataset? It is not defined in section 2.

I don’t understand this sentence:
> Consequently, the solution of the joint model resides on a local minimum so that given limited information about the new (unseen) target task.

Typos:
new a target task -> a new target task
fast adaption -> fast adaptation
where we notice -> while we notice
summary_of_the_review: The proposed algorithm is an interesting way of learning end-to-end meta-features across tasks to improve the efficiency of Bayesian Optimization on new similar tasks. The benchmarking experiments are extensive and convincing for the efficiency of the algorithm, but does not support clearly the claims about negative transfer. The latter point should be discussed more clearly. The additional hyperparameters of the algorithms (network size and REPTILE hyperparameters) should also analyzed empirically to see how importantly they affect the performance of the algorithm.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors focus on the important problem of more efficient hyperparamter optimization through (meta) transfer learning. More specifically, the author propose a new method (Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM)) that embeds a meta-feature extractor model within the kernel function of a deep GP. This meta-feature extractor is able to learn to extract characteristics of a dataset such that the similarity of a dataset to other ones can be (implicitly) captured and used by the rest of the model. To demonstrate the effectiveness of their proposed model, they evaluate their model on the large-scale HPO-B-v3 benchmark against 10 other baselines plus a variant of their method with a randomly-initialized meta extractor.
main_review: Strengths:
- Adequately builds upon previous work such that the proposed change (the meta-extractor within the kernel of a deep GP) is easy to assess and compare (as opposed to larger set of changes that are not adequate ablated).
- Large-scale benchmark (935 tasks, 16 search spaces) compared to 10 baselines.
- Ablations with and within a randomly-initialized meta-extractor.
- Results included error bars (Table 1).

Weaknesses:
- Experimental setup could be made clearer as to which tasks are used for training and eval in the transfer setting (i.e., which tasks are used to originally train the meta-extractor, and on which tasks is it transferred).

Minor:
- p. 3: In the fourth line from the bottom, the reference "2020" needs to be fixed.
summary_of_the_review: Overall, the authors propose a new method that is well-scoped within existing literature, and provide positive empirical results on a large-scale benchmark against an adequate set of baseline tasks. It is a reasonable paper for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles the issue of speeding up black-box hyper-parameter optimization (HPO) by leveraging results obtained by trials on different datasets. It casts the problem into a few-shot regression one, where each task is a dataset and the goal is to predict the performance of a model trained on this dataset from the values of hyper-parameters. Meta-learning (REPTILE) is then used to initialize the (meta-)parameters of a deep kernel Gaussian process. Experiments on the HPO-B-v3 benchmark show this procedure equals or beats 10 baselines when using 20 to 100 trials.
main_review: 
Strengths
========
The paper very clearly explains the background, existing methods, and issues. It shows how various existing techniques and ideas (deep kernel Gaussian processes, landmark meta-features, deepset) are combined with meta-learning to combat the "negative transfer" phenomenon.

Experiments on the large HPO-B benchmark, as well as ablations (on synthetic few-shot regression problems, and in a non-transfer setting), are well-designed and support the paper's conclusions. The research question is clearly stated.

Weaknesses
==========
One would expect that the transfer/few-shot learning setting may be most useful with a small number of trials, however we see that the improvements (wrt. baselines) seem to only appear after about 10-20 trials, and are sustained afterwards. This may be investigated, or maybe addressed as a limitation in the context of the stated goal of limiting the number of HP configuration trials.

Minor points
==========
- Some confusion in notations in section 3.2
  * In Eq (1), is $K_x$ supposed to be $K_*$?
  * A mean function $m$ is introduced, but not used, and Eq (1) shows the mean as $0$.
  * In Eq (2), $y$ vs. ${\mathbf y}$
- It would be nice to keep the colors consistent between Figures 2 and 3 for the common curves
summary_of_the_review: Overall good, well-written, self-contained article that presents a novel combination of existing techniques, and addresses the issue of negative transfer when using transfer learning for Bayesian HPO. The experiments conducted are well designed and support the conclusion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on doing transfer learning from related tasks for the optimization of ML model hyperparameters.

Previous work (e.g., Vanschoren, 2018) has considered meta-features as representation of related datasets or tasks; these features are incorporated into the model when transferring to the task at hand. This work proposes to _learn_ these meta features using a deep-set representation (Zaheer et al., 2017). These meta features are then passed to a kernel function that, finally, parameterizes the Gaussian process used during Bayesian optimization.

Experimentally, the authors compare their method to several baselines, both in the transfer and non-transfer learning settings; aggregated metrics show that the proposed method (DKLM) outperforms other baselines on average.


main_review: Strengths
-------------
- This paper combines two intuitive ideas, and the method is fairly easy to understand
- Experimentally, aggregated metrics showcase the strength of the proposed DKLM method compared to other baselines
- The ablation test which removes the _learned_ meta features convincingly shows that investing additional effort into obtaining trained meta-features improves performance.

Weaknesses 
-----------------
- The major weakness of this paper is that it seems to simply combine to well-known ideas (deep sets + deep kernel GPs), and as such has limited novelty. 
- The clarify of the paper could be improved. In particular, the experimental setup was not entirely obvious to me. Is the optimization done as Bayesian optimization (with a DKLM in the Gaussian process)? If so, which acquisition function was used?

Comments/questions
----------------------------
- You mention the "negative transfer phenomenon" several times as a major motivation for this work; I would recommend describing it in more detail in the paper.
- I'm curious about the performance variability of DKLM: do you have any insight into which types of datasets or experimental settings DKLM tends to perform better or worse on, compared to other baselines?
- Did you compare DKLM to a GP parameterized with the learned metafeatures from Dataset2Vec?

summary_of_the_review: This is an interesting paper, but its novelty is limited. A more detailed analysis of the proposed method (in particular, ablations that use a deep kernel GP with different learned meta-features) would increase the impact of this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces DKLM, a novel approach to hyperparameter optimisation based on transfer learning.
DKLM uses a Deep Kernel GP (Wilson et al., 2016) surrogate to predict test losses / response functions for given hyperparameter configurations.
The embeddings of the Deep Kernel GP are obtained by aggregating over hyperparameter-response pairs of previously observed datasets.
This is the main mechanism through which the HPO is conditioned on previously observed experiments.
The embedding mechanism relies on a Deepset (Zaheer et al., 2017) style neural network encoding, that sums over all hyperparameter-response pairs.
Experiments demonstrate DKLM improves performance over prior work.
main_review: ## Strengths

I believe that there are merits to the proposed DKLM approach.
It makes intuitive sense to me that the Deepset style inclusion of prior observations is preferable to using prior observations as initialisation for the GP as done by prior work.

The hpo-b-v3 experiments look somewhat convincing.
DKLM performs decently in the "non-transfer setting" and impressively well in the transfer setting.
I'm inclined to believe the validity of these results, although I wish the authors would have included code / more details of the setup.

## Weaknesses

Unfortunately, I feel that many parts of this paper are severely lacking in clarity.
The writing often feels unnecessarily convoluted and I am sometimes unsure of what exactly the authors are trying to tell me with a particular paragraph.
Further, the lack of clarity unfortunately leads to open questions about the proposed approach itself.
Below I give specific details of this.
I hope the authors understand my criticisms as constructive and take them as an opportunity to improve the current draft of the paper.

### Introduction

By the end of the introduction, I neither understand what open challenges in HPO are, nor what DKLM does differently to established works.
Concretely, I find the following sentences in the introduction more confusing than helpful:
* "we propose a novel architecture for deep GP kernels (Wilson et al., 2016a) that are enriched with novel end-to-end neural network components that generate meta-features only from the tuples of past hyperparameter configurations and their evaluated performances" --> How do I enrich a deep GP kernel with neural network components? Don't Deep Kernel GPs already have neural networks do produce the encoding?
* "Introduce the first paper that tackles the negative transfer phenomenon in Bayesian HPO" → What is the negative transfer phenomenon? This is not explained here.
* "Propose an end-to-end deep GP which implicitly learns networks that generate metafeatures," → What does it mean when a GP "implicitly learns networks"? Are you referring to the neural network of the deep kernel? Why does this learn this only "implicitly"?
In contrast,  Jomaa et al. (2021a) and Wistuba and Grabocka (2021), which might be the most related work to this submission, are much much more approachable.

### Negative Transfer

This paper often uses terms without introducing them. 
For example, one of the contributions listed in the introduction is that this paper is "the first paper that tackles negative transfer phenomenon in Bayesian HPO".  They do not explain anywhere what the 'negative transfer phenomenon is'. The authors do give a citation but I do think it would make sense to briefly explain this phenomenon to make the paper more accessible to readers.
(The negative transfer paper (Wang et al., 2019) itself gives a great summary in the first two sentences of the papers abstract.)
However, it is not clear to me how exactly DKLM tackles negative transfer in ways that prior transfer learning HPO works have not?
(The following sentence of the paper does *not* answer my questions: "By adding the task-specific information of the meta-features, the GP surrogate can infer a more accurate response surface on a new task based on similar source tasks that share similar meta-features. Therefore, our method is the first to tackle the negative transfer phenomenon for Bayesian HPO.")
The negative transfer problem is also not once mentioned in the experimental evaluation.

### Landmark  Features / Meta-Features

The authors repeatedly claim that DKLM (**D**eep **K**ernel Gaussian Process surrogate with **L**andmark **M**eta-features) makes use of "landmark meta-features" (Pfahringer et al., 2000; Feurer et al., 2014).

In prior work, landmark features refer to features that characterise a dataset via the predictive performance obtained by a variety of simple models on that dataset.
The idea here is that the performance of different machine learning models can be used to characterise the dataset itself: dataset A works well with tree-based models, Dataset B works well with simple parametric models, and so on.
This information can then be used as features in HPO (Feurer et al., 2014).
Note that, here, these cheap models are not the model for which we want to perform HPO, but instead are additional models that are only used to characterise datasets.

This is, to my understanding, the definition of landmark features.
In this paper, no definition is given, except that landmark features are " typically estimated by measuring the response of given datasets to machine learning algorithms".
It is entirely unclear to me, how anything that DKLM does can be called "landmark features" given that no simple auxiliary models are trained whose performance is used as a dataset-specific feature.
Again, maybe this misunderstanding could have been resolved if the authors had discussed these terms in more detail.

Additionally, the paper claims that DKLM learns to generates *meta-features* which it defines as "capturing the similarity between datasets".
However, DKLM learns embeddings that depend only on hyperparameters and loss values.
No dataset-specific information is available for constructing these embeddings.
The argument of the paper here seems to be that the embedding network learns a representation that learns to 'cluster' different dataset types.
However, given that no information of the dataset is available to the embedding, this seems unnecessarily complex/indirect.
The experiments of Figure 5 (weakly) support the idea that this clustering does occur.
(Also why not apply the approach of Jomaa et al. to actually include learned dataset-specific features? If DKLM is confronted with a novel dataset, unlike the approach of Jomaa et al., it cannot make any judgement of it before observing loss-response pairs.)
In any case, I feel that this would need to be exposed much more clearly. 

In contrast, the approach of Jomaa et al. (2021a) also claims to learn meta-features.
They construct a neural network architecture that take as input a literal dataset (with all rows/datapoitns and columns/features) and outputs a learned representation.
This per-dataset representation is then useful for HPO.
The definition of Jomaa et al. (2021a) for learning meta-features makes sense to me.

What are the definitions of landmark features/landmark meta-features/meta-features used in this paper and how are they consistent with those of earlier works?

### Missing DKLM Details

The exposition of the approach is lacking both clarity and detail.
In that regard, equations (4-7) are extremely helpful in understanding what's going on.
Without them, I'm not sure if I would have any idea of what happens in DKLM.

However, the exposition of DKLM is, in my eyes, not complete and lacking many details.
For example, this is a list of questions that I still have about how exactly DKLM works:
* Algorithm 1 gives a recipe for meta-learning with DKLM. But what does applying DKLM look like? Do you continue to train $w$ or $\theta$ when applying DKLM to a HPO task? (E.g. is anything retrained for the acquisition steps of Figure 3?) Maybe an Algorithm 2 illustrating the application of DKLM would be helpful here.
* Also, when applying DKLM to a dataset, you update the history $\mathcal{H}_t$ with observations from the current dataset, right? If so, what happens if you don't do this? You would still add new observations to the GP (just not use them in $\phi_2$). 
* However if you don't do update $\mathcal{H}_t$ as you apply DKLM to a dataset,  what does the "DKLM *without* transfer learning" setting of Figure 2 look like? This is not explained in the main body of the paper.
* What acquisition strategy do you use for HPO? Expected improvement? As far as I can tell this is not stated in the paper.
* From Algorithm 1, I can see that you are using REPTILE to train DKLM. Why are you using REPTILE? This is not discussed in the main text at all.
* The information in 6.2 is not enough to replicate the experiments. How do you set the deep kernel GP hyperparameters? What are the training schedules? Do you optimise the GP hypers? It would be great if an appendix with this information and/or code of DKLM were provided by the authors.

### Experimental Results

For the toy experiments, it seems to me that DKLM should be able to predict close to perfect after having found a single peak value of the sinusoid in the transfer learning setting.
After all, a single peak value completely defines the function. (Technically speaking, obtaining a single derivative of the function, e.g. approximated by two close observations, should also be sufficient.)
However, as is evident from Fig. 1 (top left), the predicted function is far from perfect despite observations near the peak. 
Based on my understanding, DKLM should be able to learn to perform similarly to a maximum likelihood fit of  $a * \sin(x)$ to the observations.
However, the results displayed here are looking far worse.
Weirdly, going from DKLM + 2 trials to DKLM + 3 trials, the uncertainty of the predictions *increases* despite this not making much intuitive sense.


## Smaller Comments

* "Nevertheless, extracting meta-features requires direct access to the datasets, which might be difficult in real settings where only the meta-dataset is available. " Why is direct access to datasets not available in real settings? I thought a meta-dataset directly *contained* datasets (and gives direct access to them)? Unfortunately the term meta-dataset is not further defined in the context of this paper.
* S.3.2: Why is the noise variable $\sigma_n$ indexed with $n$? 
* "We also notice how the performance improves gradually with the increasing number of trials, indicating the impact of the posterior variance modeling of our method (Section 5) as more observations are present on the black-box responses."
    * It's unclear to me why "posterior variance modeling" is needed for gradually increasing performance? Also, other HPO approaches also rely on GPs with posterior variances. What is unique about DKLM here?

## Language & Style

I believe the writing style of this paper needs improvement. Below, I present some specific examples and suggestions of where I think the writing could be improved.

* "however, it [HPO] remains an open problem" --> Why is HPO an open problem? When would it no longer be an open problem? I feel like saying that HPO is an open problem is either trivial or not true, and so best avoided. 
* "In contrast to existing approaches, we propose a novel Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM) that can be jointly meta-trained on a set of source tasks and then transferred efficiently on a new (unseen) target task." --> Why is your work *in contrast* to existing approaches? It seems that you, *like* prior work, apply transfer learning to HPO.  It seems to me that "jointly meta-training on a set of source tasks" is also not novel.
* "prior work focus on" → focuses
* "Transfer learning for HPO has been observed by modeling tasks jointly" → maybe "has been applied". "Observed" makes it sound a bit like an event in nature.
* "Among *existing* ~~the variety of~~ acquisition functions, ~~the~~ expected improvement is widely adopted (Mockus, 1974)."
* "The standard approach of fitting GPs is to optimize the weights of the kernel function, e.g. squared exponential kernel, θ."  I think it is misleading to call the optimisation of the kernel *hyperparameters*  'fitting GPs'. Gaussian Processes perform Bayesian *inference* and are not "fit" by "optimising" any parameters. However, the hyperparameters in the GP (but not the GP itself) are often inferred by maximising the evidence.
* "Consequently, the solution of the joint model resides on a local minimum so that given limited information about the new (unseen) target task [______] " This sentence is missing an ending? What happens given limited information about the task?

summary_of_the_review: The proposed method, DKLM, for transfer-learned HPO seems interesting and is, as far as I'm aware, novel.
The experimental evaluation seems to demonstrate benefits of DKLM over prior work.
Unfortunately, the paper is (sometimes severly) lacking in clarity, and the draft requires significant further work before being, in my opinion, acceptable for publication.
The exposition leaves me with too many open questions as to how DKLM actually, as well as some of the experiments, actually work.
Further, one of the main contributions of the paper – avoiding 'negative transfer' – is not discussed sufficiently and not evaluated at all, and I have trouble following the authors in why it is appropriate to say that DKLM learns "landmark meta-features".
If the authors can address these main points of my review, I am happy to reconsider my score.

Thank you for including reproducibility and ethics statement. Please make code for reproducing experiments available as soon as possible.
review_title: null

Meta-review: The submission describes a method for tuning machine learning pipeline hyperparameters using transfer learning from related tuning tasks. In particular, the method uses learned meta features to construct a covariance function for a GP.

This was an extremely difficult case and could have gone either way. It was the closest case for any paper I serve as the AC for. Two of the reviewers recommended rejecting the paper and three recommended accepting, although during discussion one of the reviewers recommending accepting the paper seemed to actually be more on the reject side.

Ultimately, I have decided to recommend rejecting this submission. However, if either the clarity (especially concerning the neural network setup) or the experiments were somewhat improved I would have recommended accepting it. I view clarity as an extremely important factor when weighing whether a submission should be accepted. I concur with the reviewers on the following weaknesses of the experiments: (1) the lack of an ablation test when considering ad-hoc meta features and (2) the experimental evaluation is based on mostly aggregated metrics.

I know this recommendation must be disappointing, but I encourage the authors to polish the work a bit more and resubmit it somewhere.
Decision: Reject