Transfer Learning for Bayesian HPO with End-to-End Meta-Features
Authors: Hadi Samer Jomaa
Sebastian Pineda Arango
Lars Schmidt-Thieme
Josif Grabocka
Venue: ICLR 2022 Submitted (2022)
Keywords: meta-learning
hyperparameter optimization
meta-features
deep kernel learning
Bayesian optimization
transfer learning

Abstract: Hyperparameter optimization (HPO) is a crucial component of deploying machine learning models, however, it remains an open problem due to the resource-constrained number of possible hyperparameter evaluations. As a result, prior work focus on exploring the direction of transfer learning for tackling the sample inefficiency of HPO. In contrast to existing approaches, we propose a novel Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM) that can be jointly meta-trained on a set of source tasks and then transferred efficiently on a new (unseen) target task. We design DKLM to capture the similarity between hyperparameter configurations with an end-to-end meta-feature network that embeds the set of evaluated configurations and their respective performance. As a result, our novel DKLM can learn contextualized dataset-specific similarity representations for hyperparameter configurations. We experimentally validate the performance of DKLM in a wide range of HPO meta-datasets from OpenML and demonstrate the empirical superiority of our method against a series of state-of-the-art baselines.

Under review as a conference paper at ICLR 2022
TRANSFER LEARNING FOR BAYESIAN HPO WITH END-TO-END META-FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Hyperparameter optimization (HPO) is a crucial component of deploying ma- chine learning models, however, remains an open problem due to the resource- constrained number of possible hyperparameter evaluations. As result, prior work focuses on exploring the direction of transfer learning for tackling the sam- ple inefficiency of HPO. In contrast to existing approaches, we propose a novel Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM) that can be jointly meta-trained ciently on new (unseen) target task. We design DKLM to capture the similarity between hyperparameter configurations with an end-to-end meta-feature network that embeds the set of evaluated configurations and their respective performance. As result, our novel DKLM learn contextualized dataset-specific similarity representations for hyperparameter configurations. We experimentally validate the performance of DKLM in a wide range of HPO meta-datasets from OpenML and demonstrate the empirical superiority of our method against series of state- of-the-art baselines.
set of source tasks and then transferred effi-
INTRODUCTION
Hyperparameter optimization (HPO) an essential open problem in machine learning (ML) due to the black-box nature of methods' empirical performances as a function of their hyperparame- ters. The major challenge lies in the computational infeasibility of training and evaluating large sample of hyperparameters in order to identify the best generalization performances. As result, transfer learning lends itself as promising direction for improving the sample efficiency of HPO methods (Wistuba et al. 2016 Perrone et al. 2018 Wistuba & Grabocka 202T)
Prior approaches for transfer learning in HPO rely on exploring existing evaluations on a pool of datasets where the model under investigation is evaluated. The similarity between datasets is of- ten captured via features describing their characteristics (a.k.a. meta-features). such as descriptive statistics of dataset features (Michie al. 1994, Wistuba et al.) 2016). or landmark measures in the form of the accuracies gathered from a set of basic classifiers (ncarest neighbors, decision trees, SVMs, etc.) the datasets (Pfahringer al. 2000 Feurer al. 2014). A recent trend high- lights the potential of learning parametric meta- feature extractors for tabular datasets (Jomaa et al.) 202 which further meta-trained (Finn al. 2017) to improve the HPO transferability to new datasets (Jomaa et al. 2021b).
Unfortunately, typical transfer learning from set of unrelated source tasks suffers from the neg- ative transfer phenomenon (Wang et al.) 2019), which implies a poor generalization performance on target tasks that are dissimilar to the source tasks, according a predefined dissimilarity mea- sure, e.g. similarity of response curves. This can happen, for example, when a model is learned jointly across tasks without task-specific attributes. To resolve the negative transfer of HPO per- formance predictors (a.k.a. surrogates) we introduce a novel direction that conditions Gaussian Process (GP) surrogates Bayesian Optimization on the meta-features of datasets. In that manner, we can transfer knowledge only from similar datasets, and hence conditioned on the similarity of meta-features. However, in contrast to ad-hoc dataset meta-features that are hand-crafted by domain experts, we propose novel architecture for deep GP kernels (Wilson et al. 2016a) that enriched with novel end-to-end neural network components that generate meta- reaturee only from the tuples of past hyperparameter configurations and their evaluated performances. Our meta-feature network

---

Under review as a conference paper at ICLR 2022
is set-based neural network that is invariant to the permutation//sequenee of past hyperparameter evaluations.
We jointly train Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM) through HPO meta-lcarning (Wistuba & Grabocka 202T). To validate the empirical performance of large-scale benchmark that involves 16 different search spaces and 101 datasets from OpenML. for a total of 3.4 million hyperparameter evaluations (Pineda- Arango al. 2021). Detailed experiments against a series of traditional HPO methods, as well as recent transfer HPO baselines, demonstrate the superiority of meta-learning the initialization of DKLM. Overall, we make the following contributions:
method we present extensive results
Introduce the first paper that tackles the negative transfer phenomenon in Bayesian HPO, by conditioning GP surrogates meta- meta-features, i.c. on dataset characteristics;
Propose end-to-end deep GP which implicitly learns networks that generate meta- features, with no ad-hoc inductive bias from experts on manually designing meta-features.
Demonstrate the empirical superiority of our method on a very-large-scale experimental protocol (3,4 million hyperparameter evaluations), against large number of baselines.
RELATED WORK
Hyperparameter optimization (HPO) has been extensively studied over the past decade for improv- ing the performance of machine learning models beyond simple search techniques (Larochelle al. 2007 Bergstra & Bengio 2012). Non-transfer learning solutions often define a probabilistic surro- gate that estimates the true hyperparameter response surface using Gaussian Processes (Rasmussen) 2003), Bayesian Neural Networks (Snoek et al. 2015) Springenberg et al. (0166), or tree-based mod-
els (Hutter et al. 2011
Bergstra et al.) 2011). Hyperparameters are then selected via an acquisition function (Wilson et al. 2017) and the process is reiterated with the new set of observations until a specified budget is exhausted, e.g. runtime or number of trials.
HPO is further expedited when defined within the context of transfer learning, i.c. by leverag- ing related tasks (or datasets) to improve the generalization over unseen tasks. Transfer learning for HPO has been observed by modeling tasks jointly (Swersky et al.) 2013) Yogatama & Mann, 2014 Perrone et al. 2018 Salinas et al.] 2020), or through some weighted-combinatioe
the sur- rogates (Schilling al. 2016 Wistuba et al. 2016 Feurer et al., 2018). Other directions include pruning the hyperparameter search space (Wistuba et al., 2015a Perrone Shen 2019), or learn- ing to initialize the surrogate by identifying good initial hyperparameters (Wistuba al., 2015b).
Apart from learning a transferable surrogate, recently, transferable acquisition functions (Wistubal al. 2018 Volpp et al. (020)) have also been proposed to replace engineered acquisition functions. The success of meta-learning for domain adaptation has also been investigated for HPO. Wistuba & Grabocka (202T) explore few-shot Bayesian Optimization by learning deep kernel Gaussian Process surrogate across set of tasks to quickly adapt to new target task. Similarly, JJoma et al. (2021 learn a shared neural network surrogate jointly coupled with a meta-feature extractor de- fined over the dataset itself.
cacilarlyyJJmaaa
Meta-features (Vanschoren 2018), or dataset characteristics. have also been widely adopted in HPO algorithms for warm-start initialization (Feurer et al., 2015 Wistuba et al. 2016) as additional attributes to better marginalize the surrogate on individual tasks (Bardenet theless, extracting meta-features requires direct access to the datasets, which might be difficult in real settings where only the meta-dataset is available. this paper, we propose to extract landmark meta-features from existing evaluations (Leite et al. 2012 Sun & Pfahringer 2013) in an end-to-end fashion using a deep Gaussian kernel approach.
2013). Never-
3 PRELIMINARIES
3.1 HYPERPARAMETER OPTIMIZATION
We denote by = ((x,, y:)) a task of interest, such that 2; E X S RR represents a hyperpa- rameter configuration in the domain of (bounded) hyperparameter search space for some model
2

---

Under review as a conference paper at ICLR 2022
under investigation. Furthermore, let y/s = fzz)) + be the response of an unknown black-box function f := X' + R* with as an additive iid Gaussian noise with some homoscedastic vari- ance a². Typically, y represents a metric of interest that should be optimized to obtain better model generalization, e.g. validation loss. The objective of hyperparameter optimization is then to find the optimal hyperparameter such that J. arg min (x) given fixed budget T of trials. HPO is commonly treated as sequential decision-making process, where a surrogate model 9 : X' - I iteratively fit the history H. == { 9:) of evaluated hyperparameters and a policy (or acquisition function) A : (A' x R)* + X' is used to select the next candidate which minimizes the expected hyperparameter response. Among the existing acquisition functions, expected improve- ment is widely adopted (Mockus 1974).
3.2 DEEP KERNEL GAUSSIAN PROCESSES
Given a training task D {(I; Pzz the response can be modeled using a Gaussian process (GP), i.e. as multivariate Gaussian distribution, such that y ~ *(( X)). GP is a non parameeaettic approach that defines prior over functions directly, and is defined by its mean function, m, and kernel function k. Given some observed data points, it possible compute the posterior over these functions to approximate unobserved data points as
y fo
N (m(x),
Kn K.
)
(1)
K
with K, x + I K, k(x 0), and K,* =(X,; X, 0). The mean and covariance of the posterior predictive distribution is then estimated as
Elf. X, y, X.) K. K cov(s, X, K., K2 K
(2)
The standard approach of fitting GPs is to optimize the weights of the kernel function, e.g. squared exponential kernel, Nevertheless, these engineered kernels are often employed under false as- sumptions (Cowen-RRin Rivers et al.) 2020), which leads to sub-optimal performances.
Recently deep kernel learning WWilson al. 2016b) has emerged as a powerful extension that lever- ages the representative capacity of non-linear function approximation, e.g. neural networks, and facilitates learning the kernel directly. Specifically, we denote by Φ : X - R.N. a mapping from the domain to a latent space which serves as an input to the kernel, such that:
Kaeep (z, x' 00, w) = K(Ó(I, w), p0x',((,, w |0)
(3)
where w represents the parameters of Φ. The weights and w are then jointly optimized for maxi- mizing the marginal likelihood (Wistuba & Grabocka 202T).
DEEP KERNEL GAUSSIAN PROCESS WITH LANDMARK META-FEATURES
Inspired by landmark meta-features (Pfahringer al. 2000), which are typically estimated by mea- suring the response of given datasets to machine learning algorithms, we propose a novel deep ker- GP that is conditioned on task-specific landmark meta-features. However, instead of computing meta-features through ad-hoc approaches, network that integrated into the kernel function of a GP and subsequently meta-learned over a set of source tasks together with the parameters of the GP kernel. In that manner, we learn meta-features that describe a set of tasks in terms of minimizing the estimation of the tasks' response functions. By adding the task-specific information of the meta-features, the GP surrogate can infer more curate response surface on a new task based on similar source tasks that share similar meta-features. Therefore, our method is the first to tackle the negative transfer phenomenon for Bayesian HPO.
introduce novel parametric meta-feature extractor
4.1 LANDMARK HEAT-RE NETWORKS
We propose simple idea to lear landmark meta-features by learning deep representation of the evaluated set of pnpepparamtee-respooe pairs as part of a deep kernel Gaussian process. With the
success of set-based algorithms (Lee et al. 2020 Zaheer et al. 2017 Lee et al. 2019) for function
3

---

Under review as a conference paper at ICLR 2022
approximation, we propose use Deepset (Zaheer et al. 2017) formulation that provides fixed- size vector representation from the dynamic set of observations. Although other methods have been developed for set-based function estimation, we focus here deepsets because they have already been shown to perform well for learning meta-feature in task-agnostic settings (Jomaa et al. 2021a) as well for hyperparameter optimization (Jomaa al., 2021b).
Suppose that we are given a collection of data points (<,,,)))) U:) where x E X' an observed hyperparameter and y E R its corresponding response. We denote by H = ((.),))) an associated set of data points that have been observed prior to 24. Furthermore, we formulate the proposed meta-feature network as:
(1x, H _w) = Φ1 (Ix, (2( ( )
(4)
t-1
s.t. ; =
(1z, Mil: wy): Wy
(5)
t
i=1
where [] symbolizes standard concatenation, Φ2 (X x X)* + R and Φ1 X' X RM are parametric neural networks with respective weights w, and where (X' x R)" represents the set of evaluated hyperparameter and their responses. With this formulation, we ensure that the rela- tionship of the covariates and the responses in H is properly encoded, and thus á is conditioned also important to note that de is permutation invariant, i.e. 22 ( ()) with π == (X' x R)" - (X' x RR) as a random permutation func- tion. This is critical, as the ordering of the data points in H--1 should not affect the landmark meta-features. Additionally, given de (H( this information about the marginal distribution of the meta-features can be encoded with the specific attribute x which in turn allows the deep kernel GP to be marginalized over individual tasks given the context, and thus transfer (joint) learning becomes easier with minimal overhead.
on these latent representations.
4.2
META-LEARNING OUR DEEP GPs
The parameters 0 and w are optimized jointly by maximizing the following log marginal likelihood:
arg max log (y w. H; 0, w)
arg max Ex.u((......)) log (ya add Ha; 0, wr)
(6)
0,00
0,0
arg min Es-U((i..))
K ya + log Ka
0,00
s.t. KaL K (o ( M7 :w) o (aa, H²-1; ww) :8)
Patacchiola et al.) 2020), we can opti- mize Equation 7 in terms of 0 via stochastic gradient descent (SGD), that is proven to maintain convergence guarantees (Chen et al. 2020). We direct the interested reader to the prior work for more details optimizing the parameters of deep GPs (Wilson et al. 2016b).
By using established practices (Wistuba Grabocka 2021
Given the diverse number of tasks, which vary in the number of available data points, we propose to jointly learn the shared surrogate via first-order meta-learning (Nichol et al. 2018). Meta-learning has found resounding success in the research community as an initialization scheme (Finn et al., 2017 Wistuba & Grabocka 2 Jomaa al., 2021b). which allows for fast adaption new domains. Consequently, the meta- trained model resides on joint minimum across all the source tasks, such that given limited information about the new (unseen) target task, it can converge faster to the new task's 's local optima. In this direction, we show the pseudocode of our meta-learning optimization in Algorithm [
MOTIVATION
Meta-features help to model the posterior uncertainty. To motivate our approach, we present an ablation of the effect of our deep GP kernel with meta-features, compared to the same deep GP kernel without meta-features (i.e. Ours vs. FSBO (Wistuba & Grabocka 202T)). We created synthetic meta-dataset of K 50 tasks in the form of randomly sampled sinusoidal functions
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Meta-learning DKLM via REPTILE (Nichol et al. 2018
Require:
training dataset &; kernel parameters 0, network parameters wy learning rate yy; inner
update steps v; meta-batch size n, batch size b.
2: while not converged do
3: t~ U mii mmxx]
4: D1. Di...., Dn do Sample t
5:
for i
6:
data points to form H!-11 Dj
7:
Sample batch B = ((1;, D; 0, k-60 0 ;0;; 4- w for j =
8:
9:
do
10:
Define as L the objective function of Equation 7 0, f- 0, + VoC
12:
t- wi V W.
Update 8 + 11 Σn (8;
14:
Update w +- w + 17 == (w)s - w)
DKLM Initial
DKLM Trials
DKLM + Trials
DKLM Trials
Trials
Meta-features Amplitude
-1
-2
2
0
4
-3
22
4
-4
-2
2
Amplitude
t(X)
Seeds
iix)
EI(x
Figure 1: (top) Sequential model-based optimization of an unseen sine wave using our approach. (bottom left and middle) Our fitted surrogate after trials, compared to FSBO after 3 trials given the same initial seeds. (bottom right) Correlation between amplitude and landmark meta-features.
f²(x) (8) sin(x b(K)), k {I....,K) by drawing each (k) 2(001,55 and ((k) 2T). Furthermore, we meta-learn our deep GP on these source functions and then transfer the surrogate as an initialization for a new sinusoidal curve (with new parameters a, b) as shown in Figure [ (top). We show the comparison of our surrogate with meta-features after 3 trials (for total of data points, including 5 initial configurations) to an FSBO deep GP that has been meta-trained identically. We notice that our surrogate (bottom row, leftmost plot) computes better posterior variance compared to FSBO (bottom row, middle plot). The effect of the superior modeling of the uncertainty leads to better exploration in Bayesian Optimization setup, and consequently to better empirical accuracies of the discovered hyperparameters (as will be shown in Section
We postulated that proposed meta-features capture task characteristics. To illustrate the argument, we create another simpler collection of K = 50 sinusoidal functions (x)x = (*) sin(x), k E (1...., K} by drawing each (k) ~ 4(0.1,5) 5). As these sine waves change only in terms of the amplitude parameter then if meta- train our meta-feature network with a 1-dimensional (1D) output layer from these source functions, must strongly learn correlate the 1D meta-feature with the sinusoidal amplitude As can
Meta-features capture task characteristics.

---

Under review as a conference paper at ICLR 2022
be seen in the rightmost plot of the bottom in Figure 1 this is exactly the case. In this plot, the y-axis shows ID meta-feature values computed from only random pairs of configurations and responses (z, f())) from one random task, and the x-axis shows the amplitude of that respective task. Although our meta-feature networks have no design bias in terms of modeling sinusoidal functions, they are perfectly able to extract latent representation of the amplitude, based the end-to-end meta-learning of the deep GP for approximating random observations on the sine waves.
EXPERIMENTS
Our experimental protocol is designed to primarily address one simple research question: Do deep GPs with our meta-feature networks outperform state-of-the-art HPO algorithms in the trans- fer and non-transfer learning settings?
6.1 META-DATASET AND BASELINES
We evaluate approach HPO-B-v3, for comparing black-box HPO methods (Pineda-Arango et al. 2021). The benchmark contains a collection of 935 black-box tasks for 16 hyperparameter search spaces (algorithms) evaluated on 101 datasets and divided into predefined training, validation, test splits. Following the same experimental protocol specified the HPO-B benchmark, compare our approach to the following large set of 10 HPO baselines:
hyperparameter optimization benchmark designed
Random Search (Bergstra & Bengio 2012);
2. GP (Rasmussen) 2003) is a hyperparameter tuning strategy that relies on Gaussian Pro- cess as surrogate model with squared exponential kernels (Matern 5/2 kernel) with auto- matic relevance determination;
3. DNGO (Snoek et al. (015)) utilizes a neural network to extract adaptive basis function of hyperparameters, which in turn are fed to Bayesian linear regression model to generate posterior distribution;
4. BOHAMIANN (Springenberg et al. 2016) is based on Bayesian neural networks that are trained via stochastic gradient Hamiltonian Monte Carlo;
5. DGP PPatacchiola et al.
2020) fits a deep kernel Gaussian process as a surrogate;
6. TST-R (Wistuba al. 2016 is an ensemble approach where the Gaussian process sur- rogate of the target task is weighted with surrogates the training datasets based on the ranking similarity of the evaluated hyperparameters:
7. RGPE (Feurer et al. 2018) is another ensemble approach, similar to TST-R, which es- timates the weights by optimizing a ranking loss between the surrogates of the training datasets and that of the target task;
8. ABLR (Perrone et al., 2018) is a multi-task Bayesian linear regression approach that opti- mizes a shared feature extractor across the training datasets an initialization strategy for the target task;
9. GCP+Prior (Salinas al. 2020) utilizes Gaussian Copula process (iilson & Ghahra- mani 2010) trained jointly on the training tasks, where quantile-transformatinn is applied their respective responses. The pretrained process used as parametric prior for the target dataset;
10. FSBO (Wistuba & Grabockal 2021) uses deep Kernel Gaussian processes (Patacchiola et al. 2020) to estimate the response of the target dataset. The parameters are initialized via meta-lea the joint response surface over the training datasets.
In nutshell, experimental protocol based HPO-B is large scale one by the standards of the prior papers, as involves 10 baselines, 16 search spaces (algorithms whose hyperparameters we tune). 101 datasets, and totally 935 box tasks containing 6.3 million evaluations.

---

Under review as a conference paper at ICLR 2022
6.2 IMPLEMENTATION DETAILS
We implement the Deep Kernel Gaussian Process using GPyTorch 1.5 (Gardner et al 2018) with Matern 5/2 kernel. As described in Equation DKLM is composed of two modules, = @1 d2-- The parameters of the network have been selected based on the performance on a held-out validation Function f is 4 dense layers with 32 hidden units and ReLU activation functions, whereas is 1 dense layer with 32 hidden units. Finally, &1 is 4 dense layers with 32 hidden units and ReLU activation. All parameters of the Deep Kernel are estimated by maximizing the marginal likelihood. We achieve this by using gradient ascent and Adam (Kingma & Ba 2015) with learning rate of 0.001 and batch size of 64 with E [2, 100].
6.3 HPO RESULTS AND DISCUSSION
We start by comparing non-transfer HPO methods our deep GPs with meta-features DKLM (RI) that are randomly initialized (no meta-learning). As depicted in Figure 2 DKLM (RI) outperforms the baselines after 100 hyperparameter trials in terms of both the mean normalized regret and mean rank metrics. We also notice how the performance improves gradually with the increasing number trials, indicating the impact of the posterior variance modeling of our method (Section more observations present the black-box responses.
Mean Normalized Regret
Mean Average Rank
4.5
Random
BOHAMIANN
10-1
4.0
Deep Kemel GP
3.5
DKLM (RI)
3.0
25
50
75
100
25
50
75
100
Number
Number Trials
Figure 2: Aggregated comparisons of normalized regret and mean ranks across all search spaces for the non-transfer learning HPO methods HPO-B-v3
Afterwards, demonstrate the comparison of state-of-the-art transfer against our method in Fig- ure 3 DKLM outperforms the rest of the baselines with lower mean normalized regret and lower mean rank. The superiority of landmark meta-features also becomes evident after larger number of trials (more than 50).
Normalized Regret
Average
Random
Deep Kemel GP
10-1
DKLM (RI)
6
TST-R
ABLR
5
RGPE
GCP+ Prior
10-2
FSBQ
0
50 Nurbee of Trials
100
0
50
100
DKLM
Number of Trials
Figure 3: Aggregated comparisons of normalized regret and mean ranks across all search spaces for the transfer learning HPO methods on HPO-B-v3
To further inspect the results we show the performance of DKML and all other baselines in the se- lected individual search spaces of Figure 4 We notice primarily that meta-learning the initialization in DKLM improves the general performance most cases. Nevertheless, we notice in 4796 that effect of transfer learning is not evident, DKLM (RT) and Deep Kernel GP are better than the

---

Under review as a conference paper at ICLR 2022
4796
$527
5633
5891
Normalized Regret
10-1
10-2
10
10
10
100
50 Number Trials
Number of Trials
Number Trials
Number of Trials
5906
6767
7609
5889
Normalized Regret
10-2
10-1
10-2
10-1
10-2
10
10-1
10
10-3
10-4
50
100
10"
50
100
50
100
10-4
50
100
Number of Trials
Number Trials
Number Trials
Number Trials
Random
DKLM (RI)
TST-R
RGPE
FS8O
Deep Kernel
TAF-R
GCP+
Figure 4: Normalized regret comparison of transfer learning HPO methods selected benchmarks
from HPO-B-v3
4796
5527
5636
5891
Latent Dimension
5906
6767
7609
5889
Latent Dimension
Latent Dimension 1
Latent Dimension
Latent Dimension 1
Latent Dimension
Figure 5: 2D illustration of meta-features extracted from each task selected search spaces. For each task, we sample 100 sets of 5 data points to extract landmark meta-features. We reduce the dimensionality of the meta-features into 2D representation via TSNE (Liu et al. 2017).
meta-initialized variants, DKLM, and FSBO respectively. Still, landmark meta-features prove yet again that they are effective in single task setting.
6.4 END-TO-ENT -EN LANDMARK HETA-FEATURES
To motivate the importance landmark meta-features, we illustrate in Figure 5 2D latent dimen- sions of the landmark meta-featur teta-features for every test task in the 16 spaces of HPO-B-v3. Each point on the graph represents a set of meta-features extracted from 100 randomly sampled data points, i.e. t = 100, from the individual tasks after meta-initializatoon of the weights of DKLM. We observe that the same color-coded meta-features. i.e. belonging to the same task, lie generally within the vicinity of each other, and distant from other tasks. As pointed out by Jomaa al. (2021a), any meta-feature extractor should able to preserve inter-and intra-dataset similarity, a property that is evident here.
8

---

Under review as a conference paper at ICLR 2022
6.5 RESILIENCE TO NEGATIVE TRANSFER
To test the phenomenon of negative transfer we design a specific ablation experiment, where the datasets in the meta-test split are uncorrelated the datasets in the meta-train split. First of all, we define the correlation of two datasets using Kendall's т correlation coefficient [Kendall 1945). We sample 100 random nypee-parameter configurations and query the validation accuracies of these configurations on both datasets from an HPO-B search space, to compute the Kendall's т coefficient between the accuracies' vectors. Afterward, for every HPO-B search space, we i) compute the correlation between each meta-test dataset and all the meta- metattraiiing datasets, ii) keep the 10% of the meta-training datasets that have the smallest mean Kendall's T coefficient to the meta-test datasets.
We report the ablation results in Figure 6 where DKLM (NT) and FSBO (NT) denote methods trained using only the 10% most uncorrelated meta-training datasets, In contrast, DKML and FSBO trained using all the 100% meta-trast tratt-t dataset. We notice that performance gap of the ranks between DKLM (NT) and FSBO (NT) is larger than that between DKLM and FSBO, validating DKML's resilience to the negative transfer phenomenon.
Additionally, we report the the critical difference diagrams @ 100 trials. We notice that the perfor- mance of DKLM (NT) is statistically significant compared to FSBO (NT), further illustrating the resilience of DKLM to negative transfer.
Rank@100
Meen Norallzzdd Regret
Mean Average Rank
FSB0 7917
1.7081 OKLM INT)
10-1
3.5
Rankgy 100
Random 2 20103
1,000 DKLM
3.0
Rankg100
10
50 Number Trials
50
100
Ror dom DKLM
25100 DKUM
Number of Trials
Random
FSBO (NT)
DKLM (NT)
Figure 6: Aggregated comparisons of normalized regret and mean ranks across all HPO-B search spaces, where the gap between DKLM (NT) and FSBO (NT) demonstrates the resilience to negative transfer.
7 LIMITATIONS
Despite the fact that our method significantly reduces the time for fitting Machine Learning, we caution practitioners against overtuning their model for large number of configuration trials, only to get a very small improvement in accuracy, unless absolutely necessary from business need.
CONCLUSION
In this paper, however, we propose DKLM as simple yet effective method to better condition deep kernel Gaussian Processes on tasks. Inspired by landmark meta-features, we design set-based meta-feature extractor that captures the interaction between the available hyperparameters and their respective responses, and consequently generates distinct task-specific attributes. DKLM is meta- learned on set of source tasks in an end-to-end fashion to jointly approximate the response surface over the shared hyperparameter and landmark meta-feature space. We show in a battery of experi- ments the significance of landmark feta-fcatures, outperforming state-of -fate-of-the-art afrtthe-rt HPO baselines in non-transfer and transfer learning settings.
ETHICS STATEMENT
In our work, only publicly available data without privacy concern.. Furthermore, algo- rithm reduces the overall time for fitting machine learning algorithms, therefore, saving computa- tional resources and yielding positive impact energy consumption.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We promote reproducibility as detailed below:
We use only publicly available datasets.
All our baselines are publicly available and provided by the HPQ-B benchmark (Pineda- Arango et al. 202I).
We clearly describe our method in Section 4 and provide implementation details in Sec- 6.2
Finally, plan make the source code of our method publicly available.
REFERENCES
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sebag. Collaborative hyperparameter tuning. In Proceedings of the 30th International Conference Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 199-207, 2013.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parametts optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December
2011, Granada, Spain, pp. 2546-2554, 2011.
Hao Chen, Lili Zheng, Raed Al Kontar, and Garvesh Raskutti. Stochastic gradient descent in cor- related settings: A study gaussian processes. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Alexander Cowen-Rivers, Wenlong Lyu, Zhi Wang, Rasul Tutunov, Hao Jianye, Jun Wang, and Haitham Bou Ammar. Hebo: Heteroscedastic evolutionary bayesian optimisation. arXiv preprint arXiv:2012.008826, 2020. winning submission to the NeurlPS 2020 Black Box Optimisation Chal- lenge.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta-learning initialize bayesian optimization of hyperparameters. In Proceedings of the International Workshop on Meta-learning and Algoritimm Selection co-located with 21st European Conference on Artificial Intelligence, MetaSel@ ECAI 2014, Prugue, Czech Republic, August 19, 2014, pp. 3-10, 2014.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pp. 1128-1135. 2015.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian opti- 2018.
mization. CoRR,
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learnigg for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1126-1133 135, 2017.
Jacob R. Gardner, Geoff Pleiss, Kilian Q. Weinberger, David Bindel, and Andrew Gordon Wil- son. Gpytorch: Blackbox matrix-matrix gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurIPS 2018. December 3-8, 2018, Montréal, Canada, pp- 7587-7597, 2018.
Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization 5tth International Conference, LION 5, Rome, Italy, January 17-21, 201 I. Selected Papers, pp. 507-523, 2011.
10

---

Under review as a conference paper at ICLR 2022
Hadi S. Jomaa, Lars Schmidt-Thient and Josif Grabocka. Dataset2vec: learning dataset meta- features. Data Min. Knowl. Discov, 35(3):964 2021a.
Hadi S. Jomaa, Lars Schmidt-Thieme, and Josif Grabocka. Hyperparameter optimization with dif- ferentiable metafeatures. CoRR, abs/2102.00777. 2021b.
Maurice G Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239-251, 1945.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Hugo Larochelle, Dumitru Erhan, Aaron C. Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Machine Learning, Proceedings of the Twenty-Fone Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24, 2007, pp. 473 480, 2007.
Hacbeom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning balance: Bayesian meta-learning for imbalanced and -uitt--distriuution tasks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Scungjin Choi, and Yee Whye Tch. Set transformer: framework for attention-based permutation-invarian neural networks. In Pro- ceedings the 36th International Conference on Machine Learning, ICML.2019, 9-15 June 2019, Long Beach, Califormia, USA, pp. 3744-3753, 2019.
Rui Leite, Pavel Brazdil, and Joaquin Vanschoren. Selecting classification algorithms with active testing. In Machine Learning and Data Mining in Pattern Recognition 8th International Con- ference, MLDM 2012, Berlin, Germany, July 13-20, 2012. Proceedings, pp. 117-131, 2012.
Shusen Liu, Dan Maljovec, Bei Wang, Peer-Timo Bremer, and Valerio Pascucci. Visualizing high- dimensional data: Advances in the past decade. IEEE Trans. Vis. Comput. Graph., 23(3):1249 1268, 2017.
Donald Michie, David J. Spiegelhalter, and C. C. Taylor. Machine Learning, Neural and Statistical Classification. Ellis Horwood, 1994.
Jonas Mockus. On bayesian methods for seeking the extremum. In Optimization Techniques, IFIP Technical Conference, Novosibirsk, USSR, July 1-7, 1974, pp. 400-404, 1974.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta leaaning algorithms. CoRR, abs/1803.00 808802999, 2018.
Massimiliano Patacchiola, Jack Turner, Elliot Crowley, Michael F. P. O'Boyle, and Amos J. Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.
Valerio Perrone and Huibin Shen. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Prucessing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12751 12751-12761 2019.
Valerio Perrone, Rodolphe Jenatton, Mathias W. Seeger, and Cédric Archambeau. Scalable hyper- parameter transfer learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6846 6856, 2018.
Bernhard Pfahringer, Hilan Bensusan, and Christophe G. Siraud-Carries Meta-learning by land- marking various learning algorithms. In Proceedings the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29. July 2, 2000, pp. 743-750, 2000.
11

---

Under review as a conference paper at ICLR 2022
Sebastian Pineda-Arango Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A large- scale reproducible benchmark for black-box HPO based on openml. CoRR, abs/2106.06257, 2021.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Advanced Lectures on Ma- chine Learning, ML Summer Schools 2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germany, August 4-16, 2003, Revised Lectures, pp. 63-71,200 2003.
David Salinas, Huibin Shen, and Valerio Perrone. A quantile-based approach for hyperparameter transfer learning. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pp. 8438 8448, 2020.
Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thiene Scalable hyperparameter optimiza- tion with products of gaussian process experts. In Machine Learning and Knowledge Discovery in Databases- European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part pp. 48, 2016
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan P. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 2171-21808 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4134 4142, 2016.
Quan Sun and Bernhard Pfahringer. Pairwise meta-rules for better meta-learning-bart based algorithm ranking. Mach. Learn., 93(11;141-16,, 2013.
Kevin Swersky, Jasper Snock, and Ryan Prescott Adams. Multi-task bayesian optimization. In Advances in Neural Information Processing Systems 26: 27th Annual Conference Neural In- formation Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2004-2012, 2013.
Joaquin Vanschoren. Meta-learning: survey. CoRR, abs/ 1810.03548 2018.
Michael Volpp, Lukas Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in bayesian optimiza- tion. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Zirui Wang. Zihang Dai, Barnabás Póezos, and Jaime G. Carbonell. Characterizing and avoiding negative transfer. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11293-11302, 2019.
Andrew Gordon Wilson and Zoubin Ghahramani. Copula processes. In Advances in Neural In- formation Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pp. 2460, 2010.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learn ing. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-1 1, 2016, pp. 370-378, 201
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learn ing. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-1 2016, pp. 370-378, 2016b.
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4148 4158, 2017.
12

---

Under review as a conference paper at ICLR 2022
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kemell surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Hyperparameter search space prun- ing A new component for sequential model-based hyperparameter optimization. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part II, pp. 104-119, 2015a.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimiza- tion initializations. 2015 IEEE International Conference on Data Science and Advanced Ana- lytics. DSAA 2015, Campus des Cordeliers, Paris, France, October 19-21, 2015, pp. 1-10, 2015b.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiene Two-stage transfer surrogate model for automatic hyperparameter optimization. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I, pp. 199-214, 2016.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiemee Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Mach. Learn., 107(1):43778, 2018
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, Reykjavik, Iceland, April 22-25, 2014, pp- 1077-1085, 2014.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov. and Alexander J. Smola. Deep sets. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA pp. 3391-3401, 2017.
13

---

Under review as a conference paper at ICLR 2022
A ALGORITHMS
We describe in Algorithm 2 the general procedure for BO optimization. Our surrogate model, DKLM, is fit to the history of observations H at each trial via Algorithm
Algorithm 2: Sequential Model-Based Optimization with DKLM
Input: kernel parameters 0, network parameters w. black-box function J N initial evaluations 7 {1 yt } on target task, acquisition function A(:), number of trials T,
Output: Best configuration and response (I+, V+)
for f 1 to do
Fit c and kernel using H in Algorithm 3
Obtain next configure configuration to evaluation arg max A(x)
Observe response y fz));
Update History H. 7t U[x,,))]:
end
Return best configuration (Ir., y/o) arg (ax(x.y)>**
Algorithm 3: Fitting DKLM to a target task
Input: Learning rates ?i and T72, N evaluations H/ {x,,, y1) y:) parameters 0, network parameters w
on target task, kernel
Output: Gaussian Process posterior p (S. lar,; H)
while not converged do
Compute marginal likelihood C on (Equation 6
0
0
in VoL:
w
t w T2V
end
Compute Gaussian Process posterior (f+(z,, p., H) = NN,, x2) (Equation
B EXPERIMENTAL PROTOCOL
We followed the protocol presented in HPO-B, (Pineda-Arango et al. 2021). benchmark collection of tasks collected from OpenML. The tasks are grouped in 16 search spaces, and every search space contains three splits: meta-training, meta-validation and meta-testing. According to the authors, the splitting for the percentage of tasks 80%, 10% and 10%, respectively. We refer the repository to know the exact task IDs for every split.
In our experiments, we meta-train model per search space, whereas we used the meta-validation data for early-stopping. We use 1 0.001, v 1, = 1000, Timin = 2, Tmax 100, b = 64 for meta-training (Algorithm[]] We optimize the black-box function following Sequential Model Based Optimization (SMBO, Algorithm 2) use Expected Improvement the acquisition function:
A(x) E |max (f) Ymax, 0}]
which uses the mean and variance computations presented in section 3.2 for estimating the expec- tation. We fit DKLM using ADAM (Kingma & Ba 2015) with learning rates 7)1 7/2 0.001, (see Algorithm B). We use the meta-validation dataset for choosing the number of hidden layers in Equation and the learning rates 1).
C IMPACT OF SAMPLE SIZE
We mentioned in Section 6.5 that the performance of DKLM improves with the increasing number of observed data points. To further examine this observation, we present Table T the negative log
14

---

Under review as a conference paper at ICLR 2022
likelihood given history of size t. To get these results, sample points for the history Ht and compute the negative log- siglliielihodd on 100 separate data points upon meta-initialization of DKLM. The process is repeated 100 times for each task, for total of 10000 data points. We highlight the trend of the negative log-likelihood by fitting a linear model to the 10000 points and reporting the slope. In most search spaces, the performance improves with the increasing number of observed data points, where we notice some over-fitting in others, c.g. space 5860 that can be alleviated by further regularization.
Table 1: Mean Negative Log- Likelihood upon metaiinitialization of DKLM with unit of standard deviation, lower is better.
Space
t =
{ = 10
f = 20
slope
4796
0.38 + 2.64
-0.19 + 1.80
-0.78 +: 0.99
-0.075
5527
2.92 6.12
1.22
0.9 + 2.44
-0.12
5636
-1.8 3.96
-.70
-3.11 0.55
-0.077
5859
28.51 40.70
+0.37 10.09
-1.473
5860
21.8
1
16.13 18.1
0.215
5889
0
1.84
3.75 2.97
-0.039
5891
0.33 0.90
0.02
-0.28 0.69
0.414
5906
4.004
5.52
9.22 5.19
-0.003
5965
1.12
0.75
0.77
-0.015
5970
0.12
+.11
-1.22
0.021
5971
1.08
0.77
-0.81 1.10
0.14
6766
0.39
0.44
2.38 0.59
-0,118
6767
+.59
-0.05 +
-0.39 + 0.92
-0.032
6794
+
+
-1.12 + 0.67
0.019
7609
-0.59 2.42
1.749
-1.89 - 0.61
-0.08
7607
-0.05
0.77
-0,06 0.87
0.21 + 0.76
0.229
Average
2.41 + 6.40
1.57 + 4.69
1.37 + 4.86
-0.062
D DETAILED EXPERIMENTAL RESULTS
We present the detailed results for the 16 search spaces in Figure landmark meta-feature in Figure g
Additionally, we plot the 2D
15

---

Under review as a conference paper at ICLR 2022
4796
5527
5636
5859
10-1
10"
Normalized
20-1
10-1
10-2
10-2
10"2
5860
5891
5906
5965
Normalized Regret
10-1
10-1
10-2
10
10-2
10
10-1
10-3
10-4
10
5970
5971
6766
6767
10-1
Normalized Regret
10-2
10-2
10
10
10-3
10
6794
7607
7609
5889
Normalized Regret
10
10-1
10-1
10-1
2
10-2
10
10-
10-1
10
10
Random
Deep
DKLM-Att (RI)
BOHAMIANN
DNGO
DKLM (RI)
TAH-R
ABLR
GCP+Prior
DKLM
Figure 7: Normalized regret comparison of transfer learning HPO methods on all benchmarks from HPO-B-v3
16

---

Under review as a conference paper at ICLR 2022
4796
5527
5636
5859
Latent Dimension 2
5860
5891
5906
5965
Latent Dimension
5970
5971
6766
6767
Latent Dimension 2
Boo
6794
7607
7609
5889
Latent Dimension
Latent Dimension 1
Latent Dimension
Latent Dimension 1
Latent Dimension
Figure 8: 2D illustration of meta-feature extracted from each task in the 16 selected search spaces. For each task, we sample 100 sets of 5 data points extract landmark meta-features We reduce the dimensionality of the meta-features into 2D representation via TSNE (Liu et al. 2017
17


Reviews:
[{"review_id": "jbKkZNd12bJ", "reviewer": "Reviewer_bzFz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Adequately builds upon previous work such that the proposed change (the meta-extractor within the kernel of a deep GP) is easy to assess and compare (as opposed to larger set of changes that are not adequate ablated).\n- Large-scale benchmark (935 tasks, 16 search spaces) compared to 10 baselines.\n- Ablations with and within a randomly-initialized meta-extractor.\n- Results included error bars (Table 1).\n\nWeaknesses:\n- Experimental setup could be made clearer as to which tasks are used for training and eval in the transfer setting (i.e., which tasks are used to originally train the meta-extractor, and on which tasks is it transferred).\n\nMinor:\n- p. 3: In the fourth line from the bottom, the reference \"2020\" needs to be fixed.", "summary_of_the_paper": "The authors focus on the important problem of more efficient hyperparamter optimization through (meta) transfer learning. More specifically, the author propose a new method (Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM)) that embeds a meta-feature extractor model within the kernel function of a deep GP. This meta-feature extractor is able to learn to extract characteristics of a dataset such that the similarity of a dataset to other ones can be (implicitly) captured and used by the rest of the model. To demonstrate the effectiveness of their proposed model, they evaluate their model on the large-scale HPO-B-v3 benchmark against 10 other baselines plus a variant of their method with a randomly-initialized meta extractor.", "summary_of_the_review": "Overall, the authors propose a new method that is well-scoped within existing literature, and provide positive empirical results on a large-scale benchmark against an adequate set of baseline tasks. It is a reasonable paper for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eHYFVqRznP_", "reviewer": "Reviewer_1rkp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nI believe that there are merits to the proposed DKLM approach.\nIt makes intuitive sense to me that the Deepset style inclusion of prior observations is preferable to using prior observations as initialisation for the GP as done by prior work.\n\nThe hpo-b-v3 experiments look somewhat convincing.\nDKLM performs decently in the \"non-transfer setting\" and impressively well in the transfer setting.\nI'm inclined to believe the validity of these results, although I wish the authors would have included code / more details of the setup.\n\n## Weaknesses\n\nUnfortunately, I feel that many parts of this paper are severely lacking in clarity.\nThe writing often feels unnecessarily convoluted and I am sometimes unsure of what exactly the authors are trying to tell me with a particular paragraph.\nFurther, the lack of clarity unfortunately leads to open questions about the proposed approach itself.\nBelow I give specific details of this.\nI hope the authors understand my criticisms as constructive and take them as an opportunity to improve the current draft of the paper.\n\n### Introduction\n\nBy the end of the introduction, I neither understand what open challenges in HPO are, nor what DKLM does differently to established works.\nConcretely, I find the following sentences in the introduction more confusing than helpful:\n* \"we propose a novel architecture for deep GP kernels (Wilson et al., 2016a) that are enriched with novel end-to-end neural network components that generate meta-features only from the tuples of past hyperparameter configurations and their evaluated performances\" --> How do I enrich a deep GP kernel with neural network components? Don't Deep Kernel GPs already have neural networks do produce the encoding?\n* \"Introduce the first paper that tackles the negative transfer phenomenon in Bayesian HPO\" → What is the negative transfer phenomenon? This is not explained here.\n* \"Propose an end-to-end deep GP which implicitly learns networks that generate metafeatures,\" → What does it mean when a GP \"implicitly learns networks\"? Are you referring to the neural network of the deep kernel? Why does this learn this only \"implicitly\"?\nIn contrast,  Jomaa et al. (2021a) and Wistuba and Grabocka (2021), which might be the most related work to this submission, are much much more approachable.\n\n### Negative Transfer\n\nThis paper often uses terms without introducing them. \nFor example, one of the contributions listed in the introduction is that this paper is \"the first paper that tackles negative transfer phenomenon in Bayesian HPO\".  They do not explain anywhere what the 'negative transfer phenomenon is'. The authors do give a citation but I do think it would make sense to briefly explain this phenomenon to make the paper more accessible to readers.\n(The negative transfer paper (Wang et al., 2019) itself gives a great summary in the first two sentences of the papers abstract.)\nHowever, it is not clear to me how exactly DKLM tackles negative transfer in ways that prior transfer learning HPO works have not?\n(The following sentence of the paper does *not* answer my questions: \"By adding the task-specific information of the meta-features, the GP surrogate can infer a more accurate response surface on a new task based on similar source tasks that share similar meta-features. Therefore, our method is the first to tackle the negative transfer phenomenon for Bayesian HPO.\")\nThe negative transfer problem is also not once mentioned in the experimental evaluation.\n\n### Landmark  Features / Meta-Features\n\nThe authors repeatedly claim that DKLM (**D**eep **K**ernel Gaussian Process surrogate with **L**andmark **M**eta-features) makes use of \"landmark meta-features\" (Pfahringer et al., 2000; Feurer et al., 2014).\n\nIn prior work, landmark features refer to features that characterise a dataset via the predictive performance obtained by a variety of simple models on that dataset.\nThe idea here is that the performance of different machine learning models can be used to characterise the dataset itself: dataset A works well with tree-based models, Dataset B works well with simple parametric models, and so on.\nThis information can then be used as features in HPO (Feurer et al., 2014).\nNote that, here, these cheap models are not the model for which we want to perform HPO, but instead are additional models that are only used to characterise datasets.\n\nThis is, to my understanding, the definition of landmark features.\nIn this paper, no definition is given, except that landmark features are \" typically estimated by measuring the response of given datasets to machine learning algorithms\".\nIt is entirely unclear to me, how anything that DKLM does can be called \"landmark features\" given that no simple auxiliary models are trained whose performance is used as a dataset-specific feature.\nAgain, maybe this misunderstanding could have been resolved if the authors had discussed these terms in more detail.\n\nAdditionally, the paper claims that DKLM learns to generates *meta-features* which it defines as \"capturing the similarity between datasets\".\nHowever, DKLM learns embeddings that depend only on hyperparameters and loss values.\nNo dataset-specific information is available for constructing these embeddings.\nThe argument of the paper here seems to be that the embedding network learns a representation that learns to 'cluster' different dataset types.\nHowever, given that no information of the dataset is available to the embedding, this seems unnecessarily complex/indirect.\nThe experiments of Figure 5 (weakly) support the idea that this clustering does occur.\n(Also why not apply the approach of Jomaa et al. to actually include learned dataset-specific features? If DKLM is confronted with a novel dataset, unlike the approach of Jomaa et al., it cannot make any judgement of it before observing loss-response pairs.)\nIn any case, I feel that this would need to be exposed much more clearly. \n\nIn contrast, the approach of Jomaa et al. (2021a) also claims to learn meta-features.\nThey construct a neural network architecture that take as input a literal dataset (with all rows/datapoitns and columns/features) and outputs a learned representation.\nThis per-dataset representation is then useful for HPO.\nThe definition of Jomaa et al. (2021a) for learning meta-features makes sense to me.\n\nWhat are the definitions of landmark features/landmark meta-features/meta-features used in this paper and how are they consistent with those of earlier works?\n\n### Missing DKLM Details\n\nThe exposition of the approach is lacking both clarity and detail.\nIn that regard, equations (4-7) are extremely helpful in understanding what's going on.\nWithout them, I'm not sure if I would have any idea of what happens in DKLM.\n\nHowever, the exposition of DKLM is, in my eyes, not complete and lacking many details.\nFor example, this is a list of questions that I still have about how exactly DKLM works:\n* Algorithm 1 gives a recipe for meta-learning with DKLM. But what does applying DKLM look like? Do you continue to train $w$ or $\\theta$ when applying DKLM to a HPO task? (E.g. is anything retrained for the acquisition steps of Figure 3?) Maybe an Algorithm 2 illustrating the application of DKLM would be helpful here.\n* Also, when applying DKLM to a dataset, you update the history $\\mathcal{H}_t$ with observations from the current dataset, right? If so, what happens if you don't do this? You would still add new observations to the GP (just not use them in $\\phi_2$). \n* However if you don't do update $\\mathcal{H}_t$ as you apply DKLM to a dataset,  what does the \"DKLM *without* transfer learning\" setting of Figure 2 look like? This is not explained in the main body of the paper.\n* What acquisition strategy do you use for HPO? Expected improvement? As far as I can tell this is not stated in the paper.\n* From Algorithm 1, I can see that you are using REPTILE to train DKLM. Why are you using REPTILE? This is not discussed in the main text at all.\n* The information in 6.2 is not enough to replicate the experiments. How do you set the deep kernel GP hyperparameters? What are the training schedules? Do you optimise the GP hypers? It would be great if an appendix with this information and/or code of DKLM were provided by the authors.\n\n### Experimental Results\n\nFor the toy experiments, it seems to me that DKLM should be able to predict close to perfect after having found a single peak value of the sinusoid in the transfer learning setting.\nAfter all, a single peak value completely defines the function. (Technically speaking, obtaining a single derivative of the function, e.g. approximated by two close observations, should also be sufficient.)\nHowever, as is evident from Fig. 1 (top left), the predicted function is far from perfect despite observations near the peak. \nBased on my understanding, DKLM should be able to learn to perform similarly to a maximum likelihood fit of  $a * \\sin(x)$ to the observations.\nHowever, the results displayed here are looking far worse.\nWeirdly, going from DKLM + 2 trials to DKLM + 3 trials, the uncertainty of the predictions *increases* despite this not making much intuitive sense.\n\n\n## Smaller Comments\n\n* \"Nevertheless, extracting meta-features requires direct access to the datasets, which might be difficult in real settings where only the meta-dataset is available. \" Why is direct access to datasets not available in real settings? I thought a meta-dataset directly *contained* datasets (and gives direct access to them)? Unfortunately the term meta-dataset is not further defined in the context of this paper.\n* S.3.2: Why is the noise variable $\\sigma_n$ indexed with $n$? \n* \"We also notice how the performance improves gradually with the increasing number of trials, indicating the impact of the posterior variance modeling of our method (Section 5) as more observations are present on the black-box responses.\"\n    * It's unclear to me why \"posterior variance modeling\" is needed for gradually increasing performance? Also, other HPO approaches also rely on GPs with posterior variances. What is unique about DKLM here?\n\n## Language & Style\n\nI believe the writing style of this paper needs improvement. Below, I present some specific examples and suggestions of where I think the writing could be improved.\n\n* \"however, it [HPO] remains an open problem\" --> Why is HPO an open problem? When would it no longer be an open problem? I feel like saying that HPO is an open problem is either trivial or not true, and so best avoided. \n* \"In contrast to existing approaches, we propose a novel Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM) that can be jointly meta-trained on a set of source tasks and then transferred efficiently on a new (unseen) target task.\" --> Why is your work *in contrast* to existing approaches? It seems that you, *like* prior work, apply transfer learning to HPO.  It seems to me that \"jointly meta-training on a set of source tasks\" is also not novel.\n* \"prior work focus on\" → focuses\n* \"Transfer learning for HPO has been observed by modeling tasks jointly\" → maybe \"has been applied\". \"Observed\" makes it sound a bit like an event in nature.\n* \"Among *existing* ~~the variety of~~ acquisition functions, ~~the~~ expected improvement is widely adopted (Mockus, 1974).\"\n* \"The standard approach of fitting GPs is to optimize the weights of the kernel function, e.g. squared exponential kernel, θ.\"  I think it is misleading to call the optimisation of the kernel *hyperparameters*  'fitting GPs'. Gaussian Processes perform Bayesian *inference* and are not \"fit\" by \"optimising\" any parameters. However, the hyperparameters in the GP (but not the GP itself) are often inferred by maximising the evidence.\n* \"Consequently, the solution of the joint model resides on a local minimum so that given limited information about the new (unseen) target task [______] \" This sentence is missing an ending? What happens given limited information about the task?\n", "summary_of_the_paper": "This paper introduces DKLM, a novel approach to hyperparameter optimisation based on transfer learning.\nDKLM uses a Deep Kernel GP (Wilson et al., 2016) surrogate to predict test losses / response functions for given hyperparameter configurations.\nThe embeddings of the Deep Kernel GP are obtained by aggregating over hyperparameter-response pairs of previously observed datasets.\nThis is the main mechanism through which the HPO is conditioned on previously observed experiments.\nThe embedding mechanism relies on a Deepset (Zaheer et al., 2017) style neural network encoding, that sums over all hyperparameter-response pairs.\nExperiments demonstrate DKLM improves performance over prior work.", "summary_of_the_review": "The proposed method, DKLM, for transfer-learned HPO seems interesting and is, as far as I'm aware, novel.\nThe experimental evaluation seems to demonstrate benefits of DKLM over prior work.\nUnfortunately, the paper is (sometimes severly) lacking in clarity, and the draft requires significant further work before being, in my opinion, acceptable for publication.\nThe exposition leaves me with too many open questions as to how DKLM actually, as well as some of the experiments, actually work.\nFurther, one of the main contributions of the paper – avoiding 'negative transfer' – is not discussed sufficiently and not evaluated at all, and I have trouble following the authors in why it is appropriate to say that DKLM learns \"landmark meta-features\".\nIf the authors can address these main points of my review, I am happy to reconsider my score.\n\nThank you for including reproducibility and ethics statement. Please make code for reproducing experiments available as soon as possible.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dmpwcedmskr", "reviewer": "Reviewer_5B4g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\n- This paper combines two intuitive ideas, and the method is fairly easy to understand\n- Experimentally, aggregated metrics showcase the strength of the proposed DKLM method compared to other baselines\n- The ablation test which removes the _learned_ meta features convincingly shows that investing additional effort into obtaining trained meta-features improves performance.\n\nWeaknesses \n-----------------\n- The major weakness of this paper is that it seems to simply combine to well-known ideas (deep sets + deep kernel GPs), and as such has limited novelty. \n- The clarify of the paper could be improved. In particular, the experimental setup was not entirely obvious to me. Is the optimization done as Bayesian optimization (with a DKLM in the Gaussian process)? If so, which acquisition function was used?\n\nComments/questions\n----------------------------\n- You mention the \"negative transfer phenomenon\" several times as a major motivation for this work; I would recommend describing it in more detail in the paper.\n- I'm curious about the performance variability of DKLM: do you have any insight into which types of datasets or experimental settings DKLM tends to perform better or worse on, compared to other baselines?\n- Did you compare DKLM to a GP parameterized with the learned metafeatures from Dataset2Vec?\n", "summary_of_the_paper": "This paper focuses on doing transfer learning from related tasks for the optimization of ML model hyperparameters.\n\nPrevious work (e.g., Vanschoren, 2018) has considered meta-features as representation of related datasets or tasks; these features are incorporated into the model when transferring to the task at hand. This work proposes to _learn_ these meta features using a deep-set representation (Zaheer et al., 2017). These meta features are then passed to a kernel function that, finally, parameterizes the Gaussian process used during Bayesian optimization.\n\nExperimentally, the authors compare their method to several baselines, both in the transfer and non-transfer learning settings; aggregated metrics show that the proposed method (DKLM) outperforms other baselines on average.\n\n", "summary_of_the_review": "This is an interesting paper, but its novelty is limited. A more detailed analysis of the proposed method (in particular, ablations that use a deep kernel GP with different learned meta-features) would increase the impact of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GX0IeSgi_7P", "reviewer": "Reviewer_ZdAZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The topic is important. Hyperparameter is expensive and transfer learning is a promising solution to reduce the cost of HPO. \n- The solution is relatively complex, involving deep Kernels using neural networks and trained with meta-learning, and yet it is presented clearly and is easy to understand.\n- The benchmark is large, including 10 different relevant algorithms on 935 tasks with 16 different search spaces. \n- The experiments cover different aspects of the contributed algorithm; T-SNE projections of the meta-features and the impact of the sample size.\n\nWeaknesses:\n- A lot of emphasize in introduction is on the fact that DKLM would address negative transfer. I do not see this discussed explicitly in any of the experiments. If it is important, it should be supported by the experiments.\n- The proposed solution is most certainly more computationally expensive than random search, or the fast Bayesian Optimization algorithm ABLR. The experiments should also show the improvement in terms of running time so that we can see how it performs compared to these faster algorithm. \n- The standard deviation on Table 1 is way too large to be able to conclude anything. I could run again the same experiments with different seeds and observe an upward trend instead. \n- The functions f and g introduces many hyperparameters to DKLM. The paper does not discuss how to chose these hyperparameters and whether tuning them is important for the performance of the algorithm. \n\nMinor comments:\nWhat is a meta-dataset? It is not defined in section 2.\n\nI don’t understand this sentence:\n> Consequently, the solution of the joint model resides on a local minimum so that given limited information about the new (unseen) target task.\n\nTypos:\nnew a target task -> a new target task\nfast adaption -> fast adaptation\nwhere we notice -> while we notice", "summary_of_the_paper": "This paper proposes a new Kernel for Bayesian Optimization to incorporate 'Landmark meta-features' of groups of tasks, so that the Kernel can be used to improve the efficience of the Bayesian Optimization on a new related task. The Kernel is based on 2 nested neural networks that are jointly trained via first-order meta-learning method REPTILE. The authors show that the method is overall better performing on the benchmark HPO-B-v3.", "summary_of_the_review": "The proposed algorithm is an interesting way of learning end-to-end meta-features across tasks to improve the efficiency of Bayesian Optimization on new similar tasks. The benchmarking experiments are extensive and convincing for the efficiency of the algorithm, but does not support clearly the claims about negative transfer. The latter point should be discussed more clearly. The additional hyperparameters of the algorithms (network size and REPTILE hyperparameters) should also analyzed empirically to see how importantly they affect the performance of the algorithm.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BgxV24OoTU8", "reviewer": "Reviewer_kJb9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n========\nThe paper very clearly explains the background, existing methods, and issues. It shows how various existing techniques and ideas (deep kernel Gaussian processes, landmark meta-features, deepset) are combined with meta-learning to combat the \"negative transfer\" phenomenon.\n\nExperiments on the large HPO-B benchmark, as well as ablations (on synthetic few-shot regression problems, and in a non-transfer setting), are well-designed and support the paper's conclusions. The research question is clearly stated.\n\nWeaknesses\n==========\nOne would expect that the transfer/few-shot learning setting may be most useful with a small number of trials, however we see that the improvements (wrt. baselines) seem to only appear after about 10-20 trials, and are sustained afterwards. This may be investigated, or maybe addressed as a limitation in the context of the stated goal of limiting the number of HP configuration trials.\n\nMinor points\n==========\n- Some confusion in notations in section 3.2\n  * In Eq (1), is $K_x$ supposed to be $K_*$?\n  * A mean function $m$ is introduced, but not used, and Eq (1) shows the mean as $0$.\n  * In Eq (2), $y$ vs. ${\\mathbf y}$\n- It would be nice to keep the colors consistent between Figures 2 and 3 for the common curves", "summary_of_the_paper": "The paper tackles the issue of speeding up black-box hyper-parameter optimization (HPO) by leveraging results obtained by trials on different datasets. It casts the problem into a few-shot regression one, where each task is a dataset and the goal is to predict the performance of a model trained on this dataset from the values of hyper-parameters. Meta-learning (REPTILE) is then used to initialize the (meta-)parameters of a deep kernel Gaussian process. Experiments on the HPO-B-v3 benchmark show this procedure equals or beats 10 baselines when using 20 to 100 trials.", "summary_of_the_review": "Overall good, well-written, self-contained article that presents a novel combination of existing techniques, and addresses the issue of negative transfer when using transfer learning for Bayesian HPO. The experiments conducted are well designed and support the conclusion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission describes a method for tuning machine learning pipeline hyperparameters using transfer learning from related tuning tasks. In particular, the method uses learned meta features to construct a covariance function for a GP.

This was an extremely difficult case and could have gone either way. It was the closest case for any paper I serve as the AC for. Two of the reviewers recommended rejecting the paper and three recommended accepting, although during discussion one of the reviewers recommending accepting the paper seemed to actually be more on the reject side.

Ultimately, I have decided to recommend rejecting this submission. However, if either the clarity (especially concerning the neural network setup) or the experiments were somewhat improved I would have recommended accepting it. I view clarity as an extremely important factor when weighing whether a submission should be accepted. I concur with the reviewers on the following weaknesses of the experiments: (1) the lack of an ablation test when considering ad-hoc meta features and (2) the experimental evaluation is based on mostly aggregated metrics.

I know this recommendation must be disappointing, but I encourage the authors to polish the work a bit more and resubmit it somewhere.
Decision: Reject