Few-Shot Bayesian Optimization with Deep Kernel Surrogates
Authors: Martin Wistuba
Josif Grabocka
Venue: ICLR 2021 Poster (2021)
Keywords: bayesian optimization
metalearning
few-shot learning
automl

Abstract: Hyperparameter optimization (HPO) is a central pillar in the automation of machine learning solutions and is mainly performed via Bayesian optimization, where a parametric surrogate is learned to approximate the black box response function (e.g. validation error). Unfortunately, evaluating the response function is computationally intensive. As a remedy, earlier work emphasizes the need for transfer learning surrogates which learn to optimize hyperparameters for an algorithm from other tasks. In contrast to previous work, we propose to rethink HPO as a few-shot learning problem in which we train a shared deep surrogate model to quickly adapt (with few response evaluations) to the response function of a new task. We propose the use of a deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of a collection of training data sets. As a result, the novel few-shot optimization of our deep kernel surrogate leads to new state-of-the-art results at HPO compared to several recent methods on diverse metadata sets.

Published as a conference paper at ICLR 2021
FEW-SHOT BAYESIAN OPTIMIZATION
WITH DEEP
KERNEL SURROGATES
Martin Wistuba
Josif Grabocka
IBM Research
University of Freiburg
Dublin, Ireland
Freiburg, Germany
ma hartin, w sbab h.m. com
grabocka cs uni uni-freiburg
ABSTRACT
Hyperparameter optimization (HPO) is a central pillar in the automation of ma- chine learning solutions and is mainly performed via Bayesian optimization, where parametric surrogate is learned to approximate the black box response function (e.g. validation error). Unfortunately, evaluating the response function is computationally intensive. As remedy, earlier work emphasizes the need for transfer learning surrogates which learn to optimize hyperparameters for an algo- rithm from other tasks. In contrast to previous work, we propose to rethink HPO as a few-shol learning problem in which we train a shared deep surrogate model quickly adapt (with few response evaluations) to the response function of a new task. We propose the use of a deep kernel network for a Gaussian process surro- gate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of collection of training data sets. As result, the novel few-shot optimization of our deep kernel surrogate leads to new state-of-the-art results HPO compared to several recent methods on diverse metadata sets.
INTRODUCTION
Many machine learning models have very sensitive hyperparameters that must be carefully tuned for efficient use. Unfortunately, finding the right setting a tedious trial and error process that requires expert knowledge. AutoMI. methods address this problem by providing tools to automate hyper- parameter optimization, where Bayesian optimization has become the standard for this task (Snoek) et al. 2012). It treats the problem of hyperparameter optimization a black box optimization problem. Here the black box function is the hyperparameter response function, which maps hy- perparameter setting the validation loss. Bayesian optimization consists of two parts. First, surrogate model, often a Gaussian process, used to approximate the response function. Second, acquisition function is used that balances the trade-off between exploration and exploitation. In a sequential process, hyperparameter settings are selected and evaluated, followed by an update of the surrogate model.
Recently, several attempts have been made to extend Bayesian optimization to account for a transfer learning setup. It is assumed here that historical information on machine learning algorithms is available with different hyperparameters. This can either be because this information is publicly available (e.g. OpenML) because the algorithm repeatedly optimized for different data sets. To this end, several transfer learning surrogates have been proposed that use this additional information reduce the convergence time of Bayesian optimization.
We propose a new paradigm for accomplishing the knowledge transfer by reconceptualizing the process as a few-shot learning task. Inspiration is drawn from the fact that there are limited number of black box function evaluations for new hyperparameter optimization task (i.e. few shots) but there are ample evaluations of related black box objectives (i.e. evaluated hyperparameters on other data sets). This approach has several advantages. First, single model is learned that is trained to quickly adapt to a new task when few examples are available. This is exactly the challenge face when optimizing hyperparameters. Second, this method can be sealed very well for any number of considered tasks. This not only enables the learning from large metadata sets but also enables the problem of label normalization to be dealt with in way. Finally, we present

---

Published as a conference paper at ICLR 2021
evolutionary algorithm that can use surrogate models to get warm start initialization for Bayesian optimization. All of our contributions are empirically compared with several competitive methods in three different problems. Two ablation studies provide information about the influence of the individual components. Concluding, our contributions in this work are:
This is the first work that, in the context of hyperparameter optimization (HPO), trains the initialization of the parameters of a prohabilistic surrogate model from a collection of meta-tasks by few-shot learning and then transfers it by fine-tuning the initialized kernel
parameters to a target task.
We are the first to consider transfer learning in HPO as a few-shot learning task.
We set the new state of the in transfer learning for the HPO and provide ample evidence that we outperform strong baselines published at ICLR and NeurIPS with a statistically significant margin.
We present an evolutionary algorithm that can use surrogate models to get warm start initialization for Bayesian optimization.
RELATED WORK
The idea of using transfer learning improve Bayesian optimization is being investigated in several papers. The early work suggests learning a single Gaussian process for the entire data (Bardenet al. 2013 Swersky et al. 2013 Yogatama & Mann 2014). Since the training of a Gaussian process is cubic in the number of training points, this idea does not scale well. This problem has recently been addressed by several proposals to use ensembles of Gaussian processes where Gaussian process is learned for each task (Wistuba et al., 2018 Feurer et al. 2018). This idea scales linearly the number of tasks but still cubically the number of training points per task. Thus, the problem persists in scenarios where there is lot of data available for each task.
Bayesian neural networks are a possible more scalable way of learning with large amounts of data. For example, examplc.Schilling et al. (2015) propose to use neural network with task embedding and vari- able interactions. To obtain mean and variance predictions, the authors propose using an ensemble of models. In contrast, Springenberg et al. (2016) use Bayesian multi-task neural network. How- ever, since training Bayesian neural networks is computationally intensive, Perrone et al.] (2018) propose more scalable approach. They suggest using neural network that is shared by all tasks and using Bayesian linear regression for each task. The parameters are trained jointly on the entire data. While work shares some similarities with the previous work, algorithm has unique properties. First of all, a meta-learning algorithm is used, which is motivated by recent work on model-agnostic meta-learning for few-shot learning (Finn al.] 2017). This will allow to inte- grate all task-specific parameters out such that the model does not grow with the number of tasks. As a result, our algorithm scales very well with the number of tasks. Second, while we are also using a neural network, we combine it with a Gaussian process with a nonlinear kernel in order to obtain uncertainty predictions.
A simpler solution for using the transfer learning idea in Bayesian optimization is initializa- tion (Feurer et al., 2015) Wistuba al., 2015a). The standard Bayesian optimization routine with a simple Gaussian process is used in this case but it is warm-started by number of hyperparameter settings that work well for related tasks. We also explore this idea in the context of this paper by proposing a simple evolutionary algorithm that can warm start initialization sequence. The use of an evolutionary algorithm is motivated by its ease of implementation and natural capability deal with continuous and discrete hyperparameters.
surrogate model to estimate data-driven
PRELIMINARIES
3.1 BAYESIAN OPTIMIZATION
Bayesian optimization is an optimization method for computationally intensive black box functions that consists of two main components, the surrogate model and the acquisition function. The sur- rogate model is a probabilistic model with mean j and variance o², which tries to approximate the unknown black box function f, in the following also response function. The acquisition function
2

---

Published as a conference paper at ICLR 2021
provide score for each feasible solution based on the prediction of the surrogate model. This score balances between exploration and exploitation. The following steps are carried out sequen- tially. First, the feasible solution that maximizes the acquisition function is evaluated. In this way new observation (x, f())) is obtained. Then, the surrogate model is updated based on the entirety of all observations D. This sequence of steps is carried out until a previously defined convergence cri- terion occurs (c.g. exhaustion of the time budget). In Figure I we provide an example how Bayesian optimization used maximize sine wave.
Expected Improvement (Jones et al. 1998) is one of the most commonly used acquisition functions and will also be used in all our experiments. It is defined
(x E |mma (f(x) y/max, 0}}
where Umax the largest observed value of f.
3.2 GAUSSIAN PROCESSES
where yr are noisy obser- vations of the function values yi = fxx) + s. We assume that the noise e is additive independent identically distributed Gaussian with variance o For Gaussian processes (Rasmussen & Williams, 2006) we consider Y/ to be random variable and the joint distribution of all yi is assumed to be multivariate Gaussian distributed:
We have a training set D of observations, = {(x; 8:)li -, .....,,... nn.
y N (m (X,))(XXX)
Gaussian process is completely specified by its mean function m, its covariance function kt and may depend parameters 9. A common choice is to set m(x,) 0. At inference time for instances x., the assumption is that their ground truth f. is jointly Gaussian with y
y P
~ L~NO.
(0,
K K
K, K,.
))
(3)
where
K₂ h: X(0) + II, K. = 1 (X,x.10), K., = t (X., (
(4)
for brevity. Then, the posterior predictive distribution has mean and covariance
y, K,,,,,, cov [f. K.. K.
(5)
Examples for covariance functions are the linear squared exponential kernel. However, these kernel functions are designed by hand. The idea of deep kernel learning (Wilson al., 2016)) is to learn the kernel function. They propose to use neural network yp to transform the input x to a latent representation which serves as input for the kernel function.
keoop x' x/10, w) k((p(x, w) w)
4 FEW-SHOT BAYESIAN OPTIMIZATION
Hyperparameter optimization is traditionally either tackled an optimization problem without prior knowledge about the response function or alternatively as a multi-task or transfer learning problem. In the former, every search basically starts from scratch. In the latter, or multiple models trained that attempt to reuse knowledge of the source tasks for the target task.
In this work we will address the problem as a few-shot problem. Given T related source tasks and very few examples of the target task, we want to make reliable predictions. For each of the source tasks we have observations (() {x(" (t)
where x(" is hyperparameter setting and y, is the noisy observations of (() (t) i... a validation score of a machine learning model on data set f. In the following we will denote the set of all data points by D : U"_, D(t), Model- agnostic meta-learning (Finn et al. 2017) has become a popular choice for few-shot learning and propose to model within the Bayesian optimization framework. The idea is simple. A deep kernel & is used to learn parameters across tasks such that all its parameters O and are task-independent. All
adaptation for Gaussian processes (Patacchiola et al.] 2020)
surrogate
3

---

Published as a conference paper at ICLR 2021
task-dependent parameters are kept separate which allows to marginalize its corresponding variable out when solely optimizing for the task indeepnden parameters. If we assume that the posteriors over A and w are dominated by their respective maximum likelihood estimates i and w, we can approximate the posterior predictive distribution by
p (ff.lx,,)) D) - (fflxx, (0,, w = D,)
We estimate θ and w by maximizing the log marginal likelihood for all tasks.
logp (y(),...,, y( (()) X 0,
logp (y(*) w)
(8)
T α ((0 k(0)"² y(t) + log K
We maximize the marginal likelihood with stochastic gradient ascent (SGA). Each batch contains data for one task. It is important to note that this batch data does not correspond to the data in D(t), but rather a subset of With small batch sizes, this is an efficient way to train the Gaussian process, regardless of how many tasks knowledge is transferred from or how large D(4) is. It is important to note that there is a correlation between all data per task and not just one batch. Hence the stochastic gradient is biased estimator of the full gradient. For long time we lacked the theoretical understanding of how SGA would behave in this situation. Fortunately, recent work (Chen et al., 2020) proved that SGA still successfully converges and restores model parameters in these cases. The results by Chen et al. (2020) guarantee O(1/K) optimization margin of error, where K is the number of iterations.
The training works follows. In each iteration, task t is sampled uniformly at random from all T tasks. Then sample a batch of training instances uniformly random from ((t), Finally, calculate the log marginal likelihood for this batch (Equation 8 and update the kernel parameters with one step in the direction of the gradient.
In hyperparameter optimization, we are only interested in the hyperparameter setting that works best according to a predefined metric of interest. Therefore, only the ranking of the hyperparameter settings is important, not the actual value of the metric. Therefore, we are interested in a surrogate model whose prediction strongly correlate with the response function and the squared error of little to no interest for us. In practice, however, the minimum / maximum score and the range of values differ significantly between the tasks which makes it challenging surrogate model. The most common way to address this problem is to normalize the labels, e.g. by z-normalizing or scaling the labels to [0, 1] per data set. However, this does not fully solve the problem. The main problem is that this label normalization must also be applied to the target task. This is not possible with a satisfactory degree of accuracy, especially when only few examples are available, since the approximate values for minimum / maximum mean / standard deviation are insufficiently accurate.
obtain a strongly correlating
Since proposed method scales easily to any number of tasks, option. We propose a task augmentation strategy that addresses the label normalization issue by randomly scaling the labels for each batch. Since Ymin and Ymux are the minimum and maximum values for all T tasks, we can generate augmented tasks for each batch B = (x;, ys Jim lim..... D(t), where b the batch size, as follows. lower and upper limit is sampled for each sample batch,
afford to consider another
~ (ymmn: Ummx), ), ~ U (ymins Ummx)
(10)
such that < u holds. Then the labels for that batch are scaled to this range,
(11)
No further changes are required. We summarize this procedure in Algorithm [ The idea here is to leam representation that is invariant with respect to various offsets and ranges that the target task does not require normalization. This strategy is not possible for other hyperparameter opti- mization methods, since this either increases the training data set size further and thus becomes
4

---

Published as a conference paper at ICLR 2021
Algorithm 1: Few-Shot GP Surrogate
Learning rates a and B, training data D, kernel k; and neural network sp. while not done do Sample task ~ (1((1 ................ ));
Input:
Estimate / and (Equation 10
for by times do
Sample batch = {(x,, yi)). D(t) and scale labels y using and u;
Compute marginal likelihood L on B. (Equation 8);
r + VaCC;
WC
end end
2.5
0.0
2.5
2.5
0.0
2.5
-5
-5
-5
Response Function
Posterior Mean StDev
Expected Improvement
Posterior Mean
Observations
Figure 1: Demonstration of five steps of Bayesian Optimization with FSBO for maximizing sine wave (blue). One maximum discovered within only three steps. Expected Improvement has been scaled to improve readability. In black the predictions of the surrogate model. Bottom right examples of the source tasks. The deep kernel consists of a spectral kernel (Wilson & Adams 2013) combined with two-layer neural network (1 - 64 64).
computationally impossible (Bardenet et al. 2013 Swersky et al. 2013 Yogatama & Mann 2014) or thousands of new tasks would have to be generated, which is is not practical (Springenberg
et al. 2016 Perrone et al. 2018 Wistuba et al. 2018 Feurer et al. 2018).
At test time, the posterior predictive distribution or the target task T + computed according to Equation []] In practice, the target data set can be very different to the learned prior. For this reason, the deep kernel parameters are fine-tuned on the target task's data for few epochs. Our task augmentation strategy described earlier has resulted in model that has become invariant for different scales of y. For this reason we do not apply this strategy to target task it for all source tasks to T. We discuss the usefulness of this step in more detail when discuss the empirical results.
and only
A MOTIVATING EXAMPLE
We would like to first motivate our method, FSBO, with an example, similar the one described in Nichol et al. 20188. The aim is to find the argument æ that maximizes a one-dimensional sine wave of the form (()( = ( sin(x + 6())). not known that the function a sine wave and that one only has determine the amplitude and phase b. However, information about other sine waves
5

---

Published as a conference paper at ICLR 2021
Mutation
Crossover
2M
Replace one clement
at random
Merge two sets
random
JF32
JBA
Jits
JBa
Figure 2: Examples for the mutation and crossover operation with / 3.
available to the optimizer which are considered to be related. These source tasks / = , generated randomly with ((001, 5) and b 24(0, 2x). 50 examples (a. (x) available for each task t, whereby the points (2) E [- 5] are evenly spaced.
It should be noted this point that the expected value for each 2; is 0. Thus, training the data described above leads to model predicting 0 for every Ii. However, the used meta-learning after only a few examples
procedure allows for accurately reconstructing the underlying sine (see Figure D.. Provided that the response function of the original task is similar to the target task, this is very promising technique for accelerating the search for good hyperparameter settings.
A DATA-DRIVEN INITIALIZATION
The proposed few-shot surrogate model requires only a few examples before reliable predictions can be made for the target data set. How do you choose the initial hyper parameter settings? The simplest idea would be to pick them at random use Latin Hypereube Sampling (LHS) (McKay et al. 1979). Since we already have data from various tasks and a surrogate model that can impute missing values, we propose data-driven warm start approach. we evaluate the performance of hyperparameter optimization algorithm with loss function L, we use an evolutionary algorithm to find a set with I hyperparameter settings which minimizes this loss on the source tasks.
T Σ c C arg min
x (init)
arg min XeX!
Σ min (x)
(12)
XeX!
The loss of a set of hyperparameters depends on the response function values for each of the ele- ments. Since these are not available for every x, we approximate fx9 with the prediction of the surrogate model described in the last section whenever this is necessary.
Arbitrary loss function can considered. The specific loss function used our experiments normalized regret and defined on the right side of Equation 12 where f(t) is derived from (() by scaling it to [0, 1] range:
f(0(x)
p(t)
f(0 =
min
(13)
(())
(t)
where ((() and (()) are the maximum and minimum f()), respectively.
The evolutionary algorithm works follows. We initialize the population with sets containing I random settings, the settings being sampled in proportion to their performance according to the predictions. Precisely, setting is sampled proportional
exp (-
min minn
j(0 (x)
(14)
The best sets are then selected to be refined using evolutionary algorithm. The algorithm ran- domly chooses either to mutate a set or to perform a crossover operation between two sets. When mutating set, setting is removed uniformly random and new setting is added proportional to its predicted performance (Figure g left). The crossover operation creates a new set and adds elements from the union of the parent sets until the new set has I settings (Figure right). The

---

Published as a conference paper at ICLR 2021
set is added to the population. After 100,000 steps, the algorithm is stopped and the best set is used as the set of initial hyperparameter settings. In Figure 4 we compare this warm start initialization
with the simple random or LHS initialization.
7 EXPERIMENTS
We used three different optimization problems compare the different hyperparameter optimiza- tion methods: AdaBoost, GLMNet, and SVM. We created the GLMNet and SVM metadata set by downloading the 30 data sets with the most reported hyperparameter settings from OpenML for each problem. The AdaBoost data set is publicly available (Wistuba et al. 2018). The number settings per data set varies, the number of settings across all tasks for the GLMNet problem is approximately 800,000. See the appendix for more details about the metadata sets. We compare the following list of hyperparameter optimization methods.
Random Search This is a simple but strong baseline for hyperparameter optimization (Bergstra & Bengio 2012). Hyperparameters settings are selected uniformly at random from the search space.
Gaussian Process (GP) Bayesian optimization with a Gaussian process as a surrogate model is a standard and strong baseline (Snoek al.) (012). We
Matérn 5/2 kernel with ARD and rely on the scikit-optimize implementation. We compare with two variations. The first is the vanilla method, which uses Latin Hypercube Sampling (LHS) with design size of 10 initialize. The second method uses the warm start (WS) method described in Section 6 to find an initial set of 5 settings to use the knowledge from other data sets.
RGPE RGPE (Feurer et al., 2018) is one of the latest examples of methods that use GP ensembles to transfer knowledge across task (Wistuba et al.) 2016)] Schhiling et al.) [016] Wistuba al.) 2018). In this case GP is trained for each individual task and in final step the predictions of the GPs are aggregated. These ensembles scale linearly the number of tasks, but the computation time still cubic and the space requirement is still quadratic in the number of data points per task. For this reason we can only present results for AdaBoost and not for the other optimization problems, which have significantly more data points.
MetaBO Using the acquisition function instead of the surrogate model for transfer learning is an- other option. MetaBO is the oftte-of-theaan transfer acquisition function and presented last year at ICLR (Volpp et al. 2020). We use the implementation provided by the authors.
ABLR The state-of ahet surrogate model for scalable hyperparameter transfer learning (Perrone jet al. 2018). This method uses a multi-task GP which combines a linear kernel with a neural network and scales very large data sets. Transfer learning is achieved by sharing the neural network parameters across different tasks. By ABLR (WS) we are referring to a version of ABLR that uses the same warm start FSBO.
Multi-Head GPs This closely resembles ABLR but uses the same deep kernel as our proposed method. The main difference from our proposed method that it using a GP for every task, only shares the neural network across tasks, and uses standard stochastic gradient ascent.
Few-Shot Bayesian Optimization (FSBO) Our proposed method described in Section 4 The deep kernel is composed of two-layer neural network (128 - 128) with ReLU activations and a exprneniaap kernel. We use the Adam optimizer with learning rate 10 10-3 and a batch size of fifty. The warm start length is five.
The experiments are repeated ten times and evaluated in a leave-one-task-out means that all transfer learning methods use one task as the target task and all other tasks as source tasks. For AdaBoost we use the same train/test split as used by Volpp et al. (2020) instead. We report the aggregated results for all tasks within one problem class with respect to the mean of normalized regrets. The normalized regret for task obtained by first scaling the response function values between 0 and before calculating the regret (Wistuba et al. 2018). i.e.
cross-validation. This
p(() (xmmix))))) -(())
(15)
7

---

Published as a conference paper at ICLR 2021
Method
AdaBoost
GLMNet
SVM
15
33
50
33
67
100
33
67
100
RANDOM
4.87
3.02
2.16
0.85
0.40
0.29
2.01
1.12
0.83
GP (LHS)
3.87
2.49
2.23
1.32
1.01
0.73
1.94
1.40
1.14
GP (WS)
3.25
1.66
1.02
0.86
0.36 0
0.24
1.10
0.81
0.65
RGPE
5.29
3.26
2.83
METABO
5.27
3.52
1.96
11.02
10.97
10.96
12.39
12.39
11.93
ABLR
4.56
1.44
1.24
1.77
0,50
0.40
1.81
1.23
0.84
ABLR (WS)
3.17
1.72
1.17
0.54
0.36
0.29
1.47
1.08
0.87
MULTI-HEAD GPs (WS)
6.78
3.45
1.80
2.83
2.80
2.68
11.20
9.82
8.72
FSBO
3.10
1.13
0.80
0.42
0.22
0.16
0.79
0.51
0.36
Table 1: FSBO obtains better results for all hyperparameter optimization problems. The best results are in bold. Results that are not significantly worse than the best are in italics. Used initialization in parentheses, (LHHS) Latin Hypercube Sampling, (WS)- Warm Start.
where f()) is the normalized response function (Equation 13
z(t) min is the global minimum for task and Xmin is the best discovered hyperparameter setting by the optimizer.
7.1 HYPERPARAMETHS OPTIMIZATION
We conduct experiments on three different metadata sets and report the aggregated mean of normal- ized regrets Table [ The best results are bold. Results that are not significantly worse than the best are in italics. We determined the significance using the Wilcoxon signed rank test with confi- dence level of 95%. Results are reported after every 33 trials of Bayesian optimization for GLMNett and SVM. Since AdaBoost has fewer test examples, we report its results in shorter intervals.
Our proposed FSBO method outperforms all other transfer methods all three tasks. Results are significantly better but in the case of AdaBoost where GP (WS) achieves similar but on average worse results. The results obtained for MetaBO are the worst. We contacted Volpp et al. (2020) and they confirmed that results are obtained correctly. The problem is apparently that the Re- inforcement Learning method does not work well for larger number of trials. We provide a longer discussion in the appendix. Also ABLR and the very related Multi- GP not performing very well. As we show in the next section, one possible reason for this might be because the neural network parameters are fixed which will prohibit a possibly required adaptation to the target task. The vanilla GP and its transfer variant that uses a warm start turn out to be among the strongest baselines. This is particular true for the warm start version which is the second best method. This simple baseline of knowledge transfer is often not taken into account when comparing transfer surrogates, although it is easy to implement. Due to the very competitive results, we recommend using it as standard baseline to assess the usefulness of new transfer surrogates.
7.2 COMPONENT CONTRIBUTIONS
In this section we highlight the various components that make a significant difference to ABLR. The results for each setup reported in Figure The Multi-Head GP is surrogate model that shares the neural network between tasks but uses a separate Gaussian process for each task. closely resembles the idea of ABLR but is closer our proposed implementation FSBO. Starting from this configuration, we add various components that will eventually lead to our proposed model FSBO. We consider Multi-Hes Head GP (WS), a version that additionally uses the warm start method described in Section instead of a random initialization. Multi- Head GP (fine-tune) not only updates the kernel parameters when a new observation received but also fine-tunes the parameters of the neural network. Finally, FSBO is our proposed method, which uses only one Gaussian process for all tasks. We see that Multi-Head GP versions have a problem adapting to the target tasks efficiently. Fine-tuning the deep kernel is important part of learning FSBO. Although FSBO outperforms all Multi-Head GP versions without ane-tunnng, the additional use of it makes for a significant improvement. We analyzed the reason for this in more detail. We observed that the learned neural
Out-of-memory exception due large data.
8

---

Published as a conference paper at ICLR 2021
AdaBoost
GLMNet
SVM
Normalized Regret
10
10"
100
10a
0
20
Number of Trials
Number of Trials
Number of Trials
Head GPs
Multi-Heat Head GPs [warm start siart-ffine finette)i-
FSBO (warm start start---ine-turr -ine-tuune)
Head (warm start)
FSBO (warm start)
Figure 3: Comparison of the contribution of the various FSBO components to the final solution.
Each component makes its own orthogonal contribution.
AdaBoost
GLMNet
SVM
Normalized Rogret
30
30-
20-
20-
20-
10-
8
10
2
10
Initialization Length
Initia lizaaion Length
Initialization Length
Random
LHS
Warm Start
Figure 4: The warm start initialization yields the best results on GLMNet and SVM for all initial- ization lengths. For AdaBoost it is comparable LHS.
network provides strong prior that leads to strong exploitation behavior. Fine-tuning prevents this and thus ensures that the method does not get stuck in local optima.
7.3 DATA-DRIVEN WARM START
The use of a warm start is not the main contribution this paper but it interesting to explore fur- ther nonetheless. First of all, for some methods this is the only differentiating factor. We compare our suggested warm start initialization with a random and Latin hypercube sampling (LHS) initial- ization in Figure 4 Considering that GP (WS) always performed better than GP (LHS) in Table T one would expect that the warm start itself also performs better than LHS. While this is the case for significant differences for the AdaBoost problem. Our assumption
GLMNet and SVM, there that in this case the warm start might not have always found good settings as part of the initializa- tion. it still explored areas in the search space close to the optimum. This would facilitate finding better solution in one of the subsequent steps of Bayesian optimization.
CONCLUSIONS
In this work, we propose to rethink hyperparameter optimization as a few-shot learning problem in which we train shared deep surrogate model to quickly adapt (with few response evaluations) to the response function of new task. We propose the use of deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion order to jointly approximate the response functions of a collection of training data sets. This few-shot surrogate model is used for two different purposes. First, we use it combination with an evolutionary algorithm in order to estimate warm start initialization for Bayesian optimization. Second, we it directly for Bayesian optimization. our empirical evaluation on three hyperparameter optimization problems, observe significantly better results than with state-of-the-art methods that use transfer learning-

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work has been supported by European Union's Horizon 2020 research and innovation pro- gramme under grant number 951911 A14Media. Prof. Grabocka is thankful to the Eva Mayr-Stihl Foundation for their generous research grant.
REFERENCES
Rémi Bardenet, Mátyás Brendel, Balázs Kégl, and Michèle Sebag. Collaborative hyperparameter tuning. In Proceedings of the 30th International Conference Machine Learning, /CML 2013, Atlanta, GA. USA, 16-21 June 2013, pp. 199- 2013. URL http://proceecinge mmlr.
press/ www/b//ddeeeeel.ea
html
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012. URL http::ttp//ll acm. wtt/://at/cooooo....o///.. cfm?id=
2188395
Hao Chen, Lili Zheng, Raed AL Kontar, and Garvesh Raskutti. Stochastic gradient descent in correlated settings: A study gaussian processes. In Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, 2020. URL httos://papers: nips. wwtppper.pappppppppppp /2tp//2a2h///// 33554455a3f3r82beee22 035554063367705 Abs.teeeetaattttt.tta...... rtmt.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings the Twenty-Ninth AAAI Conference on Artificial the
Intelligence, Jamuary 25-30, 2015, Austin, Texas, USA, pp. 1128-1135, 2015. URL http://
www. www.aaai.co www.ai.orr//cco /cs/ ndex. php/AAAI/ AAAAAAAAAAAAAAAAAA AAAAAAAATIII /ttpa//AAAA////pe//eeeee////////
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian opti- mization. CoRR, abs/1 1ls/0202,,229,, 2018. URL htt.:/tp://aaxi..orrr/bb//////// 802 02219
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1126-1135, 2017. URL http:
//proceedings ml ppreas...rsp.....mmmmmmmm.mm...e 7 ninnn11 7a httll
Donald R. Jones, Matthias Schonlau, and William Welch. Efficient global optimization of pensive black-box functions. J. Global Optimization, 3344):55-492 1998. doi: 10.1023/A 1008306431 147. URL nttps://ddet www.orrr.... 02.3/: 008306431144
Balázs Kégl and Róbert Busa-Fekete. Boosting products base classifiers. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 18-18, 2009, pp. 497-504, 2009, doi: 10.1 00.1144/1553374.1555555 1553439,, URL https:
7/doi. .rg. 10. 1145 00155/1553744 1553439
M. D. McKay, R. J. Beckman, and W. Conover. comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21 239-245, 1979. ISSN 10401706. URL htRL:ttp:// poctp://wee /wwww//wwwww/..../ jstor. wwwps/ttaaeeeeeeeeree
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
Massimiliano Patacchiola, Jack Turner, Elliot Crowley, and Amos Storkey. Bayesian meta-learning for the few-shot setting via deep kernes. In Advances Neural Informa- tion Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020. URL lattss://papers nips wttpp//pppp//////aass.. http:p//wws...aaaoo.eaaaaae/////
html
Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cédric Archambeau. Scalable hy- perparameter transfer learning. In Advances Neural Information Processing Systems 31: An- nual Conference Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 6846 6856, 2018 URL /ttpt::://ppppaaa.// paperspppps.aaaa nips cC cc/ccc//ppp/o paper/
791 s778-cal1 abl http::/eelllllyrrrrrrrraataaaaaa
10

---

Published as a conference paper at ICLR 2021
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning, MIT Press, 2006. ISBN 026218253X.
Nicolas Schilling, Martin Wistuba, Lucas Drumond, and Lars Schmidt-Thieme. Hyperparameter optimization with factorized multilayer perceptrons. In Machine Learning and Knowledge Dis- covery Databases European Conference, ECML PKDD 2015, Porto, Portugal, September doi: 10. 1011077978-3331--2555-5 3- 319 23525- 7_6
7-11, 2015, Proceedings, Part 11. pp. 87-103, 2015.
URL
https: / //oii owo.oogg//ooooo /0. 111.0.111100000
Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thieme Scalable hyperparameter optimiza- tion with products of gaussian process experts. In Machine Learning and Knowledge Discovery in Databases- - European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19- doi: 10. 10.1007/978-3-399-461
23, 2016, Proceedings, Part I, pp. 48.48, 2016.
URL
https: www.oi.oooooo woo..org/l00 11..0.110000 1007 9788 3- 311---128--1111
Jasper Snock, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimiza- tion of machine learning algorithms. In Advances in Neural Information Process- ing Systems 25: 26th Annual Conference on Neural Information Processing Sys- tems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 2960-2968, 2012. URL http:/ http::/papprs.... nww.. ntpp..c.pppp.rrrr.///ccpcpppppp. 4522 practical seaetical-bayesiar opt Limizaation of machine -aaciiin lewrning -ennng
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimiza- tion with robust bayesian neural networks. In Advances in Neural Information Processing Sys- lems 29; Annuul Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4134 4142, 2016. URL httt.http://peeee
nips cc cc/paper/ 6117 1117-bayesian-optinne zat ion with www.ssttbayeseeeee oow.oooww.....tttotoaaaaaaaaaaaa networks
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Multi-task bayesian optimization. In Ad- vances Neural Information Processing Systems 26: 27th Annual Conference on Neural Infor- mation Processing Systems
2013. Proceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2004 2012, 2013. URL http::/pppprrstett/////// nips cc cc/paper/
5086 multi task bayesiaabaaaaa-aa--- opt imi zation
Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning bayesian op- timization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL htl.https:// /openrvvieeeeeeeeeooeeeeee httpps:///pperreviieet ryeYpJSKwr
net/ forum?
Andrew Gordon Wilson and Ryan Prescott Adams. Gaussian process kernels for pattern dis- covery and extrapolation. In Proceedings of the 30th International Conference on Machine Learning. ICML 2013, Atlanta, GA, USA, 16-21 June 2013, pp. 1067-1075, 2013. URL v28 hwtp//wwwwon...mmm/oo/oo/.. html
http://proceedinng wwt.smll...r.rmm..
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernll learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, pp. 370-378, 2016. URL http: html
mnr. attpeedi.n...mm.ppprsss///////
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. Learning hyperparameter optimiza- tion initializations. In 2015 IEEE International Conference Data Science and Advanced An- alytics, DSAA 2015, Campus des Cordeliers, Paris, France, October 19-21, 2015, pp. 1-10, 2015a. doi: 10.1100/DSA..215..77 2015 7344817
URL httts:://doi.org//2.
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thise Thiemm. Sequential model-free hyperparam- eter tuning. In 2015 IEEE International Conference Data Mining. ICDM 2015, Atlantic City, NJ, USA. November 14-17, 2015, pp. 0038-1038, 2015b. doi: 10.1109/CDM22015.200 URL
https: :ttps://doi.. .ar.ii.rg//oio www./oo...o..... 1109 CDM. 201 5.250.20

---

Published as a conference paper at ICLR 2021
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme Two-stage transfer surrogate model for automatic byperparameter optimization. In Machine Learning and Knowledge Discovery in Databases European Conference. ECML PKDD 2016, Riva del Garda, Italy, September 19. doi: 10. 1111007977-93331--666888 https: htt./oi.oggg//0/ 1007 978 --
23, 2016, Proceedings, Part I, pp. 199-214, 2016.
13. URL
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thiemee Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Mach. Learn, 107())433-7,, 2018. doi: 10. 10077/119994--77-55664 URI https:://do .argiiorg11... 007 1077/10199994017--58888888
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, Reykjavik, Iceland, April 22-25, 2014, pp. 1077-1085, 2014. URL
http://proceeddingss ml www.mm.........e....ee wttpppppes/s/s//peeeeee yogga//at////yooaammmmmaa html
12

---

Published as a conference paper at ICLR 2021
AdaBoost
GLM
SVM
10'
Nommalized Regret
10'
10P
10f
10P
10
0
20
40
100
100
Number of Trials
Number of Trials
Number of Trials
Random
MetaBO ABLR
Multi-Head GPs (warm start)
GP (LHS)
FSBO
GP (warm start)
Figure 5: FSBO is the best of all considered methods.
A HYPERPARAMETER OPTIMIZATION
In the main paper we report the normalized regret after every 33 trials of Bayesian optimization in a table. This allows us to also report statistical significance. In Figure we show results for all 100 trials. The conclusions remain unchanged. FSBO provides consistently the best results.
B CHALLENGES WITH METABO
The results reported in the main paper for MetaBO seem not to align with the numbers reported by Volpp et al. (2020), in particular for AdaBoost. In order to understand this behavior, conducted a deeper analysis for the AdaBoost metadata set which was used in the evaluation of Volpp et al.) (2020) as well. Using the authors' code we executed the same scripts used by the authors report their results for 7' = 15 trials. Furthermore, we changed two lines in the scripts to train MetaBO for 50, the number of trials considered for AdaBoost in our experiment. We report the outcome of these two experiments in Figure 6 In the two left plots we show how the learned policies for 15 and T 50 perform on validation and test during training. The authors' default setting for the total number of iterations is 2000. However, we noticed a sudden drop in regret for the setting 7 50 around 2000 iterations. For that reason we assumed that further training might help to further improve MetaBO and increased it to 5000 iterations. As can be seen in the middle plot, that was not the case. We observe that the policy learned for T 15 has a lower regret than the one for T' 50 even though it is only trying 15 compared to 50 trials. In the right plot show the results compared to a simple random search baseline. The results for T = 15 are line with those reported by Volpp al. (2020). We contacted the authors and they confirmed that we their code correctly. Their explanation is that the increase of episode length makes this a harder problem for Reinforcement Learning. This results in worse results for T = 50 and even worse for T= 100 in case of the GLMNet and SVM problem. They proposed to randomly vary T between and 50 during the training phase. We also considered this training protocol and report the results in Figure 6 This improves over training only with T = 50 but does not improve over a simple random search.
C METADATA
We created the GLMNet and SVM metadata using OpenML. We chose the 30 OpenML flows with the most observations. We also limited ourselves to the uploaded results from user with ID 2702 (OpenML.Boo) R), who uploaded the majority all runs, to ensure that the metadata was generated under similar cireumstances. In some cases the choice of single hyperparameter is not indicated.
In such a case, we assume that the default value was used. The GL.Ner metadata has two conti- nous hyperparameters: the elastic-net mixing parameter [0, and the regularization parameter E [2-10, 22"). The SVM metadata has two mandatory hyperparameters and two conditional
13

---

Published as a conference paper at ICLR 2021
T = 15
T 50
AdaBoost
0.05
0.05
Simple Regret
0.01
0.01
Normalized Regret
0.08
0.003
0.02
0.02
10
0.01
0.01
0.00
0.00
2000
2000
4000
Iterations
Iterations
Number of Trials
MetaBO (Valid)
MetaBO (T=15)
MetaBO (Test)
MetaBO (T=50)
Random (Test, T=50)
MetaBO (T=5. 50)
Random
Figure 6: Left two plots:
Training MetaBO for different number of trials T. Right plot: comparing
the final policy against a random search. Results for T = 15 match the results reported by Volpp al. 2020).
Method
AdaBoost
GLMNet
SVM
15
33
50
67
100
33
100
FSBO
3.10
1.13
0.80
0.42
0.22
0.16
0.79
0.51
0.36
FSBO (REPTILE)
3.80
2.13
1.21
0.36
0.19
0.14
1.48
0.89
0.72
Table 2: Comparing different model-agnostic meta-learning approaches. Again, the best results are in bold and results that are not significantly worse than the best are in italics.
hyperparameters. The kernel (linear, polynomial or RBF) and the continuous trade-off parameter C' E [2-10, 21 must always be selected. The degree d E [2, 5, is only considered for the polynomial kernel and the bandwidth 7 E [2-10, 2200 is only considered for the RBF kernel.
We further use the AdaBoost metadata which has been used to evaluate MetaBO (Volpp et al., 2020). According to Wistuba et al. (2015b), this metadata was created using AdaBoost (Kégl & Busa-Fekete| 2009) with decision products weak learners. The authors designed a grid over the two continuous hyperparameters: the number of iterations and the number product terms. The number of iterations was chosen from {2, 5, 10, 20, 50, 100, 200, 500, 10 2 103, 5 10 10*), the number product term was chosen from (2, 3, 4, 5, 7, 10, 15, 30).
The metadata was created by rumning grid search on 50 different classification data sets, using clas- sification accuracy as an objective. Compared to the other two metadata sets, this one is significantly smaller. Its use mainly motivated by comparing to evaluate MetaBO. For that reason, we do not use the leave-one-data-set--n cross-validation pro- tocol but instead use the same fixed split created by Volpp et al. (222
MetaBO under the same cireumstances used
Statistics about these metadata sets are provided in Table 3
D META-LEARNING WITH REPTILE
In principle our few-shot surrogate can be combined with any model-agnostic meta-learning ap- proach. In this section we compare our proposed meta-learning approach against Reptile (Nichol et al,, 2018). first-order approximation of MAML. We use the same hyperparameters and data augmentation as described in Algorithm[] Reptile introduces a new hyperparameter, an outer learn- ing rate. We set it to 0.1 and decay it linearly to O. As summarized in Table 2 FSBO with the meta-learning approach described in Algorithm iss either better or not significantly worse than its variation with Reptile.
14

---

Published as a conference paper at ICLR 2021
AdaBoost DATA SET
GLMNet
SVM
RUNS
OPENML ID
RUNS
OPENML ID
RUNS
A9A
108
3
12796
37
2429
W8A
108
31
60721
3485
1000
ABALONE
108
37
17950
3492
3217
APPENDICITIS
108
219
13963
3493
AUSTRALIAN
108
3492
16968
3494
1854
AUTOMOBILE
108
3493
19931
3889
1951
BANANA
108
3889
13942
3891
3083
BANDS
108
3891
13917
3899
2487
BREAST-CANCER
108
3899
15954
3902
1105
BUPA
108
3903
39155
3903
1018
CAR
108
3913
22949
3913
1448
CHESS
108
3917
16960
3918
1459
COD-RNA
108
3918
20984
3950
COIL2000
108
9914
37635
6566
2702
COLON-CANCER
108
9946
25532
9889
2485
CRX
108
9952
25854
9914
1498
DIABETES
108
9967
21994
9946
1291
ECOLI
108
9978
19956
9952
2483
GERMAN-NUMER
108
9980
31961
9967
1492
HABERMAN
108
9983
37134
9971
1381
HOUSEVOTES
108
10101
66277
9976
1473
JCCNII
108
25923
35555
9978
KR-VS-K
108
145847
29927
9980
2943
LED7DIGIT
108
145857
31180
9983
987
LETTER
108
145862
12954
10101
1871
LYMPHOGRAPHY
108
145872
17835
14951
1765
MAGIC
108
145953
47439
34536
1499
MONK-2
108
145972
19972
34537
1489
PENDIGITS
108
145979
12979
145878
1244
PHONEME
108
146064
43785
146064
1435
PIMA
108
RING
108
SAHEART
108
SEGMENT
108
SEISMIC
108
SHUTTLE
108
SONAR-SCALE
108
SPAMBASE
108
SPECTFHEART
108
SPLICE
108
TIC-TAC-TOE
108
TITANIC
108
TWONORM
108
USPS
108
VEHICLE
108
WDRC
108
WINE 108
WINEQUALIT-REDD
108
WISCONSIN
108
YEAST 108
TOTAL
5400
ToTAL 804159
TOTAL
54418
Table 3: Metadata set statistics. OpenMI. refers a task on openml.org.
15


Reviews:
[{"review_id": "b_wz6PMS-h8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to solve the expensive Bayesian optimization from the view of few-shot learning by resorting to the usage of deep kernel learning for hyperparameter optimization. The major concerns regarding this paper are listed as below.\n\nThe description of GP in sec. 2.2 is confusing. The paper considers the noise variance \\sigma_n^2 for K_n while ignoring it for K_* in eq. 5, which is inconsistent. The authors are recommended to refer to the GP model described in the book Gaussian Processes for Machine Learning by Rasmussen.\n\nThe second para of sec. 3 defines T source tasks, leaving no symbols for indicating the interested target task. Similarly, the reviewer cannot find the definition of target task in the following equations.\n\nThe paper states that the parameters \\theta and w of the deep kernel are task-independent, and the task-dependent parameters are kept separate. Where is the definition of the task-dependent parameters? The authors should make it clear.\n\nThe reviewer did not clearly found the novelty of this paper. How to transfer knowledge from source tasks (simply sharing the parameters of deep kernel?), and what is the difference to existing transfer learning works in the studied few-shot Bayesian optimization framework?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "K3Hdp5lKQwd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to simply learn/optimize the deep kernel parameters and hyperparameters of the GP using the data from all tasks (equation 9) and use such a deep GP kernel for BO. Instead of maximizing the log marginal likelihood of a single dataset (as is typically the case for a learning task), they propose to maximize the sum of log marginal likelihoods over the datasets of all tasks, which I view to be an incremental technical contribution. This paper is not about innovations in the acquisition function in BO. Their proposed approach outperforms the tested methods on 3 benchmark datasets.\n\nA drawback of their proposed method (equation 9) is that in contrast to some existing meta-BO algorithms, it cannot exploit the GP posterior means and variances when such information is available from previous BO tasks. \n\nConsidering the diversity of the types of datasets in the AdaBoost experiment, can the authors give an interpretation of the max likelihood estimates of theta and w in equations 8 and 9 in the context of this experiment?\n\nFor the GLMNet and SVM experiments, would it be possible to instead combine all the datasets over tasks and construct a *joint* likelihood over them in equation 9 instead of a sum of likelihoods over tasks? How would the results differ in this case?\n\nFor the experiments conducted, large amounts of data are drawn from many available previous BO tasks, especially for GLMNet and SVM. This seems to be in disagreement with the setting of BO where the unknown objective functions are expected to be costly to evaluate (e.g., in hyperparameter optimization). In the context of BO, it would be meaningful to consider the practical setting where only small amounts of data are available from a few expensive BO tasks. In this case, how would the proposed approach perform compared to the tested methods?\n\nAn empirical comparison with the state-of-the-art meta-BO algorithm: weighted GP ensembles (Feurer et al. 2018) should be included.\n\nFrom the results presented in Table 1 and Fig. 2, it does not seem like only a few shots/BO iterations are needed for the experiments performed in this paper. How do the results compare when only a few shots are used?\n\nEquations 9 and 10: Shouldn't the log marginal likelihood be over y's instead of f's?\n\nEquation 13: Exactly how is the loss function L(f^(t),X) defined? The author mentions that it is a normalized regret. The exact expression is needed here. In particular, I have noticed that f^(t) is not in bold: does this mean that only 1 \"test\" point is selected per task t?\n\nPage 6: Can the authors provide the exact details on \"the settings being sampled in proportion to their performance according to the predictions\"?\n\nThe authors can consider doing experiments on hyperparameter optimization of larger-scale CNNs, which is commonly seen in BO works.\n\n\nMinor issues\n\nPage 3: adaption or adaptation\n\nThe following reference would be relevant to the context of meta/transfer BO:\n\nZ. Dai, B. K. H. Low, and P. Jaillet (2020). Federated Bayesian optimization via Thompson sampling. In Proc. NeurIPS.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D3UHSebD7C", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Quality and Clarity**\n\n The authors clearly describe the problem and the proposed solution. The explanation in some parts can be improved (see Queries and Suggestions below) but overall the paper is well written.\n\n**Originality and Significance**\n\nWhile the problem of hyperparameter optimization is extremely important and well studied, the main contribution of this work appear to be some modifications to the transfer learning based approaches that leverage the performance of hyperparameter settings on related tasks to tune the hyperparameters for a new task. The techniques like Warm Start, Data Augmentation, and sharing of NN and GP parameters across all tasks are modifications that can potentially improve the practical performance of many approaches in this space and thus positioning this work as a set of techniques that can be applied to multiple models in this space might make it more impactful than the current approach of positioning it as a single model that can outperform other models.\n\n**Strengths**\n\n1. The idea of using task independent parameters for both the neural network and the GP appears to be novel and enables the model to learn from the data across all the old tasks and the initial data from the new task.\n\n2. The data augmentation and warm start approaches are intuitive and appear to give clear gains in practice.\n\n**Weaknesses**\n\n1. I do not think it is correct to apply mini-batch SGD to the task-wise losses in the manner of Algorithm 1, since the individual losses (GP log-likelihood) will not be additive over the samples of a given task and so the gradient estimates from the mini-batch would be biased. It might work in practice but I would appreciate some discussion/clarification on this in the paper.\n\n2. While using mini-batches is based on the rationale that the number of samples for each task might be large, the data augmentation approach proposed seems to be intended for the setting when \"only a few examples are available\". Thus it is not clear if the targeted setting involves few or many examples per task.\n\n3. The data augmentation and warm start approaches are largely heuristic and it is not clear whether they will always be beneficial.\n\n**Queries/Suggestions**\n\n1. Please explain the mutation and crossover operations in the Warm Start approach more clearly (preferably with an example). I am not entirely clear on what they are doing.\n\n2. Please provide a mathematical formulation of the normalized regret loss used to evaluate the methods in the Experiments.\n\n3. Can the Warm-Start and fine-tuning steps be performed with the ABLR baseline as they have been performed with the Multi-Head GP baseline? If so, please include those results as well.\n\n4. Please include some discussion/clarification on the points 1 and 2 under Weaknesses above.\n\n**Comments after Author Response**\n\nI thank the authors for their response. The explanation of the main approach has certainly improved and the details of task augmentation and warm start are much clearer now. I also appreciate the added discussion on bias in Stochastic Gradients for GP training. The reference cited does seem to indicate that the training will converge despite bias in gradients. The warm start and task augmentation approaches still seem a bit heuristic to me. The task augmentation approach seems to assume an inherent linearity in errors/noise for the metric of interest which may not hold in practice. The choice of loss function for warm start also seems rather arbitrary. It is not entirely clear why choosing $\\mathbf{X}^{\\text{(init)}}$ that minimizes (12) is a good idea. However as my main concern about the validity of SGD in training the model has been addressed and both the task augmentation and warm start approaches seem somewhat intuitive, at least at the idea level, I am increasing my score by one point.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4JW7Fv8o-S1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I appreciate the response to address my concerns carefully. My major concerns were the lack of novelty and some unclear descriptions on details in methods and experiments, but most of them have been well addressed. At first, I didn't see the technical challenges when applying DKL or any multi-task GPs into hyperparamter optimization. After reading the response and the manuscript again, I'm convinced the task augmentation plays a critical role in this setting. And, additional information from the revised manuscript helps to understand the details in method and experiments. So, I increased my score by two points.\n\n**Strengths**\nSolving few-shot regression tasks is interesting for warm-starting Bayesian optimization.\n\n**Weaknesses**\n1. Technical novelty seems to be weak. I have some concerns on the proposed approach.\n2. Compared to baselines, the experiments are not strong.\n3. The presentation needs to be improved.\n\n**Major comments**\n\n“A deep kernel ϕ is used to learn parameters across tasks such that all its parameters θ and w are task-independent. All task-dependent parameters are kept separate which allows to marginalize its corresponding variable out when solely optimizing for the task-independent parameters.” -> It didn’t clearly state which parameters are task-dependent and which are not. \\theta and w are the only learnable parameters by maximizing a marginal log-likelihood. So, which parameters are task-dependent?\n\nCompared to few-shot regression in a meta-learning framework, I didn’t find the benefits of the proposed approach. In addition, the proposed method doesn’t have a regularzing effects for few-shot tasks, since there is no meta-learning phase and no parameters shared across tasks.\n\n“In hyperparameter optimization, we are only interested in the hyperparameter setting that works best according to a predefined metric of interest. Therefore, only the ranking of the hyperparameter settings is important, not the actual value of the metric.” -> If we discretize all continuous hyperparameters, this argument makes sense. But, I couldn’t find the hyperparameter setting for AdaBoos, GLMNet, and SVM in the paper. How many discrete/continuous hyperparameters? And, how to determine the range for each hyperparameter? Did you follow the same setting in this paper? Initializing Bayesian Hyperparameter Optimization via Meta-Learning, M. Feurer et al. AAAI’15.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses deep kernel learning to develop a compelling framework for hyperparameter optimization in a few-shot setting, with empirically strong results. Please carefully account for all reviewer comments in the final version.

Decision: Accept (Poster)