Probabilistic Meta-Learning for Bayesian Optimization
Authors: Felix Berkenkamp
Anna Eivazi
Lukas Grossberger
Kathrin Skubch
Jonathan Spitz
Christian Daniel
Stefan Falkner
Venue:  (2021)
Keywords: meta-learning
bayesian optimization
probabilistic modelling

Abstract: Transfer and meta-learning algorithms leverage evaluations on related tasks in order to significantly speed up learning or optimization on a new problem. For applications that depend on uncertainty estimates, e.g., in Bayesian optimization, recent probabilistic approaches have shown good performance at test time, but either scale poorly with the number of data points or under-perform with little data on the test task. In this paper, we propose a novel approach to probabilistic transfer learning that uses a generative model for the underlying data distribution and simultaneously learns a latent feature distribution to represent unknown task properties. To enable fast and accurate inference at test-time, we introduce a novel meta-loss that structures the latent space to match the prior used for inference. Together, these contributions ensure that our probabilistic model exhibits high sample-efficiency and provides well-calibrated uncertainty estimates. We evaluate the proposed approach and compare its performance to probabilistic models from the literature on a set of Bayesian optimization transfer-learning tasks. 

at test time Under review as a conference paper at ICLR 2021
PROBABILISTIC META-LEARNING FOR BAYESIAN OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Transfer and meta-learning algorithms leverage evaluations related tasks in order to significantly speed up learning or optimization on a new problem. For applications that depend on uncertainty estimates, e.g., in Bayesian optimization, recent probabilistic approaches have shown good performance at test time, but either scale poorly with the number of data points or under-perform with little data on the test task. In this paper, we propose novel approach to probabilistic transfer learning that uses a generative model for the underlying data distribution and simultaneously learns latent feature distribution to represent unknown task properties. To enable fast and accurate inference at test-time, we introduce a novel meta-loss that structures the latent space match the prior used for inference. Together, these contributions ensure that our probabilistic model exhibits high sample-efficiency and provides well-calibrated uncertainty estimates. We evaluate the proposed approach and compare its performance to probabilistic models from the literature on a of Bayesian optimization transfer-leaming tasks.
INTRODUCTION
Bayesian optimization (BO) is arguably one of the most proven and widely used blackbox optimization frameworks for expensive functions (Shahriari et al., 2015) with applications that include materials design (Frazier Wang, 2016), reinforcement learning (Metzen et al., 2015), and automated machine learning (ML) (Hutter et al., 2019). In practical applications, BO is repeatedly used to solve variations of similar tasks. In these cases, the sample efficiency can be further increased by not starting the optimization from scratch, but rather leveraging previous runs to inform and accelerate the latest one.
Several approaches to this emerged under the name of transfer-learning (Weiss et al., 2016) and meta-learning (Vanschoren, 2018). Compared early work by Swersky et al. (2013); Golovin et al. (2017), recent publications leverage the representative flexibility of neural networks, which allows for powerful models and impressive results (Gordon et al., 2019; Rusu al., 2019; Garnelo et al., 201 8b;a; Zintgraf et al., 2019). Despite these significant advances, only a small subset of algorithms offers the well-calibrated uncertainty estimates which BO relies to guide its sampling strategy efficiently. Additionally, BO benefits greatly from a meaningful prior over tasks that quickly converges to the true function to provide the highest sample efficiency. Existing work mostly focuses on deterministic models and, for those providing uncertainty estimates, often challenge.
Contributions We set out to close this gap and introduce BAyesian optimization with Neural Networks and Embedding Reasoning (BaNNER), flexible meta-learning method for BO. We go beyond previous work of Perrone et al. (2018) and introduce generative regression model explicitly conditioned on low-dimensional latent representation for the tasks. This allows our model to (i) encode meaningful prior over tasks and (ii) remain highly sample-efficient, since each new task only requires inference over a low-dimensional latent representation. To ensure robust training of model, we introduce novel loss function to regularize the latent distribution and optimize model's 's hyper syper-parameters parameters using the available meta-data. We evaluate BaNNER on a set of synthetic benchmarks and two meta-learning problems and compare with the state-of-the-an in the literature.

---

Under review as a conference paper at ICLR 2021
Function valuos
f(x, T() go(x) (BaNNER)
Function valuie
f(x,T) g0(x) (BaNNER)
Parameters x
Parameters x
(a) Example tasks and meta-learned prior.
(b) Meta-learned posterior after two observations.
Figure 1: Example application BaNNER. Fig. la shows functions ((. based samples ~ p(T) for parameterized Forrester function together with the 2a confidence interval of the meta-leamed prior. In Fig. 1b we see the corresponding posterior distribution after two data points (blue circles) for specific test-function f(x, r). The confidence intervals contain the true function and collapse quickly, which enables highly-efficient Bayesian optimization. More plots in Fig. 6 (Appendix A.2).
PROBLEM STATEMENT AND BACKGROUND
Our goal is to efficiently optimize an unknown function f(x, r) over a domain x E X for some unknown but fixed task parameters that are sampled from unknown distribution To this end, at each iteration n we can select function parameters x,, and observe a noisy function value y/. = fx,, 7) Cn. with en drawn i.idd from some distribution p.. While our method can handle arbitrary noise distributions, we assume a Gaussian distribution, i.c. p. = =N o"), for the remainder of the paper.
p(T)-
We assume that each evaluation of f is expensive either in terms of monetary cost or time, so that we want to minimize the number of evaluations of f during the optimization process. The most data-efficient class of algorithms for this setting are Bayesian optimization (BO) algorithms, which use the observations collected up to iteration n, Dr = {x,, U: Y==1 order to infer a posterior belief over the function values f(x, -). To select parameters x,, that are informative about the optimum maxx (x, r). BO algorithms define an acquisition function ax-) that uses the posterior belief to select parameters as X,, argmax ((((f(x, D..).
While BO algorithms have been studied extensively, their performance crucially depends on the properties of the statistical model used for f- The two key requirements for BO algorithms to be data-efficient are i), that the prior belief over f concentrates quickly on the true function f as we observe data in D, and ii), that the posterior uncertainty estimates calibrated, that the model always considers the true function f to be statistically plausible. The latter requirement means that the true function f(, 7) must always be contained in the model's confidence intervals with high probability. Since the task parameters r are unknown, in general this requires a conservative model that works well for all possible tasks
We propose to use meta-leann metannee in order to learn an effective prior (Fig. la) that can quickly adapt to new task 7 (Fig. 1b). We are given data from T previous tasks T₁ p(T) with N, observations Dimeta {( (x,,,t, Vn. - each. We show the resulting generative model on the left in Fig. 2. Meta- learning aims to distill the information in Dmeta into model ga by optimizing the meta-parameters At test time, we then keep these parameters fixed and use the learned model Go to speed up the optimization of the new function f(,,T).
RELATED WORK
There are several approaches to improve the sample efficiency of BO methods based on information from related tasks. We refer to (Vanschoren, 2018) for broad review of meta-learning in the context of automated machine learning and focus on the most relevant approaches below.
One strategy to improve sample-efficiency is initialize the BO algorithm with high-quality query points. These initial configurations can be either constructed to be complementary (Feurer et al., 2014; 2015; Lindauer & Hutter, 2018) learned based on data set features (Kim et al., 2017). An
2

---

Under review as a conference paper at ICLR 2021
True, unknown generative model with f(x,7) 7)
Approximated
T
T
Zt
generative model with go (x, 2)
Meta- teta-training
Xn.
yrat
Xp,1
Yn,tt
N
N₂
Figure 2: Illustration of the generative models. We approximate the unknown function f(x, 7) with ge(x, z) by meta-learning the parameters based on the noisy observations f(x. r) + c. We regularize the model so that infer reliable uncertainty prediction for function [(, 7) through approximate inference on z in our model. Our method results in reliable confidence intervals able adapt quickly, see Fig.
alternative strategy is to transfer knowledge between tasks by adapting the acquisition function. These approaches balance exploration and exploitation by weighting the usefulness of given x not only on the current task, but also the related tasks. This trade-off can either be heuristically motivated (Wistuba et al., 2015; Wistuba et al., 2018), or learned directly (Volpp et al., 2020). All of these approaches change the BO algorithm, but not the underlying probabilistic model.
Most related to our work are methods that focus on alternative ways to model the observations. Several methods build global Gaussian process (GP) models across all tasks (Swersky al., 2013; Golovin et al., 2017; Marco et al., 2017; Feurer et al., 2018; Law et al., 2019), which requires approximations due the cubic scaling of GPs. A method to avoid this scaling is to use shared neural network to learn specialized features for Bayesian Linear Regression (BLR). The method, dubbed adaptive BLR (PerRoe al., 2018), learns a new BLR model for each task based on the shared neural network features. This allows ABLR to quickly adapt new tasks and scale better than GPs, but limits its predictive power for small number of test data points. We consider ABLR to be the state-of-the ooftthe-art art for our setting and use it as baseline in our experiments.
The problem of adapting a model to new problem is not unique to BO, but arises in other fields too. Recent progress in transfer learning (Weiss et al., 2016) and meta-learning Vanschoren, 2018) is concerned with the adaptation of a ML model from one or multiple related tasks to a new one. This transfer often focuses data-efficiency, which makes them similar BO. While Frameworks for general (probabilistic) model adaption (Finn et al., 2017: 7; 2018) exist, our contribution relates to a subset of approaches that directly model latent task distributions (Gordon et al., 2019; Rusu et al., 2019; Garnelo et al., 201 8b;a; Zintgraf et al., 2019). All of them model the relationship between tasks using the task specific variables, latent variables, and model that adapts to new tasks by using these latent variables (often as an input). Most of these meta- meta-learnnng algorithms do not consider active learning as an application and focus on deterministic models go- the following, we propose a method to meta-learn a probabilistic prior model that we can use together with approximate inference in order to estimate a posterior distribution for BO.
4 PROBABILISTIC META-LEARNING FOR BAYESIAN OPTIMIZATION
In this section, we discuss how to approximate f(x, r) with a generative model based on the metadata Dieta and how use this model infer predictions for BO on a new task, given test-data D,,- We use neural network go (x, z) to approximate the unknown function f(x, r). Next to the function inputs x, the network depends on trainable meta- peraamarsseterss 0 shared between all tasks, and an encoding x E 2 C Rdd of the task-specific parameters We refer to the latter as task embeddings in the following, since they represent unknown task parameters т. addition to being used as additional inputs to the network g, the latent parameter space Z can also include task-dependent network parameters, c.g., weights the output layer. Since we do not know 7 and p(T), we can not match to directly. Instead, assume fixed and known prior distribution p( Z turn go into generative network. Without loss of generality, we focus on a Gaussian prior p( 2 = (o, I) in the following. The resulting approximative generative model is shown the right in Fig. 2.
Meta training The goal meta-learning is to select appropriate global parameters so that for each task there exists corresponding latent encoding E 2 with f(x, r) go go(x,2 z).
3

---

Under review as a conference paper at ICLR 2021
Since have no information about we train separate task embedding z, for each task 7, by optimizing them jointly with the parameters 6. We can view the resulting model as a variational autoencoder, where directly optimize over the outputs 2, of the encoder, without committing to a specific parametric form. We can optimize the model's predictive performance on the meta-data by maximizing the log-likelihood [(x,,tz Vn,t; 0, 24) log Pe (Vv.e gx 2t)) of the parameters and observations for each of the I"" meta-tasks together with the corresponding task embedding:
T
max P.R1....
1
Σ (;0,); -
zr) ll)(
(1)
No
(x.D)
where we rescale the likelihood for each meta-task by N₁, the number of data points per task, to account for imbalanced data which skews the loss towards tasks with many evaluations. Without regularization, this model could easily overfit to each task by placing the task embeddings z, in disjoint areas of the domain. In that case, samples from the Gaussian prior p(2) would lead to function samples with low probability-mas under p(7). To avoid this, we introduce a regularization term R scaled by appropriate constant À E IR in order get a more uniform loss approximation over Z. We discuss the specific choice that we use to enable reliable and efficient inference below.
Inference Given the meta-traines generative model, we can make predictions about new task T() That is, after n 2 0 noisy observations of function values in D,, we can infer a posterior belief over the task embeddings, p(z D,,, f). While this is generally intractable analytically, approximate inference methods are generally reliable enough to make this tractable. For example, we use Hamiltonian Monte Carlo (HMC) (Neal, 2011) in our experiments. Given samples from this posterior belief, we obtain posterior belief over function values f(x, T) from a monte-carlo approximation of
p(go(x) Xn) go 2)) p(z p(z
D,,, 0) Oz Ez>p(z/D,..)) (90, ())
We can view this posterior inference as a task-specific adaptation of the parameters z in order to match g to f. In contrast deterministic meta-learning approaches, here the adaptation does not happen directly through an optimization process such as gradient descent, but through approximate Bayesian inference. This enables us to naturally capture uncertainty depending on the amount of data in D,, that we condition on.
Regularizing the Latent space We can only expect the inference over the predictive distribution in (2) to work well if the two generative models in Fig. 2 encode similar distributions over functions; that is, if fx, 7) with
p(T) has a similar distribution as go(x, z) with z ~ p(Z) for all X. This generally requires each sample z ~ p(Z) to be associated with a meaningful function ga (x, z). In this paper, we achieve this through regularizing z, to be close to the prior distribution p( 2 and regularizing the network go to be smooth. Regularizing p( 2) encourages a meaningful prior over functions where each sample has probability mass under P(T), while the regularization of g ensures that samples from the posterior (xg)(x) Dn, 0) vary smoothly across Z.
To regularize the task embeddings one might tempted to use the log-likelihood of the prior p( ()). However, Tolstikhin al. (2018) show that this can lead to poor inference results, since Gaussian prior regularizes ||z l|2. rather than covering the probability mass of p(Z) uniformly. Instead, Ghosh et al. (2020) propose to a deterministic regularizer during training and use density estimation to determine the prior p(2).
We propose novel regularizer that is both deterministic (does not require stochastic approximations) and regularizes the empirical distribution of z, to a Gaussian prior p( ) = N (O, I) a tractable way. We are inspired by the Kolmogoroy-Smimno test for one-dimensional distributions, which we use to compare the empirical cumulative distribution function (CDF) of the elements [2:]e in z, with the marginal CDFs over the dimensions of the prior p(Z). We show an example in Fig. 5 in the appendix. The empirical CDF is defined as F(z,d) d) = DT [[zz]a S 2). where I[2x]] S 2) is the indicator function that returns one if the ith component of z; is smaller equal than and zero otherwise. In addition to the marginals, we account for correlations by regularizing the empirical covariance matrix (oz((())... 2, to be close to that of p( 2), which is the identity matrix in
4

---

Under review as a conference paper at ICLR 2021
Algorithm Probabilistic Meta-learning for Bayesian Optimization
1: Given meta-data Dintaa for tasks =1,.... T
2: 0 < meta-train by minimizing loss (1) on the metadata
New, unknown task T P ). D1 +
4: for iteration n 1.... do
5: Approximate predictive distribution p(go(x) [D,x, in (2)
6: Xn. argmax (x) 0, Dn))
D Optimize BO acquisition function
7: Dn+1 f Dn (Xn, (xx,; 7) + En
setting. The resulting regularizer,
d R((Z)....,ZTT) [1 = (F(())) ((zi)))) + Xel Cov((z).... 1r
(3)
marginal CDF p(2)
second moment of p22)
trades off the loss for the empirical, marginal CDF of the task embeddings relative to the one for the second moment through scaling parameter Aç Unlike the original Kolmogorov-Smimtt uses the average distance between the CDFs the points z, We found this to lead to more stable training than the original formulation which uses the maximum over 2. In practice, many different tests for assessing multivariate normality exist (Korkmax al., 2014) and could be used instead of (3). For example, Marida's test (Mardia, 1970) also considers higher-order moments. However, we found it be computationally expensive relative the cheap (dd (log(T) + d) complexity of (3) and the latter was sufficient to achieve high performance in our experiments.
test, (3)
While the log-likelihood term C in (I) encourages a good fit for each meta-task and the regularization R in (3) forces the meta-task embeddings conform with the prior p(Z), we also have to ensure that go interpolates between the different task embeddings smoothly in order obtain meaningful functions go(- z) for p(z) that are not part of the tass-embeddinngs during training. A simple trick enforce smoothness regularizing the Hessian 0 go x, z)) 0zdz for Gaussian likelihoods Pe (Webb, 1994; Bishop, 1995; An, 1996).
add noise to the embeddings during training, which is equivalent to
Bayesian optimization algorithms can directly use the predictive distribution (2) in the acquisition function a in order to select informative parameters X,, Most acquisition functions only depend on the mean and variance of the predictions or samples from the posterior, all of which can be directly computed from (2). The overall algorithm is summarized in Algorithm 1: We first use the meta-data to minimize the regularized meta-loss in Line 2. In Line 3 get
task 7 and start without any test data D, = 0. After that, we proceed iteratively and approximate the predictive distribution, select new parameters
by maximizing BO's acquisition function, evaluate test-task I (Xn; T), and add the data point to our test data set.
Like most machine learning methods, BaNNER depends on hyperpa- rameters that include the regularization constants X, Ae. parameters of the BO acquisition function a, the amount of noise add for regularization, training-specific hyperparameters like learning-rates and batch-size sizes, and inference-specife hyperparameters such as the length and number of the chains that we sample from. In practice, these have to be selected based on the meta-data only, since we typically cannot generate additional meta-training data easily cheaply. For our experiments, we split the metadata into train and validation sets and select hyperparameters by comparing the average log likelihood of the trained models. For BO tasks, split each validation task randomly into data points that we condition on in (2) and points that we use to evaluate the likelibood.
Selecting hy hyperparameters
To efficiently optimize all these parameters we use BOHB (Falkner et al., 2018), highly parallel framework to optimize hyperparameters. To speed up computation, we use BOHB's s fultitfidelity capabilities and scale the number of meta-training iterations and validation tasks used to compute the validation loss with the fidelity. This allows us to quickly find promising regions of the hyperparameter search space and focus the compute resources there.
5

---

Under review as a conference paper at ICLR 2021
5 EMPIRICAL EVALUATION
In this section, we evaluate BaNNER in different scenarios and compare it to other methods. We focus specifically on low-dimensional problems where BO outperforms other global optimization methods. As benchmarks we designed five synthetic function ensembles from well-known benchmark function. These include functions where the optimum of tasks varies only locally and benchmarks where the minima vary over the entire domain that the model must adapt globally. Additionally we evaluate simulated meta- veeta-larnnng tasks, where the objective is to efficiently optimize the hyper- papeeeprsssserss of two machine learning algorithms across different data sets.
We use two variants of BaNNER. One that only considers task embeddings as input and one that additionally considers BLR over the last layer, dubbed BaNNER-BLE in the following. We compare both to several baselines: Random search is a simple yet often surprisingly competitive baseline in the hyperparameter optimization settings (Bergstra & Bengio. 2012: Li al., 2017). We also consider Gaussian process based BO (GPBO), which is the de-facto default for low-dimensional problems in the BO community. Lastly, we compare to ABLR by Perrone et al. (2018), which we see as the state-of-the-arr for our problem setting. We do not consider approaches like stacked GPs (Golovin et al., 2017) and weighted GP ensembles (Feurer al., 2018) that do not scale to the large meta-data sets we consider.
We implement ABLR and BaNNER PyTorch (Paszke et al., 2019) and use the open source Gaussian Process implementation from Emukit for GPBO (Paleyes et al., 2019). The latter is modular BO framework, which we use to evaluate and compare all methods. All experiments use the default BO parameters with expected improvement (EI) as the acquisition function. We will make all model implementations and code to reproduce the experiments presented below openly available after acceptance of the paper.
5.1 SYNTHETIC FUNCTION ENSEMBLES
We derive all ensembles of function in this section from well-known functions by replacing parameters of the function with distributions over them. These include one-diment me-dimensional quadratic functions and an ensemble of Forrester functions (Forrester et al., 2008), as well as extended multi-dimmesional functions Branin, Hartmann3 and Hartmannó (Dixon & Szego, 1978). We provide details about these functions, their parameter distributions, and the meta data in Appendix B.
The Forrester ensemble shown in our illustrative example in Fig. represents the easiest benchmark. The provided meta data covers both X' and T well and the optimum often falls into relatively small region of the search space. As one would expect, the results in the left panel of Fig. 3 show that BO benefits strongly form any meta-learning model. Both variants of BaNNER start with already low regret and keep improving, while ABLR starts consistently worse than random sampling, but improves more quickly than GPBO. The reason for this is that ABLR starts with an uninformed BLR layer that has symmetric prior over all weights. Since the meta-data is normalized, this means that the mean prediction of ABLR without tests points is equal to the mean of the meta data. Since the weight prior is uniform, the predictive variance is large whenever the neural network features attain large absolute values, which occurs at points where the training data is most extreme. For Forrester, this coincides with the large function values at x = 1, which is not close the optimum.
Our second example consists of one-dimensional quadratic functions. For this benchmark, we provide less meta data per task and allow shifts, so that the minimum of the function can vary over the domain. Methods that quickly estimate the overall global shape of the function can expect small regret. We show results in the right panel of Fig. for 256 different runs. We can see that both variants of BaNNER achieve the lowest mean regret. Surprisingly, ABLR performs worse than random search, indicating that the small number of meta-data points did not allow ABLR to learn representative features, even though the network is capable of approximating the optimal features 1, and x² casily. We suspect that the regularization in BaNNER helped mitigate these problems.
The third ensemble based on the two-dimensional Branin function, has true latent dimensionality six, T E Rd. The results in the bottom-left panel in Fig. 3 show a similar picture to the two benchmarks above: Both variants of BaNNER perform best from the beginning and GPBO eventually closes the gap. ABLR's 's initial evaluation are poor, but it quickly improves to outperform Random Search and GPBO for the first 16 iterations, at which point it stagnates without further improvement.

---

Under review as a conference paper at ICLR 2021
Forrester ensemble
Quadratic ensemble
10
10
Simple regret
Simple regret.
100
10
10
10"3
10
10-5
10
12
14
16
2
8
Iteration
Iteration
Branin ensemble
Hardmann6 ensemble
10a
10
Simple regret
Simple regret.
10
10
10
10
10 -3
10 2
4
8
12
16 Iteration
20
24
28
32
32
64
96
128
Iteration
Random search
GPBO
ABLR
BaNNER
BaNNER-BLR
Figure 3: Performance on two synthetic function ensembles. All show the mcan + the standard error of the mean (95% confidence) for 256 independent evaluations. For the Forrester ensemble, strong prior without any test data leads to strong performance right from the start for both variants of our method. For the quadratic functions, the strength of the meta-models lies in knowing the global shape of the functions, rather than knowing a-priory the location of the optimum. ABLR failed to learn useful features from the provided meta-data, possibly due the wer-parametrized network and more broadly distributed y values. With growing complexity and smaller ratio of number of points per task per dimension, the performance of the meta-lcarning models degrades and they fail to improve over time, especially for Hartmann6. In all cases, the GPBO method eventually catches up and usually outperforms the other methods, but meta- meaanneg leads to strong early performance in all cases. Additional plots including also ensemble of Hartmann3 functions be found in Appendix A.
Our last two synthetic ensembles are based on the Hartmann3 and Hartmann6 functions, where we deliberately deliberately does not contain near-optimal points for teatattaaiiing. All meta-learning algorithms perform similarly (see Fig. 3 for Hartmannó and Appendix A for Hartmann3): After short improvement, BaNNER and ABLR stagnate and GPBO outperforms them. We attribute this to the growing complexity of the functions in terms of the search space size and the size of the training data set, which does not allow the models to adapt to test points beyond certain accuracy. For Hartmann6, all three meta-learning algorithms exhibit nearly the same performance, which could indicate that the type of probabilistic model becomes less relevant, if the meta-model lacks the necessary precision around the optimum.
In summary, the synthetic experiments demonstrate the potential improvements a probabilistic meta-model in BO and also explored its limitations when the meta data does not cover the minima sufficiently. We would like to highlight that in the case of good coverage of both, t and æ, the additional task parameters in the BLR layer of BaNNER-BI BLR did provide any benefit.
5.2 META-LEARNING SURROGATE BENCHMARKS
We now consider a more practical meta-learning application: tuning the hyperparameters of ML. algorithms across different data sets. this setting, f((x,7) represents the performance of an algorithm on specific dataset that has unknown properties algorithms directly, we use HPOlib2 (Eggensperger et al., 2013), a library dedicated to the evaluation of hyperparameter optimization algorithms. HPOlib2 replaces the costly training step by cheap lookup based on a large number of hyperparameter evaluations.
Instead of optimizing the ML
We evaluate method and the baselines two benchmarks: The training of GLMNET (Genaralized Linear Models with elastic NET regularization) (Friedman et al., 2010) and Ranger, a random forest implementation, (Wright & Ziegler, 2017) and their evaluations on 37 data Setn(Kühn al., (0)8)..
7

---

Under review as a conference paper at ICLR 2021
GLMNETI471
Ranger 1489
10
10
Simple regret
Simple regret
10 -
10
10-2
10
10
10
12
16
32
64
96
128
Ieration
Ileration
Random search
CPBO
ABLR
BaNNER
BaNNER-BLR
Figure 4: Performance on two surrogate meta-lcarning problems. Both show the mean + the standard error of the mean (95% confidence). We ran every method 256 times with different seeds.
Both of the benchmarks have data sets available on OpenML (Van Rijn et al.. 2013) and the collected evaluation data is licensed under creative commons (Kühn al., 2018). Moreover, the two algorithms are well understood MI. methods that usually benefit from optimizing their hyperparameters. We provide more details in Appendix C.
For our experiments, we randomly split the data sets into 32 meta-data sets and 5 test data sets. We refer to these benchmarks as, e.g., GLMNET335 or Ranger| 487, where the id corresponds to the data set that the method was trained on. We refer to (Kühn et al., 2018b) details about the data sets and their properties. For the meta training data, we sampled 128, and 512 points for GLMNET and Ranger respectively using the surrogate model. Instead of validating on unseen tasks with unseen hyperparameter settings, we reuse the training tasks, but with an unseen set of x. In this scheme, the hyperparameter optimization tuning BaNNER's parameters cannot estimate the generalization to new tasks. We do this to avoid a computationally expensive cross-valie tross-validation scheme that would require several fits of the meta-learning models. Our results verfy this simplified validation in our scenario, suggesting that the test tasks are sufficiently similar to the training tasks.
Figure 4 shows representative results one of data sets per ML method. We see strong performance on the very first iterations for BaNNER and BaNNER- -LR compared to random sampling. The performance of ABLR's initial guess fluctuates depending the data set. We attribute this again to the uninformed BLR layer at the beginning, as discussed above. The plots for all data sets can be found in Appendix A. Notably, in this scenario BaNNER-BLE seems outperform BaNNER. We suspect that the good coverage in x, but the rather poor coverage of 7 (due to the small number of datatsets) could be an explanation. Another difference to the experiments in Section 5.2 lies in the inherent noise of the evaluations, which could also be more effectively countered by adapting the output layer of the network rather than the embedding.
Besides this, the results look similar to the ones above: our method consistently learns better prior for the first iterations and usually adapts quickly to competitive regret values. Unlike above, GPBO does not perform much better in the shown number of iterations then the other methods. There are few examples (GLMNET148% and Ranger1485) where ABLR has the best initial guess and performs the best. We can only speculate for the reasons, but want to point out that 1485 is quite large data set with many features compared to the meta training data sets, and 1489 is slightly more imbalanced then the average. Data set 1504 turned out to be easy and all methods, even Random Search, perfeetly tune both models within a few iterations.
6 CONCLUSION
We have presented BaNNER, a novel approach to probabilistic meta-lcarning by training a generative model that mimics the data generation process. Our meta-loss contains regularization inspired by statistical tests and enables training the model deterministically while still allowing for efficient inference based on multivariate normal prior. In our experiments we use BaNNER as a model in Bayesian optimization (BO) and on meta-learning tasks, which demonstrate higher sample-efficiency than both standard BO with Gaussian processes and ABLR, the ofate-of-the-aa probabilistic model for scalable transfer- transfer-learming learning in BO.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Guozhong An. The effects of adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643-674, 1996.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13((6b):281-30)) 2012.
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7 (1):108-116, 1995.
L.C.W. Dixon and G.P. Szego. The global optimisation problem: introduction. In L.C.W. Dixon and G.P. Szego (eds.), Towards Global Optimisation 2, pp. 1-15, North-Holland Pub, New York, 1978.
Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger H Hoos, and Kevin Leyton-brown. Towards an empirical foundation for assessing bayesian optimization of hyperparameters. In In NeurIPS Workshop on Bayesian Optimization Theory and Practice. Citeseer, 2013.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter opti- mization at scale. International Conference on Machine Learning, pp. 1437-1446, 2018.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Using meta-leamiin initialize bayesian optimization of hyperparameters. In Proceedings of the 2014 International Conference on Meta-learning and Algorithm Selection-Voluee 1201, pp. 3-10. Citeseer, 2014.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in neural information processing systems, pp. 2962-2970, 2015.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian optimiza- tion. arXiv preprint arXiv: 802.02219, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pp. 9516-9527, 2018.
Alexander Forrester, Andras Sobester, and Andy Keane. Engineering design via surrogate modelling: practical guide. John Wiley & Sons, 2008.
Peter Frazier and Jialei Wang. Bayesian optimization for materials design. In Information Science for Materials Discovery and Design, pp. 45-75. Springer, 2016.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1-22, 2010. URL
http://www. jot.ttttttttog/oooooooot. org.v33/// /10//.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In 2018a.
International Conference on Machine Learning. pp.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee whye Teh. Neural processes. arXiv preprint arXiv: 2018b.
P. Ghosh, M. S. M. Sajjadi, A. Vergari, M. J. Black. and B. Schölkopf. From variational determin- istic autoencoders. In 8th International Conference on Learning Representations (ICLR), April 2020. *equal contribution.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google vizier: service for black-box optimization. In Proceedings of the 23nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 1487-1495, 2017.

---

Under review as a conference paper at ICLR 2021
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Tumer. Meta- learning probabilistic inference for prediction. In International Conference on Learning Represen- tations, 2019. URL https: / /wwpnreviewwww..eee/eeee net/ 2tt.:e/wwwweeeeeeeoouuummmmee wtw.:/oondddddlldtooo..
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated Machine Learning. Springer, 2019.
Jungtaek Kim, Saehoon Kim, and Seungjin Choi. Learning to warm-start bayesian hyperparameter optimization. arXiv preprint arXiv:1710.062)) 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Selcuk Korkmax, Dincer Goksuluk, and Gokmen Zarasiz. MVN: An R package for assessing multivariate normality. The R Journal, 6(2):151-122, 12 2014. ISSN 2073-4859.
Daniel Kühn, Philipp Probst, Janek Thomas, and Bernd Bischl. Automatic exploration of machine learning experiments on openml. ArXiv, abs/1806. 10961, 201 8a.
Daniel Kühn, Philipp Probst, Janek Thomas, and Bernd Bischl. Automatic exploration of machine learning experiments on openml. arXiv preprint arXiv: /806. /0961, 2018b.
Daniel Kühn, Philipp Probst, Janek Thomas, and Bernd Bischl. OpenML. R Bot Benchmark Data (final subset). figshare, 3 2018. doi: @Bot__enchhark-
figgsaree.5822230.2222 URL httpttts/s//fssheeeeee
Ho Chung Law, Peilin Zhao, Leung Sing Chan, Junzhou Huang, and Dino Sejdinovic. Hyperparameter learning via distributional transfer. In Advances in Neural Information Processing Systems, pp. 6801-6812 12, 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-6866 2017.
Marius Lindauer and Frank Hutter. Warmstarting of model-based algorithm configuration. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Alonso Marco, Felix Berkenkamp, Philipp Hennig, Angela P. Schoellig, Andreas Krause, Stefan Schaal, and Sebastian Trimpe. Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with Bayesian optimization. In Proc. of the International Conference on Robotics and Automation (ICRA), pp. 1557-1563, 2017.
K. V. Mardia. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3):519 -30,0,,,,,,,,,, 12 1970. ISSN 0006-3444. doi: 10.1093/bioee/57.3.519 URL https: //doi. org/10gg///ooo 10. 1093/09///// //biomet///////e /57.3.599.
Jan Hendrik Metzen, Alexander Fabisch, and Jonas Hansen. Bayesian optimization for contextual policy search. In Proceedings of the Second Machine Learning in Planning and Control of Robot Motion Workshop. IROS Hamburg, 2015.
Radford M Neal. Meme using hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, pp. 113, 2011.
Andrei Paleyes, Mark Pullin, Maren Mahsereci, Neil Lawrence, and Javier González. Emulation of physical processes with emukit. In Second Workshop on Machine Learning and the Physical Sciences, NeurIPS, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems 32, pp. 0024-8035. Curran Associates, Inc., 2019. URL 9015-pytorch an- www.orcc-aaa--mprattie-e-eeeeeee htppppaciii.e.tt.eee..ooooorrrre deep hww.r/orraanreeeeee-eeeaaann-nn- pdf.
neurips cc/ hatp//ape////////e
10

---

Under review as a conference paper at ICLR 2021
Valerio Perrone, Rodolphe Jenatton, Matthias W Seeger, and Cedric Archambeau. Scalable hyperpa- rameter transfer learning. Advances in Neural Information Processing Systems, pp. 6845 6855, 2018.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Confer- ence on Learning Representations, 2019. URL https: http////pppp/eeeeeeeeeee/. BJgk1hAcK7.
/ net forum?ice
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 175, 2015.
Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task bayesian optimization. In J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 2004-2012. Curran Associates, Inc., 2013. URL http:// pdf.
papers. nips wwtp:ccpppppppppeeeeeeeeeeeeeeee wttppppeeeeemmtttttaaaaaaaaaeeee
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto auto-encoders. In International Conference on Learning Representations, 2018. httttstt:////eeeeeeee net/ / Corrmmidd-HL1Pnn 0b.
Jan N Van Rijn, Bernd Bischl, Luis Torgo, Bo Gao, Venkatesh Umaashankar, Simon Fischer, Patrick Winter, Bernd Wiswedel, Michael R Berthold, and Joaquin Vanschoren. Openml: A collaborative science platform. In Joint european conference on machine learning and knowledge discovery in databases, pp. 645-649. Springer, 2013.
Joaquin Vanschoren. Meta-learning: A survey. arXiv preprint arXiv:1810.03544. 2018.
Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning bayesian optimization. In International Conference Learning Representations, 2020. URL https: opwp/eeiee......eeeeeee/e. net. feot/form?? id=rydeeJSKKN
Andrew R Webb. Functional approximation by feed-forward networks: a least-squares approach to generalization. IEEE transactions on Neural Networks, 5(3):363-371, 1994.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):9 2016.
M. Wistuba, N. Schilling, and L. Schmidt-Thieme. Learning hyperparameter optimization initializa- tions. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 1-10, Oct 2015. doi: 10.1109/DSAA.2215.774555
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thient Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Machine Learning, 107(():43-78, 2018.
Marvin N. Wright and Andreas Ziegler. ranger: A fast implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, 77(1):1-17, 2017. doi: 10. (0.18637/jss.v077)
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pp. 7693-7702, 2019.
11


Reviews:
[{"review_id": "tl56YgVksmy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce a technique for multitask Bayesian optimization based on meta learning. In general, there are two broad approaches to multi task learning in Bayesian optimization: sharing information between tasks by modelling the correlation between tasks (e.g., Swersky et al 2013), and sharing information between tasks via global weight sharing across over some embedding (e.g., Perrone et al., 2018). This paper falls in to the latter approach, learning a meta learning style task embedding (eqn. 1) to be able to make predictions over a new task given auxiliary task data (eqn. 2).\n\nOverall, my biggest concern is whether the empirical comparison is as well set up as it could be. In particular, this paper compares primarily to ABLR. However, ABLR avoids comparing to standard multitask Bayesian optimization primarily for scalability reasons, using pooled datasets with tens or hundreds of thousands of examples for transfer learning. In this paper, as far as I can tell, each individual task in section 5.2 is limited to 128 and 512 examples respectively. The total dataset size here is significantly smaller, which (1) does not match the setting ABLR was originally used for, and more importantly (2) is well within reach of multitask BO.\n\nFirst, before jumping in to specific details on this, let me admit outright that some details of the experimental setup in Section 5.2 are simply not clear to me. You say \"For the meta training data, we sampled 128, and 512 points for GLMNET and Ranger respectively using the surrogate model.\" Does this mean you had 32 (number of meta datasets) * (128 data points) =4096 total hyperparameter observations before starting observation, or 128 total? I'm assuming the former? Or did you use the full set of hyperparameter evaluations? I also wasn't clear on what was meant by \"Instead of validating on unseen tasks with unseen\nhyperparameter settings, we reuse the training tasks, but with an unseen set of x.\" Does this refer just to the mechanism for setting BaNNER hyperparameters and not to how test tasks / data sets were selected, or were training tasks actually reused as test tasks?\n\nIf my assumptions about the empirical set up of the paper are correct (e.g., 4096 and 16384 meta examples), then my biggest issue is whether the playing field that ABLR was compared to was level. Part of the reason ABLR avoids comparing to the \"directly learn correlations across tasks\" style of BayesOpt that uses multitask kernels is an emphasis on scalability. In their OpenML tasks for warm start on SVM and XGBoost hyperparameter optimization, their meta dataset contains 6.5x10^4 and 5.9x10^5 data points respectively. In this paper, it seems as though the meta datasets are order(s) of magnitude smaller. This may not be the most efficient regime for ABLR, which can rely on larger feature extractors with more total pooled data.\n\nNow, there's an argument to be made for a data efficiency improvement over ABLR (requiring less overall pooled data), but in these settings with on the order of thousands of pooled data points, it becomes trickier in my opinion to justify not comparing to standard multitask Bayesian optimization using multitask Gaussian processes, which works quite well in the data sparse regime and will have no trouble scaling to 32 tasks with 128-512 examples each.\n\nSome of the standard error plotting (Forrester ensemble, GLMNET1471 in the main text, much of figure 8 in the supplementary materials) is clearly broken for some methods. Is this a bug in plotting or actually large standard errors spanning multiple orders of magnitude despite being over 256 trials? Additionally, the y axes on some plots (e.g., Figure 8 bottom row, Figure 9 bottom row) do not have appropriate scales.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VURK2CDdWwk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes BaNNER, a new transfer learning approach to accelerate hyperparameter optimization based on related tasks. This is based on a generative model learning the data and task distributions. A number of experiments against standard BO and random search, as well as against ABLR, the most closely related transfer learning baseline, indicate that the proposed approach tends to find a good hyperparameter configuration more quickly.\n\nPositives \n\n1. **Significance.** The paper tackles the cold-start problem in BO by leveraging related tasks. This is a highly relevant problem that fits an active line of research. The paper makes a step forward in this direction. \n\n2. **Reproducibility.** The experimental evaluation is very well detailed, with clearly defined error bars and a large number of repetitions (i.e., 256). Implementations of the models and code to reproduce the experiments will also be made evaluable, which will further foster reproducibility. \n\n3. **Clarity**. The paper is generally well-motivated, clear and easy to follow. \n\nNegatives\n\n1. **Missing baselines.** The key concern with the current version of the paper is that more transfer learning baselines would be needed to fully justify the proposed method. Beyond ABLR, a number of additional follow-up and prior works have addressed the transfer learning problem. Many of these are indeed referenced in the related work, such as the initialization strategies to warm-start BO. It is true that these are different as they change the BO algorithm and not the underlying probabilistic model. Nonetheless, this is not a real reason not to compare against at least some of them. Without these comparisons, it is not clear if BaNNER should be the transfer learning method of choice, considering the many alternatives. One could argue that the proposed approach and warm-starting could be seen as complementary, but the authors would need to show in experiments whether there are mutual benefits from combining them or not. The same considerations hold for the following transfer learning HPO methods which prune or construct a search space based on previous related tasks:   \n[Wistuba, et al.: Hyperparameter search space pruning–a new component for sequential model-based hyperparameter optimization. In Machine Learning and Knowledge Discovery in Databases, 2015]\n[Perrone, et al.: Learning Search spaces for Bayesian optimization: another view of hyperparameter transfer learning. In NeurIPS, 2019.  ]\n\nThere are also some more recent closely related baselines that are not compared against nor discussed in the related work. For instance, the following work achieves transfer learning while scaling well in the number of hyperparameter evaluations:  \n[Salinas, et al.: A Quantile-based Approach for Hyperparameter Transfer Learning. In ICML, 2020.]\n\n  Lastly, while it is true that approaches like stacked GPs (Golovin et al., 2017) and weighted GP ensembles (Feurer et al., 2018) do not scale well to large datasets, they scale cubically only in the number of evaluations per task (as opposed to the total number of evaluations across tasks). In the considered experiments this is not necessarily unfeasible. To be comprehensive, one could also subsample the number of data points per task to make these baselines scale and be able to compare against them. It might be the case the using more datapoints will help BaNNER, but this needs to be shown; otherwise, the advantage of scaling to a large number of past evaluations remains theoretical but not demonstrated in practice. Varying the number of evaluations per task could also be seen as a useful ablation study (point 3 below).\n\n2. **Non-challenging problems**. Apart from synthetic examples, the authors consider tuning generalized linear models and random forests with a low-dimensional hyperparameter space. While these are insightful experiments, most approaches for transfer learning in BO are also evaluated on more high-dimensional and ambitious tuning problems, including neural networks tuning/NAS problems. \n\n3. **Not enough ablation studies**. The paper focuses on useful performance comparisons but does not have enough ablation studies. These would give further insights into the proposed method. For instance, it is not clear how performance is affected by changing the number of hyperparameter evaluations available from related tasks. Also, it is not clear how robust BaNNER is to noisy or unrelated tasks. Without these empirical studies, one only gets a shallow understanding of the proposed approach.\n\n\nOther comments\n\na. Figure 4 shows some specific datasets per ML method. I appreciate that all results are reported in the appendix. However, ABLR outperforms BaNNER on some datasets and several other datasets lead to very noisy results. To give a sense of overall performance, I would advise the authors to also report global metrics. One possibility is ranking different methods on each dataset and averaging the ranks (to account for heterogeneous loss values).\n\nb. The authors report that ABLR performs best on a few example problems, such as GLMNET1489 and Ranger1485, and conjecture that this is because 1485 is a large dataset with many features and 1489 is more imbalanced. It is not clear to me why dataset dimension and class imbalance should favor ABLR over BaNNER. \n\nc. The method comes with a number of hyper-hyperparameters, which the authors learn on meta-data by running an additional BO loop powered by BOHB. From the perspective of deploying BaNNER, is this not too cumbersome in practice? Do the authors recommend a set of (robust) defaults for their method to be used off-the-shelf or is an additional BO loop always required?\n\nOverall, I vote for rejection for the current version of the paper. The proposed idea is sound, promising and well-motivated, but the method is evaluated against a very limited selection of baselines and it lacks ablation studies. It is not clear with the current experiments how the BaNNER's performance is placed within the context of general transfer learning methods in the literature. I am ready to change my score if the authors compare to a wider range of baselines.\n\nMinor:\n1. Some in-line references are not in the correct format (e.g., \"We refer to (Vanschoren, 2018)\").\n2. Typos in this sentence at page 3: \"where we deliberately deliberately does not contain near-optimal points for meta-training\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "RzbQgK-_I2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY OF REVIEW\n\nThis paper proposes a probabilistic meta-learning algorithm to establish an informative prior to be used for Bayesian optimization (BO). \n\nThe technical contribution lies in the development of the probabilistic meta-learning algorithm instead of innovations made to BO's acquisition function: nearly the entire technical formulation in Section 4 pertains to meta-learning instead of BO. One may argue that this is a probabilistic meta-learning paper in \"disguise\".\n\nIf the aim of this paper is to set up an informative prior, it would be necessary to produce empirical comparisons with the state-of-the-art probabilistic meta-learning algorithms in terms of probabilistic predictions, which is lacking in this paper. In this regard, it would be necessary to perform experiments on higher-dimensional tasks/problems.\n\nOne drawback of their proposed work in the context of BO appears to be that unlike some existing meta-BO algorithms, it cannot exploit the GP posterior means and variances when made available from previous BO tasks due to equation 1.\n\nThere are a number of major concerns about the technical formulation and the experimental setup and results, as detailed below.\n\n\nDETAILED COMMENTS\n\nPage 3: The authors say that \"Most of these meta-learning algorithms do not consider active learning as an application and focus on deterministic models g_theta.\" Exactly which probabilistic meta-learning algorithms in the same paragraph as this claim cannot be used for providing a probabilistic prior? Do existing probabilistic meta-learning algorithms not satisfy the requirements of data-efficiency and well-calibrated uncertainty estimates?\n\nPages 3 & 4: The authors have placed a prior on latent task embedding z but did not seem to consistently apply the Bayesian treatment on z: The task embedding z_t for each task tau_t is optimized using equation 1 without a Bayesian treatment, while the inference or adaptation (equation 2) applies the Bayesian treatment on z. Can the authors discuss and justify this inconsistency? How does it affect the choice of the optimized theta?\n\nEquation 2: If g_theta approximates f, then should the posterior belief of g_theta(x) be an expectation of g_theta(x,z)? Furthermore, g_theta(x) is not defined.\n\nPage 5: The authors say that \"we use BOHB's multi-fidelity capabilities and scale the number of meta-training iterations and validation tasks used to compute the validation loss with the fidelity.\" How is the fidelity exactly determined? Are all the hyperparameters optimized prior to BO? Some hyperparameters need to updated as more BO samples are gathered in every BO iteration. How are these dealt with?\n\nPage 6: The authors say that \"We do not consider approaches like stacked GPs (Golovin et al., 2017) and weighted GP ensembles (Feurer et al., 2018) that do not scale to the large meta-data sets we consider.\" This lack of comparison, especially with the weighted GP ensembles approach, is not well-justified in the context of BO where it is costly to evaluate the unknown objective functions and hence we do not expect large number of samples. The authors are encouraged to perform this empirical comparison.\n\nPage 7: What is the reason for introducing BaNNER-BLR and when would BaNNER-BLR be expected to perform better? The authors have discussed this only briefly in the second last paragraph of Section 5.2.\n\nFigure 3: The fundamental difference between Algorithm 1 vs. GPBO lies in the former exploiting an informative prior based on probabilistic meta-learning (albeit not containing near-optimal points for meta-training) while the latter utilizes an uninformative prior. It is surprising to see that their proposed Algorithm 1 is not being able to undo the effects of the informative prior and hence performing worse than GPBO with an uninformative prior. The authors' justification in the second last paragraph of Section 5.1 was not able to allay my above concern. The use of an informative prior hurts in this case.\n\nAppendix B: The authors seem to have used a large number of meta tasks T=256 and for some synthetic ensembles, a large number of datapoints N_t=1024 per task. In meta-learning, this seems fine. But, in BO tasks where functions are expected to be expensive to evaluate (Section 2), these numbers do not seem practical or reasonable. Can the authors present results with varying, smaller T and N_t?\n\nThe authors should consider performing empirical evaluation on hyperparameter optimization of the larger-scale CNNs, which is commonly seen in BO works.\n\n\nMinor quibbles\n\nPage 2: The *most* data-efficient class of algorithms for this setting are Bayesian optimization (BO) algorithms... Isn't this too strong a claim to make?\n\nPage 3: adaption\n\nPage 6: form\n\nFigure 3: Hardmann, Ouadratic\n\nPage 7: deliberately deliberately \n\nPage 8: verfy\n\nPage 8: datatsets", "extra_scores": {}, "extra_text": {}}, {"review_id": "6KOrxTefImZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper, a probabilistic method is proposed for Bayesian optimization transfer learning (or meta-learning). A latent representation is modeled separately for each task. The surrogate used in BO will be approximated by a neural network with its input being the test point and the latent representation of the task. The meta-training procedure optimizes the prediction loss over the neural network weight and the latent representations of the source tasks. Additional regularizer is added to induce smooth latent representations. For a new task, predictions are made on the new data point using the surrogate model g_\\theta(x,z) with a fully Bayesian treatment. The p(z|D,\\theta) is approximated by Hamiltonian Monte Carlo. In the experiments, the proposed method is compared to random search, GP-BO, and adaptive Bayesian linear regression. \n\nThe latent representation here could be regarded as some model hyperparameter for the neural surrogate, and the p(z|D,\\theta) adaptively changes with the including of new data points into the dataset D. The meta-training procedure resembles that of VAE with z being something that can be directly optimized. The experiment shows that the proposed method performs better than the baselines. \n\nMy biggest concern is the novelty of the proposed probabilistic approach. It appears that the framework is very similar to that of Neural processes where the latent representation is conditioned on the observations and the prediction is conditioned on the latent representation and the new input data. I think a very clear distinction between this method and Neural processes needs to be made. To me, the only difference is that the method in this paper proposes to directly optimize over z instead of over a variational posterior q(z|D), which is adopted from Ghosh et al. The Neural processes method is directly applicable to the BO problem, contrary to what it is claimed by the authors as “most of these meta-learning algorithms do not consider active learning as an application”. Therefore, NP should serve as the most relevant baseline to compare with in terms of experiments and methodology. \n\nRegarding the experiments, I am curious why the proposed method has a very large variance on the most simple task Forrester. Also, I am not sure why the initial performance of ABLR is even worse than GP-BO, which means meta-learning hurts its performance initially. Some explanation or ablations studies would be helpful here. One motivation of the work is faster transfer learning for BO by using the neural model. However, there is no runtime comparison with the baselines. \nOverall, the paper is easy to follow but lacks a direct comparison with NP which most resembles the proposed method. The experiments would be more convincing if more baselines (such as the NPs) could be included. \n\nTypo: Sec 5.1, strongly \"form\" any meta-learning model --> from\n\n-------------After author's response--------------\n\nAs the other reviewers also pointed out, some baselines are missing and ablation studies on meta-data are missing as well. Meanwhile, my concern is that the presented idea in the paper looks very similar to that of the neural process, except the paper is optimizing over a point estimate of z while NP is optimizing over q(z|D). This remains unexplained in the author's response. Therefore I am keeping my original evaluation at the moment.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propose an approach to probabilistic meta-learning for Bayesian optimization. The goal is to accelerate Bayesian optimization under the assumption that multiple related tasks require optimization. In terms of strengths, the paper addresses an important problem as it has applications to efficiently optimizing hyper-parameters over multiple related data sets or models. In terms of weaknesses, the proposed approach is closely related to neural processes, but this connection was not made in the original paper. The authors were unfortunately not able to provide additional insights or results regarding this point during the limited discussion period and as a result, the novelty of the method at the core of this approach is in question. The reviewers also note that the experimental designs and comparisons performed are limited. For some smaller problems, more standard baseline methods like multitask GPs should be applied. The authors have also not compared to a number of other recent methods aimed at scalable transfer learning for hyper-parameter optimization, as detailed in the comments of Reviewer 1. The reviewers agree that the paper is not yet ready for publication.
Decision: Reject