Automatic prior selection for meta Bayesian optimization with a case study on tuning deep neural network optimizers
Authors: Zi Wang
George Edward Dahl
Kevin Swersky
Chansoo Lee
Zelda E Mariet
Zachary Nado
Justin Gilmer
Jasper Snoek
Zoubin Ghahramani
Venue: ICLR 2022 Submitted (2022)
Keywords: Bayesian optimization
Gaussian process
hyperparameter tuning
meta learning
transfer learning
multi task

Abstract: The performance of deep neural networks can be highly sensitive to the choice of a variety of meta-parameters, such as optimizer parameters and model hyperparameters. Tuning these well, however, often requires extensive and costly experimentation. Bayesian optimization (BO) is a principled approach to solve such expensive hyperparameter tuning problems efficiently. Key to the performance of BO is specifying and refining a distribution over functions, which is used to reason about the optima of the underlying function being optimized. In this work, we consider the scenario where we have data from similar functions that allows us to specify a tighter distribution a priori. Specifically, we focus on the common but potentially costly task of tuning optimizer parameters for training neural networks. Building on the meta BO method from Wang et al. (2018), we develop practical improvements that (a) boost its performance by leveraging tuning results on multiple tasks without requiring observations for the same meta-parameter points across all tasks, and (b) retain its regret bound for a special case of our method. As a result, we provide a coherent BO solution for iterative optimization of continuous optimizer parameters. To verify our approach in realistic model training setups, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset. Our results show that on average, our method is able to locate good hyperparameters at least 3 times more efficiently than the best competing methods.

Under review as a conference paper at ICLR 2022
AUTOMATIC PRIOR SELECTION FOR META BAYESIAN OPTIMIZATION WITH A CASE STUDY ON TUNING DEEP NEURAL NETWORK OPTIMIZERS
Anonymous authors Paper under double-blind review
ABSTRACT
The performance of deep neural networks can be highly sensitive to the choice of a variety of meta-parameters. such optimizer parameters and model hy- perparameters. Tuning these well, however, often requires extensive and costly experimentation. Bayesian optimization (BO) is principled approach to solve such expensive hyperparameter tuming problems efficiently. Key to the performance of BO is specifying and refining a distribution over functions, which is used to reason about the optima of the underlying function being optimized. In this work, we consider the scenario where we have data from similar functions that allows us specify tighter distribution a priori. Specifically, potentially costly task of tuning optimizer parameters for training neural networks. Building on the meta BO method from Wang al. (2018b), develop practi- cal improvements that (a) boost its performance by leveraging tuning results on multiple tasks without requiring observations for the same meta-parameter points across all tasks, and (b) retain its regret bound for special case of our method. As a result, we provide coherent BO solution for iterative optimization of continuous optimizer parameters. To verify our approach in realistic model training setups, we collected large multi-task hyperparameter tuning dataset by training tens models popular image
focus the common but
of thousands of configurations of near-state-of-thee and text datasets, as well as protein sequence dataset. Our results show that on average, our method is able to locate good hyperparameters at least 3 times more efficiently than the best competing methods.
INTRODUCTION
The careful tuning variety of mnta-parametert such as optimizer parameters and model hy- perparameters has become a basic necessity for deep learning (Bergstra et al., 2011; Feurer et al., 2015). Such tuning requires extensive experimentation, retraining models repeatedly with different configurations, and can be challenging realistic budgets because the tuning landscape is typically non-stationary, noisy and ill-behaved. Tuning has become sufficiently costly that finding more efficient and effective tuning procedures has the potential save a substantial amount of resources or, alternatively, improve the accuracy of the final models at a given budget.
Some hyperparameters might be common across large number of tuning problems, such those pertaining to an optimization algorithm. For example, Adam is used across many deep learning applications and has four parameters that require careful tuning (Nado et al., 2021). Thus, if we have access to the performance of different optimizer-speciffe hyperparameters on different model training tasks, we may able to transfer the knowledge among those tasks. This kind of meta-level learning is common among practitioners themselves: when faced with a new tuning problem, one might first try reusing hyperparameter settings that worked well on another problem. The underlying assumption is that hyperparameters should perform similarly across tasks. In this work, we aim to formalize this assumption and automate optimizer hyperparameter tuning by leveraging knowledge from previous experiments. Although our experiments consider optimizer parameter tuning practically important sub sub-proble problem of hyperparameter tuning for deep neural networks, our method applies any hyperparameters that are common across multiple tasks.

---

Under review as a conference paper at ICLR 2022
Bayesian optimization (BO) has become popular methodology for optimizing the hyperparameters of machine learning models (Snoek et al., 2012; Bergstra et al., 201 and represents the state-of- the-art (Turner et al., 2021). BO involves specifying probabilistic model over the function to be optimized and using this to reason about the location of the optimum. The optimization proceeds by iteratively updating the model with new data and using the posterior distribution to reason about where to next evaluate, trading off exploration and exploitation. The model is typically specified with only a-priori assumptions of smoothness, for example using a Gaussian process (GP) with smooth covariance function. Even if the model is well- seel-specified, BO can be slow to converge due to the generality of the prior assumptions. This would seem wasteful for problems that are repeated often or share considerable structure with previous experiments.
One natural option is to cast our problem as meta Bayesian optimization, where the goal is to learn to optimize a function by generalizing from past experience with other similar functions. Indeed, several BO methods exist in the literature, but they are unsuitable for our scenario where we envision potentially thousands of related tasks within e.g. the context of a hyperparameter tuning service. Existing meta BO methods either scale cubically in the number of evaluations and tasks (Swersky al., 2013; Bardenet al., 2013) (See $4.3 for more details), impose restrictive set of assumptions on the available data (Wang et al., 2018b; Swersky et al., 2013) to obtain efficient solutions, or make assumptions the availability of GP parameters (Volpp et al., 2020) or descriptive task-level features (Brazdil et al., 1994; Bardenet et al., 2013; Yogatama & Mann, 2014).
To address these issues, we introduce HyperBO: meta BO method that builds upon Wang al. (2018b) with a relatively simple assumption: all the related functions being optimized are samples from the same GP prior distribution over functions. Concretely, HyperBO assumes the functions are conditionally independent given the hyperparameters. mean and covariance function of the GP. Compared to Wang et al. (2018b), HyperBO does not impose any strict conditions on data or model structures, and a special case of HyperBO retains strong regret bounds. From a computational perspective, HyperBO scales linearly in the number of tasks during training, and does not depend on the number of tasks when deployed. By not imposing assumptions about the data collection conditions, it can be used with large offline datasets or a few related optimization trajectories.
To evaluate HyperBO, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of wear-state-of-the-ar models on popular image and text datasets, as well as a protein sequence dataset. We compare HyperBO several hyperparameter tuning baselines in the sequential BO setting. Our results show that optimizers that use hyperparameters suggested by our method are able to obtain better performing models requiring at least 3 times fewer function evaluations than other baselines.
Our main contributions are two-fold: (1) a practical meta BO approach that makes minimal assump- tions: and (2) a large multi-task hyperparameter tuning dataset that not only benefits our method but also serves an ideal benchmark to test future multi-task or meta-learning BO methods.'
Related work There is rich literature of innovative methodologies to improve the efficiency of BO given related tasks or additional context. Here we discuss the most closely related work and explain why these don't 't solve the specific scenario which we envision. Specifically, our goal is a methodology that is scalable enough to share information across thousands of tasks, each with potentially hundreds of observations, such as in the context of large BO service or library.
In this work we use the term me to refer to the class of BO methods that use data from existing tasks to optimize new task. Multi-task BO (Swersky et al., 2013; Poloczek et al., 2017: Yogatama & Mann, 2014) and transfer learning BO using contextual GPs (Krause & Ong, 2011; Bardenet et al., 2013; Poloezek et al., 2016) are both meta BO approaches. Some meta BO methods have also been studied for hyperparameter tuning tasks in machine learning (Feurer et al., 2015).
HyperBO assumes all tasks are independent (after conditioning on the GP), whereas both multi-task and contextual BO rely heavily on the assumption that tasks are related. Thus the latter approaches typically scale cubically in both the number of tasks and observations each task, meaning that they cannot gracefully scale across both without heavy approximations. When assuming that all inputs are equal across tasks, multi-task BO can be sped up using a Kronecker decomposition of the kernel
We are working on open-sourcing the code hase and dataset. The dataset is collected based on an open- sourced code base (Gilmer al., 2021).
2

---

Under review as a conference paper at ICLR 2022
to task kernel and an imput kernel which can be inverted separately; similar assumption is made by Wang et al. (2018b). In comparison, HyperBO scales linearly in the number of tasks (see BB).
End-to-end learning (Chen et al., 2017; Volpp et al., 2020) is another popular meta BO approach for hyperparameter tuning that learns a strategy to suggest new query points based on past history of BO. One limitation of such approaches is that the total number of BO iterations must be determined a-priori. Furthermore, by nature of using a highly parameterized model to train the strategy, we lose the interpretability of intermediate steps that GPs and acquisition functions provide.
HyperBO builds upon Wang et al. (2018b) and Kim et al. (2017; 2019). We resolve their issues with optimizing over continuous space rather than discrete set and limitations on using the same set of inputs across tasks. Kim et al. (2017: 2019) estimated multivariate Gaussian that models values for search strategies in robot manipulation tasks, and thus only considered discrete inputs. Wang et al. (2018b) provided regret bounds for Kim et al. (2017: 2019), which was identified as meta BO without the knowledge of the GP prior. For both finite discrete search spaces and continuous ones, Wang et al. (2018b) requires observations on the same set of inputs across tasks, which is an assumption that is not required for HyperBO; HyperBO still inherits the same regret bound as Wang et al. (201 1b) for the special case where the same-inputs assumption is satisfied. Similar ideas also appeared in Perrone et al. (2018), which can be viewed as a special case of HyperBO or Wang et al. (2018b).
2 PROBLEM FORMULATION
We consider the standard black-box function optimization scenario: given a real-valued function f defined over compact, hyper-rectangular space X C Rd and given observations of similar functions fi.... . JNN. we seek an x E X optimizing f. We inherit our problem formulation from Wang et al. (2018b), but relax impractical assumptions data availability (we do not require all observations to be made on the same inputs across tasks) and model restrictions.
Assumptions and the goal. Concretely, we assume that there exists Gaussian process GP(n, k) with unknown mean function J : X + IK and kernel k: X x X - IR. Let N be the number of tasks and let Mj be the number of observations we have for the ith task. Conditioned on independent function samples Ji ~ GP(,, k) and inputs z; E E E [N], E [Mi], we observe evaluations (,) Vj" ~ (((((a))) ),²²) perturbed by i..dd additive Gaussian noise A N(0,o²). Taken together, the collection of sub-datasets Dy. = {(a])) )ر² define a dataset DN = [D), W Finally, our goal is to maximize a new function independently sampled from the same GP, f~ GP k)); that is, solve arg max,ex (xx given dataset D but unknown functions μu, k and unknown parameter o².
An example. In our optimizer hyperparameter tuning application, a task corresponds to finding the best optimizer hyperparameters to train a given model on a particular dataset,² e.g. training a ResNet (He et al., 2016) on ImageNet (Russakovsky et al., 2015). Notice that we do not assume that the mean function μ., kernel & and noise variance o² are given. This is consistent with the reality of solving real-world black-box optimization problems including hyperparameter tuning. We must learn those unknown functions and parameters from data. However, in practice, searching in functional spaces to find the right mean μ or kernel k is daunting task. Hence for practical concerns, a well defined search space for functions is required. More details on this can be found at $3.1.
Metrics. For simplicity, throughout this paper, we focus on the setting where the target function f can only be optimized by iteratively choosing where to evaluate, and defer batch evaluation setups to Sec. F. As run BO on the target function ff for T' iterations, we accumulate set of observations Dj = {( v)) 3/+ ~ ((f(2)), d²). We evaluate the quality of the optimization using the simple regret metric: Rr maxzex I (xx f()), where å is the final recommendation at the end of the optimization process. There are various ways of setting â based on the observations Dy; we use the input that achieved the best evaluation: â IriT arg max,e Vt-
Bayesian viewpoint. As mentioned above, the observed functions fi, fN and the evaluation target f are assumed to be independent draws from the same GP. This assumption is consistent with a hierarchical Bayes interpretation (Fig. 1), where all observed functions are independent conditioned on the GP. Notice that for BO, each selected input (=(()) depends on all previous observations. But we only describe the generative model of hierarchical GP for simplicity.
Technically, we also consider different batch sizes be different tasks.
3

---

Under review as a conference paper at ICLR 2022
More specifically, assume that underlying functions of hy- perparameter optimization tasks are defined by parameter 8 ~ p(0; a); mean and kernel functions μ and k are defined by deterministic functions parameterized by 0. The inddpee- dent function samples {fi [EN]] are draws from (P k:). We then learn function f from observations DN on all other con- ditionally i.i.d. function samples f1, .N. We forgo a fully Bayesian approach that samples from the posterior over 0 at every BO iteration. Our method can be viewed as a type-Il max- imum likelihood approximation of such a Bayesian solution.
Figure I: Graphical model for a hi- erarchical Gaussian process.
αx
yJ
fi
M N
Notations. Let [ri] denote (1.-..,7) .n), An E z+, For concise-
k
ness, we write the evaluation of a function X + R on matrix
x = [=>]/=1 == E 1n as u(x)= == [u(
Similarly, for two matrices (i.e., the concatenation of input vectors) æ, x', we write the corresponding kernel matrix as k(x, x') := [k<,,, 2 (e)) and shorten (=(x)) : (xx 2²). We denote (multivariate) Gaussian distribution with mean u and variance Σ by [u,, Σ), and a Gaussian process (GP) with mean function μ and covariance function by GP(u.k) k). Let o² be the noise variance in observations. Given a set of observations D 1=))
{112, (et,, 1=11 1. yr [u-]?" (N((())), o² ØT = [rre]== and ʃ ~ GP(u,,) we denote the corresponding conditional GP distribution as GP(u, & D). Let (xx) = k(z, xr) (k(())))) + +²2I) I)- Recall that the conditional distribution, GP(u, k D) GP KD). is given for any x, x. X as
p = u(x) (() - p(xT)), kn(w,²²) k(x, (z))((TTT x').
3 OUR METHOD
As shown in Alg. 1, our approach trains the GP hyperparameters on a representative set of datasets and fixes them for the duration of the optimization procedure; we refer to this approach as HyperBO. HyperBO runs in two steps. First, we learn a GP model GP k)) to approximate the ground-truth (unknown) GP that generated the dataset
Algorithm 1 HyperBO with acquisition function at-).
1:
function HYPERBO (f, DN)
2:
GP(p, k) + TRAINGP(DN)
3:
Dj +
4:
for 1 = =1, ,..,,ddo
St 5- arg max a (I:GP(R,k)D)) (D)) NEX
5:
DN. Then, we do standard BO to opti- mize a new function ʃ with the learned
6:
y: + OBSERVE(f(at)
7:
Dy < D3 U ((1),
GP GP k). The initial learning process (line 2) is the critical difference between HyperBO and standard BO algorithms, as well as the key contribution of this paper.
8:
end for
9:
return D5
10: end function
Based on the Bayesian graphical model interpretation (Fig. 1), our goal is to obtain a point estimate for the parameter 0. Given this estimate, we can then estimate the mean function j and the kernel k, which defines our learned model GP(A, ie). During the BO iterations (Alg. 1, lines 4-8), we update the conditional GP, but do not re-estimate the GP mean and kernel. By separating the data for the conditional GP update and GP parameter training, we minimize the computational cost while still maintaining good performance both theoretically and empirically. Moreover, avoid the BO chicken-and-egg dilemma ( (Wang et al., 2018b) where the search strategy is trained on data collected in the BO process and the data points are selected by the search strategy.
Next, we introduce our GP training strategy based on two types of objectives: marginal data likelihood ($ 3.1) and distance between estimates and model predictions ($ 3.2). In the appendix, $B reveals the complexity of HyperBO that is linear in the number of tasks and $ 3.3 shows that a special case of HyperBO retains strong regret bounds (Wang et al., 2018b).
3.1 MARGINAL LIKELIHOOD
A straightforward way to train a GP is by optimizing the log marginal likelihood over the GP's hyperparameters. This is also known as type II maximum likelihood approximation (Rasmussen
4

---

Under review as a conference paper at ICLR 2022
& Williams, 2006). In our case, derive the data likelihhod for the observations from multiple functions that are assumed to be given, which is a key difference to regular GP or BO setups. The log marginal likelihood for our method is
pDN Mr, k,
-²) -
1 1- K y(o) log |K| 96) (-
log 2T
(2)
2
where v()) =y (zx == (z( I2 [-
(8) M, 1=1 and y) 1 1
j=
Our solution to the choice of mean function, kernel function and noise variance then becomes
n. k.k 0² = arg ma p(g (r.k,o²)
For mean function μ and kernel k. this optimization is done in functional space. While methods exist to search for functional structures (Kemp& & Tenenbaum, 2008; Malkomes & Garnett, 2018), one may opt for a simple search strategy within group of functional structures (e.g. mean μl E (linear, constant) and kernel & (exponentiated quadratic, Matérn)). For all combinations of mean/kernel structures or functional classes, we then optimize the parameterization of them and noise variance d² to eventually solve Eq. 3. Details of how we defined the search space can be found in $4.
3.2 DISTANCE BETWEEN ESTIMATES AND MODEL PREDICTIONS
Although the marginal likelihood is a straightforwwar objective to optimize, it may not be straightfor ward interpret how high of a likelihood is high enough for us stop our search for decent model. Nevertheless, we may be able to directly estimate the sample mean and covariance, and the distance between those estimates and model predictions could be good indicator of how good the model is. We will show in $3.3 that distance objective may lead us to better theoretical properties.
Here we consider a special case of dataset DN where part of it has matching inputs across some sampled functions. More formally, suppose we have matching dataset D'N {(xx, yر)} where
M is positive integer, I E X, yı wy 1 E R. and yj D'v can be constructed by querying set of functions f1, f....., fN at the same set of input locations [ 1xdx to obtain observation matrix y ول E R MxN.
(6) (f(zz)), o²)) Empirically, dataset
By definition of a GP, the vector of all function queries f(xr) is distributed according to multivariate Gaussian distribution (N(())), k(x)). With our observation model, we get the distribution for observations (u(x), k(x) ) for some unknown mean function μ and kernel k.
However, given that we have access to all observations y. we can estimate the mean on inputs æ as jr yylN E TRM and estimated covariance as K = \ j ) RMM here 1N is column vector of size N filled with 1s. We use biased estimate of covariance to be consistent with the corresponding maximum likelihood estimator in Eq. Notice that the estimated covariance includes in diagonal terms the variance of the observation noise.
For any distance function between the estimate (i, K) and model prediction (u(()), +o²) we obtain an objective to minimize, D (N (),A(((())) k(x) + Io²)). While there are different
measures of distributional discrepancy, we adopt the KL divergence. Let μ = p(xx) and K =
k(x) + Io?. The KL divergence is defined as
DRL. (N(P,
= (I(K->k) + (4- ( K (1 (1) In
K K
M):
(4)
and we can estimate the mean, kernel and noise variance by minimizing DxL. While is difficult to gauge how much a probability density is enough to obtain good model, Eq. 4 is "distance" that goes to as the difference between two distributions reduces. One may choose to do early stopping or model selection based on how close Eq. 4 is to O. Through information theory, we also know that the KL divergence in Eq. 4 describes the number of extra bits (or nats) to encode the multivariate normal A K). Overall we found the KL divergence in Eq. 4 relatively more interpretable than the marginal likelihood in Eq. 3.
NN "One may choose re-scale learned kernel by N to he unbiased.
5

---

Under review as a conference paper at ICLR 2022
The KL divergence in Eq. 4 introduces different optimization landscape than the marginal likelihood in Eq. 3. The KL divergence also makes use of the matching dataset D'y in a way that the marginal likelihood cannot. In fact, all matching inputs in the marginal likelihood in Eq. 3 are implicit: all inputs are passed in to mean/kernel functions, and so there is no way that Eq. can be informed that some inputs the same across tasks. As shown in $4, the KL divergence in Eq. 4 interestingly led to better results in our experiments.
3.3 THFORETICAL ANALYSES
While it is nontrivial to prove regret bounds for general scenarios without strict assumptions, is straightffrward to show a regret bound for our method with objective DxL of Eq. 4 in the matching dataset case where BO running on a finite set of inputs (Wang et al., 2018b).
Theorem 1. Let N 2 log & + T + 2. With probability at least - 6, simple regret in T iterations of Alg. with special cases of either GP-UCB or PI satisfies
Rr <0 (VN-
1 + + log
(0
O 2T ACT,JA/=T
max log I =k(A) +)(())+
(5)
More details can be found at $D. Theorem shows that the regret bound has linear dependency on the observation noise a. This is expected because in practice, we select the best observation rather than best function value (before observing a noisy version of it) to compute the simple regret. Another reason that we learn the noise parameter o jointly with the kernel, as shown by Eq. Hence when computing acquisition functions, the noise is always included in the predicted variance.
Intuitively, the more sub-datasets we have in the dataset, the larger is, the better we are able to estimate the GP model, and the closer the regret bound is to the case where the GP model is assumed known. Interestingly, the number of BO iterations makes the regret smaller in the second term but larger in the first term in Eq. 5. Usually as we get more observations, we get more information about the maximizer, and we are able to optimize the function better. However, as we get more observations on the function, GP conditional predictions have more freedom to deviate from the ground truth (see Lemma of Wang et al. (201 8b)). J. As a result, we get less and less confident about our predictions, which is eventually reflected in looser regret upper bound.
is tempting to prove similar bounds for more general settings where inputs are not the same across all sub-datased and BO happens in continuous space. Though the only prerequisite is to show that the difference between the learned mean/kernel and the ground truth mean/kernel is small, this prerequisite is as difficult as showing we can find model that has bounded generalization error across the entire continuous input space of an arbitrary function. Instead of making unrealistic assumptions just to satisfy such prerequisite, we leave the regret bound for general settings as an open question.
EXPERIMENTS
Our goal in this paper is to provide practical approach for hyperparameter optimization when we are given data on range of tasks over the same search space. To analyze the effectiveness of our proposal, we take the optimizer hyperparameter tuning problem in deep learning case study. Our implementation of HyperBO is based on JAX (Bradbury et al., 2018).
To reduce ambiguity, we distinguish between datasets that individual neural networks trained on and the dataset we collected that includes optimizer hyperparameter points with their validation errors (and other metrics). We will call the former (e.g. MNIST, CIFAR10) task datasets and call the latter the tuning dataset. The tuning dataset is what we described as dataset Dx in $2.
4.1 HYPERPARAMETER TUNING DATASET
In order to collect our hyperparameter tuning dataset, the PDI Neural Net Tuning Dataset, defined a set of 24 neural network tuning tasks and a single, broad search space for Nesterov
We are working on open-soureing our code as well as trained GP models.
'The hatch size 1024 ResNet50 ImageNet task only has 100 hyperparameter points because ahandoned it when scaling up data collection in order save compute resources. is used in training, but evaluation.

---

Under review as a conference paper at ICLR 2022
momentum. Each task is defined by a task dataset (e.g. ImageNet), specific neural network model (e.g. ResNet50), and a batch size. Tab. shows all the tasks that we consider in the tuning dataset. We used existing code base (Gilmer et al., 2021) for neural network model training. The dataset used roughly 12,000 machine-days of computation for approximately 50,000 hyperparameter evaluations.
For each task, we trained the model the task dataset repeatedly using Nesterov momentum (Nesterov, 1983; Sutskever et al., 2013), with the task's 's minibatch size, with different hyperparameter settings drawn from the 4-dimensional search space detailed in Tab. 2. We tuned the base learning rate, 7, on a log scale, the momentum, B, with 1 - 3 on a log scale, and the polynomial learning rate decay schedule power p and decay steps fraction We used polynomial decay schedule with the
following form: 7/-
7 + (1)
Io00) (1- A P where 7 is the training step and T is
1000
the total number of training steps for the task.
Table 1: Tasks
Dottsse
Model
Rotch Sizes
CIFARIO
Wide ResNet
[256, 2048]
Table 2: 4-dimensional input search
CIFAR1OD
Wide ResNet
1256. 2048]
MNIST
Mas, pool CNN ReLU
2048]
space (see text for more details)
Fushion MNIST
Max pool CNN unh
[256. 2048]
Fashion MNIST
Simple CNN
[256. 20:88
Hyperparaneeerr
Range
Scaling
ImageNet
ResNet50
(512, 1024, 20:18
10 5 10
Log
LMIB
Tranformmr
(2046)
(0.1.2.0
Lincar
MNIST
MNIST
Mas pool CNN unch
Max pool CNN reln
[256. 20:88
[256, 2046)
1 1-
8
10 9, 1.0
Log
MNIST
Simple CNN
2048)
0.91
Linear
SVHN extra)
ResNet
1024]
WMT15 Germun English
xlomeer
[64]
uniret50
Transformer
[128]
We collected two types of data: matched and unmatched data. Matched data used the same set of
hyperparameter points across all tasks and unmatched data sampled new points
for each task. All other training pipeline hyperparameters were fixed to hand-selected, task-specific default values. All of our tasks are classification problems, so they all used the same training loss, although occasionally task-specific regularization terms were added. For each trial (training run for a single hyperparameter point), we recorded validation error (both cross entropy error and misclassification rate). In many cases, poor optimizer hyperparameter choices can cause training to diverge. We detected divergent training when the training cost became NaN and then marked the trial but did not discard it. Please see the Appendix, supplementary material, and code (Onomous, 2021) for additional details about the tasks and training procedure. The different tuning tasks vary in difficulty and numbers of data points, but generally there are roughly 500 matched datapoints and 1500 unmatched datapoints per tuning task. For unmatched data only, we attempted generate roughly similar numbers of non-divergent points across tasks, so tasks with a higher probability of sampling a hyperparameter point that causes training diverge will tend have more trials.
4.2 DESCRIPTION OF ALL COMPARED METHODS
Our method HyperBO has several variants including using different acquisition functions and different objectives. In $4, unless otherwise mentioned, we used a thresholded probability of improvement
(PI) as the acquisition function. We set PI in line of Alg. as evaluated a variety of acquisition functions, but found PI thresholded at 0.1 to be surprisingly effective, Because we model the observations as log error rate, this actually trades off exploration and exploitation i.e. with larger error rates this seeks relatively more substantial improvements than with small error rates. The list of 5 different acquisition functions we tested is as follows: PI with 0.1 threshold, expected improvement and UCB with 2, 3, 4 coefficients. See their comparisons at $E.5.
We empirically
(2)
We use H' NLL to denote HyperBO with negative log marginal likelihood as the objective and H* KI. to denote HyperBO with KI. divergence on matching datapoints as objective. Both ob- jectives are optimized via L-BFGS (Nocedal, 1980) with u(z) = 0 tanh(0) 2z), k(z,, 2²) Matém33((tanh(() x), tanh (0, z²)), 0, E 124x8 These two settings of HyperBO are relatively representative of the performance of variants of HyperBO. See comparisons over objectives at $E.4.
Our baselines include (a) Rand: Random search in the corresponding scaled space in Tab. 2. (b) STBO: Single-task BO where in every BO iteration, STBO optimizes the GP hyperparameters via marginal likelihood on data of the test task. This implementation corresponds to the basic off-the-shell BO setups. (c) STBOH: Single-task GP-UCB with hand-tuned prior on hyper-parameters including
7

---

Under review as a conference paper at ICLR 2022
(8)
Band SmR0 STEOH
[9]
STBO S18GM
Iters 100
0,0175
Dog Dag vt ere MMc
10-
0:012
10/4
0.0000
0.0075
0.0
0.0000
23
100
30'
20
90 cutpertfim
Figure 2: (a) The performance profile for outperforming the median of best error rates at the 100th BO iteration. (b) BO convergence of all methods: the median and 20/80 percentiles of the regrets on error rates over 115 BO runs: 23 tasks and each with repeats of different random seeds. (c) A violin plot the vertical slices of (b) at the 100th iteration; the white dot is the median and the black line is the 20/80 percentile. Overall, HyperBO methods H* NLL and H* KI. are able to achieve the lowest regret on error rate on the majority of tasks with fewer iterations.
UCB coefficient. (d) MIMO: Multi-task BO with GP bases as an ensemble of feedforward neural networks with shared subnetworks (Havasi et al., 2020). (e) RFGP: Multi-task BO with GP bases as random features. More details can be found at $E.1.
4.3 RESULTS ON OFFLINE OPTIMIZER HYPERPARAMETER TUNING TASKS
Many tasks in $4.1 can use up lot of compute resources and time, which makes it infeasible to perform a wide variety of experiments to analyze the characteristics of BO methods. Hence we adopt an offline approximation, which runs BO only on the finite set of points that each tuning sub-dataset contains. More details and analyses are available Appendix E.2.
Holding out relevant tasks. Fig. 2 (a) shows the performance profiles, the fraction of all test tasks that each method from $4.2 is able to outperform a baseline criterion at each BO iteration. We can see that MIMO able to outperform other methods in the beginning 20 BO iterations, but its leading position soon gets surpassed by HyperBO (H* NILL and H* KI.). Fig. 2 (b,c) illustrates the BO convergence curves of all competing methods, together with the vertical slice the 100th iterations. REGP and STBO are both falling much behind Rand. STBO trains the GP on the data that the GP suggests to query. Optimizing the marginal data likelihood on at most 100 datapoints in fact may not lead to better model than random initialization (see Tab. 5 in $F). Surprisingly. the contextual information learned by RFGP did not generalize to a new task. On the other hand, MIMO is able to obtain a slightly better error rate than STBOH. Overall, learning the GP prior through data with HYperBO outperforms other meta BO methods, and is a more principled and effective way to obtain the GP prior when compared with hand-tuning.
Effect of number of training tasks. We now investigate the impact of the number of training tasks on the performance of meta BO methods. In Fig 3 show the BO simple regrets tasks from Table (except ImageNet ResNet50 2048) that use meta BO models trained different number of training tasks. To analyze the performance of all methods on less-related tasks, we first remove training tasks that have the same task dataset our current tuning task for testing, and then remove randomly selected training datasets from the rest.
Figure 3: Medians and 20/80 per- centiles of regrets on best valida- tion error rates for methods that uses models trained on 3 to 23 training tasks.
Hard
MEMO
ALL
20
STEOH
HOperBO variants reduced the simple regret as more training tasks are given. Interestingly, H' NLL and H* KL are already slightly better than Rand and STBOH when they started off with only 3 training tasks. There are reasonable fluctuations in the results but overall the trend of regret is going down as the number of training tasks increases. MIMO also reduced regret when the number of tasks increased from 8 to 18. RFGP, however, fails to learn from training tasks possibly because it did not learn good task embeddings for GP regression models.
13
18
Mamber feariing weks
8

---

Under review as a conference paper at ICLR 2022
Inapeler Ras
2008 MELL 1024
US handome 323
10
STBO STEOH MEMD
3x:00
STBO
2.05x10'
STEOH
RFOP
Error PATH
MIMD
STBOH
2
2,9x10
MIMRO
NILKI
RGP
2,0x10'
H" KLLKL
+ NLL
NLUKL
3x.10"1
x.65
40
80
100
80 Ilees
B0 Hler's
50 liers
Figure 5: Results of running BO methods in the online setting on 3 different tasks. The image based tasks use best validation enror rate objective while the text based ones including Uniref50 use best validation CE loss. In all 3 tasks, HyperBO methods achieved better results.
Effect of number of data points in training fasks. One maining question is, how does M, in $2, the number of data points in each training tasks, affect the performance of meta BO methods. We analyze the impact of M; by removing portion of all data that have access for each task. In particular, we set the percentage of remaining data to be 0.2%, 0.5%, 1 1%, 3%, 5%, 10%, 30%, 50%, 70%, 90%. Remain- ing datapoints are selected uniformly randomly, which breaks the structure of matching data. Hence we do not include Ha KL in this comparison, as H* KI. only makes use of matching data.
Figure 4: Medians and 20/80 percentiles of simple regrets for methods that uses models trained on 0.2% 90% of data in each
task.
Rand
MNC
KL
10
STBOH
RFCP
Fig. shows how the simple regret changes as the fraction of training data grows. Below 10% training data, we observe clear
Ragper on stror
20-2
trend that more data lead to lower regret for both H* NLL and MIMO, and relatively no change for RFGP. We also found that the performance of HyperBO (H* NLL) does not change much as the fraction of training data increases from 5% 90%. However, MIMO and RFGP suffers significantly from more data as the fraction of training data increases from 5% to 50%. It is not entirely clear why MIMO and REGP have such behaviors. One conjecture that neural network based Bayesian linear regression models may get too confident once the amount of data reaches a certain threshold. This means much less exploration if those models are used for BO.
10-4
10
Fraction of oftrainit data
4.4 RESULTS ON ONLINE OPTIMIZER HYPERPARAMETER TUNING TASKS
We now evaluate HyperBO methods, H' NLL and H* NLLKL (NLL plus KL divergence matching datapoints as the objective), in the online setting, where we optimize over the full hypercube and some hyperparameters may be infeasible evaluate. See full results $E.3. Fig. shows that can be difficult for STBO or MIMO to recover from a "bad" datapoint, but HyperBO methods are robust and performed the best among all methods being compared.
5 DISCUSSION AND CONCLUSION
While we focused obtaining better prior in BO in this work, the following directions are orthogo- what we studied: different search spaces across tasks, batch evaluation, high-dimensional or large scale data, etc. However, it should be straightforward to combine their solutions with HyperBO. Please find more discussions at SF together with implications our assumptions.
HyperBO is novel meta BO approach that supports practical applications that involve continuous inputs queried at possibly non-au non-aligned locations across tasks. HyperBO uses a simple yet effective idea that is easy to implement and efficient to run. We evaluated HyperBO real-world big model optimizer tuning tasks, and the results demonstrated its superior performance over state-of- of-th---rrr-tttttt competing methods.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rémi Bardenet, Mátýás Brendel, Balázs Kégl, and Michele Sebag. Collahorative hyperparameter tuning. In JCML, 2013.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, 2011.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http: /github. com/ homm//oole//////// jax.
Pavel Brazdil, João Gama, and Bob Henery. Characterizing the applicability of classification algorithms using meta-level learning. In ECML, 1994.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap. Matt Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. ICML, 2017.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In NeurlPS, 2015.
Justin M. Gilmer, George E. Dahl, and Zachary Nado. init2winit: jax codebase for initialization, op- timization, and tuning research, 2021. URL www.:httttttc /ttp://gtttttt com/ hwtoo///ooll/////ooooooooooeeee/ http//ininttiiiiiiimm
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. Coco: A platform for comparing continuous optimizers in black-box setting. Optimization Methods and Software, 36(1):114-144 14-144, 2021. URL https::///// /waxiviivvv.../////// org/ pdf/1603 08785 85., pdf.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010.06611 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Tarun Kathuria, Amit Deshpande, and Pushmeet Kohli. Batched gaussian process bandit optimization via determinantal point processes. NeurIPS, 2016.
Charles Kemp and Joshua B Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences, 105(31):10687-1062, 2008
Beomjoon Kim, Leslic Pack Kaclbling, and Tomás Jozano-Pééee, Learning to guide task and motion planning using score-space representation. In ICRA, 2017.
Beomjoon Kim, Zi Wang, Leslie Pack Kaelbling, and Tomás Lozano-Pérez Learning to guide task and motion planning using score-space representation. The International Journal of Robotics Research, 38(7):7938812 2019.
Andreas Krause and Cheng S Ong. Contextual Gaussian process bandit optimization. In NeurIPS, 2011.
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax- Bedrax-Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint arXiv:2006. 2020.
Gustavo Malkomes and Roman Garnett. Automating Bayesian optimization with Bayesian optimiza- tion. Advances in Neural Information Processing Systems, 31:5984-5999 2018.
Zachary Nado, Justin Gilmer, Christopher J. Shallue, Rohan Anil, and George E. Dahl. A large batch optimizer reality check: Traditional, generic optimizers suffice across batch sizes. CoRR, abs/2102.06356, 2021. URL https: :enttss:/ae /tt///aaivvv/v/////v////aa//awaa wtw.vv.rr./////o.oeav 02. 06356.
10

---

Under review as a conference paper at ICLR 2022
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate 0(1/k22). In Dokl. akad. nauk Sssr, volume 269, pp. 543-547, 1983.
Jorge Nocedal. Updating quasi-newton matrices with limited storage. Mathematics of computation, 35(151):773-782, 1980.
ChangYong Oh, Efstratios Gavves, and Max Welling. Bock: Bayesian optimization with cylindrical kernels. In 2018.
Anne Onomous. Anonymized, 2021. URL
Valerio Perrone, Rodolphe Jenatton, Matthias Seeger, and Cédric Archambeau. Scalable hyperparam- eter transfer learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6846-6856, 2018.
Matthias Poloczek, Jialei Wang, and Peter Frazier. Warm starting Bayesian optimization. In Winter Simulation Conference (WSC). IEEE, 2016.
Matthias Poloczek, Jialei Wang, and Peter Frazier. Multi- source optimization. In NeurIPS, 2017.
Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NeurlPS, 2007.
Carl Edward Rasmussen and Christopher Williams. Gaussian processes for machine learning. The MIT Press, 2006.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3)221,-252, 2015.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In NeurlPS, 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171-2180. PMLR, 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. Advances in neural information processing systems, 29: 4134 4142, 2016
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In ICML, 2013.
Kevin Swersky, Jasper Snock, and Ryan P Adams. Multi-task Bayesian optimization. In NeurIPS, 2013.
Ryan Turner. David Eriksson, M. McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and I. Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. ArXiv, abs/2104. 2021.
Michael Volpp, Lukas Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-learning acquisition functions for transfer learning in Bayesian optimization. In International Conference on Learning Representations (ICLR), 2020.
Zi Wang, Chengtao Li, Stefanie Jegelka, and Pushmeet Kohli. Batched high-dimensional Bayesian optimization via structural kernel learning. In ICML, 2017.
Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian optimization in high-dimensional spaces. In AISTATS, 2018a
Zi Wang, Beomjoon Kim, and Leslie Pack Kaelbling. Regret bounds for meta Bayesian optimization with unknown gaussian process prior. In NeurlPS, 2018b.

---

Under review as a conference paper at ICLR 2022
Ziyu Wang. Frank Hutter, Masrour Zoghi, David Matheson, and Nando de Feitas. Bayesian optimiza- tion in billion dimensions via random embeddings. Journal or Artificial Intelligence Research, 55:361-387, 2016.
Dani Yogatama and Gideon Mann. Efficient transfer learning method for automatic hyperparameter tuning. In AISTATS, 2014.
A HIERARCHICAL GP, A BAYESIAN PERSPECTIVE
In Bayesian viewpoint of $2, we assume that the overall setting of the hyperparameter optimization task is defined by a parameter ~ p(0; a); mean and kernel functions μ and k are drawn from p(u, 0). The independent function samples [fihie[)] are themselves draws from (p, k). The generative story is as follows:
Draw GP parameter 0 from p(0;a) a) and observation noise parameter from a.).
Draw mean function jr and kernel function from y(u, 0).
For each task from 1 to N,
Draw a function ft from GP(u, k).
-
For each data point j from 1 to M,.
* Given input ( we draw the observation u N(I(())))
We simplify this hierarchical setting by defining p(u, k. 0) to be sum Dirac delta functions: both mean function and kernel k are deterministic functions parameterized by 0. Thus, we can infer GP parameter and noise from their posterior p(0.o o DN D a) and obtain an informed prediction for the target function
/ p(f 10)p(0 DN DD
pff DNUD
J pls 00) p(0, Dx (D;a)
In other words, we learn function ff from observations on all other conditionally i.i.d. function samples f,...-,fx: We forgo fully Bayesian approach that samples from the posterior over 0 at every BO iteration, although our method, HyperBO, can be viewed as a type-Il maximum likelihood approximation of such Bayesian solution.
B COMPUTATIONAL COMPLEXITY
The marginal likelihood in Eq. 2 naturally decomposes into a sum of GP data likelihood terms on each sub-dataset Dj.- - The time complexity to compute Eq. 2 ((M N), where N is the number of sub-datasets and M max M; is the maximum number of data points for these sub-datasets. Notice that our method scales linearly in the number of tasks, N, in contrast to the cubic ((M 333 scaling multi-task contextual BÒ methods (Swersky al., 2013; Bardenet al., 2013; Poloezek et al., 2016; Yogatama & Mann, 2014). The only cubic cost of HyperBO is on the number of data points in sub-datasets.
To train GP with K optimization steps on Eq. 2, the time complexity is O(M The distance regularizers introduced in $3.2 requires estimating mean and covariance, which takes O(M N) for matrix multiplication. The KI. divergence in Eq. 4 has complexity ((M3) to compute and O(M3K) K) to optimize.
If there is any better probabilistic model than a GP to fit the data with less compute time, we can easily swap it in and reduce the (MM)) complexity that the GP contributed to the ((M N) complexity of Eq. 2. For example, if we approximate GP with linear model on V random features (Rahimi et al., 2007), the complexity of Eq. becomes O(V3 Another example is to train Eq. 2 with stochastic optimization methods, where the complexity of Eq. 2 on the full dataset can be reduced to O B² MN), where B the mini batch size. Running stochastic optimization will then take ( MNK), where K is the number of optimization epochs.
12

---

Under review as a conference paper at ICLR 2022
C OBJECTIVE FUNCTIONS
In $3, we presented NLL and KL divergence as objectives. Below we derive the KL divergence between regular multivariate Gaussian and degenerate multivariate Gaussian, which is the case for most of our matching data settings in $4.1: the number matching data points is greater than the number of training tasks. In the end of this section, introduce objective function, combining NLL and KL, interpreted as MAP with a data-dependent prior.
KL divergence for degenerate multivariate Gaussian Eq. 4 of $3.2 gives the KL divergence between two Gaussians in the non-degenerate case. In practice, when we minimize Eq. 4, we can simply remove the constants and do the following
nr, k 6² arg min k 2 DKI. (N ((,,
K K
arg min k
tr(K-IK) + (u (1) TIKKK(( (n ( +
In
M
-arg min min Mok,
K) + (μ (1) K (1 (t) + In K|
(6)
Here the variables we care about, μe, k o², only appear in mean vector μ and covariance matrix K over the matching data. Even if the sample mean and covariance estimate M(a, (ūr K) is degenerate, the optimization objective stays the same as reflected by the derivation below.
If (t, K') is degenerate, its base measure is at most N N-dimensional rather than M M-dimensional, given that there exists a full rank matrix MRR such that K = AAT T (R N). Note that M the number of matching data points, N the number of training tasks and R is the rank of matrix A and K. The KL divergence DRI. (N(ãa, R),, K)) is not well-defined because the base measure of (p, K) is different from the base measure of (iu, K), given K is full-rank. However, it is still possible to derive pseudo KL divergence DiL (N K),N(H, K)) below.
Let the degenerate Gaussian be p(x) = NN, K) |] exp (-) x - ())T) and the non-degenerate one be q(x) = (1², K), where | the pseudo-determinant and K+ the peudo-inverse of K. We define the support of distribution as S(p) = (p + Kłv 1oe E R MM. The pseudo KL divergence between p(x) and ((x) now becomes
Da. (Nii, R),N(M K3)
p(w) (lup(x) In lx)()))
sim
T))x=ttu K (a ( (x n) x (= m)) 2 2 p(a) (In [m|22 - In |2 (ar T)TT
1
1
(aNs (IM- I)) 11) In
K
Ep/in(K (an(K+(ee (1 Ax Ai) uil 1 (x- n)(( 4)
2
A|
2 (M Rn
I
K
tr(k R) E.(t((K 1(x 1)(z - u) T)
TT
I
((M 13) 15 In
K]
ir(AA+) Ep ,)(r x - p) + K- (2mp 22x - pp + up |n]
2
AAT A]
2 (M R) 1m
1
K|
m k- (f x) m)T)
=
A A]
2 (u(K 'R) (H R)TK L(1 -m A| R (M R) In
(7)
If the covariance matrix K. is in fact full rank, i.e. R = M, pseudo KL in Eq. 7 then recovers the KL divergence DKL (N(P, ( (H,K) K)) in Eq. for non-degenerate Gaussians. If we were to
K)). we still would get the minimization task in Eq. However, pseudo KI. divergence DiL does not satisfy properties including non- negativity. In practice we may also choose add small epsilon values to the diagonal terms of both
minimize this pseudo KL divergence DiL
K and K to make K "less degenerate" and enable the existence of DRI. (N K),N(u, (H,K).
13

---

Under review as a conference paper at ICLR 2022
Combining marginal likelihood and distance metrics It also possible to additively combine the KL divergence with the negative log marginal likelihood objective, and treat this distance as a regularizer. In the of KL divergence. it equivalent to adding detendennndnnt prior on the GP for some normalization constant Z, and the posterior is itself: ,a, K ~ (Dpl (.N(..)))((
p(u,k: k. o² DN: K) x pDDN μ. k, o²) exp(-XDas (N(x. K). K)))
(8)
We then obtain MAP estimation for the unknown functions and variables μu, o²
D DETAILS OF REGRET BOUNDS
Theorem is a direct result of Theorem 16 in Wang et al. (2018b). The only subtle difference is that we adopted biased estimate of the covariance matrix, factor of N-1 different from the unbiased estimate. But this can be corrected in the acquisition functions. We provide more details below.
N
Proposition 2. For any M, d, E z+ I E IR xxd E RM, RRM.V E N and V there exists a Gaussian process (p, ii) such that DEL (N(H, ((
=0
Proposition 2 is easy to show. We can train simple memory based model for mean function ji and kernel h. The model stores each element of vector and matrix K at the corresponding locations in input æ. When making a prediction on a new input x' Rd, the model simply retrieves the values of the closest element in Given Proposition 2, regret bound follows (Wang et al., 2018b).
By Proposition 2, we are able to obtain a () k) where (a)) and k(x) are equal to the sample mean and covariance problem setup is consistent with the case with discrete input space in Wang et al. (2018b). The following theorem is rewrite of Theorem 16 in Wang et al. (2018b), taking into account our biased estimators.
matching dataset (z, y). following the notations in $3.2. Hence,
Theorem 3. Assume there exist constant 2 max k(z) and a training dataset is available whose 6
size is 2 4 log
8 + T 2. Define
6 N 3 + 2 tlog o + 2lo b
1
6 7" for any & E [T], log
8N(N - 1)
N
and pr max AEX.[A|=T2
log II (k))( Then, with probability at least the best- sast-sample
simple regret in T iterations of meta BO with GP-UCB that uses
(6N(N- - ++ + 2 1 log _ 2log ()(((N(N-- -1-1)))
(2N (aWlog())) ())
(9)
(N-1)) 1)(1 2( 2 log 2)
as its hyperparameter in at
GP.UCR
(x)1(x)) (ike-1(z)1 satisfies
(2 (log(())) 22)
FT < i GP-UCB
.P-UUCB
Tlog(1 + 20- + +
2cpr
Ve+a² 02
where iu GP-UUCB = ( (+(2lo(( =(
21-2/0r-1 1112//bb-11 + 2br + IT-1 (ogg( (o))
With probability at least - 8, the best-sample simple regret in T iterations of meta BO with PI that uses fO 2 max f fxx ) as its target value satisfies
#l pp V
2cpT
+ 02
3 (2
28
20
Tlog(1 co
22
2
where T = ( 8r - 1(5a) + 4-1)
1+26 +2
+ 4-1
+ (2 ( 31)) 7 =
(hr=i(za)to?
2
argmingerr hi- (xt)
14

---

Under review as a conference paper at ICLR 2022
The proof can be found in Wang et al. (2018b). Theorem 1 is condensed version of Theorem 3.
While Theorem 3 provides us with some understanding of HyperBO in specific setting, in practice, need to query in continuous input space that goes beyond the finite set of points present in the training dataset. It may or may not be possible to obtain data on a wide range tasks to ensure N 2 4 log A T + 2. In fact, in all of experiment, this criterion number of tasks is not satisfied. However, we still obtained good performance.
E EXPERIMENT DETAILS AND MORE RESULTS
In this section, we provide more detailed setups and empirical results on the impact of objective functions and acquisition functions in HyperBO. All experiment setups are the same as $E.2.1: offline and holding out related tasks.
E. BASELINES
Our first set of baselines include those that do not use information from training tasks:
Rand:
Random search in the corresponding scaled space in Tab. 2.
STBO: Single task BO with constant mean function, Matern32 kernel and PI acquisition function (same as above). Every BO iteration, STBO optimizes the GP hyperparameters via marginal likelihood on data of the test task. This implementation corresponds to the basic off-the-shelf BO setups.
Single task GP-UCB (coefficient=1.8) with constant mean, Matern52 kernel and hand-tumed prior on hyper-parameters including UCB coefficient. Specifically, log amplitude follows Normal(-1) 1, 1), log length scale (one per input parameter) follows 10.m and log observation noise variance follows Normal(-6,, 3). The hyperparameters are post- processed by ensorffow-probaaility SoftClis ip bijector to constrain the values between 1-st and 99-th quantiles. These prior distributions were manually tuned to obtain reasonable convergence rates on 24 analytic functions COCO (Hansen et al., 2021). The GP parameters are then optimized via maximum marginal likelihood every BO iteration.
STBOH:
For multi-task BO baselines, we included scalable methods that replace the GP with a regression model that be trained using SGD and thus scales linearly in the number of observations. Following the multi-task setup of Springenberg et al. (2016) 2), we jointly trained a 5-dimensional embedding of each task, which was then added the input of the following two models.
MIMO: We trained an ensemble of feedforward neural networks with shared subnetworks (Havasi et al., 2020). We used shared dense layer of size 10 and 2 unshared layers of size 10. We used tanh activation based on (Snoek et al., 2015, Figure 2). The network has one output unit with linear activation and another with moft 1) activation, corresponding respectively to the mean and standard deviation parameters of normal distribution. We trained for 1000 epochs using the Adam optimizer with learning rate 10-4 and batch size 64.
RFGP: We used the open-source implementation of approximate GP by Liu et al. (2020). We trained for 1000 epochs using the Adam optimizer with learning rate 10 3 and batch size 64.
All methods share the same input and output warping. The input warping is done according to the scaling function in Tab. 17 < log 17, 8 log(1 B). The output warping is done for the best validation eтror rate +- log(r + 10 10-10).
E.2 OFFLINE BO EXPERIMENTS
In all offline experiments, we ran offline BO on the data from the test task starting from zero initial data from this task. Each method was repeated 5 times with different random seeds to initialize its model. We ran all methods without de-duplication to best simulate online BO. We evaluate on regret on error rate which denotes the simple regret on the finite set of data points in each tuning sub-dataset.
15

---

Under review as a conference paper at ICLR 2022
15)
111
STRO
0.3
STOOM
HENO
5180
S1826
IP NLL
6:3
0.5
20 to sdpertar meder
100
Uic
50
25th for
has ressed octae medien sust une 18te
turo necced outartem medkan rate 10CH
Figure 6: Performance profiles for outperforming the median of best error rates at the (a) 25th BO iteration, (b) 50th BO iteration and (c) 100th BO iteration.
Rand
MINO
1P KLL
BC Iters 50
DO iters 130
10"
STB0
K:GP
17 RL
0.16
STSOH
0.12
0.059
on
10.7
0.08
0.0003
0.02
0.023
10
20
00 30
bara 8C
100
STRON
Figure 7: The left most a summary of the BO convergence of all methods: the median and 20/80 percentiles of the regrets eπror rates over 115 BO runs: 23 tasks and each with 5 repeats of different random seeds. We also show violin plots on its two vertical slices at 50th and 100th iteration, where the white dot is the median and the black line is the 20/80 percentile. Overall, HyperBO methods H* NLL and H* KL are able to achieve the lowest regret on error rate on the majority of tasks.
E.2.1 HOLDING OUT RELEVANT TASKS
We first conducted experiments in setting where a new task dataset is presented, and a BO method trying to tune the optimizer hyperparameters for selected model that task dataset. A training dataset for meta BO is composed of at most 18 tuning sub-datasets on training tasks that do not involve the same task dataset as the test task. All methods then proceed solve the test task on the new task dataset. Fig. 6 shows performance profiles of the BO methods described in $4.2. The performance profiles show the fraction of all test tasks that each method is able to outperform baseline criterion at each BO iteration. We chose the criterion to be the median of best error rates achieved by all methods at different BO iterations: 25th, 50th or 100th. The larger the fraction of tasks at each BO iteration, the better the method is. From all 3 criteria, we can see that MIMO is able to outperform other methods in the beginning 10 to 20 BO iterations, but its leading position soon gets surpassed by HyperBO (H* NLL and H* KL) and STBOH. HyperBQ methods are gaining similar not larger fraction than the best alternative. STBOH, throughout BO iterations. Fig. 6 (c) has the most stringent performance criterion, and it shows that HyperBO with the KI. objective outperforms HyperBO with the NLL objective in this set of experiments with a small margin. And both methods HyperBO are doing considerably better than others.
Fig. 7 illustrates the BO convergence curves of all competing methods, together with the vertical slices the 50th and 100th iterations. RFGP and STBO both falling much behind Random search. STBO trains the GP on the data that the GP suggests to query, which creates a loop that could be harmful for data acquisition. Optimizing the marginal data likelihood on at most 100 datapoints in fact may not lead to better model than random initialization (see Tab. 5 in $F). Surprisingly, RFGP, though equipped with the tuning dataset and initially reached some good values, performed similarly to STBO in the end. Clearly, the contextual information learned by RFGP did not generalize to a new task. On the other hand, MIMO is able to obtain a slightly better error rate than STBOH.
Fig. 6 and Fig. 7 both show that learning the GP prior through data like what HyperBO does is performing much better than other meta BO methods, and it is a more principled and effective
We show performance relative baseline because of varying scales tasks.
16

---

Under review as a conference paper at ICLR 2022
approach to obtain the GP prior than hand-tuning, As reference, we include Tab. 3 which shows the task best validation eπror rates obtained by the top 5 methods in 100 BO iterations.
Table 3: The mean and standard error of best validation error rates (%) for each test task in the offline optimizer hyperparameter tuning experiments. Meta BO methods including MIMO and HyperBO variants (H+ NLL and H* KL) have access to training tasks that do not share the same task dataset as the test task. We show results of the top 5 methods, and we highlight the lowest error rates in bold.
Rand
STBOH
MIMO
H
H KL
WMT XFommme 64
34.27 + 6.16
34.16 = 0.15
34.29 E 0.16
33.94 = 0.01
33.99 + 0.03
Unire150 Trameformer 128
79.06 0.04
78.92
0.12
78.93
0.11
78.64 + 0.00
78.74 1. 0.09
LMIB Transformer 2048
61,96 4 0.03
61.95 = 0.04
61.95
4 0.01
61.83 = 0,01
61.82 = 0.01
SVIN WRN 1024
3.99 1: 0.04
4.05 1: 0.10
3.82
4.11
4.06 1: 0.02
SVHN WRN 256
3.71 3.71 0.00
3.72 0.02
+.00
3.79 3.0910000
3.77 : 0.02
ImageNer ResNer50 256
23.03
22.66 = 0.07
22.69:0.000
22.37 0.02
22.55 + 0.02
ImageNNet RexNet50 512
23.02
22.74 1: 0.05
22.99 1. 0.05
22.65 -: 0.02
22.75 1 0.03
MNIST CNNPoolTanh 2048
0.55 0.00
0.53 0.01
0.52 0.01
0.53 0.01
0.54
MNIST CNNPoooTann 256
0.51 0.00
0.00
0.400
0.46
0.46
MNIST CNNPoolRRLU 2048
0.69 4 0.01
0.73
0.66
0.01
0.65:00001
MNIST CNNPoolRel.U 256
0.51 0.00
0.55 0.03
0.001
0.48
0.49 0.00
MNIST CNNRLLU 2048
1.14 0.03
1.20
0.02
1.06
1.08
MNIST CNNReL.U 256
1.09 4: 0.02
1.06 0.01
1.08
1.03
1.03 0.00
Fashion CNNPoolTanh 2048
7.14 1: 0.06
7.10 0.05
7.05 + 0.06
7.03 +.03
7.16
Fashion INaPoollannh 256
6.51 + 0.03
6.67 0018
0.007
0.38
6.28
Fashion CNNPoolReLU
7.47
0.002
7.48 0.001
0.506
7.42
7.53 0.01
Fashion CNNPoolReLU 256
1.78+0.00
6.74
0.00
0.79:0004
6.70 0.01
Fashion CNNRel.U U 2048
7.70 1: 0.03
7.47 0.09
7.64
0.006
7.57 0.01
7.56 1: 0.02
Fashion CNNReLU 256
7.70 4 0.04
7.46
0.11
7.83 0.05
7.44
7.25 0.05
CIFAR 100 WRN 2048
21.28
0.27
20.78 + 0.19
20.94 + 0.13
21.26
0.23
20.98 1. 0.24
CIFARIOU WRN 256
19.17 4 0.19
19.02
= 0.03
19.15
+ 0.06
19.07
: 0.04
18.98 +
0.00 0.01
CIFARTO WRN 2048
3.13 + 0.06
3.43
0.07
3.40 + 0.04
3.66
- 0.11
3.43 1: 0.07
CIFARIO WRN 256
2.84 1. 0.04
2.88
0.06
2.84
0.05
2.83 1: 0.03
2.80 0.00
To more precisely quantify HyperBO's ' advantage, we also computed how much faster HyperBO
can get a better error rate than best alternatives, which can be different from task to task. We found that on average, on over 50% tasks, H NLL is at least 2.86 times faster than best non-HyperBO alternatives; while on over 57% tasks, H8 KL is at least 3.26 times faster than best non- non-HyperBO alternatives. Moreover, over 73% tasks, H NLL is at least 7.74 times faster than random search; and on over 75% tasks, H KL is at least 6.07 times faster than random search.
E.2.2 EFFECT OF NUMBER OF TRAINING TASKS
We now investigate the impact of number of training tasks on the performance of meta BO methods. In Fig 8 we show the BO simple regrets on tasks from Table (except ImageNet ResNet50 2048) that use meta BO models trained on different number of training tasks. To analyze the performance of all methods on less-related tasks, we first remove training tasks that have the same task dataset as our current tuning task for testing, and then remove randomly selected training datasets from the rest.
HyperBO variants were able to reduce the simple regret as more training tasks are given. Interestingly, both H* NLL and H KL are already slightly better than Rand and STBOH when they started off with only training tasks. There are reasonable fluctuations in the results but overall the trend of regret is going down as the number of training tasks increases. MIMO also reduced regret when the number of tasks increased from 8 to 18. RFGP, however, fails to learn from training tasks possibly because it did not learn good task embeddings for GP regression models.
E.2.3 EFFECT OF NUMBER OF DATA POINTS IN TRAINING TASKS
One remaining question is. how does Mj in $2, the number of data points in each training tasks, affect the performance of meta BO methods. We analyze the impact of M; by removing portion of all data that we have access to for each task. In particular, we set the percentage of remaining data to be 0.2%, 0.5%, 1%, 3%, 5%, 10%, 30%, 50%, 70%, 90%. Remaining datapoints are selected uniformly randomly, which breaks the structure of matching data. Hence we do not include H* KL in this comparison, as H* KL only makes use of matching data.
Fig. 9 shows how the simple regret changes as the fraction of training data grows. Below 10% training data, observe clear trend that more data lead to lower regret for both H* NLL and MIMO. and relatively no change for RFGP. We also found that the performance of HyperBO (H* NLL) does not change much the fraction of training data increases from 5% to 90%. However, MIMO and RFGP
17

---

Under review as a conference paper at ICLR 2022
Number
Number
STBOM
KPGP
RL
0.043
0.035
0.000
0.123
0.025
0.003
0.0273
0.0093
0.0233
6.000
0.002
MB4D
MBKO
ye WLLL
Number
Number
Number trairing
Number tasks
c.18
6.014
0.013
3.005
0.010
0.003
3.355
c.00
0.005
0.005
0.004
2.005
0.004
1880
SEL
te BLL
Figure 8: Aggregated BO results on 23 tasks (all in Table except ImageNet ResNet50 2048 because of insufficient data) that uses models trained on 3 to training tasks. Note that the models are never te
trained on the data from the test task that we run BO on. If the number of training tasks is less than 23, we first remove the tasks that involve the same task dataset as the test task and then remove others randomly until reach the designated number of training tasks. The top left shows the median and 20/80 percentiles of regret on best validation error rate for each method. The rest are violin plots showing the regret for MIMO, H* NLL and H KL, where white dots indicate the median and black lines the 20/80 percentiles.
suffers significantly from more data as the fraction of training data increases from 5% to 50%. It is entirely clear why MIMO and RFGP have such behaviors. One conjecture is that neural network based Bayesian linear regression models may get too confident once the amount of data reaches certain threshold. This means much less exploration those models are used for BO.
E.2.4 TRAINING ON ALL BUT ONE TASK
We also studied the where meta BO approaches have access both training tasks that do not use the same task dataset and training tasks that use the same task dataset but different model configurations. This especially common when we do architecture search: we aim find the best model and we are tuning the optimizer hyperparameters for new machine learning model given tuning data of the same task dataset on some other models.
For this section only, we added new baseline, MAF: al. (2020) as MAF (Meta Acquisition Function) to avoid confusion. MAF used reinforcement learning to learn acquisition function modeled by neural network over a set of transfer learning tasks. All MAF results were generated using the code from Volpp et al. (2020). See App. E,6 for experimental details. As MAF takes significantly longer than HyperBO and other methods, only include its results for this section.
We refer to the meta BO method from Volpp
We carried out series of leave-one-out experiments, where picked task the BO test function and let meta BO methods train on the remaining tasks. In Fig. 10, we aggregated results from all 23 tasks to show the trend of how each method performs.
The conclusions are similar to those from $E.2.1. As expected, STBO without any tricks to avoid pitfalls of vanilla BO did not show very good results. We inspected its learned GP which mimicked
18

---

Under review as a conference paper at ICLR 2022
Praction training 0.002
Froction 0.005
STBOM
0.003
0.015
Regiet
0.013
0.02
0.002
Fract training
Fraction
Fraction data
0.055
0.025
2.005
6,025
0.023
0.000
2019
0.013
0.010
0.005
0.00
SEL
1290
Figure 9: Aggregated BO results on 23 tasks (all in Table except ImageNet ResNet50 2048 because of insufficient data) that uses models trained on 0.2% to 90% of data in each task. Note that the models never trained the data from the test task that and 20/80 percentiles of simple regret in log scale. The rest of the figures are simple regret violin plots for MIMO and H NLL
BO The top left is the median
00175
00175
10-1
00125
0.223
00075
0.073
1071
0 0000
00009
0005
0.0000
0.0003
ST8OH
MBMO
MLL
STROS
Mato
30 lars
Figure 10: Aggregated leave-one-out BO convergence results 23 tasks, each with 5 repeats using different random seeds. The left most is the median and 20/80 percentiles of the regrets on error rates. We also show violin plots its two vertical slices 50th and 100th iteration, where the white dot is the median and the black line is the 20/80 percentile.
Dirac function that is flat almost everywhere except some locations, and hence it got very confident that it landed at a good spot and lost its ability to explore.
STBOH, the other hand, achieved very competitive results. This is because it used hand-tuned priors on all of its GP parameters, although they were tuned on somewhat different problems than the consider. As part of the goals of meta learning, would like to show that it is possible for meta BO methods to exceed or at least match STBOH.
Both HyperBO variants obtained better results than the hand-tuned STBOH. Especially in the beginning few BO iterations, it is able to locate much better hyperparameters than other methods.
Tab. 4 presents mean and standard error of the best validation
rates achieved in 100 BO iterations on the 23 tasks. HyperBO and its variants were able to achieve the best performance on 20 out of 23 tasks. In Fig. 11, we show the optimization curves of 4 individual tasks that considered most
19

---

Under review as a conference paper at ICLR 2022
ImageNet ResNet50 512
LM1B Transformer 2048
Rand
STBO
10-1
10
STBOH
MIMO
RFGP
Regret on error rate
MAF
10:3
H" NLL
10
H" KL
Rand
10
STBO
Regret
STBOH
MIMO
10
10
RFGP
MAF
H" NLL
H" KL
10
80
100
80 Iters
80 Iters
WMT XFormer 64
Uniref50 Transformer 128
10-1
10
10
Rand
Regret on error rate
10-3
STBO
STBOH
10-4
MIMO
Rand
10-5
STBO
Regret
MAF
ST8OH
H"
MIMO
10
H" KL
10
10
H" NLL
20
40
60
80
100
0
20
60
80
100
80 iters
BD Iters
Figure 11: Leave-one-out log regret mean and standard deviation results on ImageNet ResNet50 512, LMIB Transformer 2048, WMT XFormer 64 and Uniref50 Transformer 128. All methods were repeated 5 times with different random seeds to initialize their models. In LM1B Transformer 2048, H* NLL and H* KL disappeared around 60 to 80 BO iterations because they reached 0 regret.
difficult because few similar task datasets present in their training data. On all of these 4 difficult tasks, HyperBO identified good hyperparameters much sooner than its competitors.
E.3 RESULTS ON ONLINE OPTIMIZER HYPERPARAMETER TUNING TASKS
Finally, we look into the online BO setting where we optimize over the full hypercuhe. the online setting, some combinations of hyperparameters may be infeasible evaluate. For example, an overly big learning rate may lead to divergence in gradients, in which case we do not obtain valid model. To address this, we pre-process the function values to 2- 2) such that infeasible evaluations map to -2, while bad evaluations approach asymptotically to 22 More precisely, for each subdataset D,,- applied for each successful y {y 1
the following mapping:
softplus(y- - T) +4 2
softplus ymnax y)
where y the median of (1y0 Mr M
20

---

Under review as a conference paper at ICLR 2022
Table 4: The mean and standard error of best validation error rates (%) for each test task the offline leave-one-out experiments. We show results of the top 6 methods, and we highlight the lowest error rates in bold.
Rand
STROH
MIMO
MAF
H NL.
H² KL.
WMT XTvemer 64
34.27
0.16
34.15 1 0.15
34.40 E 0.13
34.09 4 0.09
33.91
0.00
33,97 I 0.02
UnireF90 Transformer
70.05
0.04
78.92 = 0.12
79.17 = 0.13
79.34
0.22
78.71 0.06
78.64
LM Transformer 2018
61.99 2 0.03
61,95 0.04
61.06
1 0.05
62.02
1 1.0.1
61.81 #0.01
61.81 + 0.01
SVHN WINN 1024
0.04
4.05
1 0.10
3.83
0.04
4.10
0.09
4.10 10001:0000 0.02
+ 0.01
SVIIN WINN
3.71 0.01
3.72
0.02
3.65 0.01
3.09
1 0.03
3.18 1 0.01
3.72 1: 0.03
ImageeNr ResNetbii 256
24.04
0.07
22.06
4 0.07
22.73 + 0.07
26.44
4 1.98
22.53 4. 0.00
22.38 # 0.04
Image.Net ResNet50
23.02
22.74
23,01
0.05
25.46
22.63 0.002
22.79
MNIST I LNNPoolTann 2048
0.05 + 0.01
0.63
0.01
0.63 1663:100 0.01
0.62
+ 0.00
0.59 D0.02
0.54 # 0.00
MINSTT UNNPooe/inn Tunhh 256
0.01
0.48 1 0.01
0.47 1.774.00
0.47
3 0,001
0.46 1 0.01
0.47 0.001
MNIST LNNPoolReLL 2048
0.01
0.73 + 0.02
0.67 0.00
0.68
0.01
0.64 + 0.00
0.70 0.03
MINITT LNNPociReLL 256
0.51 + 0.01
0.55 + 0.03
0.50 + 0.01
0.51 0.01
0.49 I 0.00
0.49 1. 0.000
CNNRReLU 2048
1.06 + 0.00
GINISTUUNReLLA LNNKRLLU 256
1.09 + 0.02
1.06 + 0.01
1.08 + 0.02
1.07
0.02
1.09 + 0.00
1,04 # 0.01
Prolhicn CNNPosiTanh 2048
7.14 0.06
7.10 + 0.05
7.01 + 0.04
7.12 0.04
7.00 - 0.04
7.02 0.007 0.07
CNNFoolTanl
0.03
0.18
=
6.47
0.03
6.40 0.04
6.34 0.04
Frohticn CNNPoolRed 2048
7.47 + 0.02
7.48 1.48::000 0.04
7.54 + 0.06
7.63 0 0.04
7.47 + 0.03
7.47 + 0.02
CNNPooIReLU 256
6.78 0.04
6.74
0.07
6,84 0.05
6.74 0.08
0.06
Fishion CNNRel 2048
7.70 0.03
T.47 0.009
7.60 0.04
40.40 17.80
7.54 + 0.01
7.57 0.02
Fishaoo CNNRoLU 256
7.70 0.004
7.46 0.11
7,84
24.13 14.64
7.20 = 0.06
7.26 0.005
21.75
15.44
20.82
CIFARTIN TII WRN 256
10.17
19.02 + 0.03
19.12 0.2200004 0.04
19.84
0.13
19.00
0.000
19.04 # 0.05
CIFARIO WRN 2018
3.73 E 0.05
3.43 1:43:00 0.07
3.46 0.05
3,40 0,00
3.55 0.10
3.43 0.005
CIFARTO WRN 256
2.84 = 0.04
2.88 0.06
= 0.06
3.04 2.04::00 0.05
2.82 2: 0.03
2.74 0.01
In this section, we set HyperBO variants and STBO to share exactly the same GP-UCB acquisition function STBOH, MIMO and RFGP. The UCB coefficient for all methods is 1.8. The variants of HyperBO are as follows:
H* NLL: HyperBO with UCB the acquisition function and negative log marginal likeli- hood (NLL) as the objective function.
H* NLLKL: HyperBO with UCB as the acquisition function and NLLL plus 10 times KI. divergence on matching datapoints as the objective function. See SC for more details.
In Fig. 12, we include the online tuning results for selected tasks due to limited compute resources. We noticed that for some methods, e.g. STBO and MIMO, it is very difficult for them to recover from a "bad" datapoint. This partly because predictions from these models are significantly tied the initial observations. For example, STBO may overfit to the initial bad value and believe there are bad values the entire search space. Nevertheless, in 7 out of 9 tasks, HyperBO methods performed the best among all methods being compared.
E.4
IMPACT OF OBJECTIVE FUNCTIONS
Here we investigate how different objective functions in HyperBO can impact its performance. Besides NLL and KL, which are already described in details in $3, we also include NLL+KL, which corresponds Eq. 8 with À= = 10. À is an arbitrary choice, and one may find other better options to set X in Eq. 8.
Figure 13 shows the performance profiles and BO simple regret curves of NLL, KL and NLL+KL when HyperBO uses different acquisition functions. As comparisons, we also included the better performing baselines: Rand, STBOH and MIMO. While the ranks of performance by different objectives do vary depending which acquisition function used, it is clear that all HyperBO variants outperform the baselines. For El and PI, the KL objective gives better performance, but NLL or NLL+KL might be preferred for UCB with coefficient 2, 4.
E.5 IMPACT OF ACQUISITION FUNCTIONS
As explained briefly in $4, we used 5 acquisition functions in our experiments: the vannila El method, Ao (x) (a)) 1o--maxx. +0.1) (v.1)) and UCB with coefficient
Pl with coefficient 0.1, P (2: (p; & Dj)
< = 2, and 4 in a UCB (;; (x;PP(R (Dr) = (D)(2) + (o),
21

---

Under review as a conference paper at ICLR 2022
CIFARLOO WRN 256
Fashion CNNReLU 2048
MNISTT CHIVRELU 256
STB0
STBO
ST3O STOOH MIND
STOOH
STEOH
MIMD
MINO
NILL
11" NLLKL
NILK.
FE" NLLKI
22.0
MNIST CNNPPo Tann 2048
ImageNet ResNet50 512
SVHN WAN 1024
5TB0
RTOP
1ST NILL
STAO
H1t NILK..
STRO
SISCH
STSOM
MIRD
MIPOO
RTGP
REGP
H" NLLKL
KLLKL
LM1B Tranformer 2048
Uniref5C Transformer 128
WMT Xformer 64
S190
STBO
S130
STACH
OTACH
MRMD
MINO
MEMO
RFGP
RFGP
RFGP
NLL
17:17
20.00
3205
Figure 12: Results of running BO methods in the online setting 9 different tasks. The image based tasks all use best validation eπrоr rate as objective while the text based tasks (LMIB, Uniref50 and WMT) use best validation ce loss. HyperBO methods achieved better results in of tasks.
Our goal is to verify that HyperBO maintains good performance across different choices of acquisition functions. To do so, we investigated in the performance HyperBO variants under different objectives. We avoid over cluttering the figures by only including STBOH as baseline, since it is roughly the best baseline according to the main results Fig.
As shown in Figure 14, HyperBO with El and PI acquisition functions perform relatively better than HyperBO with UCB variants. However, HyperBO with UCB3 can still be very competitive when it is coupled with NLL objective. Overall, HyperBO with all of the 5 acquisition function options outperforms the best performing baselines.
E.6 MAF IMPLEMENTATION DETAILS
We compared (Volpp et al., 2020) using the code and default hyperparameters provided by the authors." This code assumes that each task is additionally accompanied by the optimal set of hyperparameters for the GP used to model the task (including the task used for evaluation). Following the MAF approach, we learned these hyperparameters using the GPY library, and provided them to
22

---

Under review as a conference paper at ICLR 2022
"Sperto y0r1 with iiition
37
Hypero 1911 wtth scouation
80 Hers =
D0 hara 320
2410
MLL
0033
Esperio 2/r1 with libiion
Ihpprro acOUE
DO Por 350
57884
00030
2:010
00050
D0 lan rwwddd oupatan Dest wra ude tw
60
Praperest
0.255
6.00
202
Tyyc:/50 IHF), with oucasation U134
Papee 20 (ITT with wwppaiaaaa un
Bi 230
Sead
2.0113
MM10
MLL.
0433
21015
MLL
MLL-3)
0375
0000
Hyyoortt Hett with pccuuittion unss
39"
hyppered (-1) with anc 20:00
WO Eric 100
STBBO
0.030
MLL
ML--1
5.0915
0.055
5221
We compare the performance of different objective functions in HyperBO under settings of acquisition functions. For EI and Pl, using KI. as the objective for HyperBO is slightly better than NLL or NLL+KL. However, different conclusions can be drawn for UCB2, UCB3 and UCB4. Nevertheless, all HyperBO variants still outperform the best alternatives.
Figure 13:
the MAF algorithm. Given that MAF takes significantly longer to run than HyperBO, each subdataset was evaluated using only one random seed.
Each neural acquisition function was trained for a total of 1000 iterations. As was done in (Volpp et al., 2020), we selected the optimal training iteration for the neural acquisition function by cross-validation on the transfer learning tasks; this case, we randomly sampled 3 transfer learning task, and chose the training iteration with the lowest average simple regret.
Finally, to reuse the MAF code, we also had to ensure that (a) each subtask had the same number of evaluation points, and (b) that there were duplicated hyperparameters. For this reason, first removed all duplicate hyperparameters within each subdataset, then capped each subdataset to the first 1559 points (the size of the smallest sub-dataset).
23

---

Under review as a conference paper at ICLR 2022
Pyper30 ('1) with oéjective NLL.
HyserBO (11) with otjective 11
Typer50 (P-1 with oojective VLL+-LL
2TE44
STECK
00
113000 UCD4
Trague ci
644
50 reaced rusddd croteeeters meder but rate 1000 our
lan oo oaguiam tam mades bec racea tw
13tmm reaced www.opooooooooooo radeer at 2020
Eypertt oing bejectioo
57334
3.35
23
3,00
9.122
1n UCB
1.313
3.80
1000
STEO
tryerrio
0.000
2013
117
3312
3,000
X0X
140
HyperrB cijective MiLWHEL
0.000
2.375
UCBS
3.500
8.2022
3.322
8.224
271
3.05
8000
0,000
30
20
300
Figure 14: We compare the performance of different acquisition functions under settings of objectives in HyperBO. Overall, PI and EI outperform UCB with different coefficient values. But HyperBO with UCB variants still outperforms STBOH, which roughly the best baseline according to the main results in Fig. 6.
DISCUSSION
In this work, we focused on the question of how to make use of multi-task data to enable better Bayesian optimization. For the convenience of our investigation, we made simplifications such as sequential evaluations and shared search space across tasks, although these are mostly unnecessary (see below). Our method also relies on an important assumption: functions of all tasks are i.i.d. samples from the same GP. In this section, explore how reasonable the i.i.d. assumption is and discuss extensions to our work that would enable even more flexible uses.
Assumption i.i.d. GP samples. To get better idea how much assumptions helped training the GP, we compare NLLs associated with 23 tasks in $4.1 I with models obtained via scenarios:
24

---

Under review as a conference paper at ICLR 2022
(a) No training: randomly initialized model with training:
(b) Single task: models trained on 100 randomly selected data points of the test task;
(c) H*: models trained on 18 irrelevant tasks selected in $E.2.2.
Here case (c) corresponds to the method HyperBO used for training a GP and case (b) corresponds to the model STBO can obtain with 100 initial observations. In Tab. 5, we show NLLs of these 3 methods on all tasks and NLLs on the test task. Note that the held-out tasks for some test tasks are the same because of the held-out rules in $E.2.1.
Comparing NLLs of the test tasks using models without training and trained via marginal likelihood like STBO, it is perhaps surprising to see that training on subset of data points of the sub-dataset of the test task not only did not contribute to lowering NI.L on the entire sub-dataset, but it even made worse in 20 out of 23 test tasks. The training process by optimizing the NLL on part of a sub-dataset leads to severe wver-fitting. We can observe the same results of NLLs on all tasks. Without any training, our NLL is 148211.2. Yet single-task training leads to higher NLLs for all models trained on different dat-satsssss.
Our method H*, on the other end, consistently achieves lower NLLs both the test task and all tasks. Although it is not entirely clear what the relation is between a better NLL of the GP and better BO results, achieving lower NLLS typically means that the model has better fit the dataset. Hence, by the assumption of typical BO methods, the test function should look like a sample from our model, and so lower NLLs of model will contribute to matching the assumption of typical BO methods. By enhancing the assumption with ours on i.i.d. GP samples, Tab. 5 shows we then will be able to obtain models with a much better fit to the data. Table 5: NLLs on 23 tasks and (pseudo) KL divergences on matching datasets with trained and randomly initialized GP models. The NLL of randomly initialized model (No training) on all tasks is 148211.2. The KL value of randomly initialized model (No training) is 2177.2. Training on a subset of a sub-dataset in the test task (Single task) often leads to much worse marginal likelihood on the entire sub-dataset. Training on irrelevant tasks (H*) achieves much lower (pseudo) KLs on matching datasets and lower NLLs for both the test task only and all tasks.
NLL the test task only
NLL all tasks
(Pocddo) KL
Test task
No training
Single task
11
Single tusk
11
Single task
H
WMT XiSormer 64
301.1
159.1
1735.0
1147900.5
2264.5
9651.9
40.22
Uniref30 Transtormer
-65177
-6829.4
185000
1063418128.0
867.9
310072.2
25.1
LMIB Transformer 2048
-510.6
2009.7
1692.7
18840158. D
3565.7
57744.1
23.5
SVHN WRN 1024
9703. 1
72407.5
4267.1
3399330.0 0
9346.5
4677.9
-0.9
SVHN WRN
10770.0
53245.5
3794.8
1164804.5
9346.5
3092.7
ImageNNt ResNet50 256
1196.7
7483.0
-746.3
7925583.5
7442
15028,1
30.6
ImageNet Res 312
1300.2
6930.3
673.1
1778823.5
-74.2
9462.1
- 30.6
MNIST CNN Janh 2048
10079.7
38871.9
794.8
1375930.1 1
97.0
3163.5
32.4
MNIST CNNNoool Tanh 256
12147.7
25607.9 9
550.0
556264.6
606.0
T
41.9
MNIST CNNPoolReL.L 2048
26870.5
7149.3
5506.6
46538.2
1542.2
113.8
59.4
MNIST CNNPooIReLU 256
15601.6 6
6734.6
51.0
88687.7
782.2
361.2
41.5
MNISTT CNNRRLU 2048
13939.2
40619.2
3153.2
743233.1
231.4
877.6
61.7
MNIST CNNRRLU 256
10111.0
3411244
1365.3
977295.0
-779.8
1373.3
46.2
Fashion CNNNool Tanh 2048
2072.8
T1433.0
381.0
1139702.4
1051.7
1910.5
37.8
Fashion CNNPool Tanh
2800.7
4115.6
251.4
1278018.0
1051.7
4208.3
37.8
Fashion CNNPoolRel.t 2048
4677.4
725.2
405.2
691 73.3
1051.7
205.1
37.8
Fastioon CNNPoolR 256
3925.7 7
4254.4
755.7
296739.. 1
1051.7
1027.1
33.8
Fashion CNNRcLU 2048
1607.3
6778.1
251.9
193/188. 4
1051.7
597.0
33.8
Fashion CN 256
3295. 1
29348.6
235.1
1526820.2 2
1051.7
3341.4
-37.8
CIFARTOD WRN 2048
1271.3
16813.7
467.4
3306556.6
312.3
25503.7
- 19.2
CIFARIOD WRN 256
1957.6
5950.8
510.9
3468309.0
11.7
9288.4
25.9
CIFARIO 2048
5220.6 6
4917.6
832.9
334488
1127.1
1040.4
CIFARIO WRN 256
7819.1 1
32996.8
463.4
895691.
2
847.4
1946.0
19.6
We also computed the (pseudo) KL divergence across all matching datasets in the last columns of Tab. 5. See Appendix C for a comprehensive analysis on pseudo KL divergence for degenerate multivariate Gaussians. Note that pseudo KL. divergence can be negative. Here we use pseudo KL divergence if required by the matching dataset. Again, single-task training leads to unstable (pseudo) KL values, sometimes even higher than without training (2177.2). On the contrary, training with H* leads to much more stable and lower values for KL. This indicates that the model learned to predict
All tasks include ImageNet ResNet50 2048. But it is excluded in the test tasks in Tah. 5 because it has much fewer data points than the others.
25

---

Under review as a conference paper at ICLR 2022
similarly to the sample mean/covariance estimate, which is known help better selection of BO query points by Theorem 1.
Batch evaluation. For simplicity of this paper, we did consider batch evaluation but rather only focused on the prior selection dimension of the challenges in BO. However, it is straightforward to adopt any batch BO methods in conjunction with HyperBO support obtaining observations in parallel, For example, we can directly use batch methods in Snoek et al. (2012), Kathuria et al. (2016), or Wang al. (2017) to replace line Alg.
High- ligesiiimeniooall and large scale data. Similar to batch BO, method can also be naturally combined with most high-ding sigh-dimensional and large scale BO methods to offer more capabilities. For these cases, typically probabilistic model different from vanilla GPs may be adopted. In line 2 of Alg. 1, it is straightforward to adapt our method to optimize the cumulative marginal likelihood in Eq. 3 instead for the model. Our meta-learning idea in this paper in fact also brings benefit high-dimensional and large scale BO methods so that they can better identify their critical special structures, e.g. low-dimensional embedding Wang al. (2016), cylindrical kernels Oh et (2018) or additive Mondrian kernels Wang et al. (2018a).
Different search spaces. Roughly speaking, there could two circumstances for difference search spaces. Case I is that tasks share the same search variables, but the search ranges for some variables are different. For example, we may have each function fa X, - R, E [A] and x; = 14 hayl C Rd. In this case, our solution still applies by simply setting a union search space as X = - X; for learning and use the designated search space of new tasks for optimization.
Case more complicated: the search space for each function fi is X, C Rd: and each dimension of X, may have different meaning than another search space X2 (i / j). This paper does have solution for this scenario. Further research will be needed to reduce Case II to Case I which can be then immediately combined with HyperBO.
26


Reviews:
[{"review_id": "xgngFKZvUD0", "reviewer": "Reviewer_jdBZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Reasons to Accept\n\n+ It is well-written and well-organized.\n+ It solves a very interesting problem, which transfers a history to the current task in Bayesian optimization setup.\n+ Compared the work by Wang et al. (2018b), it solves more realistic setups.\n+ It provides promising numerical results and sound theoretical results.\n\n### Reasons to Reject\n\n- I do not think that it degrades the contributions much, but four-dimensional search space is relatively small, compared to other Bayesian optimization or hyperparameter optimization papers.\n- Following the above point, is there any specific reason why the authors use four-dimensional search space? I do not think this algorithm is not scalable. Moreover, for example, batch size can be one of the meta-parameters to be optimized.\n\n### Questions to Authors\n\n1. Can you elaborate why the proposed method does not train a GP model every iteration, e.g., every $t = 1, \\ldots, T$? I think that it can be possible without (relatively) expensive computational costs.\n1. H* NLL does not use a matching dataset, right? If you did not use multi-task GP regression, which has an additional input to indicate task information, does H* NLL (i.e., optimizing Equation (2) with $D_N$) work appropriately? ", "summary_of_the_paper": "This paper suggests a meta Bayesian optimization strategy that optimizes free parameters of GP including a prior function and noise variance, where multiple sets of historical observations are given. In particular, the proposed method chooses a free parameters using one of three approaches: (i) optimizing a marginal likelihood, (ii) measuring KL divergence, (iii) considering both marginal likelihood and KL divergence. The authors finally show the theoretical analyses on regret bounds and the numerical results on hyperparameter optimization.", "summary_of_the_review": "I think that this paper addresses an interesting problem and suggests a novel method as described above. Thus, I would like to recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d1IEzpn0Fen", "reviewer": "Reviewer_7A1X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of \"warmstarting\" HPO by making use of data from previous experiments is an obvious idea, and it has seen a large amount of past work, much of which the authors of this submission do not seem to be aware of, neither apparently was (Wang, 2018b) which seems more of a theoretical paper. In particular, there is quite a lot of work which uses GP models and scales linearly in terms of the number of past experiments, contrary to what is stated in the introduction. Two of the most interesting ones are maybe [1], [2]. The authors here cite (Perrone, 2018), which has these citations and more, so it is pretty odd the authors do not mention (or compare against) any of them.\n\nGiven the straightforward nature of what is proposed here (a setup closely related to what is done in [3]), I'd be quite surprised if for example [1] would not outperform it. After all, the assumption that data from experiments on quite different models can be modeled by the same mean and covariance function, is pretty strong. There are all sorts of issues with this idea, for example what if data from some tasks is much larger than data from others? Moreover, in what is proposed here, the surrogate model parameters do not even seem to be adapted to the current task, even as data from it becomes available. Here, methods like [1], [2] seem much more compelling to me, as they try to for example rank previous experiments by closeness to the current one. [1] is doing this without having to define any meta-features of the dataset, and also of course without relying on observations at the same configurations (given you model your data with a GP, you should certainly not need that anyway).\n\nThe experiments are not meaningful, because essentially all relevant prior work is missing for comparison. The authors more or less compare their proposal (in two variants) against a bunch of baselines, as if there was no revelant prior work. In fact, they even seem to invent on their own methods to compare against, such as \"MIMO\", in a way which has never been used for transfer HPO. Why? Please read about and compare against relevant prior work. Given they cite work (e.g., Perrone 2018), they should have been aware.\n\nApart from that, I also do not get much out of the experimental setup. Why was it chosen that way? Does it have any practical relevance? Does anybody else use this learning rate schedule, or was it just made up for this paper? I also did not find a discussion of a pretty critical point: how are the datapoints chosen for tasks you offline train on? In order to be realistic, these would have to be active choices themselves, because that is data we could have been obtained by running BO on them. Instead, my suspicion is that past data was sampled randomly, which would correspond to pure exploration (random search). Such data is obviously more valuable to obtain a good surrogate model fit, but also more expensive to obtain in the real world (one would have to run random search).\n\n[1] Feurer etal: Practical Transfer Learning for BO, https://arxiv.org/abs/1802.02219\n[2] Wistuba etal: Two-stage transfer..., ECML 2016\n[3] Golovin etal: Google Vizier, KDD 2017", "summary_of_the_paper": "This paper is concerned with speeding up Bayesian optimization by using evaluation data from previous, related tasks defined over the same configuration space. The authors propose to model the data from each experiment (or \"task\") by independent Gaussian processes, which all share the same mean and covariance function. This surrogate model can be learned from past data.\n\nThe paper also presents experiments on a fairly simple search space of 4 optimizer parameters. This is done for a bunch of datasets and NN models. And there is a pretty simple extension of theoretical results from (Wang, 2018b).", "summary_of_the_review": "This paper proposes a simple idea for wam-starting BO by fitting the parameters of a GP surrogate model on past data. Unfortunately, a lot of relevant prior work is ignored here and not compared against. Instead, the proposed approach is compared against simple baselines, as well as methods that mostly seem to have been made up (such as \"MIMO\").\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DsgJW48s8ud", "reviewer": "Reviewer_dwbG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The reviewer appreciates the authors putting effort into the empirical evaluation of the proposed method. However, the proposed approach is not interesting to the Bayesian optimization community and is trivial to some degree. The reviewer believes that the targeting problem presented in this work is a very important one and an effective method could be of great practical value. \n\nIn the abstract, authors claim that \"data from similar functions\" could lead to a better prior for GP. Obviously a better prior for GP is desirable and that is why the marginal likelihood is used to optimize  parameters of a GP. From such a claim, it is expected that an efficient method for BO will be presented by exploring novel similarities between tasks. However, throughout this paper, there is no definition of a similarity between tasks and tasks are treated as independent. This raises my concern on this work's novelty, which is my biggest concern on this paper.  \n\nAuthors claim that the critical difference between this work and standard BO algorithms is the initial learning process in line 2 of Algorithm 1. The corresponding likelihood of this approach is given in eq(2). I do not get the point how this approach is different from existing GP modeling and eq(2) is simply the unnormalized marginal likelihood for all data points since all tasks are assumed to be independent. Such a formulation is not only trivial to the GP community , but also to the empirical Bayes community. \n\n\nAdditional (minor) issues:\n1. the graphical model for GP in Figure 1 is wrong\n2. there exist a lot of inconsistencies in this paper. In assumptions section, it is assumed the variance is known however the variance is a hyper-parameter in the marginal likelihood. \n3. lots of claims and statements are superfluous. For example, authors claim one limitation of existing approaches is the total number of BO iterations must be set in a manual way. However, throughout this paper, the number of iterations is still pre-defined. What is the point of saying this is a limitation while not touching it at all? Another example, authors claim  \"interpretability of intermediate steps\" is lost in existing methods, however, this problem is not touched either. \n4. Another contribution of this paper is a tuning dataset. I can see the value of such a dataset, however, failing to explicitly describe the required computation resources makes claiming this being a contribution less convincing.", "summary_of_the_paper": "This paper presents a Bayesian optimization method based on meta-BO. The motivation is tasks can share the same parameter structure and this shared information, e.g. correlation between tasks, can be transferred to new and similar tasks. An example is to optimize the the hyper-parameters of a same optimizer across different architectures and different datasets. This problem is a very important one in the community of Bayesian optimization and a reasonable method can lead to a potentially dramatic decrease in the required computation, especially when the objective function is very expensive. This work tries to overcome limitations of existing methods. For example, the method proposed in this work does not need to evaluate all objective functions associated with all tasks on the same parameters.", "summary_of_the_review": "The proposed method is trivial. The theoretical part presented in this paper is very minimal and incremental. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9GOaIhix4m_", "reviewer": "Reviewer_Neri", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have a few key concerns about this paper. \n\n- Why fixed hyperparameters? This is clearly the bottleneck of Metalearned BO, and if these hyperparameters are learned offline, this seems to (A) somewhat eliminate the strength of HyperBO which is the linear scaling per task ---obviously this still helps significantly during the offline training, but still a point of concern of mine, and (B) seems not robust, especially if the set of representative completed tasks is heavily biased. \n\n- HyperBO, in the experiments, uses the PI acquisition function. Is there a particular reason why this is? PI is quite greedy (even more than EI), so is there any intuition as to why PI is appropriate in this situation. \n\n- In Figure 2b, I am somewhat concerned about the empirical performance of HyperBO. Though it beats the baselines, it does so in a 4D search space, using thousands of tasks; this seems like overkill. The error bars are also all over the place. This is somewhat unfair of me to ask for I admit, but I am curious if a much simpler approach involving restricting the search space (given that it is fixed) will help (see the paper “Learning search spaces for Bayesian optimization, Perrone et al., 2019). I feel like there is definitely enough data for this to make a difference. \n\n- Also, the experiments only really concern one optimization problem involving optimizer hyperparameters. Though this one experiment is quite impressive in terms of the data involved, iit would be nice to see another experiment (say for tasks that might be easier like tuning a random forest). \n", "summary_of_the_paper": " HyperBO assumes the tasks are independent given the hyperparameters, unlike typical metalearning approaches which assume tasks are related. This allows for an efficient Kronecker decomposition of the kernel and thus linear, rather than cubic scaling, across tasks. \n\nUsing this model, HyperBO performs BO as usual; maximize the acquisition function to obtain the next point to evaluate. HyperBO also makes the critical assumption of an offline pre-training of hyperparameters on a representative set of completed tasks; during optimization itself the hyperparameters are fixed. \n", "summary_of_the_review": "I have some concerns about the assumptions used in the methodology, as well as the experiments, which leave a number of open questions. In particular, the fixing of GP hypers seems to largely remove the need for scaling, which is the primary strength of HyperBO. Furthermore, though the experimental set up uses a large amount of data to achieve somewhat unconvincing results in my mind, and only one optimization problem is presented (though worth noting, is thoroughly analyzed). Thus, I can't recommend acceptance at the time. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper claims a practical improvement over one of earlier meta BO methods. Warm-starting BO or HPO by making use of data from past experiments or tasks seems to be interesting and useful for some applications. In fact, there are a large amount of work on this topic, but a lot of relevant prior work is ignored in this paper unfortunately. I appreciate the authors for making efforts in responding to reviewers’ comments. However, after the discussion period, most of reviewers had serious concerns in this work, pointing out that the proposed method is rather trivial and the comparison is made only against a simple baseline. It was also suggested to improve the experiments. While the idea is interesting, the paper is not ready for publication at the current stage.
Decision: Reject