$\pi$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization
Authors: Carl Hvarfner
Danny Stoll
Artur Souza
Marius Lindauer
Frank Hutter
Luigi Nardi
Venue: ICLR 2022 Poster (2022)
Keywords: Bayesian Optimization
Hyperparameter Optimization
Meta-Learning

Abstract: Bayesian optimization (BO) has become an established framework and popular tool for hyperparameter optimization (HPO) of machine learning (ML) algorithms. While known for its sample-efficiency, vanilla BO can not utilize readily available prior beliefs the practitioner has on the potential location of the optimum.  Thus, BO disregards a valuable source of information, reducing its appeal to ML practitioners. To address this issue, we propose $\pi$BO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user. In contrast to previous approaches, $\pi$BO is conceptually simple and can easily be integrated with existing libraries and many acquisition functions. We provide regret bounds when $\pi$BO is applied to the common Expected Improvement acquisition function and prove convergence at regular rates independently of the prior. Further, our experiments show that $\pi$BO outperforms competing approaches across a wide suite of benchmarks and prior characteristics. We also demonstrate that $\pi$BO improves on the state-of-the-art performance for a popular deep learning task, with a $12.5\times$ time-to-accuracy speedup over prominent BO approaches.

Published as a conference paper at ICLR 2022
TBO: AUGMENTING ACQUISITION FUNCTIONS WITH USER BELIEFS FOR BAYESIAN OPTIMIZATION
Carl Hvarfner', Danny Stoll?, Artur Souza", Marius Lindauer', Frank Hutter & Luigi Nardi Lund University, University of Freiburg, Federal University of Minas Gerais, *Leibniz University Hannover, Bosch Center for Artificial Intelligence, Stannford University (carl. ..varferr, luigi. nardi } @cs. lth.ee, (stolld, 0c) mmni-freibur de, arturluis@den .fng br, Lindauer@tnt. uni haahoaen.nv.e.. de
ABSTRACT
Bayesian optimization (BO) has become established framework and popular tool for hyperparameter optimization (HPO) of machine learning (ML) algorithms. While known for its sample-efficiency, vanilla BO can not utilize readily available prior beliefs the practitioner has on the potential location of the optimum. Thus, BO disregards valuable source of information, reducing its appeal ML practitioners. To address this issue, we propose TBO, an acquisition function generalization which incorporates prior beliefs about the location of the optimum in the form of a probability distribution, provided by the user. In contrast to previous approaches, x BO is conceptually simple and can casily be integrated with existing libraries and many acquisition functions. We provide regret bounds when rBO is applied to the common Expected Improvement acquisition function and prove convergence at regular rates independently of the prior. Further, rBO outperforms competing approaches across a wide suite of benchmarks and prior characteristics. We also demonstrate that BBO improves the state-of-the- art performance for a popular deep learning task, with a 12.5x time-to-accuracy speedup over prominent BO approaches.
experiments show that
INTRODUCTION
The optimization of expensive black-box functions is prominent task, arising across wide range of applications. Bayesian optimization (BO) is a sample-efficient approach to cope with this task, and has been successfully applied to various problem settings, including hyperparameter optimization (HPO) (Snoek et al. 2012), neural architecture search (NAS) (Ru et al.
202T), joint NAS and HPO (Zimmer al. 2021) algorithm configuration (Hutter al. [011] I) hardware design (Nardi
et al.
2019), robotics (Calandra et al., 2014), and the game of Go (Chen et al.
2018).
Despite the demonstrated effectiveness of BO for HPO (Bergstra al. 2011 Turner al. 2021) its adoption among practitioners remains limited. In a survey covering NeurIPS 2019 and ICLR 2020 (Bouthillier & Varoquaux] (2020), manual search was shown to be the most prevalent tuning method, with BO accounting for less than 7% of all tuning efforts. As the understanding of hy- perparameter settings in deep learning (DL) models increase (Smith) (0088), so does the tuning proficiency of practitioners (Anand et al.) 2020 As previously displayed (Smith) 2018 Anand al.) 2020: Souza al.) 2021 Wang et al. 2019). this knowledge manifests choosing single configurations or regions of hyperparameters that presumably yield good results, demonstrating belief over the location of the optimum. BO's deficit to properly incorporate said beliefs is reason why practitioners prefer manual search BO (Wang et al., 2019), despite its documented shortcomings (Bergstra Bengio 2012). To improve the usefulness automated HPO approaches for ML practictioners, the ability to incorporate such knowledge is pivotal.
Well-established BO frameworks (Snoek et al. 2012 Hutter et al. 2011 I The GPyOpt authors 2016 Kandasamy et al. 2020; Balandat al. 2020) support user input to limited extent, such as by biasing the initial design, or by narrowing the search space; however, this type of hard prior can lead poor performance by missing important regions. BO also supports a prior functions p(f) via

---

Published as a conference paper at ICLR 2022
the Gaussian Process kernel. However, this option for injecting knowledge is not aligned with the knowledge that experts possess: they often know which ranges of hyperparameter values tend to work best (Perrone et al. 2019; Smith 2018 Wang al. 2019), and are able to specify a probability distribution to quantify these priors. For example, many users of the Adam optimizer (Kingma & Ba 2015) know that its best learning rate is often in the vicinity of x 10-3 In practice, DL experiments are typically conducted in a low-budget setting of less than 50 full model trainings (Bouthillier & Varoquaux 2020). As such, practitioners want exploit their knowledge efficiently without wasting early model trainings on configurations they expect to likely perform poorly. Unfortunately, this suits standard BO poorly, as BO requires a moderate number of function evaluations to learn about the response surface and make informed decisions that outperform random search.
While there a demand to increase knowledge injection possibilities further the adoption of BO, the concept of encoding prior beliefs over the location of an optimum is still rather novel: while there are some initial works (Ramachandran al., 2020 Li et al., 2020 [Suzaa et al. 2021), no approach exists so far that allows the integration of arbitrary priors and offers flexibility in the choice of acquisition function; theory is also lacking. We close this gap by introducing a novel, remarkably simple, approach for injecting arbitrary prior beliefs into BO that is easy implement, agnostic to the surrogate model used and converges at standard BO rates for any choice of prior.
Our contributions After discussing our problem setting, related work, and background (Section 2 we make the following contributions:
1. We introduce rBO, novel generalization of myopic acquisition functions that accounts for user-specified prior distributions over possible optima, is demonstrably simple-to- simple-to-implement, and can be easily combined with arbitrary surrogate models (Section 3.1 & (.2);
2. We formally prove that rBO inherits the theoretical properties of the well-established Expected Improvement acquisition function (Section 3.3):
3. We demonstrate on a broad range of established benchmarks and in DL case studies that BOO can yield 12.5x time-to-accuracy speedup over vanilla BO (Section 4).
BACKGROUND AND RELATED WORK
2. BLACK-BOX OPTIMIZATION
We consider the problem of optimizing a black-box function f across a set of feasible inputs X C Rd:
x= E arg min f(x)..
#Ex
We assume that (a: is expensive to evaluate, and can potentially only be observed through a noisy estimate, y. In this setting, we wish to minimize f in an efficient manner, typically adhering to a
budget which sets a cap on the number of points that can be evaluated.
Black-Box Box Optimization with Probabilistic User Beliefs In our work, we consider an augmented version of the optimization problem in Eq. ( where we have access to user beliefs in the form of a probability distribution on the location of the optimum. Formally, we define the problem of black-box optimization with probabilistic user beliefs as solving Eq. (I) given a user-specified prior probability on the location of the optimum defined as
π (xx) =P (f(z)= min
f(a')) x'ex
(2)
where regions that the user expects to likely to contain an optimum will have a high value. We note that, without loss of generality, we require π to be strictly positive on all of X, i.e., any point in the search space might be an optimum. Since the user belief (xx) can be inaccurate or even misleading. optimizing Eq. (1) given
a challenging problem.
2.2 BAYESIAN OPTIMIZATION
We outline Bayesian optimization Mookus et al. 1978 Brochu et al. 2010 Shahriari al. 2016b).
2

---

Published as a conference paper at ICLR 2022
Model BO aims to globally minimize f by an initial experimental design Du {(x,, 34 1 and thereafter sequentially deciding on new points 2n to form the data D,, = - U { yn for the n-th iteration with É {l...N]. After each new observation, BO constructs a probabilistic surrogate model of f and uses that surrogate to evaluate an acquisition function (< D.)) The combination of surrogate model and acquisition function encodes the policy for selecting the next point + When constructing the surrogate, the most common choice is Gaussian processes (Rasmussen & Williams 2006), which model f as p())D,) GP k), with prior mean m (which is typically 0) and positive semi- covariance kernel k. The posterior mean mn and the variance s are
N...
mn (2) (3) (KA + o
(x) - k(x, x) (x) (K,, a (x),
(3)
where (Kn)i k( Zy), kn (x) [A æ1). k(x, xx)] and o the estimation of the observation noise variance o². Alternative surrogate models include Random forests (Hutter et al. 2011) and Bayesian neural networks (Springenberg et al. 2016).
Acquisition Functions To obtain new candidates to evaluate, BO employs a criterion, called an acquisition function, that encapsulates explore-cxploi trade-off. By maximizing this criterion at each iteration, one or more candidate point are obtained and added to observed data. Several acquisition functions are used in BO; the most common of these is Expected Improvement (EI) (Jones et al., 1998). For a noiseless function, El selects the next point 1n+1, where ft is the minimal objective function value observed by iteration ni, as
5 En+11 arg mag IE [[VF- f(x)]+] arg max Zs, ( (z)))(2) TEX
where Z = (f% mn, (x)) sn, (z²). Thus, El provides a myopic strategy for determining promis- ing points; it also comes with convergence guarantees (Bull 201T). Similar myopic acquisition
functions are Upper Confidence Bound (UCB) (Srinivas et al. 2012). Probability of Improvement
(PI) (Jones] 2001 Kushner 1964) and Thompson Sampling (TS) (Thompson) [933). A different
class of acquisition functions is based on non-myopic criteria, such as Entropy Search (Hennig & Schuler, 2012). Predictive Entropy Search (Hernández-Lobens et al./ (2014) and Max-value Entropy Search WWag & Jegelkal 2017). which select points to minimize the uncertainty about the optimum, and the Knowledge Gradient (Frazier et al.) (2008), which aims to minimize the posterior mean of the surrogate at the subsequent iteration. Our work applies to all acquisition functions in the first class, and we leave its extension to those in the second class for future work.
2.3 RELATED WORK
There are two main categories of approaches that exploit prior knowledge in BO: approaches that use records of previous experiments, and approaches that incorporate assumptions the black-box function provided either directly or indirectly by the user. As rBO exploits prior knowledge from users, briefly discuss approaches which utilize previous experiments, and then comprehensively discuss the literature on exploiting expert knowledge.
Learning from Previous Experiments Transfer learning for BO aims automatically extract and use knowledge from prior executions of BO. These executions can come, for example, from learning and optimizing the hyperparameters of a machine learning algorithm on different datasets
(van Rijn & Hutter, 2018; Swersky et al.) 22113] Wistuba et al. 2015 Perrone et al., 2019 Feurer al. 2015]2018) 2018), from optimizing the hyperparameters
different development stages (Stoll et al. 2020). For a comprehensive overview of meta learning for hyperparameter optimization, please see the survey from Vanschoren (2018). In contrast to these transfer learning approaches, aBO and
the related work discussed below does not hinge on the existence of previous experiments, and can therefore be applied more generally.
Incorporating Expert Priors over Function Structure BO can leverage structural priors on how the objective function is expected to behave. Traditionally, this is done via the surrogate model's prior over functions, e.g., the kernel of the GP. However, there are lines of work that explore additional structural priors for BO to leverage. For instance, both SMAC (Hutter et al. 2011) and iRace (López) Ibáñez al.) 2016) support structural priors in the form of log transfornnatiinss Li et al. (2018) propose to use knowledge about the monotonicity of the objective function as a prior for BO, and Snoek al. (2014) model non-stationary covariance between inputs by warping said inputs.
3

---

Published as a conference paper at ICLR 2022
Oh al. (2018) and SSivola et al. (2018) both propose structural priors tailored to high- digh-dimensional problems, addressing the issue of over-exploring the boundary described by Swersky (2017). Ohh et al. (2018) propose cylindrical kernel that expands the center of the search space and shrinks the edges, while Siivola et al. (2018) propose adding derivative signs to the edges of the search space to Lastty,SSahriari search spaces which uses regularizer to penalize points based on their distance to the center of the user-defined search space. All of these approaches incorporate prior information on specific properties of the function or search space, and are thus not always applicable. Moreover, they do not generally direct the search to desired regions of the search space, offering the user little control over the selection of points to evaluate.
steer BO towards the center. Lastly, (Shahriari et al. (2016a) propose BO algorithm for unbounded
Incorporating Expert Priors over Function Optimum Few previous works have proposed to inject explicit prior distributions over the location of an optimum into BO. In these cases, users explicitly define a prior that encodes their beliefs on where the optimum more likely to be located. Bergstra et al. (0011) suggest an approach that supports prior belief's from a fixed set of distributions. However, this approach cannot be combined with standard acquisition functions. BOPrO (Souza et al. 2021 employs similar structure that combines the user user-provices provided prior distribution with data-driven model into a
From the aseudo-posterion configurations are selected using the El acquisition function, using the formulation in [Bergstra al. (201 While BOPrO is able to recover from misleading priors, its design restricts it to only use EI. Moreover, it does not provide the convergence guarantees of rBO.
Li et al. (2020) propose to infer a posterior conditioned on both the observed data and the user prior through repeated Thompson sampling and maximization under the prior. This method displays robustness against misleading priors but lacks in empirical performance. Additionally, it is restricted to only one specific acquisition function. [Ramachandran et al. (2020) use the probability integral transform to warp the search space. stretching sigh-probbbility regions and shrinking others. While the approach is model- and acquisition function agnostic, requires invertible priors, and does not empirically display the ability to recover from misleading priors. In Section 4 we demonstrate that rBO compares favorably for priors over the function optimum, and shows improved empirical performance. Additionally, we do complete comparison of all approaches in Appendix C
In summary, BBO sets itself apart from the methods above by being simpler (and thus easier to implement in different frameworks), flexible with regard to different acquisition functions and different surrogate models, the availability of theoretical guarantees, and, as we demonstrate in Section 4 better empirical results.
METHODOLOGY
We now present BOO, which allows users to specify their belief about the location of the optimum through any probability distribution. A conceptually simple approach, nBO be easily implemented in existing BO frameworks and can be combined directly with the myopic acquisition functions listed above. rBO augments an acquisition function to emphasize promising regions under the prior, ensuring such regions are to be explored frequently. As optimization progresses, the TBO strategy increasingly resembles that of vanilla BO, retaining its standard convergence rates (see Section 3.3) rBO is publicly available as part of the SMAC (ttttp::/// com/ omt /SMAC3 and HyperMapper
HPO frameworks.
3.1 PRIOR-WEIGHTED ACQUISITION FUNCTION
In rBO, we consider ((x) in Eq. b to be a weighting scheme on points in X. The heuristic provided by acquisition function (ax, P,), such as El in Eq. (2.2). can then be combined with said weighting scheme to form prior- prightee version of the acquisition function. The resulting strategy then becomes:
Fy E arg max aαx (a)=( wEX
(5)
This emphasizes good points under π7 (2x) throughout the optimization. While this property is suitable for well-located priors ग, it risks incurring a substantial slowdown for poorly-chosen priors; we will now show how to counter this by decaying the prior over time.
4

---

Published as a conference paper at ICLR 2022
3.2 DECAYING PRIOR-WEIGHTE ACQUISITION FUNCTION
As the optimization progresses, we should increasingly trust the surrogate model over the prior; the model improves with data while the user prior remains fixed. This cannot be achieved with the formulation in Eq. ( as poorly-chosee priors would permanently slow down the optimization. Rather, to accomplish this desired behaviour, the influence of the prior needs to decay over time. Building on the approaches of Lee et al. (2020) and [Souza et al. (202T), we accomplish this by raising the prior to a power "/n E IR, which decays towards zero with growing n. Thus, the resulting prior (x) < reflects a belief on the location of an optimum that gets weaker with time, converging towards a uniform distribution. We set % = 3/n, where 3 E R+ is a hyperparameter set by the user, reflecting their confidence in π (2:). We provide sensitivity study on / in Appendix A For a given acquisition function a: (z, Dn) and user-specified prior πα)), we define the decaying wrior-weighted acquísition function at iteration
T (x, P..) = a( (n)Tm(() = axx Pn)
(6)
and its accompanying strategy as the maximizer of Or,,. With the acquisition function in Eq. ( the prior will assume large importance initially, promoting the selection of points close to the prior mode. With time, the exponent on the prior will tend to zero, making the prior tend to uniform. Thus, with increasing n, the point selection of (x . becomes increasingly similar to that of a. Algorithm [ displays the simplicity of the new strategy, highlighting the required one-line change (Line in the main BO loop. In Line 3 the mode of the prior is used first initial sample if available. Otherwise, only sampling is used for initialization.
Algorithm 1 BBO Algorithm
1: Input: Input space X, prior distribution over optimum ((x), prior confidence parameter B, size M of the initial design, max number of optimization iterations N.
2: Output: Optimized design x².
3: {x,)-= ~ T (z). {ys- + f(x;)) + MMM E ~ (0,007
4: Do < (3.,, M1
5: for (n = 1,2 1.2....,. N do
6: Knew + arg max a(x (n-1)
7: Vnew 5 (I +G
8: Dn t Dn U ( knew)}
9: end for
10: return x= +- arg W.) y4
To illustrate the behaviour of rBO, consider toy problem with Gaussian priors on three different locations of the ID space (center, left and right) as displayed in Figure [ We define a OD-Log-Branin toy problem by setting the second dimension of the 2D Branin function to the global optimum D2 2.275 and optimizing for the first dimension. Initially (iteration in the top row), BBO amplifies the acquisition function a in high-probability regions, putting lot of trust in the prior. As the prior decays (iteration 6 and 8 in the middle and bottom rows, respectively). the influence of the prior on the point selection decreases. By later iterations, rBO has searched substantially around the prior mode, and moves gradually towards other parts of the search space. This is of particular importance for the scenarios in the right column. where rBO recovers from a misleading prior. In Appendix B we show that nBOO is applicable to different surrogate models and acquisition functions.
3.3 THEORETICAL ANALYSIS
We now study the rBO method from theoretical standpoint when paired with the El acquisition function. For the full proof, we refer the reader to Appendix E To provide convergence rates, we rely on the set of assumptions introduced by Bull 2011). These assumptions are satisfied for popular kernels like the Matérn 1960) class and the Gaussian kernel, which is obtained in the limit v/ - oc, where the rate controls the smoothness of functions from the GP prior. Our theoretical results apply when both length scales l and the global scale of variation o are fixed; these results can then be extended to the case where the kernel hyperparameters learned using Maximum Likelihood
5

---

Published as a conference paper at ICLR 2022
Well-loated Prior
Officenter Prior
Iteration 6 lltration
Iteration
X
XXXXX
1D Branin
Prior-weighted Acquisition Function
Acquisition Function
Decaying Prior
Figure Rescaled values of wrigr-weighted EI (purple), (blue) and mh (red) on ID-Branin in logscale (grey) with global optimum in the center of the search space. Runs with two different prior locations ("Well-located" slightly right of optimum, "Off-center" significantly left of optimum) shown in the two columns. Each row represents optimization run with 3 = The current selection can be seen as a vertical violet line, and all marked crosses. =BO amplifies El in gradually increasing region around the prior, and moves away from the prior as iterations progress. This is particularly evident in the Off-center example.
iteration (iteration 4, and 8), for
previous observations
Estimation (MLE) following the same procedure in Bull 2011) (Theorem 5). We define the loss over the ball By for function f of norm (lfll S R in the reproducing kernel Hilbert space (RKHS) H)(X)) given symmetric positive-definite kernel Ke as
Cu, Dn, He(x), R)
sup Eylf(x))) min fl, Illlege
where n is the optimization iteration and a strategy. We focus on the strategy that maximizes EI,, the prior-weighted EI, and show that the loss Equation (7) can, at any iteration n, be bounded by the vanilla El loss function. We refer to EI,,, and EI, when we want to emphasize the iteration n for the acquisition functions EI, and EI, respectively.
Theorem 1. Given Dr, Ke. T. 3, a, l, R and the compact set X C Rd as defined above, the loss Cn incurred iteration n by Eln, can be bounded from above as
|max x (ar)) min π (xx)
8/n
C. (El Dn, He(X), R) S Ca Dny He((),, Cr,.
Using Theorem T we obtain the convergence rate of El,. This trivially follows when considering the fraction of the losses in the limit and inserting the original convergence rate on EI as in Bull (011):
Corollary 1. The loss of decaying wrighteeighted Expected Improvement strategy. El,, is asymp- totically equal to the loss of an Expected improvement strategy. El:
L (n Dn, He(()), R) ~ (n Dn, He(X)),)
so we obtain a convergence rate for Elş of C, D,, He(x)) R) On ((n)
Thus, we determine that the weighting introduced by EI, does not negatively impact the worst-case convergence rate. The short-term performance controlled by the user in their choice of x(x) and 3. This result is coherent with intuition, performance closer to that of EL. In contrast, a stronger prior or slower decay does not guarantee the same short-term performance, but can produce better empirical results, shown in Section 4
weaker prior or quicker decay will yield short-term

---

Published as a conference paper at ICLR 2022
4 RESULTS
We empirically demonstrate the efficiency of rBO in three different settings, As rBO is a general method augment acquisition functions, it can be implemented in different parent BO packages, and the implementation in any given package inherits the pros and cons of that package. To minimize confounding factors concerning this choice of parent package, we keep comparisons within the methods in one package where possible and provide results in the other packages in Appendix [ In Sec. 4.2] using Spearmint as a parent package, we evaluate BBO against three intuitive baselines to assess its performance and robustness on priors with different qualities, ranging from very accurate to purposefully detrimental. To this end, we use toy functions and cheap surrogates, where priors of known quality can be obtained. Next, in Sec. 4.3 we compare BBO against two competitive approaches (BOPrO and BOWS) that integrate priors over the optimum similarly to л rBO, using HyperMapper (Nardi et al. 2019) BOPrO is implemented. For these experiments we adopt a Multilayer Perceptron (MLP) benchmark on various datasets, using the interface provided by HPOBench (Eggensperger et al. 2021). with priors constructed around the defaults provided by the library. Lastly, in Sec. 4.4 we apply BBO and other approaches two deep learning tasks, also using priors derived from publicly available defaults, Further, we demonstrate the flexibility of rBO in Appendix B where we evaluate tBO in SMAC (Hutter et al. 2011 Lindauer et al. 2021) with random forests, as another framework with another surrogate model, and adapt it to use the UCB, TS and PI acquisition functions instead of EI.
parent framework, in which the most competitive baseline
4.1 EXPERIMENTAL SETUP
Priors For our surrogate and toy function tasks, we follow the prior construction methodology in BOPrO (Souza et al. 202T) and create three main types of prior qualities, all Gaussian: strong, weak and wrong. The strong and weak priors are located to have a high and moderate density on the optimum, respectively. The wrong prior a narrow distribution located in the worst region of the search space. For the OpenML MLP tuning benchmark, we utilize the defaults and search spaces provided in HPOBench (Eggensperger et al. 2021). and construct Gaussian priors for each hyperparameter with their mean on the default value, and standard deviation of 25% of the hyperparameter's s domain. For the DL case studies, we utilize defaults from each task's repository and, for numerical hyperparameters, once again set the standard deviation to 25% of the hyperparameter's domain. For categorical hyperparameters, we place a higher probability on the default. As such, the quality of the prior is ultimately unknown, but serves as a proxy for what a practitioner may choose and has shown to be a reasonable choice (Anastacio & Hoos 2020). For all experiments, we run nOO with 5 N/10, where N is the total number of iterations, in order to make the prior influence approximately equal in all experiments, regardless of the number of allowed BO iterations. We investigate the sensitivity to 3 in Appendix A and the sensitivity to prior quality in Appendix G
Baselines We empirically evaluate BBO against the most competitive approaches for priors over the optimum described in Section 2.3 BOPrO (Souza et al. 202T) and BO in Warped Space (BOWS) (Ramachandran et al., 2020 To contextualize the performance of BBO, we provide additional, simpler baselines: random sampling, sampling from the prior and BO with prior- based initial design. The latter initialized with the mode of the prior in addition its regular initial design. In our main results, we choose Spearmint (with El) (Snoek et al.) (012)) for this mode-initialized baseline, simply referring to it as Spearmint, See Appendix F for complete details on the experiments.
4.2 ROBUSTNESS OF BBO
First, we study the robustness of rBO. To this end, we show that rBO benefits from informative priors and can recover from wrong priors, being consistent with our theoretical results in Section 3.3 To this end, we consider a well-known Mack-box optimization function, Branin (2D), as well as two surrogate HPO tasks from the Profet suite (Klein et al., 2019): FC-Net (6D) and XGBoost (8D). For these tasks, we exemplarily show results for rBO implemented in the Spearmint framework. As Figure 2 shows, BOO is able to quickly improve over sampling from the prior. Moreover, it improves substantially over Spearmint (with mode initialization) for all informative priors, staying up to an order of magnitude ahead throughout the optimization for both strong and weak priors. For wrong priors, BBO displays desired robustness by recovering to approximately equal regret as Spearmint. In contrast, Spearmint frequently fails to substantially improve from its initial design on the strong and
7

---

Published as a conference paper at ICLR 2022
weak prior, which demonstrates the importance of considering the prior throughout the optimization procedure. This effect is even more pronounced on the higher-dimensional tasks FCNet and XGBoost, where BO typically spends many iterations the boundary (Swersky 2017). Here, mBO rapidly improves multiple orders of magnitude over the initial design, displaying its ability to efficiently exploit the information provided by the prior.
Speasmint
nBO
Proo Sampling
+ Randon Sampling
Strong prior
Week prior
Wrong prior
Branin
-2.5
RONes
XGBoos:
20
100
60
201
200
100
Beratton
Boraton
Figure 2: Comparison BBO, Spearmint, and two sampling approaches on Branin, FCNet and XGBoost for various prior strengths. Mean and standard error of log simple regret is displayed over 100 iterations, averaged over runs. The vertical line represents the end of the initial design phase.
4.3 COMPARISON BBO AGAINST OTHER PRIOR-GUIDED APPROACHES
Next, we study the performance of BBO against other state-of-the-art prior-guided approaches. To this end, we consider optimizing 5 hyperparameters of an MLP for classification (Eggensperger et al. 2021 different OpenML datasets /Vanschoren al. 2014) and compare against BOPrO (Souza et al. 202T and BOWS (Ramachandran et al.) 2020). For minimizing confounding factors, we implement & BO and BOWS in HyperMapper (Nardi al. 2019), the same framework that BOPrO runs on. Moreover, we let all approaches share BO's ' mnitiolizatinn procedure. We consider a budget of 50 iterations as it common with ML practitioners (Bouthillier & Varoquaux 2020). In Figure 3 we see that rBO offers the best performance on four out of six tasks, and displays the most consistent performance across tasks, In contrast them BOWS and BOPrO, rBO also comes with theoretical guarantees and is flexible in the choice of framework and acquisition function.
BoPiO
BOWS
Prine
KC1 Software
Image Segmentation
Vehicle Silhouette
Valication Accursoy
50.5%
85.2%
50
90.5%
94.1%%
50.2%
94.0%
90.5%
20.0%
20
30
12
30
40
50
84.0%
10
29
30
50
Iteration
Australian Credit
Blood Transfusion
German Credit
Valiatten Accurscy
54 56.8%
10.0%
12.5%
94.%%
26.2 96.2%
10.0%
90.0%
50 19.8%
0%
88.0%
20
30
32
20 Iteration
Nerstion
Reertion
Figure 3: Comparison of =BO, BOPrO, BOWS, and prior sampling for 5D MLP tuning various OpenML datasets for a prior centered on default values. We show mean and standard error of the accuracy across 20 runs. The vertical line represents the end of the initial design phase.
8

---

Published as a conference paper at ICLR 2022
4.4 CASE STUDIES ON DEEP LEARNING PIPELINES
Last, we study the impact of xBO on deep learning applications, which are often fairly expensive, making efficiency even more important than in HPO for traditional machine learning. To this end, we consider two deep learning case studies: segmentation of neuronal processes in electron microscopy images with a U-Net(6D) (Ronneberger et al. 2013), with code provided from the NVIDIA deep learning examples repository (Przemek et a and image classification on ImageNette-122 (6D) (Howard 2019). a light- weight adaptation of ImageNet (Deng et al. 2009). with code from the repository of the popular FastAl library (Howard et al., 2018). We mimic the setup from Section 4.3 by using the HyperMapper framework and identical initialization procedures across approaches. Gaussian priors are set publicly available default values, which results of previous tuning efforts of the original authors. We again optimize for practical budget of 50 iterations (Bouthillier & Varoquaux 2020). As test splits for both tasks were not available to us, we report validation scores.
क BO achieves a 2 time-to-accuracy speedup over Vanilla BO. For Ima- geNette, the performance of =BO at iteration 4 already surpasses the performance of Vanilla BO Iteration 50, demonstrating a 12.5x time-to-accurae speedup. Ultimately, BBO's final performance establishes a new state-of-the-art validation performance on ImageNette with the provided pipeline, with a final accuracy of 94. 14% (vs. the previous state of the art with 93.55%
As shown in Figure 4
+
Sampling
U-Net Medical
ImageNette- 128
90,5/
Speedup
94.2%
Vilidation DJCR Score
90.2%
Validation, Acouracy
69.1 84
Speeduup
90.07%
69.0%
89.8%
88.9%%
09.500
$9.2%
83.7%
19.174
58.66%
08.87
10
20 Iteration
30
40
50
99.5%
10
20
30
40
50
[teera Jiou
Figure 4: Comparison of approaches for U-Net Medical and ImageNette- 128 for a prior centered on default values. We show mean and standard error of the accuracy across 20 runs for U-Net Medical and 10 runs for ImageNette- 128. The vertical line represents the end of the initial design phase.
5 CONCLUSION AND FUTURE WORK
We presented TBO, conceptually very simple Bayesian optimization approach for leveraging user beliefs about the location of an optimum, which relies on a generalization of myopic acquisition functions. rBO modifies the selection of design points through decaying weighting scheme, promoting high- tigh-probability regions under the prior. Contrary to previous approaches, BBO imposes only minor restrictions on the type of priors, surrogates or frameworks that can be used. =BO provably converges at regular rates, displays state-of-the art performance across tasks, and effectively recovers from poorly specified priors. Moreover, we have demonstrated that =BO can yield substantial performance gains for practical low-budget settings, improving on the state-of-the-ee CNN tuning tasks even with trivial choices for the prior. For practitioners who have historically relied on manual or grid search for HPO, we hope that rBO will serve as an intuitive and effective tool for bridging the gap between traditional tuning methods and BO.
for a real-world
BOO sets the stage for several follow-up studies. Amongst others, we will examine the extension of BO to non-myopic acquisition functions, such as entropy-based methods. Non-myopic acquisition functions do not fit well in the current BBO framework, as they do not necessarily benefit from evalu- ating inputs expected to perform well. We will also combine BBO with multi-fidelity optimization methods to yield even higher speedups, and with multi-objectiv optimization to jointly optimize performance and secondary objective functions, such as interpretability or faimess of models.
https: thub. com faatta/aaaaaiiia htsgrtaaaammmgeeetteeeeeeeeeeeee www.mmaggeeneeeeeeeeeeaeeaee leaderboard 80 Epochs, 128 Resolution

---

Published as a conference paper at ICLR 2022
6 ETHICS STATEMENT
Our work proposes an acquisition function generalization which incorporates prior beliefs about the location of the optimum into optimization. The approach foundational and thus will not bring direct societal or ethical consequences. However, =BO will likely be used in the development of applications for wide range of areas and thus indirectly contribute to their impacts on society. In particular, we envision that BOO will impact multitude of fields by allowing ML experts to inject their knowledge about the location of the optimum into Bayesian Optimization.
We also note that we intend for rBO to be a tool that allows users to assist Bayesian Optimization by providing reasonable prior knowledge and beliefs. This process induces user bias into the optimization, as rBO will inevitably start by optimizing around this prior. As some users may only be interested in optimizing in the direct neighborhood of their prior, BBO could allow them to do so if provided with high 3 value in relation to the number of iterations. Thus, if improperly specified, n BO could serve to reinforce user's beliefs by providing improved solutions only for the user's region of interest. However, if used properly, BBO will reduce the computational resources required to find strong hyperparameter settings, contributing to the sustainability of machine learning.
REPRODUCIBILITY
In order to make the experiments run rBO as reproducible as possible, we have included links to repositories of our implementations in both Spearmint and HyperMapper, with instructions on how to rum our experiments. Moreover, we have included in said repositories all of the exact priors that we have used for our runs, which out of the box. The priors we used were, in our opinion, well motivated as to avoid subjectivity, which we hope serves as a good frame of reference for similar works in the future. Specifically, Appendix A descriies how ran our DL experiments, Appendix|i [TT goes into the implementation further detail, and Appendix [D displays the exact priors for all our experiments and prior strengths. Our Spearmint implementation of both rBO and BOWS is available at htttt::/ //i thub. com cwmpp//boatthors/eeee..ooo and our HyperMapper implementation available at htttpss: //gtthub. com/ p piotttttoss P PiBO-Hypermapper For our results on the convergence of rBO, we have provided a complete proof Appendix E
Spearmint
ACKNOWLEDGEMENTS
Luigi Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project - Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Carl Hvarfner and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Artur Souza was supported by CAPES, CNPq, and FAPEMIG. Frank Hutter acknowledges support by the European Research Council (HRC) under the European Union Horizon 2020 research and innovation programme through grant no. 716721, through TAILOR, project funded by the EU Horizon 2020 research and innovation programme under GA No 952215, by the Deutsche Forschungsgemminnchaatt (DFG, German Research Foundation) under grant number 417962828 and by the state of Baden- Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG. Marius Lindauer acknowledges support by the European Research Council (ERC) under the Europe Horizon programme. The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC partially funded by the Swedish Research Council through grant agreement no. 2018-05973.
REFERENCES
K. Anand, Z. Wang, M. Loog, and J. van Gemert. Black magic in deep learning: How human skill impacts network training. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.
10

---

Published as a conference paper at ICLR 2022
M. Anastacio and H. Hoos. Combining sequential model- algorithm configuration with default- guided probabilistic sampling. In GECCO '20: Genetic and Evolutionary Computation Conference, Companion Volume, Cancúín, Mexico, July 8- 8-12, 2020, pp. 301-302. ACM, 2020.
I. Arganda-Carreras, Carreras, S.C. Turaga, D.R. Berger, D. Cireşan, A. Giusti, L.M. Gambardella, J. Schmid- huber, D. Laptev, S. Dwivedi, J.M. Buhmann, T. Liu, M. Seyedhosseini, T. Tasdizen, L. Kamentsky, R. Burget, V. Uher, Tan, C. Sun, T.D. Pham, E. Bas, M.G. Uzunbas, Cardona, Schindelin, and H.S. Seung. Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in Neuroanatomy, 9:142, 2015.
M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. Botorch: framework for efficient monte-carlo bayesian optimization. In Advances in Neural Information Processing Systems, 2020.
J. Bergstra and Y. Bengio. Random search for hyper- srper-parameter optimization. Journal of Machine Learning Research, 13(10))281-30 2012.
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper- hyper-paramenn parameter optimization. In Shawe-Taylor, Zemel, Bartleu, Pereira, and K. Weinberger (eds.), Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS'II) pp. 2554, 2011.
X. Bouthillier and G. Varoquaux. Survey of machine-learning experimental methods at Neur1PS2019 and ICLR2020. Research report, Inria Saclay Ile de France, 2020.
E. Brochu, Cora, and N. Freitas. tutorial on Bayesian optimization expensive cost functions, with application active user modeling and hierarchical reinforcement learning. arXiv:1012.25999/ [C), 2010.
A. D. Bull. Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research, (188))2879-29001 2011.
R. Calandra, N. Gopalan, A. Seyfarth, Peters, and M. Deisenroth. Bayesian gait optimization for bipedal locomotion. In P. Pardalos and M. Resende (eds.), Proceedings of the Eighth International Conference on Learning and Intelligent Optimization (LION' 14), 2014.
A. Cardona, S. Saalfeld, S. Preibisch, B. Schmid, A. Cheng, J. Pulokas, P. Tomancak, and V. Harten- stein. An integrated micro-and macroarchitectural analysis the drosophila brain by computer- assisted serial section electron microscopy. PLoS Biol, 11005500, 2010.
Y. Chen, A. Huang, Z. Wang, I. Antonoglou, Schrittwieser, D. Silver, and D. Freitas, Bayesian optimization in alphago. ArXiv, abs/ 8 12.66855, 2018.
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009.
K. Eggensperger, P. Müller, Mallik, M. Feurer, R. Sass, Klein, N. Awad, M. Lindauer, and F. Hutter. HPOBench: collection of reproducible multi-fidelity benchmark problems for HPO, 2021.
M. Feurer, Jost Tobias Springenberg, and F. Hutter. Initializing bayesian hyperparameter optimization via meta-lcarning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 1128-1135 2015.
M. Feurer, B. Letham, F. Hutter, and E. Bakshy. Practical transfer learning for bayesian optimization. ArXiv 0ttp/8802.2222 2018.
P. Frazier, W. Powell, and S. Dayanik. A knowledge-gradient policy for sequential information collection. SIAM Control and Optimization, 47:2410-2439, 01 2008.
Hennig and C. J. Schuler. Entropy search for information-eeficienn global optimization. Journal of Machine Learning Research, 13(1):1809-1837, 2012.
2012.

---

Published as a conference paper at ICLR 2022
J. M. Hernandez-Lobato, M. W. Hoffman, and Z. Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in Neural Information Processing Systems, 2014.
Howard.
Howard al. fastai, hTtpas///gttbuuocc////aaaaaaaaaa 2019.
2018.
Hutter, H. Hoos, and K. Leyton-Brown. Sequential model-based optiiiiiziionn for general algorithm
configuration. In Learning and Intelligent Optimization, pp. 507-523, 2011.
D. Jones, Schonlau, and Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13:455-492. 12 1998.
D. R. Jones. A taxonomy of global optimization methods based on response surfaces. Journal of global optimization, 2144)3345-383, 2001.
K. Kandasamy, K. R. Vysyaraju, W. Neiswanger, B. Paria, C. R. Collins, J. Schneider, B. Poczos, and E. Xing. Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly. Journal of Machine Learning Research, 21(8)):1-21 2020.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR, 2015.
A. Klein, Z. Dai, F. Hutter, N. Lawrence, and J. Gonzalez. Meta-surrogate benchmarking for hyperparameter optimization. In Advances Neural Information Processing Systems, volume 32, 2019.
H. J. Kushner. A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise. Journal of Basic Engineering, 86(1):97-106, 03 1964.
A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 1xxiv: 2019.
Eric Hans Lee, Valerio Perrone, Cedric Archambeau, and Matthias Seeger. Cost-aware bayesian optimization, 2020.
Li, S. Rana, S. Gupta, V. Nguyen, S. Venkatesh, A. Sutti, D. R. de Celis, T. Slezak, M. Height, M. Mohammed, and I. Gibson. Accelerating experimental design by incorporating experimenter hunches. In IEEE International Conference on Data Mining, ICDM, pp. 257-266. IEEE Computer Society, 2018.
C. Li. S. Gupta, S. Rana, Nguyen, A. Robles-Kelly. and S. Venkatesh. Incorporating expert prior knowledge into experimental design via posterior sampling. ArXiv, abs/2002. 1256, 2020.
M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp. D. Deng, C. Benjaminss, R. Sass, and Hutter. SMAC3: A versatile bayesian optimization package for hyperparameter optimization. CoRR, abs/2109.0983. 2021.
M. López-Ibáñez, J. Cibbois-Lacoste... L. Cáceres, T. Stützle, and M. Birattari. The iRace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43-58, 2016.
B. Matérn. Spatial variation. Meddelanden fran Statens Skogsforsknningsinstinu,, 1960.
J. Mockus, V. Tiesis, and A. Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2(117-129):2, 1978.
L. Nardi, D. Koeplinger, and Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunica- fion Systems (MASCOTS), pp. 347-358. IEEF, 2019.
C. Oh. E. Gavves, and M. Welling. BOCK Bayesian optimization with cylindrical kernels. In International Conference Machine Learning, pp. 3865-3874, 2018.
12

---

Published as a conference paper at ICLR 2022
Andrei Paleyes, Mark Pullin, Maren Mahsereci, Neil Lawrence, and Javier González. Emulation of physical processes with emukit. In Second Workshop on Machine Learning and the Physical Sciences, NeurIPS, 2019.
V. Perrone, H. Shen, M. Seeger, C. Archambeau, and R. Jenatton. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Advances in Neural Information Processing Systems, 2019.
S. Przemek al. Nvidia deep learning examples.
http:://tttbb..comm
DeeplearningExapples
A. Ramachandran, S. Gupta, S. Rana, C. Li, and S. Venkatesh. Incorporating expert prior in bayesian optimisation via space warping. Knowledge- Based Systems, 195: 105663, 2020.
C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention MICCAI, volume 9351 Lecture Notes Computer Science, pp. 234-241. Springer, 2015.
B. Ru, X. Wan, X. Dong, and M. Osborne. Interpretable neural architecture search via bayesian opti- misation with weisfeiler-lehman kernels. In International Conference on Learning Representations, 2021.
B. Shahriari, A. Bouchard-Côté, and N. Freitas. Unbounded bayesian optimization via regularization. In Artificial intelligence and statistics, pp. 1168-1177 1176, 201 6a.
B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148-173 2016b.
E. Siivola, A. Vehtari, J. Vanhatalo, J. González, and M. R. Andersen. Correcting boundary over- exploration deficiencies in bayesian optimization with virtual derivative sign observations. In International Workshop on Machine Learning for Signal Processing, 2018.
L. Smith. A disciplined approach to neural network byper--arameters: Part l-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv: 098000 2018.
Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L.. Bottou, and K. Weinberger (eds.), Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS'12) 12), pp. 2960-2968, 2012.
J. Snoek, K. Swersky, R. Zemel, and R. Adams. Input warping for bayesian optimization of non- stationary functions. In Eric Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp- 1682-1682. PMLR, 22-24 Jun 2014.
A. Souza, L. Nardi, I.. Oliveira, K. Olukotun, M. Lindauer, and F. Hutter. Bayesian optimization with prior for the optimum. In Machine Learning and Knowledge Discovery in Databases. Research Track European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III, volume 12977 of Lecture Notes in Computer Science, pp. 265-296. Springer, 2021.
J. Springenberg, A. Klein, S. Falkner, and F. Hutter. Bayesian optimization with robust Bayesian neural networks. In D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett (eds.), Proceedings of the 30th International Conference Advances in Neural Information Processing Systems (NeurIPS' 16), 2016.
N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger. Information-theorre regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions Information Theory, 58 (5):3250-3265, 2012.
D. Stoll, J. KH Franke, D. Wagner, S. Selg, and F. Hutter. Hyperparameter transfer across developer adjustments. In NeurIPS 2020 Workshop Meta-Learning, 2020.
13

---

Published as a conference paper at ICLR 2022
K. Swersky, J. Snoek, and R. Adams. Multi-task Bayesian optimization. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger (eds.), Proceedings of the 27th International Conference Advances in Neural Information Processing Systems (NeurIPS 13), pp. 2004-2012, 2013.
K. J. Swersky. Improving Bayesian Optimization for Machine Learning using Expert Priors. PhD thesis, 2017.
The GpyOpt authors. GPyOpt: bayesian optimization framework in python. http://githui com/
GPyptt 2016.
W. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4) 1933.
R. Turner, D. Eriksson, M. McCourt, J. Kiili. E. Laaksonen. Z. Xu, and I. Guyon. Bayesian optimization is superior random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In Hugo Jair Escalante and Katja Hofmann (eds.), Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pp. 3-26. PMLR, 06-12 Dec 2021.
Jan N. Rijn and Frank Hutter. Hyperparameter importance across datasets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, KDD '18, pp. 2367-2376, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355520.
doi: 10.1145/321981992220055 URLhtttss: http:///ooo.../10111/1//1
321 981 3220058
Vanschoren. Meta-learning: survey, 2018.
Vanschoren, van Rijn, Bischl, and L. Torgo. OpenML.: Networked science machine learning. SIGKDD Explor. Newsl, 15(2):49-66) -00),,9996..11 2014.
Q. Wang, Y. Ming, Z. Jin, Q. Shen, D. Liu, M. J. Smith, K. Veeramachaneni, and H. Qu. Atmseer: Increasing transparency and controllability in automated machine learning. Proceedings of the 2019 CHI Conference Human Factors in Computing Systems, CHI 19, pp. 1-12. Association for Computing Machinery, 2019.
Z. Wang and S. Jegelka. Max-value entropy search for efficient bayesian optimization, In Proceedings of the 34th International Conference Machine Learning, ICML, volume 70 of Proceedings of Machine Learning Research, pp. 3627-3635. PMLR, 2017.
M. Wistuba, N. Schilling, and L. Sichnidt-Thienee Hyperparameter search space pruning - A new component for sequential model-basee hyperparameter optimization. In A. Appice, P. Rodrigues, V. Costa, J. Gama, A. Jorge, and C. Soares (eds.), Machine Learning and Knowledge Discovery in Databases (ECML/PKDD) 15), volume 9285, pp. 104-119, 2015.
L. Zimmer, M. Lindauer, and F. Hutter. Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl, IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):3079 3090, 2021.
14

---

Published as a conference paper at ICLR 2022
A BETA ABLATION STUDY
We consider the effect of the 3 hyperparameter of TBO introduced in Section controlling the speed of the prior decay. To show the effect of this hyperparameter, we display the performance of rBO for the toy and surrogate-based benchmarks across all prior qualities. We emphasize the trade-off between high-end performance good priors and robustness to bad priors. In general, higher value 3 yields better performance for good priors, but makes TBO slower to recover from bad priors. This behaviour follows intuition and the results provided in Section 3.3
In Figure 5 we display how BBO performs for different choices of 3, and once again provide sampling from the prior and Spearmint baselines. Following the prior decay parameter baseline by (Souza et al. 2021), ,. we show that the choice of 3 = 10 onsistently gives one of the best performances for strong priors, while retaining good overall robustness. Nearly all choices of give final performance better than that of Spearmint for good priors. Additionally, there a clear relationship between final performance and all good priors. This is best visualized in the weak XGBoost experiment, where the final performances are distinctly sorted by increasing 3. Similar patterns are not as apparent in the final performance on wrong priors. This behaviour highlights the benefits of slowly decaying the prior. Overall, n rBO is competitive for a wide range of B3 but suffers slightly worse final performance on good priors for low values of B.
Speamnint
Beta
Beta
Beta
Beta 10
Beta 20
Prior Sampling
Strong prior
Weak prior
Wrong prior
Branin
SVM
0.0
-1.5
-2.0
0.5
FCNet
2.0
1.0
-1.5
2.5
2.0
2-
XGBoost
0
-2
10
80
100
40
60
20
80
100
Iteration
Iteration
Iteration
Comparison of xBO and Spearmint with varying values / for Branin and Profet benchmarks for the strong, weak and wrong prior qualities. The mean and standard error of log simple regret is displayed over 100 iterations, averaged over 10 repetitions. Iteration is removed for visibility purposes. The vertical line represents the end of the initial design phase.
Figure
15

---

Published as a conference paper at ICLR 2022
Sampling
Strong prior
Weak prior
Wrong prior
Branin
-2
-61
01
-3
SVM
0.0
11.
-2.0
-1.5
0.5
FCNet
-25
-1.0
-3.0
-1.5
-3.5
-25
2.0
-4.0
XGBoost
20
40
60
90
100
20
40
60 Iteration
10
100
20
40
60
80
100
Iteration
Iteration
Figure 6: rBO on SMAC using a GP surrogate for the EI, UCB, PI and TS acquisition functions, 10 for Branin and Profet benchmarks for varying prior qualities. The mean and standard error of log simple regret is displayed over 100 iterations, averaged over 10 repetitions. Iteration 1 is removed for visibility purposes. The vertical line represents the end of the initial design phase.
B BBO VERSATILITY
We show the versatility of rBO by implementing it in numerous variants of SMAC Hutter et al. (01T), a well-established HPO framework which supports both GP and RF surrogates, and majority of the myopic acquisition functions mentioned in Section 2 We showcase the performance of r BO-EI, BO-PI, BOO-CC and =BOOTTS on the general formulation of π nBO with GP surrogate, well =BO-EI with an RF surrogate, which requires minor adaptation.
B. GENERAL FORMULATION OF BBO
To allow for the universality of rBO across several acquisition function, we must consider the various magnitudes of acquisition functions. As UCB and TS typically output values in the same order of magnitude and sign as the objective function, we do not want the behaviour of rBO to be affected by such variations.
The solution to the problem referenced above is to add simple affine transformation to the observa- tions, { by subtracting by the incumbent, y As such, we consider at each time step not the original dataset, D, = {(x; } but the augmented dataset D., {(x,, y: ) With this formulation, get the desired scale- and sign-invariance in the UCB and TS acquisition functions, without changing the original strategy of any of the acquisition function. Notably, this change leaves rriorweiigted El and Pl unaffected.
16

---

Published as a conference paper at ICLR 2022
Expected Iegrooment
Probsbiilyy of Inporemnnt
Taper Confilecc Bound
Thompson Sanyging x
Beredtn
Ierrotton Beroetton
x
1D Branin
Prior weighted Acquisition Function
Acqusitinn Funccion
Docasing
Figure 7: Point selection strategy evolution for rBO SMAC with GP surrogate for the EI, PI, UCB and TS acquisition functions on the Log- ID-Branin function. The plots show rescaled values of the prior-weighted acquisition function (purple), the regular acquisition function (blue) and n, (red) on a ID-Branin (grey) with global optimum in the center of the search space and the current selection vertical violet line. The rows represent iterations 4, 6 and for run with 2.
B.2 RANDOM FOREST SURROGATE
We now demonstrate x BO with RF surrogate model. In the SMAC implementation of the RF surrogate, the model forms piece-wiss wise constant mean and covariance functions. Naturally, this leads to the EI, PI or UCB acquisition function surface being piece-wise constant as well. Consequently, an acquisition function with a RF surrogate will typically have a region of global optima. The choice of the next design point is then selected uniformly at random among the candidate optima. We wish to retain this randomness when applying BBO. As such, we require the prior to be piece-wise constant, too. To do so, we employ binning approach, that linearly rounds prior values after applying the decay term. The granularity of the binning decreases at the same rate as the prior, allowing the piece-wise constant regions of the prior grow in size as optimization progresses. In Figure 9 we demonstrate the importance of the piece-wise constant acquisition function by showing the point selection when applying BO with a continuous prior an RF surrogate (left) and when applying the binning approach (right). Notably, the smooth prior on the left repeatedly proposes design points very close to previous points, as the prior forces the selection of points near the boundary of a promising region. Thus, the surrogate model rarely improves, and the optimization gets stuck at said boundary for multiple iterations at a time. This best visualized at iteration 5 and 10, where similar points have been selected for all iterations in the time span. With the binned prior the right, the selection of design points occurs randomly within a region, avoiding the static point selection and updating of mon-modified approach. In Figure g we report the performance of #BO with RF surrogate and the binning approach. This approach is competitive, as it provides substantial improvement over S.AA,, improves over sampling from the prior, and quickly recovers from misleading priors. Notably, the binning is not required for discrete parameters, as the piece-wise constant property holds by default. Thus, this adaptation is only necessary for continuous parameters.
C OTHER PRIOR-BASED APPROACHESS
We now demonstrate the performance of BBO for five different functions and HPO Surrogates: Branin, Hartmann-6, 6, as well as three tasks from the Profet suite- SVM. FCNet and XGBoost. We compare all frameworks for priors over the optimum- namely BOPrO /Suza et al. (021), BOWS Ramachandran et al.] (2020). TPE |ergstra et al. (2011), PS-G Li et al. (2020). The performance of BOO is shown on two different frameworks Spearmint and Hypermapper to allow for fair comparison and display cross-framewort consistency. As BOWS is implemented in Spearmint and BOPrO in Hypermapper, they appear in the plots retaining to their framework. We display each approach with vanilla Spearmin//Hypemapperr with normal initialization, display the performance of BBO implemented in Spearmint, as well as Mode + Spearmint, on the MI P tuning tasks.
additional baseline. Moreover,
17

---

Published as a conference paper at ICLR 2022
Sampling
Strong prior
Weak prior
Wrong prior
Branin
0
-3
-1
SVM
35
-3
-4
0.0
-1.5
-0.5
FCNet
1.6
-2.
1.8
1.0
-3.0
2.0
-1.5
-3.5
7.2
2
XGBoost
1
1.
0
0
20
40
60 Iteration
90
100
0 20
40
60
10
100
20 40
60
80
100
Iteration
Iteration
Figure 8: RBO with binning and SMAC using and RF surrogate and El acquisition function, 3 = 10. and the supporting frameworks for Branin and Profet benchmarks for varying prior qualities. The mean and standard error of log simple regret is displayed over 100 iterations, averaged over 10 repetitions. Iteration is removed for visibility purposes. The vertical line represents the end of the initial design phase.
D PRIOR CONSTRUCTION
We now present the method by which we construct our priors. For the synthetic benchmarks, we mimic (Souza et al. 202T) by offsetting a Gaussian distribution from the optima. For our case studies, we choose Gaussian prior with zero correlation between dimensions. This was required in order to have a simple, streamlined approach that was compatible with all frameworks. We constructed the priors once before conducting the experiments, and kept them fixed throughout.
Synthetic and Surrogate- HPO Benchmarks For these benchmarks, the approximate op- tima of all included functions could obtained advance, either analytically or empirically through extensive sampling. Thus, the correctness of the prior is ultimately known in advance. For function of dimensionality with optimum at x*, the strong and weak prior qualities were constructed by using quality-spechi noise term { and quality- -spelity- standard deviation fraction of the search space. For the strong prior Ty (2:,, we use small standard deviation 0, = 1% and construct the prior as
T. (z)
Aixx + +,,0s), e; -~~N ~0,
(10)
We construct the weak priors analogously by using larger standard deviation aw 10%. For 20 runs of the strong and weak prior, this procedure yielded us 20 unique priors per quality type, with varying offsets from the true optimum. Additionally, the density the optimum is substantially
18

---

Published as a conference paper at ICLR 2022
Expected Improvement
Probability of Improvement X
llrratoon
Iteration 6
X
Iteration.
1D Branin
wrior-wiighted Acquisition Function
Acquisition Function
Decaying Prior
Figure 9: Point selection strategy evolution BBO on SMAC with a RF surrogate, when employing smooth prior (left) and binned prior (right) for the ID-Branin function. The plots show rescaled values of prigh-weighted EI (purple). (blue) and Tn (red) on a ID-Branin (grey) with global optimum in the center of the search space and the current selection vertical violet line. The rows represent iterations 5, 10, 15 and 20, for run with B = 5.
250
Spearmint
BOWS
PS-G Week prior
Hyperogt TPE
Prior Sampling
Strong prior
Wrong prior
0.0
Branin
-2.5
-5.0
Hartmann
SVM
PCNet.
6
XCBoost
201
40
50
20
100
42 83 10
100
20 421
80
200
Figure 10: Comparison of #BO, Spearmint and other approaches for priors the optimum for Branin, Hartmann and Profet benchmarks. rBO implemented in Spearmint. The mean and standard error of log simple regret is displayed over 100 iterations, averaged over 20 repetitions. Iteration is removed for visibility purposes. The vertical line represents the end of the initial design phase.
larger for the strong prior than the weak prior. No priors with a mean outside the search space were allowed, such priors were simply replaced. For Branin, we only considered one the three Branin optima for this procedure, since not all included frameworks support multi-modal distributions. For the wrong prior, we construct similarly the strong prior, but around the empirical maximum, x*. of the objective function in the search space. Since this point was far away from the optimum for all benchmarks, did not add additional noise. So, the wrong prior um is constructed
Tm (z) Na ox))
(11)
which means that the wrong prior is identical across runs for given benchmark.
19

---

Published as a conference paper at ICLR 2022
Strong prior
Woek prior
Wrong pro
0.0
Branin
2.5
5.0
Hartmann 6
0
-2
-7.5
-25
SVM
-2.
5.0
55.0
FONet
-1
-2
XCBoost
Boration
Beration
Itration
Figure 11: Comparison of BBO, Hypermapper and other approaches for priors over the optimum for Branin, Hartmann and Profet benchmarks. =BO implemented in Spearmint. The mean and standard error of log simple regret is displayed over 100 iterations, averaged over 20 repetitions. Iteration removed for visibility purposes. The vertical line represents the end of the initial design phase.
Spearmint
Driee Sampling
Australian Credit
Blood Transfusion
German Credit
Validaionn Accuracy
94.
10.0%
00.3%
50.095
Dberation
Software
Image Segmentation
Vehicle Silhouette
Validation Acquracy
94.0%
93.0%
16.259
Iteraton
Rerraton
Deration
Figure 12: Comparison =BO, BOPrO. BOWS, and prior sampling for SD MLP tuning on various OpenML datasets for a prior centered around default values. We show mean and standard error of the accuracy across 20 repetitions. Iteration is removed for visibility purposes. The vertical line represents the end of the initial design phase.
E
PROOFS
Here, we provide the complete proofs for the Theorem and Corollary introduced in 3.3 In addition, provide insight into the interplay between 3, the prior n, and the value of the derived bound Cx,
20

---

Published as a conference paper at ICLR 2022
Theorem 1. Given Dn. Ke. nT, L, R. and the compact set X Rd as defined above, the loss L incurred at iteration n. by El can be bounded from above as
|max x x(x) 8/n
LE DR. He(X), R) < Ca, Ca He(A), R), Cr,"
min T(x)
(12)
Proof. To bound the performance of EI, to that of EI, we primarily need to consider Lemma 7 and Lemma 8 by Bull Bull 2011). In Lemma 7, it stated that for any sequence of points {z; 1 dimensionality d, kernel length scales L, and p N, the posterior variance sy 2n+1 will, for large value C, satisfy the following inequality at most times,
VSS sn Sn(xn+1i: E) 2 Cp Cp-(uAl)/e (logp)", 7 10 1>1*
fa. a,
(13)
Thus, we can bound the posterior variance by assuming a point in time ni where Eq. 13 has held p times. We now consider Lemma 8 where, through a number of inequalities, El is bounded by the actual improvement In
x(RRjo) In T(R/O)
max I Tn-Rs, Rs,
S El, - I, + (R + 0)s,
(14)
where I, = -[(=)))(())+= ((x))=+((=>=====--)-
+=2) and 8n (0n E). Since BOO re-weights EI, by Tye these bounds need adjustment to hold for El,,.. For the upper bound provided Lemma make use of m (x) to bound EI,, for any point X::
EI (z)
(z)
EI, In + (R+
(15)
max Tn (x) maX n. (x)
For the lower bounds, we instead rely on minngx To (T())) a similar manner:
max In Rs. ((Ro/)) Im
(x)
El,,(()))
S (lI,(() < min (x)
(16)
T(R/o)
min (x)
Consequently, EI, can be bounded by the actual improvement as
min n. (z)) max I, - Rs, T(-R/o) In S (x) S max n., mm)x, + (R+ T(R/O)
(17)
#EX
#EX
With these bounds in place, we consider the setting as in the proof for Theorem 2 in Bull Bull 2011 which proves an upper bound for the EI strategy in the fixed kernel parameters setting. At iteration np: p S np S 3p, the posterior variance will be bounded by Cp-(Cu (lol)/((ogpp) Furthermore, since I, 2 and [[fll.((x) I R, we can bound the total improvement as
Σ4 E fi<If(a)) f(z)+ S f(x)) - min f S 2l - 2R.
(18)
leaving us maximum of p times that I, 2 2Rp Consequently, both the posterior variance sa, and the improvement In, are bounded at ny. For a future iteration n, 3p I n S 3(p + 1), we use the bounds on El,, Sn, and In, to obtain the bounds the El, loss:
(EI,, Dn, He(X), R)
=((")) min f
S (1th)) min f
EIm,n, (x*)
T(R/o)
mingex Tn (x) T(-R/o)
Eln,m, (2n+1) T(R/o)
minDex nn (x) T(-R/o)
maxnex nn (x) T(R/o)
(In, + ( t))
min Tn (x) T(-R/o)
maX (xx)
8/n
T(R/O)
(2Rp + 1 CR (vAl)/d (1 (logp)))
mingex (xx)
T( R/o)
21

---

Published as a conference paper at ICLR 2022
8/n
where the last inequality is factor Can (EI, Dn, He( R)..
maxaex m(x)
larger than the bound on
mingex m(æ)
Corollary 1. The loss of a decaying prior-weighted Expected Improvement strategy, Elg, is asymp- totically equal to the loss of Expected Improvement strategy, El:
L. (n Dn, He(A)), R) ~ (n Dn, He(X'),
(19)
so we obtain convergence rate for Elg of D Dr, R) = O( (WA (log)/
Proof. We simply compute the fraction of the losses in the limit,
lim La (EI, Dn, He({)), < lim maxxexx T(x) 8/n
n2-00 (EI, Dn. He(x), R)) 2=000 minagx (x)) = =1.
(20)
E. SENSITIVITY ANALYSIS CR...
We now provide additional insight into how Ca, depends the choices of prior and made by the user. To do so, we consider typical low-budget setting and display values of Cr.n at iteration 50. We consider of the search space. In the plot below, we display how the choice of given as percentage of the search space, and 3, the prior confidence parameter, yield different values of Cr... We see that, for approximately half of the space, the upper bound on the loss is at least 80% (bright green or yellow) the upper bound of EI, and only a small region of very narrow priors (dark blue) give low guaranteed convergence rate.
deeedmmnniional search space where with Gaussian prior located the center
Value of C at iteration 50
10
1.0
0.8
3-value
0.4
0.2
40%
O's proportion to search space
Figure 13: Value of Cs, at iteration 50 for centered Gaussian prior in one-dimensional search space, varying and B. The value of Ca,, upper bounds the loss incurred by relative El at the at the given iteration, given the same data. Regions dark represent pairs of (a, () where the upper bound on prightee El is low relative to El, whereas regions in yellow represent pairs of (a, B) where the upper bound is approximately the same.
22

---

Published as a conference paper at ICLR 2022
F EXPERIMENT DETAILS
F.1 FRAMEWORKS
Our implementations of #BO require little change the supporting frameworks, Spearmint and HyperMapper, and we stay as close to the default settings as possible for each framework. For both Spearmint and HyperMapper, we consider Matérn 5/2 Kernel. For particularly strong priors, rounding errors can cause the prior to be zero in parts of the search space, potentially affecting rBO's convergence properties. To avoid these rounding errors and ensure a strictly positive prior, we add small constant, e 10 12, to the prior throughout the search space for all prior qualities. For the initial sampling from the prior, we truncate the distribution by disallowing sampled points from outside the search space, instead re-sampling such points. During optimization, we do not to explicitly truncate the prior, as points outside the search space are never considered during acquisition function maximization. Thus, the prior is effectively truncated fit the search space without requiring additional consideration.
To the best of our knowledge, there is publicly available implementation of BOWS, so re- implemented it in Spearmint. For the Spearmint implementation of BOWS, we provide warped versions of each benchmark, obtaining 20 unique warpings per prior quality and benchmark. We truncate the prior by restricting the warped search space to only include the region which maps back to the original search space through the inverted warping function. For all other approaches, we use the original, publicly available implementations. Notably, the available implementation of Hyperopt TPE does not support bounded search spaces under our priors; as compromise, when asked to evaluate outside the search space we return an empirically obtained maximum on the objective function inside the search space.
We use the search spaces, prior locations and descriptions used by (Souza al. 2021 for the toy and surrogate HPO problems. We now provide additional details about the benchmarks and case study tasks used, their associated search spaces and priors, and the resources used to run these studies.
F.2 BENCHMARKS AND CASE STUDIES
Branin The Branin function is well-known synthetic benchmark for optimization problems. The Branin function has two input dimensions and three global minima.
Hartmann-6 The Hartmann-6 function is a well-known synthetic benchmark for optimization problems, which has one global optimum and six dimensions.
benchmark in 2D based on Profet (Klein al. 2019). This benchmark is generated by a generative meta-model built using a set of SVM classification models trained on 16 OpenML tasks. The benchmark has input parameters, corresponding to SVM hyperparameters.
SVM A byperparametee-oopiiizaatt
FCNet A hyperparameter and architecture optimization benchmark in 6D based on Profet. The FC-Net benchmark is generated by a generative meta-model built using set of feed-forward neural networks trained on the same 16 OpenML tasks as the SVM benchmark. The benchmark has input parameters corresponding to network hyperparameters.
XGBoost A hyyerparameteeroppiiizztii benchmark in 8D based on Profet. The XGBoost bench- mark is generated by a generative meta-mon meta- built using a set of XGBoost regression models in 11 UCI datasets. The benchmark has eight input parameters, corresponding to XGBoost hyperparame- ters.
OpenML. ML MLP
The OpenML. MI. tuning tasks are provided through FPOBeenttFgggenggegerr al. (202I), and train binary classifiers on real-world datascts. The 5D parameter space consists of four continous parameters and one integer parameter.
U-Net Medical The U-Net (Ronneberger et al. 2015) is popular convolutional neural network architecture for image segmentation. We use the implementation and evaluation setting from the popular NVIDIA deep learning examples repository (Przemek et al.) build study for optimizing hyperparameters for U-Net. The NVIDIA repository is aimed towards the segmentation of neuronal processes in electron microscopy images for the 2D EM segmentation challenge dataset
23

---

Prior mean Published as a conference paper at ICLR 2022
Table 1: Search spaces for the Branin and Profet benchmarks. We report the original ranges and whether or not log scale was used. Mean of the strong and weak priors before offset are reported.
Benchmark
Range
Parameter values
Log scale
Branin
T1
-5, 10] [0, 15)
J2
2.275
SVM
C
le 10 e
7.84
le 10 10
9.35
FCNet
learning rate batch size units layer 1
10-6 10-1
10-6
[8, 128)
8.7
[16,512
210
units layer 2 dropout rate 11 dropout rate 12
16,512
205
00.0, 0.99
0.0007
0..0,0.99
0.852
XGBoost
learning rate gamma
110-,,10"1
10
[0.2]
2
LI regularization 1.2 regularization number of estimators
[10-5, 10
100.088
103
10 10-1.78
[10 500]
500
subsampling maximum depth minimum child weight
0.1,
0.1
1, 15)
00, 20)
8.42
Table 2: Search spaces for the two case studies. We report the original ranges and whether or not a log scale was used. Mean and standard deviation of the priors are reported, where the standard deviation is reported probabilities for activation functions U-Net were set to uniform and the probabilities pooling, [Avg, Max], were set to [0.2, 0.8), respectively.
percentage of the search space. For the categorical variable pooling, the
Benchmark
Parameter name
Range
Log scale
Prior mean
Prior st.dev.
OpenML MIP
alpha batch size depth initial learning rate width
10-6 10-2
10-1
25%
10-6 10-2
10-1.5
25%
[1,2,3]
0.5
25%
0.6, 0.99]
0.9
25%
0.9,9.9999
0.999
25%
(Arganda- Carrerrs al. 2015 Cardona et al. 2010). We optimize 6 hyperparameters of the U-Net pipeline.
ImageNette ImageNette (Howard) 2019) is subset of 10 classes of ImageNet (Deng al. 2009 and is primarily used for algorithm development for the popular FastAl library (Howard et al. 2018). The FastAl library contains convolutional neural network pipeline for ImageNette, that used by all competitors on the ImageNette leaderboard. We base our case study on the 80 epoch, 128 resolution setting of this leaderboard and optimize 6 of the hyperparameters of the FastAl ImageNette pipeline.
F.3 SEARCH SPACES AND PRIORS
The search spaces for each benchmark are summarized in Table [ (Branin and Profet), Table (OpenML MLP). and Table 3 (ImageNette and U-Net). For the Profet benchmarks, we report the original ranges and whether not log scale used. However, in practice, Profet's generative model transforms the range all hyperparameters to a linear [0, 1] range. We use Emukit's public implementation for these benchmarks (Paleyes al. 2019
24

---

Published as a conference paper at ICLR 2022
Table 3: Search spaces for the two case studies. We report the original ranges and whether or not a log scale was used. Mean and standard deviation of the priors are reported, where the standard deviation reported
pereentage of the search space. For the categorical variable pooling, the probabilities for activation functions in U-Net were set to uniform and the probabilities for pooling, [Avg, Max], were set to [0.2, 0.8], respectively.
Benchmark
Parameter name
Range
Log scale
Prior mean
Prior st.dev.
U-Net Medical
learning rate weight decay
[10 10
21
10 10
25%
10 10-6,10 10
V
10 10 4.5
25%
dropout 81 82 activation
0.0.99
0.5
25%
0.6, 0.99]
0.9
25%
0.9, 0.9999)
0.999
25%
3 options
25%
ImageNette- 128
learning rate squared momentum
[10 [10-4.0
10 10-2.10 2.10
25%
0.9, 0.999]
V
0.99
25%
momentum
0, 0.999
0.95
25%
epsilon
10 1007,10-5 $]
10
25%
mixup
0.0, 0.5
0.4
25%
pooling
{Avg, Max}
F.4
CASE STUDY DETAILS
Training details deep learning case studies Both case studies are based on existing deep learning code, whose hyperparameters we vary according to the HPO. In both case studies, we enabled mixed precision training, and for ImageNette-128 to work in conjunction with Spearmint, we had to enable the MKL_SERVICE, SERVICC_FFRCCEEEEEEE FORCE INTEL environment flag. For all further details, we refer to the PORCE_INTEL
supplementary material containing our code.
Resources used for deep learning case studies For U-Net Medical we used one GeForce RTX 2080 Ti GPU, whereas for ImageNette- 128 we used two GeForce RTX 2080 Ti GPU's. Also, used 4 cores and 8 cores respectively, of an AMD EPYC 7502 32-Core Processor. In Table 4 we list the GPU hours needed for running the deep learning case studies as well as the emitted CO2 equivalents.
Table Approximate GPU hours required perform an evaluation for one approach for the deep learning case studies. Additionally we report approximate carbon footprints using the MachineLearn ing Impact calculator presented in Lacoste al. (2019) based on OECD's 2014 yearly average carbon efficiency.
Benchmark
Repetitions
GPUh's Per Repetition
Total GPUh
Total kgCOgeq
U-Net Medical
20
9
180
19.4
ImageNette-125 128
10
40
400
43.2
Assets deep learning case studies In addition to the assets we list in the main paper, the U-Net Medical code base we used employs the 2D EM segmentation challenge dataset (Arganda-Carrera) ct al. 2015 Cardona et al, 22010), which is available for for the purpose of generating or testing non-commercial image segmentation software. We include licenses of all existing code assets used in the supplementary material containing our code.
G SENSITIVITY TO PRIOR STRENGTH
We investigate the performance BBO when providing priors over the optimum of various qualities. To show the effect of decreasing the prior strength, a grid of prior qualities, with varying widths and
25

---

Published as a conference paper at ICLR 2022
Branin
nBO
Spearmint
Prior Sampling
Mode
1% Width
2% Width
5% Width
10% Width
20% Width
1% Offset
2
1
2% Offict
5% Offset
10% Offset
-2
20% Offset
-2
Nteratin
Iteration
Eseation
Iteration
Iteration
Figure 14: Comparison Branin of priors with varying widths and offsets the optimum TBO, Spearmint, Spearmint with mode initialization, and sampling from the prior. The mean and standard error of log simple regret is displayed over 100 iterations, averaged over 20 repetitions. Iteration is removed for visibility purposes.
offsets from the optimum, are provided. Thus, priors range from the strong prior the results, to weak, correct priors and sharp, misplaced priors.
shown that nBO provides substantial performance across most prior qualities for all benchmarks but Branin, and recoups its early losses the worst priors in the bottom left corner. BO demonstrates sensitivity to the width of the prior, as the optimization does not progress quickly for well-located priors with larger width. Additionally, BBO's improvement over the Spearmint + improve over the mode in early iterations.
From Figures 14 18
Mode baseline is further emphasized, as this baseline often fails to meaningfully
26

---

Published as a conference paper at ICLR 2022
H
IBO
Spearmint
Prior Sampling
Mode Spearmint
1% Width
Width
5% Width
10% Width
20% Width
1% Offse!
2% Offset
5% Offset
10% Offset
20% Offsct
20
60 lteration
40
80
100
20
80 Itoration:
4B
100
20
40
60
80
160
20 40
G0
80
100
20
40
60
100
lteration
Itrraioo
lteration
Figure 15: Comparison on Hartmann- 6 of priors with varying widths and offsets over the optimum on rBO. Spearmint, Spearmint with mode initialization, and sampling from the prior. The mean and standard enror of log simple regret is displayed over 100 iterations, averaged over 20 repetitions. Iteration removed for visibility purposes.
27

---

Published as a conference paper at ICLR 2022
SVM
IBO
Spearmint Width
Prior Sampling
Made Spearmint
1% Width
Width
10% Width
20% Width
1% Offset
2% Offset
5% Offsct
-4
10% Offset
20% OffseL
-8
20
40
60
80
100
20
49
80
100
20
40
66
80
160
20
40
60
80
180
20
40
60
100
lteration
Itoration:
lteration:
Iteration:
lteration
Figure 16: Comparison on SVM of priors with varying widths and offsets over the optimum on rBO, Spearmint, Spearmint with mode initialization, and sampling from the prior. The mean and standard emor of log simple regret is displayed over 100 iterations, averaged over 20 repetitions. Iteration is removed for visibility purposes.
28

---

Published as a conference paper at ICLR 2022
FCNet
IBO
Spearmint
Prior Sampling
Mode Spearmint
1% Width
Width
Width
10% Width
20% Width
1% Office
2% Offict
5% Offsct
10% Offsct
20% Offset
-3
20
60 lteration
40
80
100
20
4B
80
100
20
40
60
80
160
20
40
60
80
180
20
40
60
100
Itoration:
Iteration
Iteration:
lteration
Figure 17: Comparison on FCNet of priors with varying widths and offsets over the optimum on TBO, Spearmint, Spearmint with mode initialization, and sampling from the prior. The mean and standard error of log simple regret is displayed over 100 iterations, averaged over 20 repetitions. Iteration removed for visibility purposes.
29

---

Published as a conference paper at ICLR 2022
XGBoost Prior Sampling
IBO
Spearmint
Mode Spearmint
1% Width
Width
5% Width
10% Width
20% Width
1% Offse!
2% Offset
5% Offset
10% Offsct
0
20% Offsct
20
60 lteration
40
80
100
20
40
80
100
20
40
60
80
160
20
40
60
80
180
20
40
60
100
Itoration:
Iteration:
Iteration
lteration
Figure 18: Comparison on XGBoost of priors with varying widths and offsets over the optimum on TBO, Spearmint, Spearmint with mode initialization, and sampling from the prior. The mean and standard enror of log simple regret is displayed over 100 iterations, averaged over 20 repetitions. Iteration removed for visibility purposes.
30


Reviews:
[{"review_id": "o1Ae-2PL_sc", "reviewer": "Reviewer_VqSx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is well-written and appropriately discusses the related literature. \n\nWhile the problem of incorporating the optimum location makes sense, I am not sure if the experts can provide a proper probability distribution about the optimum. Finding this information in practice seems rare. For example, even the experiments in section 4.4 rely on the priors which are based on already (manually) tuned deep networks. Finding useful priors may thus be hard in practice.\n\nThe method seems simple and as stated in the paper is easy to implement. However, it seems to me that while the paper claims the proposed method to work with any acquisition function, it gives analysis only for EI acquisition function? Is this true?\n\nAlso, there is a claim that the method can recover from any misleading prior, but it may be true only when there is a nonzero support for the true optimum x*. If the prior likelihood for x* is zero, the method may not converge. In case of misleading prior with nonzero support, is the convergence rate still sublinear?\n\nIn Eq 4, what is y^*_{n+1}? It seems that left-hand side of the EI expression in Eq 4 does not depend on x, which is strange.\n\nIn the experiments section, all the methods seem to be starting from different initialization points, if this is true then the comparison between the methods may not be fair. Why does \\pi-BO often start from a higher starting value after the initial design? We do not know if \\pi-BO does well due to the proposed algorithm or due to the better initialization?\n\n\nMinor Comments:\nOn page 1: “documentesd” should be “documented”\nOn page 4: in the third para, “show” should be “shows”\nOn page 5: before Eq 6, “prior-weighed” should be “prior-weighted”", "summary_of_the_paper": "This paper proposes a new method to incorporate prior knowledge about optima location in Bayesian optimization. The paper claims that the proposed method is simple to implement, can be used with any acquisition functions and has efficient convergence rate.", "summary_of_the_review": "While the proposed method for this problem makes sense, it may be hard for experts to provide such information in practice. Further, the convergence analysis seems limited to Expected improvement (EI) acquisition function only. Last but very important, different methods seem to be starting from different initialization points, which may be problematic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gubNCvduwDA", "reviewer": "Reviewer_eisp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Review:\n\n### Strengths:\nThe method is simple and sensible, and experiments convincingly demonstrate that it works with a well-specified prior, and more importantly, demonstrate that a poor prior could lead to problems. I know it doesn’t have a lot of moving parts or complicated mathematical formulations that the BO community tends to favor these days, but I am in favor of acceptance for the reasons I mentioned above. It’s a rigorous scientific work that could have a significant impact on the way BO is performed in the HPO community today. \n\n### Weaknesses:\nMy only criticism is that priors used (on continuous params) are Gaussian. The authors may be leaving performance on the table by using such a simple prior. But this does not affect my score that much. \n\n\n## Additional Notes:\n* The decay method you have was used in this workshop paper Cost-aware Bayesian optimization, by Lee et al., 2020 (in a slightly different context, to decrease the impact of a cost model), so you might consider citing that if you want to reinforce that this decaying type modification has precedence in the literature. \n* There is a stronger connection here with metalearning, in the context of HPO, (which seeks to build and then exploit priors) as a whole, and perhaps this could be included as a short paragraph in the related work section w/ some additional citations (e.g., more of Hutter’s work). ", "summary_of_the_paper": "## Summary:\n\nPiBO is a very straightforward paper that seeks to incorporate prior knowledge about the optimum to accelerate Bayesian optimization. \n\nThey achieve this by simply multiplying the acquisition function (Expected Improvement, in the paper) by the prior distribution, and then maximizing that. They then decay the prior in order to deal with mis-specified priors. The authors also provide some theory that this weighted acquisition function performs (for some particular definition of performance) no worse than EI times a constant that depends on the decay rate. It’s a reasonable idea; honestly I’m surprised no paper has tried this before. \n\nSynthetic and real-world experiments indicate that:\n-With a well-specified prior, PiBO demonstrates superior performance over it’s basic EI counterpart. \n-With a poorly-specified prior, PiBO performs poorly worse than it’s EI counterpart but is usually able to recover given enough iterations. \n\n", "summary_of_the_review": "The method is simple and sensible, and experiments convincingly demonstrate that it works with a well-specified prior, and more importantly, demonstrate that a poor prior could lead to problems. It’s a rigorous scientific work that could have a significant impact on the way BO is performed in the HPO community today. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TtxFzO_EY4h", "reviewer": "Reviewer_gZca", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am quite aware of research in BO and I think integrating prior information is an important aspect that has not been looked into much. Existing solutions are either restricted or do not admit a convergence analysis. In that respect, I quite like the idea of the paper. It's quite simple and yet lend itself to an analysis of the convergence. The experiments are sufficient in my opinion to expose the behaviour of the algorithm at different scenarios. \n\nTheoretical analysis is convincing. Although I should say that the upper bound can become uselessly loose if the prior is not designed properly. The constant C_{\\pi, n} can be arbitrarily large if the prior is Gaussian like and narrow. This may make the anytime upper bound not useful. The authors should use this insight to restrict the shape of the prior function. Another limitation of the theoretical analysis is sticking to EI acquisition function. An analysis that admit GP-UCB acquisition function would have completed the work because unfortunately, EI is not proven to converge for noisy function case. \n\nAnother important question is how do the authors propose to scale the prior function \\pi so that can influence the acquisition function. The EI functions are notoriusly peaky in the sense that it can have sharp peaks among the valley of mostly small values. In that case, to make prior count, it needs to be scaled properly. I would like the authors comment on this aspect.", "summary_of_the_paper": "The paper proposes a method to incorporate prior information about the optimum in the standard Bayesian optimisation (BO) setting. The prior is specified as a smooth function over the range having high values where the experimenter think the optimum may exist with high probability and low values otherwise. This prior function is incorporated in the BO workflow through multiplication with the EI acquisition function. The authors show that when the prior function is decayed then this BO achieves the usual sublinear regret rate of a standard BO asymptotically. The synthetic experiments are extensive and convincing. The real case studies are limited, but sufficient.", "summary_of_the_review": "The paper proposes a simple but elegant method to incorporate prior information about optimum in the BO workflow. It's a long-standing research gap and the proposed solution fills the gap to some degree. The best part is that the simple way of integrating prior makes its analysis straightforward using the current techniques. However, it has got some limitations: 1) lack of extension to GP-UCB acquisition function, 2) lack of discussion around the extreme looseness in the bound that can happen when the prior is chosen improperly, and 3) lack of discussion around the relative scaling between the acquisition function and the prior function and its impact on the influence of the prior function in the process. I would go for weak accept. But I am ready to move up depending on the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-OetwftPlex", "reviewer": "Reviewer_vMEc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear and with extensive empirical evaluation. An asymptotic convergence results shows that there is only a constant factor difference with the regular method. The main drawback lies in the apparent simplicity of the approach, in terms of originality. Giving more weights to areas supposed to be better in the acquisition function optimization could already exist in application oriented papers (though I did not found such example with a quick search).\n\nMinor:\n- Consistent colors/symbols for methods across figures would help.\n- Additional details on how to define the prior would make the paper more self contained.\n- Note that the unbounded BO method would apply with the definition of starting bounds (say taking 50% of the volume under the prior).\n- The differences with BOPrO could be better highlighted.\n- I appreciate the theoretical result albeit one could hope to show some improvement from using a good prior.\n\nTypos:\nP1: documentesd", "summary_of_the_paper": "In this article the authors propose to incorporate domain knowledge on where good configurations are located in Bayesian optimization (BO). They do so by weighting the acquisition function by a prior that decays and reverts to uniform as iterations go. The asymptotic rate of convergence is shown to be of the same order as the non-prior version. Empirical tests are provided on various synthetic and  hyperparameter tuning tasks, with diverse baseline methods and ablation studies.", "summary_of_the_review": "If not particularly original, the proposition is well motivated and linked to the state of the art, plus it comes with extensive empirical results supporting the approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates Bayesian optimization where a prior distribution over the optimal is available. The authors conducted a systematic study on a very intuitive prior-augmented acquisition function that multiplication the prior probability with the EI heuristic --- including an asymptotic analysis on the regret, comprehensive (controlled) synthetic experiments, and moderate empirical support on several real-world case studies.

All reviewers find the paper well-written and appreciate the rigor of the empirical evaluation. The theoretical analysis is also helpful to provide additional justification for the proposed approaches. I would like to add that the paper also included a brief but comprehensive survey on prior work related to leveraging prior in BO, which I find useful for the general audience.

Reviewers noted that such a bound could become trivial with a bad prior in practice, and further suggest that one may leverage these theoretical insights as general guidelines to practitioners in designing the prior. I think this is a valuable message to convey and suggest authors take it into account in the revision.

There were initial confusions pertaining to the experimental details, mainly concerning the effect of the quality of the prior on the performance of the proposed algorithm. The authors provided an effective rebuttal with much concrete empirical support, and after a few rounds of interaction during the discussion phase, the reviewers are convinced about the empirical significance of the proposed work. Overall, this makes a solid work.
Decision: Accept (Poster)