Prior-guided Bayesian Optimization
Authors: Artur Souza
Luigi Nardi
Leonardo Oliveira
Kunle Olukotun
Marius Lindauer
Frank Hutter
Venue:  (2021)
Keywords: Bayesian Optimization
Automated Machine Learning

Abstract: While Bayesian Optimization (BO) is a very popular method for optimizing expensive black-box functions, it fails to leverage the experience of domain experts. This causes BO to waste function evaluations on bad design choices (e.g., machine learning hyperparameters) that the expert already knows to work poorly.  To address this issue, we introduce Prior-guided Bayesian Optimization (PrBO). PrBO allows users to inject their knowledge into the optimization process in the form of priors about which parts of the input space will yield the best performance, rather than BO’s standard priors over functions (which are much less intuitive for users). PrBO then combines these priors with BO’s standard probabilistic model to form a pseudo-posterior used to select which points to evaluate next. We show that PrBO is around 12x faster than state-of-the-art methods without user priors and 10,000x faster than random search on a common suite of benchmarks, and achieves a new state-of-the-art performance on a real-world hardware design application. We also show that PrBO converges faster even if the user priors are not entirely accurate and that it robustly recovers from misleading priors.

Under review as a conference paper at ICLR 2021
PRIOR-GUIDED BAYESIAN OPTIMIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
While Bayesian Optimization (BO) is very popular method for optimizing ex- pensive black-box functions, it fails to leverage the experience of domain experts. This causes BO to waste function evaluations on bad design choices (e.g.. machine learning hyperparameters) that the expert already knows to work poorly. To ad- dress this issue, we introduce Prior-guided Bayesian Optimization (PrBÓ). PrBO allows users to inject their knowledge into the optimization process in the form of priors about which parts of the input space will yield the best performance, rather than BO's standard priors over functions (which are much less intuitive for users). PrBO then combines these priors with BO's standard probabilistic model to form a pseudo-posterior posterior used to select which points to evaluate next. We show that PrBO is around 12x faster than state-of oftte-of-the-ate the-hhhtrtt art methods without user priors and 10,000 faster than random search state-of-the-ar performance on a real-world hardware design application. We also show that PrBO converges faster even if the user priors are not entirely accurate and that it robustly recovers from misleading priors.
common suite of benchmarks, and achieves new
INTRODUCTION
Bayesian Optimization (BO) is data-efficient method for the joint optimization of design choices that
gained great popularity in recent years. It is impacting a wide range of areas, including hyperparameter
optimization (Snoek et al. 2012 Falkner et 2018), AutoML (Feurer et al. 2015a Hutter et al.)
2018), robotics (Calandra et al. 2016). computer vision (Nardi et al. 2017: Bodin et al.) [0166). environmental monitoring (Marchant & Ramos] (012).. combinatorial optimization (Hutter al. 2011. experimental design (Azimi et
RI. (Brochu et al., 2010). Computer Go (Chen et al. 2018), hardware design (Koeplinger et al. 2018 Nardi et al., 2019) and many others. It promises
greater automation so as to increase both product quality and human productivity. As result, BO is also established in many large tech companies, e.g., with Google Vizier (Golovin et al. 2017) and
Facebook BoTorch (Balandat et al. 2019).
Nevertheless domain experts often have substantial prior knowledge that standard BO cannot incor- porate. Users can incorporate prior knowledge by narrowing the search space; however, this type of hard prior can lead to poor performance by missing important regions. BO also supports a prior over functions p(J), e.g., via a kernel function. However, this is not the prior experts have: users often know which ranges of hyperparameters tend to work best, and are able to specify a probability distribution (a))((=))
quantify these priors. E.g., many users of the Adam optimizer (Kingma & Ba 2015) know that its best learning rate is often in the vicinity of le-3 (give or take one order of magnitude), yet one may not know the accuracy one may achieve in new application. Similarly,
Navruzyan et al. (2019) derived neural network hyperparameter priors for image datasets based on their experience with five datasets. In these cases, users know potentially good values for new application, but cannot be certain about them.
As a result, many competent users instead revert to manual search, which can fully incorporate their prior knowledge. A recent survey showed that most NeurIPS 2019 and ICLR 2020 papers that reported having tuned hyperparameters used manual search, with only a very small fraction using BO (Bouthillier & Varoquaux] 2020). In order for BO to be adopted widely, and help facilitate faster progress in the ML community by tuning hyperparameters faster and better, it is therefore crucial to devise method that fully incorporates expert knowledge into BO. In this paper, we introduce Prior-guided Bayesian Optimization (PrBO), a novel BO variant that combines user prior knowledge with probabilistic model of the observations made. Our technical contributions with PrBO are:

---

Under review as a conference paper at ICLR 2021
1. PrBO bridges the TPE methodology and standard BO probabilistic models, such as GPs, RFs or Bayesian NNs, instead of Tree Parzen Estimators only.
2. PrBO is flexible w.r.t. how the prior is defined, allowing previously hard-to-inject (e.g. exponential) priors.
3. PrBO gives more importance to the model as iterations progress, gradually forgetting the prior and ensuring robustness against misleading priors.
We demonstrate the effectiveness of PrBO on a comprehensive set of real-world applications and synthetic benchmarks, showing that accurate prior knowledge helps PrBO to achieve similar perfor- mance to current state-of-the-art average x faster synthetic benchmarks and 1.49x faster on a real-world application. PrBO also achieves equal or better final performance in all but one the benchmarks tested.
BACKGROUND
2.1 BAYESIAN OPTIMIZATION
Bayesian Optimization (BO) is an approach for optimizing an unknown function f X + IR that expensive to evaluate over an input space X. In this paper, we aim minimize f. i.e., find x arg min x fx)) BO approximates x" with an optimal sequence of evaluations 21, .2.... E X. with each new 1 depending on the previous function values y1, 1/2 ... Un at æ1, Rn. BO achieves this by building a posterior on based on the set of evaluated points. At each BO iteration, a new point is selected and evaluated based on the posterior, and the posterior is updated to include the new point (Ix+1, 2n++))
The points explored by BO are dictated by the acquisition function, which attributes a value to each X by balancing the predicted value and uncertainty of the prediction for each In this work, as the acquisition function we choose Expected Improvement (EI) (Mockus et al. 1978). which quantifies the expected improvement over the best function value found so far: E1 fine x
1 max(Sine y, O)p(y jdyy,, where fine is the incumbent function value, i.e., the best objective function value found so far, and p(y|æ) ) is probabilistic model, e.g., a GP. Alternatives to El would be Probability of Improvement (PI) (200), upper-confidence bounds (UCB) (Srinivas et al.. 2010), and entropy based methods (e.g. -ernández-Lobato al. (2014)).
2.2
PARZEN ESTIMATOR
Whereas the standard probabilistic model in BO directly models p(ylæ)). the Tree-structured Parzen Estimator (TPE) approach of Bergstra al. (2011) models p(ælly) and p(y) instead" This done by constructing two parametric densities, g(x) and ((w), which are computed using the observations with function value above and below given threshold, respectively. The separating threshold y" is defined as a quantile of the observed function values. TPE uses the densities g g(x) and ((x: to define p(all)) as:
p(zly) = lxx)I(V < <²) + g(x)(1 Ily < v²)),
where I (y < y²) is when y < y" and 0 otherwise. The parametrization of the generative model p(æ, y) = p(xly)p(() facilitates the computation of EI as it leads to FTy (m) o 1(x:) gm)) and, thus, arg max FT y- (mx) arg max (x)).
3 BAYESIAN OPTIMIZATION WITH PRIORS
We propose BO approach dubbed PrBO that allows field experts to inject user prior knowledge into the optimization in the form of priors. PrBO combines this user-defined prior with a probabilistic
Note that, technically, the model does not parameterize p(y), since is computed based on the observed data points, which are heavily hiased towards low values due to the optimization process. Instead, it parameterizes dynamically changing Mu which helps constantly challenge the model to yield better observations.
2

---

(z) is normalized to [0, 1] by min-max Under review as a conference paper at ICLR 2021
model that captures the likelihood of the observed data (z,, y:) PrBO is independent from the probabilistic model being used, i.c., it can be freely combined with, e.g., GPs, RFs, or Bayesian Neural Networks.
3.1 PRIORS
PrBO allows users to inject prior knowledge into BO. This is done via prior distribution that informs where the input space we expect to find good f((x)) values. A point considered "good" if leads to low function values. We denote the prior distribution P,(x), where 9 denotes that this is prior on good points and a E X' is a given point. Similarly, we define a prior on where in the input space expect to have "bad" points. Although we could have separate user-defined probability distribution Po (z). we aimed to keep the load on users low and thus, for simplicity, only require the definition of Pa (a)) and compute (x(x)
Pa (x)
scaling before computing (2)..
In practice, ær contains several dimensions but it is difficult for domain experts provide multi- variate distribution (x). Users can easily specify, e.g., draw, univariate or bivariate probability distribution for continuous dimensions or provide list of probabilities for discrete dimensions. In PrBO, users are free to define a complex multivariate distribution, but we expect the standard use case to be that users only want to specify univariate distributions, implicitly assuming a prior that factors as (x( = I (x:), where is the number of dimensions in X' and r; is the i-th input dimension of X. We show examples of continuous and discrete priors in Appendices and F respectively. We use factorized priors in our experiments to mimic what we expect most users will provide. In Appendix G we show that these factorized priors lead to similar performance compared to multivariate priors.
3.2 MODEL
Whereas the standard probabilistic model in BO, e.g., a GP, quantifies p(y/æz) directly, that model is hard to combine with the user-defined prior Py (a:). We therefore now introduce method to translate the standard probabilistic model p [x) into a model that easier to combine with this prior. Similar the TPE work described in Sec. 2.2 our generative model combines p(æly) and p(y) instead of directly modeling p(yle [z).
The computation perform for this translation is to quantify the probability that given input is "good" under our standard probabilistic model p(y|æ)). As in TPE, we define configurations as "good" if their observed y-value is below a certain quantile y of the observed function values (so that p(y < f-) = 7). We in addition exploit the fact that our standard probabilistic model p(y|æz) (z) has Gaussian form, and under this Gaussian prediction we can compute the probability My (2) of the function value lying below a certain quantile using the standard closed-form formula for PI (Kushner 1964):
Mg(x) M((x)))(((()) Salae, (m,, v:) +
dx
where (x.; y:) are the evaluated configurations and their function value, Ma and 0a are the mean and standard deviation of the probabilistic model at and & is the standard normal CDF, see Figure] Note that there are two probabilistic models here:
by The standard probabilistic model of BO, with prior over functions p(f), updated by data (D), yi to yield a posterior over functions px),( (.)..)))))))) allowing us to quantify the probability M,( = p(f < f (N,, y:)) in Equation 2
The TPE-like generative model that combines p(y) and p(ælyy) instead of directly modelling p(y]æ=).
Equation 2 bridges these two models by using the probability of improvement from BO's standard model the probability MG (z:) ar) in TPE's model. Ultimately, this is heuristic since there no formal
We note that for continuous spaces. this Pb(x) is not probability distribution, and therefore only pseudo- prior, it does not integrate to 1. For discrete spaces, we normalize Ps(x)) so that it sums to 1 and therefore is proper probability distribution and prior.
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 PrBO Algorithm. D keeps track of all function evaluations so far: (z,, W:)
Input: Input space X, user-defined prior distributions (z(x) and (x)x).. quantile and BO budget B.
[(x)
2: Output: Optimized point Wine-
3: D f Initialize(A))
4: for l = 1 B do
5: Mg(x)) + fil_model good D)
fr
M2(a)) fil_model bad (D)
(2(a)
T
7:
g(x)) + Pg(x) Mg(xa)
8:
((x) + Pa( Ma(x))
Figure Our model composed by a prob- abilistic model and the probability of improv-
3r E arg max ET (ac)
10:
y1 + f(x1)
ing over the threshold fy. i.c., right tail the Gaussian. The black curve is the probabilistic variance.
11: D= DU (D
12: end for
model's mean and the shaded area is the model's
13: Zinc f- ComputeBest((D
14: return Wine
connection between the two probabilistic models. However, we believe that the use of BO's familiar, theoretically sound framework of probabilistic modelling of p(v] followed by the computation of the familiar formula a very intuitive choice for obtaining the probability of an input achieving at least given performance threshold exactly the term need for TPE's (x) Similarly, also define a probability My(x) of æ being bad as Mp(x)= = (2)
3.3 PSEUDO-POSTERIOR
PrBO combines the prior in Section [3. and the model in Eq. 2 into a pseudo-posters on "good" points. This postterro represents the updated beliefs on where we can find good points, based on the prior and data that has been observed. The pseudo-posterior is computed as the product of the prior and the model:
g(x)) x (a))
where is the current optimization iteration, is optimization hyperparameter, Mz( is defined in Eq. (2). and (x) is the prior defined in Sec 3.1 rescaled to [0, 1]. We note that this pseudo- posterior is not normalized, but this suffices for PrBO determine the next at; the normalization constant cancels out (c.f. Section3.44 Since g(x) is not normalized and we include the exponent t/3 in Eq. 3 we refer g(x) as a pseudo-posterior to emphasize that it is not standard posterior probability distribution.
The t/ß fraction in Eq. (3) controls how much weight is given to the model. As the optimization progresses, more weight is given to the model over the prior. Intuitively, we put more emphasis on the model as it observes more data and becomes more accurate. We do this under the assumption that the model will eventually be better than the user at predicting where to find good points. This also allows to recover from misleading priors as we show Section [.22 similar to, and inspired by Bayesian models, the data ultimately washes out the prior. The 3 hyperparameter defines the balance between prior and model, with higher 3 values giving more importance to the prior and requiring more data overrule it.
We note that, computing Equation 3 directly can lead to numerical issues. Namely, the pseudo- posterior can reach extremely low values if the prior and model probabilities are low, especially as the t/B exponent grows. To prevent this, in practice, PrBO uses the logarithm the pseudo-posterian instead: log(g(=)) log( (og(P)())) llog(Mg (2:).
4

---

Under review as a conference paper at ICLR 2021
Once again, also define analogous gseudo-posterni distribution bad b(x) & (a(r)) 3. We then use these quantities to define density model p(æly) as follows:
p(zly) x [(()) if < Jn ((xz) if 2 87
(4)
3.4 ACQUISITION FUNCTION
We adopt the EI formulation used in (Bergstra et al. 2011) by replacing their Adaptive Parzen Estimators with our computation of the pscudo-posterior Eq. (3). Namely, we compute El as:
E1,, (x) = {
cinf
max(f,-8) -1) [' inf
p(xl))p(() dy xc (7+ b(z) g(x)
(f, 1---y) p())
(1 ?)
inf
The full derivation of Eq. (5) is shown in Appendix C Eq. 15 shows that to maximize improvement we would like points æ with high probability under g(x) and low probability under b(x). i.e.. minimizing the ratio b(z)/o ((x)/((1)) (x)) We note that the point that minimizes the ratio for our unnormalized pseudo-posteriors will be the same that minimizes the ratio for the normalized pseudo-postering and,
thus, the computation of the normalized ppeudo-posterion is unnecessary.
The dynamics of the PrBO algorithm can be understood in terms of the following proposition (proof in Appendix C
Proposition 1 Given fr, Pz(2), Po(z), Mg(x), Mo(x), g(x), b(xz), p(zly), and / as above, then lim arg F.T,, (ar) lim arg max My(ar) t- =EX
t-00
#EX
where EL5,, is the Expected Improvement cnniisittionfuncciio defined in Eq. and Mg(w)) is as
defined in Equation
In early BO iterations the prior will have predominant role, but in later BO iterations the model will grow more important, and as Proposition shows, if PrBO is run long enough the prior washes out and PrBO only trusts the probabilistic model informed by the data.
3.5 PUTTING IT ALL TOGETHER
Algorithm 1 shows the PrBO algorithm, based the components defined in the previous sections. In Line 3, PrBO starts with a design of experiments (DoF) phase, where it randomly samples a number of points from the user-defined prior P, (a:). After initialization, the BO loop starts Line 4. In each loop iteration, PrBO fits the probabilistic model on the previously evaluated points (lines 5 and 6) and computes the pseudo-posteriors g(æ) and b(xz) (lines and 8 respectively). The El acquisition function is computed next, using the poauddr-ssrrrors.... and the point that maximizes El is selected as the next point to evaluate line The black-bol function evaluation is performed at Line 10. This BO loop is repeated for pre-defined number of iterations, according to the user-defined budget B. In Appendix A we show a visualization of PrBO on I-dimensional function.
EXPERIMENTS
We implement both GPs and RFs as predictive models and use GPs in all experiments, except for our real-world experiments (Section[4.4 4.4) where we use RFs for a fair comparison. We set the model weight 10 and the model quantile to 7 = 0.05, see our sensitivity hyperparameter study in Appendices D and M Before starting the main BO loop in PrBO, we randomly sample D + 1 points Appendix we compare PrBO to other baselines with the same prior initialization and show that it leads to similar results. We optimize our EI acquisition function using a multi-start local search, similar to SMAC (Hutter et al., 201 We start with four synthetic benchmarks: Branin, SVM, FC-Net, and XGBoost, which are 2. 2. 6, and dimensional, respectively. The last three are part of the Profet benchmarks (Klein et al. 2019). generated by generative model built using performance data on OpenML or UCI datasets. See
from the prior consistently on all benchmarks.
5

---

Under review as a conference paper at ICLR 2021
Prior
Model
Log Posterior
Model
Log Posterior
Model
Log Posterior
1.20
1.90
1.20
1,00
Log Posterior
Log Posterior
Model
0.50
0.50
0.50
J0
0.50
0.25
0.25
0.25
-15
0.5
0.00
0.50
0.30
0.00
Real Function
GP
Optimum
Incumbent
R5
X
8D
Threchald
X
Next point
8.81
200
0.43
8.40
Tbx)
100
X
(a) No samples
(b) 0 BO iterations
(c) 10 BO iterations
(d) 20 BO iterations
Figure 2: PrBO on the 1D) Branin function. The leftmost column shows the exponential prior. The other columns show the model and the log pseudo-posterion after (RS only), 10, and 20 BO iterations. PrBO forgets the wrong prior on the local optimum and converges the global optimum.
Appendix D for more details on the experimental setup. Due to space constraints, defer the (qualitatively similar) results for the SVM benchmark to Appendix
4.1 PRIOR SELECTION
In this section we study the effect of choosing a prior. A suitable property of the prior is that, by would expect sampling from the prior to have an
selecting a tighter prior around an optimum, increased performance. To the limit, if the prior is composed by only one point which is one of the global optima, then the first sample (and all them) from the prior will hit the optimum. To have a sanity check of this property, we build an artificial prior in controlled way. We rely on an automated computation of the prior by computing a univariate Kernel Density Estimation (KDE) using Gaussian kernel the synthetic benchmarks introduced above. We note that the goal of these synthetic priors is to have an unbiased prior for our experiments, whereas manual priors would be biased by our expertise of these benchmarks. In practice, users will manually define these priors without needing additional experiments. Appendix G shows that using univariate KDEs for the prior lead similar performance compared multivariate KDEs.
We experiment with an array of varying quality priors. We select a constant 10D points in each prior and vary the size of the random sample dataset so that we can make the priors more sharply peaked around the optima in a controlled environment. We use the best performing 10/ samples to create the prior from a uniform random sample dataset size of 10.100, we refer this prior as x% in Figure 3 As an example the XGBoost benchmark has d = 8, so, 100% means we sample 80 points and use all 80 create the prior, 10% means we sample 800 points and use the best performing 80 to create the prior, 1% means we sample 8,000 and use the best 80 to create the prior, and so on.
Figure 3 shows the performance of purely sampling from the prior and running PrBO, respectively, after 107) function evaluations with different priors. A bigger random sample dataset and smaller percentage leads to tighter prior around the optimum, making the argument for stronger prior. This is confirmed by Figure 3 where sharply peaked prior (right side of the figure) leads to better performance in both scenarios. PrBO achieves smaller regret by being able evolve from the initial prior and making independent steps towards better values of the objective function. More extensive experiments with a similar trend, including the rest of the benchmarks, are in Appendix K
addition we observe that in contrast to sampling from the prior,

---

Under review as a conference paper at ICLR 2021
Simple Regret
XGBoost Prior Sampling
RS (10,000x)
Spearmint
Prior Sampling
No Prior
PrBO Weak Prior mitalizttoon
PrBO Prior
Log Simple Regret
10
Branin
FC-Net
XGBoost
12
100%
10%
8.1%
0014 0.0%
0.001
-10
200 Number of Evaluations
300
100
200
Number Evaluations
Number Evaluations
Figure 3: Regret of prior sam- pling and PrBO with different priors (11 + on reps.).
Figure 4: Log regret comparison of PrBO with and without priors, RS, and Spearmint (mean +/- one std on 5 repetitions). We run the benchmarks for 100D iterations, capped at 300.
4.2 PRIOR FORGETTING
In this section, we show that PrBO can recover from a misleading prior, thanks to our predictive model and the t/3 parameter in the predictive model becomes more accurate and receives more weight, guiding optimization away from the wrong prior and towards better values of the objective function. Showws PrBO on the ID Branin function with an exponential prior. Columns (b). (c), and (d) show PrBO after + = initial samples and 0, 10, 20 BO iterations, respectively. After initialization, as shown in Column (b), the pseudo-posterrnn is nearly identical to the exponential prior and guides PrBO towards the region of the space on the right, which is towards the local optimum. This happens until the predictive model becomes certain there will be no more improvement from sampling that region (Columns (c) and (d)). After that, the predictive model guides the pseudo-posterie posterior towards exploring regions with high uncertainty. Once the global minimum region is found, the peeeud--osterior starts balancing exploiting the global minimum and exploring regions with high uncertainty, as shown in 2 (bottom). Notably, the pseudo-posterior after x >> 4 falls to 0 in 2 (top), as the predictive model is certain there will be no improvement from sampling the region of the local optimum. We provide additional examples of forgetting in Appendix B and comparison of PrBO with misleading priors, no prior,
computation Eq. 3 As BO progresses, the
and correct priors in Appendix H
4.3 COMPARISON AGAINST STRONG BASELINES
Figure - compares PrBO to other optimizers using the log simple regret on five runs (mean and std error reported) on the synthetic benchmarks. We consider two priors in our experiments, a strong prior, computed using KDE on the best 102 out of 10,000,000 uniform random samples, and weak prior, computed using a KDE on the best 10D out of 1,000D uniform random samples. We emphasize that we only used these artificial priors in these experiments guarantee that our prior is not biased by our own expertise for the benchmarks we used. In practice, the prior is defined by the user. We compare the results of PrBO with and without priors (both weak and strong) to 10,000x random search (RS. i.c., for each BO sample we draw 10,000 uniform random samples), sampling from the strong prior only, and Spearmint (Snock al. 2012) which is well-adopted BO approach using GPs and the EI acquisition function. In Appendix we show a comparison of PrBO to TPE and SMAC.
PrBO Prior beats 10,000x RS and PrBO weak prior on all benchmarks. It also outperforms the performance of sampling from the prior; this is expected because prior sampling cannot recover from non-ideal prior. The two methods are identical up to the initialization phase because they both sample from the same prior in that phase.
PrBO Prior is more sample efficient and finds better or equal results than Spearmint on three out of the four benchmarks. On the fourth benchmark, XGBoost, PrBO leads the performance until 139 BO iterations, where Spearmint catches up and achieves slightly better results in the end (function values of 8.986 for Spearmint vs. 9.026 for PrBO after 300 iterations). Importantly, in all our experiments, PrBO with good prior consistently shows tremendous speedups in the early phases of the optimization process, typically only requiring on average 8.25 iterations to reach the performance that Spearmint reaches after 100 iterations (12. (12.12x faster). Thus,, in comparison other traditional
7

---

Under review as a conference paper at ICLR 2021
RS
HyperMapper
PrBO
Expert Configuration
Initialization
24M
Shallow CNN
Deep CNN
9M
MD Grid
18M
15M
Cycles (log)
15M
Cycles (log)
9M
Cvcles (log)
1M
11M
5M
400k
9M
3M
200k
SM
1M
20
20%
100
150
Number of Evaluations
Number of Evaluations
Number of Evaluations
Figure 5: Log regret comparison of RS, HyperMapper, PrBO, and manual optimization on Spatial. The line and shaded regions show mean and std after 5 repetitions. Vertical lines are initialization.
BO approaches, PrBO makes use of the best of both worlds, leveraging prior knowledge and efficient optimization based on BO.
4.4 THE SPATIAL USE-CASE
We next apply PrBO to the Spatial (Koeplinger et al., 2018) real-world application. Spat programming language and corresponding compiler for the design of application accelerators, i.e., FPGAs. We apply PrBO to three Spatia benchmarks, namely, 7D shallow and deep CNNs, and 10D molecular dynamics grid application. We compare the performance of PrBO to RS, manual optimization, and HyperMapper (Nardi et al.) 2019). the current state-of oof-the-ar art BO solution for Spatial. For fair comparison, both PrBO and HyperMapper use RFs predictive probabilistic model. The manual optimization and the prior for PrBO were provided by an unbiased Spatial developer, who is not an author of this paper. The priors were provided once and kept unchanged for the whole project. More details on the setup, including the priors used, are presented in Appendix
F shows the log regret the Spatial benchmarks. PrBO vastly outperforms RS in all benchmarks; notably, RS does not improve over the default configuration in MD Grid. PrBO is also able leverage the expert's prior and outperform the expert's configuration in all benchmarks (2.68x, x, 1.06x, and 10.4x speedup for shallow CNN, deep CNN, and MD Grid, respectively). Compared to HyperMapper, PrBO achieves better performance in the MD Grid benchmark (1.28x speedup). For context, this is significant improvement in the FPGA field, where 10% improvement could qualify for acceptance in a top-tier conference. In the CNN benchmarks, PrBO converges to minima regions faster than HyperMapper (1.58x and 1.4x faster for shallow and deep respectively). Thus, PrBO leverages the best of both worlds (the expert's 's prior knowledge and BO) to provide a new state of the art for Spatial.
RELATED WORK
TPE by Bergstra et al. (011), the default optimizer in the popular HyperOpt package (Bergstra et al. 3013) supports limited hand-designed priors in the form of normal or log-normal distributions. We make three technical contributions that make PrBO more flexible than TPE. First, we generalize over the TPE approach by allowing more flexible priors; second, our approach is aodel-agnostic (i.e.. PrBO is not limited to the TPE model; we use GPs and RFs in our experiments); and third, PrBO is inspired by Bayesian models that give more importance to the data as iterations progress. We also show that PrBO outperforms HyperOpt's TPE in Appendix! [
In parallel work, Li et al. (2020) allow users to specify priors via probability distribution, similarly to our approach. Their two-level approach first samples number of configurations by maximizing Thompson samples from a GP highest prior the next to evaluate. In contrast, method leverages the information from the prior more directly; is agnostic to the probabilistic model used, which is important for applications with many discrete variables like our world application, where RFs outperform GPs; and provably
and then chooses the configuration that has the
8

---

Under review as a conference paper at ICLR 2021
recovers gracefully from misspecified priors, while the approach of Li trusts the prior and never gets washed out by more data..
Li et al. (2020) fundamentally
The work of Ramachandran et al. (2020) also supports priors in the form of probability distributions. Their work uses the probability integral transform to warp the search space, stretching regions where the prior has high probability, and shrinking others. Once again, compared their approach, PrBO is agnostic to the probabilistic model used and directly controls the balance between prior and model via the 3 hyperparameter. Additionally, PrBO's s probabilistic model is fitted independently from the prior, which ensures it is not biased by the prior, while their approach fits the model to a warped version of the space, transformed using the prior, making more difficult recover from misleading priors.
Black-box optimization tools, such as SMAC (Hutter et al. 201T) or iRace (López-lbäñez al. 2016) also support simple hand-designed priors, e.g. ing-transsfommations However, these are not properly reflected in the predictive models and both cannot explicitly recover from bad priors.
Oh et al. (2018) and Siivola et al. (2018) propose structural priors for high- -igensinnsi problems.
They assume that users always place regions they expect to be good at the center of the search space and then develop BO approaches that favor configurations near the center. However, this is rigid
assumption about optimum locality, which does not allow users to freely specify their priors.
Similarly, Shahriari et al. ( focus on unbounded search spaces. The priors in their work are not about good regions of the space, but rather a regularization function that penalizes configurations based on their distance to the center of the user-defined search space. The priors are automatically
derived from the search space and users may not even be aware of the priors that are used.
Our work also relates to meta- meta-learning for BO /Vanschoren, 2019), where BO is applied to many similar optimization problems in a sequence such that knowledge ge about the general problem structure be exploited in future optimization problems. In contrast meta-learning, PrBO is the first method that allows human experts to explicitly specify their priors. Furthermore, PrBO does not depend on any meta-features (Feurer et al. 2015b) and only incorporates a single prior instead of many priors from different experiments (Lindauer & Hutter 2018).
CONCLUSIONS AND FUTURE WORK
We have proposed novel BO variant, PrBO, that allows users to inject their expert knowledge into the optimization in the form of priors about which parts of the input space will yield the best performance. These are different than standard priors over functions which are much less intuitive for users. BO failed so far to leverage the experience of domain experts, not only causing inefficiency but also getting users away from applying BO approaches because they could not exploit their years of knowledge in optimizing their box functions. PrBO addresses this issue and will therefore facilitate the adoption of BO. We showed that PrBO is 12.12x more sample efficient than state-of-the-an art methods, and 10,000x faster than random search, on common suite of benchmarks and achieves a new of-te----the-rr performance on a real-world hardware design application. We also showed that PrBO converges faster and that it robustly recovers from misleading priors.
In future work, we will study how our approach can be used to leverage prior knowledge from meta-learning. Bringing these worlds together will likely boost the performance of BO even further.
REFERENCES
Javad Azimi, Ali Jalali, and Xiaoli Zhang Fern. Hybrid batch bayesian optimization. In Proceedings of the 29th International Conference on Machine Learning, ICML. icml.cc / Omnipress, 2012.
Maximilian Balandat, Brian Karrer, Daniel R Jiang, Samuel Daulton, Benjamin Letham, Andrew Gor- don Wilson, and Eytan Bakshy. Botorch: Programmable bayesian optimization in pytorch. arXiv preprint arXiv:1910 00400, 2019.
J. Bergstra, D. Yamins, and D. D. Cox. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference International Conference on Machine Learning, 2013.

---

Under review as a conference paper at ICLR 2021
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for optimization. In Advances in neural information processing systems, pp.
2011.
Bruno Bodin, Luigi Nardi, M Zeeshan Zia, Harry Wagstaff, Govind Sreekar Shenoy, Murali Emani, John Mawer, Christos Kotselidis, Andy Nisbet, Mikel Lujan, al. Integrating algorithmic parame- ters into benchmarking and design space exploration in 3d scene understanding. In Proceedings of the 2016 International Conference Parallel A Architectures and Compilation, pp. 57-69, 2016.
Xavier Bouthillier and Gaël Varoquaux. Survey of machine-leariing experimental methods at NeurIPS2019 and ICLR2020. Research report, Inria Saclay Ile de France, January 2020. URL
futtps://hal www..a.acc.vveeouveeeeettet.
Eric Brochu, Vlad M Cora, and Nando De Freitas. tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv: 1012.2599 2010.
Roberto Calandra, André Seyfarth, Jan Peters, and Mare Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence, 76(1-2):5-23, 2016.
Yutian Chen, Aja Huang, Ziyu Wang, Ioannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in alphago. CoRR, abs/1812.00855. 2018.
Laurence Charles Ward Dixon. The global optimization problem: an introduction. Toward global optimization, 2:1-15, 1978.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter opti- mization at scale. In Proceedings of the 35th International Conference on Machine Learning, pp. 1436-1445, 2018.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2962 2970. Curran Associates, Inc., 2015a.
Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 1135, 2015b.
Jacob Gardner, Matt Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John P Cunningham. Bayesian optimization with inequality constraints. In Proceedings of the 31st International Conference on Machine Learning, ICML, 2014.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley, Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
GPy. GPy: A gaussian process framework in python. http: Nttp://githube /gthtttttbbb......gggg www.e/ocom/sseeeeeee Shmfffefffi CPy since 2012.
José Miguel Hernández-Lobato. Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Information Process- ing Systems 27, pp. 918-926. Curran Associates, Inc., 2014.
F. Hutter, L. Xu, H. Hoos, and K. Leyton-Brown. Algorithm runtime prediction: Methods & evaluation. Artificial Intelligence, 206:79-111, 2014.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Browe Sequential model-based optimization for general algorithm configuration. In International conference learning and intelligent optimization, pp. 507-523. Springer, 2011.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2018. In press, available at http://attoml.org/hor
10

---

Under review as a conference paper at ICLR 2021
Donald R Jones. A taxonomy of global optimization methods based on response surfaces. Journal of global optimization, 21(4):345-33) 3,,,,38,,020 2001.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR, 2015.
Aaron Klein, Zhenwen Dai, Frank Hutter, Neil D. Lawrence, and Javier Gonzalez. Meta-surrogate benchmarking for hyperparameter optimization. In Advances in Neural Information Processing Systems NeurIPS, pp. 6267-6277,2019.
David Koeplinger, Matthew Feldman, Raghu Prabhakar, Yaqi Zhang, Stefan Hadjis, Ruben Fiszel, Tian Zhao, Luigi Nardi, Ardavan Pedram, Christos Kozyrakis, and Kunle Olukotun. Spatial: A Language and Compiler for Application Accelerators. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), June 2018.
Harold Kushner. A new method of locating the maximum point an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, 86(1):97-106, 1964.
Cheng Li, Sunil Gupta, Santu Rana, Vu Nguyen, Antonio Rebles-Kelly, and Svetha Venkatesh. Incorporating expert prior knowledge into experimental design via posterior sampling. arXiv preprint arXiv:2002. / 1256, 2020.
Marius Lindauer and Frank Hutter. Warmstarting model-based algorithm configuration. In Proceedings of the Thirty-Seconl AAAI Conference on Artificial Intelligence, pp. 1355-1362, 2018.
Manuel López-lhäñez, Jérémie Dubois-Lacoon Leslie Pérez Cáceres, Thomas Stützle, and Mauro Birattari. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43-58, 2016
Roman Marchant and Fabio Ramos. Bayesian optimisation for intelligent environmental monitoring. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 2242-2249. IEEE, 2012.
Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for seeking the extremum. Towards global optimization, 2(117-129):2, 1978.
Luigi Nardi, Bruno Bodin, Sajad Saeedi, Emanuele Vespa, Andrew J Davison, and Paul HJ Kelly. Algorithmic performance-accoee accuracy trade-off in 3d vision applications using hypermapper. In 2017 IEEE International Purallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 1434-1443. IEEE, 2017.
Luigi Nardi, David Koeplinger, and Kunle Olukotun. Practical design space exploration. In 27th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecom- munication Systems, MASCOTS, 2019.
Arshak Navruzyan, Frank Sharp, Jeremy Howard, and Antoine Saliou. Optimizing hy- perparams for image datasets in fastai. httts::/plattorrs a wt/blo//////eee/aaeeee httpll//lg///////////// page/1/ optimi hwag--hhpppprraassffeee
-tteeeddatttttts--ddat-ee- 2019.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
C. Oh, E. Gavves, and M. Welling. BOCK: Bayesian optimization with cylindrical kernels. In
Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML, volume 80 of Proceedings of Machine Learning Research, pp. 3865-3874. PMLR, 2018.
Andrei Paleyes, Mark Pullin, Maren Mahsereci, Neil Lawrence, and Javier González. Emulation of physical processes with emukit. In Second Workshop on Machine Learning and the Physical Sciences, NeurIPS, 2019.
11

---

Under review as a conference paper at ICLR 2021
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Anil Ramachandran, Sunil Gupta, Santu Rana, Cheng Li, and Svetha Venkatesh. Incorporating expert prior in bayesian optimisation via space warping. Knowledge-Bast -asee Systems, 195:105663, 2020.
Bobak Shahriari, Alexandre Bouchard Côté, and Nando Freitas. Unbounded bayesian optimization via regularization. In Artificial intelligence and statistics, pp. 1168-1176, 2016.
Eero Siivola, Aki Vehtari, Jarno Vanhatalo, Javier González. and Michael Riis Andersen. Correcting boundary over-exploration deficiencies in bayesian optimization with virtual derivative sign obser- vations. In 2018 IEEE 28th International Workshop Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2018.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems (NeurIPS), pp. 2960-2968, 2012.
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Johannes Fürnkranz and Thorsten Joachims (eds.). Proceedings of the 27th International Conference Machine Learning (ICML-10), pp. 1015-1022. Omnipress, 2010.
Joaquin Vanschoren. Meta-lcarning, In Automated Machine Learning Methods, Systems, Challenges, pp. 35-61. 2019.
12

---

Under review as a conference paper at ICLR 2021
Prior
Model
Log Posterior
Model
Log Posterior
Model
Log Posterior
1.00
1.00
2.00
1.00
Log Posterior
Model
Model
0.50
0.50
1o
0.50
0.50
A25
0.2
A25
0.00
0.00
2.00
0.03
Real Function
GP
Optimum
Incumbent
RS
X
8D
Threshald
X Next point
EEG0
$1.00
5.40
0.40
3.26
9.40
9.40
200
0.40
100
100
01
20
3.35
0
0.71
Dig
10
0
(a) 0 BO iterations
(b) 5 BO iterations
(c) 10 BO iterations
(d) 20 BO iterations
Figure 6: Breakdown of the prior Po(2) = (3, 3). the model M2(x), and the pseudo-posterne g(m) (top row) for the 1-dimensional Branin function (bottom row) and their evolution the optimization iterations. Two random points from the prior are sampled to initialize the GP model (bottom row) before starting PrBO. The blue/green crosses denote PrBO samples, with green samples denoting later iterations, The blue line and shaded area show the mean and uncertainty of the GP model.
A MODEL AND PSEUDO-POSTERING VISUALIZATION
This section visualizes the prior (ar), the model Mg and the pseudo-posterior g(x) for a 1 1-dimensional Branin function and their evolution over the optimization iterations. We define this function by setting the second dimension of the Branin function to the global optimum x2 2.275 and optimizing the first dimension. We perform an initial design of D from the prior and use a GP as predictive model. We use a Beta distribution prior Pz(x) ( 3) which resembles truncated Gaussian centered close the global optimum and compute the model Mg (æ) and iseudo-posterior g(x> following Eq. and respectively. Figure 6 shows the optimization at different stages.
random points sampled
Figure 6a shows the initialization phase (bottom) and the Beta prior (top). After 5 BO iterations, in Figure 6 (top), the pseudo-postering is high near the global minimum, around x = t, where both the prior and the model agree there are good points. After 10 BO iterations in Figure 6C (top), there are three regions with high sneudo-posterion the optimum is found, and two regions the right and left, which will lead to future exploration as shown in Figure bl (bottom) on the right and left of the global optimum in light green crosses. After 20 iterations, see Figure 6d (top), the weudo-posteriig vanishes where the model is certain there will be no improvement, but it is high wherever there is uncertainty in the GP. Note that the influence of the prior after 20 iterations is weaker, because of t/3 Eq.
The middle region, where PrBO is exploiting until
B PRIOR FORGETTING SUPPLEMENTARY EXPERIMENTS
In this section, we show additional evidence that PrBO can recover from wrongly defined priors so to complement section 4.2 Figure shows PrBO on the ID Branin function as in Figure 2 but with decay prior. Column (a) of Figure 7 shows the decay prior and the 1D Branin function. This prior emphasizes the wrong belief that the optimum is likely located on the left side around x 5 while the optimum is located at the orange dashed line. Columns (b), (c), and (d) of 7 PrBO on the ID Branin after D D+1 1 = 2 initial samples and 0, 10, and 20 BO iterations, respectively, In the beginning of BO, as shown in column (b), the pseudo-poste posterior is nearly identical to the prior and guides PrBO towards the left region of the space. As becomes more accurate and starts guiding the pseudo-posterin away from the wrong prior (column (c)). Notably, the pseudo-posterior before falls to O, the predictive model is certain there will
points sampled, the model
13

---

Under review as a conference paper at ICLR 2021
Prior
Model
Log Posterior
Model
Log Posterior
Model
Log Posterior
1.30
1.00
1400
1.00
Log Posterior
Log Posterior
Model
0.50
0.50
050
-5
0.50
-13
0.25
0.25
-11
0.25
0.00
0.50
0.00
0.00
0:40
200
200
200
100
100
(a) No samples
(b) 0 BO iterations
10 BO itcrations
(d) 20 BO itcrations
Figure 7: PrBO the ID Branin function with decay prior. The leftmost column shows the log pseudo-posterior before any samples are evaluated, in this case, the pseudo-posterior is equal to the decay prior. The other columns show the model and pseudo-posterior after 0 (only random samples), 10, and 20 BO iterations. 2 random samples are used to initialize the GP model.
be no improvement from sampling this region. After 20 iterations, PrBO finds the optimum region, despite the poor start (column (d)). The peak in the pseudo-posterior in column (d) shows PrBO will continue to exploit the optimum region as it is not certain if the exact optimum has been found. The pseudo-posterior is also high in the high uncertainty region after 4, showing PrBO will explore that region after it finds the optimum.
Random Samples
X
Next point Incumbent
Optima
B Bayesian Optimization
Prior
0.0
Posterior
14
12
X
12
-500
550
-1000
10
10
1550
8
8-
10.0
-000
X
-12.5
-2500
X
2
-300
0s
5
10
-17.5
5
5
10
-3500
X1
X1
Figure 8: PrBO on the Branin function with exponential priors for both dimensions. (a) shows the log pseudo-posterior before any samples are evaluated, in this case, the pseudo-posterine is equal to the prior; the green crosses are the optima. (b) shows the result of optimization after initialization samples drawn from the prior at random and 50 BO iterations. The dots in (b) show the points explored by PrBO, with greener points denoting later iterations. The colored heatmap shows the log of the pseudo- posttrrro g(x).
Figure g shows PrBO on the standard 2D Branin function. We use exponential priors for both dimensions, which guides optimization towards region with only poor performing high function values. 8 shows the prior &p shows optimization results after + and 50 BO iterations. Note that, again, optimization begins near the region incentivized by the prior, but moves away from the prior and towards the optima as BO progresses. After 50 BO iterations, PrBO finds all three optima regions of the Branin.
initialization samples
14

---

Under review as a conference paper at ICLR 2021
C MATHEMATICAL DERIVATIONS
C.I EI DERIVATION
Here, provide full derivation of Eq. (5)
inf
max(fy 2)) - inf
(fr (f~-w) y) p(azl))pl)) p(x))
EI (z)
dy.
(6)
inf
As defined in Section
3.2
ply < [7) 7 and is quantile of the observed objective values (1)).
Then p()) = Jg p( y PP pyyddy = 7g(x)) + (1 7))(2)), where g(æx and b(x: are the posteriors
introduced in Section 3.3 Therefore
LT ())==-))())))ppp) 9
1 /" (fa viplu)dy = (fng(x)= g(xx) L" Inf
vp(v)dy, (7)
inf
inf
so that finally
(x) = (/g(()) - g(xz) 14y inf up(u)dy
x (x b(xx) g(x) (1 - 7) -1
(8)
(g(x) + (1 - (1-))(((z)
C.2 PROOF OF PROPOSITION
Here, provide the proof of Proposition !
1 (Sr - -y)((al))p())d
lim arg max FTy, (a)) (+000 sEx
lim arg max
(9)
1-000 #EX
inf
1 (fr (f~- y)p(y)dy
lim arg max g(x)
(10)
aEx
int
(Tho(l)) - g((x)) 1"hh sp(u)dy
lim arg max
(11)
#EX
int
7f-9(z) - g(x) S up(y)dy
lim
arg max
(12)
t-000
=ex
(g(x) + (1 - (()(e)))
b(xr)
lim
arg max
7) g(x)
(13)
2-000
sEX
b(x))
-
= lim arg max
(1 7) ((x)
(14)
t-000
#EX
Po(x)
= lim arg max
(15)
aEX
Py(x)MM(z) (x)
Po(x))o(2)) (1-7)
lim arg max
(16)
aEX
Pa(x)
(x(x)
Mo(x)
lim arg max
(1- = (17)
sEX
Pa(x))
Mg(z)
(M
lim arg max
(18)
t-
sEx
M2(x))
1 1- M,
-
lim arg max
(19)
1-000
TEX
M2
15

---

Under review as a conference paper at ICLR 2021
-
-1)
lim arg max
(00)
1-00
wEX
M2(x)
= lim arg max (M₂(
(21)
t-000
sEX
lim
arg max M2
(22)
70000
DEX
(23)
This shows that as iterations progress, the model grows more important. If PrBO is run long enough, the prior washes out and PrBO only trusts the probabilistic model.
D EXPERIMENTAL SETUP
We use a combination of publicly available implementations for our predictive models. For our Gaus- sian Process (GP) model. we use GPy's (GPy) since 2012) GP implementation with the Matérn5/2 kernel. We use different length-scales for each input dimensions, learned via Automatic Relevance Determination (ARD) (Neal, 2012). For our Random Forests (RF), we use scikit-learn's 's RF imple- mentation (Pedregosa et al. 2011). We set the fraction of features per split to 0.5, the minimum number of samples for a split to and disable bagging. We also adapt our RF implementation to use the same split selection approach as Hutter et al. (2014).
For our constrained Bayesian Optimization (cBO) approach, we use seikit-leam's RF classifier, trained on previously explored configurations, to predict the probability of a configuration being feasible. We then weight our El acquisition function by this probability of feasibility, as proposed by Gardner et al. (2014). We normalize our El acquisition function before considering the probability of feasibility, to ensure both values are in the same range. This cBO implementation is used in the Spatial use-case as in Nardi et al. (011)).
For all experiments, we set the model weight hyperparameter to 3 = 10 and the model quantile to y 0.05, see Appendices and random sampling D + 1 points from the prior, where is the number of input variables. We use the public implementation of Spearmin O
and [ Before starting the main BO loop, PrBO is initialized by
which by default uses 2 random samples for initialization. We
set the bandwidth of our KDE priors to 100n- where D is the number of input dimensions, see Appendix K We normalize our KDE priors before computing the pseudo-posterior, to ensure they are the same range as our model. We also implement interleaving which randomly samples point to explore during BO with a 10% chance.
We optimize our EI acquisition function using a multi-start local search, similar to the one used in SMAC (Hutter et al. 2011). Namely, we start local searches on the 10 best points evaluated in previous BO iterations, on the 10 best performing points from set of 10,000 random samples and on the 10 best performing points from 10,000 random samples drawn from the prior. To compute the neighbors of each of these 30 total points, we normalize the range of each objective to [o, 1 and randomly sample four neighbors from truncated Gaussian centered at the original value and with standard deviation a = 0.2.
We use four synthetic benchmarks in our experiments.
Branin. The Branin function is a dell-known synthetic benchmark for optimization problems (Dixon 1978). The Branin function has two input dimensions and three global minima.
benchmark in 2D based on Profet (Klein et al. 2019). This benchmark is generated by generative meta-model built using set of SVM classificttion models trained on 16 OpenML tasks. The benchmark has two input parameters, corresponding to SVM hyperparameters.
SVM. is a hyperparameter-oppiniizze
FC-Net. is a hyperparameter and architecture optimization benchmark in 6D based on Profet. The FC-Net benchmark is generated by generative meta-model built using set of feed-forward neural networks trained on the same 16 OpenML tasks as the SVM benchmark. The benchmark has six input parameters corresponding to network hyperparameters.
hhags//gitthh.ccm//PPPSSSSaam
16

---

Under review as a conference paper at ICLR 2021
Table 1: Search spaces for our synthetic benchmarks. For the Profet benchmarks, we report the original ranges and whether not log scale was used.
Benchmark Branin
Parameter name
Parameter values
Log scale
5, 10
22
05]
SVM
e 10 e
(e 20
Y
FCNet
learning rate batch size
10-" 10
88 128]
units layer
[16, 112
units layer 2
16, [12]
dropout rate 11
0.9.
dropout rate 12
0.9,
XGBoost
learning rate gamma
10-0, 10
[0,2]
LI regularization L2 regularization number of estimators
[10 10
10, 500]
subsampling maximum depth
0.1,1
1,15
minimum child weight
[0,20]
XGBoost. is txperparameetrroptiiizziii benchmark 8D based on Profet. The XGBoost bench- mark is generated by generative meta-model built using set of XGBoost regression models in 11 UCI datasets. The benchmark has eight input parameters, corresponding to XGBoost hyperparame- ters.
The search spaces for each benchmark are summarized Table For the Profet benchmarks, we report the original ranges and whether or log scale used. However, in practice, Profet's generative model transforms the range of all hyperparameters to a linear [0, 1] range. We use Emukit's public implementation for these benchmarks (Paleyes al. 2019).
E SVM REGRET COMPARISON
In addition to the experiments in Section we show the performance of PrBO on the SVM benchmark. Figure | shows log regret comparison PrBO, Spearmint, Prior Sampling and 10,000x RS. We note that the results are similar to the other benchmarks in Figure Namely, PrBO with strong prior outperforms RS and spearmint. PrBO also outperforms Spearmint with a weak prior and even with uniform prior.
F SPATIAL REAL-WORLD APPLICATION
Spatial (Koeplinger et al., 2018) is a programming language and corresponding compiler for the design of application accelerators reconfigurable architectures. e.g. peelamargrrm gate arrays (FPGAs). These reconfigurable architectures are a type of logic chip that can be reconfigured via software to implement different applications. Spatial provides users with high-level of abstraction for hardware design, so that they can easily design their own applications on FPGAs. It allows users to specify parameters that do not change the behavior of the application, but impact the runtime and resource-usage (e.g. logic units) of the final design. During compilation, the Spatia compiler estimates the ranges of these parameters and estimates the resource-usage and runtime of the application for different parameter values. These parameters can then be optimized during compilation order to achieve the design with the fastest runtime. We fully integrate PrBO as pass in Spatial's compiler, that Spatial Pal can automatically compilation. This enables Spatial to seamlessly call PrBO during the compilation of any new application to guide the search towards the best design application-specifie basis.
PrBO for the optimization during
17

---

Under review as a conference paper at ICLR 2021
RS (10,000x)
Prior Sampling
PrBO Weak Prior
Initialliatiinn
Spearmint
PrBO No Prior
PYBO Prior
Log Simple Regret
SVM
-10
-20
100
200
Number of Evaluations
Log regret comparison of PrBO with and without priors, 10, 000 RS, and Spearmint (mean +/- one std on repetitions). We run the benchmark for 200 iterations.
Figure 9:
Table 2:
Search space, priors, and expert configuration for the MD Grid application. The default value for each parameter is shown in bold.
Parameter name
Type
Values
Expert
Prior
loop_grid0__
Ordinal
[1. 2, 15, 16]
[0.2, 0.1, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05]
loop_q
Ordinal
[1. .... 32]
[0.08, 0.08, 0.02, 0.I, 0.02, 0.02, 0.02,
0.1, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02,
0.02, 0.1, 0.02, 0.02, 0.02, 0.02, 0.02,
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02,
0.02, 0.02, 0.02, 0.02]
par load
Ordinal Ordinal
[1 4]
[0.45, 0.1, 0.45]
loop_p
[1, 31, 32]
2
[0.1, O.1, 0. 0.05, 0,03, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02,
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02,
0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02,
0.02, 0.02, 0.02]
sop__grid0___
Ordinal
[1, .... 15, 16]
[0.2, 0.1, 0.05, 0.05, 0.05, 0.05, 0.05,
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05]
loop_grid1
Ordinal
[1, 15, 16]
[0.2, 0.2, 0.1, 0.1, 0.07, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03, 0.03]
loop_grid0__
Ordinal
[1, 15, 16]
[0.2, 0.1, 0.05, 0.05, 0.05, 0.05, 0.05.
0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05,
0.05, 0.05]
ATOMILOOP
Categorical
false, true]
true
[0.1, 0.9]
ATOM2L0OP
Categorical
False, true]
true
[0.1, 0.9]
PLOOP
Categorical
[false, true]
[0.1, 0.9]
In our experiments, we introduce for the first time the automatic optimization of three Spat i a real-world upplications, namely, 7D shallow and deep CNNs, and 10D molecular dynamics grid application. Previous work by Nardi et al.] (2019) had applied automatic optimization of Spat ial parameters on set of benchmarks but our work we focus on real-worke applications raising the bar of ttat-oof-the-ar automated hardware design optimization. PrBO is used to optimize the parameters to find a design that leads to the fastest runtime. The search space for these three applications is based ordinal and categorical parameters; to handle these discrete parameters in the best way we implement and use Random Forest surrogate instead Gaussian Process one as explained in Appendix [ These parameters application specific and control how much of the FPGAs' resources we want to use to parallelize each step of the application's 's computation. The goal here is to find which steps more important parallelize in the final design, in order achieve the fastest
18

---

Under review as a conference paper at ICLR 2021
Table Search space, priors, and expert configuration for the Shallow CNN application. The default value for each parameter is shown in bold.
Parameter name
Type
Values
Expert
Prior
LP
Ordinal
[. 4, 8, 16, 32]
[0.4, 0.063, 0.07, 0.065, 0.4]
Pl
Ordinal
[1, 2, 3, 4]
[0.1, 0.3, 0.3, 0.3]
SP
Ordinal
[1, 4, 8, 16, 32]
16
[0.4, 0.065, 0.07, 0.065, 0.4]
P2
Ordinal
[1, 2, 4]
[0.1, 0.3, 0.3, 0.3]
P3
Ordinal
31, 32]
[0.1, 0.033, 0.1, 0.021,
0,021, 0.021, 0.1, 0.021, 0.021,
0,021, 0,021, 0.021,
0.021, 0.021, 0.021, 0.021,
0.021, 0.021, 0.021, 0.021,
0,021, 0.021, 0,021, 0.021,
0.021, 0.021, 0.021, 0.021,
0.021, 0.021]
P4
Ordinal
[1, 47, 48]
[0.08, 0.0809, 0,0137, 0.1,
0.0137, 0.0137, 0.0137, O.L,
0.0137, 0.0137, 0.0137, 0.05,
0.0137, 0.0137, 0.0137, 0.0137,
0.0137, 0.0137, 0.0137, 0.0137,
0.0137, 0.0137, 0.0137, 0.0137,
0.0137, 0,0137, 0,0137, 0.0137,
0.0137, 0.0137, 0.0137, 0.0137,
0.0137, 0.0137, 0.0137, 0.0137,
0.0137, 0.0137, 0.0137, 0.0137.
0.0137, 0.0137, 0.0137, 0.0137,
0.0137, 0.0137, 0.0137.
0.0137]
x276
Categorical [false, true]
true
[0.1, 0.9]
Table 4: Search space, priors, and expert configuration for the Deep CNN application. The default value for each parameter is shown in bold.
Parameter name
Type
Values
Expert
Prior
LP
Ordinal
[1. 4, 8, 16, 32]
[0.4, 0.065, 0.07, 0.065, 0.4]
P1
Ordinal
[1, 2, 4]
[0.4, 0.3, 0.2, 0.1]
SP
Ordinal
11,4,8, 16, 32]
8
[0.4, 0.065, 0.07, 0.065, 0.4]
P2
Ordinal
[1,2,3,4]
[0.4, 0.3, 0.2, 0.1]
P3
Ordinal
[1,2, 31, 32
[0.04, 0.01, 0.1,0,00,0 0.01, 0.01,
0.01,
0.01, 0.01,
0.01, 0.01, 0.01, 0.01, 0.2, 0.01,
0.01, 0.01, 0.01.
0.11, 0.01, 0.01, 0.01,, 0.01, 0.01,
0.01, 0.01, 0.01, 0.2]
P4
Ordinal
[1, 2, 47, 48]
4
[0.05, 0.005, 0.005, 0.005,
0.005, 0.005, 0.005, 0.13.
0.005, 0.005, 0.005, 0.005,
0.005, 0.005, 0.005, 0.2, 0.005,
0.005, 0.005, 0.005, 0.005,
0.005, 0.005, 0. 0.005,
0.005, 0.005, 0.005, 0.005.
0.005, 0.005, 0.2, 0.005, 0.005.
0.005, 0.005, 0.005, 0.005,
0.005, 0.005, 0.005, 0.005,
0.005, 0.005, 0.005, 0.005.
0.005, 0.1]
x276
Categorical
false, true
true
[0.1, 0.9]
19

---

Under review as a conference paper at ICLR 2021
PrBO Weak Multivariate Prior
PrBO Weak Univariate Prior
Initialization
PrBO Multivariate Prior
PrBO Univariate Prior
Branin
SVM
10
Log Simple Regret
Log Simple Regret
-10
-15
-10
-20
-15
0
50
100
150
200
50
100
150
200
Number of Evaluations
Number of Evalluations
FC-Net
XGBoost
Log Simple Regret
Log Simple Regret
-3
0
-1:
-2
100
200
300
100
200
300
Number Evaluations
Number of Evaluations
Figure 10: Log regret comparison of PrBO with multivariate and univariate KDE priors. The line and shaded regions show the mean and standard deviation of the log simple regret after runs. All methods were initialized with D 1 random samples, where D is the number of input dimensions, indicated by the vertical dashed line. We run the benchmarks for 100l) iterations, capped at 300.
runtime. Some parameters also control whether we want to enable pipeline scheduling or not, which consumes resources but accelerates runtime. We refer to Koeplinger et al. (2018) and Nardi et al. (2019) for more details on Spatial's 's parameters.
The three Spat ial benchmarks also have fcasibility constraints in the search space, meaning that some parameter configurations are infeasible. configuration is considered infeasible if the final design requires more logic resources than what the FPGA provides, i.e., it is not possible to perform FPGA synthesis because the design does not fit in the FPGA. To handle these constraints, we use our cBO implementation (Appendix D). Our goal is thus to find the design with the fastest runtime under the constraint that the design fits the FPGA resource budget.
The priors for these Spatia iiliiall applications take the form of list of probabilities, containing the probability of each ordinal or categorical value being good. Each benchmark also has default configuration, which ensures all methods start with least one feasible configuration. The priors and the default configuration for these benchmarks were provided once by an unbiased Spat ippatial developer, who not an author of this paper, and kept unchanged during the entire project. The search space, priors, and the expert configuration used in our experiments for each application are presented in Tables and
G MULTIVARIATE PRIOR COMPARISON
Figure T0 shows log regret comparison of PrBO with univariate and multivariate KDE priors. We show results for univariate and multivariate versions of our weak and strong KDE priors. We use the best 10/ points out of 1,000D and 10,000,000D randomly sampled points to create our weak and strong priors, respectively. We use the same points to create the univariate and multivariate priors. We recall that the goal of these synthetic priors is have unbiased prior for our experiments, whereas manual priors would be biased by our own expertise of these benchmarks. In practice, users will manually define these priors without needing additional experiments.
20

---

Under review as a conference paper at ICLR 2021
PrBO Wrong Prior
PrBO Weak Prior
Initialization
PrBO No Prior
PrBO Prior
Branin
SVM
10
Log Simple Regret
Log Simple Regret
-5
-10
-15
-10
-20
-15
50
100
150
200
50 Number Evalluations
100
150
200
Number of Evaluations
FC-Net
XGBoost
8
Log Simple Regret
Log Simple Regret
6.
4
-3
0
-4
-2
-5
Number of Evaluations 300
Number of Evaluations
Figure 11: Log regret comparison of PrBO with varying prior quality. The line and shaded regions show the mean and standard deviation of the log simple regret after 5 runs. All methods were initialized with D + random samples, where D the number of input dimensions, indicated by the vertical dashed line. We run the benchmarks for 100D iterations, capped at 300.
note that all cases PrBO achieves similar performance with univariate and multivariate priors. For the Branin and SVM benchmarks, the weak multivariate prior leads to slightly better results than the weak univariate prior. However, we note that the difference is small, in the order of 10 and 10 respectively.
Surprisingly, for the XGBoost benchmark, the univariate version for both the weak and strong priors lead to better results than their respective multivariate counterparts, though, once again, the difference in performance small, around 0.2 and 0.03 for the weak and strong prior, respectively, whereas the XGBoost benchmark reach values high f(x=) 600. Our hypothesis that this difference comes from the bandwidth estimator (100n +), which leads to larger bandwidths, consequently, smoother priors, when multivariate prior is constructed.
H MISLEADING PRIOR COMPARISON
Figure 11 shows the effect of injecting a misleading prior in PrBO. We compare PrBO with misleading prior, no prior, weak prior, and strong prior. To create the misleading prior, we use univariate KDE prior built with the worst 10D out of 10,000,000D random samples. For all benchmarks, we note that the misleading prior slows down convergence, as expected, since it pushes the optimization away from the optima the initial phase. However, PrBO is still able to forget the misleading prior and achieve similar regret to PrBO without a prior.
1 COMPARISON TO OTHER BASELINES
We compare PrBO to SMAC HHtttrr et al. 201] and TPE (Bergstra et al. 2011) on our four synthetic benchmarks. We use Hyperopt's 's implementation of TPE and the SMAC3 Python implementation
21

---

Under review as a conference paper at ICLR 2021
TPE
PrBO
SMAC
Initialization
Branin
SVM
Log Simple Regret
Log Simple Regret
-5
-5
-10
-10
-15
100
200
100
200
Number of Evaluations
Number of Evaluations
FC-Net
XGBoost
Log Simple Regret
Log Simple Regret
-2
-4
100 200
300
200 300
Number of Evaluations
Number of Evaluations
Figure 12: Log regret comparison of PrBO, SMAC, and TPE. The line and shaded regions show the mean and standard deviation of the log simple regret after 5 runs. PrBO was initialized with D + 1 random samples, where D is the number of input dimensions, indicated by the vertical dashed line. We run the benchmarks for 100D iterations, capped at 300.
of $ Hyperopt defines priors one of list of supported distributions, including Uniform, Normal, and Lognormal distributions. Since it is not possible to input the KDE priors introduced in Section 4. into the TPE algorithm, instead by the HyperOpt implementation. We note that this is straightforward in PrBO, as PrBO supports any form of probability distribution as prior. We are then able perform a fair comparison between the two approaches that use the same exact prior. Since SMAC does not support probability distribution priors, do not inject any priors in SMAC.
manually defined priors in the format supported
We define the prior for each input parameter as a Gaussian distribution with mean at the optimum and with standard deviation equal to half of the parameters range. For the Branin prior. we arbitrarily choose one of the optima, i.e., the (Tr, 2.275) optimum. For the Profet benchmarks, we use the minimum out of 10,000,000/) random samples as an approximation of the optimum. We note that using Hyperopt's Gaussian priors leads to unbounded search space, which sometimes leads TPE to suggest parameter configurations outside the allowed parameter range. To prevent these values from being evaluated, convert values outside the parameter range to be equal to the upper lower range limit, depending on which limit was exceeded.
Figure 12 shows a log regret comparison between PrBO, SMAC, and TPE our four synthetic benchmarks. PrBO outperforms SMAC on three out four benchmarks, and shows consistent speedups in the early phases of the optimization proces. PrBO also outperforms TPE in three out of four benchmarks, namely, Branin, SVM, and FCNet. We note, however, that the good performance of TPE on XGBoost may be an artifact of the approach of clipping values its maximal minimal values as mentioned above. In fact, the clipping nudges TPE towards promising configurations in this case, since XGBoost has low function value near the edges of the search space. Overall, the better performance of PrBO is expected, since PrBO is able to combine prior knowledge with more sample-efficient surrogates, which leads to better performance.
hhapss//githuuhco/aaaao////////
22

---

Under review as a conference paper at ICLR 2021
PrBO
Spearmint Prior
Spearmint
Initialization
Branin
SVM
Log Simple Regret
10
Log Simple Regret
-5
-10
-15
-10
-20
100
200
100
200
Number of Evaluations
Number of Evaluations
FC-Net
XGBoost
Log Simple Regret
Log Simple Regret
-2
0
100 200
300
100 200
300
Number of Evaluations
Number of Evaluations
Figure 13: Log regret comparison of PrBO, Spearmint with prior initialization, and Spearmint with default initialization. The line and shaded regions show the mean and standard deviation of the log simple regret after 5 runs. PrBO and Spearmint Prior were initialized with D + - random samples from the prior, where D is the number of input dimensions, indicated by the vertical dashed line. We run the benchmarks for 100D iterations, capped at 300.
HyperMapper
PrBO
HyperMapper Prior
Initialization
Shallow CNN
Deep CNN
MD Grid
24M
24M
9M
Cycles (169)
15M
3M
655M
Cycles [log]
9M
(log)
1M
11M
400k
9M
3M
7M
2M
200k
50k
20
40
20
40
20k
50
100
150
Number of Evaluations
Number Evaluations
Number Evaluations
Figure 14: Log regret comparison of PrBO, HyperMapper with prior initialization, and Maperr with default initialization. The line and shaded regions show the mean and standard deviation of the log simple regret after 5 runs. PrBO and HyperMapper Prior were initialized with D + random samples from the prior, where D is the number of input dimensions, indicated by the vertical dashed line.
J PRIOR BASELINES COMPARISON
In this section, compare PrBO with Spearmint and HyperMapper using the same prior initialization on all of our benchmarks (Figs. 13 and 14). Both PrBO and the baselines were initialized with the same D + samples from the prior, indicated by the vertical dashed line. We run each optimizer for 5 times and report mean and standard deviation of the runs. For comparison, we also show the baselines with the default initialization.
Figure 13 shows the comparison between PrBO and Spearmint Prior. In most benchmarks, the prior initialization does not lead to improved final performance. In particular, for XGBoost, the prior leads
23

---

Under review as a conference paper at ICLR 2021
to improvement in early iterations, but to worse final performance. We note that for SVM, the prior leads to better performance, however, we note that the improved performance is given solely from sampling form the prior. There is no improvement for Spearmint Prior after initialization. In contrast, in all cases, PrBO is able to leverage the prior both during initialization and its Bayesian Optimization phase, leading improved performance. PrBO still outperforms Spearmint Prior in the same 3 out of 4 benchmarks.
Figure 14 shows similar results for our Spatial benchmarks. Once again, the prior does not lead HyperMapper to improved final performance. For the Shallow CNN benchmark, the prior leads HyperMapper improved performance in early iterations, compared to HyperMapper with default initialization, but HyperMapper Prior is still outperformed by PrBO. Additionaly, the prior leads to degraded performance in the Deep CNN benchmark. These results confirm that PrBO is able to leverage the prior in its seeudo-posterion during optimization, leading to improved performance in almost all benchmarks compared to state-of-the-an BO baselines.
K PRIOR BANDWIDTH SELECTION
We show the effect of different bandwidth sizes on the univariate KDE prior. For that, we compare the performance of sampling from the prior and PrBO with different bandwidth sizes. We use scipy's s Gaussian KDE implementation and modify its bandwidth size with four variations of Scott's Rule
an -t. We experiment with a 1, b = 4 (scipy's 's default); a = 1, b = O; 10, b = 0: and a = 100, b = 0. Note that larger values for a and smaller values for b lead to smaller bandwidths. For each bandwidth size, we show results for an array of varying quality priors. We select constant 10D points in each prior and vary the size of the uniform random sample dataset. We follow the following rule: we use the best performing 10D samples to create the prior from a random sample dataset size of 10D 100, we refer to this prior as x%. We experiment with dataset sizes varying from 10D to 10?) D.
Figure 15 shows the performance of purely sampling from the prior. We note that, in most cases, using a larger dataset leads to better results. This is expected, sampling more points means we find more points near the optima and, therefore, the prior will be built with points closer to the optima. Likewise, we note that smaller bandwidths often lead to better results, especially as more points are sampled. This is also expected, since smaller bandwidth means the prior distribution will be more peaked around the optima. However, there are a couple of exceptions to these trends. First, for the Branin, sampling more points does not lead a better prior when we use a = 1, 4, this is likely because the multiple minima of the Branin and the bigger bandwidth lead the prior to be oversmoothed, missing the peaks near the optima. Second, smaller bandwidths do not always lead to better performance for smaller random sample datasets. This happens because farther from the optima in these datasets and end up computing priors peaked at points that are farther from the optima, i.e., our priors become misleading. The effects of these misleading priors can be especially noticed for the 100% random samples dataset. Based on these results, we set our KDE priors to 100n b, where D is the number of input dimensions.
find points
Figure [ shows the performance of PrBO for different priors. The same observations from Figure 15 hold here. Namely, sampling more points and using smaller bandwidths lead to better performance. Also, the 100% dataset once again leads to inconsistent results, since is misleading prior for PrBO. Based on these results, we use the smallest bandwidth and largest dataset in our experiments, i.c. a ==00 = and 0.0001% %. Intuitively, this is a reasonable choice, since these priors will be our closest approximation to an ideal prior that is centered exactly at the optima, where sampling from the prior always leads to the optimum. Our results in Figures 15 and T shlss that this combination leads to the best results in all benchmarks, as expected.
Figure 17 shows a performance comparison between PrBO and sampling from the prior. For these results, we use = 100, b- 0 and compare the regret of PrBO and sampling from the prior for different dataset sizes. PrBO performs better for nearly all dataset sizes and benchmarks. This is expected as PrBO complements the prior with its probabilistic model, learning which regions within the prior are better to explore and also recovering from misleading priors. There are two exceptions on the SVM benchmark, where sampling from the prior performs slightly better for 0.01% % and
24

---

Under review as a conference paper at ICLR 2021
Branin
Branin
Simple Regret
Simple Regrett
Simple Regrot
Simple =
2
10t
0.%%
522
3.0 Oalm
0.1%
1.0
1,00
10%
oge oue s.e s.e aonoes
0.01% tn
SVM
0.10
SVM
S4M
0.50
SVM
Simple Rearet
siroo Reggro
Stroo Reope
0.00
0.00
3.04
0.04
0.04
simols
0.04
2,22
0.02
0.02
9.02
coe 159 aap S400
10.000
000
Gats
10%
eats
FCNet
0.10
FCNet
FCNet
0.10
FCNet
Sinele Regret
5.00
0.08
0.08
0.08
0.02
0.02
coe
3.00
0.00
100%s
0.1%
0.00
0.1%
0.00
100%
1on
0.1%
Xooot
XCBoodt
Simple Regret.
Sirre Regir
Sirple Regeet
Simple Regret
Gabsts amp La
100%
10e
0.1%
eats
0.00
10m
dala
0,00
100%
10%
0.1%
0.001
6.001
(a)
1.b 4
(b) 1,b=0
(c)
10, 0
(d)
100, b == 0
Figure 15: Simple regret of sampling from the prior with different priors for our synthetic benchmarks. We provide 5 repetitions for each experiment and mean +/- one std eтror bars. A more informative prior gives better results all benchmarks.
0.0001% datasets. We note, however, that the difference in performance extremely small, in the order of 10-5 and 10-7, respectively.
L 7-SENSITIVITY STUDY
We show the effect of the hyperparameter introduced in Section 3.2 for the quantile identifying the points considered to be good. To show this, we compare the performance PrBO with a weak KDE prior and different 7 values. We use our weak prior as it leads greater variation in performance, which helps to visualize better the impact the y hyperparameter. For all experiments, we initialize PrBO with D + random samples and then run PrBO until it reaches 10/) function evaluations. For each y value, we run PrBO five times and report mean and standard deviation.
Figure 18 shows the results of our comparison. We first note that values near the lower and higher extremes lead to degraded performance, this is expected, since these values will lead to an excess of exploration. Further, note that PrBO achieves similar performance for all values of y, however, ? 0.03 and 7 = 0.05 consistently lead to better performance, with = 0.05 usually leading to lower deviation.
either exploitation
25

---

Under review as a conference paper at ICLR 2021
Branin
Branin
Simple Regret
Simple Regrett
Simple Regrot
Simple Regroit
=
322 sas
ese 20 0001
ase 3.e 3.0 1.d0 0.00
ae 2.0 1.00 1,50
10t
0.%%
0.1%
10%
0.01%
SVM
0.10
SVM
S4M
0.10
SVM
Simple Rearet
simpo Reegro
Simooe Segraa
simple Regt
0.00
0.00
2.04
0.04
0.04
0.04
2.22
0.02
0.02
0.02
coe 119 saptec S4c0
10.000
0.0
eats
10%
eats
0.001
FCN
0.10
FCNet
FCNet
0.10
FCNet
Sinele Regret
5.8-
0.08
Regiid
0.08
0.08
Simees
m oat
0.02
0.02
2.00
sre
0.00
0.4%
0.00
0.1%
0.00
100%
0.1%
13
Booct
Xooot
XCBoost
Sirple Regret.
Simple Regret
Simple Regret.
Simple Regret
oat 15t 0.4
sat 021 sse
100%
10w
0.1%
0.00 eats
10m
dala
0,01%
100%
10%
0.1%
a.ts..
0.00
0.0001
(a) 1.b 4
(b) 1,b=0 0
(c) a
10,b =0 0
(d) 100, b = 0
Figure 16: Simple regret of PrBO with different priors for our synthetic benchmarks. We provide 5 repetitions for each experiment and mean +/- one std error bars, A more informative prior gives better results in all benchmarks.
Prior Sampling Pri
100%
1%
Intialization
0.01%
0.0001%
Branin
SVM
FC-Net
XGBoost
Log Simple Ruguet
Las Simple Ragget
5.0
10
-15
-25
Number of Evalust mm
Namber Evalustion
Number of Evallsstions
Number Evaluatione
Figure 17: Log simple regret comparison between PrBO and sampling from the prior. The shaded lines are mean +/- one std error. PrBO was initialized with D + 1 random samples, indicated by the vertical dashed line.
M 3-SENSITIVITY STUDY
We show the effect of the hyperparameter introduced in Section 3.3 for controlling the influence of the prior over time. To show the effects of 3, we compare the performance of PrBO with a weak KDE prior and different 3 values our four synthetic benchmarks. We weak prior it leads to
26

---

Under review as a conference paper at ICLR 2021
Branin
FCNet
XGBoost
Simple Recgeet
Simple Regrot
Simple Regret
420
Sampl Weggee
22
800
Figure 18: Comparison of PrBO with a weak KDE prior and different values for the y hyperparameter on four synthetic benchmarks. We rum PrBO with budget of 10D function evaluations, including D + I randomly sampled DoE configurations.
FCNet
XGBloo
Simple Regnet
Smmple Rogret
Simple Rogres
2u000 -0.50
8
457
12.00
Tong
100p
Comparison of PrBO with weak KDE prior and different values for the 9 hyperparameter on our four synthetic benchmarks. We run PrBO with a budget of 10D function evaluations, including D + 1 randomly sampled DoE configurations.
Figure 19:
greater variation in performance, which helps to visualize better the impact of the hyperparameter. For experiments, we initialize PrBO with + random samples and then run PrBO until reaches 10D function evaluations. For each value, run PrBO five times and report mean and standard deviation.
Figure 99 shows the results of our comparison. We note that values of / that are too low (near 0.01)
or too high (near 1000) lead to lower performance. This shows that putting too much emphasis on the model the prior will lead degraded performance, expected. Further, lead to the best performance in three out our four benchmarks. This result is reasonable, as 3 = 10
note that 3 10
means PrBO will put more emphasis on the prior in early iterations, when the predictive model is still not accurate, and slowly shift towards putting more emphasis on the model as the model sees more data and becomes more accurate.
27


Reviews:
[{"review_id": "p3u6RCPjNMn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel method to incorporate experts' knowledge into BO. This is done through introducing  Prior-guided Bayesian Optimization (PrBO). Different experiments where conducted to compare PrBO vs different baselines and to show the effect of the user provided priors in the cases where it is well-specified or mis-specified. The design of PrBO enables it to guide the search in the early iterations and as optimization progresses, more emphasis is given to the model and the effect of the prior is washed out.\n\nStrong points:\n- The paper presents a method for incorporating priors over good solutions in an elegant way.\n- The method can benefit from the prior information and incase of misspecified priors, it can still recover.\n- Extensive experiments were conducted.\n- The paper is well written.\n- This type of work is definitely needed to increase the adoption of BO methods.\n\nWeak points:\n- Although the experimental design is extensive and covers several aspects. I wish the experiments included one realistic experiment for tuning the hyperparameters of a famous architecture. This would have been a great example to demonstrate the benefits of this method versus the manual search used by most researchers. However, this is not a huge concern.\n\nClearly state your recommendation (accept or reject) with one or two key reasons for this choice.\n- I recommend to accept this paper.\n- Choice of priors affect the performance of BO. This paper provide the intuitive way to add priors by modelling probability of good configuration. This is what the user thinks. I am sure that this kind of work will encourage more researchers to use BO in their problems\n\nQuestions:\n- It is clear how PrBO is incorporated in TPE, but what about GPs and RF? How is PrBO implemented in this case?\n- Are the experiments in the main paper using GPs or TPE?\n- Could you please explain more what 10,000×random search is?\n- How are the weak and strong priors for PrBO generated? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "fOtLvsTMgN9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper incorporates a prior distribution given by experts into Bayesian optimization (BO), to leverage useful human knowledge to accelerate BO. The algorithm uses an intuitive approach to combine the prior with the probabilistic surrogate model of BO to derive a pseudo-posterior, which naturally leads the EI acquisition function. As the BO progresses, the prior information is gradually overwhelmed by the observed data, which ensures the asymptotically correct behaviour.\n\nStrong points:\n- The design of the algorithm is intuitive and natural.\n- Experiments are comprehensive. Figure 2 is a particularly nice illustration.\n- The paper is well written.\n\nWeak points/questions:\n- I'm not fully convinced whether it's really useful to ask an expert to specify a prior distribution and then run the proposed algorithm, compared with the baseline of simply asking the expert to specify the search space and then running standard BO. Firstly, asking the expert to specify a search space is intuitively easier for the expert, compared with asking them to give a distribution across the entire domain. Moreover, I think it's hard to justify why the proposed approach is better in practice than simply asking the expert for a reduced search space. I would like to see either a comparison with this baseline (preferably for the experiment in Section 4.4 if possible), or a justification regarding why this baseline isn't competitive.\n- The proposed algorithm is a straightforward combination of PI, TPE, and EI. Therefore, the technical contribution may not be novel enough.\n- Figure 4: It seems that Spearmint consistently outperforms PrBO with weak prior. Does this mean that if you don't have a good prior, you should simply run standard BO?\n- There is only one real-world experiment (Section 4.4) where an expert is asked to specify a prior, while for the other experiments, the priors are artificially constructed. I feel more real-world experiments are needed, given that I'm doubtful as to whether the proposed algorithm is truly useful in practice as mentioned in the first point above.\n- Why haven't you compared with the related work of Li et al. (2020)? It seems very similar to the proposed approach and is thus applicable in similar settings.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZJTr4NkVKNn", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method, PrBO, and it incorporates user knowledge on the location of potential optimum to the prior used for Bayesian Optimization. The authors claim that BO can be solved more efficiently using their method and showed promising empirical results on some low-dimensional benchmarks.\n\nDisclaimer: I reviewed the same paper at Neurips. While I was happy to see that the authors have resolved some of term misusage issues and wordings, I felt somewhat disappointed that many good points raised by the reviewers there before were not addressed in the new version of the paper. IMHO, it still remains to be true that this paper needs a bigger surgery to be accepted, not only for Neurips or ICLR, but also other ML venues. And I hope the authors will consider the reviewers' opinions before submitting somewhere else.\n\nPros:\n- Writing is mostly clear. Good clarity and good illustrations.\n- I still think this is a very interesting idea and a quite promising direction to incorporate human knowledge in our used-to-be black-box machineries. This paper could set a good direction if it's done properly.\n\nCons:\n- This wasn't the case of the previous version, but I found it unsettling that the 1st contribution point was \"For the first time, user prior knowledge can be combined with standard BO probabilistic models, such as Gaussian Processes (GPs), Random Forests (RFs), and Bayesian Neural Networks.\" That is just bold and wrong. User prior knowledge is always considered in those models when selecting which kernel structure to use, how many trees would be sufficient, or if we need a convolution layer or not. Even in the narrowest definition of prior, this paper is a perfect demonstration of how to incorporate human knowledge in GPs: https://papers.nips.cc/paper/2015/file/4462bf0ddbe0d0da40e1e828ebebeb11-Paper.pdf. I strongly suggest not to ever claim one's the first, and when it's really needed, be very very specific. Otherwise there will be bad consequences. (See an extreme case related to Christopher Columbus.)\n- It is still not clear what different probabilities mean to the authors. The authors described the prior P_g(x) as a prior on good points and P_b(x) as prior on bad points. What do those mean exactly? Could you define them correctly in the paper? And how does the GP-induced good/bad point probability interfere with the proposed ones?\n- Footnote on Page 3: \"P b(x) is not a probability distribution\". What do you mean? In the text it says \"prior distributionPg(x)\". \n- P_g is not independent for each x. Will that cause any issues to the method?\n- If both the \"prior\" and \"posterior\" are both pseudo, what do they mean exactly? Or what are even approximating?\n- Can the authors describe a coherent generative model of what they proposed?\n\nI strongly suggest the authors carefully read the reviews from Neurips again. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OJtpAQiCkw-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of this paper is to enable the introduction of prior expert knowledge in Bayesian optimization. This is performed by defining a prior distribution for the optimal value, which is included in the pseudo-posterior used to select new points by Expected Improvement. The number of iterations it takes to overcome a potentially wrong prior information is controlled by a parameter, whose sensitivity is studied. Extensive experiments are conducted on toy examples as well are more realistic hyper-parameter test cases.\n\nThis paper could interest practitioners, but some aspects need to be better detailed and more comparison to existing methods should be added.\n\nDetailed comments:\n\nThe proposed model in (3) amount to scale the probability of improvement with an arbitrary threshold by a probability of optimality given by the prior. This is very similar to the use of the probability of feasibility in constrained BO, see, e.g., Schonlau, M.; Welch, W. J. & Jones, D. R. Global versus local search in constrained optimization of computer models. Lecture Notes-Monograph Series, JSTOR, 1998, 11-25 and references therein. Hence couldn’t this so called pseudo-posterior be used directly?\n\nThere are additional references tackling the introduction of a prior (or lack of) to compare with, both in the state of the art and experiments, e.g.,: \n- Anil Ramachandran, Sunil Gupta, Santu Rana, Cheng Li, Svetha Venkatesh, Incorporating expert prior in Bayesian optimisation via space warping, Knowledge-Based Systems, Volume 195, 2020.\n- Bobak Shahriari, Alexandre Bouchard-Cote, Nando Freitas ; Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, PMLR 51:1168-1176, 2016.\n\nAnother naive competitor is the use of the prior on the optimum only at the design of experiments stage, keeping the usual BO loop. This baseline should be added too.\n\nMinor details:\n\nFigure 1: consider showing the density model given by (4) here too, not just in appendix\n\nExperiments: what is exactly the weak prior setting?\n\nP8: on Oh et al., this is a structural prior to cope with high-dimensional optimization, so not clearly related.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-iFwZAu8Psw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n# summary\n\nThis paper provides an interesting setting and tries to incorporate prior\nknowledge of inputs in BO. For example, in a specific application, an expert\nknows the potential good region for some parameters in this problem. More\nconcretely, to incorporate expert knowledge in BO, the authors first specify a\n\"good\" prior, which gives a large mass on more interesting regions. A \"bad\"\nprior can also be specified, and a common choice is 1 - goodprior. These two\npriors are built from expert knowledge.\n\n\n# pros\n\n1.  This paper provides an interesting direction in BO, i.e. incorporating\n    prior knowledge. Moreover, this direction is very realistic, as in many\n    applications, we do know some parameters are likely to take some specific\n    values or fall into some regions.\n2.  This paper also provides comprehensive experiments and verifies the\n    advantages of incorporating prior knowledge in BO.\n\n\n# cons\n\n1.  My major concern is the construction of the \"good\" prior and the \"bad\"\n    prior. If my understanding is correct, the \"good\" prior in all experiments\n    are fitted using KDE. Based on the independence assumption, different\n    parameters are fitted independently. The input data to KDE is {x_i}. To obtain a good fit, a large number of\n    function evaluations is needed (many objective values are needed to get \"good\" x_i) and this violates the principles of BO in\n    the first place, i.e. the objective function in BO is expensive and we\n    cannot afford a large number of evaluations. In summary, to construct a\n    reasonable \"good\" prior, we need a large number of function evaluations but\n    we cannot afford it. If the \"good\" prior is not good, we then need a large\n    number of functional evaluations to wash out such a prior in the following\n    optimizing stage, violating the main principle of BO again.\n2.  In Figure 2, the authors use a misspecified prior to show their proposed\n    method is able to recover from a \"wrong\" prior. Indeed, this property is\n    desirable. My question is if the prior might be wrong, why not use the\n    conventional BO?\n3.  Why not directly set the parameter bound to be the potentially interesting\n    region in optimizing the acquisition? I understand directly setting the\n    parameter bound in optimizing the acquisition function could rule out the\n    optimal region if our prior knowledge is misspecified. If our prior\n    knowledge is \"correct\", I guess this approach could have a good\n    performance. Can the authors comment on this?\n4.  Comparison with SMAC is lacking and I think this is a must. SMAC performs\n    better than TPE in many problems.\n\nMinor issues:\n1. The claim: 10000x faster than random search, sounds like this is an ad instead of an academic paper.\n\nOverall speaking, incorporating prior in BO is an interesting direction,\nhowever, I am afraid this work do not have sufficient theoretical analysis and\nthere are some issues stated above on the prior in this work.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was quite contentious.  While there is clearly promise in the method and the idea, and reviewers appreciate the importance of encoding non-trivial prior knowledge into BO, three reviewers express major concerns regarding the presentation (including worries about over-claiming contributions), the specification of the probabilistic model as well as to some extent about the experimental evaluation.  Lastly, the title appears to be a kind of pleonasm - arguably, the key point of using a Bayesian prior is to be able to encode prior knowledge.  The authors consider a different form of prior than perhaps usually meant in BO (in a generative rather than discriminative sense), but the terminology is still confusing.  
Decision: Reject