A General Framework for User-Guided Bayesian Optimization
Authors: Carl Hvarfner
Frank Hutter
Luigi Nardi
Venue: ICLR 2024 spotlight (2024)
Keywords: Bayesian Optimization
Hyperparameter Optimization
Gaussian Processes
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We develop a Bayesian-principled framework for the incorporation of user beliefs over the location of the optimizer into Bayesian optimization.

Abstract: The optimization of expensive-to-evaluate black-box functions is prevalent in various scientific disciplines. Bayesian optimization is an automatic, general and sample-efficient method to solve these problems with minimal knowledge of the the underlying function dynamics. However, the ability of Bayesian optimization to incorporate prior knowledge or beliefs about the function at hand in order to accelerate the optimization is limited, which reduces its appeal for knowledgeable practitioners with tight  budgets. To allow domain experts to customize the optimization routine, we propose ColaBO, the first Bayesian-principled framework for incorporating prior beliefs beyond the typical kernel structure, such as the likely location of the optimizer or the optimal value. The generality of ColaBO makes it applicable across different Monte Carlo acquisition functions and types of user beliefs. We empirically demonstrate ColaBO's ability to substantially accelerate optimization when the prior information is accurate, and to retain approximately default performance when it is misleading.

Published as a conference paper at ICLR 2024
A GENERAL FRAMEWORK FOR USER-GUIDED BAYESIAN OPTIMIZATION
Carl Hvarfner
Frank Hutter
Lund University
University of Freiburg
carl. whwarfnerccs..... se
fhêcs. uni -f Fre wweeerrrrruu.r..r bibbr.g.dd de
Luigi Nardi
DBtune, Lund University, Stanford University luigi, nardinardee. ith.
ABSTRACT
The optimization of expensive-to-evaluee black-box functions is prevalent in var- ious scientific disciplines. Bayesian optimization is automatic, general and sample-efficient method to solve these problems with minimal knowledge of the underlying function dynamics. However, the ability of Bayesian optimization to incorporate prior knowledge or beliefs about the function at hand in order to accelerate the optimization is limited, which reduces its appeal for knowledgeable practitioners with tight budgets. To allow domain experts to customize the opti- mization routine, we propose ColaBO, the first Bayesian- priniip--d framework for incorporating prior beliefs beyond the typical kernel structure, such as the likely location of the optimizer or the optimal value. The generality of Co 1 aBO makes it applicable across different Monte Carlo acquisition functions and types of user beliefs. We empirically demonstrate Co aBO's ability to substantially accelerate optimization when the prior information is accurate, and to retain approximately default performance when is misleading.
INTRODUCTION
Bayesian Optimization (BO) (Mockus et al. 1978; Jones al. 1998, Snoek et al.. 2012) is well-
established methodology for the optimization of expensive-to-evalaaae black-box functions. Known for its sample efficiency, BO has been successfully applied to a variety of domains where laborious system tuning is prominent, such hyperparameter optimization (Snoek et al. [012] Bergstra
cet al.] [2011b] Lindauer et al.) (202), neural architecture search (Ru et al. 2021] White ct al.] 2021).
roboties (Calandra al.) 20144) Mayr al. 2022), hardware design (Nardi et al. 2019 Ejjeh et al.
2022). and chemistry (Griffiths Hernández-LLoato 2020).
Typically employing Gaussian Process (Rasmussen & Williams 2006) (GP) surrogate model, BO allows the user to specify a prior over functions p(f)) via the Gaussian Process kernel, as well as an optional prior over its hyperparameters. Within the framework of the prior, the user can specify expected smoothness, output range and possible noise level of the function at hand, with the hopes of accelerating the optimization if accurate. However, the prior beliefs that can be specified within the framework of the kernel hyperparameters. do not span the full range of beliefs that practitioners may possess. For example, practitioners may know which parts of the input space tend work best (Oh et al. 2018 [Perrone et al. 2019 [Smith] [018]] Wang et al. 2019), know range or upper bound on the optimal output (Jeong & Kim 2021 Nguyen & Osborne 2020) such as a maximal achievable accuracy of 100%, or other properties of the objective, such preference relations between configurations (Huang et al.] 2022). The limited ability of practitioners to interact and valuable
collaborate with the BO machinery (Kumar al. 2022) thus runs the risk of failing domain expertise, or alienating knowledgeable practitioners altogether. While knowledge injection beyond what is natively supported by the GP kernel crucial further increase the efficiency of

---

Published as a conference paper at ICLR 2024
Moments of prior
Moments of prior
Moments prior
Price monunbs of pini
Price
Petre of
Uses Price
Prior momente
Moments of posterlor
Moments of posterior
Moments of posterior
Poststiar storennts ((l))
Posterist martaatt pt/D)
Twstersor moments of pet0, pl
Proteriar moreens pitp, p!
Posterarr tsomenta pill)
Good region uncer
Oveer maccimal valuen
Posterior tummnatt of pf10. 2
0.0
0.2
0.4
0.6
0.0
02
0.4
0.6
0.5
1.0
0.0
0.4
0.6
0.5 1.b
Figure 1: Three different ColaBO priors: (left) Prior over the optimum T4. and the induced changed in the GP for an optimum located the green region. (middle) Prior over optimal value, f 0.8. (right) Prior over preference relations f(x) ž ((22) for five preferences (green arrows, e.g. f(0.0) > f(0.1) > f(0.2).
Bayesian optimization, so far no current approach allows for the integration of arbitrary types of user knowledge. To address this gap, we propose intuitive framework that effectively allows the to reshape the Gaussian process at will to mimic their held beliefs.
Figure Iillustrats how, for the three aforementioned priors, the GP reshaped to faithfully represent the belief held by the user whether it be prior over the optimum (left), optimal value (middle), or preference relations between points (right). Our novel framework for Collaborative Bayesian Optimization (ColaBO) diverges from the typical assumption of Gaussian posteriors, and is applicable to any Monte Carlo acquisition function (Wilson et al., 2017 2018 Balandat et al. 2020). Formally, we make the following contributions:
1. We introduce Co ColaBO, a framework for incorporating user knowledge over the optimizer, optimal value and preference relations into Bayesian optimization in the form of an additional prior on the surrogate, orthogonal the conventional prior on the kernel hyperparameters,
2. We demonstrate that the proposed framework is generally applicable to Monte Carlo acqui- sition functions, inheriting MC acquisiiton function utility,
3. We empirically show that ColaBO accelerates optimization when injected with for priors over optimal location and optimal value.
2
BACKGROUND
We outline Bayesian optimization, Gaussian Processes and Monte Carlo (MC) acquisition functions, as well as the concept of prior over the optimum.
2.1 BAYESIAN OPTIMIZATION
We consider the problem of optimizing a black-box function f across set of feasible inputs X = Rd:
x arg max f(x).
We assume that f(x) is expensive to evaluate and can potentially only be observed through noise-
corrupted estimate, Ya. where Væ = f(x) + +,c (0) for some noise level o? In this setting, we wish to maximize ff in an efficient manner. Bayesian optimization (BO) aims to globally maximize f by initial design and thereafter sequentially choosing new points œ,, for some iteration
n, creating the data D,, = Dn-1 U {(I,,, Vvx) (Brochu et al. 2010 Shahriari et al. 2016 Garnett
2022). After each new observation, BO constructs a probabilistic surrogate model f Snook
ct al. 2012 Hutter et al. 2011 Bergstra et al., 221 Ia Müller et al. 22023) and uses that surrogate to build acquisition function ax; D,,) which selects the next query.
2

---

Published as a conference paper at ICLR 2024
2.2 GAUSSIAN PROCESSES
When constructing the surrogate, the most common choice is a Gaussian process (GP) (Rasmussen) & Williams 2006). The GP utilizes covariance function k, which encodes prior belief for the smoothness of J. and determines how previous observations influence prediction. Given observations Dn iteration n, the Gaussian posterior pff(P, over the objective is characterized by the posterior mean un x') and (co-)variance (x, x²) of the GP:
pm(x)) (2) (Kn +z 'y Ex, = k(o, x') - kx(x)) ( +o,)) (x,'),
where (K,))s k( Zj), (x) [,(x,,1)..... k(x, 2T,]_ and o² is the noise variance. For applications in BO and beyond, samples from the posterior are required either directly for optimization (Eriksson et al. 2019) through Thompson sampling (Thompson, 1933). or to estimate auxiliary quantities of interest (Hernández- Lobato et al. 2015 Neiswanger et al. 2202 Hvarfner et al., 2023). For a finite set of query locations (X = T1.... Tt), the classical method of generating samples is via a location-scale transform of Gaussian random variables, f(X) (,(( ) + Le,
where L is the Cholesky decomposition of K and c ~ (N0, 7). Unfortunately, the classic approach intrinsically non-scalable, incurring a (1.3) cost due to the aforementioned matrix decomposition.
2.3 DECOUPLED POSTERIOR SAMPLING
To remedy the issue of scalability in posterior sampling, O(k) weight-space approximations based on Random Fourier Features (RFF) (Rahimi & Recht 2007) obtain approximate (continuous) function
draws [(x) = Σ;n1 (erde(r)), where Φb (az) = 2 (vb x + b.). The random variables w; ~ -V(0, 1),
b; ~ (0( 2x), and eb;; are sampled proportional to the spectral density of k.
While achieving scalability, the seminal RFF approach by Rahimi & Recht (2007) suffers from the issue of variance starvation (Mutny & Krause 2018: Wang et al. 2018 Wilson et al. 2020). As remedy, Wilson et al. (2020) decouple the draw of functions from the approximate posterior p( PD)
into a more accurate draw from the prior p(J), followed by deterministic data-dependent update:
(f) d
f(x)
(x)) (K,, + ooT (y f(x)
from prior
deterministic update
Eq. 2 deviates from the distribution-first approach that is typically prevalent in GPs in favor of a variable-first approach utilizing Matheron's rule (Journel Huijbregts 1976).
2.4 MONTE CARLO ACQUISITION FUNCTIONS
Acquisition functions act on the surrogate model to quantify the utility of point in the search space. They encode a trade-off between exploration and exploitation, typically using greedy heuristic to do
so. A simple and computationally cheap heuristic is Expected Improvement (EI) (Jones et al. 1998
Bull 2011) For noiseless function and a current best observation y the EI acquisition function is 0 Eyg [(vÆ yz²+} For noisy problem settings, noise-adapted variant of ET (Letham et al. 2018) is frequently considered, where both the incumbent y and the upcoming query Ya are substituted for the non-observable noiseless incumbent f and noiseless upcoming query fa. Other frequently used acquisition functions are the Upper Confidence Bound (UCB) (Srinivas et al. 2012)) Probability of Improvement (PI) (Kushner 1964) and Knowledge Gradient (KG) (Frazier al. acquisition functions consider the mutual information to select the next query (x) [([x, ya); (D,), where can entail either the optimum Xg as in (Predictive)
2009). Information-theoree
Entropy Search (ES/PES) (Hennig & Schuler 2012 |eerandde--Loooaae
et al. 2014). the optimal
value f& as in Max-value Entropy Search (MESS Wang & Jegelka| 201
7 Moss al. 2021 or the
tuple (I+,fa fx) for Joint Entropy Scarch (JES) Hvarfner et al. 2022a Tu et al. 2022).
All the aforementioned acquisition functions compute expectations E,, (or alternatively E,,,) over some utility w( u(fa)) of the output (Wilson et al. 2017: 2018). which typically have simple, or even closed-form, solutions for Gaussian posteriors. However, approximating the expectation through Monte Carlo integration has proven useful in the context of batch optimization (Wilson et al. 2018). efficient acquisition function approximation (Balandat et al., 2020), and non- Gaussian posteriors (As tudillo & Frazier 2021). By sampling over possible outputs fax and utilizing the reparametrization
3

---

Published as a conference paper at ICLR 2024
trick (Kingma & Welling 2014, Rezende et al. 2014), utilities be easily computed across larger set applications and be optimized to greater accuracy.
2.5 PRIOR OVER THE OPTIMUM
A prior the optimum (Souza et al. 2021: Hvarfner al., 2022b Mallik al.. 2023) is user-specified belief т X R of the subjective likelihood that a given æ is optimal. Formally,
π ( TP
arg max f(xx')
This prior is generally considered be independent of observed data, but rather result of previous experimentation or anecdotal evidence. Regions that the user expects to contain the optimum will typically have a high value, but this does exclude the chance of the user belief ((Φ) to be inaccurate, or even misleading. Lastly, we require и to be strictly positive in all of X, which suggests that any point included in the search space may be optimal.
METHODOLOGY
We now introduce ColaBO, the first Bayesian-principlec principled BO framework that flexibly allows users to collaborate with the optimizer by injecting prior knowledge about the objective that substantially exceeds the type of prior knowledge natively supported by GPs. In Sec. 3.1 we introduce and derive a novel prior over function properties, which yields a surrogate model conditioned on the user belief. Thereafter, in Sec. 3.2 we demonstrate how the hierarchical prior integrates with MC acquisition functions. Lastly, in Sec. 3,3
we state practical considerations to assure the performance of ColaBo
3.1 PRIOR OVER FUNCTION PROPERTIES
We consider the typical GP prior over functions p(f) (), Σ), where the characteristics of f. such as smoothness and output magnitude, are fully defined by the kernel ht (and its associated hyper- parameters 0 which are omitted for brevity). We seek inject an additional, user-defined prior belief over f into the GP, such as the prior over the optimum in Sec. 2.5 (xx) P (æ arg max fx=)). By postulating that ' is accurate, we wish to form a belief-weighted prior a prior over functions where the distribution over the optimum is exactly (xx). We start by considering the user belief π : X - R from Eq. (3). and extend the definition to involve the integration over similarly to the Thompson sampling definition of Kandasamy al. (2018). Formally,
x (xx)
x arg max f(x')
((ba(=)
where 5x (x(l) if arg exx fz=), and zero otherwise. As such, (x maps function ʃa (p)) to its arg max, and evaluates whether this arg max is equal to ac.
However, belief over the optimum, any other property, of function is implicitly belief over the function of itself. As such, a non-uniform <(x) should reasonably induce a change in the prior p(f) reflect the non-uniform optimum. To this end, we introduce an augmented over the optimum p ø. where o is the prior over possible user beliefs, and draws are random functions P= X number quantifying the likelihood of sample fü under ग (x). Formally, we define p as
belief
R+ which themselves take a function as input, and output positive real
1 - Z
()) x arg max f(xz))
6x(xx))
(5)
where the intractible normalizing constant 7,p= arises from the fact that the integrated density イ ((x) acts on a finite-dimensional property of and not f itself. Under (f),, functions whose arg max lies in high-density region under x will be assigned a higher probability. Notably, the definition in can extend other properties of of as well: user belief P/ over the optimal value fa analogously yields belief over functions 0 (f):
(f) = = max f(x')) = 1
S P
(6)
x'
4

---

Published as a conference paper at ICLR 2024
It is worthwhile to reflect on the meaning of p(,f), and how beliefs over function properties propagate to p(f). Concretely, if the user belief p7_ (f) asserts that the maximal value lies within C1 < max f <2,, the resulting distribution over f should only contain functions whose max falls within this range. Using rejection sampling, functions which disobey this criterion are filtered out, which yields the posterior p(flp) Having defined and exemplified how user beliefs impact the prior over functions p(f), the role of p as likelihood should be apparent: given a prior over functions p(f) and a user belief over functions p(f) which places a probability on all possible draws Si p(f), we can form belief-weighted prior p(f(p)) (pp)))((()(())) Thus, we introduce the formal definition of user belief over function property:
Definition 3.1 (User Belief over Functions). The user belief over functions p( p(fle) p(T)
As the subsequent derived methodology applies independently of the specific property of f that a prior is placed on, we will henceforth consider belief over a general function property p. Having defined the role of p and the posterior over functions it produces, a natural question arises: How is p(Jlp) updated once obxervations 'D are obtained?
Since the data D is independent of the prior (the data generation process is intrinsically unaffected by the belief held by the user), application of Bayes' rule yields the following posterior (())
p(ID, p) p(D, plDP)))))( (DPl))ppP p(fle) p(f) P(ID) & p(D((DD)
(7)
p),
((D)p(p)
where the right side of the proportionality in Eq. Suggests intuitive generation process for samples (f(D, p) to approximate the density p( p). Utilizing the pathwise update from Eq 2 we note that given an approximate draw f from the prior, the subsequent data deta-dependent update is deterministic. Recalling Eq. 2 and assuming independence between p and D, P only affects the draw from the prior, whereas D only affects the update. Consequently, we obtain
()a), d (Al))a)) + (x) (K, + I '(y - (flp)(z) -6),
(8)
draw from prior
deterministic update
where (f(p) ~ p(f) are once again obtained using rejection sampling on draws from p( p() Figure 2 displays this in detail: given the typical GP prior over functions and a user belief over the optimum, obtain distribution over functions p(flpt) before having observed any data (top right). Samples from the approximate prior p(f) (light blue) are re-sampled proportionally their probability of occurring under the prior p* (f() in green, leaving samples (flpp" in navy blue, which are highly probable under Prr Once data is obtained, these samples are updates according to Eq. g which preserves the shape of the samples far away from observed data and yields the desired posterior.
3.2 PRIOR-WEIGHTE MONTE CARLO ACQUISITION FUNCTIONS
Naturally, neither the weiiht-weighted prior p()lp) nor the weiihf-weighted posterior pDf p) have a closed- form expression. Both are inherently non- Non-Gaussian for non-uniform beliefs. As such, we resort to MC acqui- sition functions to compute utilities that are amenable to BO. In the subsequent section, we focus on the prevalent acquisition functions EI, and MES.
Samples from posterior
Good rester ander
Obsevailzza
6
Effect Acquisition Function
0.10
Expected Improvement The computation of the MC-EI within the Co ColaBO framework requires only minor adaptations of the original MC acquisition function. By definition, MC-EI assigns utility u as Ugz (f(x)) max(f# f(x), 0), which yields Oegr (a;; D) T., F [pz (fa)] &
0.00
Celabo
0.00
Vendla El
0.2
3.
3.
0.
1
Figure 3: (Top) Draws from PDJ(D) (light blue) and p(flp. D) with a prior p located in the green region. (Bottom) Vanilla MC-RT and Col BO MC-EI, resulting from computing the acquisition function from sample draws from p(flp, D).
(9)
max(f** f(() 0), f( pD) (10)
5

---

Published as a conference paper at ICLR 2024
Samples from prior
Pathwise updated posterior samples
6- #ttid
1-pt0 ei
User triar
fiod rigton under Obverratizsa
Good region under
0.0
Monenis prior
0.8
1.0
0.0
0.2
0,6 Monienls of pasterior
9.8
1.0
Posterior
0.2
Figure 2: (Top left) Draws from the prior p(f) (light blue) and the helief-weighted prior p(f [() whose members are likely to have their optimum within the green region. (Top right) Pathwise updated draws based on observed data. As the green region is distant from the observed data, samples are almost unaffected by the data in this region. (Bottom left) Exact mean and standard deviation (Ha, an)) p(f) and estimated mean and standard deviation of p(/(p). (Bottom right) Exact (D) and estimated p(flp. D). As p([(p) constitutes of functions whose optimum is located within the green region the resulting model has a higher mean and lower variance within this region. Morcover, p(f globally displays lower upsidc variance compared to the vanilla GP.
Utilizing rejection sampling, we can compute the MC-ET under the ColaBO posterior accordingly,
c2: p) (/.p..p(usr(aa)
(11)
J, uer (er((e)o(()(((DD El(ro max(St 119.0) -10
10 ~ p(f(x))D),
(12)
wherein samples in Eq. 12 are drawn from the prior, retained with probability ) max and pathwise updated. In Figure we demonstrate how ColaBO-EI differs from MC-EI for an identical posterior as in Figure 2 By computing dga from samples biased by p. ColaBO substantially directs the search towards good regions under p. Derivations for and are analogous to that of EI.
Max- Value Entropy Search We derive a ColaBO-MES acquisition function by first considering the definition of the entropy, H[p(y, [D]] = Ey. |pl log p( [D)]. When considering the belief- weighted posterior, we further condition the posterior p and obtain
OWEs (x)) D P. D,,, []]
E I plg ID, p)]
(13)
E fa D2 [B p. [E plgglll f+)]]] -Eje p. E v. [log pl (( fal p)]1
(14)
1
K
1
logp(v
1
1120 11 p )(((()))))))))
Σ
Σ
logplu
(1) )() )()()))
)((
(15)
where Z, is normalizing constant Σ, p(f(t) brought by sampling optimal values, Yie Ve can trivially be obtained by sampling Gaussian noise -N(0, o²) to a noiseless observation fz D in the innermost expectation, and fa and fe are obtained through the pathwise sampling procedure outlined in Eq. 8 The samples are evaluated on p((ual (l)_=/ll)-((e fa: f))). As evident by Eq. 15 p affects the posterior distribution of both the observations Yz and the optimal values fa. PES and JES are derived analogously. However, these acquisition function require conditioning on additional, simulated data and consequently, additional pathwise updates, to compute.
3.3 PRACTICAL CONSIDERATIONS
ColaBO introduces additional flexibility to MC-based BO acquisition functions. The ColaBO framework deviates from vanilla (q-) acquisition functions (Wilson et al. 2017 Balandat et al.)
2020) by utilizing approximate sample functions from the posterior, as opposed to pointwise draws from the posterior predictive and the reparametrization trick (Kingma & Welling 2014 Rezende
ct al. 2014). ColaBo holds three shortcomings not prevalent in vanilla MC acquisition functions: (1) it cannot utilize Quasi-MC in the draws from the predictive posterior (only in the RFF weights),

---

Published as a conference paper at ICLR 2024
Algorithm 1 ColaBO iteration
1: Input: User prior p. number of function samples L, current data D
2: Output: Next query location x'.
3: for E {1..... [] do
Pathwwsseepaatee 4: pl9) (D)
p( ((f()) n), F() p(f)
Sample functions and evaluate F Per-sample update as in Eq.
5:
6: end for
7: p(f E, (f
I> Form MC estimate of pesterior
8: x' =arg max BotfID [u(fa))
Maximiize MC acquisition
(2) it cannot fix the base samples (Balandat et al., 2020) drawn from the posterior for acquisition function consistency across the search space, and (3) the RFF approximation of p(f) introduces bias. This approximation error is substantially more pronounced for the Matérn 5/2-kernel than the squared exponential, leaving ColaBO best suited for the latter. In Sec. 4.] we empirically display the impact of these shortcomings. While acquisition function optimization no longer enjoys improved accuracy resulting from reparametrization, the acquisition function can still benefit from the fact that ColaBO backpropagates through quantities computed sums of smooth functions.
RESULTS
We evaluate the performance of ColaBO on various tasks, using priors over the optimum Te obtained from known optima symthetic tasks, well as from prior work (Mallik et al. 2023 on realistic tasks. We consider two variants of ColaBO: one using LogEI (Ament et I (023). a numerically stable, smoothed logsumexp transformation of E.I. with analogous derivation, and variant using ME.S. We benchmark against the vanilla variants of each acquisition function, as well as TBC (Hvarfner et al., 20226) and decoupled Thompson sampling Thompson (1933); Wilson al.) 2020). All acquisition functions are implemented in BoTorch (Balandat et al. (0200) using a squared exponential kernel and MAP hyperparameter estimation. We present experiments with Matérn- 5/2 (Matérn 1960) kernel in App. C.I Unless stated otherwise, all methods are initialized with the mode of the prior followed by 2 Sobol samples. The experimental setup outlined in Appendix and our code is publicly available at
http:hhwb.ooo/heeeeeeeeeeeeeee www.co/....ac....co.oooooo. fne r co coo
4.1 APPROXIMATION QUALITY OF THE COTABO FRAMEWORK
Firstly, we demonstrate the approximation quality of ColaBO without user priors assert its accuracy compared to vanilla MC acquisition function. To facilitate compar- ison, randomly sample 10 points the Hartmann (3D) function, and optimize LoqEI with a large budget. We subsequently optimize Co 1 aBO-LogEI on the same of points and compare the arg max to the solution found by the gold standard. Figure 4 displays the (log10) Euclidian distance between the arg max of LogEI and its ColaBO variant. We note that, for small amounts (a 256) of poste- rior samples, the error induced by RFF bias is relatively low, which is evidenced by all RFF variants being roughly equal in distance to the true acquisition function optimizer.
Cosvergence Acquisiion Optimus
512817
10208195
60088225
125
5039
Figure 4: Mcan and 1/4 standard devi- ation of MC-induced errors of Col a LogE1 relative vanilla LogE I. mea- sured by the distance to the arg max of the acquisition function on Hartmann (3D) on 10 randomly sampled points for 40 seeds.
4.2 SYNTHETIC FUNCTIONS WITH KNOWN PRIORS
We adapt similar evaluation protocol to Hvarfner et al. (2022b), and evaluate ColaBO for two types of user beliefs for synthetic tasks: well-located and poorly located priors over the optimal location, designed to emulate well- well-informed and poorly-informed practitioner, respectively. The well-located prior offset by a small (10%) amount from the optimum, and the poorly located prior is maximally offset, while retaining its mode inside the search space. Complete details the priors can be found
7

---

Published as a conference paper at ICLR 2024
Hartmonn (4D)
Lavy (5D)
Hartmesn (RD)
Rocenbrock (BB)
Styareng (TB)
353
ColaBi-Loghi
ColeBO MES
-019
Legiti
155
0.5
LB
MISS
-1
150
30 110
Baraton
Dwrstoo
Figure 5: Performance synthetic functions with well-located priors. Both Col LBO-LEEI and ColaBO- MES offer drastic speed-ups over their vanilla variants, and offer similar performance to TBO. The ranking of ColaBO acquisition functions are generally consistent with their respective vanilla variants. This is most prominent on Rosenbrock (6D), where ColaBO-MES struggles similarly to vanilla MES.
Hartmonn (4D)
Levy (5D)
Hartmann (GD)
Rosenbrock (GD)
Stybenn (7D)
6:2
LogEl
4:33
PIBG
Derrution
Daraato
Derraboa
Dirrabioo
Figure 6: Performance on poorly located priors. ColaBO acquisition functions are more robust than TBC, as it frequently recovers the performance of the vanilla acquisition function before the total budget is depleted. ColaBO-LogEI struggles marginally on Hartmann (6D). ColaBO-MES recovers the baseline on all tasks.
On well-located priors, both Co PolaBO-LoggE) and ColaBO-MES demonstrate substantially improved performance relative to their vanilla counterparts, comparable to TBO on all benchmarks. On poorly located priors, ColaBO demonstrates superior robustness, recovering the performance of the vanilla acquisition function within the maximal budget of 20/ iterations and clearly outperforming #30, which more frequently misled by the poor prior. In Appendix C.2 also demonstrate Co lolaa aBO utilizing (accurate) beliefs over the optimal value: similarly to Figure 5 ColaBs yields increased efficiency relative baselines, albeit not substantial. Moreover, demonstrate its usage with batch evaluations on well-located priors in Sec. C.3 showing that the drop in performance from batching evaluations is marginal worst.
in Appendix B.3
4.3 HYPERPARAMETER TUNING TASKS
Lastly, evaluate ColaBO three 4D deep learning HPO tasks from the PD1 (Wang al. 2023) benchmarking suite. While the optima for these tasks are ultimately unknown, we utilize the priors provided in MF-Prior-Bente starting point for further optimization. To emulate realistic HPO setting, we consider a smaller optimization budget of 10D iterations, and initalize all methods that utilize user beliefs with only one initial sample, that being the mode of the prior. The two Cola30 variants perform best in this evaluation, producing the best terminal performance on two tasks (CIFAR, LMIB), with all methods being tied the third (CIFAR). ColaBo demonstrates consistent speed-ups compared to its vanilla counterparts, surpassing the terminal performance of the baseline within a third of the budget on CIFAR and LMIB. In App. A we benchmark on tasks from LBBench (Zimmer et al. 2020). displaying similar performance.
Mallik et al. 2023), which are intended to provide a good
5 RELATED WORK
In BO, auxiliary prior information can be conveyed in multiple ways. We outline meta learning/trans- fer learning for BO based on data from previous experiments, and data-less approaches.
Learning from Previous Experiments Transfer learning and meta learning for BO aims to automatically extract and use knowledge from prior executions of BO by pre-training the model on
https://github..... http:///wwutttm/mffprrrrrrr-eeee
8

---

Published as a conference paper at ICLR 2024
PD1 PO1-MNT
PDI-CIPAR
PD1-EMTB
Cota Bo-Logh
CoLLBO-MEB MES
Tegiti
MIS
25
78
Figure 7: Performance on the 4D PDI hyperparameter tuning tasks of various deep learning pipelines. ColaBO drastically accelerates optimization initially, finding configurations with close to terminal performance quickly. TBC offers competitive performance, but lacks the rapid initial progress ColaBO on CIFAR and LMIB.
data acquired from previous executions (Swersky et al., 2013 Wistuba et al., 2013 Perrone et al. 2019 Feurer et al.) 2015 2018 Rothfuss et al. 202 Wistuba & Grabocka| [022] Feurer et al. 2022). Typically, meta- and transfer learning exploit relevant previous data for training the GP for the current task while retaining predictive uncertainty to account for imperfect task correlation.
Expert Priors over Function Optimum Few previous works have proposed to inject explicit prior distributions over the location of an optimum into BO. In these cases, users explicitly define prior that encodes their beliefs on where the optimum is more likely to be located. [Brgsttra et al. (201 la) suggest an approach that supports prior beliefs from fixed set of distributions, which affects the very initial stage of optimization. However, this approach cannot be combined with standard acquisition functions. BOPrO (Souza et al., 2021) employs similar structure that combines the user- eserrprrvided prior distribution with a data-driven model into a pseudo-posterior. From the gpeudo-posterion configurations are selected using the acquisition function, using the formulation in Bergstra et al.) (2011a). nBO (Hvarfner et al. (2022b) suggests ceneral-purpose where the influence of the prior decreases over time. They provide convergence guarantees for when the framework is applied to the EI acquisition function. While effective, none of these approaches act fashion, but strictly as heuristics. Moreover, they
weighte acquisition function,
on the surrogate model in Bayesian-prineee solely focus priors over optimal inputs, thus offering less utility than ColaBO.
Priors over Optimal Value Similarly few works have addressed the issuc of auxilliary knowledge of the optimal value. Both Jeong& & Kim (202T) and Nguyen & Osborne (2020) propose altering the GP and accompanying it with tailored acquisition functions. Peong & Kim (2021) employ variational inference, proposing distinct variational families depending on the type of knowledge pertaining to the optimal value. [Nguen & Osborne (2020) use parabolic transformation of the output space to ensure an upper bound preserved. Unlike ColaBO, neither of these methods general enough to accompany arbitrary priors guide the optimization.
6 CONCLUSION, LIMITATIONS AND FUTURE WORK
We presented Co aBC, flexible BO framework that allows practitioners to inject beliefs over function properties in a Bayesian Bayesian-principled manner, allowing for increased efficiency in the BO procedure. ColaBO works across collection of MC acquisition functions, inheriting their flexibility in batch optimization and ability to work with non-Gaussian posteriors. It demonstrates competitive performance for well-located priors, using them to substantially accelerate optimization. Moreover, it retains approximately baseline performance when applied to detrimental priors, demonstrating greater robustness than BBO. Co 1 aBO crucially relies on multiple steps of MC. While flexible. this approach drives computational expense in order to assert sufficient accuracy, requiring tens of seconds per evaluation achieve desired accuracy, depending on the size of the benchmark. Moreover, obtaining draws from Pa scales exponentially in the dimensionality of the prior. While practitioners are unlikely to specify priors over more than handful of variables, Co aBO may become impractical when priors of higher dimensionality are employed. Paths for future work could involve more accurate and efficient sampling procedures (Lin et al., (023) from the weighted prior, as well as variational (Titsias 2009) or pre-trained Müller et al. (2022); Müller et al. (2023) approaches to obtain a representative belief-biased model with an analytical posterior. This would likely bring down the runtime of Co ColaBA and broaden its potential use. Lastly, applying ColaBo to multi-fidelity optimization (Kandasamy et al., 2016 Mallik et al.) (203) offers an additional avenue for increased efficiency which would further increase its viability on costly deep learning pipelines.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their valuable contributions. Luigi Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project - Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Carl Hvarfner, Erik Hellsten and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 5021.0348. Frank Hutter acknowledge support through TAILOR, project funded by the EU Horizon 2020 research and innovation programme under (DFG, German Research Foundation) Baden-Württembenn Württemberg through bwHPC and the German
GA No 952215, by the Deutsche Forschungsgemeinschatt under grant number 417962828, by the state of Baden- Research Foundation (DFG) through grant no INST 39/963-1 FUGG, and by the European Research Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765). The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC partially funded by the Swedish Research Council through grant agreement no. 2018-05977 Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
Funded by the European Union
REFERENCES
Sebastian Ament, Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Un- expected improvements to expected improvement for bayesian optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https::/t///// /tttps://ppeereview net/ id= 1oaddd-vvyyAc vyAA66 9PE
Raul Astudillo and Peter Frazier. Bayesian optimization of function networks. Advances in neural information processing systems, 34:1 14414466314475. 2021.
M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. Botorch:
A framework for efficient monte-carlo bayesian optimization. In Advances in Neural Information Processing Systems, 2020. URL http:/ arxiv.
/ wttp:/wwwwrr//b//11......1 06403
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In
J. Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger (eds.), Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS'I) pp. 2546-2554, 201 la.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Hyper-Paramette Parameter Optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 24. Curran Associates, Inc., 2011b.
E. Brochu, Cora, and N. de Freitas. tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. 1rXiv:1012.2599v/ [cs. 2010.
Adam D. Bull. Convergence rates of efficient global optimization algorithms. 12:2879- 2011. 1.
R. Calandra, N. Gopalan, A. Seyfarth, J. Peters, and M. Deisenroth. Bayesian gait optimization for bipedal locomotion. In Pardalos and M. Resende (eds.), Proceedings of the Eighth International Conference on Learning and Intelligent Optimization (LION' 14), 2014.
Adel Ejjeh, Leon Medvinsky, Aaron Councilman, Hemang Nehra, Suraj Sharma, Vikram Adve, Luigi Nardi, Eriko Nurvitadhi, and Rob A Rutenbar. Hpvm2fpga: Enabling true hardware- fpga programming. In Proceedings of the 33rd IEEE International Conference on Application-spectee Systems, Architectures, and Processors, 2022.
10

---

Published as a conference paper at ICLR 2024
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information
Processing Systems, pp. -507, 2019. URL http: /tetp://papee http:/ppprs...nncc www.nnisssccppppppe sca aecalabll. aaallllle e e-globalla opt mi zat onn non-ve via loca bww--bbayyennnteeeeeeaaaaa zo
pdf
M. Feurer, Jost Tobias Springenberg, and F. Hutter. Initializing bayesian hyperparameter optimization via meta- mneaa-larming. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence,
pp. 1128-1135, 2015.
M. Feurer, B. Letham, Hutter, and E. Bakshy. Practical transfer learning for bayesian optimization. ArXiv abs/1802.022199 2018.
Matthias Feurer, Benjamin Letham, Frank Hutter, and Eytan Bakshy. Practical transfer learning for Bayesian optimization. arXiv preprint 1802.02219, 2022.
Peter Frazier, Warren Powell, and Savas Dayanik. The wewwllede-grrdiiee policy for correlated normal belief'.. INFORMS journal Computing, 21(4):599-613 2009.
R. Garnett. Bayesian Optimization. Cambridge University Press, 2022. Available for free at
hatpps://bayessplook.com
Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chemical Science, 2020.
Hennig and C. J. Schuler. Entropy search for information-efficien global optimization. Journal of Machine Learning Research, 13(1):1809-189) June 2012. ISSN 1532-4435.
J. M. Hernández--Lobato, M. W. Hoffman, and Z. Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Process- ing Systems, 2014. URL
newrips. httpa/ss.ccppppppe/222202000 pdd
066 (
José Miguel Hernández- Michael Gelbart, Matthew Hoffman, Ryan Adams, and Zoubin Ghahramani. Predictive entropy search for bayesian optimization with unknown constraints. In International conference on machine learning, pp. 1699-1707. PMLR, 2015.
Daolang Huang, Louis Filstroff, Petrus Mikkola, Runkai Zheng, and Samuel Kaski. Bayesian optimization augmented with actively elicited expert knowledge, 2022.
F. Hutter, H. Hoos, and K. Beyton-Brown Sequential model-based optimization for general algorithm configuration. In C. Coello (ed.), Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION'II) 11), volume 6683, pp. 507-523, 2011.
Carl Hvarfner, Frank Hutter, and Luigi Nardi. Joint entropy search for maximally-infomme bayesian optimization. In Proceedings of the 36th International Conference on Neural Information Process- ing Systems, 2022a.
Carl Hvarfner, Danny Stoll, Artur Souza, Marius Lindauer, Frank Hutter, and Luigi Nardi. PiBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. In International Conference on Learning Representations, 2022b.
Carl Hvarfner, Erik Hellsten, Frank Hutter, and Luigi Nardi. Self-correcting bayesian optimization through bayesian active learning. In Teirth- Conference Neural Information Processing Systems, 2023. URL httttss://ppeervieeeeee http///ee/eeeeeeeeee furmmforrmmmmmm wtw.:mmmdddd99MMX__....
Taewon Jeong and Heeyoung Kim. Objective bound conditional gaussian process for bayesian optimization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4819-4828. PMLR, 18-24 Jul 2021. URL jeong21a. html
D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black-bor functions. Journal of Global Optimization, 13:455-492 12 1998. doi: 10.1023/A:1008300
11

---

Published as a conference paper at ICLR 2024
A G Journel and CJ Huijbregts. Mining geostatisties, Jan 1976.
K. Kandasamy, G. Dasarathy, J. Oliva, J. Schneider, and B. Póczos. Gaussian Process Bandit Optimisation with Multi-fidelity Evaluations. In D. Lee, M. Sugiyama, U. von Luxburg, Guyon, and R. Garnett (eds.), Proceedings of the 30th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 16), pp. 992-1000, 2016.
K. Kandasamy, A. Krishnamurthy, J. Schneider, and B. Póczos. Parallelised Bayesian optimisation via Thompson sampling. In A. Storkey and F Perez-Cruz (eds.), Proceedings of the 2 Inter- national Conference Artificial Intelligence and Statistics (AISTATS), volume 84, pp. 133-142. Proceedings of Machine Learning Research, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2014. URL https //rxiv.org/abs/////.. 6114
Arun Kumar, Santu Rana, Alistair Shilton, and Svetha Venkatesh. Human-ai collaborative bayesian optimisation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 16233-16245. Curran Asso- ciates, Inc., 2022. URL / neuri ps cc paper_ fol_fileee files/paper 2022 22222211477555a3333aaaaaaaeeeea3 Coaeeeeeeeepppeerrooeeeeeeeeeee
H. J. Kushner. A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise. Journal of Basic Engineering, 86(1):97-106, 03 1964. ISSN 0021-9223. doi: (01111/113653122 URL http::////o...orr/0.1111111....
B. Letham, K. Brian, G. Ottoni, and E. Bakshy. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis, 2018
Jihao Andreas Lin, Javier Antorán, Shreyas Padhy, David Janz, José Miguel Hernández-Lobato, and Alexander Terenin. Sampling from gaussian process posteriors using stochastic gradient descent. In Seirty-eeventh Conference on Neural Information Processing Systems, 2023. URL
https://openreview/ hetp:eeneeeeeoooooeeeeeee-- fttp:/oroummii-----g0----
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54): 1-9, 2022. URL http://pt////.rrroooppppps//////
html
Neeratyoy Mallik, Edward Bergman, Carl Hvarfner, Danny Stoll, Maciej Janowski, Marius Lindauer, Luigi Nardi, and Frank Hutter. Priorband: Practical hyperparameter optimization in the age of deep learning. arXiv preprint 2306. 2306.12370, 12370, 2023.
B. Matérn. Spatial variation. Meddelanden fran Statens Skogsforsknningssisttinu, 1960.
Matthias Mayr, Carl Hvarfner, Konstantinos Chatzilygeroudis, Luigi Nardi, and Volker Krueger. Learning skill-based industrial robot tasks with user priors. IEEE 18th International Conference on Automation Science and Engineering, 2022. URL https:://axxii www. /abs//////// /2008
01605
J. Mockus, V. Tiesis, and A. Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2(117-129):2, 1978.
Henry B. Moss, David S. Leslie, Javier Gonzalez, and Paul Rayson. Gibbon: General- pprppse information-baut bayesian optimisation. Journal of Machine Learning Research, 22(235):1 49, 2021. URLhttte imll..rrr... wttp://.rrggppppee///////// 2wt.://221120...11 0120. html
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Representations, 2022. URL wtt.t:ttss//// hnttps::/ htt.://wwww.e..e.ee.eo/oooooooo.
12

---

Published as a conference paper at ICLR 2024
Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. PFNs4BO: In-context learning for Bayesian optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 25444 25470. PMLR, 23-29 Jul 2023. htt.:htttpppptoteeeeeeeoo
mm www.aar.......ssspmm.m /20rss//v002
mu muller23a.htm.
Mojmir Mutny and Andreas Krause. Efficient high dimensional bayesian optimization with additivity and quadrature fourier features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
www.inng....uuuuuuueeeee neur ps cc/paper Paper. pdf
Files
L. Nardi, D. Koeplinger, and K. Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunica- fion Systems (MASCOTS), pp. 347-358. IEEE, 2019.
Willie Neiswanger, Ke Alexander Wang, and Stefano Ermon. Bayesian algorithm execution: Esti- mating computable properties of black-box functions using mutual information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8005-8015. PMLR, 18-24 Jul 2021. URL https: wttpp::/pproeeeeeeeeeeeee http:e////pocceeiig.e.oooo pwwss/r.esssvvvv.ee http:///www.eeeiiaeeeeeeeeeeeeee www.eeewwwwaggaaeeeeeem html
Vu Nguyen and Michael A. Osborne. Knowing the what but the where in Bayesian optimization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7317-7326. PMLR, 13-18 Jul 2020. URL httppp:://pppoeeennn............ http://s.........ooo./////////// html
C. Oh. E. Gavves, and M. Welling. BOCK Bayesian optimization with cylindrical kernels. In International Conference on Machine Learning, pp. 3865-3874, 2018.
V. Perrone, H. Shen, M. Seeger, C. Archambeau, and R. Jenatton. Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning. In Advances Neural Information Processing Systems, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural In- formation Processing Systems, volume 20. Curran Associates, Inc.. 2007. URL https: https://proceedingg nwwrips.eipp.... cc/paper httpppppp...Leee/ppp//////////11 feboaeec5553efffffeeeeee
01
.ddf
C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara (eds.), Pro- ceedings of the 31st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1278 1286, Bejing, China, 22-24 Jun 2014. PMLR. URL www.pr..........mmmmm v32/ rezendel44 html
Jonas Rothfuss, Vincent Fortuin, Martin Josifoski. and Andreas Krause. Pacoh: Bayes-optimal meta- learning with pac-guarantees. In Proceedings of the 38th International Conference on Machine Learning, pp. 9116-9126, 2021a.
Jonas Rothfuss, Dominique Heyn, Jinfan Chen, and Andreas Krause. Meta-learning reliable priors in the function space. In Advances in Neural Information Processing Systems, volume 34, 2021b.
Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural architec- ture search via bayesian optimisation with weisfeille-Iehmann kernels. In International Confer- ence Learning Representations, 2021. URL
net ftrtt//oummmmim
39Rv7qaX
B. Shahriari, K. Swersky, Z. Wang, R. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings the IEEE, 104(1):148-175, 2016.
13

---

Published as a conference paper at ICLR 2024
L. Smith. A disciplined approach to neural network hyper- syper-parameters: Part 1- 1-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.098200 2018.
Snoek, H. Larochelle, and R. Adams. Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, I.. Bottou, and K. Weinberger (eds.). Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 12), pp. 2960-2968, 2012.
A. Souza, L. Nardi, L. Oliveira, K. Olukotun, M. Lindauer, and F. Hutter. Bayesian optimization with a prior for the optimum. Machine Learning and Knowledge Discovery in Databases. Research Track European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Purt III, volume 12977 of Lecture Notes in Computer Science, pp. 265-296. Springer, 2021.
N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger. Information-theoreetee regret bounds for gaussian process optimization in the bandit setting. IEEE Trunsactions on Information Theory, 58(5)33250-3255, May 2012. ISSN 1557-9654. doi: 10.11099/2.2111.218220 URL http 111.110..1111TTT 2011 221.0318233
K. Swersky, J. Snoek, and R. Adams. Multi-task Bayesian optimization. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger (eds.), Proceedings of the 27th International Conference on Advances in Neural Information Processing Systems (NeurIPS' 13), pp. 2004-2012, 2013.
W. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294 1933.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In David van Dyk and Max Welling (eds.). Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 567-574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR. URL https::/proceedinng ml www.sppees/eaee https////ttittsttstteeee html
Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi- objective bayesian optimization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022. URL
net www.fforrmiiooommm 10
Q. Wang, Y. Ming, Z. Jin, Q. Shen, D. Liu, M. J. Smith, K. Veeramachaneni, and H. Qu. Atmseer: Increasing transparency and controllability automated machine learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 119, pp. 1-12. Association for Computing Machinery, 2019.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning (ICML), 2017.
Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian optimization in high-dimensional spaces. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings of the Twenty-Firsl International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings Machine Learning Research, pp. 745-754. PMLR, 09-11 Apr 2018.
URL htttss:/pproceec.nng. v
mlr. www.ppes/vvvv. wttps////v//waagggee www.waggal...... 18c. html
Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zachary Nado, Justin Gilmer, Jasper Snoek, and Zoubin Ghahramani. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.082/5. 2023.
C. White, W. Neiswanger, and Y. Savani. BANANAS: Bayesian optimization with neural architectures for neural architecture search. In Q Yang, K. Seyton-Brownn and Mausam (eds.), Proceedings of the Thirty-Fifth Conference on Artificial Intelligence (AAAI'2I), pp. 10293-10301. Association for the Advancement of Artificial Intelligence, AAAI Press, 2021.
14

---

Published as a conference paper at ICLR 2024
James Wilson, Frank Hutter, and Mare Deisenroth. Maximizing acquisition functions for bayesian optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL 498f2c2168BBF 45572221888444111111100033e301e3
mew.reiiiininnnnnnnnneeeeee wwcpppppe/2..///Ee.ccp //018//iee/
James T. Wilson, Riccardo Moriconi, Frank Hutter, and Mare Peter Deisenroth. The reparameteriza tion trick for acquisition functions, 2017. URL httpss:/aaxxi........ org/ wttppap//g//.....ot/p///.// 0
James T. Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Mare Peter Deisenroth. Efficiently sampling functions from gaussian process posteriors. International 09309
Conference Machine Learning, 2020. URL
M. Wistuba, N. Schilling, and L. Schmid--Thieme Hyperparameter search space pruning A new component for sequential model-based hyperparameter optimization. In A. Appice, P. Rodrigues, V. Costa, J. Gama, A. Jorge, and C. Soares (eds.), Machine Learning and Knowledge Discovery in Databases (ECML/PKDD' 15), volume 9285, pp. 104-119, 2015.
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates, In International Conference on Learning Representations, 2021. URL htttss://oppereeiiie net/ htte//oormmmdddbdxxe33333333
Lucas Zimmer, Marius Thomas Lindauer, and Frank Hutter. Auto-pytorch tabular: Multi-fidelity
metalearning for efficient and robust autodl. ArXiv, abs/2066.13799, 2020. URL https://apl
semanticschol ar. o : 220041844
15

---

Published as a conference paper at ICLR 2024
Nomso
Phoneme
Tensts
Pobert
Seymest
014
ColoBi-Loghh
1.0
ColeBO-MEB MES
Eegiti
18
600
535
045
Twoston
Berator
Hecation
Banatoo
Ewintton
Figure 8: Performance the 6D LCBench hyperparameter tuning tasks of various deep learning pipelines. ColaBO substantially improves on the non-prior baselines for five tasks. BBO performs best on aggregate, and achieves the best acceleration in performance at carly itcrations.
A LCBENCH BENCHMARKING
We evaluate all methods on five deep learning tasks (6D) from the LCBench (Zimmer et al. 2020) suite, utilizing priors from MF-Prior-Bench. The chosen tasks the five tasks with available priors of the best (good) strength, as the benchmark suite. Figure shows the performance of all methods on the LCBench tasks. CoLaBO improves substantially on the baseline approaches for of 5 tasks. TBO is the overall best-performing method, followed by Col aolaBO-T....
B EXPERIMENTAL SETUP
B.1 MODEL.
We outline the model used and the budget allocated to the various MC approximations involved with ColaBy For all experiments, the hyperparameters at every iteration of BO. All hyperparameters lengthscale, outputscale and observation noise (0 {e, o are given conventional CA((0, 1) prior, applied on normalized inputs and standardized outputs. Furthermore,
utilize MAP estimation of the hyperparameters, and update
fit the constant of the mean function, assigning a (O, 1) prior as well. In Tab. [ we display the parameters of the MC approximations for various tasks. No. f is the maximal number of functions used in the MC computation of the acquisition function. No. Reamples is the number initial posterior draws maximally used for the re-sampling of functions from the posterior p(flp). Lastly, No. f+ the number of optimal values used in the computation of ColaBO-MES.
Task
No.
No. RFFs
No. Resamples
No. f
Synthetic Good
768
2048
1.5 * 10
32
Synthetic Bad
768
2048
1.5 10
PDI
512
4096
2 105
32
Appendix
512
1024
10a
32
Table 1: Budget- parameters of the Monte Carlo approximations for all tasks.
B.2 BENCHMARKS
We outline the benchmarks used, their search spaces and the amount of synthetic noise added. When adding noise, we intend for the ratio of noise variance to total output range to be approximately equal across benchmarks.
B.3 PRIORS
For synthetic benchmarks, the approximate optima of all included functions can be obtained in advance. Thus, the correctness of the prior is ultimately known in advance. For a function of dimensionality d with optimum at &. the well-located prior is constructed by sampling offset
16

---

Published as a conference paper at ICLR 2024
Task
Dimensionality
de
Search space
Hartmann (4D)
4
0.25
[0. 1]
Levy (5D)
5
0.5
[-5, 5]
Hartmann (6D)
6
0.25
[0, [0, 1]
Rosenbrock (6D)
6
5
[-2.048, 2.048]
Stybtang (7D)
1
4, 4]
Table 2: Benchmarks used for the Bayesian optimization experiments.
direction and scaling the offset by a dimensionality- and quality-specific term cdd qvd. For the well-located prior synthetic tasks, we use 0.1, which implies that the optimum is located 10% of the distance across the search space away from the optimum, and construct a Gaussian prior as
(16) with as 25% for all tasks and prior qualities. For our 20 runs of the well-located prior, this procedure yields us 20 unique priors per quality type, with identical offsets from the true optimum. No priors with mode outside the search space were allowed, such priors were simply replaced. For the misinformed priors, set q 1, guaranteeing that the mode of the prior will be outside of the search space, and subsequently relocating to the edge of the search space by its shortest path. Priors for all tasks displayed in Tab. 3 For the PD1 tasks, the location for the priors were obtained from MF-Prior- MF-Prror-Bennch/ [tt kttpp:: //tpttub...././///// com/ aut wwwp:////wennooeeeeeeeeeeeee However, these priors require offsetting in order not be too strong, thus making subsequent BÔ obsolete. PDI priors are provided in [0, -J-mommaleeed space for simplicity.
Taz (x) ~N(x + cac/llell,o.), e~ (N(O
Task
Location
Offset, good
Offset, bad
Hartmann (4D)
[0.19, 0.19. 0.56, 0.26]
0.1vD
max
0.25
Levy (5D)
[1]
IVD
max
2.5
Hartmann (6D)
[0.20, 0.15, 0.48, 0.28, 0.31, 0.66]
0.1v/D D
max
0.25
Rosenbrock (6D)
[1]
0.4096
max
1.024
Stybtang (7D)
2.9]
0.8V D
max
2
PD1-WMT
[0.90, 0.69, 0.02, 0.97]
0.05
N/A
0.25
PD1-CIFAR
[1, 0.80, 0.0, 0.0]
0.05
N/A
0.25
PDI-LMIB
[0.91, 0.67, 0.36. 0.85
0.05
N/A
0.25
Table 3: Rag for synthetic BO tasks of both prior qualities and PDI.
C ADDITIONAL EXPERIMENTS
We provide complementary experiments those introduced in the main paper. Firstly, display results when Co cBl is used with prior Ty, over the optimal value in Sec. C.2 demonstrate Col aBOs extensibility to batch evaluations, seamlessly extending the work of (Wilson
C.2 In Sec. C.3 we
et al. 2017).
C.1
SYNTHETIC MATERN KERNEL EXPERIMENTS
We evaluate Co BOand all baselines on the synthetic tasks with a Matern-5/2 kernel and the good user belief over the optimum. We note that roughly half of all mBO runs struggle with numerical instability from iteration 60 onwards, which produces stagnation in performance and infrequent gains.
C.2 MAX-VALUE PRIORS
We evaluate ColaBO with priors over the optimal value πyz in Figure I0 For each task, we place a Gaussian prior the optimal value, centering it exactly at the optimal value. Notably, such prior
17

---

Published as a conference paper at ICLR 2024
Hortmann (1
Lavy (5D)
Hartmam (ID)
Rosenbrack (TD)
Stytlong (PB)
2.50
ColoBi-
1-30
2.25
ColeBO-MBB MES
Laghti
3.00
-.77
6.75
Hensienn
Bomtoon
Brodive
Dwrstion
Beeation
Figure 9: ColaBO on the synthetic tasks with Matern kernel. Due to the difficulty of the RFF approximation, ColaBO-LogBI struggles on Hartmann (6D), and ColaBB performance marginally worse aggregate.
Hartmman
Hartmann
Rosenbrock
433
Laght MRS
Cole3D-Lapti
Borsta
Dorska
Dereloos
Dooratoo
Figure 10: ColaBO with priors over the optimal value. Terminal performance substantially increases on out of 5 benchmarks (Levy, Hartmann (6D), Stybtang), and is approximately preserved on the final two. ColaBO-MES improves marginally more than Cola B0-lloge when utilizing prior क.. over the optimal value.
substantially influences the exploration-exploiie has value close the optimal one, encouraged to exploit samples with well-above-optiie values in exploratory will be discarded. Conversely, we are heavily encouraged to explore the current best observation holds value that believe far from optimal. On Hartmann (6D), can see this behavior at play. Initial performance poorer for Col aBC than their respective baselines, presumably due to above-average exploration, but terminal performance is better.
trade-off; the prior suggests that the incumbent
C.3 BATCH EVALUATIONS
We evaluate ColaBO batch evaluations, utilizing the sequential greedy technique for MC ac- quisition functions from Wilson et al. (2018). Drop-off from sequential to batch evaluations is not evident from the plots, ordering between sequential and batch varies with the benchmark. While unpredictable, we speculate that the altered apploration-expliiiii acquisition function is occasionally beneficial in the presence of auxilliary beliefs Tay.
trade-off provided by the batched
-33
ColaBo
Dolelt-- ngiti q -3
ColaBO-MEB q=3
-20
Pecostio
Iecaiico
Figure 11: q = (sequential) and g 3 batch evaluation on a subset of synthetic functions with well-located priors for ColaBC-Loge and ColaBO-MES. Total function evaluations plotted for both scquential and batched variants, leaving them with the same number of function evaluations.
18


Reviews:
[{"review_id": "XGFEZrirR3", "reviewer": "Reviewer_tGCg", "summary": "The paper introduces ColaBO, which allows domain experts to customize the BO optimization process by integrating prior beliefs, such as information about the probable location of the optimum or the optimal value. Through empirical experiments, it shows that ColaBO speeds up optimization when prior information is accurate and maintains reasonable performance even when the prior knowledge is misleading.", "questions": "Besides the likely location of the optimizer or the optimal value, what other forms of prior knowledge would generally be useful?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The framework's adaptability and flexibility to incorporate prior knowledge into the optimization process. The method maintains reasonable performance even when the prior knowledge is misleading, demonstrating its robustness in different scenarios.", "weaknesses": "Test functions used to evaluate the proposed framework were quite limited, only a restricted set of test functions was employed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pSv5QxXVI6", "reviewer": "Reviewer_ZcGu", "summary": "The paper considers a setting where BO is applied to tackle black-box optimization problems where good prior knowledge is available. However, the conventional GP prior might fall short in effectively incorporating this knowledge. Therefore, the authors propose a new approach to inject it, mainly based on reweighing the prior of the GP with the user-defined prior and deterministic update of the GP posterior. Empirically, two instances of the framework is tested in synthetic and hyperparameter tuning tasks.", "questions": "1.\tWhat is the difference between $f^*$ and $f_*$? What is $x_*$?\n2.\tIs there any reason the authors in favor of rejection sampling compared to other sampling methods? What is the efficiency of using rejection sampling here?\n3.\tHow $\\beta$ is determined? \n4.\tIn Figure 5, even though ColaBO-MES is injected with a poorly located priors, it still performs better than MES, how do the authors interpret that?\n5.\tThe authors show in Eq. 5 that $p(f|D, \\pi) \\propto \\pi(f)p(f|D) $. Does this proportionality still hold in Eq. 6? \n6.\tTo indicate where the optimum is located within the prior, couldn’t we achieve it by simply defining the prior mean function in GP properly? For instance, assigning higher values to points where the users believe to be good and low values to the rest.\n7.\tWhy do the authors consider log EI instead of EI?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper considers a novel black-box optimization setting where good prior knowledge exists and proposes a framework to handle this problem.\n2.\tBased on sampling, this framework is compatible with all Monte Carlo acquisition functions.\n3.\tWith acquisitions to be Log Expected Improvement and Max-Value Entropy Search, the proposed models work well in synthetic and hyperparameter tuning tasks when well-located prior to the optimal location is available, while the drop in performance is not obvious compared to the benchmark models.", "weaknesses": "1.\tAlthough the empirical performance of ColaBO looks promising in the synthetic task and hyperparameter tuning task, the theory developed in the work is limited. Therefore, how the method performs statistically is a concern, given that there are many approximations, such as the Monte Carlo acquisition and the RFF sampling.\n2.\tThe empirical section can be further enhanced by testing the algorithms on more challenging tasks, for example, higher-dimensional problems. The paper primarily focuses on the low-dimensional tasks, which might not provide a strong basis for demonstrating the algorithm’s performance in realistic problems, where the dimensions might be much higher.\n3.\tThe presentation of the work can be improved. Initially, the authors define $\\pi(\\cdot)$ in Eq. 3 with input from $\\mathcal{X}$, but it later becomes $\\pi(f)$. Moreover, the notations become messier after Eq. 4, making it more challenging to follow. Also, while rejection sampling seems to be an important step in the model, the way it works is not introduced in the paper. Similar for RFF.\n4.\tGiven the authors’ claim that this is a general framework for BO to incorporate prior, the paper should present concrete examples to demonstrate its generality. For example, the authors could show how the examples in the second paragraph of the introduction section can be solved effectively within the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iNnosw6lrq", "reviewer": "Reviewer_ysEZ", "summary": "This paper proposes a new approach to user (or prior) guided Bayesian optimization. Unlike previous approaches where the acquisition function was modified to incorporate priors, the paper proposes to sample from a modified posterior of the Gaussian process. This is achieved by combining rejection sampling with the recent line of work on path-wise conditioning.", "questions": "* Given that the authors rely on GP posterior sampling, an obvious thing that could have been tried is Thompson sampling. Have the authors tried to include it?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed technique is original and interesting. It certainly appears more principled than previous approaches to incorporating expert knowledge.", "weaknesses": "* The mathematical derivation is too informal in some places, affecting clarity. I believe this needs to be improved. For instance, $\\pi$ here represents a belief over functions. Then why is Eq. (4) a function that receives a point $x \\in \\mathcal{X}$? Shouldn't it be a probability distribution over functions? Also, Def 3.1 introduces a conditioning on $\\pi$. I found this quite confusing. Is $\\pi$ a density? a function? or a random variable?\n* There are concerns about the fairness of the experiments. The authors state that \"ColaBO and πBO are initialized with the mode of the prior followed by 2 Sobol samples, whereas LogEI and MES are conventionally initialized with D + 1 Sobol samples.\" Shouldn't logEI and MES also have been initialized on the mode of the prior? If we assume that such prior information is present in user-guided BO methods, it is fair to assume that conventional BO methods also have access to this information. \n* The evaluations are okay but not extensive (especially compared to the piBO paper). I think this is an important point, given that the utility of user-guided BO methods can only be judged empirically on a case-study basis. This connects with my next concern.\n* The paper lists a couple of different types of function priors in Section 3.1 but only seems to evaluate one type of such prior. It is, therefore, hard to judge how general this framework is for incorporating prior knowledge. Especially since the users state in the contributions that one can \"incorporate arbitrary user knowledge.\" \n\n### Major Comments\n* The contribution statement is too general except for item 2. For instance, 1 and 3 can be applied to any user-guided BO method. I expect some more technical details about what this method offers to the field of user-guided BO.\n* The derivation of importance sampling in Eq. (10) is unclear. $\\pi(f) p(f | D)$ is an unnormalized density, but the integral is taken over it. Then, the equality with Eq. (9), doesn't hold since the expectation is not normalized.\n* In Section 3.3, the authors introduce a tempering scheme based on the number of datapoints. Furthermore, they draw connections with generalized Bayesian inference (GBI). I think the authors should make it clear that the connection is very loose. A major characteristic of GBI is to find a statistically principled way to come up with the temperature, which is not the case here. It is particularly misleading as, in the last sentence of Section 3.3, the paper states that tempering is done so in a \"principled Bayesian manner.\" \n\n### Minor Comments\n* Couldn't the name collaborative Bayesian optimization be misleading as to making people to think this method involves multiple users?\n* Section 1 first sentence: Please cite classic papers that introduced BO for historical context.\n* Figure 9-12: The boundary of the error bands makes it hard to distinguish between the plots. Please consider improving the visibility here.\n* Section 2.1 equation: Generally, we aim to minimize the expectation of f or the regret. Saying that we minimize f, in the presence of noise, is too informal.\n* Section 2.2 second from the last sentence: I think replacing \"standard method\" with \"classic method\" would be better here since we have an influx of more efficient methods.\n* Eq (2): Has the \"equivalent in distribution\" sign been defined anywhere?\n* Section 3.1 second paragraph second sentence: the sentence is incomplete.\n* References: please consider making the reference more consistent and check the entries. Carl et al 2022b has a typo: joint entropy eearch -> joint entropy search; Jones et al. 2018 is missing the journal entry; Kingma and Welling 2013 was published in ICLR’14.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IV0RvG12xb", "reviewer": "Reviewer_TtXD", "summary": "This submission provides a unifying framework for encompassing user beliefs in Bayesian Optimization (BO) beyond the usual priors on kernel  hyperparameters. Previous approaches offered to augment BO with expert beliefs $\\pi$, like optimum value or location, but mostly focused on doing so at the acquisition function level. Here, the authors propose to integrate this at the surrogate level. Instead of the classical Gaussian Process prior $p(f)$, they introduce a user belief over functions $\\pi(f) \\propto \\frac{p(f|\\pi)}{p(f)}$. Thus, $p(f|\\pi)$ can be obtained by reweighting samples from $p(f)$ proportionally to their probability of occurring under $\\pi(f)$. \n\nAs the user belief is assumed independent of the data-generating mechanism, the resulting posterior $p(f|\\mathcal{D},\\pi)$ is naturally proportional to the user belief $\\pi(f)$ and the likelihood $p(f|\\mathcal{D})$. It is therefore non Gaussian for nontrivial user beliefs, an issue circumvented by the authors using a decoupled sampling scheme. Likewise, classical acquisition functions like Expected Improvement or Maximum Entropy Search are not tractable anymore, which led the authors to leverage and tailor their Monte-Carlo version to this specific case. \n\nIn the end, the proposed method, *ColaBO*, is then evaluated on a range of synthetic and real-world benchmarks, and demonstrates convincing performances against its competitors, particularly for misleading user beliefs.\nAs these benchmarks contain hyperparameter tuning of Deep Learning models, I believe this submission completely falls into the scope of ICLR.", "questions": "I do not have questions.\n\nTypos or similar:\n\n- Can you clarify what \"DoE\" means in \"[...] we consider a smaller optimization budget of $10D$ iterations, and initialize all methods that utilize user beliefs with only one DoE sample, that being the mode of the prior\".\n- Figure 6: The y-axis gives \"Accuracy\" but performances are decreasing over the BO trial. It probably should be something like 1-accuracy?\n- A.3: \"by using a sampling an offset direction $\\boldsymbol{\\epsilon}$ - > \"by sampling an offset direction $\\boldsymbol{\\epsilon}$\"?\n- Concusion: \"[...] or pre-trainedp\" -> pre-trained", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- To the best of my knowledge, this is the first attempt to integrate user prior beliefs directly at the level of the surrogate rather than at the acquisition function level. The approach is novel and encompasses multiple ways for the user to incorporate its expertise: knowledge of function optimum value, optimum location, and preferences.\n- I like Figures 1 and 2, they nicely illustrate the benefits of incorporating user beliefs and how this impacts the GP posterior and the acquisition function landscape.", "weaknesses": "I cannot think of any salient weakness in this work. *ColaBO* relies on several approximations due to non-Gaussianity of the posterior and these can probably be made more efficient, as mentioned in the limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)