Approximation-Aware Bayesian Optimization
Authors: Natalie Maus
Kyurae Kim
David Eriksson
Geoff Pleiss
John Patrick Cunningham
Jacob R. Gardner
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Bayesian optimization
variational inference
Gaussian processes
utility maximization
expected improvement
knowledge gradient
black-box optimization
Primary area: probabilistic_methods

Abstract: High-dimensional Bayesian optimization (BO) tasks such as molecular design often require $>10,$$000$ function evaluations before obtaining meaningful results. While methods like sparse variational Gaussian processes (SVGPs) reduce computational requirements in these settings, the underlying approximations result in suboptimal data acquisitions that slow the progress of optimization. In this paper we modify SVGPs to better align with the goals of BO: targeting informed data acquisition over global posterior fidelity. Using the framework of utility-calibrated variational inference (Lacoste–Julien et al., 2011), we unify GP approximation and data acquisition into a joint optimization problem, thereby ensuring optimal decisions under a limited computational budget. Our approach can be used with any decision-theoretic acquisition function and is readily compatible with trust region methods like TuRBO (Eriksson et al., 2019). We derive efficient joint objectives for the expected improvement (EI) and knowledge gradient (KG) acquisition functions in both the standard and batch BO settings. On a variety of recent high dimensional benchmark tasks in control and molecular design, our approach significantly outperforms standard SVGPs and is capable of achieving comparable rewards with up to $10\times$ fewer function evaluations.

Approximation-Awar Bayesian Optimization
Natalie Maus
Kyurae Kim University of Pennsylvania
Geoff Pleiss
University of Pennsylvania nmaus @ wew..upenn.......
University of British Columbia Vector Institute
David Eriksson Meta
John P. Cunningham Columbia University
Jacob R. Gardner University of Pennsylvania
Abstract
High-dimensional Bayesian optimization (BO) tasks such as molecular design often require > >10,000 function evaluations before obtaining meaningful results. While methods like sparse variational Gaussian processes (SVGPs) reduce computational requirements in these settings, the underlying approximations result in suboptimal data acquisitions that slow the progress of optimization. In this paper we modify SVGPs to better align with the goals of BO: targeting informed data acquisition rather than global posterior fidelity. Using the framework of utility-calibrated variational inference, we unify GP approximation and data acquisition into a joint optimization problem, thereby ensuring optimal decisions under a limited computational budget. Our approach can be used with any decision-theores is readily compatible with trust region methods like TuRBO. We derive efficient joint objectives for the expected improvement and knowledge gradient acquisition functions for standard and batch BO. Our approach outperforms standard SVGPs on high-dimensional benchmark tasks in control and molecular design.
acquisition function and
1 Introduction
Bayesian optimization (BO; Frazier, 2018; Garnett, 2023; Jones et al., 1998; Mockus, 1982; Shahriari al., 2015) casts optimization sequential decision-make problem. Many recent successes of BO have involved complex and high-dimensional problems. In contrast to "classic" low-dimensional BO problems-wher expensive black-box function evaluations far exceeded computational costs these modern problems necessitate tens of thousands of function evaluations, and it often the complexity and dimensionality of the search space that makes optimization challenging, rather than limited evaluation budget (Eriksson et al., 2019; Griffiths and Hernández-Lobato, 2020; Maus et al., 2022, 2023; Stanton et al., 2022). Because of these scenarios, BO is entering a regime where computational costs are becoming primary bottleneck (Maddox et al., 2021; Maus et al., 2023; Moss et al., 2023; Vakili et al., 2021), as the Gaussian process (GP; Rasmussen and Williams, 2005) surrogate models that underpin most of Bayesian optimization scale cubically with the number of observations.
In this new regime, we require scalable GP approximations, an that has made tremendous progress over the last decade. In particular, sparse variational Gaussian processes (SVGP; Hensman et al., 2013; Quiñonero-CCi Candela and Rasmussen, 2005; Titsias, 2009) have use (Griffiths and Honnänddz-Lobatte 2020; Maddox et al., 2021; Maus et al., 2022, 2023; Stanton al., 2022; Tripp al., 2020; Vakili et al., 2021), but many challenges remain effectively deploy SVGPs for budget BO. In particular, the standard SVGP training objective is not aligned with the goals of black-box optimization. SVGPs construct an inducing point approximation that maximizes the standard variational evidence lower bound (ELBO; Jordan et al., 1999), yielding posterior approximation q* (f) that models all observed data (Matthews al., 2016; Moss al., 2023). However, the optimal posterior approximation q* is suboptimal for the decision -aaioonn-aainggg tasks involved
increase in
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

in BO (Lacoste-Juile et al., 2011). In BO, we do not care about posterior fidelity at the majority of prior observations; rather, we only care about the fidelity of downstream functions involving the posterior, such as the expected utility. To illustrate this point intuitively, consider using the common expected improvement (EI; Jones et al., 1998) acquisition function for selecting new observations. Maximizing the ELBO might result in posterior approximation that maintains fidelity for training examples in regions of virtually zero EI, thus wasting "approximation budget."
To solve this problem, we focus on the deep connections between statistical decision theory (Robert, 2001; Wasserman, 2013, $12) and Bayesian optimization (Garnett, 2023, $6-7), where acquisition maximization can be viewed as maximizing posterior-expected utility. Following this perspective, we leverage the utility-calibratee approximate inference framework (Jaiswal et al., 2020, 2023; Lacoste-Julien et al., 201 1), and solve the aforementioned problem through variational bound (Blei et al., 2017; Jordan et al., 19ee)-the (log) expected utility lower bound (EULBO)- joint function of the decision (the BO query) and the posterior approximation (the SVGP). When optimized jointly, the EULBO automatically yields the approximately optimal decision through the minorize- maximize principle (Lange, 2016). The EULBO is reminiscent of the standard variational ELBO (Jordan et al., 1999), and can indeed be viewed as a standard ELBO for a generalized Bayesian inference problem (Bissiri et al., 2016; Knoblauch et al., 2022), where we seek to approximate the urility-weighted posterior. This work represents the first application of utility-calibrated approximate inference towards BO despite its inherent connection with utility maximization.
The benefits of our proposed approach are visualized in Fig. 1. Furthermore, it can be applied to acquisition function that admits a decision-theoretio interpretation, which includes the popular expected improvement (EI; Jones et al., 1998) and knowledge gradient (KG; Wu et al., 2017) acquisition functions, and is trivially compatible with local optimization techniques like TuRBO (Eriksson et al., 2019) for high-dimensional problems. We demonstrate that our joint SVGP/acquisition optimization approach yields significant improvements across numerous Bayesian optimization benchmarks. As an added benefit, our approach can simplify the implementation and reduce the computational burden of complex (decision-theoretti)) acquisition functions like KG. We demonstrate a novel algorithm derived from our joint optimization approach for computing and optimizing the KG that expands recent work on one-shot KG (Balandat et al., 2020) and variational GP posterior refinement (Maddox et al., 2021).
Overall, our contributions are summarized as follows:
We propose utility-calibrated variational inference of SVGPs in the context of large-budget BO.
We study this framework in two special cases using the utility functions of two common acquisition functions: EI and KG. For each, we derive tractable EULBO expressions that can be optimized.
For KG, we demonstrate that the computation of the EULBO takes only negligible additional work over computing the standard ELBO by leveraging an online variational update. Thus, as a byproduct of optimizing the EULBO, optimizing KG becomes comparable to the cost of the EI.
We extend this framework to be capable of running in batch mode, by introducing q-EULBO analogs of K- and q-EI as commonly used in practice (Wilson et al., 2018).
We demonstrate the effectiveness of our proposed method against standard SVGPs trained with ELBO maximization on nigh-dimensional benchmark tasks in control and molecular design, where the dimensionality and evaluation budget go up to 256 and 80k, respectively.
2 Background
Noisy Blx Optimization. Noisy black-box optimization refers to problems of the form: maximizexex X F (x), where X c Rd is some compact domain, F X - y is some objective function, and we assume that only zeroth-order information of F is available. More formally, for some i Noo- we assume that observations of the objective function (x, = f (x;)) have been corrupted by independently and identically distributed (i.i.d)) Gaussian noise f (xx) & F(x)) e, where c ~ (O, The noise variance o is also unknown.
Bayesian optimization. Bayesian Optimization (BO) is and iterative approach to noisy black- box optimization that iterates the following steps: At each step 0. we use a set of observations D. {xx,>} = (x,))) of f to fit surrogate supervised model f E J. Typically, J is taken to be the sample space of a Gaussian process such that the function- valued posterior distribution

---

ELBO fit (m = 4)
ELBO
EULBO (Ours)
Data
True function
Inducing points
GP mean
(approx)
El (exact)
Figure 1: (Left.) Fitting SVGP model with only 4 inducing points sacrifices modeling areas of high EI (few data points at right) because the ELBBO focuses only on global data approximation (left data) and is ignorant of the downstream decision making task. (Middle.) Because of this, (normalized) EI with the SVGP model peaks in an incorrect location relative to the exact posterior. (Right.) Updating the GP fit and selecting a candidate jointly using the EULBO (our method) results in candidate selection much closer to the exact model.
(f] 2) forms distribution over surrogate models step 4. The posterior is then used to (D) form a decision problem where we choose which point we should evaluate next, x,+1 = 6₂ (2)). by maximizing an acquisition function a IR
8 (2,) _ arg max a (x;
xET
K After selecting x,+1. evaluated to obtain the new datapoint (X(+1,)(41 = (x)+1)). This is then added the dataset, forming Di+1 D, U (x++1))(41) to be used in the next iteration.
Utility-Bass BBsee Acquisition Functions. Many commonly used acquisition functions, including El and KG, can be expressed as posterior-expected utility functions
< (x; (x. (f 2)
where u (x. (: XX R some utility function associated with a (Garnett, 2023, $6-7). In statistical decision theory, posterior-expected utility maximization policies such as 5ạ are known as Bayes policies. These are important because, for a given utility function, they attain certain notions of statistical optimality such Bayes optimality and admissibility (Robert, 2001, $2.4; Wasserman, 2013, $12). However, this only holds true if we can exactly compute Eq. (2) over the posterior. Once approximate inference is involved, making optimal Bayes decisions becomes challenging.
Sparse Variational Gaussian Processes. While the Onr)) complexity of exact Gaussian process model selection and inference is not necessarily a roadblock in the traditional regression setting with 10,000-50,000 training examples, BO amplifies the scalability challenge by requiring us to sequentially train or update many large scale GPs as we iteratively acquire more data.
To address this, sparse variational GPs (SVGP; Hensman et al., 2013; Titsias, 2009) have become commonly used in high- theouhhrou Bayesian optimization. SVGPs modify the original GP prior from p(f) to pff p1)( where assume the latent function f is "induced" by finite set of inducing values u = (u.,,.. um) E Rt located at inducing points 2 E X for i = 1,... m. Inference done through variational inference (Blei et al., 2017; Jordan al., 1999), where the posterior of the inducing points is approximated using Qı (u) = N (u;2 = (m, S)) and that of the latent functions with q (f u) p (f u). Here, the variational parameters standard practice to define 2 = (m, S) so that 2 can be used shorthand to represent all of the trainable variational parameters. As is typical in the BO literature, we use the subscript 2 E A to denote that the distribution denoted as q contains trainable parameters in d.
and S defined the learned mean
and covariance of the variational distribution qa (w).
For positive definite kernel function 3' x + Ryo, the resulting ELBO objective, which can be computed in closed form (Hensman al., 2013), is then
L ELBO 2,) - E Fg)
Σi log Ev: If (xx)) DKL (qa (4), (u)),
where E(y; | f (x))) = N (y; If (x;), o.) is a Gaussian likelihood. The marginal variational approximation can be computed as
q2(f)= = q2(f u) du f p(f u) (u) du
3

---

such that the point-wise function evaluation on some x E 2' is
9x(f(x)) =N (f(x); Uy(x) Kaz zz m,
of (x) (x) =xx + xx>22zz z2 SK 22xxxx
(4) (zz x), and the
with Exx - -xx2 2Z. Zx the vector Kzx E Rm is formed as [kzxli matrix Kzz E Rmmm is formed as [Zzzlj = Zi). Additionally, the GP likelihood and kernel contain hyperparameters, which we denote as E θ, and we collectively denote the set of inducing point locations Z (Z1.-.,Zm) E _ We therefore denote the ELBO EELBO (2, Z, 0: 2)).
XxzKzzk2
3 Approximation-Aware Bayesian Optimization
When SVGPs are used conjunction with BO (Maddox et al., 2021; Moss et al., 2023) iteration I 0, acquisition functions of the form of Eq. (2) are naively approximated
(x; f (: (f)
where q2( is the approximate SVGP posterior given by Eq. (4). The acquisition policy implied by this approximation contains two separate optimization problems:
f (xx A
x = arg max XET
Of and ELBO
arg max ELBO 2,). (5) AEA
Treating these optimization problems separately creates an artificial bottleneck that results in suboptimal data acquisition decisions. Intuitively, LEL is chosen to faithfully model all observed data (Matthews et al., 2016; Moss et al., 2023), without regard for how the resulting model performs at selecting the next function evaluation in the BO loop. For illustration of this, Figure 1. Instead, we propose a modification to SVGPs that couples the posterior approximation and data acquisition through a joint problem of the form:
(x(+1, 2²) arg max RULLO (2, D,) ΛEλXET
This results x(+1 directly approximating a solution to Eq. (2), where the expected utility lower- bound (EULBO) is an ELBO-like objective function derived below.
3.1 Expected Utility Lower-Bound
Consider an acquisition function of the form of Eq. (2), where the utility u X Rx is strictly positive. We can derive a similar variational formulation of the acquisition function maximization problem following Lacoste-Julien et al. (2011). That is, given any distribution 92 indexed by A e A and considering the SVGP prior augmentation p(f) - pff u)p(u), the acquisition function can be lower-bounde through Jensen's inequality
log (x; :)) log
u (x, f;: df
u (x, f: (,) (f.u 21) 4a (f. u)
log
df du
q2 (f,
LA (u) p (f
u)
log
u (x. (xx f; e (D) pf (u)
df du - log
DA (u) p (f u))
[xx (,) (D, p
log
p pf u) (u) df du - duglog
(7)
91 (1))
where Z is a normalizing constant. restriction on u comes from the inequality in Eq. (7), where the utility needs to be strictly positive. This means that non-strictly positive utilities need to be modified to be incorporated into this framework. (See the examples by Kušmierczyk et al., 2019.) Also, notice that the derivation is reminiscent of expectation-maxine maximization (Dempster et al., 1977) and variational lower bounds (Jordan et al., 1999). That is, through the minorize-maximize principle (Lange, 2016), maximizing the lower bound with respect to x and 2 approximately solves the original problem of maximizing the posterior-expected utility.

---

Expected Utility Lower-Bound. Up to a constant and rearranging terms, maximizing Eq. (7) is equivalent to maximizing
EULBO (,) _ E ( [log (2) ( + log - log qı (u) +
= =x(() - log ((y; [) DKL (qA (4), p(u)) + (r) log (x, 22) LELBO (A; 2,) 200 log (x)
(8)
which is the joint objective function alluded to in Eq. (6). We maximize EULBO to obtain (x1+1, EULLO (x, 2). where x1+1 corresponds our next BO "query".
arg max XEX TEXAEA AEA
From Eq. (8), the connection between the EULBO and ELBO is obvious: the EULBO is now "nudging" the ELBO solution toward high utility regions. An alternative perspective is that we are approximating generalized posterior weighted by the utility (Table. 1 by Knoblauch al., 2022; Bissiri al., 2016). Furthermore, Jaiswal et al. (2020, 2023) prove that the resulting actions satisfy consistency guarantees under assumptions typical in such results for variational inference (Wang and Blei, 2019).
Hyperparameters and Inducing Point Locations. For the hyperparameters @ and inducing point locations Z, we use the marginal likelihood to perform model selection, which is common practice in BO (Shahriari al., 2015, SV.A). (Optimizing over Z was popularized by Snelson and Ghahramani, 2005.) Following suit, we also optimize the EULBO as a function of 0 and Z as
maximize { CEULBO (2, x, 0, D;) - 2x,00Z
LELOO (2, Z,0; + E qa(
We emphasize here that the SVGP-associated parameters 2, have gradients that are determined by both terms above. Thus, the expected log-utility term Ef~91() log (x, fx 1D) simultaneously results in acquisition of x and directly influences the underlying SVGP regression model.
3.2 EULBO for Expected Improvement (EI)
The EI acquisition function can be expressed as a posterior-expected utility, where the underlying "improvement" utility function given by the difference between the objective value of the query, fxx, and the current best objective value y," = =a.. 1 E D
MEI f:2)) A ReLU (f)x)
(x) - ,;,,
(EI; Jones et al., 1998)
where ReLU (x) max (x, 0). Unfortunately, this utility is not strietly positive whenever f (x) I y." Thus, we cannot immediately plug Mgi into the EULBO. While is possible to add small positive constant to ugi and make it strictly positive as done by Kušmierczyk et al. (2019), this results in looser Jensen gap Eq. (7), which could be detrimental. This also introduces the need for tuning the constant, which is not straightforward. Instead, we define the following "soft" EI utility:
UsEI (x, f: 2,) - softplus (f (x) (x))))").
where the ReLU in Eq. (9) is replaced with softplus (x) - log (1 exp(x)). softplus(x) converges to the ReLU in both extremes of x + to0. Thus, USEI will behave closely to ugr. while being slightly
more explorative due positivity.
Computing the EULBO and its derivatives now requires the computation 5 log UsEa fx D,), which, unlike EI, does not have a closed-form. However, since the utility function only depends on the function values of f, the expectation can be efficiently computed high precision through one-dimensional Gauss-Hermite quadrature. Crucially. the expensive K=- m and SK solves that dominate both the asymptotic and practical rumning time of both the ELBO and the EULBO are fixed across the log utility evaluations needed by quadrature. Because quadrature only depends these precomputed moments, the additional work necessary due to lacking a closed form solution is negligible: Gauss-Hermite quadrature converges extremely quickly in the number of quadrature sites, and only requires on the order of 10 or so of these post-solve evaluations to achieve near machine precision.
3.3 EULBO for Knowledge Gradient (KG)
Although non-trivial, the KG acquisition is also a posterior-expected utility, where the underlying utility function is given by the maximum predictive mean value any in the input domain after
5

---

conditioning on new observation (x.y) E X' x y: max EE [f(x' D, u y)]
Ugg (x. D
(KG; Frazier, 2009; Garnett, 2023)
Fet
Note that the utility function as defined above not non- neg-neeativ:: the maximum predictive mean of Gaussian process can be negative. For this reason, the utility function is commonly (and originally, egg. Frazier, 2009, Eq. 11) written the literature as the difference between the new maximum mean after conditioning on (x,y) and the maximum mean beforehand:
UgG yx D - max X't
[f((') Dr U ((x,))))) uT,
where u² max E [f(") D. Note that u plays the role of a simple constant as it depends on neither x nor y. Similarly to the acquisition, this utility is still not strictly positive, and we thus define its "softplus-ed" variant:
USKG (x) - softplus (ukG (x.
Here, c* acts as jrr" by making Ugg positive as often as possible. This is particularly important when consequence of the objective values being negative. One
the GP predictive mean is negative natural choice of constant is using 14: however, we find that simply choosing c* y, works well and is more computationally efficient. Here, y is the highest value of y (the highest objective value observed so far).
One-Shot KG EULBO. The EULBO using Usko results in an expensive nested optimization problem. To address this, we use an approach similar to the one-shot knowledge gradient method of Balandat al. (2020). For clarity, we will define the reparameterization function
Ya (x) Ma, (x) g. (x) Ej.
where, for an i..dd sample e . (0, 1), computing y; ya (x, e,) is equivalent to sampling yi ~ N (M (x), G (x)). This enables the use of the reparameterization gradient estimator (Kingma and Welling, 2014; Rezende et al., 2014; Titsias and Gredilll,,,,, 2014). Now, notice that the KG acquisition function can be approximated through Monte Carlo as
axo(x;;) 2)
UKo y2 (x; 5
Σ max E [f(x') D. U yx
where, for i = 1,...,,, & N' 1) are i.i.d. The one-shot KG approach absorbs the nested optimization over x' into simultaneous joint optimization over x and mean maximizer for each of the S samples, *'. such that maxy axo(x; D,) x max maxx,((pmn.. (D) where
xx D1)
Σ 1=1
(x:e;;;; 2,)
1 Σ [f(()) 1=1
1 yxx,2 (x;e;))
Evidently, there no longer an inner optimization problem over x'. estimate the ith term of this sum, we draw sample of the objective value of x, (2(x;;; E.), and condition the model this sample. We then compute the new posterior predictive mean at X After summing, we compute gradients with respect to both the candidate x and the mean maximizers x'I....''' X. Again, we use the "soft" version of one-shot KG in our EULBO optimization problem:
UJ-SKG (x, y; D,) softplus (E [f(x)) ((x,)) -ct),
where this utility function is crucially a function both x and a free parameter x'. As with jj.KG- maximizing the EULBO can be set up joint optimization problem:
maximize A.z.
(2, (,, Σ
log uKG xx;
(10)
The computation time of the non-ELBO term Eq. (10) is dominated by having to compute E [[f(x() D, (xx (x) S-times. Notice that we only need to compute an updated posterior predictive mean, and can ignore predictive variances. For this, we can leverage the online updating strategy of Maddox et al. (2021). In particular, the predictive mean can be updated in (m=)) time using simple Cholesky update. The additional OSm²² cost of computing the EULBO is therefore amortized by the original O(m²) cost of computing the ELBO.
Efficient KG-EULBO Computation.

---

3.4 Extension to Q-EULBO for Batch Bayesian Optimization
The EULBO can be extended to support batch Bayesian optimization by using the Monte Carlo batch mode analogs of utility functions as discussed e.g. by Balandat et al. (2020); Wilson et al. (2018). Given a set of candidates
(x),,.,,))) 229 the q-EI utility function is given by:
Mq-EI (: D,)
max ReLU (f (xx) yt)
(q qq-El; Balandat et al., 2020; Wilson et al., 2018)
j=1..q
This utility can again be softened as:
Mg-SEI (X, (X. DD) _ max softplus (f );;
j=l..q
Because this is now q-dimensional integral, Gauss-Hermite quadrature is no longer applicable. However, we can apply Monte Carlo as
1 Σ max softplus (y2 (x; - ), j=1...
Eaxc) log u (X. 21)
S
As done in the BoTorch software package (Balandat al., 2020), we observe that fixing the set of base samples Eg. during each BO iteration results in better optimization performance at the cost of negligible q Q-EULBO bias. Now, optimizing the q-EULBO done over the full set of q candidates (xI,... jointly, as well the GP hyperparameters, inducing points, and variational parameters.
Knowledge Gradient. The KG version of the EULBO can be similarly extended. The expected log utility term in the maximization problem Eq. (10) becomes:
1 Σ max log (xgu (x;&));;,) ELBO (2, 2)
maximize
xi X x 22000
j=L.q
resulting in a similar analog to q- KG as described by Balandat et al. (2020).
3.5 Optimizing the EULBO
Optimizing the ELBO for SVGPs is known to be challenging (Galy-Fajou and Opper, 2021; Terenin et al., 2024) as the optimization landscape for the inducing points is non-convex, multi-modal, and non-smooth. Naturally, these are also challenges for EULBO; we found that care must be taken when implementing and initializing the EULBO maximization problem. In this subsection, we outline some key ideas, while detailed description with pseudocode is presented in Appendix A.
Initialization and Warm-Starting. We warm-start the EULBO maximization procedure by solving the conventional two-step scheme in Eq. (5): At each BO iteration, we obtain the "warm" initial values for (A, Z, 9) by optimizing the standard ELBO. Then, use this maximize the conventional acquisition function corresponding to the chosen utility function u (the expectation of u over (2),,, which provides the warm-start initialization for
Alternating Maximization Scheme. To optimize LEULBO (x, A, Z, 9), we alternate between opti- mizing over the query x and the SVGP parameters 2, 8. We find this block-coordinate descent scheme to be more stable and robust than jointly updating all parameters, though the reason why this is more stable than jointly optimizing all parameters requires further investigation.
4 Experiments
We evaluate EULBO- based SVGPs on a number of benchmark BO tasks, described in detail in Section 4.1. These tasks include standard low-dimensional BO problems, e.g., the 6D Hartmann function, as well as 7 high-dimensional and high-througghee optimization tasks.
Baselines. We compare EULBO to several baselines with the main goal of achieving high reward using as few function evaluations as possible. Our primary point of comparison is ELBO-based SVGPs. We consider two approaches for inducing point locations: 1. optimizing inducing point locations via the BLBOO (denoted as ELBBO, 2. placing the inducing points using the strategy proposed by Moss et al. (2023) at each stage of ELBO optimization (denoted as Moss et al.). The latter offers improved BO performance over standard ELBO-SVGP in BO settings, unlike our method it exclusively
7

---

335
0.0
-311
380
315
Mean Roward
0.1
Oeteenees M470
Foofrradie MPO
Nodian Millaci
Modiinnn Molanden
4.70
6.90
<9t
0.0
6.0
03
2800
Number of Oracle Calls
Tati3 (2UL
TuBtD
(11..00
(Marcctaa
Trati1 102.1011300 KCO) Te2.00 (Mox
(10120
00:01.011
[D lib
Figure 2: Optimization results on the considered tasks. We compare all methods for both standard BO and TuRBO-based BO (on all tasks except Hartmann). Each line/shaded region represents the mean/standard error over 20 runs See subsection B.1 for additional molecule results.
targets inducing point placement and does not affect variational parameters hyperparameters of the model. In addition, we compare to BO using exact GPs using 00000 function evaluations as the use of exact GP is intractable beyond this point due the need to repeatedly fit models.
Acquisition Functions and BO algorithms. For EULBO, we test the versions based on both the Expected Improvement (EI) and Knowledge Gradient (KG) acquisition functions as well as the batch variant. We test the baseline methods using EI only. On high-dimensional tasks (tasks with dimensionality above 10), run EULBO and baseline methods with standard BO and with trust region Bayesian optimization (TuRBO) (Eriksson et al., 2019). For the largest tasks (Lasso, Molecules) we use acquisition batch size of 20 (q 20), and batch size 1 (q 1) for all others.
Implementation Details and Hyperparameters. Code to reproduce all results in the paper available attps://gis thub. www.comm.aa../cccc http//aaaalmmauaaaaaaaaaa amaus/ We implement EULBO and baseline methods using the GPyTorch (Gardner et al., 2018) and BoTorch (Balandat et al., 2020) packages. For all methods, we initialize using set of 100 data points sampled uniformly at random in the search space. We use the same trust region hyperparameters as in (Eriksson et al., 2019). In Appendix B.1, we also evaluate an additional initialization strategy for the molecular design tasks. This alternative initialization matches prior work in using 10, 000 molecules from the GuacaMol dataset Brown et al. (2019) rather than the details we used above for consistency across tasks, but does achieve higher overall performance.
4.1 Tasks
Hartmann 6D. The widely used Hartmann benchmark function (Surjanovic and Bingham, 2013).
Lunar Lander. The goal of this task is to find an optimal 12-dimensional control policy that allows an autonomous lunar lander to consistently land without crashing. The final objective value we optimize the reward obtained by the policy averaged over a of 50 random landing terrains. For this task, we use the same controller setup used by Eriksson et al. (2019).
The rover trajectory optimization task introduced by Wang et al. (2018) consists of finding 60-dimensional policy that allows rover to move along some trajectory while avoiding set of obstacles. We use the same obstacle set up as in Maus et al. (2023).
Rover.

---

-023
Mean Reward
-0.20
0033
0.00
-0480
0423
Number Orucle Calls
(all parometers))
TBRBO (BULBO (variational parometers only)
TURBO (JULBO EI (varattonal paremeters indaccnng poirra))
TvRBO (EL9O BD
Ablation study measuring the impact of EULBO optimization on various SVGP parameters. At each BO iteration, use the standard ELBO objective optimize the SVGP hyperparameters, variational parameters, and inducing point locations. We then refine some subset of these parameters by further optimizing them with respect to the EULBO objective.
Figure 3:
Lasso DNA. We optimize the 180-dimensional DNA task from the LassoBench library (Schić et al., 2022) of benchmarks based on weighted LASSO regression (Gasso et al., 2009).
Molecular design tasks (x4). We select four challenging tasks from the Guacamol benchmark suite of molecular design tasks (Brown et al., 2019): Osimertinib MPO, Fexofenadine MPO, Median Molecules I, and Median Molecules 2. We use the SELFIES-VAE introduced by Maus et al. (2022) to enable continuous 256 dimensional optimization.
4.2 Optimization Results
In Figure 2, we plot the reward of the best point found by the optimizer after given number of function evaluations. Error bars show the standard error of the mean over 20 replicate runs. EULBO with TuRBO outperforms the other baselines with TuRBO. Similarly, EULBO with standard BO outperforms the other standard BO baselines. One noteworthy observation is that neither acquisition function appears to consistently outperform the other. However, EULBO-SVGP almost always dominates ELBO-SVGP and often requires a small fraction of the number of oracle calls to achieve comparable performance. These results suggest that coupling data acquisition with approximate inference/model selection results in significantly more sample-efficient optimization.
4.3 Ablation Study
While the results in Fig. 2 demonstrate that EULBO-SVGP improves the BO performance it is not immediately clear to what extent joint optimization modifies the posterior approximation beyond what is obtained by standard ELBO optimization. To that end, in Fig. 3 we refine an ELBO-SVGP model with varying degrees of additional EULBO optimization. every BÔ iteration we begin by obtaining SVGP model (where the variational parameters, inducing point locations, and GP hyperparameters all obtained by optimizing the standard ELBO objective). We then refine some subset of parameters (either the inducing points, the variational parameters, the GP hyperparameters, all of the above) through additional optimization with respect to the EULBO objective. Interestingly, we find that tasks respond differently to the varying levels of EULBO refinement. In the case of Lasso DNA, there is not much a difference between EULBO refinement on all parameters versus refinement on the variational parameters alone. On the other hand, the performance Median Molecules 2 clearly dominated by refinement on all parameters. Nevertheless, we see that EULBO is always beneficial, whether applied to all parameters or some subset.
Related Work
Scaling Bayesian Optimization the Large-Budget Regime, BO has traditionally been confined to the amall-budges optimization regime with few hundred objective evaluations at most. However, recent interest in high-dimensional optimization problems has demonstrated the need scale BO to large data acquisition budgets. For problems with ~00 data acquisitions, Hernández-LLobat al. (2017); Snoek et al. 2); Springenberg et al. (2016) consider Bayesian neural networks (BNN; Neal, 1996), McIntire et al. (2016) use SVGP, and Wang et al. (2018) turn to ensembles of

---

subsampled GPs. For problems with > 10 acquisitions, SVGP has become the de facto approach alleviate computational complexity (Griffiths and Hernández-Lobato, 2020; Maus al., 2022, 2023; Stanton et al., 2022; Tripp et al., 2020; Vakili et al., 2021). As in this paper, many works have proposed modifications to SVGP to improve its performance in BO applications. Moss et al. (2023) proposed an inducing point placement based on a heuristic modification of determinantal point processes (Kulesza and Taskar, 2012), which we used for initialization, while Maddox et al. (2021) proposed method for fast online update strategy for SVGPs, which we utilize for the KG acquisition strategy.
Approximate Inference. The utility-calibraeed VI objective was first proposed by Lacoste-Julien et al. (2011), where they used coordinate ascent algorithm to maximize it. Since then, various extensions have been proposed: Kušmierczyk et al. (2019) leverage black-box variational inference (Ranganath et al., 2014; Titsias and LLázaro-Gredille
2014); Morais and Pillow (EP; Minka, 2001); Abbasnejad et al. (2015) and Rainforth et al. (2020) employ importance sampling: Cobb et al. (2018) and Li and Zhang (2023) derive a specific variant for BNNs; and (Wei et al., 2021) derive a specific variant for GP classification. Closest to our work is the GP-based recommendation model learning algorithm by Abbasnejad et al. (2013), which sparsifies an EP-based GP approximation by maximizing a utility similar to those used in BO.
(2022) use pnppectttioo-poopgattiiii
Limitations and Discussion
The main limitation of our proposed approach increased computational cost. While EULBO-SVGP still retains the O(m²) computational complexity of standard SVGP, our practical implementation requires warm-start: first fitting SVGP with the ELBO loss and then maximizing the acquisition function before jointly optimizing with the EULBO loss. Furthermore, EULBO optimization currently requires multiple tricks such as clipping and block-coordinate updates. future work, we aim to develop better understanding of the EULBO geometry in order to develop developing more stable, efficient, and easy-to-use EULBO optimization schemes. Nevertheless, our results Section demonstrate that the additional computation of EULBO yields substantial improvements in BO data- efficiency, a desirable trade-off in many applications. Moreover, EULBO-SVGP modular, and experiments capture fraction of its potential use. It can be applied to any decision -theorrtic acquisition function, and is likely compatible with non-standard Bayesian optimization problems such cost-constrained BO (Snoek et al., 2012), causal BO (Aglietti al., 2020), and many more.
More importantly, our paper highlights a new avenue for research in BO, where surrogate modeling, approximate inference, and data selection are jointly determined from unified objective. Extending this idea to GP approximations beyond SVGP and acquisition functions beyond EI/KG may yield further improvements, especially in the increasingly popular high-throughhut BO setting.
10

---

Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for suggestions that improved the quality of the work.
N. Maus was supported by the National Science Foundation Graduate Research Fellowship; K. Kim was supported by gift from AWS Al Penn Engineering's s ASSET Center for Trustworthy AI; G. Pleiss was supported by NSERC and the Canada CIFAR Al Chair program; J. P. Cunningham was supported by the Gatsby Charitable Foundation (GAT3708), the Simons Foundation (542963), the NSF AI Institute for Artificial and Natural Intelligence (ARNI: NSF DBI 2229929), and the Kavli Foundation; R. Gardner was supported by NSF awards IIS-2145644 and DBI-2400135.
References
Ehsan Abbasnejad, Justin Domke, and Scott Sanner. Loss-calibrated Monte Carlo action selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29 of AAAI. AAAI Press, March 2015. (page 10)
M. Ehsan Abbasnejad, Edwin V. Bonilla, and Scott Sanner. Decision-theoretio sparsification for Gaussian process preference learning. In Machine Learning and Knowledge Discovery in Databases, volume 13717 of LNCS. pages 515-530, Berlin, Heidelberg. 2013. Springer. (page 10)
Virginia Aglietti, Xiaoyu Lu, Andrei Paleyes, and Javier González. Causal Bayesian optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 108 of PMLR. pages 3155-3164. JMLR, June 2020. (page 10)
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A framework for efficient Monte-Carie Bayesian optimization. In Advances in Neural Information Processing Systems, volume 33, pages 11524-21533 Curran Associates, Inc., 2020. (pages 2, 6. 7, 16)
P. G. Bissiri, C. C. Holmes, and S. G. Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5): 103 103--130, 2016. (pages 2, 5)
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-88) April 2017. (pages 2, 3)
Nathan Brown, Marco Fiscato, Marwin H.S. Segler, and Alain C. Vaucher. Guacamol: Benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling, 59(3): 1096-1108, Mar 2019. (pages 8, 9)
Adam D. Cobb, Stephen J. Roberts, and Yarin Gal. Bayesian Neural Networks. arXiv Preprint arXiv: 10Xv:1805..0901, arXiv, May 2018. (page 10)
Approximate Inference in
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-25 September 1977. (page 4)
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing
Systems, volume 32, pages 5496-5507. Curran Associates, Inc., 2019, (pages 1, 2, 8)
Peter Frazier. Knowledge-gradient methods for statistical learning. PhD thesis, Princeton University Princeton, 2009. (page
Peter Frazier. A tutorial on Bayesian optimization. arXiv Preprint arXiv: ArXiv, 2018.
(page !)
Théo Galy-Fajou and Manfred Opper. Adaptive inducing points selection for Gaussian processes. arXiv Preprint arXiv:2107.10066 arXiv, 2021. (page 7)
Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G. Wilson. GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. Advunces in Neural Information Processing Systems, volume 31, pages 7576-7586. Curran Associates, Inc., 2018. (pages 8, 16)
Roman Garnett. Bayesian Optimizationn Cambridge University Press, Cambridge, United Kingdom:
New York, NY, 2023. (pages 2,

---

Gilles Gasso, Alain Rakotomamonjy. and Stéphane Canu. Recovering sparse signals with a certain family of nonconvex penalties and DC programming. IEEE Transactions on Signal Processing. 57 (12):4686-465) 2009. (page 9)
Ryan-Rhys Griffiths and José Miguel Hernández--Lobato. Constrained Bayesian optimization for
automatic chemical design using variational autoencoders. Chemical Science, 112)5577-5866 2020. (pages 1, 10)
James Hensman, Nicolo Fusi, and Neil D. Lawrence. Gaussian processes for big data. In Proceedings of the Conference Uncertainty in Artificial Intelligence, pages 282-290, AUAI Press, 2013.
(pages 1.3)
José Miguel Hernández-Lobato, James Requeima, Edward O. Pyzer-Knan and Alán Aspuru- -uzik. Parallel and distributed Thompson sampling for large-scale accelerated exploration of chemical space. In Proceedings of the International Conference on Machine Learning, volume 70 of PMLR, pages 1470-1479. JMLR, July 2017. (page 9)
Prateek Jaiswal, Harsha Honnappa, and Vinayak A. Rao. Asymptotic consistency of loss-calibrated variational Bayes. Stat, 9(1):e258, 2020. (pages 5)
Prateck Jaiswal, Harsha Honnappa, and Vinayak Rao. On the statistical consistency of risk-sensitive bayesian decision-making In Advances in Neural Information Processing Systems, volume 36, pages 53158 532000 Curran Associates, Inc., December 2023. (pages 5)
Martin Jankowiak, Geoff Pleiss, and Jacob R. Gardner. Parametric gaussian process regressors. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020. (page 19)
Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455-492 1998. (pages 1, 2, 5)
Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183225 1999. (pages 2,
3,4)
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations, San Diego, California, USA, 2015. (pages 15, 16)
Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations, Banff, AB, Canada, April 2014. (page 6)
Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centric view on Bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23 (132):1-109, 2022. (pages 5)
Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. Foundations and Trends@ in Machine Learning, 5(2-3):123-286, 2012. (page 10)
Tomasz Kušmierczyk, Joseph Sakaya, and Arto Klami. Variational Bayesian decision-making for continuous utilities. In Advances in Neural Information Processing Systems, volume 32, pages Curran Associates, Inc., 2019. (pages 4,5, 10)
Simon Lacoste-Juliest Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the loss-calibrated Bayesian. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 15 of PMLR. pages
JMLR, June 2011. (pages 2, 4, 10)
Kenneth Lange. MM Optimization Algorithms. Society for Industrial and Applied Mathematics, Philadelphia, 2016. (pages 2, 4)
Bolian Li and Ruqi Zhang. Long-tailed Classification from Bayessan-ddecisoon-tteery Perspective. arXiv Preprint arXiv:2303.005 00675,, Xrv, 2023. (page 10)
Wesley J Maddox, Samuel Stanton, and Andrew G Wilson. Conditioning sparse variational Gaussian processes for online decision-
In Advances in Neural Information Processing Systems, volume 34, pages 6365-6379. Curran Associates, Inc., 2021. (pages 1,2, 4, 6, 10)
Alexander G. de G. Matthews, James Hensman, Richard Turner, and Zoubin Ghahramani. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 51 of PMLR, pages 231-239. JMLR, May 2016. (pages 1,
12

---

Natalie Maus, Haydn Jones, Juston Moore, Matt J. Kusner, John Bradshaw, and Jacob Gardner. Local latent space Bayesian optimization over structured inputs. In Advances Neural Information Processing Systems, volume 35, pages 34505-34515 December 2022. (pages 1,9, 10)
Natalie Maus, Kaiwen Wu, David Friksson, and Jacob Gardner. Discovering many diverse solutions with Bayesian optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 206, pages 1779-179.. PMLR, April 2023. (pages 1, 8. 10)
Mitchell McIntire, Daniel Ratner, and Stefano Ermon. Sparse Gaussian Processes for Bayesian Optimization. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, Jersey City, New Jersey, USA, 2016. AUAI Press. (page
Thomas P. Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pages 362-369, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. (page 10)
Jonas Mockus. The Bayesian approach to global optimization. In System Modeling and Optimization. pages 473-481 Springer, 1982. (page 1)
Michael J. Morais and Jonathan W. Pillow. Loss-calibrated expectation propagation for approximate
Bayesian decision-making. Technical Report arXiv:2201.08 03128, arXiv, January 2022. (page 10)
Henry B. Moss, Scbastian W. Ober, and Victor Picheny. Inducing point allocation for sparse Gaussian processes in high throuhhtruughuut Bayesian optimisation. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 206 of PMLR, pages 5213-5230. JMLR, April 2023. (pages 1, 4, 7, 10, 16, 17, 18)
Radford M. Neal. Bayesian Learning for Neural Networks, volume 118 of Lecture Notes in Statistics. Springer New York, New York, NY, 1996. (page 9)
Joaquin Quiñonero-Candele and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6(65):1939- 1959, 2005.
(page 1)
Tom Rainforth, Adam Golinski, Frank Wood, and Sheheryar Zaidi. TargetàC bayesian inference: How to beat optimal conventional estimators. Journal of Machine Learning Research, 21 21(8))11-54, 2020.
wwt. ptpppps/v21/1911///// 102. html. (page 10)
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 33 of PMLR, pages 814 822. JMLR, April 2014. (page 10)
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, November 2005. (page 1)
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the International Conference on Machine Learning, volume 32 of PMLR, pages 1278-122 1286. JMLR, June 2014. (page 6)
Christian P. Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation. Springer Texts in Statistics. Springer, New York Berlin Heidelberg. 2. ed edition, 2001. (pages 3)
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015, (pages 5)
Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems, volume 18, pages 1257-1264. MIT Press, 2005. (page 5)
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25:2951-2959, 2012.
(page 10)
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian optimization using deep neural networks. In Proceedings of the International Conference on Machine Learning, volume 37 of PMLR, pages 2171-2188 JMLR, June 2015, (page 9)
13

---

Jost Tobias Springenberg. Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian Optimization with Robust Bayesian Neural Networks. In Advances Neural Information Processing Systems, volume 29, pages
Curran Associates, Inc., 2016. (page 9)
Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In Proceedings of the International Conference on Machine Learning, volume 162 of PMLR, pages 20459-20478. JMLR, June 2022. (pages I, 10)
Sonja Surjanovic and Derek Bingham. Virtual library of simulation experiments: Test functions and datasets, 2013. (page 8)
Alexander Terenin, David R. Burt, Artem Artemev, Seth Flaxman, Mark van der Wilk, Carl Edward Rasmussen, and Hong Ge. Numerically stable sparse Gaussian processes via minimum separation using cover trees. Journal of Machine Learning Research, 25(26):1-36, 2024. (page 7)
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 5 of PMLR, pages 567-574. JMLR, April 2009. (pages 1,
Michalis Titsias and Miguel
Doubly stochastic variational Bayes for non-conjugate inference. In Proceedings of the International Conference on Machine Learning, volume 32 of PMLR, pages 1971-1979. JMLR, June 2014. (pages 6, 10)
Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobace Sample-cfficient optimization in the latent space deep generative models via weighted retraining. In Advances in Neural Information Processing Systems, volume 33, pages 11259-11272. Curran Associates, Inc., 2020. (pages 1, 10)
Sattar Vakili, Henry Moss, Artem Artemev, Vincent Dutordoir, and Victor Picheny. Scalable Thompson sampling using sparse Gaussian process models. In Advances in Neural Information Processing Systems, volume 34, pages 5631-5643, 2021. (pages 1, 10)
Kenan Sehic, Alexandre Gramfort, Joseph Salmon, and Luigi Nardi. Lassobench: high-dimensional hyperparameter optimization benchmark suite for LASSO. In Proceedings of the International Conference on Automated Machine Learning, volume 188 of PMLR, pages 2/1-24. JMLR, 25-27
Jul 2022. (page 9)
Yixin Wang and David M. Blei. Frequeentistcoossittennyy
consistency of variational Bayes. Journal of the American Statistical Association, 14(527):1 147-1161, July 2019. (page
Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian optimization in digh-dimensional spaces. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 84 of PMLR, pages 745-754. JMLR, March 2018. (pages 8, 9)
Larry Wasserman. All of statistics: a concise course statistical inference. Springer Science Business Media, 2013. (pages 2,
Yadi Wei, Rishit Sheth, and Roni Khardon. Direct loss minimization for sparse Gaussian processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 130 of PMLR, pages 2566-2574, JMLR, March 2021. (page 10)
James Wilson, Frank Hutter, and Marc Deiscnroth. Maximizing acquisition functions for Bayesian optimization. In Advances in Neural Information Processing Systems, pages 9884 9895. Curran Associates, Inc., 2018. (pages 2, ?)
Jian Wu, Matthias Poloczek, Andrew G Wilson, and Peter Frazier. Bayesian optimization with gradients. In Advances in Neural Information Processing Systems, volume 30, pages 5267-5278. Curran Associates, Inc., 2017. (page) 2)
14

---

A Implementation Details
We will now provide additional details on the implementation. For the implementation, we treat the SVGP parameters, such as the variational parameters 2, inducing point locations Z, and hyperparameters 6, equally. Therefore, for clarity, we will collectively denote them w (2, Z. 8) such that w E W _ A x 3 x O and the resulting SVGP variational approximation as qw. Then, the ELBO and EULBO are equivalently denoted as follows:
ELBO (w: D) - LELBO (A, 02
EEULBO (x, w: xx Dw) - F--qw(f) logu (x: f; EELBO (w).
Also, notice that the EULBO separately denote the dataset to be passed to the utility and the ELBO. (Setting D, = w = Dx retrieves the original formulation in Eq. (8).)
Alternating Updates We perform block-coordinatt ascent on the EULBO by alternating between maximizing over x as w. Using vamilla gradient descent, the x-update equivalent
x + Yx V EULBO (x, w; D) + v E f~qu(() log u
where Yx is the stepsize. On the other hand, for the w- wpuatete, we subsample the data such that we optimize the ELBO over minibatch S D of size B |S|
w - w + yw V EULBO (x, W; D) = w w V u ( (E)~gu()) logu (:)) + ELBO (w; Sw)
where Yw is the stepsize. Naturally, the w-update stochastic due to minibatching, while the x-update is deterministic. In practice, we leverage the Adam update rule (Kingma and Ba, 2015) instead of simple gradient descent. Together with gradient clipping, this alternating update scheme is much more robust than jointly updating (x, w).
Algorithm 1: EULBO Maximization Policy
Input: SVGP parameters wa = (Ag, Zo. 9,), Dataset 2,, BO utility function Output: BO query x
Compute Warm- Stam-ttart Initializations
w - arg max weW CELOO (w; with wo as initialization.
x - arg max xer J u (x, f: (D) qw df)
Maximize EULBO
repeat
Update posterior approximation qu
Fetch minibatch S from D, Compute Iw VW (x, w; D,)
Clip Iw with threshold Gelip (w, gw)
AdamStep,, you
10
Update BO query x Compute 9x + EULBO (x, S, 2,)
11
12
Clip 9x with threshold Gelip
13
x - AdamStep,, (x.gx)
14
x - projx (x)
until until converged
16
x,1
Overview of Pseudocode. The complete high-level view of the algorithm is presented in Algorithm 1, except for the acquisition-speeiff details. AdamStep, (x, g) applies the Adam stepsize rule (Kingma practice,
and Ba, 2015) to the current location x with the gradient estimate g and the stepsize Adam is "stateful" optimizer, which maintains two scalar-valued states for each scalar parameter. For this, we re-initialize the Adam states at the beginning of each BO step.
15

---

Initialization. In the initial BO step = O. we initialize Zo with the DPP-based inducing point selection strategy of Moss et al. (2023). For the remaining SVGP parameters 2g and o.. we used the default initialization of GPy Torch (Gardner et al., 2018). For the remaining BO steps > 0, we use w from the previous BO step as the initialization wo of the current BO step.
Warm-Starting. Due to the non-convexity and multi-modality both ELBO and the acquisition function, it is critical to appropriately initialize the EULBO maximization procedure. As mentioned in Section 3.5, warm-start
EULBO maximization procedure, we use the conventional 2-step scheme Eq. (5), where maximize the ELBO and then maximize the acquisition function. For ELBO maximization, we apply Adam (Kingma and Ba, 2015) with the stepsize set as Yw until the convergence criteria (described below) are met. For acquisition function maximization, we invoke the highly optimized BoTorch. owthmiieimmiiiiii optimize_aaceem function (Balandat et al., 2020).
Minibatch Subsampling Strategy. As commonly done, we use the reshuffling subsampling strategy where the dataset 1,, is shuffled and partitioned into minibatches of size B. The number of minibatches constitutes an "epoch." The dataset is reshuffleddrppartttonnee after going through a full epoch.
Convergence Determination. For maximizing the ELBO during warm-starting and maximizing the EULBO, continue optimization until stop making progress exceed Kepochs number of epochs. That is if the ELBO//ULBB function value fails to make progress for Afaill number of steps.
Table 1: Configurations of Hyperparameters used for the Experiments
Hyperparameter
Value
Description
Yx
0.001
ADAM stepsize for the query
Y B
0.01
ADAM stepsize the SVGP parameters w
32
Minibatch size
Gelip Kepochs
2.0
Gradient clipping threshold
30
Maximum number of epochs
Mtail
Maximum number of failure to improve
m no 12ol
100
Number of inducing points
100
Number of observations for initializing BO
# quad.
20
Number of Hermiie quadrature points
optimize_acqf: restarts
10
optimize_acqEf raw_samples
256
optimize_acqf: batch_size
1/20
Depends on task; see details in Section 4
The hyperparameters used in our experiments are organized in Table 1. For the full-extent of the implementation details and experimental configuration, please refer to the supplementary code.
Hyperparameters.
16

---

B
Additional Plots
We provide additional results and plots that were omitted from the main text.
B.1 Additional Results on Molecule Tasks
In Fig. 4, we provide plots on additional results that are similar to those in Fig. 2. On three of the molecule tasks, we use 10,000 random molecules from the GuacaMol dataset as initialization. This more consistent with what has been done in previous works and achieves better overall optimization performance.
Osimertinth
Fesoferadine
Molian Molecales
3
Mcan Reward
1.2
9.81
0.31
0.38
0.76
0.77
20000
.0033
80000
20333
99393
50000
Number of Oracle Calls
TURRO (1TTL20
TURRO (BL
TURROO (Moer 2005
Figure 4: Additional optimization results on three molecule tasks using 10,000 random molecules from the GuacaMol dataset initialization. Each linc/shaded region represents the mean/ standard error over 20 runs. We count oracle calls starting after these initialization evaluations for all methods.
B.2 Separate Plots for BO and TuRBO Results
In this section, we provide additional plots separating out BO and TuRBO results to make visualization easier.
125
4.12
Meaan Reward
6.54
Onnerrmnt MPU
2wwuloedez MIN
Madis Moluako
Medaas Midemie
2.14
6.4
0.55
C114
1.2
111
6.4
100
606
6.56
688
600
3008
RUD8
0400
9000
20018
3098
1080
Number of Oracle Calls
(Mou
Figure 5: BO-only optimization results of Fig. 2. We compare EULBO-SVGP, ELBO-SVGP, ELBO-SVGP with DPP inducing point placement (Moss et al., 2023), and exact GPs. These are subset of the same results shown Fig. 2. Each line/shaded region represents the mean/standard eтror over 20 runs.
17

---

20
TuRBO (PK,LBC Ei) (EULBB
TuKBO (ELBO ED
Tuk:00 (Preet RI)
Mcan Reward
Orkesttil NIPO
Faetladee $00
Madia Meloales
Media Mekuale
080
1.70
1.70
1030
0.2
1450
24800
1980
1000
80.00
20.08
8008
1000
1080
2000
1900
10000
1.800 9.00
Number of Oracle Calls
Figure 6: TuRBO-only optimization results of Fig. 2. We compare EULBO-SVGI SVGP, ELBO-SVGP, ELBO-SVGP with DPP inducing point placement (Moss et al., 2023), and exact GPs. These are a subset of the same results shown in Fig. 2. Each line/shaded region represents the mean/standard error over 20 runs.
B.3 Efffet Number of Inducing Points
For the results with approximate- GPs in Section we used m 100 inducing points. In Fig. evaluate the effect of using larger number of inducing points (m 1024) for EULEO-SVGP and ELBO-SVGP.
Lasso DNA
TuRDO (BULBO 1024 inducing
TuRBO (BGLBBO ET)
L024 induxing points
TuRRO (FLBO EI)
-0.30
Moan Reward
-0.32
0.53
5000
10000
Number of Calls
Figure 7: Ablating the number inducing points used by EULBO-SVGP and ELBO-SVGP. As in Fig. 2, we compare running TuRBO with EULBO-SVGI and with ELBO-SVGP using 100 inducing points used for both methods. We add two additional curves for TuRBO with EULBO-SVGP and TuRBO with ELBO-SVGP using 1024 inducing points. Each line/shaded region represents the mean/standard error over 20 runs.
Fig. 7 shows that the number of inducing points has limited impact on the overall performance of TuRBO, and EULBO-SVGP outperforms ELBO-SVVP regardless of the number of inducing points used.
18

---

B.4 Effect of GP Objective
The results in Section 4 used standard SVGP objective. In this section, we evaluate the effect of using an alternative objective: the parametric Gaussian process regressor (PPGPR: Jankowiak et al., 2020) objective. PPGPR differs from the standard SVGP objective in that the variational approximation is optimized to maximize the predictive accuracy instead of matching the posterior.
Lasso DNA
TuRBO (BULBO ED PPGPR
TuRBO (BULBO
-0.29
(ELBO
-0.30
Mean Reward
-0.33
-0.34
5000
10000
15000
20000
Number Oracle Calls
Figure 8: Effect of using the PPGPR objective instead of the SVGP objective for EULBO-EI and ELBO-EI. As in Fig. 2, we compare running TuRBO with EULBO-EI and with ELBO-EI using an SVGP model for both methods. We add two additional curves for TuRBO with EULBO-EI with PPGPR model, and TuRBO with ELBO-EI using PPGPR model. Each line/shaded region represents the mean/standard error over 20 runs.
We compare the choice of objective (PPGPR vs SVGP) in Fig. 8 and observe that the objective has limited impact on the overall performance of TuRBO. In particular, EULBO-EI EI outperforms ELBO-EI regardless of the GP objective.
19

---

C
Compute Resources
Table Internal Cluster Setup
Type
Model and Specifications
System Topology
20 nodes with 2 sockets each with 24 logical threads (total 48 threads) Intel Xeon Silver 4310, 2.1 GHz (maximum 3.3 GHz) per socket
Processor
Cache
1.1 MiB LI, 30 MiB L2, and 36 MiB L3
Memory
250 GiB RAM
Accelerator
NVIDIA RTX A5000 per node, 2 GHZ, 24GB RAM
Type of Compute and Memory. All results in the paper required the use of GPU workers (one GPU per run of each method on each task). The majority of runs were executed on an internal cluster, where details are shown in Table 2, where each node was equipped with an NVIDIA RTX A5000 GPU. In addition, we used cloud compute resources for short period leading up to the subsmission of the paper. We used 40 RTX 4090 GPU workers from runpod. io, where each GPU had approximately 24 GB of GPU memory. While we used GB GPUs for our experiments, each run of our experiments only requires approximately 15 GB of GPU memory.
Execution Time. Each optimization run for non-molecule tasks takes approximately one day to finish. Since we run the molecule tasks out to much larger number of function evaluations than other tasks (80000 total function evaluations for each molecule optimization task), each molecule optimization task run takes approximately 2 days of execution time. With all eight tasks, ten methods run, and 20 runs completed per method, results in Fig. 2 include 1600 total optimization runs (800 for molecule tasks and 800 for non-molecule tasks). Additionally, the two added curves in each plot in Fig. required 160 additional runs (120 for molecule tasks and 40 for non-molecule task). Completing all of the runs needed to produce all of the results in this paper therefore required roughly 2680 total GPU hours.
Compute Resources Used During Preliminary Investigations. In addition to the computational resources required to produce experimental results in the paper discussed above, we spent approximately 500 hours of GPU time preliminary investigations. This was done the aforementioned internal cluster shown in Table 2.
D Wall-clock Run Times
In Table 3, we provide average wall-clock run times of different methods on the Lasso DNA optimization task.
Table 3: Average wall-clock run times for one full of TuRBO on the Lasso DNA task. We compare the average wall-clock time of TuRBO on all TuRBO methods from Figure 2. Note that we do not include the wall clock run time for TuRBO with Exact EI here because we only ran this method calls (rather than the full budget of 20k oracle calls).
2k oracle
Method
Wall-clock Run Time in Minutes
BULBO EI EULBO KG
267.30 2.53
296.95 + 1.31
ELBO EI
184.40 0.59
Moss et al. 20203 EI
194.32 + 0.77
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All stated claims are backed-up with results in Section 4 and the stated focus/scope of the paper accurately reflects what discussed throughout the rest of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings,
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
21

---

Justification:
This work does not contain a formal theoretical analysis.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: We provide detailed explanation of how our method works in Section 3 and all additional required details to reproduce results in Section and Appendix A. Additionally, we have included a link to public GitHub repository containing all of the source code used in the work in Section 4. This source code allows any reader run our code reproduce all results in the paper. Additionally, the README in the repository provides detailed instructions to make setting up the proper environment and running the code easy for users.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path reproducing verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes] Replace by [Yes], , [No], or [NA].
Justification: We have included a link to public GitHub repository containing all of the source code used in the work in Section 4. This source code allows any reader to
code to reproduce all results in the paper. Additionally, the README in the repository provides detailed instructions make setting up the proper environment and running the code easy for users.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
aublic//uiides/Coeeuuuu www.:uuubiiisiiinnni sesssssiooooo sisooliyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurlPS code and data submission guidelines (https://nipa wwtppblic/guiuee.eeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All chosen hyper- parameeasssss and implementation details are stated in section and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: On all plots, we plot the mean taken over multiple random runs and include егror bars to show the standard error over the runs.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer:
[Yes]
Justification:
See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and made sure to adhere to them in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No].
Justification: The paper is methodological, where the considered algorithm does not immediately pose societal risks.

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper does not use data with potential societal concerns.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators of assets used to produce our results are cited in Section 4. All assets used are open source software or models.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. wwt.coom/asseee datasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve live participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "i6DwDRpDUU", "reviewer": "Reviewer_veTg", "summary": "The paper proposes a new approach for scaling Bayesian Optimisation to large datasets. Contrary to previous approaches, which fitted a sparse GP and optimised acquisition function independently, the paper proposes a method, which jointly optimises the variational parameters of sparse GP and searches for the next point to query. This is done by adding a utility term to ELBO. Authors discuss how this objective can be connected to generalised Bayesian Inference. Authors evaluate the proposed method on a number of benchmark with high number of datapoints and show it outperforms other baselines.", "questions": "- Can the authors provide the running time of the proposed method (and baselines) ? It is ok if the proposed method is slower, but it would be good to quantify exactly how much slower it is.\n\n- Do authors have any expectations on how would the method perform in comparison to an exact GP on a smaller dataset? While I understand that the method is particularly designed for large datasets, it would be interesting to see how the utility VI-based acquisition strategy compares to standard acquisition functions like EI. It would be nice to have at least such experiment in the camera-ready version to judge whether the improvement delivered by the method comes from better acquisition strategy or better modelling", "limitations": "Authors admit that the increased difficulty of optimising EULBO is a limitation of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper touches on a very important subject of scaling Bayesian Optimisation to large datasets. I find the proposed solution to be very elegant. I also think the connection to generalised Bayesian Inference is very interesting.\n\n- The method seems to be delivering a significant improvement in sample efficiency over baselines. I believe the choice of benchmarks is sufficiently diverse and the number of baselines compared against is sufficient.\n\n- The paper is well-written and I really like that authors share the details of the tricks they used for improved optimisation of EULBO. I also like the fact that authors are very open about admitting the limitations of the proposed method.", "weaknesses": "- As mentioned by the authors, optimisation of the EULBO objective is a bit cumbersome and seems this process is also slower than the optimisation of standard ELBO. However, it might be worth to sacrifice a bit of computational efficiency for an increase in the sample complexity.\n\n- While authors cite previous work, which proved that the selected action satisfies convergence guarantee, it does not directly prove anything about the performance of optimisation process choosing action in such a way (e.g. regret bound or convergence to optimum). The paper could be made much stronger if authors managed to connect those notions in some way. However, I also do not think it is critical, as the main contribution of the paper is empirical (although it would be greatly appreciated).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> While authors cite previous work, which proved that the selected action satisfies convergence guarantee, it does not directly prove anything about the performance of optimisation process choosing action in such a way (e.g. regret bound or convergence to optimum). The paper could be made much stronger if authors managed to connect those notions in some way. However, I also do not think it is critical, as the main contribution of the paper is empirical (although it would be greatly appreciated).\n\nFrom the best of our knowledge, non-asymptotic convergence proofs under approximate inference, both in Bayesian optimization and bandits are rare. Furthermore, most existing works assume that one can explicitly control the fidelity of the approximation or that the true posterior is within the variational family. The benefits of utility-calibrated variational inference, on the other hand, only exists when the approximation is imperfect. Therefore, we are currently unsure how to analyze the theoretical properties of utility-calibrated Bayesian inference. However, we believe this would be a very interesting avenue for future research and will definitely explore this direction. We also point out that some asymptotic analyses of consistency exist as mentioned in Line 151-152.\n\n> Can the authors provide the running time of the proposed method (and baselines) ? It is ok if the proposed method is slower, but it would be good to quantify exactly how much slower it is.\n\nHere are the wall-clock run times for running TuRBO on the Lasso DNA task from the paper using the standard ELBO compared to using EULBO with EI and using EULBO with KG: \n\n| method | execution time (min) |\n|---|---|\n| ELBO | 184.40 $\\pm$ 0.59 |\n| EULBO-EI | 267.30 $\\pm$ 2.53 |\n| EULBO-KG | 296.95 $\\pm$ 1.31 |\n\nWe agree that providing a full table of average wall-clock runtimes for EULBO and all baselines would be useful, we will plan to add this table to the appendix for the camera ready version. Thank you for the suggestion! \n\n> Do authors have any expectations on how would the method perform in comparison to an exact GP on a smaller dataset? While I understand that the method is particularly designed for large datasets, it would be interesting to see how the utility VI-based acquisition strategy compares to standard acquisition functions like EI. It would be nice to have at least such experiment in the camera-ready version to judge whether the improvement delivered by the method comes from better acquisition strategy or better modelling\n\nWe point out to the top left pane in Figure 2, where we present results on the classic Hartmann 6 function. Somewhat surprisingly, our proposed approach does better than exact GPs in this one instance. Because SVGPs are not only a variational approximation, but a distinct model from exact GPs due to the use of inducing points, there is no reason to expect that exact GPs are a limiting case for the BO performance of SVGPs. Indeed, Exact GPs arguably have their own “mismatches” between learning and acquisition like the ones pointed out in this paper: hyperparameter learning in exact GPs is not done in a utility aware fashion, for example, while it is with EULBO. In general, we still expect exact GPs to be quite performant in settings where it can be applied. (Also note the response to Reviewer TCW1 on a similar question.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6QBzwNMpx5", "reviewer": "Reviewer_TCW1", "summary": "This paper proposes a modification to sparse variational Gaussian processes (SVGPs) used in Bayesian optimization (BO) to better align the SVGP posterior approximation with the goal of optimizing an acquisition function. The key idea is to jointly optimize the SVGP and the acquisition function using a unified objective called the expected utility lower bound (EULBO). This approach ensures the posterior approximation is well-suited for the downstream decision-making task. The authors derive efficient EULBO objectives for the expected improvement (EI) and knowledge gradient (KG) acquisition functions and demonstrate improved performance over standard SVGPs on several high-dimensional optimization benchmarks.", "questions": "Can the proposed EULBO approach be extended to other sparse GP approximations beyond SVGPs, even if they do not have a tractable ELBO? If so, what challenges would need to be addressed, and how might the optimization problem be formulated?\n\nWhy does the exact GP perform worse than the proposed method and some other baselines in the BO experiments (Figure 2)? Is this due to the limitations of the exact GP in high dimensions, or are there other factors at play?\n\nWhat are the key factors to consider when choosing between the different variants of the proposed method (e.g., EI-EULBO, KG-EULBO, batch versions)? In what scenarios would one variant be preferred over another?", "limitations": "The authors have addressed the limitations of their work in Section 6, acknowledging the increased computational cost of the EULBO approach compared to standard SVGPs. They also mention the need for multiple optimization tricks and the potential instability of the EULBO optimization problem. These are important limitations that users should be aware of when considering this method.\n\nHowever, the authors do not discuss any potential negative societal impacts of their work. While the proposed method is primarily methodological and does not pose immediate societal risks, it would be valuable for the authors to briefly comment on any broader implications of improved BO in high-dimensional spaces, such as the potential for misuse or unintended consequences in certain application domains.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel approach to aligning SVGP approximations with the goals of BO by jointly optimizing the posterior approximation and acquisition function. This is a creative combination of ideas from variational inference and decision theory.\n\nThe proposed method is well-motivated and grounded in sound theoretical principles. The derivations of the EULBO objectives for EI and KG are clear and technically sound. The experiments are comprehensive, covering a range of high-dimensional optimization tasks, and the results convincingly demonstrate the effectiveness of the approach.\n\nThe paper is well-written and easy to follow. The authors provide a clear exposition of the problem, the proposed solution, and the experimental setup. The figures and tables effectively communicate the key results.\n\nScaling BO to high-dimensional spaces is an important problem, and this work represents a significant step towards more efficient and effective BO in such settings. The proposed approach is general and could potentially be applied to other acquisition functions and GP approximations, making it a valuable contribution to the field.", "weaknesses": "The paper focuses specifically on SVGPs, but it would be interesting to explore whether the proposed approach can be extended to other sparse GP approximations, even those without a tractable ELBO. This would broaden the applicability of the method and strengthen the contribution.\n\nIn Figure 2, the exact GP performs worse than the proposed method and some other baselines in the BO setting. This is counterintuitive, as one would expect the exact GP to be the gold standard when computationally feasible. The authors should provide more discussion and insights into this unexpected result.\n\nThe paper presents several variations of the proposed method (e.g., EI-EULBO, KG-EULBO, batch versions), but it lacks a clear recommendation on which variant to use in practice. A more in-depth discussion of the trade-offs between these variants and guidance on when to use each one would enhance the practical value of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper focuses specifically on SVGPs, but it would be interesting to explore whether the proposed approach can be extended to other sparse GP approximations, even those without a tractable ELBO. This would broaden the applicability of the method and strengthen the contribution. \n\n> Can the proposed EULBO approach be extended to other sparse GP approximations beyond SVGPs, even if they do not have a tractable ELBO? If so, what challenges would need to be addressed, and how might the optimization problem be formulated?\n\nOur focus on SVGP was motivated by the fact that SVGPs are the most widely used sparse GP approximation in the high-throughput BO literature, but your question is a good one. SVGPs have been extended and improved several times, and it’s a natural question whether the EULBO can be adapted to all of these settings. Our method should be readily applicable when an ELBO exists mathematically but can only be approximated through Monte Carlo or quadrature. For sparse approximations where the objective is not a canonical ELBO, it is possible that a similar variational utility lower-bound could be derived, but it may look very different from this paper. We conjecture that some recent SVGP-like models like ODSVGP admit relatively straightforward EULBO adaptations. Other examples, like Vecchia approximations, would require pretty orthogonal machinery to what we introduce and would probably be interesting in their own right. We are happy to provide more detailed comments on specific sparse GP approximations if the reviewer has one in mind, and will add a discussion to the camera ready version.\n\n> In Figure 2, the exact GP performs worse than the proposed method and some other baselines in the BO setting. This is counterintuitive, as one would expect the exact GP to be the gold standard when computationally feasible. The authors should provide more discussion and insights into this unexpected result.\n> Why does the exact GP perform worse than the proposed method and some other baselines in the BO experiments (Figure 2)? Is this due to the limitations of the exact GP in high dimensions, or are there other factors at play?\n\nWhile we agree that this is counter-intuitive, we point to the fact that the true posterior of an SVGP is not an exact GP. Strictly speaking, SVGPs not only provide a variational approximation, but also modify the GP model to have an additional layer of latent variables. Therefore, there is no reason to expect that the performance of SVGPs would be the same as exact GPs, even if one has access to their true posterior. Additionally, and perhaps most interestingly, Exact GPs arguably have their own “mismatches” between learning and acquisition like the ones pointed out in and addressed by this paper: hyperparameter learning in exact GPs is not done in a utility aware fashion, for example. It seems plausible that, for some problems, utility aware hyperparameter learning might outweigh the loss incurred by using a sparse model. We do note that whether our method is better or worse than exact GPs is unlikely to be consistent across different tasks, and we suspect exact GPs are still quite competitive in settings where they can be used. (Also note the response to Reviewer veTg on a similar question.)\n\n> The paper presents several variations of the proposed method (e.g., EI-EULBO, KG-EULBO, batch versions), but it lacks a clear recommendation on which variant to use in practice. A more in-depth discussion of the trade-offs between these variants and guidance on when to use each one would enhance the practical value of the work.\n> What are the key factors to consider when choosing between the different variants of the proposed method (e.g., EI-EULBO, KG-EULBO, batch versions)? In what scenarios would one variant be preferred over another?\n\nThe best choice for which acquisition function to use, whether or not to use batch BO, what batch size to use, etc., often vary widely across problems and it’s hard for us to give guidance beyond what exists in the literature: KG is often less myopic than EI, which can be better on some problems and worse on others. Definitively comparing acquisition strategies probably deserves its own investigation. One thing we do note, however, is that the use of the EULBO switches the balance in the computational-performance trade-off of acquisition functions. For instance, the cost of computing the EULBO amortizes the cost of computing KG, making it a more sensible choice in some settings than it was before (see section 3.3 last paragraph). But again, whether one should use EI or KG given this tipping of the scale will highly depend on the problem and this paper probably doesn’t help definitively answer that question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aHwqvrcWLt", "reviewer": "Reviewer_jWXp", "summary": "The paper proposes a new method for training sparse Gaussian processes (GPs) for large-budget Bayesian Optimization (BO), designed to facilitate the sequential decision tasks inherent in BO. This is achieved by introducing the “expected utility lower bound (EULBO),” which formulates sparse GP training as a joint optimization problem of BO queries and posterior approximation. The proposed framework is compatible with other significant advancements in the field, such as Expected Improvement (EI) and Knowledge Refinement (KR) acquisition functions, TuRBO, and batch optimization. This compatibility was demonstrated across several BO benchmarks.", "questions": "Introduction:  \n\n(line 51): '\\emph{minorize-maximize}'\n\nSection 2:\n\n(Fig. 1): It would be beneficial to show the SVGP model fit for EULBO as well.\n\n(line 110): Could you define \\lambda, m and S?\n\n(Eq. between 114-115): q_\\lambda is initially presented with one argument, then two. If the intention is for q_\\lambda to represent all approximate distributions  (in this case, p(f, u) ), please state this explicitly. It is natural to assume that q_\\lambda(.) is just the Gaussian pdf (as defined in line 110), which creates confusion.\n\n(line 116): k(.,.) as a function also needs to be explicitly defined.\n\nSection 3:\n\n(line 123): This should probably refer to the equation between lines 114-115, not Equation 3.\n\n(Eq 4): Please define \\Lambda.\n\n(Eq 6): Please use double integrals to indicate nested integration. Additionally, I don't understand the emergence of l(D_t |f) and the normalizing constant Z. I was under the impression that l(.|.) is the Gaussian likelihood as defined in line 113. Notice that D_t, as defined in line 92, contains both x and y.\n\n(line 147): 'corresponds to'\n\n(line 165): Please clarify where exactly the requirement for the utility to be strictly positive plays a role or comes from.\n\n(lines 173-175): Would using Gauss-Hermite quadrature for EI make EULBO significantly slower than ELBO?\n\n(line 189): What is y^{+}_t and what its relation to y_t?\n\nQuestion:\n\nBecause you formulate the BO queries and GP approximation as a joint optimization problem without necessarily reducing the dimensionality of the problem, how much slower is this method compared to ‘approximation-ignorant’ BO?", "limitations": "The authors are open about the limitations of their work, which they leave for future exploration. Specifically, the increased computational complexity seems to be a primary limitation. It would be beneficial if the authors could quantify this increased complexity, perhaps through reporting empirical runtimes or other relevant metrics.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The method proposed in the paper is novel and original. One of the main strengths is how the newly proposed EULBO integrates with important points in the literature. The authors effectively demonstrate how EULBO can be used in various contexts, such as with EI, KR acquisitions, and batch optimization. This helps distinguish their work from others in the field. The submission is technically sound, with derivations and experiments convincingly supporting the claims of the paper. The work is significant, and I anticipate it will be extended in the future or that some of the ideas presented will be adopted elsewhere.", "weaknesses": "While the paper is generally well-written, many derivations were abbreviated and some of the notation was not properly explained, making the paper more difficult to read than necessary. These issues, however, are easily amendable. I provide some suggestions for improvement in the following subsections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (line 110): Could you define \\lambda, m and S?\n\n$m$ and $S$ are defined as the (learned) mean and covariance of the variational distribution $q(u)$ in our SVGP model. $\\lambda$ should have been defined as $\\lambda = (m, S)$, a shorthand for “all of the variational parameters,” which we omitted. We will fix this in the next version, and clarify the definition of $m, S$.\n\n> (Eq. between 114-115): q_\\lambda is initially presented with one argument, then two. If the intention is for q_\\lambda to represent all approximate distributions (in this case, p(f, u) ), please state this explicitly. It is natural to assume that q_\\lambda(.) is just the Gaussian pdf.\n\nIn Bayesian methods, it is fairly typical to overload the notation of a pdf/measure with its argument. Therefore, $q(u)$ and $q(f, u)$, should be without ambiguity as per common practice. The subscript \\lambda is to denote that the distribution denoted as q contains trainable parameters in $\\lambda$. We will clarify this.\n\n> (line 116): k(.,.) as a function also needs to be explicitly defined.\n\nThank you for pointing this out. We will define the kernel function properly in the next version.\n\n> (line 165): Please clarify where exactly the requirement for the utility to be strictly positive plays a role or comes from.\n\nThis is due to the log in Eq 6. Thank you for pointing this out, this is pretty subtle and we don’t draw much attention to it. We will clarify this requirement in the next version.\n\n> (lines 173-175): Would using Gauss-Hermite quadrature for EI make EULBO significantly slower than ELBO?\n\nThis is a great question that we missed addressing in the original text. Very crucially, the expensive $K_{zz}^{-1}m$ and $K_{zz}^{-1}SK_{zz}^{-1}$ (or $K_{zz}^{-1/2}$ with whitening) solves that dominate both the asymptotic and practical running time of both the ELBO and the EULBO are fixed across the log utility evaluations needed by quadrature (and Monte Carlo in the q-EULBO case). As a result, the additional work of quadrature is pretty negligible. Because Gauss-Hermite quadrature converges extremely quickly in the number of quadrature sites, it only requires on the order of 10 or so of these post-solve evaluations to achieve near machine precision. We will add an explanation of this.\n\n> (Eq 6): Please use double integrals to indicate nested integration. Additionally, I don't understand the emergence of l(D_t |f) and the normalizing constant Z. I was under the impression that l(.|.) is the Gaussian likelihood as defined in line 113. Notice that D_t, as defined in line 92, contains both x and y.\n\nYes, that is correct. $\\pi(f,u|D)$ denotes the “normalized” posterior, which is essentially $\\ell(D|f,u) p(f,u) / Z$. Plugging these into the derivation of Eq 6 should clarify things.\n\n> (line 189): What is y^{+}_t and what its relation to y_t?\n\nThank you for catching this! We forgot to define $y^{+}_t$, which is the highest value of $y_t$ observed so far. We will properly define this in the next version.\n\n> Because you formulate the BO queries and GP approximation as a joint optimization problem without necessarily reducing the dimensionality of the problem, how much slower is this method compared to ‘approximation-ignorant’ BO?\n\nIn general, you’re right that the added cost of EULBO is largely due to optimization challenges, rather than concerns like quadrature etc. A single EULBO forward and backward calculation has essentially the same cost as an ELBO calculation; however, as we describe in Section 3.5 and our limitations section, we currently “warm start” EULBO optimization by first optimizing the ELBO, leading to increased computational cost. Here are the wall-clock run times for running TuRBO on the Lasso DNA task using the standard ELBO compared to EULBO with EI and EULBO with KG: \n\n| method | execution time (min) |\n|---|---|\n| ELBO | 184.40 $\\pm$ 0.59 |\n| EULBO-EI | 267.30 $\\pm$ 2.53 |\n| EULBO-KG | 296.95 $\\pm$ 1.31 |\n\nFor the camera ready version of the paper, we plan to add a table of wall-clock runtimes comparing EULBO to all baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FQMFP9BCA2", "reviewer": "Reviewer_6opG", "summary": "The authors present an extension of the sparse Variational Gaussian Process framework (SVGP) that is better suited for Bayesian optimization (BO) tasks. This is achieved by a new optimization criterion (EULBO) which aims to optimize the parameters of SVGPs s.t. the training data is fit well while achieving a better performance in selecting new solution candidates. Two concrete instantiations of EULBO (with Expected Improvement and Knowledge Gradient) are derived. An extensive empirical evaluation is conducted to demonstrate the effectiveness of the proposed approach.", "questions": "see weaknesses", "limitations": "the authors discussed the limitations of their approach in a detailed waysssssssssssssssssssssss", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- though being dense, the paper is clearly written and straightforward to follow\n- the proposed approach is technically solid and mathematically sound\n- a relevant problem is addressed by an elegant solution\n- both, methodology and experiments are presented well", "weaknesses": "## Methodology\n- there are some details unclear/missing regarding figure 1:\n\t- which points were used as inducing points?\n\t- why is the uncertainty relatively constant at the right side of the plot (left plot)? Depending on the choice of the inducing points, I'd expect the predicted variance (and thus the EI) to increase towards the right edge of the plot\n\t- what was the exact problem definition leading to Fig. 1? It would be good to see the exact parameterizations for reproducibility as it would help to gauge whether the motivational example is rather cherry-picked or whether the shown issue manifests across different parameterizations\n- missing $t$ in Eq. 2? shouldn't it be $\\pi(f | \\mathcal{D}_t)$?\n- in the equation after line 221 it seems that the sum does not depend on $j$. shouldn't it be $\\mathbf{x}_j$?\n\n## Experiments\n- why were the baselines only applied in conjunction with EI? (see line 259)\n- it seems that, depending on the choice of the objective and method, there are quite significant differences in the variance of EULBO. For example, in Fig. 2 \"Media Molecules 1\", TuRBO + ELBO EI has much higher variance than TuRBO + EULBO EI. Do you account this observation to EULBO?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Figure 1: which points were used as inducing points?\n\nThe inducing points are initialized to a set of points selected uniformly at random in the search space and then they are learned via gradient descent by minimizing the loss (ELBO or EULBO) when the GP model is trained on the data. \n\n> Figure 1: why is the uncertainty relatively constant at the right side of the plot (left plot)? Depending on the choice of the inducing points, I'd expect the predicted variance (and thus the EI) to increase towards the right edge of the plot\n\nIn the large interval between the rightmost inducing points, the variance remains fairly constant at the prior variance. At the rightmost inducing point, the variance dips, and then to the right of this rightmost inducing point, the variance increases back up to the prior variance. We think this is fairly expected behavior, and perhaps the dip in variance at the rightmost inducing point was missed due to the relatively large observation noise in this data? We are happy to discuss this with you in more detail. \n\n> Figure 1: what was the exact problem definition leading to Fig. 1? It would be good to see the exact parameterizations for reproducibility as it would help to gauge whether the motivational example is rather cherry-picked or whether the shown issue manifests across different parameterizations\n\nWe agree that reproducibility is important for this example problem as well as for other experiments. We plan to include a simple notebook with code to quickly reproduce Figure 1 in our code release which will be available on github once the paper is accepted. We specifically designed the toy dataset to illustrate the point of the paper (large amount of data on the left, very little on the right so that an SVGP with only 4 inducing points would underfit on the right so that the large objective value out right would be missed). Our hope is that this figure is illustrative and aids reader understanding, which we will make more clear in the next version. Of course, the actual experimental results in the paper are actual challenging benchmark problems and not designed for our method. Thank you for the suggestion! \n\n> missing 𝑡 in Eq. 2? shouldn't it be 𝜋(𝑓|𝐷𝑡)?\n\nThanks for catching this. We will fix this in the next version.\n\n> in the equation after line 221 it seems that the sum does not depend on 𝑗. shouldn't it be 𝑥𝑗?\n\nThe maximum (denoted by “max”; not “maximize”!) operation in Line 221 is taken over j.\n\n> why were the baselines only applied in conjunction with EI? (see line 259)\n\nEI is the most commonly used acquisition function used in the BO literature, often because it is cheap to compute and performs comparably with other KG and other acquisition functions. A common motivation in the literature for choosing EI over KG is because KG is significantly more expensive to compute. Running EULBO provides a unique opportunity to make computing KG much more efficient such that it becomes approximately as cheap as EULBO with EI  (see last paragraph of section 3.3). This is not the case for other baselines so running them with KG wouldn’ve been much more expensive. Beyond KG and EI, we note that the EULBO is limited by construction to acquisition functions that admit decision theoretic formulations, i.e. they must be phrasable as posterior expected utilities. This appears to exclude some popular acquisition functions like UCB.\n\n> it seems that, depending on the choice of the objective and method, there are quite significant differences in the variance of EULBO. For example, in Fig. 2 \"Media Molecules 1\", TuRBO + ELBO EI has much higher variance than TuRBO + EULBO EI. Do you account this observation to EULBO?\n\nThis is not something that we have explored specifically, but one plausible explanation is that EULBO consistently converges to a high-performing region of the search space with similar high-scoring molecules, while ELBO converges to a wider variety of lower-scoring regions of the search space, thus resulting in higher variance and lower final objective values. In other words, EULBO may just have lower variance because it consistently performs better on this problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work scales up Bayesian Optimisation to large datasets. The paper is clear and claims supported. All reviewers were enthusiastic about this work and all concerns raised were adequately addressed during the rebuttal. This is a solid piece of work that is relevant to the community.
Decision: Accept (spotlight)