Title: Approximation-Aware Bayesian Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bayesian optimization

variational inference

Gaussian processes

utility maximization

expected improvement

knowledge gradient

black-box optimization
Abstract: High-dimensional Bayesian optimization (BO) tasks such as molecular design often require $>10,$$000$ function evaluations before obtaining meaningful results. While methods like sparse variational Gaussian processes (SVGPs) reduce computational requirements in these settings, the underlying approximations result in suboptimal data acquisitions that slow the progress of optimization. In this paper we modify SVGPs to better align with the goals of BO: targeting informed data acquisition over global posterior fidelity. Using the framework of utility-calibrated variational inference (Lacoste–Julien et al., 2011), we unify GP approximation and data acquisition into a joint optimization problem, thereby ensuring optimal decisions under a limited computational budget. Our approach can be used with any decision-theoretic acquisition function and is readily compatible with trust region methods like TuRBO (Eriksson et al., 2019). We derive efficient joint objectives for the expected improvement (EI) and knowledge gradient (KG) acquisition functions in both the standard and batch BO settings. On a variety of recent high dimensional benchmark tasks in control and molecular design, our approach significantly outperforms standard SVGPs and is capable of achieving comparable rewards with up to $10\times$ fewer function evaluations.

Approximation-Aware Bayesian Optimization

Abstract

High-dimensional Bayesian optimization (BO) tasks such as molecular design often require > 10,000 function evaluations before obtaining meaningful results. While methods like sparse variational Gaussian processes (SVGPs) reduce computational requirements in these settings, the underlying approximations result in suboptimal data acquisitions that slow the progress of optimization. In this paper we modify SVGPstobetter align with the goals of BO: targeting informed data acquisition rather than global posterior fidelity. Using the framework of utility-calibrated variational inference, we unify GP approximation and data acquisition into a joint optimization problem, thereby ensuring optimal decisions under a limited computational budget. Our approach can be used with any decision-theoretic acquisition function and is readily compatible with trust region methods like TuRBO. We derive efficient joint objectives for the expected improvement and knowledge gradient acquisition functions for standard and batch BO. Our approach outperforms standard SVGPs on high-dimensional benchmark tasks in control and molecular design.

1 Introduction

Bayesian optimization (BO; Frazier, 2018; Garnett, 2023; Jones et al., 1998; Mockus, 1982; Shahriari et al., 2015) casts optimization as a sequential decision-making problem. Many recent successes of BO have involved complex and high-dimensional problems. In contrast to 'classic' low-dimensional BO problems-where expensive black-box function evaluations far exceeded computational costs-these modern problems necessitate tens of thousands of function evaluations, and it is often the complexity and dimensionality of the search space that makes optimization challenging, rather than a limited evaluation budget (Eriksson et al., 2019; Griffiths and Hernández-Lobato, 2020; Maus et al., 2022, 2023; Stanton et al., 2022). Because of these scenarios, BO is entering a regime where computational costs are becoming a primary bottleneck (Maddox et al., 2021; Maus et al., 2023; Moss et al., 2023; Vakili et al., 2021), as the Gaussian process (GP; Rasmussen and Williams, 2005) surrogate models that underpin most of Bayesian optimization scale cubically with the number of observations.

In this new regime, we require scalable GP approximations, an area that has made tremendous progress over the last decade. In particular, sparse variational Gaussian processes (SVGP; Hensman et al., 2013; Quiñonero-Candela and Rasmussen, 2005; Titsias, 2009) have seen an increase in use (Griffiths and Hernández-Lobato, 2020; Maddox et al., 2021; Maus et al., 2022, 2023; Stanton et al., 2022; Tripp et al., 2020; Vakili et al., 2021), but many challenges remain to effectively deploy SVGPs for large-budget BO. In particular, the standard SVGP training objective is not aligned with the goals of black-box optimization. SVGPs construct an inducing point approximation that maximizes the standard variational evidence lower bound (ELBO; Jordan et al., 1999), yielding a posterior approximation /u1D45E ∗ (/u1D453) that models all observed data (Matthews et al., 2016; Moss et al., 2023). However, the optimal posterior approximation /u1D45E ∗ is suboptimal for the decision-making tasks involved

in BO (Lacoste-Julien et al., 2011). In BO, we do not care about posterior fidelity at the majority of prior observations; rather, we only care about the fidelity of downstream functions involving the posterior, such as the expected utility. To illustrate this point intuitively, consider using the common expected improvement (EI; Jones et al., 1998) acquisition function for selecting new observations. Maximizing the ELBO might result in a posterior approximation that maintains fidelity for training examples in regions of virtually zero EI, thus wasting 'approximation budget.'

To solve this problem, we focus on the deep connections between statistical decision theory (Robert, 2001; Wasserman, 2013, §12) and Bayesian optimization (Garnett, 2023, §6-7), where acquisition maximization can be viewed as maximizing posterior-expected utility. Following this perspective, we leverage the utility-calibrated approximate inference framework (Jaiswal et al., 2020, 2023; Lacoste-Julien et al., 2011), and solve the aforementioned problem through a variational bound (Blei et al., 2017; Jordan et al., 1999)-the (log) expected utility lower bound ( EULBO ) -a joint function of the decision (the BO query) and the posterior approximation (the SVGP). When optimized jointly, the EULBO automatically yields the approximately optimal decision through the minorizemaximize principle (Lange, 2016). The EULBO is reminiscent of the standard variational ELBO (Jordan et al., 1999), and can indeed be viewed as a standard ELBO for a generalized Bayesian inference problem (Bissiri et al., 2016; Knoblauch et al., 2022), where we seek to approximate the utility-weighted posterior. This work represents the first application of utility-calibrated approximate inference towards BO despite its inherent connection with utility maximization.

The benefits of our proposed approach are visualized in Fig. 1. Furthermore, it can be applied to acquisition function that admits a decision-theoretic interpretation, which includes the popular expected improvement (EI; Jones et al., 1998) and knowledge gradient (KG; Wu et al., 2017) acquisition functions, and is trivially compatible with local optimization techniques like TuRBO (Eriksson et al., 2019) for high-dimensional problems. We demonstrate that our joint SVGP/acquisition optimization approach yields significant improvements across numerous Bayesian optimization benchmarks. As an added benefit, our approach can simplify the implementation and reduce the computational burden of complex (decision-theoretic) acquisition functions like KG. We demonstrate a novel algorithm derived from our joint optimization approach for computing and optimizing the KG that expands recent work on one-shot KG (Balandat et al., 2020) and variational GP posterior refinement (Maddox et al., 2021).

Overall, our contributions are summarized as follows:

We propose utility-calibrated variational inference of SVGPs in the context of large-budget BO.

Westudy this framework in two special cases using the utility functions of two common acquisition functions: EI and KG. For each, we derive tractable EULBO expressions that can be optimized.

For KG, we demonstrate that the computation of the EULBO takes only negligible additional work over computing the standard ELBO by leveraging an online variational update. Thus, as a byproduct of optimizing the EULBO , optimizing KG becomes comparable to the cost of the EI.

We extend this framework to be capable of running in batch mode, by introducing q-EULBO analogs of q-KG and q-EI as commonly used in practice (Wilson et al., 2018).

We demonstrate the effectiveness of our proposed method against standard SVGPs trained with ELBO maximization on high-dimensional benchmark tasks in control and molecular design, where the dimensionality and evaluation budget go up to 256 and 80k, respectively.

2 Background

Noisy Black-Box Optimization. Noisy black-box optimization refers to problems of the form: maximize/u1D499∈/u1D4B3 /u1D439 (/u1D499) , where /u1D4B3 ⊂ ℝ /u1D451 is some compact domain, /u1D439 ∶ /u1D4B3 → /u1D4B4 is some objective function, and we assume that only zeroth-order information of /u1D439 is available. More formally, for some /u1D456 ∈ ℕ>0 , we assume that observations of the objective function (/u1D499 /u1D456 , /u1D466 /u1D456 = ˆ /u1D439 (/u1D499 /u1D456 )) have been corrupted by independently and identically distributed (i.i.d.) Gaussian noise /u1D439 ˆ (/u1D499 /u1D456 ) ≜ /u1D439(/u1D499 /u1D456 ) + /u1D716 , where /u1D716 ∼ /u1D4A9(0, /u1D70E 2 n ) . The noise variance /u1D70E 2 n is also unknown.

Bayesian optimization. Bayesian Optimization (BO) is and iterative approach to noisy black-box optimization that iterates the following steps: ❶ At each step /u1D461 ≥ 0 , we use a set of observations /u1D49F /u1D461 = { ( /u1D499 /u1D456 , /u1D466 /u1D456 = ˆ /u1D439 (/u1D499 /u1D456 ) ) } /u1D45B /u1D461 /u1D456 =1 of /u1D439 ˆ to fit a surrogate supervised model /u1D453 ∈ ℱ . Typically, ℱ is taken to be the sample space of a Gaussian process such that the function-valued posterior distribution

ELBO fit (

m

=4)

ELBO

EULBO (Ours)

Data

True function

Inducing points

GP mean

EI (approx)

EI (exact)

/u1D70B (/u1D453 ∣ /u1D49F ) forms a distribution over surrogate models at step /u1D461 . ❷ The posterior is then used to form a decision problem where we choose which point we should evaluate next, /u1D499 /u1D461 +1 = /u1D6FF /u1D6FC (/u1D49F /u1D461 ) , by maximizing an acquisition function /u1D6FC ∶ /u1D4B3 →ℝ as

❸ After selecting /u1D499 /u1D461 +1 , /u1D439 ˆ is evaluated to obtain the new datapoint (/u1D499 /u1D461 +1 , /u1D466 /u1D461 +1 = ˆ /u1D439 (/u1D499 /u1D461 +1 )) . This is then added to the dataset, forming /u1D49F /u1D461 +1 = /u1D49F /u1D461 ∪ (/u1D499 /u1D461 +1 , /u1D466 /u1D461 +1 ) to be used in the next iteration.

Utility-Based Acquisition Functions. Many commonly used acquisition functions, including EI and KG, can be expressed as posterior-expected utility functions

where /u1D462 (/u1D499, /u1D453 ; /u1D49F) ∶ /u1D4B3 × ℱ → ℝ is some utility function associated with /u1D6FC (Garnett, 2023, §6-7). In statistical decision theory, posterior-expected utility maximization policies such as /u1D6FF /u1D6FC are known as Bayes policies . These are important because, for a given utility function, they attain certain notions of statistical optimality such as Bayes optimality and admissibility (Robert, 2001, §2.4; Wasserman, 2013, §12). However, this only holds true if we can exactly compute Eq. (2) over the posterior. Once approximate inference is involved, making optimal Bayes decisions becomes challenging.

Sparse Variational Gaussian Processes. While the /u1D4AA(/u1D45B 3 ) complexity of exact Gaussian process model selection and inference is not necessarily a roadblock in the traditional regression setting with 10 , 000 -50 , 000 training examples, BO amplifies the scalability challenge by requiring us to sequentially train or update many large scale GPs as we iteratively acquire more data.

To address this, sparse variational GPs (SVGP; Hensman et al., 2013; Titsias, 2009) have become commonly used in high-throughput Bayesian optimization. SVGPs modify the original GP prior from /u1D45D(/u1D453 ) to /u1D45D(/u1D453 ∣ /u1D496)/u1D45D(/u1D496) , where we assume the latent function /u1D453 is 'induced' by a finite set of inducing values /u1D496 = (/u1D462 1 , … , /u1D462 /u1D45A ) ∈ ℝ /u1D45A located at inducing points /u1D49B /u1D456 ∈ /u1D4B3 for /u1D456 = 1, … , /u1D45A . Inference is done through variational inference (Blei et al., 2017; Jordan et al., 1999), where the posterior of the inducing points is approximated using /u1D45E /u1D740 (/u1D496) = /u1D4A9 (/u1D496; /u1D740 = (/u1D48E, /u1D47A)) and that of the latent functions with /u1D45E (/u1D453 ∣ /u1D496) = /u1D45D (/u1D453 ∣ /u1D496) . Here, the variational parameters /u1D48E and /u1D47A are defined as the learned mean and covariance of the variational distribution /u1D45E /u1D740 (/u1D496) . It is standard practice to define /u1D740 = (/u1D48E, /u1D47A) so that /u1D740 can be used as shorthand to represent all of the trainable variational parameters. As is typical in the BO literature, we use the subscript /u1D740 ∈ Λ to denote that the distribution denoted as /u1D45E contains trainable parameters in /u1D740 .

For a positive definite kernel function /u1D458 ∶ /u1D4B3 × /u1D4B3 →ℝ>0 , the resulting ELBO objective, which can be computed in a closed form (Hensman et al., 2013), is then

where /u1D4C1(/u1D466 /u1D456 ∣ /u1D453 (/u1D499 /u1D456 )) = /u1D4A9 (/u1D466 /u1D456 ∣ /u1D453 (/u1D499 /u1D456 ) , /u1D70E /u1D716 ) is a Gaussian likelihood. The marginal variational approximation can be computed as

such that the point-wise function evaluation on some /u1D499 ∈ /u1D4B3 is

with /u1D458 ˜ /u1D499/u1D499 ≜ /u1D458 (/u1D499, /u1D499) -/u1D48C /u1D499/u1D481 /u1D472 -1 /u1D481 /u1D481 /u1D48C ⊤ /u1D481 /u1D499 , the vector /u1D48C /u1D481 /u1D499 ∈ ℝ /u1D45A is formed as [/u1D48C /u1D481 /u1D499 ] /u1D456 = /u1D458 (/u1D49B /u1D456 , /u1D499) , and the matrix /u1D472 /u1D481 /u1D481 ∈ ℝ /u1D45A×/u1D45A is formed as [/u1D472 /u1D481 /u1D481 ] /u1D456 /u1D457 = /u1D458(/u1D49B /u1D456 , /u1D49B /u1D457 ) . Additionally, the GP likelihood and kernel contain hyperparameters, which we denote as /u1D73D ∈ Θ , and we collectively denote the set of inducing point locations as /u1D481 = (/u1D49B 1 , … , /u1D49B /u1D45A ) ∈ /u1D4B3 /u1D45A . We therefore denote the ELBO as ℒELBO (/u1D740, /u1D481, /u1D73D; /u1D49F /u1D461 ) .

3 Approximation-Aware Bayesian Optimization

When SVGPs are used in conjunction with BO (Maddox et al., 2021; Moss et al., 2023) at iteration /u1D461 ≥ 0 , acquisition functions of the form of Eq. (2) are naïvely approximated as

where /u1D45E /u1D740 (/u1D453) is the approximate SVGP posterior given by Eq. (4). The acquisition policy implied by this approximation contains two separate optimization problems:

Treating these optimization problems separately creates an artificial bottleneck that results in suboptimal data acquisition decisions. Intuitively, /u1D740 ∗ ELBO is chosen to faithfully model all observed data (Matthews et al., 2016; Moss et al., 2023), without regard for how the resulting model performs at selecting the next function evaluation in the BO loop. For an illustration of this, see Figure 1. Instead, we propose a modification to SVGPs that couples the posterior approximation and data acquisition through a joint problem of the form:

This results in /u1D499 /u1D461 +1 directly approximating a solution to Eq. (2), where the expected utility lowerbound ( EULBO ) is an ELBO-like objective function derived below.

3.1 Expected Utility Lower-Bound

Consider an acquisition function of the form of Eq. (2), where the utility /u1D462 ∶ /u1D4B3 ×ℱ → ℝ>0 is strictly positive. We can derive a similar variational formulation of the acquisition function maximization problem following Lacoste-Julien et al. (2011). That is, given any distribution /u1D45E /u1D740 indexed by /u1D740 ∈ Λ and considering the SVGP prior augmentation /u1D45D(/u1D453 ) →/u1D45D(/u1D453 ∣ /u1D496)/u1D45D(/u1D496) , the acquisition function can be lower-bounded through Jensen's inequality as

where /u1D44D is a normalizing constant. A restriction on /u1D462 comes from the inequality in Eq. (7), where the utility needs to be strictly positive. This means that non-strictly positive utilities need to be modified to be incorporated into this framework. (See the examples by Kuśmierczyk et al., 2019.) Also, notice that the derivation is reminiscent of expectation-maximization (Dempster et al., 1977) and variational lower bounds (Jordan et al., 1999). That is, through the minorize-maximize principle (Lange, 2016), maximizing the lower bound with respect to /u1D499 and /u1D740 approximately solves the original problem of maximizing the posterior-expected utility.

Expected Utility Lower-Bound. Up to a constant and rearranging terms, maximizing Eq. (7) is equivalent to maximizing

which is the joint objective function alluded to in Eq. (6). We maximize EULBO to obtain (/u1D499 /u1D461 +1 , /u1D740 ∗ ) = arg max /u1D499∈/u1D4B3 ,/u1D740∈Λ ℒEULBO (/u1D499, /u1D740) , where /u1D499 /u1D461 +1 corresponds our next BO 'query'.

From Eq. (8), the connection between the EULBO and ELBO is obvious: the EULBO is now 'nudging' the ELBO solution toward high utility regions. An alternative perspective is that we are approximating a generalized posterior weighted by the utility (Table. 1 by Knoblauch et al., 2022; Bissiri et al., 2016). Furthermore, Jaiswal et al. (2020, 2023) prove that the resulting actions satisfy consistency guarantees under assumptions typical in such results for variational inference (Wang and Blei, 2019).

Hyperparameters and Inducing Point Locations. For the hyperparameters /u1D73D and inducing point locations /u1D481 , we use the marginal likelihood to perform model selection, which is common practice in BO (Shahriari et al., 2015, §V.A). (Optimizing over /u1D481 was popularized by Snelson and Ghahramani, 2005.) Following suit, we also optimize the EULBO as a function of /u1D73D and /u1D481 as

We emphasize here that the SVGP-associated parameters /u1D740, /u1D73D , /u1D481 have gradients that are determined by both terms above. Thus, the expected log-utility term /u1D53C /u1D453 ∼/u1D45E /u1D740 (/u1D453 ) log /u1D462 (/u1D499, /u1D453 ; /u1D49F /u1D461 ) simultaneously results in acquisition of /u1D499 /u1D461 +1 and directly influences the underlying SVGP regression model.

3.2 EULBO for Expected Improvement (EI)

The EI acquisition function can be expressed as a posterior-expected utility, where the underlying 'improvement' utility function is given by the difference between the objective value of the query, /u1D453 (/u1D499) , and the current best objective value /u1D466 ∗ /u1D461 = max/u1D456=1,…,/u1D461 { /u1D466 /u1D456 ∣ /u1D466 /u1D456 ∈ /u1D49F /u1D461 } :

where ReLU(/u1D465) ≜ max (/u1D465, 0) . Unfortunately, this utility is not strictly positive whenever /u1D453 (/u1D499) ≤ /u1D466 ∗ . Thus, we cannot immediately plug /u1D462 EI into the EULBO . While it is possible to add a small positive constant to /u1D462 EI and make it strictly positive as done by Kuśmierczyk et al. (2019), this results in a looser Jensen gap in Eq. (7), which could be detrimental. This also introduces the need for tuning the constant, which is not straightforward. Instead, we define the following 'soft' EI utility:

where the ReLU in Eq. (9) is replaced with softplus (/u1D465) ≜ log (1 + exp(/u1D465)) . softplus(/u1D465) converges to the ReLU in both extremes of /u1D465→±∞ . Thus, /u1D462 SEI will behave closely to /u1D462 EI , while being slightly more explorative due to positivity.

Computing the EULBO and its derivatives now requires the computation of /u1D53C /u1D453 ∼/u1D45E /u1D740 (/u1D453 ) log /u1D462 SEI (/u1D499, /u1D453 ; /u1D49F /u1D461 ) , which, unlike EI, does not have a closed-form. However, since the utility function only depends on the function values of /u1D453 , the expectation can be efficiently computed to high precision through one-dimensional Gauss-Hermite quadrature. Crucially, the expensive /u1D43E -1 /u1D467/u1D467 /u1D45A and /u1D43E -1 /u1D467/u1D467 /u1D446 /u1D43E -1 /u1D467/u1D467 solves that dominate both the asymptotic and practical running time of both the ELBO and the EULBO are fixed across the log utility evaluations needed by quadrature. Because quadrature only depends on these precomputed moments, the additional work necessary due to lacking a closed form solution is negligible: Gauss-Hermite quadrature converges extremely quickly in the number of quadrature sites, and only requires on the order of 10 or so of these post-solve evaluations to achieve near machine precision.

3.3 EULBO for Knowledge Gradient (KG)

Although non-trivial, the KG acquisition is also a posterior-expected utility, where the underlying utility function is given by the maximum predictive mean value anywhere in the input domain after

conditioning on a new observation (/u1D499, /u1D466 ) ∈ /u1D4B3 × /u1D4B4 :

Note that the utility function as defined above is not non-negative: the maximum predictive mean of a Gaussian process can be negative. For this reason, the utility function is commonly (and originally, e.g. Frazier, 2009, Eq. 4.11) written in the literature as the difference between the new maximum mean after conditioning on (/u1D499, /u1D466 ) and the maximum mean beforehand:

where /u1D707 + /u1D461 ≜ max/u1D499 ′′ ∈/u1D4B3 /u1D438 [ /u1D453 (/u1D499 ′′ ) ∣ /u1D49F /u1D461 ] . Note that /u1D707 + /u1D461 plays the role of a simple constant as it depends on neither /u1D499 nor /u1D466 . Similarly to the EI acquisition, this utility is still not strictly positive, and we thus define its 'softplus-ed' variant:

Here, /u1D450 + acts as /u1D707 + /u1D461 by making /u1D462KG positive as often as possible. This is particularly important when the GP predictive mean is negative as a consequence of the objective values being negative. One natural choice of constant is using /u1D707 + /u1D461 ; however, we find that simply choosing /u1D450 + = /u1D466 + /u1D461 works well and is more computationally efficient. Here, /u1D466 + /u1D461 is the highest value of /u1D466 /u1D461 (the highest objective value observed so far).

One-Shot KG EULBO . The EULBO using /u1D462SKG results in an expensive nested optimization problem. To address this, we use an approach similar to the one-shot knowledge gradient method of Balandat et al. (2020). For clarity, we will define the reparameterization function

where, for an i.i.d. sample /u1D716 /u1D456 ∼ /u1D4A9 (0, 1) , computing /u1D466 /u1D456 = /u1D466 /u1D740 (/u1D499, /u1D716 /u1D456 ) is equivalent to sampling /u1D466 /u1D456 ∼ /u1D4A9 ( /u1D707 /u1D45E /u1D706 (/u1D499), /u1D70E /u1D45E /u1D706 (/u1D499) ) . This enables the use of the reparameterization gradient estimator (Kingma and Welling, 2014; Rezende et al., 2014; Titsias and Lázaro-Gredilla, 2014). Now, notice that the KG acquisition function can be approximated through Monte Carlo as

where, for /u1D456 = 1, … , /u1D446 , /u1D716 /u1D456 ∼ /u1D4A9(0, 1) are i.i.d. The one-shot KG approach absorbs the nested optimization over /u1D499 ′ into a simultaneous joint optimization over /u1D499 and a mean maximizer for each of the S samples, /u1D499 ′ 1 , ..., /u1D499 ′ /u1D446 such that max/u1D499 /u1D6FC KG(/u1D499; /u1D49F /u1D461 ) ≈ max /u1D499,/u1D499 ′ 1 ,...,/u1D499 ′ /u1D446 /u1D6FC 1-KG (/u1D499; /u1D49F), where

Evidently, there is no longer an inner optimization problem over /u1D499 ′ . To estimate the /u1D456 th term of this sum, we draw a sample of the objective value of /u1D499 , /u1D466 /u1D740 (/u1D499; /u1D716 /u1D456 ) , and condition the model on this sample. We then compute the new posterior predictive mean at /u1D499 ′ /u1D456 . After summing, we compute gradients with respect to both the candidate /u1D499 and the mean maximizers /u1D499 ′ 1 , ..., /u1D499 ′ /u1D446 . Again, we use the 'soft' version of one-shot KG in our EULBO optimization problem:

where this utility function is crucially a function of both /u1D499 and a free parameter /u1D499 ′ . As with /u1D6FC 1-KG , maximizing the EULBO can be set up as a joint optimization problem:

Efficient KGEULBO Computation. The computation time of the non-ELBO term in Eq. (10) is dominated by having to compute /u1D53C [ /u1D453 (/u1D499 ′ /u1D456 ) ∣ /u1D49F /u1D461 ∪ {(/u1D499, /u1D466 /u1D740 (/u1D499; /u1D716 /u1D456 ))} ] /u1D446 -times. Notice that we only need to compute an updated posterior predictive mean, and can ignore predictive variances. For this, we can leverage the online updating strategy of Maddox et al. (2021). In particular, the predictive mean can be updated in /u1D4AA(/u1D45A 2 ) time using a simple Cholesky update. The additional /u1D4AA(/u1D446/u1D45A 2 ) cost of computing the EULBO is therefore amortized by the original /u1D4AA(/u1D45A 3 ) cost of computing the ELBO.

3.4 Extension to q-EULBO for Batch Bayesian Optimization

The EULBO can be extended to support batch Bayesian optimization by using the Monte Carlo batch mode analogs of utility functions as discussed e.g. by Balandat et al. (2020); Wilson et al. (2018). Given a set of candidates /u1D47F = ( /u1D499 1 , ..., /u1D499 /u1D45E ) ∈ /u1D4B3 /u1D45E , the /u1D45E -EI utility function is given by:

This utility can again be softened as:

Because this is now a /u1D45E -dimensional integral, Gauss-Hermite quadrature is no longer applicable. However, we can apply Monte Carlo as

As done in the BoTorch software package (Balandat et al., 2020), we observe that fixing the set of base samples /u1D716 1 , ..., /u1D716 /u1D446 during each BO iteration results in better optimization performance at the cost of negligible q-EULBO bias. Now, optimizing the q-EULBO is done over the full set of /u1D45E candidates ( /u1D499 1 , ..., /u1D499 /u1D45E ) jointly, as well as the GP hyperparameters, inducing points, and variational parameters.

Knowledge Gradient. The KG version of the EULBO can be similarly extended. The expected log utility term in the maximization problem Eq. (10) becomes:

resulting in a similar analog to q-KG as described by Balandat et al. (2020).

3.5 Optimizing the EULBO

Optimizing the ELBO for SVGPs is known to be challenging (Galy-Fajou and Opper, 2021; Terenin et al., 2024) as the optimization landscape for the inducing points is non-convex, multi-modal, and non-smooth. Naturally, these are also challenges for EULBO ; we found that care must be taken when implementing and initializing the EULBO maximization problem. In this subsection, we outline some key ideas, while a detailed description with pseudocode is presented in Appendix A.

Initialization and Warm-Starting. We warm-start the EULBO maximization procedure by solving the conventional two-step scheme in Eq. (5): At each BO iteration, we obtain the 'warm' initial values for (/u1D740, /u1D481 , /u1D73D ) by optimizing the standard ELBO. Then, we use this to maximize the conventional acquisition function corresponding to the chosen utility function /u1D462 (the expectation of /u1D462 over /u1D45E /u1D740 (/u1D453) ), which provides the warm-start initialization for /u1D499 .

Alternating Maximization Scheme. To optimize ℒEULBO (/u1D499, /u1D740, /u1D481 , /u1D73D) , we alternate between optimizing over the query /u1D499 and the SVGP parameters /u1D740, /u1D481 , /u1D73D . We find this block-coordinate descent scheme to be more stable and robust than jointly updating all parameters, though the reason why this is more stable than jointly optimizing all parameters requires further investigation.

4 Experiments

We evaluate EULBO -based SVGPs on a number of benchmark BO tasks, described in detail in Section 4.1. These tasks include standard low-dimensional BO problems, e.g., the 6D Hartmann function, as well as 7 high-dimensional and high-throughput optimization tasks.

Baselines. We compare EULBO to several baselines with the main goal of achieving a high reward using as few function evaluations as possible. Our primary point of comparison is ELBO-based SVGPs. We consider two approaches for inducing point locations: 1. optimizing inducing point locations via the ELBO (denoted as ELBO ), 2. placing the inducing points using the strategy proposed by Moss et al. (2023) at each stage of ELBO optimization (denoted as Moss et al. ). The latter offers improved BO performance over standard ELBO-SVGP in BO settings, yet-unlike our method-it exclusively

targets inducing point placement and does not affect variational parameters or hyperparameters of the model. In addition, we compare to BO using exact GPs using 2, 000 function evaluations as the use of exact GP is intractable beyond this point due to the need to repeatedly fit models.

Acquisition Functions and BO algorithms. For EULBO , we test the versions based on both the Expected Improvement (EI) and Knowledge Gradient (KG) acquisition functions as well as the batch variant. We test the baseline methods using EI only. On high-dimensional tasks (tasks with dimensionality above 10), we run EULBO and baseline methods with standard BO and with trust region Bayesian optimization ( TuRBO ) (Eriksson et al., 2019). For the largest tasks (Lasso, Molecules) we use acquisition batch size of 20 ( /u1D45E = 20 ), and batch size 1 ( /u1D45E = 1 ) for all others.

Implementation Details and Hyperparameters. Code to reproduce all results in the paper is available at https://github.com/nataliemaus/aabo . We implement EULBO and baseline methods using the GPyTorch (Gardner et al., 2018) and BoTorch (Balandat et al., 2020) packages. For all methods, we initialize using a set of 100 data points sampled uniformly at random in the search space. We use the same trust region hyperparameters as in (Eriksson et al., 2019). In Appendix B.1, we also evaluate an additional initialization strategy for the molecular design tasks. This alternative initialization matches prior work in using 10, 000 molecules from the GuacaMol dataset Brown et al. (2019) rather than the details we used above for consistency across tasks, but does achieve higher overall performance.

4.1 Tasks

Hartmann 6D. The widely used Hartmann benchmark function (Surjanovic and Bingham, 2013).

Lunar Lander. The goal of this task is to find an optimal 12 -dimensional control policy that allows an autonomous lunar lander to consistently land without crashing. The final objective value we optimize is the reward obtained by the policy averaged over a set of 50 random landing terrains. For this task, we use the same controller setup used by Eriksson et al. (2019).

Rover. The rover trajectory optimization task introduced by Wang et al. (2018) consists of finding a 60 -dimensional policy that allows a rover to move along some trajectory while avoiding a set of obstacles. We use the same obstacle set up as in Maus et al. (2023).

Lasso DNA. We optimize the 180dimensional DNA task from the LassoBench library (Šehić et al., 2022) of benchmarks based on weighted LASSO regression (Gasso et al., 2009).

Molecular design tasks (x4). We select four challenging tasks from the Guacamol benchmark suite of molecular design tasks (Brown et al., 2019): Osimertinib MPO, Fexofenadine MPO, Median Molecules 1, and Median Molecules 2. We use the SELFIES-VAE introduced by Maus et al. (2022) to enable continuous 256 dimensional optimization.

4.2 Optimization Results

In Figure 2, we plot the reward of the best point found by the optimizer after a given number of function evaluations. Error bars show the standard error of the mean over 20 replicate runs. EULBO with TuRBO outperforms the other baselines with TuRBO . Similarly, EULBO with standard BO outperforms the other standard BO baselines. One noteworthy observation is that neither acquisition function appears to consistently outperform the other. However, EULBO -SVGP almost always dominates ELBO-SVGP and often requires a small fraction of the number of oracle calls to achieve comparable performance. These results suggest that coupling data acquisition with approximate inference/model selection results in significantly more sample-efficient optimization.

4.3 Ablation Study

While the results in Fig. 2 demonstrate that EULBO -SVGP improves the BO performance it is not immediately clear to what extent joint optimization modifies the posterior approximation beyond what is obtained by standard ELBO optimization. To that end, in Fig. 3 we refine an ELBO-SVGP model with varying degrees of additional EULBO optimization. At every BO iteration we begin by obtaining a SVGP model (where the variational parameters, inducing point locations, and GP hyperparameters are all obtained by optimizing the standard ELBO objective). We then refine some subset of parameters (either the inducing points, the variational parameters, the GP hyperparameters, or all of the above) through additional optimization with respect to the EULBO objective. Interestingly, we find that tasks respond differently to the varying levels of EULBO refinement. In the case of Lasso DNA, there is not much of a difference between EULBO refinement on all parameters versus refinement on the variational parameters alone. On the other hand, the performance on Median Molecules 2 is clearly dominated by refinement on all parameters. Nevertheless, we see that EULBO is always beneficial, whether applied to all parameters or some subset.

5 Related Work

Scaling Bayesian Optimization to the Large-Budget Regime. BO has traditionally been confined to the small-budget optimization regime with a few hundred objective evaluations at most. However, recent interest in high-dimensional optimization problems has demonstrated the need to scale BO to large data acquisition budgets. For problems with ∼10 3 data acquisitions, Hernández-Lobato et al. (2017); Snoek et al. (2015); Springenberg et al. (2016) consider Bayesian neural networks (BNN; Neal, 1996), McIntire et al. (2016) use SVGP, and Wang et al. (2018) turn to ensembles of

subsampled GPs. For problems with ≫10 3 acquisitions, SVGP has become the de facto approach to alleviate computational complexity (Griffiths and Hernández-Lobato, 2020; Maus et al., 2022, 2023; Stanton et al., 2022; Tripp et al., 2020; Vakili et al., 2021). As in this paper, many works have proposed modifications to SVGP to improve its performance in BO applications. Moss et al. (2023) proposed an inducing point placement based on a heuristic modification of determinantal point processes (Kulesza and Taskar, 2012), which we used for initialization, while Maddox et al. (2021) proposed a method for a fast online update strategy for SVGPs, which we utilize for the KG acquisition strategy.

Utility-Calibrated Approximate Inference. The utility-calibrated VI objective was first proposed by Lacoste-Julien et al. (2011), where they used a coordinate ascent algorithm to maximize it. Since then, various extensions have been proposed: Kuśmierczyk et al. (2019) leverage black-box variational inference (Ranganath et al., 2014; Titsias and Lázaro-Gredilla, 2014); Morais and Pillow (2022) use expectation-propagation (EP; Minka, 2001); Abbasnejad et al. (2015) and Rainforth et al. (2020) employ importance sampling; Cobb et al. (2018) and Li and Zhang (2023) derive a specific variant for BNNs; and (Wei et al., 2021) derive a specific variant for GP classification. Closest to our work is the GP-based recommendation model learning algorithm by Abbasnejad et al. (2013), which sparsifies an EP-based GP approximation by maximizing a utility similar to those used in BO.

6 Limitations and Discussion

The main limitation of our proposed approach is increased computational cost. While EULBO -SVGP still retains the /u1D442(/u1D45A 3 ) computational complexity of standard SVGP, our practical implementation requires a warm-start: first fitting SVGP with the ELBO loss and then maximizing the acquisition function before jointly optimizing with the EULBO loss. Furthermore, EULBO optimization currently requires multiple tricks such as clipping and block-coordinate updates. In future work, we aim to develop a better understanding of the EULBO geometry in order to develop developing more stable, efficient, and easy-to-use EULBO optimization schemes. Nevertheless, our results in Section 4 demonstrate that the additional computation of EULBO yields substantial improvements in BO dataefficiency, a desirable trade-off in many applications. Moreover, EULBO -SVGP is modular, and our experiments capture a fraction of its potential use. It can be applied to any decision-theoretic acquisition function, and it is likely compatible with non-standard Bayesian optimization problems such as cost-constrained BO (Snoek et al., 2012), causal BO (Aglietti et al., 2020), and many more.

More importantly, our paper highlights a new avenue for research in BO, where surrogate modeling, approximate inference, and data selection are jointly determined from a unified objective. Extending this idea to GP approximations beyond SVGP and acquisition functions beyond EI/KG may yield further improvements, especially in the increasingly popular high-throughput BO setting.

Acknowledgments and Disclosure of Funding

The authors thank the anonymous reviewers for suggestions that improved the quality of the work.

N. Maus was supported by the National Science Foundation Graduate Research Fellowship; K. Kim was supported by a gift from AWS AI to Penn Engineering's ASSET Center for Trustworthy AI; G. Pleiss was supported by NSERC and the Canada CIFAR AI Chair program; J. P. Cunningham was supported by the Gatsby Charitable Foundation (GAT3708), the Simons Foundation (542963), the NSF AI Institute for Artificial and Natural Intelligence (ARNI: NSF DBI 2229929), and the Kavli Foundation; J. R. Gardner was supported by NSF awards IIS-2145644 and DBI-2400135.

References

Ehsan Abbasnejad, Justin Domke, and Scott Sanner. Loss-calibrated Monte Carlo action selection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29 of AAAI . AAAI Press, March 2015. (page 10)

Ehsan Abbasnejad, Edwin V. Bonilla, and Scott Sanner. Decision-theoretic sparsification for Gaussian process preference learning. In Machine Learning and Knowledge Discovery in Databases , volume 13717 of LNCS , pages 515-530, Berlin, Heidelberg, 2013. Springer. (page 10)

Virginia Aglietti, Xiaoyu Lu, Andrei Paleyes, and Javier González. Causal Bayesian optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 108 of PMLR , pages 3155-3164. JMLR, June 2020. (page 10)

Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. In Advances in Neural Information Processing Systems , volume 33, pages 21524-21538. Curran Associates, Inc., 2020. (pages 2, 6, 7, 8, 16)

G. Bissiri, C. C. Holmes, and S. G. Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology , 78(5):1103-1130, 2016. (pages 2, 5)

David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, April 2017. (pages 2, 3)

Nathan Brown, Marco Fiscato, Marwin H.S. Segler, and Alain C. Vaucher. Guacamol: Benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling , 59(3): 1096-1108, Mar 2019. (pages 8, 9)

Adam D. Cobb, Stephen J. Roberts, and Yarin Gal. Loss-Calibrated Approximate Inference in Bayesian Neural Networks. arXiv Preprint arXiv:1805.03901, arXiv, May 2018. (page 10)

P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EMalgorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1-22, September 1977. (page 4)

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems , volume 32, pages 5496-5507. Curran Associates, Inc., 2019. (pages 1, 2, 8)

Peter I Frazier. Knowledge-gradient methods for statistical learning . PhD thesis, Princeton University Princeton, 2009. (page 6)

Peter I Frazier. A tutorial on Bayesian optimization. arXiv Preprint arXiv:1807.02811, ArXiv, 2018. (page 1)

Théo Galy-Fajou and Manfred Opper. Adaptive inducing points selection for Gaussian processes. arXiv Preprint arXiv:2107.10066, arXiv, 2021. (page 7)

Jacob Gardner, Geoff Pleiss, Kilian Q. Weinberger, David Bindel, and Andrew G. Wilson. GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems , volume 31, pages 7576-7586. Curran Associates, Inc., 2018. (pages 8, 16)

Roman Garnett. Bayesian Optimization . Cambridge University Press, Cambridge, United Kingdom ; New York, NY, 2023. (pages 1, 2, 3, 6)

Gilles Gasso, Alain Rakotomamonjy, and Stéphane Canu. Recovering sparse signals with a certain family of nonconvex penalties and DC programming. IEEE Transactions on Signal Processing , 57 (12):4686-4698, 2009. (page 9)

Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained Bayesian optimization for automatic chemical design using variational autoencoders. Chemical Science , 11(2):577-586, 2020. (pages 1, 10)

James Hensman, Nicolo Fusi, and Neil D. Lawrence. Gaussian processes for big data. In Proceedings of the Conference on Uncertainty in Artificial Intelligence , pages 282-290. AUAI Press, 2013. (pages 1, 3)

José Miguel Hernández-Lobato, James Requeima, Edward O. Pyzer-Knapp, and Alán Aspuru-Guzik. Parallel and distributed Thompson sampling for large-scale accelerated exploration of chemical space. In Proceedings of the International Conference on Machine Learning , volume 70 of PMLR , pages 1470-1479. JMLR, July 2017. (page 9)

Prateek Jaiswal, Harsha Honnappa, and Vinayak A. Rao. Asymptotic consistency of loss-calibrated variational Bayes. Stat , 9(1):e258, 2020. (pages 2, 5)

Prateek Jaiswal, Harsha Honnappa, and Vinayak Rao. On the statistical consistency of risk-sensitive bayesian decision-making. In Advances in Neural Information Processing Systems , volume 36, pages 53158-53200. Curran Associates, Inc., December 2023. (pages 2, 5)

Martin Jankowiak, Geoff Pleiss, and Jacob R. Gardner. Parametric gaussian process regressors. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020. (page 19)

Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization , 13(4):455-492, 1998. (pages 1, 2, 5)

Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning , 37(2):183-233, 1999. (pages 1, 2, 3, 4)

Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations , San Diego, California, USA, 2015. (pages 15, 16)

Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations , Banff, AB, Canada, April 2014. (page 6)

Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centric view on Bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research , 23 (132):1-109, 2022. (pages 2, 5)

Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. Foundations and Trends ® in Machine Learning , 5(2-3):123-286, 2012. (page 10)

Tomasz Kuśmierczyk, Joseph Sakaya, and Arto Klami. Variational Bayesian decision-making for continuous utilities. In Advances in Neural Information Processing Systems , volume 32, pages 6395-6405. Curran Associates, Inc., 2019. (pages 4, 5, 10)

Simon Lacoste-Julien, Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the loss-calibrated Bayesian. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 15 of PMLR , pages 416-424. JMLR, June 2011. (pages 2, 4, 10)

Kenneth Lange. MM Optimization Algorithms . Society for Industrial and Applied Mathematics, Philadelphia, 2016. (pages 2, 4)

Bolian Li and Ruqi Zhang. Long-tailed Classification from a Bayesian-decision-theory Perspective. arXiv Preprint arXiv:2303.06075, arXiv, 2023. (page 10)

Wesley J Maddox, Samuel Stanton, and Andrew G Wilson. Conditioning sparse variational Gaussian processes for online decision-making. In Advances in Neural Information Processing Systems , volume 34, pages 6365-6379. Curran Associates, Inc., 2021. (pages 1, 2, 4, 6, 10)

Alexander G. de G. Matthews, James Hensman, Richard Turner, and Zoubin Ghahramani. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 51 of PMLR , pages 231-239. JMLR, May 2016. (pages 1, 4)

Natalie Maus, Haydn Jones, Juston Moore, Matt J. Kusner, John Bradshaw, and Jacob Gardner. Local latent space Bayesian optimization over structured inputs. In Advances in Neural Information Processing Systems , volume 35, pages 34505-34518, December 2022. (pages 1, 9, 10)

Natalie Maus, Kaiwen Wu, David Eriksson, and Jacob Gardner. Discovering many diverse solutions with Bayesian optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 206, pages 1779-1798. PMLR, April 2023. (pages 1, 8, 10)

Mitchell McIntire, Daniel Ratner, and Stefano Ermon. Sparse Gaussian Processes for Bayesian Optimization. In Proceedings of the Conference on Uncertainty in Artificial Intelligence , Jersey City, New Jersey, USA, 2016. AUAI Press. (page 9)

Thomas P. Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the Conference on Uncertainty in Artificial Intelligence , pages 362-369, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. (page 10)

Jonas Mockus. The Bayesian approach to global optimization. In System Modeling and Optimization , pages 473-481. Springer, 1982. (page 1)

Michael J. Morais and Jonathan W. Pillow. Loss-calibrated expectation propagation for approximate Bayesian decision-making. Technical Report arXiv:2201.03128, arXiv, January 2022. (page 10)

Henry B. Moss, Sebastian W. Ober, and Victor Picheny. Inducing point allocation for sparse Gaussian processes in high-throughput Bayesian optimisation. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 206 of PMLR , pages 5213-5230. JMLR, April 2023. (pages 1, 4, 7, 10, 16, 17, 18)

Radford M. Neal. Bayesian Learning for Neural Networks , volume 118 of Lecture Notes in Statistics . Springer New York, New York, NY, 1996. (page 9)

Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research , 6(65):1939-1959, 2005. (page 1)

Tom Rainforth, Adam Golinski, Frank Wood, and Sheheryar Zaidi. Targetâ = C'aware bayesian inference: How to beat optimal conventional estimators. Journal of Machine Learning Research , 21(88):1-54, 2020. URL http://jmlr.org/papers/v21/19-102.html . (page 10)

Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 33 of PMLR , pages 814-822. JMLR, April 2014. (page 10)

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning . The MIT Press, November 2005. (page 1)

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the International Conference on Machine Learning , volume 32 of PMLR , pages 1278-1286. JMLR, June 2014. (page 6)

Christian P. Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation . Springer Texts in Statistics. Springer, New York Berlin Heidelberg, 2. ed edition, 2001. (pages 2, 3)

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 104(1): 148-175, 2015. (pages 1, 5)

Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems , volume 18, pages 1257-1264. MIT Press, 2005. (page 5)

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. Advances in neural information processing systems , 25:2951-2959, 2012. (page 10)

Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian optimization using deep neural networks. In Proceedings of the International Conference on Machine Learning , volume 37 of PMLR , pages 2171-2180. JMLR, June 2015. (page 9)

Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian Optimization with Robust Bayesian Neural Networks. In Advances in Neural Information Processing Systems , volume 29, pages 4134-4142. Curran Associates, Inc., 2016. (page 9)

Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In Proceedings of the International Conference on Machine Learning , volume 162 of PMLR , pages 20459-20478. JMLR, June 2022. (pages 1, 10)

Sonja Surjanovic and Derek Bingham. Virtual library of simulation experiments: Test functions and datasets, 2013. (page 8)

Alexander Terenin, David R. Burt, Artem Artemev, Seth Flaxman, Mark van der Wilk, Carl Edward Rasmussen, and Hong Ge. Numerically stable sparse Gaussian processes via minimum separation using cover trees. Journal of Machine Learning Research , 25(26):1-36, 2024. (page 7)

Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 5 of PMLR , pages 567-574. JMLR, April 2009. (pages 1, 3)

Michalis Titsias and Miguel Lázaro-Gredilla. Doubly stochastic variational Bayes for non-conjugate inference. In Proceedings of the International Conference on Machine Learning , volume 32 of PMLR , pages 1971-1979. JMLR, June 2014. (pages 6, 10)

Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems , volume 33, pages 11259-11272. Curran Associates, Inc., 2020. (pages 1, 10)

Sattar Vakili, Henry Moss, Artem Artemev, Vincent Dutordoir, and Victor Picheny. Scalable Thompson sampling using sparse Gaussian process models. In Advances in Neural Information Processing Systems , volume 34, pages 5631-5643, 2021. (pages 1, 10)

Kenan Šehić, Alexandre Gramfort, Joseph Salmon, and Luigi Nardi. Lassobench: A high-dimensional hyperparameter optimization benchmark suite for LASSO. In Proceedings of the International Conference on Automated Machine Learning , volume 188 of PMLR , pages 2/1-24. JMLR, 25-27 Jul 2022. (page 9)

Yixin Wang and David M. Blei. Frequentist consistency of variational Bayes. Journal of the American Statistical Association , 114(527):1147-1161, July 2019. (page 5)

Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian optimization in high-dimensional spaces. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 84 of PMLR , pages 745-754. JMLR, March 2018. (pages 8, 9)

Larry Wasserman. All of statistics: a concise course in statistical inference . Springer Science & Business Media, 2013. (pages 2, 3)

Yadi Wei, Rishit Sheth, and Roni Khardon. Direct loss minimization for sparse Gaussian processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 130 of PMLR , pages 2566-2574. JMLR, March 2021. (page 10)

James Wilson, Frank Hutter, and Marc Deisenroth. Maximizing acquisition functions for Bayesian optimization. In Advances in Neural Information Processing Systems , pages 9884-9895. Curran Associates, Inc., 2018. (pages 2, 7)

Jian Wu, Matthias Poloczek, Andrew G Wilson, and Peter Frazier. Bayesian optimization with gradients. In Advances in Neural Information Processing Systems , volume 30, pages 5267-5278. Curran Associates, Inc., 2017. (page 2)

A Implementation Details

We will now provide additional details on the implementation. For the implementation, we treat the SVGP parameters, such as the variational parameters /u1D740 , inducing point locations /u1D481 , and hyperparameters /u1D73D , equally. Therefore, for clarity, we will collectively denote them as /u1D498 = (/u1D740, /u1D481 , /u1D73D ) such that /u1D498 ∈ /u1D4B2 ≜ Λ × /u1D4B3 /u1D45A × Θ , and the resulting SVGP variational approximation as /u1D45E /u1D498 . Then, the ELBO and EULBO are equivalently denoted as follows:

Also, notice that the ℒEULBO separately denote the dataset to be passed to the utility and the ELBO. (Setting /u1D49F /u1D461 = /u1D49F /u1D498 = /u1D49F /u1D499 retrieves the original formulation in Eq. (8).)

Alternating Updates We perform block-coordinate ascent on the EULBO by alternating between maximizing over /u1D499 as /u1D498 . Using vanilla gradient descent, the /u1D499 -update is equivalent to

where /u1D6FE /u1D499 is the stepsize. On the other hand, for the /u1D498 -update, we subsample the data such that we optimize the ELBO over a minibatch /u1D446 ⊂ /u1D49F of size /u1D435 = |/u1D446| as

where /u1D6FE /u1D498 is the stepsize. Naturally, the /u1D498 -update is stochastic due to minibatching, while the /u1D499 -update is deterministic. In practice, we leverage the Adam update rule (Kingma and Ba, 2015) instead of simple gradient descent. Together with gradient clipping, this alternating update scheme is much more robust than jointly updating (/u1D499, /u1D498) .

Algorithm 1: EULBO Maximization Policy

Overview of Pseudocode. The complete high-level view of the algorithm is presented in Algorithm 1, except for the acquisition-specific details. AdamStep /u1D6FE (/u1D499, /u1D488) applies the Adam stepsize rule (Kingma and Ba, 2015) to the current location /u1D499 with the gradient estimate /u1D488 and the stepsize /u1D6FE . In practice, Adam is a 'stateful' optimizer, which maintains two scalar-valued states for each scalar parameter. For this, we re-initialize the Adam states at the beginning of each BO step.

Initialization. In the initial BO step /u1D461 = 0 , we initialize /u1D481 0 with the DPP-based inducing point selection strategy of Moss et al. (2023). For the remaining SVGP parameters /u1D740 0 and /u1D73D 0 , we used the default initialization of GPyTorch (Gardner et al., 2018). For the remaining BO steps /u1D461 > 0 , we use /u1D498 from the previous BO step as the initialization /u1D498 0 of the current BO step.

Warm-Starting. Due to the non-convexity and multi-modality of both the ELBO and the acquisition function, it is critical to appropriately initialize the EULBO maximization procedure. As mentioned in Section 3.5, to warm-start the EULBO maximization procedure, we use the conventional 2-step scheme Eq. (5), where we maximize the ELBO and then maximize the acquisition function. For ELBO maximization, we apply Adam (Kingma and Ba, 2015) with the stepsize set as /u1D6FE /u1D464 until the convergence criteria (described below) are met. For acquisition function maximization, we invoke the highly optimized BoTorch.optimize.optimize_acqf function (Balandat et al., 2020).

Minibatch Subsampling Strategy. As commonly done, we use the reshuffling subsampling strategy where the dataset /u1D49F /u1D461 is shuffled and partitioned into minibatches of size /u1D435 . The number of minibatches constitutes an 'epoch.' The dataset is reshuffled/repartitioned after going through a full epoch.

ConvergenceDetermination. For both maximizing the ELBO during warm-starting and maximizing the EULBO, we continue optimization until we stop making progress or exceed /u1D458 epochs number of epochs. That is if the ELBO/EULBO function value fails to make progress for /u1D45B f ail number of steps.

Hyperparameters. The hyperparameters used in our experiments are organized in Table 1. For the full-extent of the implementation details and experimental configuration, please refer to the supplementary code.

B Additional Plots

We provide additional results and plots that were omitted from the main text.

B.1 Additional Results on Molecule Tasks

In Fig. 4, we provide plots on additional results that are similar to those in Fig. 2. On three of the molecule tasks, we use 10,000 random molecules from the GuacaMol dataset as initialization. This is more consistent with what has been done in previous works and achieves better overall optimization performance.

B.2 Separate Plots for BO and TuRBO Results

In this section, we provide additional plots separating out BO and TuRBO results to make visualization easier.

B.3 Effect of Number of Inducing Points

For the results with approximate-GPs in Section 4, we used /u1D45A = 100 inducing points. In Fig. 7, we evaluate the effect of using a larger number of inducing points ( /u1D45A = 1024 ) for EULBO -SVGP and ELBO-SVGP.

0

5000

10000

15000

20000

Number of Oracle Calls

0.34

0.33

0.32

0.31

0.30

0.29

Mean Reward

TuRBO (EULBO EI) w/ 1024 inducing points

TuRBO (EULBO EI)

TuRBO (ELBO EI) w/ 1024 inducing points

TuRBO (ELBO EI)

Fig. 7 shows that the number of inducing points has limited impact on the overall performance of TuRBO , and EULBO -SVGP outperforms ELBO-SVGP regardless of the number of inducing points used.

B.4 Effect of GP Objective

The results in Section 4 used a standard SVGP objective. In this section, we evaluate the effect of using an alternative objective: the parametric Gaussian process regressor (PPGPR; Jankowiak et al., 2020) objective. PPGPR differs from the standard SVGP objective in that the variational approximation is optimized to maximize the predictive accuracy instead of matching the posterior.

0

5000

10000

15000

20000

Number of Oracle Calls

0.34

0.33

0.32

0.31

0.30

0.29

Mean Reward

TuRBO (EULBO EI) PPGPR

TuRBO (EULBO EI)

TuRBO (ELBO EI) PPGPR

TuRBO (ELBO EI)

We compare the choice of objective (PPGPR vs SVGP) in Fig. 8 and observe that the objective has limited impact on the overall performance of TuRBO . In particular, EULBO -EI outperforms ELBO-EI regardless of the GP objective.

C Compute Resources

Type of Compute and Memory. All results in the paper required the use of GPU workers (one GPU per run of each method on each task). The majority of runs were executed on an internal cluster, where details are shown in Table 2, where each node was equipped with an NVIDIA RTX A5000 GPU. In addition, we used cloud compute resources for a short period leading up to the subsmission of the paper. We used 40 RTX 4090 GPU workers from runpod.io , where each GPU had approximately 24 GB of GPU memory. While we used 24 GB GPUs for our experiments, each run of our experiments only requires approximately 15 GB of GPU memory.

Execution Time. Each optimization run for non-molecule tasks takes approximately one day to finish. Since we run the molecule tasks out to a much larger number of function evaluations than other tasks ( 80000 total function evaluations for each molecule optimization task), each molecule optimization task run takes approximately 2 days of execution time. With all eight tasks, ten methods run, and 20 runs completed per method, results in Fig. 2 include 1600 total optimization runs ( 800 for molecule tasks and 800 for non-molecule tasks). Additionally, the two added curves in each plot in Fig. 3 required 160 additional runs ( 120 for molecule tasks and 40 for non-molecule task). Completing all of the runs needed to produce all of the results in this paper therefore required roughly 2680 total GPU hours.

Compute Resources Used During Preliminary Investigations. In addition to the computational resources required to produce experimental results in the paper discussed above, we spent approximately 500 hours of GPU time on preliminary investigations. This was done on the aforementioned internal cluster shown in Table 2.

D Wall-clock Run Times

In Table 3, we provide average wall-clock run times of different methods on the Lasso DNA optimization task.

Table 3: Average wall-clock run times for one full run of TuRBO on the Lasso DNA task. We compare the average wall-clock run time of TuRBO on all TuRBO methods from Figure 2. Note that we do not include the wall clock run time for TuRBO with Exact EI here because we only ran this method out to 2k oracle calls (rather than the full budget of 20k oracle calls).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All stated claims are backed-up with results in Section 4 and the stated focus/scope of the paper accurately reflects what is discussed throughout the rest of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA] .

Justification: This work does not contain a formal theoretical analysis.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All assumptions should be clearly stated or referenced in the statement of any theorems.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Theorems and Lemmas that the proof relies upon should be properly referenced.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: We provide detailed explanation of how our method works in Section 3 and all additional required details to reproduce results in Section 4 and Appendix A. Additionally, we have included a link to a public GitHub repository containing all of the source code used in the work in Section 4. This source code allows any reader to run our code to reproduce all results in the paper. Additionally, the README in the repository provides detailed instructions to make setting up the proper environment and running the code easy for users.

Guidelines:

The answer NA means that the paper does not include experiments.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] Replace by [Yes] , [No] , or [NA] .

Justification: We have included a link to a public GitHub repository containing all of the source code used in the work in Section 4. This source code allows any reader to run our code to reproduce all results in the paper. Additionally, the README in the repository provides detailed instructions to make setting up the proper environment and running the code easy for users.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All chosen hyper-parameters and implementation details are stated in section 4 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: On all plots, we plot the mean taken over multiple random runs and include error bars to show the standard error over the runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the NeurIPS Code of Ethics and made sure to adhere to them in all aspects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No] .

Justification: The paper is methodological, where the considered algorithm does not immediately pose societal risks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper does not use data with potential societal concerns.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All creators of assets used to produce our results are cited in Section 4. All assets used are open source software or models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should state which version of the asset is used and, if possible, include a URL.

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: The paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve human participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: The paper does not involve live participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an extension of the sparse Variational Gaussian Process framework (SVGP) that is better suited for Bayesian optimization (BO) tasks. This is achieved by a new optimization criterion (EULBO) which aims to optimize the parameters of SVGPs s.t. the training data is fit well while achieving a better performance in selecting new solution candidates. Two concrete instantiations of EULBO (with Expected Improvement and Knowledge Gradient) are derived. An extensive empirical evaluation is conducted to demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n- though being dense, the paper is clearly written and straightforward to follow\n- the proposed approach is technically solid and mathematically sound\n- a relevant problem is addressed by an elegant solution\n- both, methodology and experiments are presented well\n\nWEAKNESSES:\n## Methodology\n- there are some details unclear/missing regarding figure 1:\n\t- which points were used as inducing points?\n\t- why is the uncertainty relatively constant at the right side of the plot (left plot)? Depending on the choice of the inducing points, I'd expect the predicted variance (and thus the EI) to increase towards the right edge of the plot\n\t- what was the exact problem definition leading to Fig. 1? It would be good to see the exact parameterizations for reproducibility as it would help to gauge whether the motivational example is rather cherry-picked or whether the shown issue manifests across different parameterizations\n- missing $t$ in Eq. 2? shouldn't it be $\\pi(f | \\mathcal{D}_t)$?\n- in the equation after line 221 it seems that the sum does not depend on $j$. shouldn't it be $\\mathbf{x}_j$?\n\n## Experiments\n- why were the baselines only applied in conjunction with EI? (see line 259)\n- it seems that, depending on the choice of the objective and method, there are quite significant differences in the variance of EULBO. For example, in Fig. 2 \"Media Molecules 1\", TuRBO + ELBO EI has much higher variance than TuRBO + EULBO EI. Do you account this observation to EULBO?\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nthe authors discussed the limitations of their approach in a detailed waysssssssssssssssssssssss", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method for training sparse Gaussian processes (GPs) for large-budget Bayesian Optimization (BO), designed to facilitate the sequential decision tasks inherent in BO. This is achieved by introducing the “expected utility lower bound (EULBO),” which formulates sparse GP training as a joint optimization problem of BO queries and posterior approximation. The proposed framework is compatible with other significant advancements in the field, such as Expected Improvement (EI) and Knowledge Refinement (KR) acquisition functions, TuRBO, and batch optimization. This compatibility was demonstrated across several BO benchmarks.\n\nSTRENGTHS:\nThe method proposed in the paper is novel and original. One of the main strengths is how the newly proposed EULBO integrates with important points in the literature. The authors effectively demonstrate how EULBO can be used in various contexts, such as with EI, KR acquisitions, and batch optimization. This helps distinguish their work from others in the field. The submission is technically sound, with derivations and experiments convincingly supporting the claims of the paper. The work is significant, and I anticipate it will be extended in the future or that some of the ideas presented will be adopted elsewhere.\n\nWEAKNESSES:\nWhile the paper is generally well-written, many derivations were abbreviated and some of the notation was not properly explained, making the paper more difficult to read than necessary. These issues, however, are easily amendable. I provide some suggestions for improvement in the following subsections.\n\nQUESTIONS:\nIntroduction:  \n\n(line 51): '\\emph{minorize-maximize}'\n\nSection 2:\n\n(Fig. 1): It would be beneficial to show the SVGP model fit for EULBO as well.\n\n(line 110): Could you define \\lambda, m and S?\n\n(Eq. between 114-115): q_\\lambda is initially presented with one argument, then two. If the intention is for q_\\lambda to represent all approximate distributions  (in this case, p(f, u) ), please state this explicitly. It is natural to assume that q_\\lambda(.) is just the Gaussian pdf (as defined in line 110), which creates confusion.\n\n(line 116): k(.,.) as a function also needs to be explicitly defined.\n\nSection 3:\n\n(line 123): This should probably refer to the equation between lines 114-115, not Equation 3.\n\n(Eq 4): Please define \\Lambda.\n\n(Eq 6): Please use double integrals to indicate nested integration. Additionally, I don't understand the emergence of l(D_t |f) and the normalizing constant Z. I was under the impression that l(.|.) is the Gaussian likelihood as defined in line 113. Notice that D_t, as defined in line 92, contains both x and y.\n\n(line 147): 'corresponds to'\n\n(line 165): Please clarify where exactly the requirement for the utility to be strictly positive plays a role or comes from.\n\n(lines 173-175): Would using Gauss-Hermite quadrature for EI make EULBO significantly slower than ELBO?\n\n(line 189): What is y^{+}_t and what its relation to y_t?\n\nQuestion:\n\nBecause you formulate the BO queries and GP approximation as a joint optimization problem without necessarily reducing the dimensionality of the problem, how much slower is this method compared to ‘approximation-ignorant’ BO?\n\nLIMITATIONS:\nThe authors are open about the limitations of their work, which they leave for future exploration. Specifically, the increased computational complexity seems to be a primary limitation. It would be beneficial if the authors could quantify this increased complexity, perhaps through reporting empirical runtimes or other relevant metrics.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a modification to sparse variational Gaussian processes (SVGPs) used in Bayesian optimization (BO) to better align the SVGP posterior approximation with the goal of optimizing an acquisition function. The key idea is to jointly optimize the SVGP and the acquisition function using a unified objective called the expected utility lower bound (EULBO). This approach ensures the posterior approximation is well-suited for the downstream decision-making task. The authors derive efficient EULBO objectives for the expected improvement (EI) and knowledge gradient (KG) acquisition functions and demonstrate improved performance over standard SVGPs on several high-dimensional optimization benchmarks.\n\nSTRENGTHS:\nThe paper presents a novel approach to aligning SVGP approximations with the goals of BO by jointly optimizing the posterior approximation and acquisition function. This is a creative combination of ideas from variational inference and decision theory.\n\nThe proposed method is well-motivated and grounded in sound theoretical principles. The derivations of the EULBO objectives for EI and KG are clear and technically sound. The experiments are comprehensive, covering a range of high-dimensional optimization tasks, and the results convincingly demonstrate the effectiveness of the approach.\n\nThe paper is well-written and easy to follow. The authors provide a clear exposition of the problem, the proposed solution, and the experimental setup. The figures and tables effectively communicate the key results.\n\nScaling BO to high-dimensional spaces is an important problem, and this work represents a significant step towards more efficient and effective BO in such settings. The proposed approach is general and could potentially be applied to other acquisition functions and GP approximations, making it a valuable contribution to the field.\n\nWEAKNESSES:\nThe paper focuses specifically on SVGPs, but it would be interesting to explore whether the proposed approach can be extended to other sparse GP approximations, even those without a tractable ELBO. This would broaden the applicability of the method and strengthen the contribution.\n\nIn Figure 2, the exact GP performs worse than the proposed method and some other baselines in the BO setting. This is counterintuitive, as one would expect the exact GP to be the gold standard when computationally feasible. The authors should provide more discussion and insights into this unexpected result.\n\nThe paper presents several variations of the proposed method (e.g., EI-EULBO, KG-EULBO, batch versions), but it lacks a clear recommendation on which variant to use in practice. A more in-depth discussion of the trade-offs between these variants and guidance on when to use each one would enhance the practical value of the work.\n\nQUESTIONS:\nCan the proposed EULBO approach be extended to other sparse GP approximations beyond SVGPs, even if they do not have a tractable ELBO? If so, what challenges would need to be addressed, and how might the optimization problem be formulated?\n\nWhy does the exact GP perform worse than the proposed method and some other baselines in the BO experiments (Figure 2)? Is this due to the limitations of the exact GP in high dimensions, or are there other factors at play?\n\nWhat are the key factors to consider when choosing between the different variants of the proposed method (e.g., EI-EULBO, KG-EULBO, batch versions)? In what scenarios would one variant be preferred over another?\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work in Section 6, acknowledging the increased computational cost of the EULBO approach compared to standard SVGPs. They also mention the need for multiple optimization tricks and the potential instability of the EULBO optimization problem. These are important limitations that users should be aware of when considering this method.\n\nHowever, the authors do not discuss any potential negative societal impacts of their work. While the proposed method is primarily methodological and does not pose immediate societal risks, it would be valuable for the authors to briefly comment on any broader implications of improved BO in high-dimensional spaces, such as the potential for misuse or unintended consequences in certain application domains.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach for scaling Bayesian Optimisation to large datasets. Contrary to previous approaches, which fitted a sparse GP and optimised acquisition function independently, the paper proposes a method, which jointly optimises the variational parameters of sparse GP and searches for the next point to query. This is done by adding a utility term to ELBO. Authors discuss how this objective can be connected to generalised Bayesian Inference. Authors evaluate the proposed method on a number of benchmark with high number of datapoints and show it outperforms other baselines.\n\nSTRENGTHS:\n- The paper touches on a very important subject of scaling Bayesian Optimisation to large datasets. I find the proposed solution to be very elegant. I also think the connection to generalised Bayesian Inference is very interesting.\n\n- The method seems to be delivering a significant improvement in sample efficiency over baselines. I believe the choice of benchmarks is sufficiently diverse and the number of baselines compared against is sufficient.\n\n- The paper is well-written and I really like that authors share the details of the tricks they used for improved optimisation of EULBO. I also like the fact that authors are very open about admitting the limitations of the proposed method.\n\nWEAKNESSES:\n- As mentioned by the authors, optimisation of the EULBO objective is a bit cumbersome and seems this process is also slower than the optimisation of standard ELBO. However, it might be worth to sacrifice a bit of computational efficiency for an increase in the sample complexity.\n\n- While authors cite previous work, which proved that the selected action satisfies convergence guarantee, it does not directly prove anything about the performance of optimisation process choosing action in such a way (e.g. regret bound or convergence to optimum). The paper could be made much stronger if authors managed to connect those notions in some way. However, I also do not think it is critical, as the main contribution of the paper is empirical (although it would be greatly appreciated).\n\nQUESTIONS:\n- Can the authors provide the running time of the proposed method (and baselines) ? It is ok if the proposed method is slower, but it would be good to quantify exactly how much slower it is.\n\n- Do authors have any expectations on how would the method perform in comparison to an exact GP on a smaller dataset? While I understand that the method is particularly designed for large datasets, it would be interesting to see how the utility VI-based acquisition strategy compares to standard acquisition functions like EI. It would be nice to have at least such experiment in the camera-ready version to judge whether the improvement delivered by the method comes from better acquisition strategy or better modelling\n\nLIMITATIONS:\nAuthors admit that the increased difficulty of optimising EULBO is a limitation of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}