Title: Unexpected Improvements to Expected Improvement for Bayesian Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian Optimization

Gaussian Process

Multi-Objective Optimization
Abstract: Expected Improvement (EI) is arguably the most popular acquisition function in Bayesian optimization and has found countless successful applications, but its performance is often exceeded by that of more recent methods. Notably, EI and its variants, including for the parallel and multi-objective settings, are challenging to optimize because their acquisition values vanish numerically in many regions. This difficulty generally increases as the number of observations, dimensionality of the search space, or the number of constraints grow, resulting in performance that is inconsistent across the literature and most often sub-optimal. Herein, we propose LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically. We demonstrate that numerical pathologies manifest themselves in “classic” analytic EI, Expected Hypervolume Improvement (EHVI), as well as their constrained, noisy, and parallel variants, and propose corresponding reformulations that remedy these pathologies. Our empirical results show that members of the LogEI family of acquisition functions substantially improve on the optimization performance of their canonical counterparts and surprisingly, are on par with or exceed the performance of recent state-of-the-art acquisition functions, highlighting the understated role of numerical optimization in the literature.

Unexpected Improvements to Expected Improvement for Bayesian Optimization

Abstract

Expected Improvement (EI) is arguably the most popular acquisition function in Bayesian optimization and has found countless successful applications, but its performance is often exceeded by that of more recent methods. Notably, EI and its variants, including for the parallel and multi-objective settings, are challenging to optimize because their acquisition values vanish numerically in many regions. This difficulty generally increases as the number of observations, dimensionality of the search space, or the number of constraints grow, resulting in performance that is inconsistent across the literature and most often sub-optimal. Herein, we propose LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically. We demonstrate that numerical pathologies manifest themselves in 'classic' analytic EI, Expected Hypervolume Improvement (EHVI), as well as their constrained, noisy, and parallel variants, and propose corresponding reformulations that remedy these pathologies. Our empirical results show that members of the LogEI family of acquisition functions substantially improve on the optimization performance of their canonical counterparts and surprisingly, are on par with or exceed the performance of recent state-of-the-art acquisition functions, highlighting the understated role of numerical optimization in the literature.

1 Introduction

Bayesian Optimization (BO) is a widely used and effective approach for sample-efficient optimization of expensive-to-evaluate black-box functions [25, 28], with applications ranging widely between aerospace engineering [48], biology and medicine [49], materials science [3], civil engineering [4], and machine learning hyperparameter optimization [66, 72]. BO leverages a probabilistic surrogate model in conjunction with an acquisition function to determine where to query the underlying objective function. Improvement-based acquisition functions, such as Expected Improvement (EI) and Probability of Improvement (PI), are among the earliest and most widely used acquisition functions for efficient global optimization of non-convex functions [42, 58]. EI has been extended to the constrained [27, 29], noisy [52], and multi-objective [20] setting, as well as their respective batch variants [6, 13, 77], and is a standard baseline in the BO literature [25, 66]. While much of the literature has focused on developing new sophisticated acquisition functions, subtle yet critical implementation details of foundational BO methods are often overlooked. Importantly, the performance of EI and its variants is inconsistent even for mathematically identical formulations and, as we show in this work, most often sub-optimal.

Although the problem of optimizing EI effectively has been discussed in various works, e.g. [25, 31, 77], prior focus has been on optimization algorithms and initialization strategies, rather than the fundamental issue of computing EI.

In this work, we identify pathologies in the computation of improvement-based acquisition functions that give rise to numerically vanishing values and gradients, which - to our knowledge - are present in all existing implementations of EI , and propose reformulations that lead to increases in the associated optimization performance which often match or exceed that of recent methods.

Contributions

We introduce LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically. Notably, the analytic variant of LogEI, which mathematically results in the same BO policy as EI, empirically shows significantly improved optimization performance.

We extend the ideas behind analytical LogEI to other members of the EI family, including constrained EI (CEI), Expected Hypervolume Improvement (EHVI), as well as their respective batch variants for parallel BO, qEI and qEHVI, using smooth approximations of the acquisition utilities to obtain non-vanishing gradients. All of our methods are available as part of BoTorch [6].

We demonstrate that our newly proposed acquisition functions substantially outperform their respective analogues on a broad range of benchmarks without incurring meaningful additional computational cost, and often match or exceed the performance of recent methods.

Motivation

Maximizing acquisition functions for BO is a challenging problem, which is generally non-convex and often contains numerous local maxima, see the lower right panel of Figure 1. While zeroth-order methods are sometimes used, gradient-based methods tend to be far more effective at optimizing acquisition functions on continuous domains, especially in higher dimensions.

In addition to the challenges stemming from non-convexity that are shared across acquisition functions, the values and gradients of improvement-based acquisition functions are frequently minuscule in large swaths of the domain. Although EI is never mathematically zero under a Gaussian posterior distribution, 1 it often vanishes, even becoming exactly zero in floating point precision. The same

applies to its gradient, making EI (and PI, see Appendix A) exceptionally difficult to optimize via gradient-based methods. The right panels of Figure 1 illustrate this behavior on a simple onedimensional quadratic function.

To increase the chance of finding the global optimum of non-convex functions, gradient-based optimization is typically performed from multiple starting points, which can help avoid getting stuck in local optima [70]. For improvement-based acquisition functions however, optimization becomes increasingly challenging as more data is collected and the likelihood of improving over the incumbent diminishes, see our theoretical results in Section 3 and the empirical illustration in Figure 1 and Appendix D.2. As a result, gradient-based optimization with multiple random starting points will eventually degenerate into random search when the gradients at the starting points are numerically zero. This problem is particularly acute in high dimensions and for objectives with a large range.

Various initialization heuristics have been proposed to address this behavior by modifying the randomrestart strategy. Rather than starting from random candidates, an alternative naïve approach would be to use initial conditions close to the best previously observed inputs. However, doing that alone inherently limits the acquisition optimization to a type of local search, which cannot have global guarantees. To attain such guarantees, it is necessary to use an asymptotically space-filling heuristic; even if not random, this will entail evaluating the acquisition function in regions where no prior observation lies. Ideally, these regions should permit gradient-based optimization of the objective for efficient acquisition function optimization, which necessitates the gradients to be non-zero. In this work, we show that this can be achieved for a large number of improvement-based acquisition functions, and demonstrate empirically how this leads to substantially improved BO performance.

2 Background

Weconsider the problem of maximizing an expensive-to-evaluate black-box function f true : X ↦→ R M over some feasible set X ⊆ R d . Suppose we have collected data D n = { ( x i , y i ) } n i =1 , where x i ∈ X and y i = f true ( x i ) + v i ( x i ) and v i is a noise corrupting the true function value f true ( x i ) . The response f true may be multi-output as is the case for multiple objectives or black-box constraints, in which case y i , v i ∈ R M . We use Bayesian optimization (BO), which relies on a surrogate model f that for any batch X := { x 1 , . . . , x q } of candidate points provides a probability distribution over the outputs f ( X ) := ( f ( x 1 ) , . . . , f ( x q )) . The acquisition function α then utilizes this posterior prediction to assign an acquisition value to x that quantifies the value of evaluating the points in x , trading off exploration and exploitation.

2.1 Gaussian Processes

Gaussian Processes (GP) [65] are the most widely used surrogates in BO, due to their high data efficiency and good uncertainty quantification. For our purposes, it suffices to consider a GP as a mapping that provides a multivariate Normal distribution over the outputs f ( x ) for any x :

In the single-outcome ( M = 1 ) setting, f ( x ) ∼ N ( µ ( x ) , Σ( x )) with µ : X q → R q and Σ : X q →S q + . In the sequential ( q = 1 ) case, this further reduces to a univariate Normal distribution: f ( x ) ∼ N ( µ ( x ) , σ 2 ( x )) with µ : X → R and σ : X → R + .

2.2 Improvement-based Acquisition Functions

Expected Improvement For the fully-sequential ( q = 1 ), single-outcome ( M = 1 ) setting, 'classic' EI [59] is defined as

where [ · ] + denotes the max(0 , · ) operation, y ∗ = max i y i is the best function value observed so far, also referred to as the incumbent , h ( z ) = ϕ ( z ) + z Φ( z ) , and ϕ, Φ are the standard Normal density and distribution functions, respectively. This formulation is arguably the most widely used acquisition function in BO, and the default in many popular software packages.

Constrained Expected Improvement Constrained BO involves one or more black-box constraints and is typically formulated as finding max x ∈ X f true , 1 ( x ) such that f true ,i ( x ) ≤ 0 for i ∈ { 2 , . . . , M } . Feasibility-weighting the improvement [27, 29] is a natural approach for this class of problems:

Parallel Expected Improvement In many settings, one may evaluate f true on q > 1 candidates in parallel to increase throughput. The associated parallel or batch analogue of EI [30, 75] is given by where ✶ is the indicator function. If the constraints { f i } i ≥ 2 are modeled as conditionally independent of the objective f 1 this can be simplified as the product of EI and the probability of feasibility.

Unlike EI, qEI does not admit a closed-form expression and is thus typically computed via Monte Carlo sampling, which also extends to non-Gaussian posterior distributions [6, 75]:

where ξ i ( x ) ∼ f ( x ) are random samples drawn from the joint model posterior at x .

Expected Hypervolume Improvement In multi-objective optimization (MOO), there generally is no single best solution; instead the goal is to explore the Pareto Frontier between multiple competing objectives, the set of mutually-optimal objective vectors. A common measure of the quality of a finitely approximated Pareto Frontier P between M objectives with respect to a specified reference point r ∈ R M is its hypervolume HV ( P , r ) := λ (⋃ y i ∈P [ r , y i ] ) , where [ r , y i ] denotes the hyperrectangle bounded by vertices r and y i , and λ is the Lebesgue measure. An apt acquisition function for multi-objective optimization problems is therefore the expected hypervolume improvement

due to observing a batch f ( X ) := [ f ( x 1 ) , · · · , f ( x q )] of q new observations. EHVI can be expressed in closed form if q = 1 and the objectives are modeled with independent GPs [80], but Monte Carlo approximations are required for the general case (qEHVI) [13].

2.3 Optimizing Acquisition Functions

Optimizing an acquisition function (AF) is a challenging task that amounts to solving a non-convex optimization problem, to which multiple approaches and heuristics have been applied. These include gradient-free methods such as divided rectangles [41], evolutionary methods such as CMA-ES [32], first-order methods such as stochastic gradient ascent, see e.g., Daulton et al. [15], Wang et al. [75], and (quasi-)second order methods [25] such as L-BFGS-B [10]. Multi-start optimization is commonly employed with gradient-based methods to mitigate the risk of getting stuck in local minima. Initial points for optimization are selected via various heuristics with different levels of complexity, ranging from simple uniform random selection to BoTorch 's initialization heuristic, which selects initial points by performing Boltzmann sampling on a set of random points according to their acquisition function value [6]. See Appendix B for a more complete account of initialization strategies and optimization procedures used by popular implementations. We focus on gradient-based optimization as often leveraging gradients results in faster and more performant optimization [13].

Optimizing AFs for parallel BO that quantify the value of a batch of q > 1 points is more challenging than optimizing their sequential counterparts due to the higher dimensionality of the optimization problem qd instead of d - and the more challenging optimization surface. A common approach to simplify the problem is to use a sequential greedy strategy that greedily solves a sequence of single point selection problems. For i = 1 , . . . , q , candidate x i is selected by optimizing the AF for q = 1 , conditional on the previously selected designs { x 1 , ..., x i -1 } and their unknown observations, e.g. by fantasizing the values at those designs [77]. For submodular AFs, including EI, PI, and EHVI, a sequential greedy strategy will attain a regret within a factor of 1 /e compared to the joint optimum, and previous works have found that sequential greedy optimization yields improved BO performance compared to joint optimization [13, 77]. Herein, we find that our reformulations enable joint batch optimization to be competitive with the sequential greedy strategy, especially for larger batches.

2.4 Related Work

While there is a substantial body of work introducing a large variety of different AFs, much less focus has been on the question of how to effectively implement and optimize these AFs. Zhan and Xing [81] provide a comprehensive review of a large number of different variants of the EI family, but do not discuss any numerical or optimization challenges. Zhao et al. [82] propose combining a variety of different initialization strategies to select initial conditions for optimization of acquisition functions and show empirically that this improves optimization performance. However, they do not address any potential issues or degeneracies with the acquisition functions themselves. Recent works have considered effective gradient-based approaches for acquisition optimization. Wilson et al. [77] demonstrates how stochastic first-order methods can be leveraged for optimizing Monte Carlo acquisition functions. Balandat et al. [6] build on this work and put forth sample average approximations for MC acquisition functions that admit gradient-based optimization using deterministic higher-order optimizers such as L-BFGS-B.

Another line of work proposes to switch from BO to local optimization based on some stopping criterion to achieve faster local convergence, using either zeroth order [60] or gradient-based [57] optimization. While McLeod et al. [57] are also concerned with numerical issues, we emphasize that those issues arise due to ill-conditioned covariance matrices and are orthogonal to the numerical pathologies of improvement-based acquisition functions.

3 Theoretical Analysis of Expected Improvement's Vanishing Gradients

In this section, we shed light on the conditions on the objective function and surrogate model that give rise to the numerically vanishing gradients in EI, as seen in Figure 1. In particular, we show that as a BO algorithm closes the optimality gap f ∗ -y ∗ , where f ∗ is the global maximum of the function f true, and the associated GP surrogate's uncertainty decreases, EI is exceedingly likely to exhibit numerically vanishing gradients.

Let P x be a distribution over the inputs x , and f ∼ P f be an objective drawn from a Gaussian process. Then with high probability over the particular instantiation f of the objective, the probability that an input x ∼ P x gives rise to an argument ( µ ( x ) -y ∗ ) /σ ( x ) to h in Eq. (2) that is smaller than a threshold B exceeds P x ( f ( x ) < f ∗ -ϵ n ) , where ϵ n depends on the optimality gap f ∗ -y ∗ and the maximum posterior uncertainty max x σ n ( x ) . This pertains to EI's numerically vanishing values and gradients, since the numerical support S η ( h ) = { x : | h ( x ) | > η } of a naïve implementation of h in (2) is limited by a lower bound B ( η ) that depends on the floating point precision η . Formally, S η ( h ) ⊂ [ B ( η ) , ∞ ) even though S 0 ( h ) = R mathematically. As a consequence, the following result can be seen as a bound on the probability of encountering numerically vanishing values and gradients in EI using samples from the distribution P x to initialize the optimization of the acquisition function.

Theorem 1. Suppose f is drawn from a Gaussian process prior P f , y ∗ ≤ f ∗ , µ n , σ n are the mean and standard deviation of the posterior P f ( f |D n ) and B ∈ R . Then with probability 1 -δ ,

For any given - and especially early - iteration, ϵ n does not have to be small, as both the optimality gap and the maximal posterior standard deviation can be large initially. Note that under certain technical conditions on the kernel function and the asymptotic distribution of the training data D n , the maximum posterior variance vanishes guaranteeably as n increases, see [50, Corollary 3.2]. On its own, Theorem 1 gives insight into the non-asymptotic behavior by exposing a dependence to the distribution of objective values f . In particular, if the set of inputs that give rise to high objective values ( ≈ f ∗ ) is concentrated, P ( f ( x ) < f ∗ -ϵ ) will decay very slowly as ϵ increases, thereby maintaining a lower bound on the probability of close to 1. As an example, this is the case for the Ackley function, especially as the dimensionality increases, which explains the behavior in Figure 1.

4 Unexpected Improvements

In this section, we propose re-formulations of analytic and MC-based improvement-based acquisition functions that render them significantly easier to optimize. We will use differing fonts, e.g. log and log , to differentiate between the mathematical functions and their numerical implementations.

4.1 Analytic LogEI

Mathematically, EI's values and gradients are nonzero on the entire real line, except in the noiseless case for points that are perfectly correlated with previous observations. However, naïve implementations of h are numerically zero when z = ( µ ( x ) -y ∗ ) /σ ( x ) is small, which happens when the model has high confidence that little improvement can be achieved at x . We propose an implementation of log ◦ h that can be accurately computed for a much larger range of inputs. Specifically, we compute

where log_h is mathematically equivalent to log ◦ h and can be stably and accurately computed by

where c 1 = log(2 π ) / 2 , and c 2 = log( π/ 2) / 2 , ϵ is the numerical precision, and log1mexp , erfcx are numerically stable implementations of log(1 -exp( z )) and exp( z 2 ) erfc ( z ) , respectively, see App. A. Progenitors of Eq. (9) are found in SMAC 1.0 [35] and RoBO [46] which contain a logtransformed analytic EI implementation that is much improved, but can still exhibit instabilities as z grows negative, see App. Fig. 10. To remedy similar instabilities, we put forth the third, asymptotic case in Eq. (9), ensuring numerical stability throughout, see App. A.2 for details. The asymptotically quadratic behavior of log_h becomes apparent in the last two cases, making the function particularly amenable to gradient-based optimization with significant practical implications for EI-based BO.

4.2 Monte Carlo Parallel LogEI

Beyond analytic EI, Monte Carlo formulations of parallel EI that perform differentiation on the level of MC samples, don't just exhibit numerically, but mathematically zero gradients for a significant proportion of practically relevant inputs. For qEI, the primary issue is the discrete maximum over the q outcomes for each MC sample in (5). In particular, the acquisition utility of expected improvement in Eq. 4 on a single sample ξ i of f is max j [ ξ i ( x j ) -y ∗ ] + . Mathematically, we smoothly approximate the acquisition utility in two stages: 1) u ij = softplus τ 0 ( ξ i ( x j ) -y ∗ ) ≈ [ ξ i ( x j ) -y ∗ ] + and 2) ∥ u i · ∥ 1 /τ max ≈ max j u ij . Notably, while we use canonical softplus and p-norm approximations here, specialized fat-tailed non-linearities are required to scale to large batches, see Appendix A.4. Since the resulting quantities are strictly positive, they can be transformed to log-space permitting an implementation of qLogEI that is numerically stable and can be optimized effectively. In particular,

where i is the index of the Monte Carlo draws from the GP posterior, j = 1 , . . . , q is the index for the candidate in the batch, and logsoftplus is a numerically stable implementation of log(log(1 + exp( z ))) . See Appendix A.3 for details, including the novel fat-tailed non-linearities like fatplus .

While the smoothing in (10) approximates the canonical qEI formulation, the following result shows that the associated relative approximation error can be quantified and bounded tightly as a function of the temperature parameters τ 0 , τ max and the batch size q . See Appendix C for the proof.

Lemma 2. [Relative Approximation Guarantee] Given τ 0 , τ max > 0 , the approximation error of qLogEI to qEI is bounded by

In Appendix D.10, we show the importance of setting the temperatures sufficiently low for qLogEI to achieve good optimization characteristics, something that only becomes possible by transforming all involved computations to log-space. Otherwise, the smooth approximation to the acquisition utility would exhibit vanishing gradients numerically, as the discrete max operator does mathematically.

4.3 Constrained EI

Both analytic and Monte Carlo variants of LogEI can be extended for optimization problems with black-box constraints. For analytic CEI with independent constraints of the form f i ( x ) ≤ 0 , the constrained formulation in Eq. (3) simplifies to LogCEI ( x ) = LogEI ( x ) + ∑ i log( P ( f i ( x ) ≤ 0)) , which can be readily and stably computed using LogEI in Eq. (8) and, if f i is modelled by a GP, a stable implementation of the Gaussian log cumulative distribution function. For the Monte Carlo variant, we apply a similar strategy as for Eq. (10) to the constraint indicators in Eq. (3): 1) a smooth approximation and 2) an accurate and stable implementation of its log value, see Appendix A.

4.4 Monte Carlo Parallel LogEHVI

The numerical difficulties of qEHVI in (6) are similar to those of qEI, and the basic ingredients of smoothing and log-transformations still apply, but the details are significantly more complex since qEHVI uses many operations that have mathematically zero gradients with respect to some of the inputs. Our implementation is based on the differentiable inclusion-exclusion formulation of the hypervolume improvement [13]. As a by-product, the implementation also readily allows for the differentiable computation of the expected log hypervolume, instead of the log expected hypervolume, note the order, which can be preferable in certain applications of multi-objective optimization [26].

5 Empirical Results

We compare standard versions of analytic EI (EI) and constrained EI (CEI), Monte Carlo parallel EI (qEI), as well as Monte Carlo EHVI (qEHVI), in addition to other state-of-the-art baselines like lower-bound Max-Value Entropy Search (GIBBON) [61] and single- and multi-objective Joint Entropy Search (JES) [36, 71]. All experiments are implemented using BoTorch [6] and utilize multi-start optimization of the AF with scipy 's L-BFGS-B optimizer. In order to avoid conflating the effect of BoTorch's default initialization strategy with those of our contributions, we use 16 initial points chosen uniformly at random from which to start the L-BFGS-B optimization. For a comparison with other initialization strategies, see Appendix D. We run multiple replicates and report mean and error bars of ± 2 standard errors of the mean. Appendix D.1 contains additional details.

Single-objective sequential BO We compare EI and LogEI on the 10-dimensional convex Sumof-Squares (SoS) function f ( x ) = ∑ 10 i =1 ( x i -0 . 5) 2 , using 20 restarts seeded from 1024 pseudorandom samples through BoTorch's default initialization heuristic. Figure 2 shows that due to vanishing gradients, EI is unable to make progress even on this trivial problem.

50

75

100

125

150

175

200

Number of evaluations

10

2

10

1

Regret

Regret

EI

LogEI

50

75

100

125

150

175

200

Number of evaluations

10

6

10

5

10

4

10

3

Acquisition function value

Acquisition function value

EI

exp(LogEI)

In Figure 3, we compare performance on the Ackley and Michalewicz test functions [67]. Notably, LogEI substantially outperforms EI on Ackley as the dimensionality increases. Ackley is a challenging multimodal function for which it is critical to trade off local exploitation with global exploration, a task made exceedingly difficult by the numerically vanishing gradients of EI in a large fraction of the search space. We see a similar albeit less pronounced behavior on Michalewicz, which reflects the fact that Michalewicz is a somewhat less challenging problem than Ackley.

0

50

100

150

200

250

0

2

4

6

8

10

12

Ackley

Best observed value

d = 2

random

EI

LogEI

GIBBON

JES

0

50

100

150

200

250

2

4

6

8

10

12

14

16

d = 8

0

50

100

150

200

250

2

4

6

8

10

12

14

16

d = 16

0

50

100

150

200

250

Function evaluations

1.8

1.6

1.4

1.2

1.0

0.8

0.6

Michalewicz

Best observed value

0

50

100

150

200

250

Function evaluations

6

5

4

3

2

0

50

100

150

200

250

Function evaluations

7

6

5

4

3

0

50

100

Iterations

0.00

0.05

0.10

0.15

0.20

0.25

Best feasible value

3D Tension-Compression String

with 4 black-box constraints

cei

logcei

scbo

0

50

100

Iterations

5000

8000

11000

14000

17000

20000

4D Pressure Vessel Design

with 4 black-box constraints

cei

logcei

scbo

0

50

100

Iterations

2

4

6

8

10

4D Welded Beam Design

with 5 black-box constraints

cei

logcei

scbo

0

50

100

Iterations

3000

3500

4000

4500

5000

7D Speed Reducer Design

with 11 black-box constraints

cei

logcei

scbo

BO with Black Box Constraints Figure 4 shows results on four engineering design problems with black box constraints that were also considered in [22]. We apply the same bilog transform as the trust region-based SCBO method [22] to all constraints to make them easier to model with a GP. We see that LogCEI outperforms the naive CEI implementation and converges faster than SCBO. Similar to the unconstrained problems, the performance gains of LogCEI over CEI grow with increasing problem dimensionality and the number of constraints. Notably, we found that for some problems, LogCEI in fact improved upon some of the best results quoted in the original literature , while using three orders of magnitude fewer function evaluations, see Appendix D.7 for details.

Parallel Expected Improvement with qLogEI Figure 5 reports the optimization performance of parallel BO on the 16-dimensional Ackley function for both sequential greedy and joint batch optimization using the fat-tailed non-linearities of App. A.4. In addition to the apparent advantages of qLogEI over qEI, a key finding is that jointly optimizing the candidates of batch acquisition functions can yield highly competitive optimization performance, see App. D.3 for extended results.

High-dimensional BO with qLogEI Figure 6 shows the performance of LogEI on three highdimensional problems: the 6 -dimensional Hartmann function embedded in a 100 -dimensional space, a 100 -dimensional rover trajectory planning problem, and a 103 -dimensional SVM hyperparameter tuning problem. We use a 103 -dimensional version of the 388 -dimensional SVM problem considered by Eriksson and Jankowiak [21], where the 100 most important features were selected using Xgboost.

0

100

200

Function evaluations

2.5

5.0

7.5

10.0

12.5

15.0

Best observed value

q = 4

rand

qei seq

qei jnt

qlogei seq

qlogei jnt

0

100

200

Function evaluations

5.0

7.5

10.0

12.5

15.0

q = 16

0

100

200

Function evaluations

5.0

7.5

10.0

12.5

15.0

q = 32

25

50

75

100

Iterations

3.5

3.0

2.5

2.0

1.5

1.0

Best observed value

100D Embedded Hartmann6 (

q

=4)

25

50

75

100

Iterations

0.13

0.14

0.15

0.16

0.17

0.18

0.19

Test RMSE

103D SVM (

q

=4)

25

50

75

100

Iterations

2.0

2.5

3.0

3.5

4.0

Reward

100D Rover (

q

=4)

qlogei + saas

qei + saas

qlogei

qei

random

Figure 6 shows that the optimization exhibits varying degrees of improvement from the inclusion of qLogEI, both when combined with SAASBO [21] and a standard GP. In particular, qLogEI leads to significant improvements on the embedded Hartmann problem, even leading BO with the canonical GP to ultimately catch up with the SAAS-prior-equipped model. On the other hand, the differences on the SVM and Rover problems are not significant, see Section 6 for a discussion.

Multi-Objective optimization with qLogEHVI Figure 7 compares qLogEHVI and qEHVI on two multi-objective test problems with varying batch sizes, including the real-world-inspired cell network design for optimizing coverage and capacity [19]. The results are consistent with our findings in the single-objective and constrained cases: qLogEHVI consistently outperforms qEHVI and even JES [71] for all batch sizes. Curiously, for the largest batch size and DTLZ2, qLogNEHVI's improvement over the reference point (HV > 0 ) occurs around three batches after the other methods, but dominates their performance in later batches. See Appendix D.5 for results on additional synthetic and real-world-inspired multi-objective problems such as the laser plasma acceleration optimization [38], and vehicle design optimization [54, 68]

6 Discussion

To recap, EI exhibits vanishing gradients 1) when high objective values are highly concentrated in the search space, and 2) as the optimization progresses. In this section, we highlight that these conditions are not met for all BO applications, and that LogEI's performance depends on the surrogate's quality.

On problem dimensionality While our experimental results show that advantages of LogEI generally grow larger as the dimensionality of the problem grows, we stress that this is fundamentally due to the concentration of high objective values in the search space, not the dimensionality itself. Indeed, we have observed problems with high ambient dimensionality but low intrinsic dimensionality, where LogEI does not lead to significant improvements over EI, e.g. the SVM problem in Figure 6.

0

100

200

0.03

0.04

0.05

0.06

Cell Network 30D

Best observed HV

q = 4

0

100

200

0.03

0.04

0.05

0.06

q = 8

0

100

200

0.03

0.04

0.05

0.06

q = 16

0

100

200

Function evaluations

0.0

0.1

0.2

0.3

0.4

DTLZ2 16D

Best observed HV

0

100

200

Function evaluations

0.0

0.1

0.2

0.3

0.4

0

100

200

Function evaluations

0.0

0.1

0.2

0.3

0.4

Rand

qNEHVI

qLogNEHVI

JES

On asymptotic improvements While members of the LogEI family can generally be optimized better, leading to higher acquisition values, improvements in optimization performance might be small in magnitude, e.g. the log-objective results on the convex 10D sum of squares in Fig. 2, or only begin to materialize in later iterations, like for q = 16 on DTLZ2 in Figure 7.

On model quality Even if good objective values are concentrated in a small volume of the search space and many iterations are run, LogEI might still not outperform EI if the surrogate's predictions are poor, or its uncertainties are not indicative of the surrogate's mismatch to the objective, see Rover in Fig. 6. In these cases, better acquisition values do not necessarily lead to better BO performance.

Replacing EI Despite these limitation, we strongly suggest replacing variants of EI with their LogEI counterparts. If LogEI were dominated by EI on some problem, it would be an indication that the EI family itself is sub-optimal, and improvements in performance can be attributed to the exploratory quality of randomly distributed candidates, which could be incorporated explicitly.

7 Conclusion

Our results demonstrate that the problem of vanishing gradients is a major source of the difficulty of optimizing improvement-based acquisition functions and that we can mitigate this issue through careful reformulations and implementations. As a result, we see substantially improved optimization performance across a variety of modified EI variants across a broad range of problems. In particular, we demonstrate that joint batch optimization for parallel BO can be competitive with, and at times exceed the sequential greedy approach typically used in practice, which also benefits from our modifications. Besides the convincing performance improvements, one of the key advantages of our modified acquisition functions is that they are much less dependent on heuristic and potentially brittle initialization strategies. Moreover, our proposed modifications do not meaningfully increase the computational complexity of the respective original acquisition function.

While our contributions may not apply verbatim to other classes of acquisition functions, our key insights and strategies do translate and could help with e.g. improving information-based [34, 76], cost-aware [51, 66], and other types of acquisition functions that are prone to similar numerical challenges. Further, combining the proposed methods with gradient-aware first-order BO methods [5, 16, 23] could lead to particularly effective high-dimensional applications of BO, since the advantages of both methods tend to increase with the dimensionality of the search space. Overall, we hope that our findings will increase awareness in the community for the importance of optimizing acquisition functions well, and in particular, for the care that the involved numerics demand.

Acknowledgments and Disclosure of Funding

The authors thank Frank Hutter for valuable references about prior work on numerically stable computations of analytic EI, David Bindel for insightful conversations about the difficulty of optimizing EI, as well as the anonymous reviewers for their knowledgeable feedback.

References

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Largescale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow. org/ .

Sebastian Ament and Michael O'Neil. Accurate and efficient numerical calculation of stable densities via optimized quadrature and asymptotics. Statistics and Computing , 28:171-185, 2018.

Sebastian Ament, Maximilian Amsler, Duncan R. Sutherland, Ming-Chiang Chang, Dan Guevarra, Aine B. Connolly, John M. Gregoire, Michael O. Thompson, Carla P. Gomes, and R. Bruce van Dover. Autonomous materials synthesis via hierarchical active learning of nonequilibrium phase diagrams. Science Advances , 7(51):eabg4930, 2021. doi: 10.1126/sciadv.abg4930. URL https://www.science.org/doi/abs/10.1126/sciadv.abg4930 .

Sebastian Ament, Andrew Witte, Nishant Garg, and Julius Kusuma. Sustainable concrete via bayesian optimization, 2023. URL https://arxiv.org/abs/2310.18288 . NeurIPS 2023 Workshop on Adaptive Experimentation in the Real World.

Sebastian E Ament and Carla P Gomes. Scalable first-order Bayesian optimization via structured automatic differentiation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 500-516. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ament22a.html .

Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33 , 2020.

Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures, 2018.

Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi-objective bayesian optimization with constraints, 2020.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing , 16(5):1190-1208, 1995.

Carlos A Coello Coello and Efrén Mezura Montes. Constraint-handling in genetic algorithms through the use of dominance-based tournament selection. Advanced Engineering Informatics , 16(3):193-203, 2002.

Alexander I. Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, and Haitham Bou Ammar. Hebo pushing the limits of sample-efficient hyperparameter optimisation, 2022.

Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 9851-9864. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 6fec24eac8f18ed793f5eaad3dd7977c-Paper.pdf .

Samuel Daulton, Sait Cakmak, Maximilian Balandat, Michael A. Osborne, Enlu Zhou, and Eytan Bakshy. Robust multi-objective Bayesian optimization under input noise. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 4831-4866. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/daulton22a.html .

Samuel Daulton, Xingchen Wan, David Eriksson, Maximilian Balandat, Michael A. Osborne, and Eytan Bakshy. Bayesian optimization over discrete and mixed spaces via probabilistic reparameterization. In Advances in Neural Information Processing Systems 35 , 2022.

Filip De Roos, Alexandra Gessner, and Philipp Hennig. High-dimensional gaussian process inference with derivatives. In International Conference on Machine Learning , pages 2535-2545. PMLR, 2021.

Kalyan Deb, L. Thiele, Marco Laumanns, and Eckart Zitzler. Scalable multi-objective optimization test problems. volume 1, pages 825-830, 06 2002. ISBN 0-7803-7282-4. doi: 10.1109/CEC.2002.1007032.

Aryan Deshwal, Sebastian Ament, Maximilian Balandat, Eytan Bakshy, Janardhan Rao Doppa, and David Eriksson. Bayesian optimization over high-dimensional combinatorial spaces via dictionary-based embeddings. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 7021-7039. PMLR, 25-27 Apr 2023.

Ryan M. Dreifuerst, Samuel Daulton, Yuchen Qian, Paul Varkey, Maximilian Balandat, Sanjay Kasturia, Anoop Tomar, Ali Yazdan, Vish Ponnampalam, and Robert W. Heath. Optimizing coverage and capacity in cellular networks using machine learning, 2021.

M. T. M. Emmerich, K. C. Giannakoglou, and B. Naujoks. Single- and multiobjective evolutionary optimization assisted by gaussian random field metamodels. IEEE Transactions on Evolutionary Computation , 10(4):421-439, 2006.

David Eriksson and Martin Jankowiak. High-dimensional Bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence . PMLR, 2021.

David Eriksson and Matthias Poloczek. Scalable constrained Bayesian optimization. In International Conference on Artificial Intelligence and Statistics . PMLR, 2021.

David Eriksson, Kun Dong, Eric Lee, David Bindel, and Andrew G Wilson. Scaling gaussian process regression with derivatives. Advances in neural information processing systems , 31, 2018.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems 32 , NeurIPS, 2019.

Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

Tobias Friedrich, Karl Bringmann, Thomas Voß, and Christian Igel. The logarithmic hypervolume indicator. In Proceedings of the 11th workshop proceedings on Foundations of genetic algorithms , pages 81-92, 2011.

Jacob Gardner, Matt Kusner, Zhixiang, Kilian Weinberger, and John Cunningham. Bayesian optimization with inequality constraints. In Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 937-945, Beijing, China, 22-24 Jun 2014. PMLR.

Roman Garnett. Bayesian Optimization . Cambridge University Press, 2023. to appear.

Michael A. Gelbart, Jasper Snoek, and Ryan P. Adams. Bayesian optimization with unknown constraints. In Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence , UAI, 2014.

David Ginsbourger, Rodolphe Le Riche, and Laurent Carraro. A Multi-points Criterion for Deterministic Parallel Global Optimization based on Gaussian Processes. Technical report, March 2008. URL https://hal.science/hal-00260579 .

Robert B Gramacy, Annie Sauer, and Nathan Wycoff. Triangulation candidates for bayesian optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 35933-35945. Curran Associates, Inc., 2022.

Nikolaus Hansen, Sibylle D. Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary Computation , 11(1):1-18, 2003. doi: 10.1162/106365603321828970.

Tim Head, Manoj Kumar, Holger Nahrstaedt, Gilles Louppe, and Iaroslav Shcherbatyi. scikit-optimize/scikit-optimize, October 2021. URL https://doi.org/10.5281/zenodo. 5565057 .

José Miguel Henrández-Lobato, Matthew W. Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1 , NIPS'14, pages 918-926, Cambridge, MA, USA, 2014. MIT Press.

Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5 , pages 507-523. Springer, 2011.

Carl Hvarfner, Frank Hutter, and Luigi Nardi. Joint entropy search for maximally-informed bayesian optimization. In Advances in Neural Information Processing Systems 35 , 2022.

Wolfram Research, Inc. Wolfram alpha, 2023. URL https://www.wolframalpha.com/ .

F. Irshad, S. Karsch, and A. Döpp. Multi-objective and multi-fidelity bayesian optimization of laser-plasma acceleration. Phys. Rev. Res. , 5:013063, Jan 2023. doi: 10.1103/PhysRevResearch. 5.013063. URL https://link.aps.org/doi/10.1103/PhysRevResearch.5.013063 .

Faran Irshad, Stefan Karsch, and Andreas Doepp. Reference dataset of multi-objective and multi- fidelity optimization in laser-plasma acceleration, January 2023. URL https://doi. org/10.5281/zenodo.7565882 .

Shali Jiang, Daniel Jiang, Maximilian Balandat, Brian Karrer, Jacob Gardner, and Roman Garnett. Efficient nonmyopic bayesian optimization via one-shot multi-step trees. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 18039-18049. Curran Associates, Inc., 2020.

Donald Jones, C. Perttunen, and B. Stuckman. Lipschitzian optimisation without the lipschitz constant. Journal of Optimization Theory and Applications , 79:157-181, 01 1993. doi: 10.1007/BF00941892.

Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization , 13:455-492, 1998.

Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabás Póczos, and Eric P. Xing. Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly. J. Mach. Learn. Res. , 21(1), jan 2020.

Jungtaek Kim, Seungjin Choi, and Minsu Cho. Combinatorial bayesian optimization with random mapping functions to convex polytopes. In James Cussens and Kun Zhang, editors, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , volume 180 of Proceedings of Machine Learning Research , pages 1001-1011. PMLR, 01-05 Aug 2022.

Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints , page arXiv:1312.6114, Dec 2013.

A. Klein, S. Falkner, N. Mansur, and F. Hutter. Robo: A flexible and robust bayesian optimization framework in python. In NIPS 2017 Bayesian Optimization Workshop , December 2017.

Dieter Kraft. A software package for sequential quadratic programming. ForschungsberichtDeutsche Forschungs- und Versuchsanstalt fur Luft- und Raumfahrt , 1988.

Rémi Lam, Matthias Poloczek, Peter Frazier, and Karen E Willcox. Advances in bayesian optimization with applications in aerospace engineering. In 2018 AIAA Non-Deterministic Approaches Conference , page 1656, 2018.

Robert Langer and David Tirrell. Designing materials for biology and medicine. Nature , 428, 04 2004.

Armin Lederer, Jonas Umlauft, and Sandra Hirche. Posterior variance analysis of gaussian processes with application to average learning curves. arXiv preprint arXiv:1906.01404 , 2019.

Eric Hans Lee, Valerio Perrone, Cedric Archambeau, and Matthias Seeger. Cost-aware Bayesian Optimization. arXiv e-prints , page arXiv:2003.10870, March 2020.

Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. Constrained bayesian optimization with noisy experiments. Bayesian Analysis , 14(2):495-519, 06 2019. doi: 10. 1214/18-BA1110.

Qiaohao Liang, Aldair E. Gongora, Zekun Ren, Armi Tiihonen, Zhe Liu, Shijing Sun, James R. Deneault, Daniil Bash, Flore Mekki-Berrada, Saif A. Khan, Kedar Hippalgaonkar, Benji Maruyama, Keith A. Brown, John Fisher III, and Tonio Buonassisi. Benchmarking the performance of bayesian optimization across multiple experimental materials science domains. npj Computational Materials , 7(1):188, 2021.

Xingtao Liao, Qing Li, Xujing Yang, Weigang Zhang, and Wei Li. Multiobjective optimization for crash safety design of vehicles using stepwise regression model. Structural and Multidisciplinary Optimization , 35:561-569, 06 2008. doi: 10.1007/s00158-007-0163-x.

Wenlong Lyu, Fan Yang, Changhao Yan, Dian Zhou, and Xuan Zeng. Batch Bayesian optimization via multi-objective acquisition ensemble for automated analog circuit design. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3306-3314. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/lyu18a.html .

Martin Mächler. Accurately computing log (1- exp (-| a|)) assessed by the rmpfr package. Technical report, Technical report, 2012.

Mark McLeod, Stephen Roberts, and Michael A. Osborne. Optimization, fast and slow: optimally switching between local and Bayesian optimization. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3443-3452. PMLR, 10-15 Jul 2018.

Jonas Moˇ ckus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1-7, 1974 , pages 400-404. Springer, 1975.

Jonas Mockus. The application of bayesian methods for seeking the extremum. Towards global optimization , 2:117-129, 1978.

Hossein Mohammadi, Rodolphe Le Riche, and Eric Touboul. Making ego and cma-es complementary for global optimization. In Clarisse Dhaenens, Laetitia Jourdan, and Marie-Eléonore Marmion, editors, Learning and Intelligent Optimization , pages 287-292, Cham, 2015. Springer International Publishing.

Henry B. Moss, David S. Leslie, Javier González, and Paul Rayson. Gibbon: General-purpose information-based bayesian optimisation. J. Mach. Learn. Res. , 22(1), jan 2021.

Changyong Oh, Jakub Tomczak, Efstratios Gavves, and Max Welling. Combinatorial bayesian optimization using the graph cartesian product. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 2914-2924. Curran Associates, Inc., 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Victor Picheny, Joel Berkeley, Henry B. Moss, Hrvoje Stojic, Uri Granta, Sebastian W. Ober, Artem Artemev, Khurram Ghani, Alexander Goodall, Andrei Paleyes, Sattar Vakili, Sergio Pascual-Diaz, Stratis Markou, Jixiang Qing, Nasrulloh R. B. S Loka, and Ivo Couckuyt. Trieste: Efficiently exploring the depths of black-box functions with tensorflow, 2023. URL https: //arxiv.org/abs/2302.08436 .

Carl Edward Rasmussen. Gaussian Processes in Machine Learning , pages 63-71. Springer Berlin Heidelberg, Berlin, Heidelberg, 2004.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems , pages 2951-2959, 2012.

S. Surjanovic and D. Bingham. Virtual library of simulation experiments: Test functions and datasets. Retrieved May 14, 2023, from http://www.sfu.ca/~ssurjano .

Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing , 89:106078, 2020. ISSN 1568-4946.

The GPyOpt authors. GPyOpt: A bayesian optimization framework in python. http:// github.com/SheffieldML/GPyOpt , 2016.

Aimo Törn and Antanas Zilinskas. Global optimization , volume 350. Springer, 1989.

Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multiobjective bayesian optimization. In Advances in Neural Information Processing Systems 35 , 2022.

Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In NeurIPS 2020 Competition and Demonstration Track , 2021.

Andreas Wächter and Lorenz T. Biegler. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical Programming , 106 (1):25-57, 2006.

Xingchen Wan, Vu Nguyen, Huong Ha, Binxin Ru, Cong Lu, and Michael A. Osborne. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 10663-10674. PMLR, 18-24 Jul 2021.

Jialei Wang, Scott C. Clark, Eric Liu, and Peter I. Frazier. Parallel bayesian global optimization of expensive functions, 2016.

Zi Wang and Stefanie Jegelka. Max-value Entropy Search for Efficient Bayesian Optimization. ArXiv e-prints , page arXiv:1703.01968, March 2017.

James Wilson, Frank Hutter, and Marc Deisenroth. Maximizing acquisition functions for bayesian optimization. In Advances in Neural Information Processing Systems 31 , pages 9905-9916. 2018.

Jian Wu and Peter I. Frazier. The parallel knowledge gradient method for batch bayesian optimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 3134-3142. Curran Associates Inc., 2016.

Jian Wu, Matthias Poloczek, Andrew Gordon Wilson, and Peter I Frazier. Bayesian optimization with gradients. In Advances in Neural Information Processing Systems , pages 5267-5278, 2017.

Kaifeng Yang, Michael Emmerich, André Deutz, and Thomas Bäck. Multi-objective bayesian global optimization using expected hypervolume improvement gradient. Swarm and Evolutionary Computation , 44:945 - 956, 2019.

Dawei Zhan and Huanlai Xing. Expected improvement for expensive optimization: a review. Journal of Global Optimization , 78(3):507-544, 2020.

Jiayu Zhao, Renyu Yang, Shenghao Qiu, and Zheng Wang. Enhancing high-dimensional bayesian optimization by optimizing the acquisition function maximizer initialization. arXiv preprint arXiv:2302.08298 , 2023.

Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evol. Comput. , 8(2):173-195, jun 2000. ISSN 1063-6560. doi: 10.1162/106365600568202. URL https://doi.org/10.1162/106365600568202 .

A Acquisition Function Details

A.1 Analytic Expected Improvement

Recall that the main challenge with computing analytic LogEI is to accurately compute log h , where h ( z ) = ϕ ( z ) + z Φ( z ) , with ϕ ( z ) = exp( -z 2 / 2) / √ 2 π and Φ( z ) = ∫ z -∞ ϕ ( u ) du . To express log h in a numerically stable form as z becomes increasingly negative, we first take the log and multiply ϕ out of the argument to the logarithm:

Fortunately, this form exposes the quadratic factor, Φ( z ) /ϕ ( z ) can be computed via standard implementations of the scaled complementary error function erfcx , and log (1 + z Φ( z ) /ϕ ( z )) , the last term of Eq. (12), can be computed stably with the log1mexp implementation proposed in [56]:

where expm1 , log1p are canonical stable implementations of exp( x ) -1 and log(1+ x ) , respectively. In particular, we compute

where c 1 = log(2 π ) / 2 , c 2 = log( π/ 2) / 2 , and ϵ is the floating point precision. We detail the reasoning behind the third asymptotic case of Equation (14) in Lemma 4 of Section A.2 below.

Numerical Study of Acquisition Values Figure 8 shows both the numerical failure mode of a naïve implementation of EI, which becomes exactly zero numerically for moderately small z , while the evaluation via log_h in Eq. (14) exhibits quadratic asymptotic behavior that is particularly amenable to numerical optimization routines.

Expected Improvement

5000

LogEl

Naive

4000

FailurePoint

O(z^2)

((z)601-)0

3000

060l-

2000

1000

0

-100

-50

0

50

100

Z

Equivalence of Optimizers Lemma 3 states that if the maximum of EI is greater than 0, LogEI and EI have the same set of maximizers. Furthermore, if max x ∈ X EI ( x ) = 0 , then X = arg max x ∈ X EI ( x ) . In this case, LogEI is undefined everywhere, so it has no maximizers, which we note would yield the same BO policy as EI, for which every point is a maximizer.

Lemma 3. If max x ∈ X EI( x ) > 0 , then arg max x ∈ X EI( x ) = arg max x ∈ X , EI( x ) > 0 LogEI( x ) .

Proof. Suppose max x ∈ X EI ( x ) > 0 . Then arg max x ∈ X EI ( x ) = argmax x ∈ X , EI ( x ) > 0 EI ( x ) . For all x ∈ X such that EI ( x ) > 0 , LogEI ( x ) = log( EI ( x )) . Since log is monontonic, we have that arg max z ∈ R > 0 z = argmax z ∈ R > 0 log( z ) . Hence, arg max x ∈ X , EI ( x ) > 0 EI ( x ) = arg max x ∈ X , EI ( x ) > 0 LogEI ( x ) .

20.0

17.5

15.0

12.5

10.0

7.5

5.0

2.5

z

10

13

10

10

10

7

10

4

10

1

10 2

Absolute Difference

Difference of Laurent Expansion to log1mexp Implementation

O(1 / z^2)

O(1 / z^4)

Order 2

Order 4

Order 6

Order 8

Order 10

Order 12

Order 14

Order 16

Order 18

Order 20

Order 22

Order 24

Order 26

Order 28

Order 30

Order 32

A.2 Analytical LogEI's Asymptotics

As z grows negative and large, even the more robust second branch in Eq. (14), as well as the implementation of Hutter et al. [35] and Klein et al. [46] can suffer from numerical instabilities (Fig. 10, left). In our case, the computation of the last term of Eq. (12) is problematic for large negative z . For this reason, we propose an approximate asymptotic computation based on a Laurent expansion at -∞ . As a result of the full analysis in the following, we also attain a particularly simple formula with inverse quadratic convergence in z , which is the basis of the third branch of Eq. (14):

In full generality, the asymptotic behavior of the last term can be characterized by the following result.

Lemma 4 (Asymptotic Expansion) . Let z < -1 and K ∈ N , then

Proof. We first derived a Laurent expansion of the non-log-transformed z Φ( z ) /ϕ ( z ) , a key quantity in the last term of Eq. (12), with the help of Wolfram Alpha [37]:

It remains to derive the asymptotic error bound through the log-transformation of the above expansion. [ ∏ ]

The penultimate equality is due to log(1+ x ) = x + O ( x 2 ) , the last due to L ( z, K ) = Θ( | z | -2 ) .

SMAC 1.0 and RoBO's Analytic LogEI To our knowledge, SMAC 1.0's implementation of the logarithm of analytic EI due to Hutter et al. [35], later translated to RoBO [46], was the first to improve the numerical stability of analytic EI through careful numerics. The associated implementation is mathematically identical to log ◦ EI, and greatly improves the numerical stability of the computation. For large negative z however, the implementation still exhibits instabilities that gives rise to floating point infinities through which useful gradients cannot be propagated (Fig. 10, left). The implementation proposed herein remedies this problem by switching to the asymptotic approximation of Eq. (15) once it is accurate to machine precision ϵ . This is similar to the use of asymptotic expansions for the computation of α -stable densities proposed by Ament and O'Neil [2].

HEBO's Approximate Analytic LogEI HEBO [12] contains an approximation to the logarithm of analytical EI as part of its implementation of the MACE acquisition function [55], which - at the time of writing - is missing the log normalization constant of the Gaussian density, leading to a large discontinuity at the chosen cut-off point of z = -6 below which the approximation takes effect, see here for the HEBO implementation. Notably, HEBO does not implement an exact stable computation of LogEI like the one of Hutter et al. [35], Klein et al. [46] and the non-asymptotic branches of the current work. Instead, it applies the approximation for all z < -6 , where the implementation exhibits a maximum error of > 1 . 02 , or if the implementation's normalization constant were corrected, a maximum error of > 0 . 1 . By comparison, the implementation put forth herein is mathematically exact for the non-asymptotic regime z > -1 / √ ϵ and accurate to numerical precision in the asymptotic regime due to the design of the threshold value.

A.3 Monte-Carlo Expected Improvement

For Monte-Carlo, we cannot directly apply similar numerical improvements as for the analytical version, because the utility values, the integrand of Eq. (4), on the sample level are likely to be mathematically zero. For this reason, we first smoothly approximate the acquisition utility and subsequently apply log transformations to the approximate acquisition function.

To this end, a natural choice is softplus τ 0 ( x ) = τ 0 log(1 + exp( x/τ 0 )) for smoothing the max(0 , x ) , where τ 0 is a temperature parameter governing the approximation error. Further, we approximate the max i over the q candidates by the norm ∥ · ∥ 1 /τ max and note that the approximation error introduced by both smooth approximations can be bound tightly as a function of two 'temperature' parameters τ 0 and τ max , see Lemma 2.

Importantly, the smoothing alone only solves the problem of having mathematically zero gradients, not that of having numerically vanishing gradients, as we have shown for the analytical case above. For this reason, we transform all smoothed computations to log space and thus need the following special implementation of log ◦ softplus that can be evaluated stably for a very large range of inputs:

where τ is a temperature parameter and l depends on the floating point precision used, around -35 for double precision in our implementation.

Note that the lower branch of logsoftplus is approximate. Using a Taylor expansion of log(1+ z ) = z -z 2 / 2 + O ( z 3 ) around z = 0 , we can see that the approximation error is O ( z 2 ) , and therefore, log(log(1 + exp( x ))) = x + O (exp( x ) 2 ) , which converges to x exponentially quickly as x →-∞ . In our implementation, l is chosen so that no significant digit is lost in dropping the second order term from the lower branch.

Having defined logsoftplus , we further note that

Therefore, we express the logarithm of the smoothed acquisition utility for q candidates as

Applying another logsumexp to compute the logarithm of the mean of acquisition utilities over a set of Monte Carlo samples { ξ i } i gives rise to the expression in Eq. (10).

In particular for large batches (large q ), this expression can still give rise to vanishing gradients for some candidates, which is due to the large dynamic range of the outputs of the logsoftplus when x << 0 . To solve this problem, we propose a new class of smooth approximations to the 'hard' non-linearities that decay as O (1 /x 2 ) as x →-∞ in the next section.

A.4 A Class of Smooth Approximations with Fat Tails for Larger Batches

A regular softplus( x ) = log(1 + exp( x )) function smoothly approximates the ReLU non-linearity and - in conjunction with the log transformations - is sufficient to achieve good numerical behavior for small batches of the Monte Carlo acquisition functions. However, as more candidates are added, log softplus( x ) = log(log(1 + exp( x ))) is increasingly likely to have a high dynamic range as for x ≪ 0 , log softplus τ ( x ) ∼ -x/τ . If τ > 0 is chosen to be small, ( -x/τ ) can vary orders of magnitude within a single batch. This becomes problematic when we approximate the maximum utility over the batch of candidates, since logsumexp only propagates numerically non-zero gradients to inputs that are no smaller than approximately (max j x j -700) in double precision, another source of vanishing gradients.

To solve this problem, we propose a new smooth approximation to the ReLU, maximum, and indicator functions that decay only polynomially as x → -∞ , instead of exponentially, like the canonical softplus. The high level idea is to use (1 + x 2 ) -1 , which is proportional to the Cauchy density function (and is also known as a Lorentzian), in ways that maintain key properties of existing smooth approximations - convexity, positivity, etc - while changing the asymptotic behavior of the functions from exponential to O (1 /x 2 ) as x → -∞ , also known as a 'fat tail'. Further, we will show that the proposed smooth approximations satisfy similar maximum error bounds as their exponentially decaying counterparts, thereby permitting a similar approximation guarantee as Lemma 2 with minor adjusments to the involved constants. While the derivations herein are based on the Cauchy density with inverse quadratic decay, it is possible to generalize the derivations to e.g. α -stable distribution whose symmetric variants permit accurate and efficient numerical computation [2].

Fat Softplus We define

for a positive scalar α . The following result shows that we can ensure the monotonicity and convexity - both important properties of the ReLU that we would like to maintain in our approximation - of g by carefully choosing α .

1.0

0.5

0.0

0.5

1.0

0.0

0.2

0.4

0.6

0.8

1.0

fatplus

softplus

1.0

0.5

0.0

0.5

1.0

100

80

60

40

20

0

log softplus

Lemma 5 (Monotonicity and Convexity) . φ + ( x ) is positive, monotonically increasing, and strictly convex for α satisfying

Proof. Positivity follows due to α ≥ 0 , and both sumands being positive. Monotonicity and convexity can be shown via canonical differential calculus and bounding relevant quantities.

In particular, regarding monotonicity, we want to select α so that the first derivative is bounded below by zero:

First, we note that ∂ x φ + ( x ) is positive for x < 0 and any α , since both terms are positive in this regime. For x ≥ 0 , e x 1+ e x = (1 + e -x ) -1 ≥ 1 / 2 , and -1 / (1 + x 2 ) 2 ≥ -1 / (1 + x 2 ) , so that

Forcing 1 2 -α 2 x (1+ x 2 ) > 0 , and multiplying by (1 + x 2 ) gives rise to a quadratic equation whose roots are x = 2 α ± √ 4 α 2 -1 . Thus, there are no real roots for α < 1 / 2 . Since the derivative is certainly positive for the negative reals and the guaranteed non-existence of roots implies that the derivative cannot cross zero elsewhere, 0 ≤ α < 1 / 2 is a sufficient condition for monotonicity of φ + .

Regarding convexity, our goal is to prove a similar condition on α that guarantees the positivity of the second derivative:

Note that 6 x 2 -2 (1+ x 2 ) 3 is symmetric around 0 , is negative in ( -√ 1 / 3 , √ 1 / 3) and has a minimum of -2 at 0 . e -x (1+ e -x ) 2 is symmetric around zero and decreasing away from zero. Since the rational polynomial is only negative in ( -√ 1 / 3 , √ 1 / 3) , we can lower bound e -x (1+ e -x ) 2 > e -√ 1 / 3 (1+ e -√ 1 / 3 ) 2 in

( -√ 1 / 3 , √ 1 / 3) . Therefore,

Forcing e -√ 1 / 3 (1+ e -√ 1 / 3 ) 2 -2 α > 0 and rearranging yields the result. Since e -√ 1 / 3 (1+ e -√ 1 / 3 ) 2 / 2 ∼ 0 . 115135 , the convexity condition is stronger than the monotonicity condition and therefore subsumes it.

Importantly φ decays only polynomially for increasingly negative inputs, and therefore log φ only logarithmically, which keeps the range of φ constrained to values that are more manageable numerically. Similar to Lemma 7, one can show that

There are a large number of approximations or variants of the ReLU that have been proposed as activation functions of artificial neural networks, but to our knowledge, none satisfy the properties that we seek here: (1) smoothness, (2) positivity, (3) monotonicity, (4) convexity, and (5) polynomial decay. For example, the leaky ReLU does not satisfy (1) and (2), and the ELU does not satisfy (5).

Fat Maximum The canonical logsumexp approximation to max i x i suffers from numerically vanishing gradients if max i x i -min j x j is larger a moderate threshold, around 760 in double precision, depending on the floating point implementation. In particular, while elements close to the maximum receive numerically non-zero gradients, elements far away are increasingly likely to have a numerically zero gradient. To fix this behavior for the smooth maximum approximation, we propose

This approximation to the maximum has the same error bound to the true maximum as the logsumexp approximation:

Lemma 6. Given τ > 0

Proof. Regarding the lower bound, let i = arg max j x j . For this index, the associated summand in (21) is 1 . Since all sumands are positive, the entire sum is lower bounded by 1 , hence

Adding max j x j to the inequality finishes the proof for the lower bound.

Regarding the upper bound, (21) can be maximized when x i = max j x j for all i , in which case each ( x i -max j x j ) 2 is minimized, and hence each summand is maximized. In this case,

Adding max j x j to the inequality finishes the proof for the upper bound.

Fat Sigmoid Notably, we encountered a similar problem using regular (log)-sigmoids to smooth the constraint indicators for EI with black-box constraints. In principle the Cauchy cummulative distribution function would satisfy these conditions, but requires the computation of arctan , a special function that requires more floating point operations to compute numerically than the following function. Here, we want the smooth approximation ι to satisfy 1) positivity, 2) monotonicity, 3) polynomial decay, and 4) ι ( x ) = 1 / 2 -ι ( -x ) . Let γ = √ 1 / 3 , then we define

10

5

0

5

10

0.0

0.2

0.4

0.6

0.8

1.0

10

5

0

5

10

80

60

40

20

0

log fat sigmoid

log sigmoid

ι is monotonically increasing, satisfies ι ( x ) → 1 as x → ∞ , ι (0) = 1 / 2 , and ι ( x ) = O (1 /x 2 ) as x → -∞ . Further, we note that the asymptotics are primarily important here, but that we can also make the approximation tighter by introducing a temperature parameter τ , and letting ι τ ( x ) = ι ( x/τ ) . The approximation error of ι τ ( x ) to the Heaviside step function becomes tighter point-wise as τ → 0+ , except for at the origin where ι τ ( x ) = 1 / 2 , similar to the canonical sigmoid.

A.5 Constrained Expected Improvement

For the analytical case, many computational frameworks already provide a numerically stable implementation of the logarithm of the Gaussian cummulative distribution function, in the case of PyTorch, torch.special.log_ndtr , which can be readily used in conjunction with our implementation of LogEI , as described in Sec. 4.3.

For the case of Monte-Carlo parallel EI, we implemented the fat-tailed ι function from Sec. A.4 to approximate the constraint indicator and compute the per-candidate, per-sample acquisition utility using

where ξ ( k ) i is the i th sample of the k th constraint model, and τ cons is the temperature parameter controlling the approximation to the constraint indicator. While this functionality is in our implementation, our benchmark results use the analytical version.

A.6 Parallel Expected Hypervolume Improvement

The hypervolume improvement can be computed via the inclusion-exclusion principle, see [13] for details, we focus on the numerical issues concerning qEHVI here. To this end, we define

where f is the vector-valued objective function, and u k is the vector of upper bounds of one of K hyper-rectangles that partition the non-Pareto-dominated space, see [13] for details on the partitioning. Letting l k be the corresponding lower bounds of the hyper-rectangles, the hypervolume improvement can then be computed as

where X j = { X j ⊂ X cand : | X j | = j } is the superset of all subsets of X cand of size j and z ( m ) k,X j = z ( m ) k,i 1 ,...,i j for X j = { x i 1 , . . . , x i j } .

To find a numerically stable formulation of the logarithm of this expression, we first re-purpose the φ max function to compute the minimum in the expression of z ( m ) k,i 1 ,...,i j , like so φ min ( x ) = -φ max ( -x ) . Further, we use the φ + function of Sec. A.4 as for the single objective case to approximate [ z ( m ) k,X j -l ( m ) k ] + . We then have

Since we can only transform positive quantities to log space, we split the sum in Eq. (23) into positive and negative components, depending on the sign of ( -1) j +1 , and compute the result using a numerically stable implementation of log(exp( log of positive terms ) -exp( log of negative terms ) . The remaining sums over k and q can be carried out by applying logsumexp to the resulting quantity. Finally, applying logsumexp to reduce over an additional Monte-Carlo sample dimension yields the formulation of qLogEHVI that we use in our multi-objective benchmarks.

A.7 Probability of Improvement

Numerical improvements for the probability of improvement acquisition which is defined as α ( x ) = Φ ( µ ( x ) -y ∗ σ ( x ) ) , where Φ is the standard Normal CDF, can be obtained simply by taking the logarithm using a numerically stable implementation of log(Φ( z )) = logerfc ( -1 √ 2 z ) -log(2) , where logerfc is computed as

A.8 q-Noisy Expected Improvement

The same numerical improvements used by qLogEI to improve Monte-Carlo expected improvement (qEI) in Appendix A.3 can be applied to improve the fully Monte Carlo Noisy Expected Improvement [6, 52] acquisition function. As in qLogEI, we can (i) approximate the max(0 , x ) using a softplus to smooth the sample-level improvements to ensure that they are mathematically positive, (ii) approximate the maximum over the q candidate designs by norm || · || 1 τ max , and (iii) take the logarithm to of the resulting smoothed value to mitigate vanishing gradients. To further mitigate vanishing gradients, we can again leverage the Fat Softplus and Fat Maximum approximations. The only notable difference in the q EI and q NEI acquisition functions is the choice of incumbent, and similarly only a change of incumbent is required to obtain qLogNEI from qLogEI. Specifically, when the scalar y ∗ in Equation (10) is replaced with a vector containing the new incumbnent under each sample wer obtain the qLogNEI acquisition value. The i th element of the incumbent vector for qLogNEI is max n + q j ′ = q +1 ξ i ( x j ′ ) , where x q +1 , ..., x n + q are the previously evaluated designs and ξ i ( x j ′ ) is the value of the j ′ th point under the i th sample from the joint posterior over x 1 , ..., x n + q . We note that we use a hard maximum to compute the incumbent for each sample because we do not need to compute gradients with respect to the previously evaluated designs x q +1 , ..., x n + q .

We note that we obtain computational speed ups by (i) pruning the set of previously evaluated points that considered for being the best incumbent to include only those designs with non-zero probability of being the best design and (ii) caching the Cholesky decomposition of the posterior covariance over the resulting pruned set of previously evaluated designs and using low-rank updates for efficient sampling [14].

For experimental results of q LogNEI see Section D.4.

B Strategies for Optimizing Acquisition Functions

As discussed in Section 2.3, a variety of different approaches and heuristics have been applied to the problem of optimizing acquisition functions. For the purpose of this work, we only consider continuous domains X . While discrete and/or mixed domains are also relevant in practice and have

received substantial attention in recent years - see e.g. Baptista and Poloczek [7], Daulton et al. [15], Deshwal et al. [18], Kim et al. [44], Oh et al. [62], Wan et al. [74] - our work here on improving acquisition functions is largely orthogonal to this (though the largest gains should be expected when using gradient-based optimizers, as is done in mixed-variable BO when conditioning on discrete variables, or when performing discrete or mixed BO using continuous relaxations, probabilistic reparameterization, or straight-through estimators [15]).

Arguably the simplest approach to optimizing acquisition functions is by grid search or random search. While variants of this combined with local descent can make sense in the context of optimizing over discrete or mixed spaces and when acquisition functions can be evaluated efficiently in batch (e.g. on GPUs), this clearly does not scale to higher-dimensional continuous domains due to the exponential growth of space to cover.

Another relatively straightforward approach is to use zeroth-order methods such as DIRECT [41] (used e.g. by Dragonfly [43]) or the popular CMA-ES [32]. These approaches are easy implement as they avoid the need to compute gradients of acquisition functions. However, not relying on gradients is also what renders their optimization performance inferior to gradient based methods, especially for higher-dimensional problems and/or joint batch optimization in parallel Bayesian optimization.

The most common approach to optimizing acquisition functions on continuous domains is using gradient descent-type algorithms. Gradients are either be computed based on analytically derived closed-form expressions, or via auto-differentiation capabilities of modern ML systems such as PyTorch [63], Tensorflow [1], or JAX [9].

For analytic acquisition functions, a common choice of optimizer is L-BFGS-B [10], a quasi-second order method that uses gradient information to approximate the Hessian and supports box constraints. If other, more general constraints are imposed on the domain, other general purpose nonlinear optimizers such as SLSQP [47] or IPOPT [73] are used (e.g. by BoTorch ). For Monte Carlo (MC) acquisition functions, Wilson et al. [77] proposes using stochastic gradient ascent (SGA) based on stochastic gradient estimates obtained via the reparameterization trick [45]. Stochastic first-order algorithms are also used by others, including e.g. Wang et al. [75] and Daulton et al. [15]. Balandat et al. [6] build on the work by Wilson et al. [77] and show how sample average approximation (SAA) can be employed to obtain deterministic gradient estimates for MC acquisition functions, which has the advantage of being able to leverage the improved convergence rates of optimization algorithms designed for deterministic functions such as L-BFGS-B. This general approach has since been used for a variety of other acquisition functions, including e.g. Daulton et al. [13] and Jiang et al. [40].

Very few implementations of Bayesian Optimization actually use higher-order derivative information, as this either requires complex derivations of analytical expressions and their custom implementation, or computation of second-order derivatives via automated differentiation, which is less well supported and computationally much more costly than computing only first-order derivatives. One notable exception is Cornell-MOE [78, 79], which supports Newton's method (though this is limited to the acquisition functions implemented in C++ within the library and not easily extensible to other acquisition functions).

B.1 Common initialization heuristics for multi-start gradient-descent

One of the key issues to deal with gradient-based optimization in the context of optimizing acquisition functions is the optimizer getting stuck in local optima due to the generally highly non-convex objective. This is typically addressed by means of restarting the optimizer from a number of different initial conditions distributed across the domain.

A variety of different heuristics have been proposed for this. The most basic one is to restart from random points uniformly sampled from the domain (for instance, scikit-optimize [33] uses this strategy). However, as we have argued in this paper, acquisition functions can be (numerically) zero in large parts of the domain, and so purely random restarts can become ineffective, especially in higher dimensions and with more data points. A common strategy is therefore to either augment or bias the restart point selection to include initial conditions that are closer to 'promising points'. GPyOpt [69] augments random restarts with the best points observed so far, or alternatively points generated via Thompson sampling. Spearmint [66] initializes starting points based on Gaussian perturbations of the current best point. BoTorch [6] selects initial points by performing Boltzmann sampling on a set of random points according to their acquisition function value; the goal of this strategy is to achieve a

biased random sampling across the domain that is likely to generate more points around regions with high acquisition value, but remains asymptotically space-filling. The initialization strategy used by Trieste [64] works similarly to the one in BoTorch , but instead of using soft-randomization via Boltzmann sampling, it simply selects the topk points. Most recently, Gramacy et al. [31] proposed distributing initial conditions using a Delaunay triangulation of previously observed data points. This is an interesting approach that generalizes the idea of initializing 'in between' observed points from the single-dimensional case. However, this approach does not scale well with the problem dimension and the number of observed data points due to the complexity of computing the triangulation (with wall time empirically found to be exponential in the dimension, see [31, Fig. 3] and worst-case quadratic in the number of observed points).

However, while these initialization strategies can help substantially with better optimizing acquisition functions, they ultimately cannot resolve foundational issues with acquisition functions themselves. Ensuring that acquisition functions provides enough gradient information (not just mathematically but also numerically) is therefore key to be able to optimize it effectively, especially in higher dimensions and with more observed data points.

C Proofs

Theorem 1. Suppose f is drawn from a Gaussian process prior P f , y ∗ ≤ f ∗ , µ n , σ n are the mean and standard deviation of the posterior P f ( f |D n ) and B ∈ R . Then with probability 1 -δ ,

Proof. Webegin by expanding the argument to the h function in Eq. (2) as a sum of 1) the standardized error of the posterior mean µ n to the true objective f and 2) the standardized difference of the value of the true objective f at x to the best previously observed value y ∗ = max n i y i :

We proceed by bounding the first term on the right hand side. Note that by assumption, f ( x ) ∼ N ( µ n ( x ) , σ n ( x ) 2 ) and thus ( µ n ( x ) -f ( x )) / σ n ( x ) ∼ N (0 , 1) . For a positive C > 0 then, we use a standard bound on the Gaussian tail probability to attain

Therefore, ( µ ( x ) -f ( x )) /σ n ( x ) < C with probability 1 -δ if C = √ -2 log(2 δ ) .

Using the bound just derived, and forcing the resulting upper bound to be less than B yields a sufficient condition to imply µ n ( x ) -y ∗ < Bσ n ( x ) :

Letting f ∗ := f ( x ∗ ) , re-arranging and using y ∗ = f ∗ +( y ∗ -f ∗ ) we get with probability 1 -δ ,

Thus, we get

Note that the last inequality gives a bound that is not directly dependent on the evaluation of the posterior statistics of the surrogate at any specific x . Rather, it is dependent on the optimality gap f ∗ -y ∗ and the maximal posterior standard deviation, or a bound thereof. Letting ϵ n = ( f ∗ -y ∗ ) -( √ -2 log(2 δ ) -B ) max x σ n ( x ) finishes the proof.

Lemma 2. [Relative Approximation Guarantee] Given τ 0 , τ max > 0 , the approximation error of qLogEI to qEI is bounded by

Proof. Let z iq = ξ i ( x q ) -y ∗ , where i ∈ { 1 , ..., n } , and for brevity of notation, and let lse , lsp refer to the logsumexp and logsoftplus functions, respectively, and ReLU ( x ) = max( x, 0) . We then bound n | e qLogEI ( X ) -qEI ( X ) | by

First and second inequalities are due to the triangle inequality, where for the second we used | a -c | ≤ | a -b | + | b -c | with b = max q softplus ( z iq ) .

To bound the first term in the sum, note that ∥ x ∥ ∞ ≤ ∥ x ∥ q ≤ ∥ x ∥ ∞ d 1 /q , thus 0 ≤ ( ∥ x ∥ q -∥ x ∥ ∞ ) ≤ d 1 /q -1 ∥ x ∥ ∞ , and therefore

The second term in the sum can be bound due to | softplus τ 0 ( x ) -ReLU ( x ) | ≤ log(2) τ 0 (see Lemma 7 below) and therefore,

Dividing Eq. (30) by n to compute the sample mean finishes the proof for the Monte-Carlo approximations to the acquisition value. Taking n → ∞ further proves the result for the mathematical definitions of the parallel acquisition values, i.e. Eq. (4).

Approximating the ReLU using the softplus τ ( x ) = τ log(1 + exp( x/τ )) function leads to an approximation error that is at most τ in the infinity norm, i.e. ∥ softplus τ -ReLU ∥ ∞ = log(2) τ . The following lemma formally proves this.

Lemma 7. Given τ > 0 , we have for all x ∈ R ,

Proof. Taking the (sub-)derivative of softplus τ -ReLU , we get

which is positive for all x < 0 and negative for all x > 0 , hence the extremum must be at x , at which point softplus τ (0) -ReLU (0) = log(2) τ . Analyzing the asymptotic behavior, lim x →±∞ ( softplus τ ( x ) -ReLU ( x )) = 0 , and therefore softplus τ ( x ) > ReLU ( x ) for x ∈ R .

Approximation guarantees for the fat-tailed non-linearities of App. A.4 can be derived similarly.

D Additional Empirical Details and Results

D.1 Experimental details

All algorithms are implemented in BoTorch. The analytic EI, qEI, cEI utilize the standard BoTorch implementations. We utilize the original authors' implementations of single objective JES [36], GIBBON [61], and multi-objective JES [71], which are all available in the main BoTorch repository. All simulations are ran with 32 replicates and error bars represent ± 2 times the standard error of the mean. We use a Matern-5/2 kernel with automatic relevance determination (ARD), i.e. separate length-scales for each input dimension, and a top-hat prior on the length-scales in [0 . 01 , 100] . The input spaces are normalized to the unit hyper-cube and the objective values are standardized during each optimization iteration.

D.2 Additional Empirical Results on Vanishing Values and Gradients

The left plot of Figure 1 in the main text shows that for a large fraction of points across the domain the gradients of EI are numerically essentially zero. In this section we provide additional detail on these simulations as well as intuition for results.

The data generating process (DGP) for the training data used for the left plot of Figure 1 is the following: 80% of training points are sampled uniformly at random from the domain, while 20% are sampled according to a multivariate Gaussian centered at the function maximum with a standard deviation of 25% of the length of the domain. The idea behind this DGP is to mimic the kind of data one would see during a Bayesian Optimization loop (without having to run thousands of BO loops to generate the Figure 1). Under this DGP with the chosen test problem, the incumbent (best observed point) is typically better than the values at the random test locations, and this becomes increasingly the case as the dimensionality of the problem increases and the number of training points grows. This is exactly the situation that is typical when conducting Bayesian Optimization.

23

22

21

20

19

18

17

16

23

22

21

20

19

18

17

16

predicted value (+/- 2se)

Model fits, train & test data, d=8, n=60

trainig data (in sample)

test data (out of sample)

incumbent

true value

10

8

6

4

2

0

100

200

300

400

500

600

700

800

frequency

Histogram of z values

threshold: h(z) < 1e-08

threshold: h(z) < 1e-09

threshold: h(z) < 1e-10

threshold: h(z) < 1e-11

z

For a particular replicate, Figure 15 shows the model fits in-sample (black), out-of-sample (blue), and the best point identified so far (red) with 60 training points and a random subset of 50 (out of 2000) test points. One can see that the model produces decent mean predictions for out-of-sample data, and that the uncertainty estimates appear reasonably well-calibrated (e.g., the credible intervals typically cover the true value). A practitioner would consider this a good model for the purposes of Bayesian Optimization. However, while there is ample uncertainty in the predictions of the model away from the training points, for the vast majority of points, the mean prediction is many standard deviations away from the incumbent value (the error bars are ± 2 standard deviations). This is the key reason for EI taking on zero (or vanishingly small) values and having vanishing gradients.

To illustrate this, Figure 16 shows the histogram of z ( x ) values, the argument to the function h in (2). It also contains the thresholds corresponding to the values z below which h ( z ) is less than the respective threshold. Since σ ( x ) is close to 1 for most test points (mean: 0.87, std: 0.07), this is more or less the same as saying that EI ( z ( x )) is less than the threshold. It is evident from the histogram that the majority of the test points fall below these threshold values (especially for larger thresholds), showing that the associated acquisition function values (and similarly the gradients) are numerically almost zero and causing issues during acquisition function optimization.

D.3 Parallel Expected Improvement

0

50

100

150

200

250

2.5

5.0

7.5

10.0

12.5

15.0

Best observed value

q = 4

rand

qei seq

qei jnt

qlogei seq

qlogei jnt

0

50

100

150

200

250

2.5

5.0

7.5

10.0

12.5

15.0

q = 8

0

50

100

150

200

250

5.0

7.5

10.0

12.5

15.0

q = 16

0

50

100

150

200

250

5.0

7.5

10.0

12.5

15.0

q = 32

0

50

100

150

200

250

Function evaluations

0

20

40

60

80

Best observed value

0

50

100

150

200

250

Function evaluations

0

20

40

60

80

0

50

100

150

200

250

Function evaluations

20

40

60

80

0

50

100

150

200

250

Function evaluations

20

40

60

80

0

100

200

2

4

6

8

10

12

14

16

Best observed value

qei (seq)

rand

q=4

q=8

q=16

q=32

0

100

200

2

4

6

8

10

12

14

16

qei (jnt)

0

100

200

2

4

6

8

10

12

14

16

qlogei (seq)

0

100

200

2

4

6

8

10

12

14

16

qlogei (jnt)

0

100

200

2

4

6

8

10

12

14

16

0

100

200

Function evaluations

0

20

40

60

80

Best observed value

0

100

200

Function evaluations

0

20

40

60

80

0

100

200

Function evaluations

0

20

40

60

80

0

100

200

Function evaluations

0

20

40

60

80

0

100

200

Function evaluations

0

20

40

60

80

Figure 17 reports optimization performance of parallel BO on the 16-dimensional Ackley and Levy functions for both sequential greedy and joint batch optimization. Besides the apparent substantial advantages of qLogEI over qEI on Ackley, a key observation here is that jointly optimizing the candidates of batch acquisition functions can yield highly competitive optimization performance, especially as the batch size increases. Notably, joint optimization of the batch with qLogEI starts out performing worse in terms of BO performance than sequential on the Levy function, but outperforms all sequential methods as the batch size increases. See also Figure 18 for the scaling of each method with respect to the batch size q .

0

100

200

3

2

1

Hartmann 6D

Best observed value

Noise: 1.0% * Range(f)

rand

qnei

qei

qlogei

gibbon

0

100

200

3

2

1

Noise: 2.0% * Range(f)

0

100

200

3

2

1

Noise: 5.0% * Range(f)

0

100

200

2

4

6

8

10

12

14

16

Ackley 8D

Best observed value

0

100

200

2

4

6

8

10

12

14

16

0

100

200

2

4

6

8

10

12

14

16

0

100

200

Function evaluations

2

4

6

8

10

12

14

16

Best observed value

0

100

200

Function evaluations

2

4

6

8

10

12

14

16

0

100

200

Function evaluations

2

4

6

8

10

12

14

16

D.4 Noisy Expected Improvement

Figure 19 benchmarks the 'noisy' variant, qLogNEI. Similar to the noiseless case, the advantage of the LogEI versions over the canonical counterparts grows as with the dimensionality of the problem, and the noisy version improves on the canonical versions for larger noise levels.

D.5 Multi-Objective optimization with qLogEHVI

Figure 20 compares qLogEHVI and qEHVI on 6 different test problems with 2 or 3 objectives, and ranging from 2-30 dimensions. This includes 3 real world inspired problems: cell network design for optimizing coverage and capacity [19], laser plasma acceleration optimization [38], and vehicle design optimization [54, 68]. The results are consistent with our findings in the single-objective and constrained cases: qLogEHVI consistently outperforms qEHVI, and the gap is larger on higher dimensional problems.

D.6 Combining LogEI with TuRBO for High-Dimensional Bayesian Optimization

In the main text, we show how LogEI performs particularly well relative to other baselines in high dimensional spaces. Here, we show how LogEI can work synergistically with trust region based methods for high-dimensional BO, such as TuRBO [24].

Fig. 21 compares the performance of LogEI, TuRBO-1 + LogEI, TuRBO-1 + EI, as well as the original Thompson-sampling based implementation for the 50d Ackley test problem. Combining TuRBO-1 with LogEI results in substantially better performance than the baselines when using a small number of function evaluations. Thompson sampling (TS) ultimately performs better after 10 , 000 evaluations, but this experiment shows the promise of combining TuRBO and LogEI in settings where we cannot do thousands of function evaluations. Since we optimize batches of q = 50 candidates jointly, we also increase the number of Monte-Carlo samples from the Gaussian process

0

100

200

1

2

3

Hypervolume

1e7

Laser Plasma (

d

=4,

M

=3)

0

100

200

0.04

0.06

Cell Network (

d

=30,

M

=2)

0

50

100

0.2

0.4

DTLZ2 (

d

=6,

M

=2)

0

50

100

Function Evaluations

100

110

120

Hypervolume

d

=6,

M

=2)

Rand

qEHVI

qLogEHVI

JES

0

50

100

Function Evaluations

0

25

50

BraninCurrin (

d

=2,

M

=2)

0

50

100

Function Evaluations

150

200

250

Vehicle Safety (

d

=5,

M

=3)

1

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Number of evaluations

0.5

1

2

4

8

16

Regret

50D Ackley, q=50

TuRBO-1 + qEI, 1 restart

TuRBO-1 + qEI, 8 restarts

TuRBO-1 + qLogEI, 1 restart

TuRBO-1 + qLogEI, 8 restarts

qLogEI, 1 restart

qLogEI, 8 restarts

qEI, 1 restart

qEI, 8 restarts

TuRBO-1 (TS)

from 128 , the BoTorch default, to 512 , and use the fat-tailed smooth approximations of Sec. A.4 to ensure a strong gradient signal to all candidates of the batch.

Regarding the ultimate out-performance of TS over qLogEI, we think this is due to a model misspecification since the smoothness of a GP with the Matern-5/2 kernel cannot express the nondifferentiability of Ackley at the optimum.

D.7 Constrained Problems

While running the benchmarks using CEI in section 5, we found that we in fact improved upon a best known result from the literature. We compare with the results in Coello and Montes [11], which are generated using 30 runs of 80,000 function evaluations each.

For the pressure vessel design problem, Coello and Montes [11] quote a best-case feasible objective of 6059 . 946341 . Out of just 16 different runs, LogEI achieves a worst-case feasible objective of 5659 . 1108 after only 110 evaluations , and a best case of 5651 . 8862 , a notable reduction in objective value using almost three orders of magnitude fewer function evaluations.

For the welded beam problem, Coello and Montes [11] quote 1 . 728226 , whereas LogEI found a best case of 1 . 7496 after 110 evaluations, which is lightly worse, but we stress that this is using three orders of magnitude fewer evaluations.

For the tension-compression problem, LogEI found a feasible solution with value 0 . 0129 after 110 evaluations compared to the 0 . 012681 reported in in [11].

We emphasize that genetic algorithms and BO are generally concerned with distinct problem classes: BO focuses heavily on sample efficiency and the small-data regime, while genetic algorithms often utilize a substantially larger number of function evaluations. The results here show that in this case BO is competitive with and can even outperform a genetic algorithm, using only a tiny fraction of the sample budget, see App. D.7 for details. Sample efficiency is particularly relevant for physical simulators whose evaluation takes significant computational effort, often rendering several tens of thousands of evaluations infeasible.

D.8 Parallel Bayesian Optimization with cross-batch constraints

In some parallel Bayesian optimization settings, batch optimization is subject to non-trivial constraints across the batch elements. A natural example for this are budget constraints. For instance, in the context of experimental material science, consider the case where each manufactured compound requires a certain amount of different materials (as described by its parameters), but there is only a fixed total amount of material available (e.g., because the stock is limited due to cost and/or storage capacity). In such a situation, batch generation will be subject to a budget constraint that is not separable across the elements of the batch. Importantly, in that case sequential greedy batch generation is not an option since it is not able to incorporate the budget constraint. Therefore, joint batch optimization is required.

Here we give one such example in the context of Bayesian Optimization for sequential experimental design. We consider the five-dimensional silver nanoparticle flow synthesis problem from Liang et al. [53]. In this problem, to goal is to optimize the absorbance spectrum score of the synthesized nanoparticles over five parameters: four flow rate ratios of different components (silver, silver nitrate, trisodium citrate, polyvinyl alcohol) and a total flow rate Q tot .

The original problem was optimized over a discrete set of parameterizations. For our purposes we created a continuous surrogate model based on the experimental dataset (available from https: //github.com/PV-Lab/Benchmarking ) by fitting an RBF interpolator (smoothing factor of 0.01) in scipy on the (negative) loss. We use the same search space as Liang et al. [53], but in addition to the box bounds on the parameters we also impose an additional constraint on the total flow rate Q max tot = 2000 µL/min across the batch: ∑ q i =1 Q i tot ≤ Q max tot (the maximum flow rate per syringe / batch element is 1000µL/min). This constraint expresses the maximum throughput limit of the microfluidic experimentation setup. The result of this constraint is that we cannot consider the batch elements (in this case automated syringe pumps) have all elements of a batch of experiments operate in the high-flow regime at the same time.

In our experiment, we use a batch size of q = 3 and start the optimization from 5 randomly sampled points from the domain. We run 75 replicates with random initial conditions (shared across the different methods), error bars show ± two times the standard error of the mean. Our baseline is uniform random sampling from the domain (we use a hit-and-run sampler to sample uniformly from the constraint polytope ∑ q i =1 Q i tot ≤ Q max tot ). We compare qEI vs. qLogEI, and for each of the two we evaluate (i) the version with the batch constraint imposed explicitly in the optimizer (the optimization in this case uses scipy 's SLSQP solver), and (ii) a heuristic that first samples the total flow rates { Q i tot } q i =1 uniformly from the constraint set, and then optimizes the acquisition function with the flow rates fixed to the sampled values.

The results in Figure 22 show that while both the heuristic ('random Q tot ') and the proper constrained optimization ('batch-constrained Q tot ') substantially outperform the purely random baseline, it requires uisng both LogEI and proper constraints to achieve additional performance gains over the other 3 combinations. Importantly, this approach is only possible by performing joint optimization of

0

10

20

30

40

50

60

Number of evaluations

0.8

0.7

0.6

0.5

0.4

0.3

negative loss

Best observed point

Random

qEI, batch-constrained

Qtot

qLogEI, batch-constrained

Qtot

qEI, random

Qtot

qLogEI, random

Qtot

0

5

10

15

20

Number of batches

Best inferred point

Random

qEI, batch-constrained

Qtot

qLogEI, batch-constrained

Qtot

qEI, random

Qtot

qLogEI, random

Qtot

the batch, which underlines the importance of qLogEI and its siblings being able to achieve superior joint batch optimization in settings like this.

D.9 Details on Multi-Objective Problems

We consider a variety of multi-objective benchmark problems. We evaluate performance on three synthetic biobjective problems Branin-Currin ( d = 2 ) [8], ZDT1 ( d = 6 ) [83], and DTLZ2 ( d = 6 ) [17]. As described in 5, we also evaluated performance on three real world inspired problems. For the laser plasma acceleration problem, we used the public data available at Irshad et al. [39] to fit an independent GP surrogate model to each objective. We only queried te surrogate at the highest fidelity to create a single fidelity benchmark.

D.10 Effect of Temperature Parameter

In Figure 23, we examine the effect of fixed τ for the softplus operator on optimization performance. We find that smaller values typically work better.

D.11 Effect of the initialization strategy

Packages and frameworks commonly utilize smart initialization heuristics to improve acquisition function optimization performance. In Figure 24, we compare simple random restart optimization, where initial points are selected uniformly at random, with BoTorch's default initialization strategy, which evaluates the acquisition function on a large number of points selected from a scrambled Sobol sequence, and selects n points at random via Boltzman sampling (e.g., sampling using probabilities computed by taking a softmax over the acquisition values [6]. Here we consider 1024 initial candidates. We find that the BoTorch initialization strategy improves regret for all cases, and that qLogEI, followed by UCB show less sensitivity to the choice of initializations strategy. Figure25 examines the sensitivity of qEI to the number of initial starting points when performing standard random restart optimization and jointly optimizing the q points in the batch. We find that, consistent with our empirical and theoretical results in the main text, qEI often gets stuck in local minima for the Ackley test function, and additional random restarts often improve results but do not compensate for the fundamental optimality gap. The performance of qLogEI also improves as the number of starting points increases.

0

50

100

150

200

250

10

2

10

1

10 0

10 1

ackley d=2

tau_softplus = 0.1

tau_softplus = 0.01

tau_softplus = 0.001

tau_softplus = 0.0001

tau_softplus = 1e-05

tau_softplus = 1e-06

0

50

100

150

200

250

10 1

3×10 0

4×10 0

6×10 0

ackley d=16

0

50

100

150

200

250

10

7

10

6

10

5

10

4

10

3

10

2

10

1

sos d=2

0

50

100

150

200

250

10

4

10

3

10

2

10

1

10 0

sos d=16

0

100

200

0

5

10

Ackley

Best observed value

2D

EI - Random Init.

LogEI - Random Init.

GIBBON - Random Init.

JES - Random Init.

UCB - Random Init.

EI - Boltzmann Init.

LogEI - Boltzmann Init.

GIBBON - Boltzmann Init.

JES - Boltzmann Init.

UCB - Boltzmann Init.

0

100

200

2

4

6

8

10

12

14

16

8D

0

100

200

2

4

6

8

10

12

14

16

16D

0

100

200

Function Evaluations

1.5

1.0

Michalewicz

Best observed value

0

100

200

Function Evaluations

6

4

2

0

100

200

Function Evaluations

8

7

6

5

4

0

50

100

150

200

250

Iterations

16

14

12

10

8

6

4

Simple regret

0

50

100

150

200

250

Iterations

80

60

40

20

0

Simple regret

0

10

20

30

40

50

60

Iterations

16

14

12

10

8

6

4

Simple regret

ackley q=4

0

10

20

30

40

50

60

Iterations

80

60

40

20

0

Simple regret

levy q=4

0

2

4

6

8

10

12

14

16

14

12

10

8

6

4

Simple regret

ackley q=16

0

2

4

6

8

10

12

14

16

80

70

60

50

40

30

20

10

Simple regret

levy q=16

qLogEI - 1 starting points

qLogEI - 4 starting points

qLogEI - 16 starting points

qEI - 1 starting points

qEI - 4 starting points

qEI - 16 starting points

Iterations

Iterations

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a major weakness of Bayesian expected improvement acquisition functions, which are ubiquitously used for black-box optimization tasks such as computational hyperparameter tuning, materials science, and biomedical research. It is very common to use gradient-based optimizers to find local maxima of the acquisition surface, however expected-improvement acquisition functions have the very unfortunate pathology of a completely flat acquisition surface (where both the acquisition value and acquisition gradient are 0) on large regions of input space, particularly as optimization progresses and the best known solution improves. This pathology makes Bayesian optimization extremely sensitive to implementation decisions, particularly the initialization scheme of the acquisition maximization subproblem, which hinders BayesOpt practitioners in academia and industry. This paper rightfully places numerical precision and stability as one of the primary considerations in acquisition function design, and proposes simple, intuitive modifications to expected improvement acquisition functions that significantly improve performance.\n\nSTRENGTHS:\nI am strongly in favor of accepting this paper. The basic problem the authors are addressing is one I have often encountered myself, and I have even tried some similar ideas as those presented in this paper to try to address the problem, however I had to shelve the project due to competing demands for my time. I'm delighted to see the problem addressed so thoroughly here.\n\nThe greatest strength of this paper is the emphasis placed on how acquisition function design interacts with the optimization algorithms used to find their maxima. Generally speaking I feel this aspect of acquisition function design is often neglected in many Bayesian optimization papers, to the great detriment of the field.\n\nGiven the widespread use of BayesOpt across industries, and the use of EI-style acquisition functions in particular, I think this paper could have significant practical impact on multiple fields.\n\nWEAKNESSES:\nThis work is ready for publication without any significant revisions. I would encourage reviewers in general to think about the opportunity cost of burdening authors with minor or tangential concerns, slowing the development of follow-up work.\n\nIt's worth noting that the weaknesses of EI-style acquisition functions are fairly well documented in latent-space BayesOpt papers, such as [1] and [2]. Both of those works employed a heuristic I didn't see mentioned in the paper, which is to scale the max_{x_i \\in D} f(x_i) term in the acquisition function with some factor < 1 (e.g. 0.9). Some brief discussion in the related work on this point could help better communicate the potential impact of this paper.\n\n[1] Tripp, A., Daxberger, E., and Hernandez-Lobato, J. M. ´\nSample-efficient optimization in the latent space of deep\ngenerative models via weighted retraining. Advances in\nNeural Information Processing Systems, 33, 2020.\n\n[2] Stanton, S., Maddox, W., Gruver, N., Maffettone, P., Delaney, E., Greenside, P., & Wilson, A. G. (2022, June). Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning (pp. 20459-20478). PMLR.\n\nQUESTIONS:\nCan you comment on whether you expect batch acquisition value optimization to outperform sequential greedy optimization when there are strong locality constraints placed on the inner loop problem (i.e. d(x_0, x_t) < \\varepsilon for all x_t optimization iterates)? Intuitively it seems that the performance of batch acquisition optimization once again comes down to heuristics for choosing the right collection of points as the initial solution (e.g. [2]), since the iterates may not be able to move far enough from the initialization for the improved acquisition landscape to make much difference.\n\nLIMITATIONS:\nI think the discussion section could be expanded a bit. In particular I think the following rather vague sentence could be made more specific:\n\n\"While our contributions may not apply verbatim to other classes of acquisition functions, our key insights and strategies do translate and could help e.g. with improving information-based [20, 42], cost-aware [26, 36], and other types of acquisition functions that are prone to similar numerical challenges.\"\n\nI take this to mean that this paper has primarily focused on resolving numerical difficulties arising from the use of the max operator, and other acquisition functions may have numerical issues from operators that are not the max. If this is the case I think it could be stated more clearly, as it would give readers a clearer picture of avenues for future work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes LogEI, family of acquisition functions with improved numerical stability over EI that makes it more suitable for gradient-based acquisition function optimization, all while retaining similar optima as EI. Pathologies of EI are visualized and analyzed, and the approximation error between qEI and qLogEI is theoretically bounded. Empirically, LogEI clearly outperforms EI on most tasks, suggesting that it can act as a drop-in replacement for EI.\n\nSTRENGTHS:\n__Good motivation__: Acquisition function optimization is an often-overlooked aspect of Bayesian optimization, and the paper does a good job of displaying the difficulties of acquisition function optimization (Fig. 1) and how the proposed approach remedies the issue.\n__Simple, effective and extensible solution:__ Simple solutions that work are great, and LogEI (and its extensions) is good example. \n__Very good empirical performance:__ The improvement over EI in the results is striking on most tasks, suggesting it is simply a superior acquisition function to the default.\n\nWEAKNESSES:\n__Anecdotal evidence for similarity with EI:__ Intuitively, It is sensible that LogEI is similar to EI. However, there is little evidence (Fig. 1) and theory (Lemma 2) to support this.  I would greatly appreciate a similar Lemma for the analytic variant, and examples of when the two may not be identical. The performance gain of LogEI compared to EI is rather substantial on Ackley (~4x on Ackley-16!) and Mich, which suggests that the two may in fact not be very similar (but that LogEI may in fact simply be superior). Specifically, I don't believe the statement in Row 9, \"LogEI, whose members either have _identical or approximately equal optima_ as their canonical counterparts\" is well supported.\n\n__Existing LogEI and Lacking references to Related work:__ The idea of a LogEI is not novel (LogEI was an acquisition function in SMAC at one point). Admittedly, that implementation regards log-transformations of the objective [1, 2] and would not help with numerical stability in the same manner. Nevertheless, I think these warrant citation and  _comparison in the experiments_, given the similarities. Moreover, it limits the novelty of the approach.\n\n__Relevance in high dimensions:__ Currently, I am not convinced by the justification for LogEI in high dimensions. To me, LogEI aims to address the pathology where the acquisition function is zero, but I don't see that happening in high-dimensional problems due to the exuberance of high-uncertainty regions (which would make the acquisition function _non-zero, but constant?_). So, why is the proposed approach particularily important in high dimensions - i.e., why does LogEI help when the acquisition function is constant as opposed to (almost) zero? This would, in my opinion, require a separate motivation than Fig. 1, empirical results aside. Moreover, I think the _zero-value_ versus _constant-value_ distinction is very important, and should be emphasized more.\n\n  With this in mind, I find Fig. 2 striking and odd. 60 data points (which is when almost all points have a zero-valued gradient) on an 8-dimensional problem is not a large amount of data (not even a 2x2x...x2 grid), yet the uncertainty is small to the point of \"EI and its gradients become numerically zero across most of the domain\"? With all due respect, are the authors sure that this is not _just_ the gradient (and not the function value), or that the model has oddly long lengthscales?\n\n__Minor:__\n  - __Noisy tasks:__ Adaptation of LogEI to noisy problem settings are missing\n  - __Lack of conventional benchmarks__: As a potential drop-in replacement for EI, seeing its performance of the method on the most conventional low-dimensional tasks (Branin, Hartmanns) would be informative. Moreover, it would be helpful for future benchmarking.\n\nFor all of these bullets, I believe that evidence to clarify (and not necessarily disprove) the remarks would substantially strengthen the paper. \n\n[1]. An experimental investigation of model-based parameter optimisation: SPO and beyond. F Hutter, HH Hoos, K Leyton-Brown, Kevin P. Murphy. _GECCO '09: Proceedings of the 11th Annual conference on Genetic and evolutionary computation_. 2009.\n\n[2] Sequential model-based optimization for general algorithm configuration. F Hutter, HH Hoos, K Leyton-Brown. _Learning and Intelligent Optimization: 5th International Conference_. 2011.\n\nLogEI in SMAC: https://github.com/automl/SMAC3/blob/29355618b35dcf4b3ce3e773d633109f036dba17/smac/optimizer/acquisition.py#L503\n\nQUESTIONS:\n- Is there any setting where LogEI can _not_ act as a plug-in replacement for EI, or where performance would be expected to be worse?\n- Have the authors experimented with LogEI non-continous search spaces, and if so, what are the findings?\n\nLIMITATIONS:\nSome suggestions for addressing limitations have been stated in the Weaknesses section, but are otherwise adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper identifies a numerical pathology with the expected improvement (EI) family of acquisition functions: the vanishing gradients of the acquisition function leads to failure in acquisition function optimization. A set of modified EI acquisition functions that fix the numerical pathologies have been proposed. In experiments, the proposed EI acquisition functions are shown to outperform the canonical EI acquisition functions and to perform on par with other state-of-the-art acquisition functions on Bayesian optimization benchmarks.

STRENGTHS:
- This paper focuses on a previous neglected aspect of Bayesian optimization, acquisition function optimization, and identifies the numerical pathologies associated with EI acquisition functions. This paper provides a theoretical analysis of the vanishing gradient issue.
- Improvements have been proposed to EI, Monto Carlo Parallel EI, Constrained EI and EHVI. 
- Experiments clearly show the numerical pathology of EI optimization and the superior performance of the improved EI version.
- The proposed improved EI acquisition functions perform on par with state-of-the-art acquisition functions on high dimensional synthetic functions in both sequential and batch settings.

WEAKNESSES:
- The proposed treatment only works for the acquisition functions with vanishing gradient issues.

QUESTIONS:
- It is great to have an error bound of the qLogEI. However, for acquisition functions, preserving the relative order of values is more important than absolute difference. I wonder to what extent qLogEI preserves the relative order of values compared to qEI.

LIMITATIONS:
The limitation of the proposed method has been discussed in the last section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

comments: SUMMARY:
In this paper, the authors identify both through examples and theoretical analysis several numerical pathologies inherent to the computation and optimization of the Expected Improvement (EI), a popular acquisition function at the heart of Bayesian Optimization (BO) algorithms. 
They subsequently propose a numerical reformulation, LogEI, which achieves substantially better performances on a quite extensive range of benchmarks. This reformulation applies to all the member of the EI family: constrained EI for constrained BO, parallel EI for batch BO, and expected hypervolume improvement for multi-objective BO.

STRENGTHS:
- The paper is well-written and well-organised.
- The proposed numerical fix for EI is likely to have a great impact as it may benefit to all public implementations of EI. Furthermore, it does not incur excessively longer computation times except perhaps for multi-objective BO (roughly one order of magnitude larger).
- LogEI seems to produce more consistent results with respect to the initial optimization starting point compared to its canonical counterpart EI, thus reducing the need for heuristics for that matter.
- All claims are backed up by an impressive amount of numerical experiments and ablation studies in each setting (vanilla BO/ constrained BO/ batch BO/multi-objective BO).

WEAKNESSES:
I did not spot any weakness.

QUESTIONS:
I do not have any questions.

Edit: I have read the rebuttal and the discussions between the authors and other reviewers, my score remains unchanged.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06