SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The same numerical improvements used by qLogEI to improve Monte-Carlo expected improvement (qEI) in Appendix A.3 can be applied to improve the fully Monte Carlo Noisy Expected Improvement [6, 52] acquisition function. As in qLogEI, we can (i) approximate the  $\max(0, \mathbf{x})$  using a softmax to smooth the sample-level improvements to ensure that they are mathematically positive, (ii) approximate the maximum over the  $q$  candidate designs by norm  $\|\cdot\|_{\frac{1}{7\max}}$ , and (iii) take the logarithm to the resulting smoothed value to mitigate vanishing gradients. To further mitigate vanishing gradients, we can again leverage the Fat Softplus and Fat Maximum approximations. The only notable difference in the  $q$  EI and  $q$  NEI acquisition functions is the choice of incumbent, and similarly only a change of incumbent is required to obtain qLogNEI from qLogEI. Specifically, when the scalar  $y^*$  in Equation (10) is replaced with a vector containing the new incumbent under each sample we obtain the qLogNEI acquisition value. The  $i^{\text{th}}$  element of the incumbent vector for qLogNEI is  $\max_{j' = q + 1}^{n + q}\xi^i(\mathbf{x}_{j'})$ , where  $\mathbf{x}_{q + 1}, \ldots, \mathbf{x}_{n + q}$  are the previously evaluated designs and  $\xi^i(\mathbf{x}_{j'})$  is the value of the  $j'$ th point under the  $i^{\text{th}}$  sample from the joint posterior over  $\mathbf{x}_1, \ldots, \mathbf{x}_{n + q}$ . We note that we use a hard maximum to compute the incumbent for each sample because we do not need to compute gradients with respect to the previously evaluated designs  $\mathbf{x}_{q + 1}, \ldots, \mathbf{x}_{n + q}$ .
We note that we obtain computational speed ups by (i) pruning the set of previously evaluated points that considered for being the best incumbent to include only those designs with non-zero probability of being the best design and (ii) caching the Cholesky decomposition of the posterior covariance over the resulting pruned set of previously evaluated designs and using low-rank updates for efficient sampling [14].
For experimental results of  $q\mathrm{LogNEI}$  see Section D.4.
B Strategies for Optimizing Acquisition Functions
As discussed in Section 2.3, a variety of different approaches and heuristics have been applied to the problem of optimizing acquisition functions. For the purpose of this work, we only consider continuous domains  $\mathbb{X}$ . While discrete and/or mixed domains are also relevant in practice and have
received substantial attention in recent years - see e.g. Baptista and Poloczek [7], Daulton et al. [15], Deshwal et al. [18], Kim et al. [44], Oh et al. [62], Wan et al. [74] - our work here on improving acquisition functions is largely orthogonal to this (though the largest gains should be expected when using gradient-based optimizers, as is done in mixed-variable BO when conditioning on discrete variables, or when performing discrete or mixed BO using continuous relaxations, probabilistic reparameterization, or straight-through estimators [15]).
Arguably the simplest approach to optimizing acquisition functions is by grid search or random search. While variants of this combined with local descent can make sense in the context of optimizing over discrete or mixed spaces and when acquisition functions can be evaluated efficiently in batch (e.g. on GPUs), this clearly does not scale to higher-dimensional continuous domains due to the exponential growth of space to cover.
Another relatively straightforward approach is to use zeroth-order methods such as DIRECT [41] (used e.g. by Dragonfly [43]) or the popular CMA-ES [32]. These approaches are easy implement as they avoid the need to compute gradients of acquisition functions. However, not relying on gradients is also what renders their optimization performance inferior to gradient based methods, especially for higher-dimensional problems and/or joint batch optimization in parallel Bayesian optimization.
The most common approach to optimizing acquisition functions on continuous domains is using gradient descent-type algorithms. Gradients are either be computed based on analytically derived closed-form expressions, or via auto-differentiation capabilities of modern ML systems such as PyTorch [63], Tensorflow [1], or JAX [9].
For analytic acquisition functions, a common choice of optimizer is L-BFGS-B [10], a quasi-second order method that uses gradient information to approximate the Hessian and supports box constraints. If other, more general constraints are imposed on the domain, other general purpose nonlinear optimizers such as SLSQP [47] or IPOPT [73] are used (e.g. by BoTorch). For Monte Carlo (MC) acquisition functions, Wilson et al. [77] proposes using stochastic gradient ascent (SGA) based on stochastic gradient estimates obtained via the reparameterization trick [45]. Stochastic first-order algorithms are also used by others, including e.g. Wang et al. [75] and Daulton et al. [15]. Balandat et al. [6] build on the work by Wilson et al. [77] and show how sample average approximation (SAA) can be employed to obtain deterministic gradient estimates for MC acquisition functions, which has the advantage of being able to leverage the improved convergence rates of optimization algorithms designed for deterministic functions such as L-BFGS-B. This general approach has since been used for a variety of other acquisition functions, including e.g. Daulton et al. [13] and Jiang et al. [40].
Very few implementations of Bayesian Optimization actually use higher-order derivative information, as this either requires complex derivations of analytical expressions and their custom implementation, or computation of second-order derivatives via automated differentiation, which is less well supported and computationally much more costly than computing only first-order derivatives. One notable exception is Cornell-MOE [78, 79], which supports Newton's method (though this is limited to the acquisition functions implemented in  $\mathrm{C + + }$  within the library and not easily extensible to other acquisition functions).
B.1 Common initialization heuristics for multi-start gradient-descent
One of the key issues to deal with gradient-based optimization in the context of optimizing acquisition functions is the optimizer getting stuck in local optima due to the generally highly non-convex objective. This is typically addressed by means of restarting the optimizer from a number of different initial conditions distributed across the domain.
A variety of different heuristics have been proposed for this. The most basic one is to restart from random points uniformly sampled from the domain (for instance, scikit-optimize [33] uses this strategy). However, as we have argued in this paper, acquisition functions can be (numerically) zero in large parts of the domain, and so purely random restarts can become ineffective, especially in higher dimensions and with more data points. A common strategy is therefore to either augment or bias the restart point selection to include initial conditions that are closer to "promising points". GPy0pt [69] augments random restarts with the best points observed so far, or alternatively points generated via Thompson sampling. Spearmint [66] initializes starting points based on Gaussian perturbations of the current best point. BoTorch [6] selects initial points by performing Boltzmann sampling on a set of random points according to their acquisition function value; the goal of this strategy is to achieve a
biased random sampling across the domain that is likely to generate more points around regions with high acquisition value, but remains asymptotically space-filling. The initialization strategy used by Trieste [64] works similarly to the one in BoTorch, but instead of using soft-randomization via Boltzmann sampling, it simply selects the top- $k$  points. Most recently, Gramacy et al. [31] proposed distributing initial conditions using a Delaunay triangulation of previously observed data points. This is an interesting approach that generalizes the idea of initializing "in between" observed points from the single-dimensional case. However, this approach does not scale well with the problem dimension and the number of observed data points due to the complexity of computing the triangulation (with wall time empirically found to be exponential in the dimension, see [31, Fig. 3] and worst-case quadratic in the number of observed points).
However, while these initialization strategies can help substantially with better optimizing acquisition functions, they ultimately cannot resolve foundational issues with acquisition functions themselves. Ensuring that acquisition functions provide enough gradient information (not just mathematically but also numerically) is therefore key to be able to optimize it effectively, especially in higher dimensions and with more observed data points.
where  $\epsilon_{n} = (f^{*} - y^{*}) + \left(\sqrt{-2\log(2\delta)} -B\right)\max_{\mathbf{x}}\sigma_{n}(\mathbf{x}).$
Proof. We begin by expanding the argument to the  $h$  function in Eq. (2) as a sum of 1) the standardized error of the posterior mean  $\mu_{n}$  to the true objective  $f$  and 2) the standardized difference of the value of the true objective  $f$  at  $x$  to the best previously observed value  $y^{*} = \max_{i}^{n}y_{i}$ :
$$
\frac {\mu_ {n} (\mathbf {x}) - y ^ {*}}{\sigma_ {n} (\mathbf {x})} = \frac {\mu_ {n} (\mathbf {x}) - f (\mathbf {x})}{\sigma_ {n} (\mathbf {x})} + \frac {f (\mathbf {x}) - f ^ {*}}{\sigma_ {n} (\mathbf {x})} \tag {25}
$$
We proceed by bounding the first term on the right hand side. Note that by assumption,  $f(\mathbf{x}) \sim \mathcal{N}(\mu_n(\mathbf{x}), \sigma_n(\mathbf{x})^2)$  and thus  $(\mu_n(\mathbf{x}) - f(\mathbf{x})) / \sigma_n(\mathbf{x}) \sim \mathcal{N}(0,1)$ . For a positive  $C > 0$  then, we use a standard bound on the Gaussian tail probability to attain
$$
P \left(\frac {\mu_ {n} (\mathbf {x}) - f (\mathbf {x})}{\sigma_ {n} (\mathbf {x})} > C\right) \leq e ^ {- C ^ {2} / 2} / 2. \tag {26}
$$
Therefore,  $(\mu (\mathbf{x}) - f(\mathbf{x})) / \sigma_{n}(\mathbf{x}) < C$  with probability  $1 - \delta$  if  $C = \sqrt{-2\log(2\delta)}$
Using the bound just derived, and forcing the resulting upper bound to be less than  $B$  yields a sufficient condition to imply  $\mu_n(\mathbf{x}) - y^* < B\sigma_n(\mathbf{x})$ :
$$
\frac {\mu_ {n} (\mathbf {x}) - y ^ {*}}{\sigma_ {n} (\mathbf {x})} \leq C + \frac {f (\mathbf {x}) - y ^ {*}}{\sigma_ {n} (\mathbf {x})} <   B \tag {27}
$$
Letting  $f^{*} \coloneqq f(\mathbf{x}^{*})$ , re-arranging and using  $y^{*} = f^{*} + (y^{*} - f^{*})$  we get with probability  $1 - \delta$
$$
f (\mathbf {x}) \leq f ^ {*} - \left(f ^ {*} - y ^ {*}\right) - (\sqrt {- 2 \log (2 \delta)} - B) \sigma_ {n} (\mathbf {x}). \tag {28}
$$
$$
\begin{array}{l} P _ {\mathbf {x}} \left(\frac {\mu_ {n} (\mathbf {x}) - y ^ {*}}{\sigma_ {n} (\mathbf {x})} <   B\right) \geq P _ {\mathbf {x}} \left(f (\mathbf {x}) \leq f ^ {*} - (f ^ {*} - y ^ {*}) - (\sqrt {- 2 \log (2 \delta)} - B) \sigma_ {n} (x)\right) \\ \geq P _ {x} \left(f (\mathbf {x}) \leq f ^ {*} - \left(f ^ {*} - y ^ {*}\right) - \left(\sqrt {- 2 \log (2 \delta)} - B\right) \max  _ {\mathbf {x}} \sigma_ {n} (\mathbf {x})\right). \tag {29} \\ \end{array}
$$
Note that the last inequality gives a bound that is not directly dependent on the evaluation of the posterior statistics of the surrogate at any specific  $\mathbf{x}$ . Rather, it is dependent on the optimality gap  $f^{*} - y^{*}$  and the maximal posterior standard deviation, or a bound thereof. Letting  $\epsilon_{n} = (f^{*} - y^{*}) - (\sqrt{-2\log(2\delta)} - B)\max_{\mathbf{x}}\sigma_{n}(\mathbf{x})$  finishes the proof.
Lemma 2. [Relative Approximation Guarantee] Given  $\tau_0, \tau_{\mathrm{max}} > 0$ , the approximation error of qLogEI to qEI is bounded by
Proof. Let  $z_{iq} = \xi_i(\mathbf{x}_q) - y^*$ , where  $i \in \{1, \dots, n\}$ , and for brevity of notation, and let  $1 \leq \mathsf{se}, 1 \leq \mathsf{sp}$  refer to the logsumexp and logsoftplus functions, respectively, and  $\operatorname{ReLU}(x) = \max(x, 0)$ . We then bound  $n | e^{\mathrm{qLogEI}(\mathbf{X})} - \mathrm{qEI}(\mathbf{X})|$  by
$$
\begin{array}{l} \left| \exp (\mathbf {l s e} _ {i} (\tau_ {\max } \mathbf {l s e} _ {q} (\mathbf {l s p} _ {\tau_ {0}} (z _ {i q}) / \tau_ {\max }))) - \sum_ {i} \max  _ {q} \operatorname {R e L U} (z _ {i q}) \right| \\ \leq \sum_ {i} \left| \exp \left(\tau_ {\operatorname * {m a x}} \mathbf {l s e} _ {q} \left(\mathbf {l s p} _ {\tau_ {0}} \left(z _ {i q}\right) / \tau_ {\operatorname * {m a x}}\right)\right) - \max  _ {q} \mathrm {R e L U} \left(z _ {i q}\right) \right| \\ = \sum_ {i} \left| \| \text {s o f t p l u s} _ {\tau_ {0}} \left(z _ {i}\right) \| _ {1 / \tau_ {\max }} - \max  _ {q} \operatorname {R e L U} \left(z _ {i q}\right) \right| \tag {30} \\ \leq \sum_ {i} \left| \| \mathbf {s o f t p l u s} _ {\tau_ {0}} (z _ {i.}) \| _ {1 / \tau_ {\max }} - \max  _ {q} \mathbf {s o f t p l u s} _ {\tau_ {0}} (z _ {i q}) \right| \\ + \left| \max  _ {q} \mathsf {s o f t p l u s} _ {\tau_ {0}} \left(z _ {i q}\right) - \max  _ {q} \mathsf {R e L U} \left(z _ {i q}\right) \right| \\ \end{array}
$$
First and second inequalities are due to the triangle inequality, where for the second we used  $|a - c| \leq |a - b| + |b - c|$  with  $b = \max_q \text{softplus}(z_{iq})$ .
To bound the first term in the sum, note that  $\| \mathbf{x}\|_{\infty}\leq \| \mathbf{x}\|_{q}\leq \| \mathbf{x}\|_{\infty}d^{1 / q}$ , thus  $0\leq (\| \mathbf{x}\| _q - \| \mathbf{x}\|_{\infty})\leq d^{1 / q} - 1\| \mathbf{x}\|_{\infty}$ , and therefore
$$
\begin{array}{l} \left| \left\| \text {s o f t p l u s} _ {\tau_ {0}} \left(z _ {i \cdot}\right) \right\| _ {1 / \tau_ {\max }} - \max  _ {q} \text {s o f t p l u s} _ {\tau_ {0}} \left(z _ {i q}\right) \right| \leq \left(q ^ {\tau_ {\max }} - 1\right) \max  _ {q} \text {s o f t p l u s} _ {\tau_ {0}} \left(z _ {i q}\right) \\ \leq \left(q ^ {\tau_ {\max }} - 1\right) \left(\max  _ {q} \operatorname {R e L U} \left(z _ {i q}\right) + \log (2) \tau_ {0}\right) \\ \end{array}
$$
The second term in the sum can be bound due to  $|\mathrm{softplus}_{\tau_0}(x) - \mathrm{ReLU}(x)| \leq \log(2)\tau_0$  (see Lemma 7 below) and therefore,
$$
\left| \max  _ {q} \mathbf {s o f t p l u s} _ {\tau_ {0}} \left(z _ {i q}\right) - \max  _ {q} \mathrm {R e L U} _ {\tau_ {0}} \left(z _ {i q}\right) \right| \leq \log (2) \tau_ {0}.
$$
Dividing Eq. (30) by  $n$  to compute the sample mean finishes the proof for the Monte-Carlo approximations to the acquisition value. Taking  $n \to \infty$  further proves the result for the mathematical definitions of the parallel acquisition values, i.e. Eq. (4).
Approximating the ReLU using the softplus $_{\tau}(x) = \tau \log(1 + \exp(x/\tau))$  function leads to an approximation error that is at most  $\tau$  in the infinity norm, i.e.  $\| \text{softplus}_{\tau} - \text{ReLU}\|_{\infty} = \log(2)\tau$ . The following lemma formally proves this.
Lemma 7. Given  $\tau >0$ , we have for all  $x\in \mathbb{R}$
$$
\left| \operatorname {s o f t p l u s} _ {\tau} (x) - R e L U (x) \right| \leq \log (2) \tau . \tag {31}
$$
Proof. Taking the (sub-)derivative of  $\mathrm{softplus}_{\tau} - \mathrm{ReLU}$ , we get
$$
\partial_ {x} \mathsf {s o f t p l u s} _ {\tau} (x) - \mathsf {R e L U} (x) = (1 + e ^ {- x / \tau}) ^ {- 1} - \left\{ \begin{array}{l l} 1 & x > 0 \\ 0 & x \leq 0 \end{array} \right.
$$
which is positive for all  $x < 0$  and negative for all  $x > 0$ , hence the extremum must be at  $x$ , at which point  $\text{softplus}_{\tau}(0) - \text{ReLU}(0) = \log(2)\tau$ . Analyzing the asymptotic behavior,  $\lim_{x \to \pm \infty} (\text{softplus}_{\tau}(x) - \text{ReLU}(x)) = 0$ , and therefore  $\text{softplus}_{\tau}(x) > \text{ReLU}(x)$  for  $x \in \mathbb{R}$ .
Approximation guarantees for the fat-tailed non-linearities of App. A.4 can be derived similarly.
D Additional Empirical Details and Results
All algorithms are implemented in BoTorch. The analytic EI, qEI, cEI utilize the standard BoTorch implementations. We utilize the original authors' implementations of single objective JES [36], GIBBON [61], and multi-objective JES [71], which are all available in the main BoTorch repository. All simulations are ran with 32 replicates and error bars represent  $\pm 2$  times the standard error of the mean. We use a Matern-5/2 kernel with automatic relevance determination (ARD), i.e. separate length-scales for each input dimension, and a top-hat prior on the length-scales in [0.01, 100]. The input spaces are normalized to the unit hyper-cube and the objective values are standardized during each optimization iteration.
D.2 Additional Empirical Results on Vanishing Values and Gradients
The left plot of Figure 1 in the main text shows that for a large fraction of points across the domain the gradients of EI are numerically essentially zero. In this section we provide additional detail on these simulations as well as intuition for results.
The data generating process (DGP) for the training data used for the left plot of Figure 1 is the following:  $80\%$  of training points are sampled uniformly at random from the domain, while  $20\%$  are sampled according to a multivariate Gaussian centered at the function maximum with a standard deviation of  $25\%$  of the length of the domain. The idea behind this DGP is to mimic the kind of data one would see during a Bayesian Optimization loop (without having to run thousands of BO loops to generate the Figure 1). Under this DGP with the chosen test problem, the incumbent (best observed point) is typically better than the values at the random test locations, and this becomes increasingly the case as the dimensionality of the problem increases and the number of training points grows. This is exactly the situation that is typical when conducting Bayesian Optimization.
Figure 15: Model fits for a typical replicate used in generating the Fig. 1 (left). While there is ample uncertainty in the test point predictions (blue, chosen uniformly at random), the mean prediction for the majority of points is many standard deviations away from the incumbent value (red).
Figure 16: Histogram of  $z(x)$ , the argument to  $h$  in (2), corresponding to Figure 15. Vertical lines are the thresholds corresponding to values  $z$  below which  $h(z)$  is less than the respective threshold. The majority of the test points fall below these threshold values.
For a particular replicate, Figure 15 shows the model fits in-sample (black), out-of-sample (blue), and the best point identified so far (red) with 60 training points and a random subset of 50 (out of 2000) test points. One can see that the model produces decent mean predictions for out-of-sample data, and that the uncertainty estimates appear reasonably well-calibrated (e.g., the credible intervals typically cover the true value). A practitioner would consider this a good model for the purposes of Bayesian Optimization. However, while there is ample uncertainty in the predictions of the model away from the training points, for the vast majority of points, the mean prediction is many standard deviations away from the incumbent value (the error bars are  $\pm 2$  standard deviations). This is the key reason for EI taking on zero (or vanishingly small) values and having vanishing gradients.
To illustrate this, Figure 16 shows the histogram of  $z(x)$  values, the argument to the function  $h$  in (2). It also contains the thresholds corresponding to the values  $z$  below which  $h(z)$  is less than the respective threshold. Since  $\sigma(x)$  is close to 1 for most test points (mean: 0.87, std: 0.07), this is more or less the same as saying that  $\operatorname{EI}(z(x))$  is less than the threshold. It is evident from the histogram that the majority of the test points fall below these threshold values (especially for larger thresholds), showing that the associated acquisition function values (and similarly the gradients) are numerically almost zero and causing issues during acquisition function optimization.
D.3 Parallel Expected Improvement
Figure 17: Parallel optimization performance on the Ackley and Levy functions in 16 dimensions. qLogEI outperforms all baselines on Ackley, where joint optimization of the batch also improves on the sequential greedy. On Levy, joint optimization of the batch with qLogEI starts out performing worse in terms of BO performance than sequential, but wins out over all sequential methods as the batch size increases.
Figure 18: Breakdown of the parallel optimization performance of Figure 17 per method, rather than per batch size. On Levy, qLogEI exhibits a much smaller deterioration in BO performance due to increases in parallelism than the methods relying on sequentially optimized batches of candidates.
Figure 17 reports optimization performance of parallel BO on the 16-dimensional Ackley and Levy functions for both sequential greedy and joint batch optimization. Besides the apparent substantial advantages of qLogEI over qEI on Ackley, a key observation here is that jointly optimizing the candidates of batch acquisition functions can yield highly competitive optimization performance, especially as the batch size increases. Notably, joint optimization of the batch with qLogEI starts out performing worse in terms of BO performance than sequential on the Levy function, but outperforms all sequential methods as the batch size increases. See also Figure 18 for the scaling of each method with respect to the batch size  $q$ .
Figure 19: Optimization performance with noisy observations on Hartmann 6d (top), Ackley 8d (mid), and Ackley 16 (bottom) for varying noise levels and  $q = 1$ . We set the noise level as a proportion of the maximum range of the function, which is  $\approx 3.3$  for Hartmann and  $\approx 20$  for Ackley. That is, the  $1\%$  noise level corresponds to a standard deviation of 0.2 for Ackley. qLogNEI outperforms both canonical EI counterparts and Gibbon significantly in most cases, especially in higher dimensions.
D.4 Noisy Expected Improvement
Figure 19 benchmarks the "noisy" variant, qLogNEI. Similar to the noiseless case, the advantage of the LogEI versions over the canonical counterparts grows as with the dimensionality of the problem, and the noisy version improves on the canonical versions for larger noise levels.
D.5 Multi-Objective optimization with qLogEHVI
Figure 20 compares qLogEHVI and qEHVI on 6 different test problems with 2 or 3 objectives, and ranging from 2-30 dimensions. This includes 3 real world inspired problems: cell network design for optimizing coverage and capacity [19], laser plasma acceleration optimization [38], and vehicle design optimization [54, 68]. The results are consistent with our findings in the single-objective and constrained cases: qLogEHVI consistently outperforms qEHVI, and the gap is larger on higher dimensional problems.
D.6 Combining LogEI with TuRBO for High-Dimensional Bayesian Optimization
In the main text, we show how LogEI performs particularly well relative to other baselines in high dimensional spaces. Here, we show how LogEI can work synergistically with trust region based methods for high-dimensional BO, such as TuRBO [24].
Fig. 21 compares the performance of LogEI, TuRBO-1 + LogEI, TuRBO-1 + EI, as well as the original Thompson-sampling based implementation for the 50d Ackley test problem. Combining TuRBO-1 with LogEI results in substantially better performance than the baselines when using a small number of function evaluations. Thompson sampling (TS) ultimately performs better after 10,000 evaluations, but this experiment shows the promise of combining TuRBO and LogEI in settings where we cannot do thousands of function evaluations. Since we optimize batches of  $q = 50$  candidates jointly, we also increase the number of Monte-Carlo samples from the Gaussian process
Figure 20: Sequential  $(q = 1)$  optimization performance on multi-objective problems, as measured by the hypervolume of the Pareto frontier across observed points. This plot includes JES [71]. Similar to the single-objective case, qLogEHVI significantly outperforms all baselines on all test problems.