SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sebastian Ament
ament@meta.com
Samuel Daulton
sdaulton@meta.com
David Eriksson
deriksson@meta.com
Maximilian Balandat
balandat@meta.com
Eytan Bakshy
ebakshy@meta.com
Expected Improvement (EI) is arguably the most popular acquisition function in Bayesian optimization and has found countless successful applications, but its performance is often exceeded by that of more recent methods. Notably, EI and its variants, including for the parallel and multi-objective settings, are challenging to optimize because their acquisition values vanish numerically in many regions. This difficulty generally increases as the number of observations, dimensionality of the search space, or the number of constraints grow, resulting in performance that is inconsistent across the literature and most often sub-optimal. Herein, we propose LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically. We demonstrate that numerical pathologies manifest themselves in "classic" analytic EI, Expected Hypervolume Improvement (EHVI), as well as their constrained, noisy, and parallel variants, and propose corresponding reformulations that remedy these pathologies. Our empirical results show that members of the LogEI family of acquisition functions substantially improve on the optimization performance of their canonical counterparts and surprisingly, are on par with or exceed the performance of recent state-of-the-art acquisition functions, highlighting the understated role of numerical optimization in the literature.
Bayesian Optimization (BO) is a widely used and effective approach for sample-efficient optimization of expensive-to-evaluate black-box functions [25, 28], with applications ranging widely between aerospace engineering [48], biology and medicine [49], materials science [3], civil engineering [4], and machine learning hyperparameter optimization [66, 72]. BO leverages a probabilistic surrogate model in conjunction with an acquisition function to determine where to query the underlying objective function. Improvement-based acquisition functions, such as Expected Improvement (EI) and Probability of Improvement (PI), are among the earliest and most widely used acquisition functions for efficient global optimization of non-convex functions [42, 58]. EI has been extended to the constrained [27, 29], noisy [52], and multi-objective [20] setting, as well as their respective batch variants [6, 13, 77], and is a standard baseline in the BO literature [25, 66]. While much of the literature has focused on developing new sophisticated acquisition functions, subtle yet critical implementation details of foundational BO methods are often overlooked. Importantly, the performance of EI and its variants is inconsistent even for mathematically identical formulations and, as we show in this work, most often sub-optimal.
Although the problem of optimizing EI effectively has been discussed in various works, e.g. [25, 31, 77], prior focus has been on optimization algorithms and initialization strategies, rather than the fundamental issue of computing EI.
In this work, we identify pathologies in the computation of improvement-based acquisition functions that give rise to numerically vanishing values and gradients, which – to our knowledge – are present in all existing implementations of  $EI$ , and propose reformulations that lead to increases in the associated optimization performance which often match or exceed that of recent methods.
1. We introduce LogEI, a new family of acquisition functions whose members either have identical or approximately equal optima as their canonical counterparts, but are substantially easier to optimize numerically. Notably, the analytic variant of LogEI, which mathematically results in the same BO policy as EI, empirically shows significantly improved optimization performance.
2. We extend the ideas behind analytical LogEI to other members of the EI family, including constrained EI (CEI), Expected Hypervolume Improvement (EHVI), as well as their respective batch variants for parallel BO, qEI and qEHVI, using smooth approximations of the acquisition utilities to obtain non-vanishing gradients. All of our methods are available as part of BoTorch [6].
3. We demonstrate that our newly proposed acquisition functions substantially outperform their respective analogues on a broad range of benchmarks without incurring meaningful additional computational cost, and often match or exceed the performance of recent methods.
Figure 1: Left: Fraction of points sampled from the domain for which the magnitude of the gradient of EI vanishes to  $< 10^{-10}$  as a function of the number of randomly generated data points  $n$  for different dimensions  $d$  on the Ackley function. As  $n$  increases, EI and its gradients become numerically zero across most of the domain, see App. D.2 for details. Right: Values of EI and LogEI on a quadratic objective. EI takes on extremely small values on points for which the likelihood of improving over the incumbent is small and is numerically exactly zero in double precision for a large part of the domain ( $\approx [5, 13.5]$ ). The left plot shows that this tends to worsen as the dimensionality of the problem and the number of data points grow, rendering gradient-based optimization of EI futile.
Maximizing acquisition functions for BO is a challenging problem, which is generally non-convex and often contains numerous local maxima, see the lower right panel of Figure 1. While zeroth-order methods are sometimes used, gradient-based methods tend to be far more effective at optimizing acquisition functions on continuous domains, especially in higher dimensions.
In addition to the challenges stemming from non-convexity that are shared across acquisition functions, the values and gradients of improvement-based acquisition functions are frequently minuscule in large swaths of the domain. Although EI is never mathematically zero under a Gaussian posterior distribution, it often vanishes, even becoming exactly zero in floating point precision. The same
1 except at any point that is perfectly correlated with a previous noiseless observation
applies to its gradient, making EI (and PI, see Appendix A) exceptionally difficult to optimize via gradient-based methods. The right panels of Figure 1 illustrate this behavior on a simple one-dimensional quadratic function.
To increase the chance of finding the global optimum of non-convex functions, gradient-based optimization is typically performed from multiple starting points, which can help avoid getting stuck in local optima [70]. For improvement-based acquisition functions however, optimization becomes increasingly challenging as more data is collected and the likelihood of improving over the incumbent diminishes, see our theoretical results in Section 3 and the empirical illustration in Figure 1 and Appendix D.2. As a result, gradient-based optimization with multiple random starting points will eventually degenerate into random search when the gradients at the starting points are numerically zero. This problem is particularly acute in high dimensions and for objectives with a large range.
Various initialization heuristics have been proposed to address this behavior by modifying the random_restart strategy. Rather than starting from random candidates, an alternative naive approach would be to use initial conditions close to the best previously observed inputs. However, doing that alone inherently limits the acquisition optimization to a type of local search, which cannot have global guarantees. To attain such guarantees, it is necessary to use an asymptotically space-filling heuristic; even if not random, this will entail evaluating the acquisition function in regions where no prior observation lies. Ideally, these regions should permit gradient-based optimization of the objective for efficient acquisition function optimization, which necessitates the gradients to be non-zero. In this work, we show that this can be achieved for a large number of improvement-based acquisition functions, and demonstrate empirically how this leads to substantially improved BO performance.
We consider the problem of maximizing an expensive-to-evaluate black-box function  $\mathbf{f}_{\mathrm{true}}: \mathbb{X} \mapsto \mathbb{R}^M$  over some feasible set  $\mathbb{X} \subseteq \mathbb{R}^d$ . Suppose we have collected data  $\mathcal{D}_n = \{(\mathbf{x}_i, \mathbf{y}_i)\}_{i=1}^n$ , where  $\mathbf{x}_i \in \mathbb{X}$  and  $\mathbf{y}_i = \mathbf{f}_{\mathrm{true}}(\mathbf{x}_i) + \mathbf{v}_i(\mathbf{x}_i)$  and  $\mathbf{v}_i$  is a noise corrupting the true function value  $\mathbf{f}_{\mathrm{true}}(\mathbf{x}_i)$ . The response  $\mathbf{f}_{\mathrm{true}}$  may be multi-output as is the case for multiple objectives or black-box constraints, in which case  $\mathbf{y}_i, \mathbf{v}_i \in \mathbb{R}^M$ . We use Bayesian optimization (BO), which relies on a surrogate model  $\mathbf{f}$  that for any batch  $\mathbf{X} := \{\mathbf{x}_1, \dots, \mathbf{x}_q\}$  of candidate points provides a probability distribution over the outputs  $f(\mathbf{X}) := (f(\mathbf{x}_1), \dots, f(\mathbf{x}_q))$ . The acquisition function  $\alpha$  then utilizes this posterior prediction to assign an acquisition value to  $\mathbf{x}$  that quantifies the value of evaluating the points in  $\mathbf{x}$ , trading off exploration and exploitation.
Gaussian Processes (GP) [65] are the most widely used surrogates in BO, due to their high data efficiency and good uncertainty quantification. For our purposes, it suffices to consider a GP as a mapping that provides a multivariate Normal distribution over the outputs  $f(\mathbf{x})$  for any  $\mathbf{x}$ :
$$
f (\mathbf {x}) \sim \mathcal {N} (\boldsymbol {\mu} (\mathbf {x}), \boldsymbol {\Sigma} (\mathbf {x})), \quad \boldsymbol {\mu}: \mathbb {X} ^ {q} \rightarrow \mathbb {R} ^ {q M}, \quad \boldsymbol {\Sigma}: \mathbb {X} ^ {q} \rightarrow \mathcal {S} _ {+} ^ {q M}. \tag {1}
$$
In the single-outcome  $(M = 1)$  setting,  $f(\mathbf{x})\sim \mathcal{N}(\mu (\mathbf{x}),\Sigma (\mathbf{x}))$  with  $\mu :\mathbb{X}^q\to \mathbb{R}^q$  and  $\Sigma :\mathbb{X}^q\rightarrow S_+^q$ . In the sequential  $(q = 1)$  case, this further reduces to a univariate Normal distribution:  $f(\mathbf{x})\sim \mathcal{N}(\mu (\mathbf{x}),\sigma^2 (\mathbf{x}))$  with  $\mu :\mathbb{X}\to \mathbb{R}$  and  $\sigma :\mathbb{X}\to \mathbb{R}_{+}$ .
2.2 Improvement-based Acquisition Functions
Expected Improvement For the fully-sequential  $(q = 1)$ , single-outcome  $(M = 1)$  setting, "classic" EI [59] is defined as
$$
\operatorname {E I} _ {y ^ {*}} (\mathbf {x}) = \mathbb {E} _ {f (\mathbf {x})} \left[ \left[ f (\mathbf {x}) - y ^ {*} \right] _ {+} \right] = \sigma (\mathbf {x}) h \left(\frac {\mu (\mathbf {x}) - y ^ {*}}{\sigma (\mathbf {x})}\right), \tag {2}
$$
where  $[\cdot ]_{+}$  denotes the  $\max (0,\cdot)$  operation,  $y^{*} = \max_{i}y_{i}$  is the best function value observed so far, also referred to as the incumbent,  $h(z) = \phi (z) + z\Phi (z)$ , and  $\phi ,\Phi$  are the standard Normal density and distribution functions, respectively. This formulation is arguably the most widely used acquisition function in BO, and the default in many popular software packages.
Constrained Expected Improvement Constrained BO involves one or more black-box constraints and is typically formulated as finding  $\max_{\mathbf{x} \in \mathbb{X}} f_{\mathrm{true},1}(\mathbf{x})$  such that  $f_{\mathrm{true},i}(\mathbf{x}) \leq 0$  for  $i \in \{2, \dots, M\}$ . Feasibility-weighting the improvement [27, 29] is a natural approach for this class of problems:
$$
\operatorname {C E I} _ {y ^ {*}} (\mathbf {x}) = \mathbb {E} _ {\mathbf {f} (\mathbf {x})} \left[ \left[ f _ {1} (\mathbf {x}) - y ^ {*} \right] _ {+} \prod_ {i = 2} ^ {M} \mathbb {1} _ {f _ {i} (\mathbf {x}) \leq 0} \right], \tag {3}
$$
where  $\mathbb{1}$  is the indicator function. If the constraints  $\{f_i\}_{i\geq 2}$  are modeled as conditionally independent of the objective  $f_{1}$  this can be simplified as the product of EI and the probability of feasibility.
Parallel Expected Improvement In many settings, one may evaluate  $f_{\mathrm{true}}$  on  $q > 1$  candidates in parallel to increase throughput. The associated parallel or batch analogue of EI [30, 75] is given by
$$
\operatorname {q E I} _ {y ^ {*}} (\mathbf {X}) = \mathbb {E} _ {f (\mathbf {X})} \left[ \max  _ {j = 1, \dots , q} \left\{\left[ f \left(\mathbf {x} _ {j}\right) - y ^ {*} \right] _ {+} \right\} \right]. \tag {4}
$$
Unlike EI, qEI does not admit a closed-form expression and is thus typically computed via Monte Carlo sampling, which also extends to non-Gaussian posterior distributions [6, 75]:
$$
\operatorname {q E I} _ {y ^ {*}} (\mathbf {X}) \approx \sum_ {i = 1} ^ {N} \max  _ {j = 1, \dots , q} \left\{\left[ \xi^ {i} \left(\mathbf {x} _ {j}\right) - y ^ {*} \right] _ {+} \right\}, \tag {5}
$$
where  $\xi^i (\mathbf{x})\sim f(\mathbf{x})$  are random samples drawn from the joint model posterior at  $\mathbf{x}$
Expected Hypervolume Improvement In multi-objective optimization (MOO), there generally is no single best solution; instead the goal is to explore the Pareto Frontier between multiple competing objectives, the set of mutually-optimal objective vectors. A common measure of the quality of a finitely approximated Pareto Frontier  $\mathcal{P}$  between  $M$  objectives with respect to a specified reference point  $\mathbf{r} \in \mathbb{R}^{M}$  is its hypervolume  $\mathrm{HV}(\mathcal{P}, \mathbf{r}) \coloneqq \lambda \left( \bigcup_{\mathbf{y}_{i} \in \mathcal{P}} [\mathbf{r}, \mathbf{y}_{i}] \right)$ , where  $[\mathbf{r}, \mathbf{y}_{i}]$  denotes the hyperrectangle bounded by vertices  $\mathbf{r}$  and  $\mathbf{y}_{i}$ , and  $\lambda$  is the Lebesgue measure. An apt acquisition function for multi-objective optimization problems is therefore the expected hypervolume improvement
$$
\operatorname {E H V I} (\mathbf {x}) = \mathbb {E} _ {\mathbf {f} (\mathbf {X})} \left[ \left[ \operatorname {H V} (\mathcal {P} \cup \mathbf {f} (\mathbf {X}), \mathbf {r}) - \operatorname {H V} (\mathcal {P}, \mathbf {r}) \right] _ {+} \right], \tag {6}
$$
due to observing a batch  $\mathbf{f}(\mathbf{X})\coloneqq [\mathbf{f}(\mathbf{x}_1),\dots ,\mathbf{f}(\mathbf{x}_q)]$  of  $q$  new observations. EHVI can be expressed in closed form if  $q = 1$  and the objectives are modeled with independent GPs [80], but Monte Carlo approximations are required for the general case (qEHVI) [13].
2.3 Optimizing Acquisition Functions
Optimizing an acquisition function (AF) is a challenging task that amounts to solving a non-convex optimization problem, to which multiple approaches and heuristics have been applied. These include gradient-free methods such as divided rectangles [41], evolutionary methods such as CMA-ES [32], first-order methods such as stochastic gradient ascent, see e.g., Daulton et al. [15], Wang et al. [75], and (quasi-)second order methods [25] such as L-BFGS-B [10]. Multi-start optimization is commonly employed with gradient-based methods to mitigate the risk of getting stuck in local minima. Initial points for optimization are selected via various heuristics with different levels of complexity, ranging from simple uniform random selection to BoTorch's initialization heuristic, which selects initial points by performing Boltzmann sampling on a set of random points according to their acquisition function value [6]. See Appendix B for a more complete account of initialization strategies and optimization procedures used by popular implementations. We focus on gradient-based optimization as often leveraging gradients results in faster and more performant optimization [13].
Optimizing AFs for parallel BO that quantify the value of a batch of  $q > 1$  points is more challenging than optimizing their sequential counterparts due to the higher dimensionality of the optimization problem -  $qd$  instead of  $d$  - and the more challenging optimization surface. A common approach to simplify the problem is to use a sequential greedy strategy that greedily solves a sequence of single point selection problems. For  $i = 1, \dots, q$ , candidate  $\mathbf{x}_i$  is selected by optimizing the AF for  $q = 1$ , conditional on the previously selected designs  $\{\mathbf{x}_1, \dots, \mathbf{x}_{i-1}\}$  and their unknown observations, e.g. by fantasizing the values at those designs [77]. For submodular AFs, including EI, PI, and EHVI, a sequential greedy strategy will attain a regret within a factor of  $1/e$  compared to the joint optimum, and previous works have found that sequential greedy optimization yields improved BO performance compared to joint optimization [13, 77]. Herein, we find that our reformulations enable joint batch optimization to be competitive with the sequential greedy strategy, especially for larger batches.
While there is a substantial body of work introducing a large variety of different AFs, much less focus has been on the question of how to effectively implement and optimize these AFs. Zhan and Xing [81] provide a comprehensive review of a large number of different variants of the EI family, but do not discuss any numerical or optimization challenges. Zhao et al. [82] propose combining a variety of different initialization strategies to select initial conditions for optimization of acquisition functions and show empirically that this improves optimization performance. However, they do not address any potential issues or degeneracies with the acquisition functions themselves. Recent works have considered effective gradient-based approaches for acquisition optimization. Wilson et al. [77] demonstrates how stochastic first-order methods can be leveraged for optimizing Monte Carlo acquisition functions. Balandat et al. [6] build on this work and put forth sample average approximations for MC acquisition functions that admit gradient-based optimization using deterministic higher-order optimizers such as L-BFGS-B.
Another line of work proposes to switch from BO to local optimization based on some stopping criterion to achieve faster local convergence, using either zeroth order [60] or gradient-based [57] optimization. While McLeod et al. [57] are also concerned with numerical issues, we emphasize that those issues arise due to ill-conditioned covariance matrices and are orthogonal to the numerical pathologies of improvement-based acquisition functions.
3 Theoretical Analysis of Expected Improvement's Vanishing Gradients
In this section, we shed light on the conditions on the objective function and surrogate model that give rise to the numerically vanishing gradients in EI, as seen in Figure 1. In particular, we show that as a BO algorithm closes the optimality gap  $f^{*} - y^{*}$ , where  $f^{*}$  is the global maximum of the function  $f_{\mathrm{true}}$ , and the associated GP surrogate's uncertainty decreases, EI is exceedingly likely to exhibit numerically vanishing gradients.
Let  $P_{\mathbf{x}}$  be a distribution over the inputs  $\mathbf{x}$ , and  $f \sim P_f$  be an objective drawn from a Gaussian process. Then with high probability over the particular instantiation  $f$  of the objective, the probability that an input  $\mathbf{x} \sim P_{\mathbf{x}}$  gives rise to an argument  $(\mu(\mathbf{x}) - y^*) / \sigma(\mathbf{x})$  to  $h$  in Eq. (2) that is smaller than a threshold  $B$  exceeds  $P_{\mathbf{x}}(f(\mathbf{x}) < f^* - \epsilon_n)$ , where  $\epsilon_n$  depends on the optimality gap  $f^* - y^*$  and the maximum posterior uncertainty  $\max_{\mathbf{x}} \sigma_n(\mathbf{x})$ . This pertains to EI's numerically vanishing values and gradients, since the numerical support  $S_{\eta}(h) = \{\mathbf{x} : |h(\mathbf{x})| > \eta\}$  of a naive implementation of  $h$  in (2) is limited by a lower bound  $B(\eta)$  that depends on the floating point precision  $\eta$ . Formally,  $S_{\eta}(h) \subset [B(\eta), \infty)$  even though  $S_0(h) = \mathbb{R}$  mathematically. As a consequence, the following result can be seen as a bound on the probability of encountering numerically vanishing values and gradients in EI using samples from the distribution  $P_{\mathbf{x}}$  to initialize the optimization of the acquisition function.
Theorem 1. Suppose  $f$  is drawn from a Gaussian process prior  $P_{f}$ ,  $y^{*} \leq f^{*}$ ,  $\mu_{n}, \sigma_{n}$  are the mean and standard deviation of the posterior  $P_{f}(f|\mathcal{D}_{n})$  and  $B \in \mathbb{R}$ . Then with probability  $1 - \delta$ ,
$$
P _ {\mathbf {x}} \left(\frac {\mu_ {n} (\mathbf {x}) - y ^ {*}}{\sigma_ {n} (\mathbf {x})} <   B\right) \geq P _ {\mathbf {x}} \left(f (\mathbf {x}) <   f ^ {*} - \epsilon_ {n}\right) \tag {7}
$$
where  $\epsilon_{n} = (f^{*} - y^{*}) + \left(\sqrt{-2\log(2\delta)} -B\right)\max_{\mathbf{x}}\sigma_{n}(\mathbf{x})$
For any given – and especially early – iteration,  $\epsilon_{n}$  does not have to be small, as both the optimality gap and the maximal posterior standard deviation can be large initially. Note that under certain technical conditions on the kernel function and the asymptotic distribution of the training data  $\mathcal{D}_n$ , the maximum posterior variance vanishes guaranteeably as  $n$  increases, see [50, Corollary 3.2]. On its own, Theorem 1 gives insight into the non-asymptotic behavior by exposing a dependence to the distribution of objective values  $f$ . In particular, if the set of inputs that give rise to high objective values  $(\approx f^{*})$  is concentrated,  $P(f(\mathbf{x}) < f^{*} - \epsilon)$  will decay very slowly as  $\epsilon$  increases, thereby maintaining a lower bound on the probability of close to 1. As an example, this is the case for the Ackley function, especially as the dimensionality increases, which explains the behavior in Figure 1.
4 Unexpected Improvements
In this section, we propose re-formulations of analytic and MC-based improvement-based acquisition functions that render them significantly easier to optimize. We will use differing fonts, e.g. log and  $\log$ , to differentiate between the mathematical functions and their numerical implementations.
4.1 Analytic LogEI
Mathematically, EI's values and gradients are nonzero on the entire real line, except in the noiseless case for points that are perfectly correlated with previous observations. However, naive implementations of  $h$  are numerically zero when  $z = (\mu(\mathbf{x}) - y^*) / \sigma(\mathbf{x})$  is small, which happens when the model has high confidence that little improvement can be achieved at  $\mathbf{x}$ . We propose an implementation of  $\log \circ h$  that can be accurately computed for a much larger range of inputs. Specifically, we compute
$$
\operatorname {L o g E I} _ {y ^ {*}} (\mathbf {x}) = \log_ {\mathrm {h}} \left(\left(\mu (\mathbf {x}) - y ^ {*}\right) / \sigma (\mathbf {x})\right) + \log (\sigma (\mathbf {x})) \tag {8}
$$
where  $\log_{\mathbf{h}}$  is mathematically equivalent to  $\log \circ h$  and can be stably and accurately computed by
$$
\log_ {-} \mathrm {h} (z) = \left\{ \begin{array}{l l} \log (\phi (z) + z \Phi (z)) & z > - 1 \\ - z ^ {2} / 2 - c _ {1} + \log 1 \operatorname {m e x p} (\log (\operatorname {e r f c x} (- z / \sqrt {2}) | z |) + c _ {2}) & - 1 / \sqrt {\epsilon} <   z \leq - 1 \\ - z ^ {2} / 2 - c _ {1} - 2 \log (| z |) & z \leq - 1 / \sqrt {\epsilon} \end{array} \right. \tag {9}
$$