SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $c_{1} = \log (2\pi) / 2$ , and  $c_{2} = \log (\pi /2) / 2$ ,  $\epsilon$  is the numerical precision, and  $\log 1\mathrm{mexp}$ ,  $\operatorname{erfcx}$  are numerically stable implementations of  $\log (1 - \exp (z))$  and  $\exp (z^2)\mathrm{erfc}(z)$ , respectively, see App. A. Progenitors of Eq. (9) are found in SMAC 1.0 [35] and RoBO [46] which contain a log-transformed analytic EI implementation that is much improved, but can still exhibit instabilities as  $z$  grows negative, see App. Fig. 10. To remedy similar instabilities, we put forth the third, asymptotic case in Eq. (9), ensuring numerical stability throughout, see App. A.2 for details. The asymptotically quadratic behavior of  $\log_{-}\mathfrak{h}$  becomes apparent in the last two cases, making the function particularly amenable to gradient-based optimization with significant practical implications for EI-based BO.
4.2 Monte Carlo Parallel LogEI
Beyond analytic EI, Monte Carlo formulations of parallel EI that perform differentiation on the level of MC samples, don't just exhibit numerically, but mathematically zero gradients for a significant proportion of practically relevant inputs. For qEI, the primary issue is the discrete maximum over the  $q$  outcomes for each MC sample in (5). In particular, the acquisition utility of expected improvement in Eq. 4 on a single sample  $\xi_{i}$  of  $f$  is  $\max_j[\xi_i(\mathbf{x}_j) - y^*]_+$ . Mathematically, we smoothly approximate the acquisition utility in two stages: 1)  $u_{ij} = \mathrm{softplus}_{\tau_0}(\xi_i(\mathbf{x}_j) - y^*) \approx [\xi_i(\mathbf{x}_j) - y^*]_+$  and 2)  $\| u_i \|_{1 / \tau_{\mathrm{max}}} \approx \max_j u_{ij}$ . Notably, while we use canonical softplus and p-norm approximations here, specialized fat-tailed non-linearities are required to scale to large batches, see Appendix A.4. Since the resulting quantities are strictly positive, they can be transformed to log-space permitting an implementation of qLogEI that is numerically stable and can be optimized effectively. In particular,
$$
\begin{array}{l} \operatorname {q L o g E I} _ {y ^ {*}} (\mathbf {X}) = \log \int \left(\sum_ {j = 1} ^ {q} \operatorname {s o f t p l u s} _ {\tau_ {0}} \left(f \left(\mathbf {x} _ {j}\right) - y ^ {*}\right) ^ {1 / \tau_ {\max }}\right) ^ {\tau_ {\max }} d f \tag {10} \\ \approx \operatorname {l o g s u m e x p} _ {i} \left(\tau_ {\max } \operatorname {l o g s u m e x p} _ {j} \left(\operatorname {l o g s o f t p l u s} _ {\tau_ {0}} \left(\xi^ {i} \left(\mathbf {x} _ {j}\right) - y ^ {*}\right)\right) / \tau_ {\max }\right)), \\ \end{array}
$$
where  $i$  is the index of the Monte Carlo draws from the GP posterior,  $j = 1,\dots ,q$  is the index for the candidate in the batch, and logsoftplus is a numerically stable implementation of  $\log (\log (1 + \exp (z)))$ . See Appendix A.3 for details, including the novel fat-tailed non-linearities like fatplus.
While the smoothing in (10) approximates the canonical qEI formulation, the following result shows that the associated relative approximation error can be quantified and bounded tightly as a function of the temperature parameters  $\tau_0$ ,  $\tau_{\mathrm{max}}$  and the batch size  $q$ . See Appendix C for the proof.
Lemma 2. [Relative Approximation Guarantee] Given  $\tau_0, \tau_{\max} > 0$ , the approximation error of qLogEI to qEI is bounded by
$$
\left| e ^ {\mathrm {q} \operatorname {L o g E I} (\mathbf {X})} - \mathrm {q E I} (\mathbf {X}) \right| \leq \left(q ^ {\tau_ {\max }} - 1\right) \mathrm {q E I} (\mathbf {X}) + \log (2) \tau_ {0} q ^ {\tau_ {\max }}. \tag {11}
$$
In Appendix D.10, we show the importance of setting the temperatures sufficiently low for qLogEI to achieve good optimization characteristics, something that only becomes possible by transforming all involved computations to log-space. Otherwise, the smooth approximation to the acquisition utility would exhibit vanishing gradients numerically, as the discrete max operator does mathematically.
4.3 Constrained EI
Both analytic and Monte Carlo variants of LogEI can be extended for optimization problems with black-box constraints. For analytic CEI with independent constraints of the form  $f_{i}(\mathbf{x}) \leq 0$ , the constrained formulation in Eq. (3) simplifies to  $\mathrm{LogCEI}(\mathbf{x}) = \mathrm{LogEI}(\mathbf{x}) + \sum_{i} \log(P(f_{i}(\mathbf{x}) \leq 0))$ , which can be readily and stably computed using LogEI in Eq. (8) and, if  $f_{i}$  is modelled by a GP, a stable implementation of the Gaussian log cumulative distribution function. For the Monte Carlo variant, we apply a similar strategy as for Eq. (10) to the constraint indicators in Eq. (3): 1) a smooth approximation and 2) an accurate and stable implementation of its log value, see Appendix A.
4.4 Monte Carlo Parallel LogEHVI
The numerical difficulties of qEHVI in (6) are similar to those of qEI, and the basic ingredients of smoothing and log-transformations still apply, but the details are significantly more complex since qEHVI uses many operations that have mathematically zero gradients with respect to some of the inputs. Our implementation is based on the differentiable inclusion-exclusion formulation of the hypervolume improvement [13]. As a by-product, the implementation also readily allows for the differentiable computation of the expected log hypervolume, instead of the log expected hypervolume, note the order, which can be preferable in certain applications of multi-objective optimization [26].
We compare standard versions of analytic EI (EI) and constrained EI (CEI), Monte Carlo parallel EI (qEI), as well as Monte Carlo EHVI (qEHVI), in addition to other state-of-the-art baselines like lower-bound Max-Value Entropy Search (GIBBON) [61] and single- and multi-objective Joint Entropy Search (JES) [36, 71]. All experiments are implemented using BoTorch [6] and utilize multi-start optimization of the AF with scipy's L-BFGS-B optimizer. In order to avoid conflating the effect of BoTorch's default initialization strategy with those of our contributions, we use 16 initial points chosen uniformly at random from which to start the L-BFGS-B optimization. For a comparison with other initialization strategies, see Appendix D. We run multiple replicates and report mean and error bars of  $\pm 2$  standard errors of the mean. Appendix D.1 contains additional details.
Single-objective sequential BO We compare EI and LogEI on the 10-dimensional convex Sum-of-Squares (SoS) function  $f(\mathbf{x}) = \sum_{i=1}^{10}(x_i - 0.5)^2$ , using 20 restarts seeded from 1024 pseudorandom samples through BoTorch's default initialization heuristic. Figure 2 shows that due to vanishing gradients, EI is unable to make progress even on this trivial problem.
Figure 2: Regret and EI acquisition value for the candidates selected by maximizing EI and LogEI on the convex Sum-of-Squares problem. Optimization stalls out for EI after about 75 observations due to vanishing gradients (indicated by the jagged behavior of the acquisition value), while LogEI continues to make steady progress.
In Figure 3, we compare performance on the Ackley and Michalewicz test functions [67]. Notably, LogEI substantially outperforms EI on Ackley as the dimensionality increases. Ackley is a challenging multimodal function for which it is critical to trade off local exploitation with global exploration, a task made exceedingly difficult by the numerically vanishing gradients of EI in a large fraction of the search space. We see a similar albeit less pronounced behavior on Michalewicz, which reflects the fact that Michalewicz is a somewhat less challenging problem than Ackley.
Figure 3: Best objective value as a function of iterations on the moderately and severely non-convex Michalewicz and Ackley problems for varying numbers of input dimensions. LogEI substantially outperforms both EI and GIBBON, and this gap widens as the problem dimensionality increases. JES performs slightly better than LogEI on Ackley, but for some reason fails on Michalewicz. Notably, JES is almost two orders of magnitude slower than the other acquisition functions (see Appendix D).
Figure 4: Best feasible objective value as a function of number of function evaluations (iterations) on four engineering design problems with black-box constraints after an initial  $2d$  pseudo-random evaluations.
BO with Black Box Constraints Figure 4 shows results on four engineering design problems with black box constraints that were also considered in [22]. We apply the same bilog transform as the trust region-based SCBO method [22] to all constraints to make them easier to model with a GP. We see that LogCEI outperforms the naive CEI implementation and converges faster than SCBO. Similar to the unconstrained problems, the performance gains of LogCEI over CEI grow with increasing problem dimensionality and the number of constraints. Notably, we found that for some problems, LogCEI in fact improved upon some of the best results quoted in the original literature, while using three orders of magnitude fewer function evaluations, see Appendix D.7 for details.
Parallel Expected Improvement with qLogEI Figure 5 reports the optimization performance of parallel BO on the 16-dimensional Ackley function for both sequential greedy and joint batch optimization using the fat-tailed non-linearities of App. A.4. In addition to the apparent advantages of qLogEI over qEI, a key finding is that jointly optimizing the candidates of batch acquisition functions can yield highly competitive optimization performance, see App. D.3 for extended results.
High-dimensional BO with qLogEI Figure 6 shows the performance of LogEI on three high-dimensional problems: the 6-dimensional Hartmann function embedded in a 100-dimensional space, a 100-dimensional rover trajectory planning problem, and a 103-dimensional SVM hyperparameter tuning problem. We use a 103-dimensional version of the 388-dimensional SVM problem considered by Eriksson and Jankowiak [21], where the 100 most important features were selected using Xgboost.
Figure 5: Best objective value for parallel BO as a function of the number evaluations for single-objective optimization on the 16-dimensional Ackley function with varying batch sizes  $q$ . Notably, joint optimization of the batch outperforms sequential greedy optimization.
Figure 6: Best objective value as a function of number of function evaluations (iterations) on three high-dimensional problems, including Eriksson and Jankowiak [21]'s SAAS prior.
Figure 6 shows that the optimization exhibits varying degrees of improvement from the inclusion of qLogEI, both when combined with SAASBO [21] and a standard GP. In particular, qLogEI leads to significant improvements on the embedded Hartmann problem, even leading BO with the canonical GP to ultimately catch up with the SAAS-prior-equipped model. On the other hand, the differences on the SVM and Rover problems are not significant, see Section 6 for a discussion.
Multi-Objective optimization with qLogEHVI Figure 7 compares qLogEHVI and qEHVI on two multi-objective test problems with varying batch sizes, including the real-world-inspired cell network design for optimizing coverage and capacity [19]. The results are consistent with our findings in the single-objective and constrained cases: qLogEHVI consistently outperforms qEHVI and even JES [71] for all batch sizes. Curiously, for the largest batch size and DTLZ2, qLogNEHVI's improvement over the reference point  $(\mathrm{HV} > 0)$  occurs around three batches after the other methods, but dominates their performance in later batches. See Appendix D.5 for results on additional synthetic and real-world-inspired multi-objective problems such as the laser plasma acceleration optimization [38], and vehicle design optimization [54, 68]
To recap, EI exhibits vanishing gradients 1) when high objective values are highly concentrated in the search space, and 2) as the optimization progresses. In this section, we highlight that these conditions are not met for all BO applications, and that LogEI's performance depends on the surrogate's quality.
On problem dimensionality While our experimental results show that advantages of LogEI generally grow larger as the dimensionality of the problem grows, we stress that this is fundamentally due to the concentration of high objective values in the search space, not the dimensionality itself. Indeed, we have observed problems with high ambient dimensionality but low intrinsic dimensionality, where LogEI does not lead to significant improvements over EI, e.g. the SVM problem in Figure 6.
Figure 7: Batch optimization performance on two multi-objective problems, as measured by the hypervolume of the Pareto frontier across observed points. This plot includes JES [71]. Similar to the single-objective case, the LogEI variant qLogEHVI significantly outperforms the baselines.
On asymptotic improvements While members of the LogEI family can generally be optimized better, leading to higher acquisition values, improvements in optimization performance might be small in magnitude, e.g. the log-objective results on the convex 10D sum of squares in Fig. 2, or only begin to materialize in later iterations, like for  $q = 16$  on DTLZ2 in Figure 7.
On model quality Even if good objective values are concentrated in a small volume of the search space and many iterations are run, LogEI might still not outperform EI if the surrogate's predictions are poor, or its uncertainties are not indicative of the surrogate's mismatch to the objective, see Rover in Fig. 6. In these cases, better acquisition values do not necessarily lead to better BO performance.
Replacing EI Despite these limitations, we strongly suggest replacing variants of EI with their LogEI counterparts. If LogEI were dominated by EI on some problem, it would be an indication that the EI family itself is sub-optimal, and improvements in performance can be attributed to the exploratory quality of randomly distributed candidates, which could be incorporated explicitly.
Our results demonstrate that the problem of vanishing gradients is a major source of the difficulty of optimizing improvement-based acquisition functions and that we can mitigate this issue through careful reformulations and implementations. As a result, we see substantially improved optimization performance across a variety of modified EI variants across a broad range of problems. In particular, we demonstrate that joint batch optimization for parallel BO can be competitive with, and at times exceed the sequential greedy approach typically used in practice, which also benefits from our modifications. Besides the convincing performance improvements, one of the key advantages of our modified acquisition functions is that they are much less dependent on heuristic and potentially brittle initialization strategies. Moreover, our proposed modifications do not meaningfully increase the computational complexity of the respective original acquisition function.
While our contributions may not apply verbatim to other classes of acquisition functions, our key insights and strategies do translate and could help with e.g. improving information-based [34, 76], cost-aware [51, 66], and other types of acquisition functions that are prone to similar numerical challenges. Further, combining the proposed methods with gradient-aware first-order BO methods [5, 16, 23] could lead to particularly effective high-dimensional applications of BO, since the advantages of both methods tend to increase with the dimensionality of the search space. Overall, we hope that our findings will increase awareness in the community for the importance of optimizing acquisition functions well, and in particular, for the care that the involved numerics demand.
The authors thank Frank Hutter for valuable references about prior work on numerically stable computations of analytic EI, David Bindel for insightful conversations about the difficulty of optimizing EI, as well as the anonymous reviewers for their knowledgeable feedback.
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaojiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/.
[2] Sebastian Ament and Michael O'Neil. Accurate and efficient numerical calculation of stable densities via optimized quadrature and asymptotics. Statistics and Computing, 28:171-185, 2018.
[3] Sebastian Ament, Maximilian Amsler, Duncan R. Sutherland, Ming-Chiang Chang, Dan Guevarra, Aine B. Connolly, John M. Gregoire, Michael O. Thompson, Carla P. Gomes, and R. Bruce van Dover. Autonomous materials synthesis via hierarchical active learning of nonequilibrium phase diagrams. Science Advances, 7(51):eabg4930, 2021. doi: 10.1126/sciadv.abg4930. URL https://www.science.org/doi/abs/10.1126/sciadv.abg4930.
[4] Sebastian Ament, Andrew Witte, Nishant Garg, and Julius Kusuma. Sustainable concrete via bayesian optimization, 2023. URL https://arxiv.org/abs/2310.18288. NeurIPS 2023 Workshop on Adaptive Experimentation in the Real World.
[5] Sebastian E Ament and Carla P Gomes. Scalable first-order Bayesian optimization via structured automatic differentiation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 500-516. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ament22a.html.
[6] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020.
[7] Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures, 2018.
[8] Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi-objective bayesian optimization with constraints, 2020.
[9] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
[10] Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5):1190-1208, 1995.
[11] Carlos A Coello Coello and Efrén Mezura Montes. Constraint-handling in genetic algorithms through the use of dominance-based tournament selection. Advanced Engineering Informatics, 16(3):193-203, 2002.
[74] Xingchen Wan, Vu Nguyen, Huong Ha, Binxin Ru, Cong Lu, and Michael A. Osborne. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 10663-10674. PMLR, 18-24 Jul 2021.
[75] Jialei Wang, Scott C. Clark, Eric Liu, and Peter I. Frazier. Parallel bayesian global optimization of expensive functions, 2016.
[76] Zi Wang and Stefanie Jegelka. Max-value Entropy Search for Efficient Bayesian Optimization. ArXiv e-prints, page arXiv:1703.01968, March 2017.
[77] James Wilson, Frank Hutter, and Marc Deisenroth. Maximizing acquisition functions for bayesian optimization. In Advances in Neural Information Processing Systems 31, pages 9905-9916. 2018.
[78] Jian Wu and Peter I. Frazier. The parallel knowledge gradient method for batch bayesian optimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 3134-3142. Curran Associates Inc., 2016.
[79] Jian Wu, Matthias Poloczek, Andrew Gordon Wilson, and Peter I Frazier. Bayesian optimization with gradients. In Advances in Neural Information Processing Systems, pages 5267-5278, 2017.
[80] Kaifeng Yang, Michael Emmerich, André Deutz, and Thomas Bäck. Multi-objective bayesian global optimization using expected hypervolume improvement gradient. Swarm and Evolutionary Computation, 44:945 - 956, 2019.
[81] Dawei Zhan and Huanlai Xing. Expected improvement for expensive optimization: a review. Journal of Global Optimization, 78(3):507-544, 2020.
[82] Jiayu Zhao, Renyu Yang, Shenghao Qiu, and Zheng Wang. Enhancing high-dimensional bayesian optimization by optimizing the acquisition function maximizer initialization. arXiv preprint arXiv:2302.08298, 2023.
[83] Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. *Evol. Comput.*, 8(2):173-195, jun 2000. ISSN 1063-6560. doi: 10.1162/106365600568202. URL https://doi.org/10.1162/106365600568202.
A Acquisition Function Details
A.1 Analytic Expected Improvement
Recall that the main challenge with computing analytic LogEI is to accurately compute  $\log h$ , where  $h(z) = \phi(z) + z\Phi(z)$ , with  $\phi(z) = \exp(-z^2/2)/\sqrt{2\pi}$  and  $\Phi(z) = \int_{-\infty}^{z} \phi(u)du$ . To express  $\log h$  in a numerically stable form as  $z$  becomes increasingly negative, we first take the log and multiply  $\phi$  out of the argument to the logarithm:
$$
\log h (z) = z ^ {2} / 2 - \log (2 \pi) / 2 + \log \left(1 + z \frac {\Phi (z)}{\phi (z)}\right). \tag {12}
$$
Fortunately, this form exposes the quadratic factor,  $\Phi(z) / \phi(z)$  can be computed via standard implementations of the scaled complementary error function  $\operatorname{erfcx}$ , and  $\log(1 + z\Phi(z) / \phi(z))$ , the last term of Eq. (12), can be computed stably with the  $\log 1\exp$  implementation proposed in [56]:
$$
\log 1 \mathrm {m e x p} (x) = \left\{ \begin{array}{l l} \log (- \exp \mathrm {m} 1 (x)) & - \log 2 <   x \\ \log 1 \mathrm {p} (- \exp (x)) & - \log 2 \geq x \end{array} , \right. \tag {13}
$$
where  $\exp 1, \log 1p$  are canonical stable implementations of  $\exp(x) - 1$  and  $\log(1 + x)$ , respectively. In particular, we compute
$$
\log_ {-} \mathrm {h} (z) = \left\{ \begin{array}{l l} \log (\phi (z) + z \Phi (z)) & z > - 1 \\ - z ^ {2} / 2 - c _ {1} + \log 1 \operatorname {m e x p} (\log (\operatorname {e r f c x} (- z / \sqrt {2}) | z |) + c _ {2}) & 1 / \sqrt {\epsilon} <   z \leq - 1 \\ - z ^ {2} / 2 - c _ {1} - 2 \log (| z |) & z \leq - 1 / \sqrt {\epsilon}. \end{array} \right. \tag {14}
$$
where  $c_{1} = \log (2\pi) / 2$ ,  $c_{2} = \log (\pi /2) / 2$ , and  $\epsilon$  is the floating point precision. We detail the reasoning behind the third asymptotic case of Equation (14) in Lemma 4 of Section A.2 below.
Numerical Study of Acquisition Values Figure 8 shows both the numerical failure mode of a naive implementation of EI, which becomes exactly zero numerically for moderately small  $z$ , while the evaluation via  $\log_{-h}$  in Eq. (14) exhibits quadratic asymptotic behavior that is particularly amenable to numerical optimization routines.
Figure 8: Plot of the  $\log h$ , computed via  $\log \circ h$  and  $\log_h$  in Eq. (14). Crucially, the naive implementation fails as  $z = (\mu(\mathbf{x}) - f^*) / \sigma(\mathbf{x})$  becomes increasingly negative, due to being exactly numerically zero, while our proposed implementation exhibits quadratic asymptotic behavior.
Equivalence of Optimizers Lemma 3 states that if the maximum of EI is greater than 0, LogEI and EI have the same set of maximizers. Furthermore, if  $\max_{\mathbf{x} \in \mathbb{X}} \operatorname{EI}(\mathbf{x}) = 0$ , then  $\mathbb{X} = \arg \max_{\mathbf{x} \in \mathbb{X}} \operatorname{EI}(\mathbf{x})$ . In this case, LogEI is undefined everywhere, so it has no maximizers, which we note would yield the same BO policy as EI, for which every point is a maximizer.