SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 21: Combining LogEI with TuRBO on the high-dimensional on the 50d Ackley problem yields significant improvement in objective value for a small number of function evaluations. Unlike for qEI, no random restarts are necessary to achieve good performance when performing joint optimization of the batch with qLogEI ( $q = 50$ ).
from 128, the BoTorch default, to 512, and use the fat-tailed smooth approximations of Sec. A.4 to ensure a strong gradient signal to all candidates of the batch.
Regarding the ultimate out-performance of TS over qLogEI, we think this is due to a model misspecification since the smoothness of a GP with the Matern-5/2 kernel cannot express the non-differentiability of Ackley at the optimum.
D.7 Constrained Problems
While running the benchmarks using CEI in section 5, we found that we in fact improved upon a best known result from the literature. We compare with the results in Coello and Montes [11], which are generated using 30 runs of 80,000 function evaluations each.
- For the pressure vessel design problem, Coello and Montes [11] quote a best-case feasible objective of 6059.946341. Out of just 16 different runs, LogEI achieves a worst-case feasible objective of 5659.1108 after only 110 evaluations, and a best case of 5651.8862, a notable reduction in objective value using almost three orders of magnitude fewer function evaluations
- For the welded beam problem, Coello and Montes [11] quote 1.728226, whereas LogEI found a best case of 1.7496 after 110 evaluations, which is lightly worse, but we stress that this is using three orders of magnitude fewer evaluations.
- For the tension-compression problem, LogEI found a feasible solution with value 0.0129 after 110 evaluations compared to the 0.012681 reported in in [11].
We emphasize that genetic algorithms and BO are generally concerned with distinct problem classes: BO focuses heavily on sample efficiency and the small-data regime, while genetic algorithms often utilize a substantially larger number of function evaluations. The results here show that in this case BO is competitive with and can even outperform a genetic algorithm, using only a tiny fraction of the sample budget, see App. D.7 for details. Sample efficiency is particularly relevant for physical simulators whose evaluation takes significant computational effort, often rendering several tens of thousands of evaluations infeasible.
D.8 Parallel Bayesian Optimization with cross-batch constraints
In some parallel Bayesian optimization settings, batch optimization is subject to non-trivial constraints across the batch elements. A natural example for this are budget constraints. For instance, in the context of experimental material science, consider the case where each manufactured compound requires a certain amount of different materials (as described by its parameters), but there is only a fixed total amount of material available (e.g., because the stock is limited due to cost and/or storage capacity). In such a situation, batch generation will be subject to a budget constraint that is not separable across the elements of the batch. Importantly, in that case sequential greedy batch generation is not an option since it is not able to incorporate the budget constraint. Therefore, joint batch optimization is required.
Here we give one such example in the context of Bayesian Optimization for sequential experimental design. We consider the five-dimensional silver nanoparticle flow synthesis problem from Liang et al. [53]. In this problem, to goal is to optimize the absorbance spectrum score of the synthesized nanoparticles over five parameters: four flow rate ratios of different components (silver, silver nitrate, trisodium citrate, polyvinyl alcohol) and a total flow rate  $Q_{tot}$ .
The original problem was optimized over a discrete set of parameterizations. For our purposes we created a continuous surrogate model based on the experimental dataset (available from https: //github.com/PV-Lab/Benchmarking) by fitting an RBF interpolator (smoothing factor of 0.01) in scipy on the (negative) loss. We use the same search space as Liang et al. [53], but in addition to the box bounds on the parameters we also impose an additional constraint on the total flow rate  $Q_{tot}^{max} = 2000 \mu \mathrm{L / min}$  across the batch:  $\sum_{i=1}^{q} Q_{tot}^{i} \leq Q_{tot}^{max}$  (the maximum flow rate per syringe / batch element is  $1000 \mu \mathrm{L / min}$ ). This constraint expresses the maximum throughput limit of the microfluidic experimentation setup. The result of this constraint is that we cannot consider the batch elements (in this case automated syringe pumps) have all elements of a batch of experiments operate in the high-flow regime at the same time.
In our experiment, we use a batch size of  $q = 3$  and start the optimization from 5 randomly sampled points from the domain. We run 75 replicates with random initial conditions (shared across the different methods), error bars show  $\pm$  two times the standard error of the mean. Our baseline is uniform random sampling from the domain (we use a hit-and-run sampler to sample uniformly from the constraint polytope  $\sum_{i=1}^{q} Q_{tot}^{i} \leq Q_{tot}^{max}$ ). We compare qEI vs. qLogEI, and for each of the two we evaluate (i) the version with the batch constraint imposed explicitly in the optimizer (the optimization in this case uses scipy's SLSQP solver), and (ii) a heuristic that first samples the total flow rates  $\{Q_{tot}^{i}\}_{i=1}^{q}$  uniformly from the constraint set, and then optimizes the acquisition function with the flow rates fixed to the sampled values.
The results in Figure 22 show that while both the heuristic ("random  $Q_{tot}$ ) and the proper constrained optimization ("batch-constrained  $Q_{tot}$ ) substantially outperform the purely random baseline, it requires using both LogEI and proper constraints to achieve additional performance gains over the other 3 combinations. Importantly, this approach is only possible by performing joint optimization of
Figure 22: Optimization results on the nanomaterial synthesis material science problem with cross-batch constraints. While qLogEI outperforms qEI under the proper constrained ("batch-constrained  $Q_{tot}$ ) optimization, this is not the case for the heuristic ("random  $Q_{tot}$ "), demonstrating the value of both joint batch optimization with constraints and LogEI.
the batch, which underlines the importance of qLogEI and its siblings being able to achieve superior joint batch optimization in settings like this.
D.9 Details on Multi-Objective Problems
We consider a variety of multi-objective benchmark problems. We evaluate performance on three synthetic biobjective problems Branin-Currin  $(d = 2)$  [8], ZDT1  $(d = 6)$  [83], and DTLZ2  $(d = 6)$  [17]. As described in 5, we also evaluated performance on three real world inspired problems. For the laser plasma acceleration problem, we used the public data available at Irshad et al. [39] to fit an independent GP surrogate model to each objective. We only queried te surrogate at the highest fidelity to create a single fidelity benchmark.
D.10 Effect of Temperature Parameter
In Figure 23, we examine the effect of fixed  $\tau$  for the softplus operator on optimization performance. We find that smaller values typically work better.
D.11 Effect of the initialization strategy
Packages and frameworks commonly utilize smart initialization heuristics to improve acquisition function optimization performance. In Figure 24, we compare simple random restart optimization, where initial points are selected uniformly at random, with BoTorch's default initialization strategy, which evaluates the acquisition function on a large number of points selected from a scrambled Sobol sequence, and selects  $n$  points at random via Boltzman sampling (e.g., sampling using probabilities computed by taking a softmax over the acquisition values [6]. Here we consider 1024 initial candidates. We find that the BoTorch initialization strategy improves regret for all cases, and that qLogEI, followed by UCB show less sensitivity to the choice of initializations strategy. Figure25 examines the sensitivity of qEI to the number of initial starting points when performing standard random restart optimization and jointly optimizing the  $q$  points in the batch. We find that, consistent with our empirical and theoretical results in the main text, qEI often gets stuck in local minima for the Ackley test function, and additional random restarts often improve results but do not compensate for the fundamental optimality gap. The performance of qLogEI also improves as the number of starting points increases.
Figure 23: Ablation study on the convergence characteristics of LogEI on Ackley and sum of squares (SOS) problems in 2 and 16 dimensions. The study shows that it is important to choose a small  $\tau_0$  for the best convergence properties, which results in a very tight approximation to the original ReLU non-linearity in the integrand. Critically, setting  $\tau_0$  as low as  $10^{-6}$  is only possible due to the transformation of all computations into log-space. Otherwise, the smoothed acquisition utility would exhibit similarly numerically vanishing gradients as the original ReLU non-linearity.
CELL NETWORK	BRANIN-CURRIN	DTLZ2	LASER PLASMA	ZDT1	VEHICLE SAFETY
JES	21.6 (+/- 1.1)	89.6 (+/- 3.3)	33.6 (+/- 1.0)	57.3 (+/- 0.7)	72.7 (+/- 1.0)	47.0 (+/- 1.6)
QEHVI	0.6 (+/- 0.0)	0.7 (+/- 0.0)	1.0 (+/- 0.0)	3.0 (+/- 0.1)	0.6 (+/- 0.0)	0.6 (+/- 0.0)
QLOGEHVI	9.2 (+/- 0.8)	10.0 (+/- 0.4)	5.8 (+/- 0.2)	31.6 (+/- 1.7)	7.2 (+/- 0.7)	2.1 (+/- 0.1)
RAND	0.2 (+/- 0.0)	0.2 (+/- 0.0)	0.2 (+/- 0.0)	0.3 (+/- 0.0)	0.3 (+/- 0.0)	0.3 (+/- 0.0)
Table 1: Multi-objective acquisition function optimization wall time in seconds on CPU (2x Intel Xeon E5-2680 v4 @ 2.40GHz). We report the mean and  $\pm 2$  standard errors.
Figure 24: Sensitivity to the initialization strategy. Random selects random restart points from the design space uniformly at random, whereas Boltzmann initialization is the default BoTorch initialization strategy which selects points with higher acquisition function values with a higher probability via Boltzmann sampling.
EI-Random Init. EI-Boltzmann Init.
LogEI - Random Init. LogEI - Boltzmann Init.
GIBBON - Random Init. GIBBON - Boltzmann Init.
JES - Random Init. JES - Boltzmann Init.
UCB - Random Init. UCB - Boltzmann Init.
Figure 25: Sensitivity to number of starting points with multi-start optimization for the 16D Ackley and Levy test problems. Note: We plot negative regret, so higher is better.