SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ackley53 is a 53-dimensional function with 50 binary and three continuous variables. Wan et al. [79] discretized 50 continuous variables of the original Ackley function, requiring these variables to be either zero or one. This benchmark was designed such that the optimal value of 0.0 is at the origin  $x = (0, \dots, 0)$ . Here, we perturb the optimal assignment of combinatorial variables by flipping each binary variable with probability  $1/2$ . Figure 8 summarizes the performances of the algorithms. Bounce outperforms all other algorithms and proves to be robust to the location of the optimum point. Casmopolitan is a distanced runner-up. BODi initially outperforms Casmopolitan on the published benchmark version but falls behind later.
B.1.2 Contamination control
The Contamination benchmark models a supply chain with 25 stages [39]. At each stage, a binary decision is made whether to quarantine food that has not yet been contaminated. Each such intervention is costly, and the goal is to minimize the number of contaminated products and prevention cost [6, 56]. Figure 9 shows the performances of the algorithms.
Bounce, Casmopolitan, and BODi all produce solutions of comparable objective value. Bounce and Casmopolitan find better solutions than BODi initially, but after about 100 function evaluations, the solutions obtained by the three algorithms are typically on par.
Figure 9: Bounce and the other algorithms on the 25-dimensional contamination problem. Bounce performs on par with Casmopolitan and BODi on both versions of the benchmark.
Figure 10: Bounce and other algorithms on the 60-dimensional weighted maximum satisfiability problem. Bounce is the first to find an optimal solution (left). On the published version (right), Bounce comes in second after BODi.
B.1.3 The MaxSAT60 benchmark
MaxSAT60 is a 60-dimensional, weighted instance of the Maximum Satisfiability (MaxSAT) problem. MaxSAT is a notoriously hard combinatorial problem that cannot be solved in polynomial time (unless  $\mathbb{P} = \mathbb{N}\mathbb{P}$ ). The goal is to find a binary assignment to the variables that satisfies clauses of maximum total weight. For every  $i$  in  $\{1,2,\dots,d\}$ , this benchmark has one clause of the form  $x_{i}$  with a weight of 1 and 638 clauses of the form  $\neg x_{i} \lor \neg x_{j}$  with a weight of 61. Following [21, 56, 79], we normalize these weights to have zero mean and unit standard deviation. This normalization causes the one-variable clauses to have a negative weight, i.e., the function value improves if such a clause is not satisfied, which is atypical behavior for a MaxSAT problem. Since the clauses with two variables are satisfied for  $x_{i} = x_{j} = 0$  and the clauses with one variable of negative weights are never satisfied for  $x_{i} = 0$ , the normalized benchmark version has a global optimum at  $x^{*} = (0,\dots,0)$  by construction. The problem's difficulty is finding an assignment for variables such that all two-variable clauses are satisfied and as many one-variable clauses as possible are not captured by normalized weights.
Figure 10 summarizes the performances of the algorithms. The general version that attains the global optimum for a randomly selected binary assignment is shown on the left. The special case where the global optimum is set to the all-zero assignment is shown on the right.
We observe that Bounce requires the smallest number of samples to find an optimal assignment in general, followed by BODi and Casmopolitan. Only in the special case where the optimum is the all-zero assignment, BODi ranks first, confirming the corresponding result in Deshwal et al. [21].
C An evaluation of Bounce on continuous problems
Figure 11: Bounce on continuous problems with different batch sizes (plotted in terms of batch evaluations).
Figure 12: Bounce on continuous problems with different batch sizes (plotted in terms of function evaluations).
To showcase the performance and scalability of Bounce, we run it on a set of continuous problems from Papenmeier et al. [60]. The 124-dimensional Mopta08 benchmark is a constrained vehicle optimization problem. We adopt the soft-constrained version from Eriksson and Jankowiak [23]. The 388-dimensional SVM problem [23] concerns the classification performance with an SVR on the slice localization dataset. The 180-dimensional LassoDNA benchmark [69] is a sparse regression problem on a real-world dataset, and the 1000-dimensional LassoHard benchmark optimizes over a synthetic dataset. The 500-dimensional Branin2 and Hartmann6 problems are versions of the 2- and 6-dimensional benchmark problems where additional dimensions with no effect on the function value were added.
We set the number of function evaluations to  $\max(2000, 500B)$  for a batch size of  $B$  and configure Bounce such that it reaches the input dimensionality after 500 function evaluations. Figures 11 and 12 show the simple regret for the synthetic Branin2 and Hartmann6 problems, and the best function value obtained after a given number of batch (Figure 11) or function (Figure 12) evaluations for the remaining problems: Mopta08, SVM, LassoDNA, and LassoHard.
We observe that Bounce always benefits from more parallel function evaluations. The difference between smaller batch sizes, such as  $B = 1$  and  $B = 3$  or  $B = 3$  and  $B = 10$ , is more remarkable than between larger batch sizes, like  $B = 10$  and  $B = 20$ . Parallel function evaluations prove
especially effective on SVM and LassoDNA. Here, the optimization performance improves drastically. We conclude that a small number of parallel function evaluations already helps to considerably increase the optimization performance.
On the synthetic Branin2 and Hartmann6 problems, Bounce quickly converges to the global optimum. Here, we see that a larger number of parallel function evaluations also helps in converging to a better solution.
In Figure 11, we also compare to BAxUS by Papenmeier et al. [60], which does not support parallel function evaluations and is therefore run with a batch size of 1. We observe that Bounce with a batch size of 1 outperforms BAxUS on all benchmarks except for Hartmann6, showcasing Bounce's overall performance improvements even for continuous problems and single-element batches.
D Batched evaluations on mixed and combinatorial problems
In addition to Figure 6, which shows the best objective value in relation to the number of batches, Figure 13 shows the objective value in terms of function evaluations.
Figure 13: Bounce on mixed and combinatorial problems with different batch sizes (plotted in terms of function evaluations).
The figure confirms that Bounce leverages parallel function values efficiently.
E Low-sequence version of Bounce
We show how we can bias Bounce towards low-sequence solutions. A binary vector has low sequence if there are few changes from 0 to 1 or vice versa. Similarly, a solution to a categorical or ordinal problem has low sequence if there are few changes from one category or level to another. We remove the random signs (for binary and continuous variables) and the random offsets (for categorical and ordinal variables) from the Bounce embedding. We conduct this study to show a) that Bounce can outperform BODi on the unmodified versions of the benchmark problems if we introduce a similar bias towards low-sequence solutions and b) that the random signs empirically show to remove biases towards low-sequence solutions. However, we want to emphasize that the results of this section are not representative of the performance of Bounce on arbitrary real-world problems. Nevertheless, if one knows that the problem has a low-sequence structure, then Bounce can be configured to exploit this structure and outperform BODi.
Figure 14 shows the results of the low-sequence version of Bounce on the original benchmarks from Section 4. We observe that Bounce outperforms BODi and the other algorithms on the unmodified versions of the benchmark problems. This shows that Bounce can outperform BODi on the unmodified version of the benchmarks if we introduce a similar bias towards low-sequence solutions.
Figure 15 shows the results of the low-sequence version of Bounce on the flipped benchmarks from Section 4. The low-sequence version of Bounce is robust towards the randomization of the optimal point.
Figure 14: 'Low-sequence' version of Bounce on the original benchmarks from Section 4: with a bias towards low-sequence solutions, Bounce outperforms BODi on the original versions of the benchmark problems.
Figure 15: "Low-sequence" version of Bounce on the modified benchmarks from Section 4.
F Effect of trust-region management
Bounce uses a novel trust-region management that differs from previous approaches in that it allows arbitrary trust region base lengths in  $[L_{\min}, L_{\max}]$ . This strategy allows Bounce to efficiently leverage parallel function evaluations, which we refer to as batch acquisition.
We compare Bounce's trust-region management strategy with the strategy employed by BAxUS [60] for purely continuous benchmarks and an adapted version of Casmopolitan's [79] strategy for mixed or discrete-space benchmarks. For the latter, the difference is that in Bounce we reduce the failure tolerance as described by Papenmeier et al. [60]: We first calculate the number of times the TR base length needs to be reduced to reach the minimum TR base length as  $k = \left\lfloor \log_{1.5^{-1}}\frac{L_{\mathrm{min}}}{L_{\mathrm{init}}} \right\rfloor$ , and then find the failure tolerance for the  $i$ -th target space by  $\tau_{\mathrm{fail}}^i = \max \left(1,\min \left(\left\lfloor \frac{m_i^s}{k}\right\rfloor\right)\right)$ , where  $m_i^s$  is the budget for the  $i$ -th target space [60]. This change in regard to Casmopolitan is necessary because Casmopolitan's failure tolerance of 40 [79] would cause the algorithm to spend a large part of the evaluation budget in initial target spaces of low dimensionality.
We show the effect of this strategy on discrete and mixed-space benchmarks in Figure 16. Figure 17 shows the effect on continuous benchmarks. All experiments are replicated 50 times. We observe that the proposed trust-region management method not only enables efficient batch parallelism but also improves the performance for single-function evaluations when compared to the respective baseline for the TR management that we stated above. The TR management proposed here usually
Figure 16: Effect of the proposed trust region management on the discrete and mixed benchmarks. Here, the baseline is the TR management of Casmopolitan. We see that often, even large batch sizes for the old TR management strategy do not outperform the proposed new strategy with a batch size of 1.
Figure 17: Effect of the proposed trust-region management on the continuous benchmarks. Here, the baseline is the TR management of BAxUS. We observe that larger batches help to improve performance for both methods. The new strategy, however, outperforms the baseline in almost all cases with the exception of LassoDNA with a batch size of 3.
provides better solutions at the same number of batches than the respective baseline TR management. There are a few exceptions. On Labs, the previous TR management strategy outperforms Bounce's strategy by a small margin for batch sizes 5, 10, and 20, and on Ackley53, the previous strategy converges to a better solution for a batch size of three, five, and ten.
On the continuous benchmarks (Figure 17), we observe that the new TR management strategy also improves the performance for single function evaluations with the exception of Branin2. Similar to the mixed and combinatorial-space benchmarks, large batch sizes (10 and 20) bring little advantage compared to a batch size of 5. For Hartmann6, the previous strategy with a batch size of 20 performs worse than 3.
We implement Bounce in Python using the BoTorch [5] and GPyTorch [29] libraries.
We employ a  $\Gamma(1.5,0.1)$  prior on the lengthscales of both kernels and a  $\Gamma(1.5,0.5)$  prior on the signal variance. We further use a  $\Gamma(1.1,0.1)$  prior on the noise variance.
Motivated by Wan et al. [79] and Eriksson et al. [25], we use an initial trust region baselength of 40 for the combinatorial variables, and 0.8 for the continuous variables. We maintain two separate TR shrinkage and expansion parameters  $(\gamma_{\mathrm{cmb}}$  and  $\gamma_{\mathrm{cnt}})$  for the combinatorial and continuous variables, respectively such that each TR base length reaches its respective minimum of 1 and  $2^{-7}$  after a given number of function evaluations. When Bounce finds a better or worse solution, we increase or decrease both TR base lengths.
We use the author's implementations for COMBO $^2$ , BODi $^3$ , RDUCB $^4$ , SMAC $^5$ , and Casmopolitan $^6$ . We use the same settings as the authors for COMBO, RDUCB, SMAC, and BODi. For Casmopolitan, we use the same settings as the authors for benchmarks reported in Wan et al. [79] and set the initial trust region base length to 40 otherwise.
Due to its high-memory footprint, we ran B0Di on NVidia A100 80GB GPUs for 300 GPU/h. We ran Bounce on NVidia A40 GPUs for 2,000 GPU/h. We ran the remaining methods for 20,000 GPU/h on one core of Intel Xeon Gold 6130 CPUs with 60GB of memory.
G.1 Optimization of the acquisition function
We use different strategies to optimize the acquisition function depending on the type of variables present in a problem.
Continuous problems. For purely continuous problems, we follow a similar approach as TuRBO [25]. In particular, we use the lengthscales of the GP posterior to shape the TR. We use gradient descent to optimize the acquisition function within the TR bounds with 10 random restarts and 512 raw samples. For a batch size of 1, we use analytical EI. For larger batch sizes, we use the BoTorch implementation of qEI [5, 66, 86].
Binary problems. Similar to Wan et al. [79], we use discrete TRs centered on the current best solution. A discrete TR describes all solutions with a certain Hamming distance to the current best solution.
We use a local search approach to optimize the acquisition function for all problems with a combinatorial search space of only binary variables: When starting the optimization, we first create a set of  $\min(5000, \max(2000, 200 \cdot d_i))$  random solutions. The choice of the number of random solutions is based on TuRBO [25]. For each candidate, we first draw  $L_i$  indices uniformly at random from  $\{1, \dots, d_i\}$  without replacement, where  $L_i$  is the TR length at the  $i$ -th iteration. We then sample  $d_i$  values in  $\{0, 1\}$  and set the candidate at the sampled indices to the sampled values. All other values are set to the values of the current best solution. Note that this construction ensures that each candidate solution lies in the TR bounds of the current best solution. We add all neighbors (i.e., points with a Hamming distance of 1) of the current best solution to the set of candidates. This is inspired by BODi [21]. We find the 20 candidates with the highest acquisition function value and use local search to optimize the acquisition function within the TR bounds: At each local search step, we create all direct neighbors that do not coincide with the current best solution or would violate the TR bounds. We then move the current best solution to the neighbor with the highest acquisition function value. We repeat this process until the acquisition function value does not increase anymore. Finally, we return the best solution found during local search.
Categorical problems. We adopt the approach for binary problems, i.e., we first create a set of random solutions with the same size as for purely binary problems and start the local search on the 20 best initial candidates.
Suppose the number of categorical variables of the problem is smaller or equal to the current TR length. In that case, we sample, for each candidate and each categorical variable, an index uniformly at random from  $\{1,2,\dots ,|v_i|\}$  where  $|v_{i}|$  is the number of values of the  $i$ -th categorical variable. We then set the candidate at the sampled index to 1 and all other values to 0.
$^{2}$ https://github.com/QUVA-Lab/combo, unspecified license, last access: 2023-05-04
<sup>3</sup>https://github.com/aryandeshwal/bodi, no license provided, last access: 2023-05-04
4https://github.com/huawei-noah/HEBO/tree/master/RDUCB, MIT license, last access: 2023-10-20
<sup>5</sup>https://github.com/autom1/pysmac, AGPL-3.0 license, last access 2023-10-20
<sup>6</sup>https://github.com/xingchenwan/casmo, MIT license, last access: 2023-05-04
If the number of categorical variables of the problem is larger than the current TR length  $L_{i}$ , we first sample  $L_{i}$  categorical variables uniformly at random from  $[d_i]$  without replacement. For each initial candidate and each sampled categorical variable, we sample an index uniformly at random, for which we set the categorical variable to 1 and all other values to 0. The values for the variables that were not sampled are set to the values of the current best solution.
As for the binary case, we add all neighbors of the current best solution to the set of candidates, and we sample the 20 candidates with the highest acquisition function value.
We then use multi-start local search to optimize the acquisition function within the TR bounds while neighbors are created by changing the index of one categorical variable. Again, we repeat until convergence and return the best solution found during the local search.
Ordinal problems. The construction for ordinal problems is similar to the one for categorical problems.
Suppose the number of ordinal variables of the problem is smaller or equal to the current TR length. In that case, we sample an ordinal value uniformly at random to set the ordinal variable for each candidate and each ordinal variable. Otherwise, we choose as many ordinal variables as each candidate's current TR length and sample an ordinal value uniformly at random to set the ordinal variable. We add all neighbors of the current best solution, all solutions where the distance to the current best solution is 1 for one ordinal variable, to the set of candidates. We then sample the 20 candidates with the highest acquisition function value and use local search to optimize the acquisition function within the TR bounds. In the local search, we increment or decrement the value of a single ordinal variable.
Mixed problems. Mixed problems are effectively handled by treating every variable type separately. Again, we create a set of initial random solutions where the values for the different variable types are sampled according to the abovementioned approaches. This can lead to solutions outside the TR bounds. We remove these solutions and find the 20 best candidates only across the solutions within the TR bounds.
When optimizing the acquisition function, we differentiate between continuous and combinatorial variables. We optimize the continuous variables by gradient descent with the same settings as purely continuous problems. When optimizing, we fix the values for the combinatorial values.
We use local search to optimize the acquisition function for the combinatorial variables. In this step, we fix the values for the continuous variables and only optimize the combinatorial variables. We create the neighbors by creating neighbors within Hamming distance of 1 for each combinatorial variable type and then combining these neighbors. Again, we run a local search until convergence.
We do five interleaved steps, starting with the continuous variables and ending with the combinatorial variables.
G.2 Kernel choice
We use the CoCaBo kernel [67] with one global lengthscale for the combinatorial and ARD for the continuous variables:
$$
\begin{array}{l} k (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \sigma_ {f} ^ {2} \left(\rho k _ {\mathrm {c m b}} \left(\boldsymbol {x} _ {\mathrm {c m b}}, \boldsymbol {x} _ {\mathrm {c m b}} ^ {\prime}\right) k _ {\mathrm {c n t}} \left(\boldsymbol {x} _ {\mathrm {c n t}}, \boldsymbol {x} _ {\mathrm {c n t}} ^ {\prime}\right)\right) \\ + (1 - \rho) \left(k _ {\mathrm {c m b}} \left(\boldsymbol {x} _ {\mathrm {c m b}}, \boldsymbol {x} _ {\mathrm {c m b}} ^ {\prime}\right) + k _ {\mathrm {c n t}} \left(\boldsymbol {x} _ {\mathrm {c n t}}, \boldsymbol {x} _ {\mathrm {c n t}} ^ {\prime}\right)\right)) \\ \end{array}
$$
$$
\begin{array}{l} k _ {\mathrm {c m b}} \left(\boldsymbol {x} _ {\mathrm {c m b}}, \boldsymbol {x} _ {\mathrm {c m b}} ^ {\prime}\right) = \left(1 + \frac {\sqrt {5} r _ {\mathrm {c m b}}}{\ell_ {\mathrm {c m b}}} + \frac {5 r _ {\mathrm {c m b}} ^ {2}}{3 \ell_ {\mathrm {c m b}} ^ {2}}\right) \exp \left(- \frac {\sqrt {5} r _ {\mathrm {c m b}}}{\ell_ {\mathrm {c m b}}}\right) \\ r _ {\mathrm {c m b}} = \left\| \boldsymbol {x} _ {\mathrm {c m b}} - \boldsymbol {x} _ {\mathrm {c m b}} ^ {\prime} \right\| \\ k _ {\text {c n t}} \left(\boldsymbol {x} _ {\text {c n t}}, \boldsymbol {x} _ {\text {c n t}} ^ {\prime}\right) = \left(1 + \sqrt {5} r _ {\text {c n t}} + 5 r _ {\text {c n t}} ^ {2}\right) \exp \left(- \sqrt {5} r _ {\text {c n t}}\right) \\ r_{\text{cnt}} = \sqrt{\sum_{i\in[d];d_{i}\text{continuous}}\frac{(x_{i} - x_{i}^{\prime})^{2}}{\ell_{\text{cnt},i}^{2}}} \\ \end{array}
$$
where  $\pmb{x}_{\mathrm{cnt}}$  and  $\pmb{x}_{\mathrm{cmb}}$  are the continuous and combinatorial variables in  $\pmb{x}$ , respectively, i.e.,
$$
\boldsymbol {x} _ {\mathrm {c m b}} = \left(x _ {i}: d _ {i} \text {i s c o m b i n a t o r i a l}\right)
$$
$$
\boldsymbol {x} _ {\text {c n t}} = \left(x _ {i}: d _ {i} \text {i s c o n t i n u o u s}\right)
$$
and  $\rho \in [0,1]$  is a tradeoff parameter learned jointly with the other hyperparameters during the likelihood maximization. Here,  $\ell_{\mathrm{cmb}}$  and  $\ell_{\mathrm{cnt},i}$  are the lengthscale hyperparameters.
H Additional analysis of BODi and COMBO
H.1 Analysis of BODi
Binary problems. BODi by Deshwal et al. [21] uses a dictionary of anchor points  $\mathbf{A} = (a_{1},\ldots ,a_{m})$  to encode a candidate point  $z$ . In particular, the  $i$ -th entry of the  $m$ -dimensional embedding  $\phi_A(z)$  is obtained by computing the Hamming-distance between  $z$  and  $a_{i}$ . Notably, Deshwal et al. [21] choose the dimensionality of the embedding  $m$  as 128, which is larger than the dimensionality of the objective functions themselves.
The sampling procedure for dictionary elements  $\pmb{a}_i$  is chosen to yield a wide range of sequences. The sequence of a binary string is defined as the number of times the string changes from 0 to 1 and vice versa. Deshwal et al. [21] propose two approaches to generate the dictionary elements: (i) using binary wavelets, and (ii) by first drawing a Bernoulli parameter  $\theta_i \sim \mathcal{U}(0,1)$  for each  $i \in [m]$  and then drawing a binary string  $\pmb{a}_i$  from the distribution  $\mathcal{B}(\theta_i)$ . The latter approach is their default method.
We prove that BODi generates an all-zeros (or, by a similar symmetry argument, all-ones) representative point with a probability that is significantly higher than  $2^{-D}$ . Moreover, we claim without proof that a similarly increased probability holds for points with low Hamming distance to all-zeros or all-ones. This is consistent with the intention of BODi to sample points of diverse sequence and not necessarily an issue.
However, we claim without proof that having such points in the dictionary substantially increases the probability that BODi evaluates the all-zeros (and, by symmetry, the all-ones) points. That hypothesis is consistent with our observation in Section 4.6 that BODi has a much higher chance of finding good or optimal solutions when they are near the all-zero point.
Deshwal et al. [21] choose the dictionary to have  $m = 128$  dictionary elements. Given a Bernoulli parameter  $\theta_{i}$ , the probability that the  $i$ -th dictionary point  $\pmb{a}_{i}$  is a point of sequence zero is given by  $\theta_{i}^{D} + (1 - \theta_{i})^{D}$ :
$$
\mathbb {P} \left(" z e r o s e q u e n c y" \mid \theta_ {i}\right) = \prod_ {i = 1} ^ {m} \theta_ {i} ^ {D} + \left(1 - \theta_ {i}\right) ^ {D}
$$