SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: The mapping (or binning) of categorical and ordinal variables. Suppose that variable  $v_{k}$  has two categories and that  $v_{\ell}$  has three categories. Both are mapped to the target dimension  $d_{i}$  that has cardinality  $3 = \max \{2,3\}$ . While the mapping of  $v_{\ell}$  to  $d_{i}$  is a straightforward bijection,  $v_{k}$  has fewer categories than  $d_{i}$ . Thus, the mapping of  $v_{k}$  to  $d_{i}$  repeats label 1. Ordinal variables are mapped similarly.
Binary variables. Binary dimensions are represented by the values  $-1$  and  $+1$ . Each input dimension  $D_{i}$  is associated with a random sign  $s_i \in \{-1, +1\}$ , and the subspace embedding may map one or more binary input dimensions to the same binary target dimension. While the embedding for binary and continuous dimensions is similar, Bounce handles binary dimensions differently when optimizing the acquisition function.
Categorical and ordinal variables. Categorical variables that differ in the number of categories may be mapped to the same target dimension (bin). Suppose that the categorical variables  $v_{1}, \ldots, v_{\ell}$  with cardinalities  $c_{1}, \ldots, c_{\ell}$  are mapped to a single bin that is associated with the target dimension  $d_{j}$  of the subspace embedding. Then  $d_{j}$  is of categorical type and has  $\max \{c_{i} \mid 1 \leq i \leq \ell\} =: c_{\max}$  distinct categories, that is, its cardinality is given by the maximum cardinality of any variable mapped to it. Thus, the bin  $d_{j}$  can represent every category of these input variables.
Suppose that Bounce assigns the category  $k \in \{1, \dots, c_{\max}\}$  to the categorical bin (target dimension)  $d_j$ . We transform this label to a categorical assignment to each input variable  $v_1, \dots, v_\ell$ , setting  $v_i = \lceil k \cdot (c_i / c_{\max}) \rceil$ . Recall that Bounce may split up bins, i.e., target dimensions, to increase the cardinality of its subspace embedding. In such an event, every derived bin inherits the cardinality of the parent bin. This allows us to retain any observations the algorithm has taken up to this point. Analogously to the random sign for binary variables, we randomly shuffle the categories before the embedding. This reduces the risk of Bounce being biased towards a specific structure of the optimizer (see Appendix E).
We treat ordinal variables as categorical variables whose categories correspond to the discrete values the ordinal variable can take. For the sake of simplicity, we suppose here that an ordinal variable  $v_{i}$  has range  $\{1,2,\dots ,c_i\}$  and  $c_{i}\geq 2$  for all  $i\in \{1,\ldots ,\ell \}$ . Figure 1 shows examples of the binning of categorical and ordinal variables.
3.2 Maximization of the acquisition function
We use expected improvement (EI) [43] for batches of size  $B = 1$  and qEI [5, 80, 86] for larger batches. We optimize the EI using gradient-based methods for continuous problems and local search for combinatorial problems. We interleave gradient-based optimization and local search for functions defined over a mixed space; see Appendix G.1 for details.
4 Experimental evaluation
We evaluate Bounce empirically on various benchmarks whose inputs are combinatorial, continuous, or mixed spaces. The evaluation comprises the state-of-the-art algorithms BODi [21], Casmopolitan [79], COMBO [56], SMAC [41], and RDUCB [88], using code provided by the authors. We also report Random Search [8] as a baseline. For categorical problems, COMBO's implementation suffers from a bug explained in Appendix H.2. We report the results for COMBO with the correct benchmark implementation as "COMBO (fixed)".
The experimental setup. We initialize every algorithm with five initial points. The plots show the performances of all algorithms averaged over 50 repetitions except BODi, which has 20 repetitions due to resource constraints caused by its high memory demand. The shaded regions give the standard error of the mean. We use common random seeds for all algorithms and for randomizing the
Figure 2: The 50D low-autocorrelation binary sequence problem. Bounce finds the best solutions, followed by COMBO.
Figure 3: The 125D weighted ClusterExpansion maximum satisfiability problem. We plot the total negative weight of clauses. Bounce produces the best assignments.
benchmark functions. We run all methods for 200 function evaluations unless stated otherwise. The Labs (Section 4.1) and MaxSat125 (Section 4.2) benchmarks are run for 500 evaluations.
The benchmarks. The evaluation uses seven established benchmarks [21]: 53D SVM, 50D LABS, 125D ClusterExpansion [3, 4], 60D MaxSAT60 [21, 56], 25D PestControl, 53D Ackley53, and 25D Contamination [6, 39, 56]. Due to space constraints, we moved the results for the MaxSAT60, Contamination, and Ackley53 benchmarks to Appendix B.1. For each benchmark, we report results for the originally published formulation and for a modification where we move the optimal point to a random location. The randomization procedure is fixed for each benchmark for all algorithms and repetitions. For binary problems, we flip each input variable independently with probability 0.5. For categorical problems, we randomly permute the order of the categories. We motivate this randomization in Section 4.6.
4.1 50D Low-Autocorrelation Binary Sequences (LABS)
LABS has  $n = 50$  binary dimensions. It has important applications in communications engineering and mathematics; see [58] for details. LABS is a hard combinatorial problem and currently solved via exhaustive search. The goal is to find a sequence  $\pmb{x} \in \{-1, +1\}^n$  with a maximum merit factor  $F(\pmb{x}) = \frac{n^2}{2E(\pmb{x})}$ , where  $E(\pmb{x}) = \sum_{k=1}^{n-1} C_k^2(\pmb{x})$  and  $C_k(\pmb{x}) = \sum_{i=1}^{n-k} x_i x_{i+k}$  for  $k = 0, \dots, n-1$  are the autocorrelations of  $\pmb{x}$  [58]. Figure 2 summarizes the performances. We observe that Bounce outperforms all other algorithms on the benchmark's original and randomized versions.
4.2 Industrial Maximum Satisfiability: 125D ClusterExpansion benchmark
MaxSat is a notoriously hard problem for which various approximations and exact (exponential time) algorithms have been developed; see [32, 61] for an overview. We evaluate Bounce and the other algorithms on the 125-dimensional ClusterExpansion benchmark, a real-world MaxSAT instance with many applications in materials science [2]. Unlike the MaxSAT60 benchmark (see Appendix B.1.3), ClusterExpansion is not a crafted benchmark, and its optimum has no synthetic structure [1, 3]. We treat the MaxSat problems as black-box problems; hence, algorithms do not have access to the clauses, and we cannot use the usual algorithms.
Figure 3 shows the total negative weight of the satisfied clauses as a function of evaluations. We cannot plot regret curves since the optimum is unknown [4]. We observe that Bounce finds better solutions than all other algorithms. BODi is the only algorithm for which we observe sensitivity to
Figure 4: The 25D categorical pest control problem. Bounce obtains the best solutions, followed by Casmopolitan. BODi's performance degrades significantly when shuffling the order of categories.
Figure 5: The 53-dimensional SVM benchmark. Bounce, BODi, and Casmopolitan achieve comparable solutions.
the location of the optimal assignment: for the published version of the benchmark, B0Di quickly jumps to a moderately good solution but fails to make further progress.
4.3 25D Categorical Pest Control
PestControl is a more complex version of the Contamination benchmark and has 25 categorical variables with five categories each [56]. The task is to select one out of five actions  $\{1,2,\ldots,5\}$  at each of 25 stations to minimize the objective function that combines total cost and a measure of the spread of the pest. We note the setting  $x = (5,5,\dots,5)$  achieves a good value of 12.57, while the best value found in our evaluation is 12.07 is  $x = (5,5,\dots,5,1)$  and thus has a Hamming distance of one. The random seed used in our experiments is zero. Figure 4 summarizes the performances of the algorithms. Bounce is robust to the location of the global optimum and consistently obtains the best solutions. In particular, the performances of COMBO and BODi depend on whether the optimum has a certain structure. We discuss this issue in detail in Appendix H.1.
4.4 SVM - a 53D AutoML task
In the SVM benchmark, we optimize over a mixed space with 50 binary and 3 continuous parameters to tune an  $\varepsilon$ -support vector regression (SVR) model [72]. The 50 binary parameters determine whether to include or exclude an input feature from the dataset. The 3 continuous parameters correspond to the regularization parameter  $C$ , the kernel width  $\gamma$ , and the  $\varepsilon$  parameter of the  $\varepsilon$ -SVR model [72]. Its root mean squared error on a held-out dataset gives the function value. Figure 5 summarizes the performances of the algorithms. We observe that Bounce, BODi, and Casmopolitan achieve comparable solutions. BODi performs slightly worse if the ordering of the categories is shuffled and slightly better if the optimal assignment to all binary variables is one. COMBO does not support continuous variables and thus was omitted.
4.5 Bounce's efficacy for batch acquisition
We study the sample efficiency of Bounce when it selects a batch of  $B$  points in each iteration to evaluate in parallel. Figure 6 shows the results for  $B = 1,3,5,10$ , and 20, where Bounce was run for  $\min(2000,200 \cdot B)$  function evaluations. We configure Bounce to reach the input dimensionality after 100 evaluations for  $B = 1,3,5$  and after  $25B$  for  $B = 10,20$ . We observe that Bounce leverages parallel function evaluations effectively: it obtains a comparable function value at a considerably smaller number of iterations, thus saving wall-clock time for applications with time-consuming function evaluations. We also studied batch acquisition for continuous problems
Figure 6: Bounce benefits from the batch acquisition that allows parallelizing function evaluations. We show the best function values obtained after each batch for batch sizes 1, 3, 5, 10, and 20.
and found that Bounce also provides significant speed-up. Due to space constraints, we deferred the discussion to Appendix C.
4.6 The sensitivity of BODi and COMBO to the location of the optima
The empirical evaluation reveals that the performances of BODi [21] and COMBO [56] are sensitive to the location of the optima. Both methods degrade on at least one benchmark when the optimum is moved to a randomly chosen point. This is particularly unexpected for categorical variables where moving the optimum to a random location is equivalent to shuffling the labels of the categories of each variable. Such a change of representation should not affect the performance of an algorithm.
BODi is more susceptible to the location of the optimizer than COMBO. The performance of COMBO degrades only on the categorical PestControl benchmark, whereas BODi degrades on five out of seven benchmarks. Looking closer, we observe that BODi's performance degradation is particularly large for synthetic benchmarks like Ackley53 and MaxSAT60, where setting all variables to the same value is optimal. Figure 7 summarizes the effects of moving the optimum on BODi. Due to space constraints, we moved the details and a discussion of categorical variables to the appendix. Similarly, setting all binary variables of the SVM benchmark to one produces a good objective value. It is not surprising, given that the all-one assignment corresponds to including all features previously selected for the benchmark because of their high importance.
We show in Appendix H.1 that BODi adds a point to its dictionary with zero Hamming distance to an all-zero or all-one solution, with a probability that increases with the dictionary size. Deshwal et al. [21, p. 7] reported that BODi's performance 'tends to improve' with the size of the dictionary. Moreover, BODi samples a new dictionary in each iteration, eventually increasing the chance of having such a point in its dictionary. Thus, we hypothesize that BODi benefits from having a near-optimal solution in its dictionary, likely for all-zero or all-one solutions. For COMBO, Figure 4 shows that the performance on PestControl degrades substantially if the labels of the categories are shuffled. Then COMBO's sample-efficiency becomes comparable to Random Search.
BO in combinatorial spaces has many exciting and impactful applications. Its applicability to real-world problems, such as LABS that defy a closed-form solution, makes it a valuable tool for practitioners. Our empirical evaluation reveals that state-of-the-art methods fail to provide good solutions reliably. In particular, it finds that BODi and COMBO, which performed best in recent publications, are sensitive to the location of the optimizer. We identified design flaws in BODi and an implementation bug in COMBO as the root causes of the performance degradations.
The proposed Bounce algorithm is reliable for high-dimensional black-box optimization in combinatorial, continuous, and mixed spaces. The empirical evaluation demonstrates that Bounce reliably outperforms the state-of-the-art on a diverse set of problems. Using a novel TR management strategy, Bounce leverages parallel evaluations of the objective function to improve its performance. We
Figure 7: BODi's performance degrades on five out of seven benchmarks when randomizing the location of the optimal solution: BODi on the default version (orange) of the benchmark and on the modified version (green, dashed) where the optimum was moved.
anticipate headroom by tailoring the modeling of combinatorial objects, e.g., arising in the search for peptides or materials discovery [28, 34, 37, 59, 75, 78]. Here it seems particularly interesting to incorporate prior belief on the importance of decision variables while maintaining the overall scalability. Moreover, extending the present work to black-box constraints [24, 30], multiple objectives, and multiple information sources [18, 31, 62] will considerably expand the applicable use cases.
Limitations. Bounce is not designed to handle noisy evaluations of the objective function. While it seems straightforward to extend Bounce to handle noisy evaluations, e.g., by using a Gaussian process with a noise term and acquisition functions that account for noise [70], we leave this for future work. Moreover, in applications where the categorical or ordinal variables vary substantially in the number of values they can take, there may be better ways to 'bin' them.
Societal impact. Bayesian optimization has recently gained wide-spread popularity for tasks in drug discovery [53], chemical engineering [15, 33, 38, 68, 71], materials discovery [28, 34, 37, 59, 75, 78], aerospace engineering [6, 45, 49], robotics [16, 17, 48, 50, 64], and many more. This highlights the Bayesian optimization community's progress toward providing a reliable 'off-the-shelf optimizer.' However, this promise is not yet fulfilled for the newer domain of mixed-variable Bayesian optimization that allows optimization over hundreds of 'tunable levers', some of which are discrete, while others are continuous. This domain is of particular relevance for the tasks above. Bounce's ability to incorporate more such levers in the optimization significantly impacts the above practical applications, allowing for more granular control of a chemical reaction or a processing path, to give some examples. The empirical evaluation shows that the performance of state-of-the-art methods is highly sensitive to the location of the unknown global optima and often degenerates drastically, thus putting practitioners at risk. The proposed algorithm Bounce, however, achieves robust performance over a broad collection of tasks and thus will become a 'goto' optimizer for practitioners in other fields. Therefore, we open-source the Bounce code.<sup>1</sup>
Leonard Papenmeier and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348 and by affiliate members and other supporters of the Stanford DAWN project Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at the Chalmers Centre for Computational Science and Engineering (C3SE) and the National Supercomputer Centre at Linköping University, partially funded by the Swedish Research Council through grant agreement no. 2022-06725.
<https://github.com/LeoIV/bounce>
[82] Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search. Advances in Neural Information Processing Systems (NeurIPS), 33:19511-19522, 2020.
[83] Xilu Wang, Yaochu Jin, Sebastian Schmitt, and Markus Olhofer. Recent Advances in Bayesian Optimization. ACM Comput. Surv., 2023.
[84] Ziyu Wang, Frank Hutter, Masour Zoghi, David Matheson, and Nando de Feitas. Bayesian Optimization in a Billion Dimensions via Random Embeddings. Journal of Artificial Intelligence Research (JAIR), 55:361-387, 2016.
[85] Christopher K Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
[86] James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparameterization trick for acquisition functions. NeurIPS Workshop on Bayesian Optimization, 2017.
[87] David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science, 10(1-2):1-157, 2014.
[88] Juliusz Krzysztof Ziomek and Haitham Bou Ammar. Are Random Decompositions all we need in High Dimensional Bayesian Optimisation? In International Conference on Machine Learning, pages 43347-43368. PMLR, 2023.
A Consistency of Bounce
In this section, we prove the consistency of the Bounce algorithm. The proof is based on Papenmeier et al. [60] and Eriksson and Poloczek [24].
Theorem 1 (Bounce consistency). With the following definitions
Def. 1.  $(\pmb{x}_k)_{k=1}^{\infty}$  is a sequence of points of decreasing function values;
Def. 2.  $\pmb{x}^{*} \in \arg \min_{\pmb{x} \in \mathcal{X}}$  is a minimizer of  $f$  in  $\mathcal{X}$ ;
and under the following assumptions:
Ass. 1.  $D$  is finite;
Ass. 2.  $f$  is observed without noise;
Ass. 3. The range of  $f$  is bounded in  $\mathcal{X}$ , i.e.,  $\exists C \in \mathbb{R}_{++}$  s.t.  $|f(\pmb{x})| < C \forall \pmb{x} \in \mathcal{X}$ ;
Ass. 4. For at least one of the minimizers  $\pmb{x}_i^*$  the (partial) assignment corresponding to the continuous variables lies in a (continuous) region with positive measure;
Ass. 5. One Bounce reached the input dimensionality  $D$ , the continuous elements of the initial points  $\{\pmb{x}_{cont_i}\}_{n=1}^{n_{init}}$  after each TR restart are chosen
(a) uniformly at random for continuous variables; and
(b) such that every realization of the combinatorial variables has positive probability;
then the Bounce algorithm finds a global optimum with probability 1, as the number of samples  $N$  goes to  $\infty$ .
Proof. The range of  $f$  is bounded per Assumption 3, and Bounce only considers a function evaluation a 'success' if the improvement over the current best solution exceeds a certain constant threshold. Bounce can only have a finite number of 'successful' evaluations because the range of  $f$  is bounded per Assumption 3. For the sake of a contradiction, we suppose that Bounce does not obtain an optimal solution as its number of function evaluations  $N \to \infty$ . Thus, there must be a sequence of failures, such that the TRs in the current target space, i.e., the current subspace, will eventually reach its minimum base length. Recall that in such an event, Bounce increases the target dimension by splitting up the 'bins', thus creating a subspace of  $(b + 1)$ -times higher dimensionality. Then Bounce creates a new TR that again experiences a sequence of failures that lead to another split, and so on. This series of events repeats until the embedded subspace eventually equals the input space and thus has dimensionality  $D$ . See lines 12 - 16 in Algorithm 1 in Sect. 3.
Still supposing that Bounce does not find an optimum in the input space, there must be a sequence of failures such that the side length of the TR again falls below the set minimum base length, now forcing a restart of Bounce. Recall that at every restart, Bounce samples a fresh set of initial points uniformly at random from the input space; see line 18 in Algorithm 1. Therefore, with probability 1, a random sample will eventually be drawn from any subset  $\mathcal{V} \subseteq \mathcal{X}$  with positive Lebesgue measure  $(\nu(\mathcal{V}) > 0)$ :
$$
1 - \lim  _ {k \rightarrow \infty} (1 - \mu (\mathcal {Y})) ^ {k} = 1, \tag {1}
$$
where  $\mu$  is the uniform probability measure of the sampling distribution that Bounce employs for initial data points upon restart [73].
$$
\alpha = \inf  \left\{t: \nu \left[ x \in \mathcal {X} \mid f (x) <   t \right] > 0 \right\}
$$
denote the essential infimum of  $f$  on  $\mathcal{X}$  with  $\nu$  being the Lebesgue measure [73].
Following Solis and Wets [73], we define the optimality region, i.e., the set of points whose function value is larger by at most  $\varepsilon$  than the essential infimum:
$$
R _ {\varepsilon , M} = \{x \in \mathcal {X} \mid f (x) <   \alpha + \varepsilon \}
$$
with  $\varepsilon > 0$  and  $M < 0$ . Because of Ass. 4, at least one optimal point lies in a region of positive measure that is continuous for the continuous variables. Therefore, we have that  $\alpha = f(\pmb{x}^{*})$ . Note that this is also the case if the domain of  $f$  only consists of combinatorial variables (Ass. 5). Then,  $R_{\varepsilon, M} = \{\pmb{x} \in \mathcal{X} \mid f(\pmb{x}) < f(\pmb{x}^{*}) + \varepsilon\}$ .
Figure 8: Bounce the other algorithms on the synthetic Ackley53 benchmark function. Bounce outperforms all other algorithms and quickly finds excellent solutions. BODi's performance degrades upon randomization.
Let  $(\pmb{x}_k^\star)_{k=1}^\infty$  denote the sequence of best points that Bounce discovers with  $\pmb{x}_k^\star$  being the best point up to iteration  $k$ . This sequence satisfies Def. 1 by construction. Note that  $\pmb{x}_k^\star \in R_{\varepsilon, M}$  implies that  $\pmb{x}_{k'}^\star \in R_{\varepsilon, M}$  for all  $k' \geq k + 1$  [73] because observations are noise-free. Then,
$$
\begin{array}{l} \mathbb {P} \left[ \boldsymbol {x} _ {k} ^ {\star} \in R _ {\varepsilon , M} \right] = 1 - \mathbb {P} \left[ \boldsymbol {x} _ {k} ^ {\star} \in \mathcal {X} \backslash R _ {\varepsilon , M} \right] \\ \geq 1 - \left(1 - \mu \left(R _ {\varepsilon , M}\right)\right) ^ {k}, \\ \end{array}
$$
$$
1\geq \lim_{k\to \infty}\mathbb{P}\left[\boldsymbol{x}_{k}^{\star}\in R_{\varepsilon ,M}\right]\geq \underbrace{1 - \lim_{k\to\infty}\left(1 - \mu(R_{\varepsilon,M})\right)^{k}}_{= 1,\text{Eq. (1)}} = 1,
$$
i.e.,  $\pmb{x}_k^\star$  eventually falls into the optimality region [73]. By letting  $\varepsilon \to 0$ ,  $\pmb{x}_k^\star$  converges to the global optimum with probability 1 as  $k \to \infty$ .
B Additional experiments
We compare Bounce to the other algorithms on three additional benchmark problems: Ackley53 and MaxSAT60 [21]. Moreover, we run two additional studies to investigate the performance of Bounce further. First, we run Bounce on a set of continuous problems from Papenmeier et al. [60] to showcase the performance and scalability of Bounce on purely continuous problems. We then present a "low-sequence" version of Bounce to showcase how such a version can outperform its competitors on the original benchmarks by introducing a bias towards low-sequence solutions.
B.1 Bounce and other algorithms on additional benchmarks
B.1.1 The synthetic Ackley53 benchmark function