SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Leonard Papenmeier
Lund University
leonard.papenmeier@cs.lth.se
Luigi Nardi
Lund University, Stanford University, DBtune luigi.nardi@cs.lth.se
Matthias Poloczek
San Francisco, CA 94105, USA
matpol@amazon.com
Impactful applications such as materials discovery, hardware design, neural architecture search, or portfolio optimization require optimizing high-dimensional black-box functions with mixed and combinatorial input spaces. While Bayesian optimization has recently made significant progress in solving such problems, an in-depth analysis reveals that the current state-of-the-art methods are not reliable. Their performances degrade substantially when the unknown optima of the function do not have a certain structure. To fill the need for a reliable algorithm for combinatorial and mixed spaces, this paper proposes Bounce that relies on a novel map of various variable types into nested embeddings of increasing dimensionality. Comprehensive experiments show that Bounce reliably achieves and often even improves upon state-of-the-art performance on a variety of high-dimensional problems.
Bayesian optimization (BO) has become a 'go-to' method for optimizing expensive-to-evaluate black-box functions [27, 83] that have numerous important applications, including hyperparameter optimization for machine learning models [9, 27], portfolio optimization in finance [7], chemical engineering and materials discovery [13, 15, 26, 33, 37, 38, 40, 57, 68, 71, 81], hardware design [22, 36, 51], or scheduling problems [42]. These problems are challenging for a variety of reasons. Most importantly, they may expose hundreds of tunable parameters that allow for granular optimization of the underlying design but also lead to high-dimensional optimization tasks and the 'curses of dimensionality' [10, 63]. Typical examples are drug design [53, 76] and combinatorial testing [55]. Moreover, real-world applications often have categorical or ordinal tunable parameters, in addition to the real-valued parameters that BO has traditionally focused on [10, 27, 70]. Recent efforts have thus extended BO to combinatorial and mixed spaces. Casmopolitan of Wan et al. [79] uses trust regions (TRs) to accommodate high dimensionality, building upon prior work of Eriksson et al. [25] for continuous spaces. COMBO of Oh et al. [56] constructs a surrogate model based on a combinatorial graph representation of the function. Recently, Deshwal et al. [21] presented BODi that employs a novel type of dictionary-based embedding and showed that it outperforms the prior work. However, the causes for BODi's excellent performance are not yet well-understood and require a closer examination. Moreover, the ability of methods for mixed spaces to scale to higher dimensionalities trails behind BO for continuous domains. In particular, Papenmeier et al. [60] showed that nested embeddings allow BO to handle a thousand input dimensions, thus outperforming vanilla TR-based approaches and raising the question of whether similar performance gains are feasible for combinatorial domains.
In this work, we assess and improve upon the state-of-the-art in combinatorial BO. In particular, we make the following contributions:
1. We conduct an in-depth analysis of two state-of-the-art algorithms for combinatorial BO, COMBO [56] and BODi [21]. The analysis reveals that their performances often degrade considerably when the optimum of the optimization problem does not exhibit a particular structure common for synthetic test problems.
2. We propose Bounce (Bayesian optimization using increasingly high-dimensional combinatorial and continuous embeddings), a novel high-dimensional Bayesian optimization (HDBO) method that effectively optimizes over combinatorial, continuous, and mixed spaces. Bounce leverages parallel function evaluations efficiently and uses nested random embeddings to scale to high-dimensional problems.
3. We provide a comprehensive evaluation on a representative collection of combinatorial, continuous, and mixed-space benchmarks, demonstrating that Bounce is on par with or outperforms state-of-the-art methods.
Bayesian optimization. Bayesian optimization aims to find the global optimum  $\pmb{x}^{*} \in \mathcal{X}$  of a black-box function  $f: \mathcal{X} \to \mathbb{R}$ , where  $\mathcal{X}$  is the  $D$ -dimensional search space or input space. Throughout this paper, we consider minimization problems, i.e., we aim to find  $\pmb{x}^{*} \in \mathcal{X}$  such that  $f(\pmb{x}^{*}) \leq f(\pmb{x})$  for all  $\pmb{x} \in \mathcal{X}$ . The search space  $\mathcal{X}$  may contain variables of different types: continuous, categorical, and ordinal. We denote the number of continuous variables in  $\mathcal{X}$  by  $n_{\mathrm{cont}}$  and the number of combinatorial variables by  $n_{\mathrm{comb}} = n_{\mathrm{cat}} + n_{\mathrm{ord}} = D - n_{\mathrm{cont}}$ , where we denote the number of categorical variables by  $n_{\mathrm{cat}}$  and the number of ordinal variables by  $n_{\mathrm{ord}}$ .
Combinatorial domains. Extending BO to combinatorial spaces is challenging, for example, because the acquisition function is only defined at discrete locations or the dimensionality of the space grows drastically when using one-hot encoding for categorical variables. Due to its numerous applications, combinatorial BO has received increased attention in recent years. B0CS [6] handles the exponential explosion of combinations by only modeling lower-order interactions of combinatorial variables and imposing a sparse prior on the interaction terms. COMBO [56] models each variable as a graph and uses the graph-Cartesian product to represent the search space. We revisit COMBO's performance on categorical problems in Appendix H.1. CoCaBo [67] combines multi-armed bandits and BO to allow for optimization in mixed spaces. It uses two separate kernels for continuous and combinatorial variables and proposes a weighted average of a product and a sum kernel to model mixed spaces. Liu and Wang [47] show that certain, possibly combinatorial functions can be modeled by parametric function approximators such as neural networks or random forests. As a general method to optimize the acquisition function with gradient-based methods, Daulton et al. [19] propose probabilistic reparametrization.
High-dimensional continuous spaces. Subspace-based methods are primarily used for continuous spaces. Wang et al. [84] propose REMBO for HDBO in continuous spaces using Gaussian random projection matrices. REMBO suffers from distortions and projections outside the search domains that the corrections of Binois et al. [11, 12] address. The HeSBO algorithm of Nayebi et al. [52] avoids the need for corrections by using the CountSketch embedding [87]. A1ebo of Letham et al. [46] builds upon REMBO, learning suitable corrections of distortions. TuRBO [25] is a method that operates in the full-dimensional input space  $\mathcal{X}$ , relying on trust region (TR) to focus the search on promising regions of the search space. BAxUS of Papenmeier et al. [60] combines the trust region approach of TuRBO with the random subspace idea of HeSBO. BAxUS uses a novel family of nested random subspaces that exhibit better theoretical guarantees than the CountSketch embedding. While BAxUS handled a  $1000D$  problem, it only considers continuous problems and cannot leverage parallel function evaluations. GTBO [35] assumes the existence of an axis-aligned active subspace. The algorithm first identifies "active" variables and optimizes in the full-dimensional space by placing separate strong length scale priors onto active and inactive variables. Another line of recent approaches employs Monte-Carlo tree search (MCTS) to reduce the complexity of the problem. Wang et al. [82] use MCTS to learn a partitioning of the continuous search space to focus the search on promising regions in the search space. Song et al. [74] use a similar approach, but instead of learning promising regions in the search space, they assume an axis-aligned active subspace and use MCTS to select important variables. Linear embeddings and random linear embeddings [14, 46, 52, 60, 84] require little or no training data to construct the embedding but assume a linear subspace.
Algorithm 1 The Bounce algorithm
Input: initial target dimensionality  $d_{\mathrm{init}}$  , evaluation budget m, batch size B, evaluation budget to input dimensionality  $m_D$  # new bins added per dimension b, number design of experiment (DOE) points  $n_{\mathrm{init}}$    
Output: optimizer  $\pmb{x}^{*}\in \arg \min_{\pmb{x}\in \mathcal{D}}f(\pmb {x})$    
1:  $i\gets 0,d\gets d_{\mathrm{init}}$    
2:  $b\gets$  ADJUSTBINS(b,  $d_{\mathrm{init}},m_D)$  ▷ Section 3   
3:  $m_i\gets \left\lfloor \frac{b\cdot m_D\cdot d_{\mathrm{init}}}{d_{\mathrm{init}}\cdot(1 - (b + 1)^k + 1}\right]$    
4:  $S\gets$  INITIALLEMBEDDING(d0, ncont, ncat, ncomb, nbin) ▷ Section 3.1   
5:  $\mathcal{D}\gets \{(z_k,f(S^{-1}(z_k)))\}_{k\in n_{\mathrm{init}}}>$  Sample and evaluate initial points.   
6: for  $j = 1,\dots ,m$  do   
7:  $L_{\mathrm{cont}}\gets 0.8,L_{\mathrm{comb}}\gets \min (40,n_{\mathrm{comb}})$    
8: while  $L_{\mathrm{cont}} > L_{\mathrm{min}}^{\mathrm{cont}}\wedge L_{\mathrm{comb}} > L_{\mathrm{min}}^{\mathrm{comb}}$  do   
9: Find B candidates according to Sec. 3.2   
10: Evaluate f at B and update D:  $\mathcal{D}\gets \mathcal{D}\cup \{(z_k,f(S^{-1}(z_k)))\}_{k\in B}$    
11: Update  $L_{\mathrm{cont}}$  and  $L_{\mathrm{comb}}$  ▷ Section 3   
12: if  $d <   D$  then   
13:  $i\gets i + 1$  ▷ Increase index for target dimensionality.   
14:  $S\gets$  INCREASEEMBEDDING(S,b) ▷ Section 3.1   
15:  $d\gets \#$  target variables in S   
16:  $m_i\gets \left\lfloor \frac{b\cdot m_D\cdot d}{d_{\mathrm{init}}\cdot(1 - (b + 1)^k + 1}\right]$    
17: else   
18: Reset  $\mathcal{D}$  by resampling and evaluating new initial points   
19: Resample S, reset  $L_{\mathrm{cont}}$  and  $L_{\mathrm{comb}},j\gets j + n_{\mathrm{init}}$
Combinatorial high-dimensional domains. These works optimize black-box functions defined over a combinatorial or mixed space with dozens of input variables. Thebelt et al. [77] use a tree-ensemble kernel to model the Gaussian process (GP) prior and derive a formulation of the upper-confidence bound (UCB) that allows it to be optimized globally and to incorporate constraints. RDUCB [88] relies on random additive decompositions of the GP kernel to model the correlation between variables. Casmopolitan [79] follows TuRBO in using TRs to focus the search on promising regions of the search space and uses the Hamming distance to model TRs for combinatorial variables. For mixed spaces, Casmopolitan uses interleaved search and models continuous and categorical variables with two separate TRs. Kim et al. [44] use a random projection matrix to approach combinatorial problems in a continuous embedded subspace. When evaluating a point, their approach projects the continuous candidate point to the high-dimensional search space and then rounds to the next feasible combinatorial solution. Deshwal et al. [20] propose two algorithms for permutation spaces, which occur in problems such as compiler optimization [36] and pose special challenges due to the superexponential explosion of solutions. BODi [21] proposes an embedding type based on a dictionary of anchor points in the search space. The pairwise Hamming distances between the point and each anchor point  $a_i$  in the dictionary represent a point in the search space. The anchor points in the dictionary change at each iteration of the algorithm. They are sampled from the search space to cover a wide range of 'sequences', i.e., the number of changes from 0 to 1 (and vice versa) in the binary vector. The authors hypothesize that the diverse sampling procedure leads to BODi's remarkable performance in combinatorial spaces with up to 60 dimensions. To our knowledge, BODi is the only other method combining embeddings and combinatorial spaces. We show in Section 4.6 that BODi's reported good performance relies on an artificial structure of the optimizer  $x^*$  and that its performance degrades considerably when this structure is violated.
3 The Bounce algorithm
To overcome the aforementioned challenges in HDBO for real-world applications, we propose Bounce, a new algorithm for continuous, combinatorial, and mixed spaces. Bounce uses a GP [85] surrogate in a lower-dimensional subspace, the target space, that is realized by partitioning input variables into 'bins', the so-called target dimensions. Bounce only bins variables of the same type (categorical, binary, ordinal, and continuous). When selecting new points to evaluate, Bounce sets
all input variables within the same bin to a single value. It thus operates in a subspace of lower dimensionality than the input space and, in particular, maximizes the acquisition function in a subspace of lower dimensionality. Bounce iteratively refines its subspace embedding by splitting bins into smaller bins, allowing for a more granular optimization at the expense of higher dimensionality. Note that by splitting up bins, Bounce asserts that observations taken in earlier subspaces are contained in the current subspace; see Papenmeier et al. [60] for details. Thus, Bounce operates in a series of nested subspaces. It uses a novel TR management to leverage batch parallelism efficiently, improving over the single point acquisition of BAxUS [60].
The nested subspaces. To model the GP in low-dimensional subspaces, Bounce leverages BAxUS' family of nested random embeddings [60]. In particular, Bounce employs the sparse count-sketch embedding [87] in which each input dimension is assigned to exactly one target dimension. When increasing the target dimensionality, Bounce creates  $b$  new bins for every existing bin and redistributes the input dimensions that had previously been assigned to that bin across the now  $b + 1$  bins. Bounce allocates an individual evaluation budget  $m_{i}$  to the current target space  $\mathcal{X}_i$  that is proportional to the dimensionality of  $\mathcal{X}_i$ . When the budget for the current target space is depleted, and Bounce has not found a better solution, Bounce will increase the dimension of the target space until it reaches the input space of dimensionality  $D$ . Let  $d_{\mathrm{init}}$  denote the dimensionality of the first target space, i.e., the random embedding that Bounce starts with. Then Bounce has to increase the target dimension  $\left\lceil \log_{b + 1}D / d_{\mathrm{init}}\right\rceil =: k$ -times to reach the input dimensionality  $D$ . After calculating  $k$ , Bounce re-sets the split factor  $b$  such that the distance between the predicted final target dimensionality  $d_{k} = d_{\mathrm{init}}\cdot (b + 1)^{k}$  and the input dimensionality  $D$  is minimized:  $b = \lfloor \log_k(D / d_{\mathrm{init}}) - 1\rfloor$ , where  $\lfloor x\rfloor$  denotes the integer closest to  $x$ . This ensures that the predetermined evaluation budget for each subspace will be approximately proportional to its dimensionality. This contrasts to BAxUS [60] that uses a constant split factor  $b$  and adjusts the initial target dimensionality  $d_{\mathrm{init}}$ . The evaluation budget  $m_{i}$  for the  $i$ -th subspace  $\mathcal{X}_i$  is  $m_{i} := \left\lfloor \frac{b\cdot m_{D}\cdot d_{i}}{d_{\mathrm{init}}\cdot(1 - (b + 1)^{k + 1}}\right\rceil$ , where  $m_D$  is the budget until  $D$  is reached and  $b$  is the maximum number of bins added per split.
Trust region management. Bounce follows TuRBO [25] and Casmopolitan [79] in using trust regions (TRs) to efficiently optimize over target spaces of high dimensionality. TRs allow focusing on promising regions of the search space by restricting the next points to evaluate to a region centered at the current best function value [25]. TR-based methods usually expand their TR if they find better points and conversely shrink it if they fail to make progress. If the TR falls below the threshold given by the base length, TuRBO and Casmopolitan restart with a fresh TR elsewhere. Casmopolitan [79] uses different base lengths for combinatorial and continuous variables. For combinatorial variables, the distance to the currently best function value is defined in terms of the Hamming distance, and the base length is an integer. For continuous variables, Casmopolitan defines the base length in terms of the Euclidean distance, i.e., a real number. Similarly, Bounce has separate base lengths  $L_{\min}^{\mathrm{cont}}$  and  $L_{\min}^{\mathrm{comb}}$  for continuous and combinatorial variables but does not fix the factor by which the TR volume is increased or decreased upon successes or failures. Instead, the factor is adjusted dynamically so that the evaluation budget  $m_i$  for the current target space  $\mathcal{X}_i$  is adhered to. This design is crucial to enable batch parallelism, as we describe next.
Batch parallelism. We allow Bounce to efficiently evaluate batches of points in parallel by using a scalable TR management strategy and  $q$ -expected improvement (qEI) [5, 80, 86] as the acquisition function for batches of size  $B > 1$ . When Bounce starts with a fresh TR, we sample  $n_{\mathrm{init}}$  initial points to initialize the GP, using a Sobol sequence for continuous variables and uniformly random values for combinatorial variables.
The TR management strategy of Bounce differs from previous strategies [25, 60, 65, 79] in that it uses a dynamic factor to determine the TR base length. Recall that Bounce shrinks the TR if it fails to make progress and starts a fresh TR if the TR falls below the threshold given by the base length. Bounce's rule is based on the idea that the minimum admissible TR base length should be reached when the current evaluation budget is exhausted. If one employed the strategies of TuRBO [25], Casmopolitan [79], or BAxUS [60] for larger batch sizes  $B$  and Bounce's nested subspaces, then one would spend a large part of the evaluation budget in early target spaces. For example, suppose a continuous problem, the common values for the initial, minimum, and maximum TR base length, and the constant shrinkage factor of [65]. Then, such a method has to shrink the TR base length at least seven times (i.e., evaluate  $f7B$ -times) before it would increase the dimensionality of the target space. Thus, the method would risk depleting its budget before reaching a target space suitable
for the problem. On the other hand, we will see that Bounce chooses an evaluation budget that is smaller in low-dimensional target spaces and higher for later target spaces of higher dimensionality. Considering a 1000-dimensional problem with an evaluation budget of 1000, it uses only 3, 12, and 47 samples for the first three target spaces of dimensionalities 2, 8, and 32.
Bounce's strategy permits flexible TR shrinkage factors and base lengths, i.e., TR base lengths to vary within the range  $[L_{\min}, L_{\max}]$ . This allows Bounce to comply with the evaluation budget  $m_i$  for the current target space  $\mathcal{X}_i$ . Suppose that Bounce has evaluated  $j$  batches of  $B$  points each since it last increased the dimensionality of the target space, and let  $L_j$  denote the current TR base length. Observe that hence  $m_i - jB$  evaluations remain for  $\mathcal{X}_i$ . Then Bounce sets the TR base length to  $L_{j+1} \coloneqq \lambda_j^{-B}L_j$  with  $\lambda_j < 1$ , if it found a new best point whose objective value improves upon the incumbent by at least  $\varepsilon$ . We call this a 'success'. Otherwise, Bounce observes a 'failure' and sets  $L_{j+1} \coloneqq \lambda_j^{+B}L_j$ . The rationale of this rule is that if the algorithm is in iteration  $j$  and only observes failures subsequently, then we apply this factor  $(m_i - jB)$ -times, which is the remaining number of function evaluations in the current subspace  $\mathcal{X}_i$ . Hence, the last batch of the  $i$ -th target space  $\mathcal{X}_i$  will have the minimum TR base length, and Bounce will increase the target dimensionality afterward. If the TR is expanded upon a 'success', we need to adjust  $\lambda_j$  not to use more than the allocated number of function evaluations in a target space. At each iteration, we therefore set adjustment factor  $\lambda_j = (L_{\min} / L_j)^{1/(m_i - jB)}$ . Note that  $\lambda_j$  remains unchanged under this rule unless the TR expanded in the previous iteration.
The kernel choice. To harvest the sample efficiency of a low-dimensional target space, we would like to combine categorical variables into a single bin, even if they vary in the number of categories. This is not straightforward. For example, note that the popular one-hot encoding of categorical variables would give rise to multiple binary input dimensions, which would not be compatible with the above strategy of binning variables to form nested subspaces. Bounce overcomes these obstacles and allows variables of the same type to share a representation in the target space. We provide the details in Sect. 3.1.
For the GP model, we use the CoCaBo kernel [67]. In particular, we model the continuous and combinatorial variables with two separate  $5/2$ -Matérn kernels where we use automatic relevance determination (ARD) for the continuous variables and share one length scale for all combinatorial variables. Following Ru et al. [67], we use a mixture of the sum and the product kernel:
$$
k (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \sigma_ {f} ^ {2} (\rho k _ {\mathrm {c m b}} (\boldsymbol {x} _ {\mathrm {c m b}}, \boldsymbol {x} _ {\mathrm {c m b}} ^ {\prime}) k _ {\mathrm {c n t}} (\boldsymbol {x} _ {\mathrm {c n t}}, \boldsymbol {x} _ {\mathrm {c n t}} ^ {\prime}) + (1 - \rho) (k _ {\mathrm {c m b}} (\boldsymbol {x} _ {\mathrm {c m b}}, \boldsymbol {x} _ {\mathrm {c m b}} ^ {\prime}) + k _ {\mathrm {c n t}} (\boldsymbol {x} _ {\mathrm {c n t}}, \boldsymbol {x} _ {\mathrm {c n t}} ^ {\prime}))),
$$
where  $x_{\mathrm{cnt}}$  and  $x_{\mathrm{cmb}}$  are the continuous and combinatorial variables in  $x$ , respectively, and  $\sigma_f^2$  is the signal variance. The trade-off parameter  $\rho \in [0,1]$  is learned jointly with the other hyperparameters via likelihood maximization. See Appendix G.2 for additional details.
Algorithm 1 gives a high-level overview of Bounce. In Appendix A, we prove that Bounce converges to the global optimum under mild assumptions. We now explain the different components of Bounce in detail.
3.1 The subspace embedding of mixed spaces
Bounce supports mixed spaces of four types of input variables: categorical, ordinal, binary, and continuous variables. We discuss binary and categorical variables separately because we model them differently. The proposed embedding maps only variables of a single type to each 'bin', i.e., no target dimension of the embedding has variables of different types. Thus, target dimensions are homogeneous in this regard. Note that the number of target dimensions of each type is implied by the current bin size of the embedding that may grow during the execution. The proposed embedding can handle categorical or ordinal input variables that differ in the number of discrete values they can take.
Continuous variables. As common in BO, we suppose that each continuous variable takes values in a bounded interval. Thus, we may normalize each interval to  $[-1, 1]$ . The embedding of continuous variables, i.e., input dimensions, follows BAxUS [60]: each input dimension  $D_{i}$  is associated with a random sign  $s_{i} \in \{-1, +1\}$  and one or multiple input dimensions can be mapped to the same target dimension of the low-dimensional embedded subspace. Recall that Bounce works on the low-dimensional subspace and thus decides an assignment  $v_{j}$  for every target dimension  $d_{j}$  of the embedding. Then, all input variables mapped to this particular target dimension are set to this value  $v_{j}$ .