Title: Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian optimization

global optimization

Gaussian process

combinatorial optimization

high-dimensional
Abstract: Impactful applications such as materials discovery, hardware design, neural architecture search, or portfolio optimization require optimizing high-dimensional black-box functions with mixed and combinatorial input spaces.
While Bayesian optimization has recently made significant progress in solving such problems, an in-depth analysis reveals that the current state-of-the-art methods are not reliable. 
Their performances degrade substantially when the unknown optima of the function do not have a certain structure. 
To fill the need for a reliable algorithm for combinatorial and mixed spaces, this paper proposes Bounce that relies on a novel map of various variable types into nested embeddings of increasing dimensionality.
Comprehensive experiments show that Bounce reliably achieves and often even improves upon state-of-the-art performance on a variety of high-dimensional problems.

Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces

Abstract

Impactful applications such as materials discovery, hardware design, neural architecture search, or portfolio optimization require optimizing high-dimensional black-box functions with mixed and combinatorial input spaces. While Bayesian optimization has recently made significant progress in solving such problems, an in-depth analysis reveals that the current state-of-the-art methods are not reliable. Their performances degrade substantially when the unknown optima of the function do not have a certain structure. To fill the need for a reliable algorithm for combinatorial and mixed spaces, this paper proposes Bounce that relies on a novel map of various variable types into nested embeddings of increasing dimensionality. Comprehensive experiments show that Bounce reliably achieves and often even improves upon state-of-the-art performance on a variety of high-dimensional problems.

1 Introduction

Bayesian optimization (BO) has become a 'go-to' method for optimizing expensive-to-evaluate black-box functions [27, 83] that have numerous important applications, including hyperparameter optimization for machine learning models [9, 27], portfolio optimization in finance [7], chemical engineering and materials discovery [13, 15, 26, 33, 37, 38, 40, 57, 68, 71, 81], hardware design [22, 36, 51], or scheduling problems [42]. These problems are challenging for a variety of reasons. Most importantly, they may expose hundreds of tunable parameters that allow for granular optimization of the underlying design but also lead to high-dimensional optimization tasks and the 'curses of dimensionality' [10, 63]. Typical examples are drug design [53, 76] and combinatorial testing [55]. Moreover, real-world applications often have categorical or ordinal tunable parameters, in addition to the real-valued parameters that BO has traditionally focused on [10, 27, 70]. Recent efforts have thus extended BO to combinatorial and mixed spaces. Casmopolitan of Wan et al. [79] uses trust regions (TRs) to accommodate high dimensionality, building upon prior work of Eriksson et al. [25] for continuous spaces. COMBO of Oh et al. [56] constructs a surrogate model based on a combinatorial graph representation of the function. Recently, Deshwal et al. [21] presented BODi that employs a novel type of dictionary-based embedding and showed that it outperforms the prior work. However, the causes for BODi 's excellent performance are not yet well-understood and require a closer examination. Moreover, the ability of methods for mixed spaces to scale to higher dimensionalities trails behind BO for continuous domains. In particular, Papenmeier et al. [60] showed that nested embeddings allow BO to handle a thousand input dimensions, thus outperforming vanilla TR-based approaches and raising the question of whether similar performance gains are feasible for combinatorial domains.

In this work, we assess and improve upon the state-of-the-art in combinatorial BO. In particular, we make the following contributions:

We conduct an in-depth analysis of two state-of-the-art algorithms for combinatorial BO, COMBO [56] and BODi [21]. The analysis reveals that their performances often degrade considerably when the optimum of the optimization problem does not exhibit a particular structure common for synthetic test problems.

We propose Bounce ( B ayesian o ptimization u sing i n creasingly high-dimensional c ombinatorial and continuous e mbeddings), a novel high-dimensional Bayesian optimization (HDBO) method that effectively optimizes over combinatorial, continuous, and mixed spaces. Bounce leverages parallel function evaluations efficiently and uses nested random embeddings to scale to highdimensional problems.

We provide a comprehensive evaluation on a representative collection of combinatorial, continuous, and mixed-space benchmarks, demonstrating that Bounce is on par with or outperforms state-of-the-art methods.

2 Background and related work

Bayesian optimization. Bayesian optimization aims to find the global optimum x ∗ ∈ X of a black-box function f : X → R , where X is the D -dimensional search space or input space . Throughout this paper, we consider minimization problems, i.e., we aim to find x ∗ ∈ X such that f ( x ∗ ) ≤ f ( x ) for all x ∈ X . The search space X may contain variables of different types: continuous, categorical, and ordinal. We denote the number of continuous variables in X by n cont and the number of combinatorial variables by n comb = n cat + n ord = D -n cont , where we denote the number of categorical variables by n cat and the number of ordinal variables by n ord .

Combinatorial domains. Extending BO to combinatorial spaces is challenging, for example, because the acquisition function is only defined at discrete locations or the dimensionality of the space grows drastically when using one-hot encoding for categorical variables. Due to its numerous applications, combinatorial BO has received increased attention in recent years. BOCS [6] handles the exponential explosion of combinations by only modeling lower-order interactions of combinatorial variables and imposing a sparse prior on the interaction terms. COMBO [56] models each variable as a graph and uses the graph-Cartesian product to represent the search space. We revisit COMBO 's performance on categorical problems in Appendix H.1. CoCaBo [67] combines multi-armed bandits and BO to allow for optimization in mixed spaces. It uses two separate kernels for continuous and combinatorial variables and proposes a weighted average of a product and a sum kernel to model mixed spaces. Liu and Wang [47] show that certain, possibly combinatorial functions can be modeled by parametric function approximators such as neural networks or random forests. As a general method to optimize the acquisition function with gradient-based methods, Daulton et al. [19] propose probabilistic reparametrization.

High-dimensional continuous spaces. Subspace-based methods are primarily used for continuous spaces. Wang et al. [84] propose REMBO for HDBO in continuous spaces using Gaussian random projection matrices. REMBO suffers from distortions and projections outside the search domains that the corrections of Binois et al. [11, 12] address. The HeSBO algorithm of Nayebi et al. [52] avoids the need for corrections by using the CountSketch embedding [87]. Alebo of Letham et al. [46] builds upon REMBO , learning suitable corrections of distortions. TuRBO [25] is a method that operates in the full-dimensional input space X , relying on trust region (TR) to focus the search on promising regions of the search space. BAxUS of Papenmeier et al. [60] combines the trust region approach of TuRBO with the random subspace idea of HeSBO . BAxUS uses a novel family of nested random subspaces that exhibit better theoretical guarantees than the CountSketch embedding. While BAxUS handled a 1000 D problem, it only considers continuous problems and cannot leverage parallel function evaluations. GTBO [35] assumes the existence of an axis-aligned active subspace. The algorithm first identifies 'active' variables and optimizes in the full-dimensional space by placing separate strong length scale priors onto active and inactive variables. Another line of recent approaches employs Monte-Carlo tree search (MCTS) to reduce the complexity of the problem. Wang et al. [82] use MCTS to learn a partitioning of the continuous search space to focus the search on promising regions in the search space. Song et al. [74] use a similar approach, but instead of learning promising regions in the search space, they assume an axis-aligned active subspace and use MCTS to select important variables. Linear embeddings and random linear embeddings [14, 46, 52, 60, 84] require little or no training data to construct the embedding but assume a linear subspace.

Algorithm 1 The Bounce algorithm

Input: initial target dimensionality d init, evaluation budget m , batch size B , evaluation budget to

Combinatorial high-dimensional domains. These works optimize black-box functions defined over a combinatorial or mixed space with dozens of input variables. Thebelt et al. [77] use a treeensemble kernel to model the Gaussian process (GP) prior and derive a formulation of the upperconfidence bound (UCB) that allows it to be optimized globally and to incorporate constraints. RDUCB [88] relies on random additive decompositions of the GP kernel to model the correlation between variables. Casmopolitan [79] follows TuRBO in using TRs to focus the search on promising regions of the search space and uses the Hamming distance to model TRs for combinatorial variables. For mixed spaces, Casmopolitan uses interleaved search and models continuous and categorical variables with two separate TRs. Kim et al. [44] use a random projection matrix to approach combinatorial problems in a continuous embedded subspace. When evaluating a point, their approach projects the continuous candidate point to the high-dimensional search space and then rounds to the next feasible combinatorial solution. Deshwal et al. [20] propose two algorithms for permutation spaces , which occur in problems such as compiler optimization [36] and pose special challenges due to the superexponential explosion of solutions. BODi [21] proposes an embedding type based on a dictionary of anchor points in the search space. The pairwise Hamming distances between the point and each anchor point a i in the dictionary represent a point in the search space. The anchor points in the dictionary change at each iteration of the algorithm. They are sampled from the search space to cover a wide range of 'sequencies', i.e., the number of changes from 0 to 1 (and vice versa) in the binary vector. The authors hypothesize that the diverse sampling procedure leads to BODi 's remarkable performance in combinatorial spaces with up to 60 dimensions. To our knowledge, BODi is the only other method combining embeddings and combinatorial spaces. We show in Section 4.6 that BODi 's reported good performance relies on an artificial structure of the optimizer x ∗ and that its performance degrades considerably when this structure is violated.

3 The Bounce algorithm

To overcome the aforementioned challenges in HDBO for real-world applications, we propose Bounce , a new algorithm for continuous, combinatorial, and mixed spaces. Bounce uses a GP [85] surrogate in a lower-dimensional subspace, the target space , that is realized by partitioning input variables into 'bins', the so-called target dimensions . Bounce only bins variables of the same type (categorical, binary, ordinal, and continuous). When selecting new points to evaluate, Bounce sets

all input variables within the same bin to a single value. It thus operates in a subspace of lower dimensionality than the input space and, in particular, maximizes the acquisition function in a subspace of lower dimensionality. Bounce iteratively refines its subspace embedding by splitting bins into smaller bins, allowing for a more granular optimization at the expense of higher dimensionality. Note that by splitting up bins, Bounce asserts that observations taken in earlier subspaces are contained in the current subspace; see Papenmeier et al. [60] for details. Thus, Bounce operates in a series of nested subspaces. It uses a novel TR management to leverage batch parallelism efficiently, improving over the single point acquisition of BAxUS [60].

The nested subspaces. To model the GP in low-dimensional subspaces, Bounce leverages BAxUS ' family of nested random embeddings [60]. In particular, Bounce employs the sparse count-sketch embedding [87] in which each input dimension is assigned to exactly one target dimension. When increasing the target dimensionality, Bounce creates b new bins for every existing bin and redistributes the input dimensions that had previously been assigned to that bin across the now b +1 bins. Bounce allocates an individual evaluation budget m i to the current target space X i that is proportional to the dimensionality of X i . When the budget for the current target space is depleted, and Bounce has not found a better solution, Bounce will increase the dimension of the target space until it reaches the input space of dimensionality D . Let d init denote the dimensionality of the first target space, i.e., the random embedding that Bounce starts with. Then Bounce has to increase the target dimension ⌈ log b +1 D/d init ⌉ =: k -times to reach the input dimensionality D . After calculating k , Bounce re-sets the split factor b such that the distance between the predicted final target dimensionality d k = d init · ( b +1) k and the input dimensionality D is minimized: b = ⌊ log k ( D/d init ) -1 ⌉ , where ⌊ x ⌉ denotes the integer closest to x . This ensures that the predetermined evaluation budget for each subspace will be approximately proportional to its dimensionality. This contrasts to BAxUS [60] that uses a constant split factor b and adjusts the initial target dimensionality d init . The evaluation budget m i for the i -th subspace X i is m i := ⌊ b · m D · d i d init · (1 -( b +1) k +1 ) ⌉ , where m D is the budget until D is reached and b is the maximum number of bins added per split.

Trust region management. Bounce follows TuRBO [25] and Casmopolitan [79] in using trust regions (TRs) to efficiently optimize over target spaces of high dimensionality. TRs allow focusing on promising regions of the search space by restricting the next points to evaluate to a region centered at the current best function value [25]. TR-based methods usually expand their TR if they find better points and conversely shrink it if they fail to make progress. If the TR falls below the threshold given by the base length , TuRBO and Casmopolitan restart with a fresh TR elsewhere. Casmopolitan [79] uses different base lengths for combinatorial and continuous variables. For combinatorial variables, the distance to the currently best function value is defined in terms of the Hamming distance, and the base length is an integer. For continuous variables, Casmopolitan defines the base length in terms of the Euclidean distance, i.e., a real number. Similarly, Bounce has separate base lengths L cont min and L comb min for continuous and combinatorial variables but does not fix the factor by which the TR volume is increased or decreased upon successes or failures. Instead, the factor is adjusted dynamically so that the evaluation budget m i for the current target space X i is adhered to. This design is crucial to enable batch parallelism, as we describe next.

Batch parallelism. We allow Bounce to efficiently evaluate batches of points in parallel by using a scalable TR management strategy and q -expected improvement (qEI) [5, 80, 86] as the acquisition function for batches of size B > 1 . When Bounce starts with a fresh TR, we sample n init initial points to initialize the GP, using a Sobol sequence for continuous variables and uniformly random values for combinatorial variables.

The TR management strategy of Bounce differs from previous strategies [25, 60, 65, 79] in that it uses a dynamic factor to determine the TR base length. Recall that Bounce shrinks the TR if it fails to make progress and starts a fresh TR if the TR falls below the threshold given by the base length. Bounce 's rule is based on the idea that the minimum admissible TR base length should be reached when the current evaluation budget is exhausted. If one employed the strategies of TuRBO [25], Casmopolitan [79], or BAxUS [60] for larger batch sizes B and Bounce 's nested subspaces, then one would spend a large part of the evaluation budget in early target spaces. For example, suppose a continuous problem, the common values for the initial, minimum, and maximum TR base length, and the constant shrinkage factor of [65]. Then, such a method has to shrink the TR base length at least seven times (i.e., evaluate f 7 B -times) before it would increase the dimensionality of the target space. Thus, the method would risk depleting its budget before reaching a target space suitable

for the problem. On the other hand, we will see that Bounce chooses an evaluation budget that is smaller in low-dimensional target spaces and higher for later target spaces of higher dimensionality. Considering a 1000 -dimensional problem with an evaluation budget of 1000 , it uses only 3 , 12 , and 47 samples for the first three target spaces of dimensionalities 2 , 8 , and 32 .

Bounce 's strategy permits flexible TR shrinkage factors and base lengths, i.e., TR base lengths to vary within the range [ L min , L max ] . This allows Bounce to comply with the evaluation budget m i for the current target space X i . Suppose that Bounce has evaluated j batches of B points each since it last increased the dimensionality of the target space, and let L j denote the current TR base length. Observe that hence m i -jB evaluations remain for X i . Then Bounce sets the TR base length to L j +1 := λ -B j L j with λ j < 1 , if it found a new best point whose objective value improves upon the incumbent by at least ε . We call this a 'success'. Otherwise, Bounce observes a 'failure' and sets L j +1 := λ + B j L j . The rationale of this rule is that if the algorithm is in iteration j and only observes failures subsequently, then we apply this factor ( m i -jB ) -times, which is the remaining number of function evaluations in the current subspace X i . Hence, the last batch of the i -th target space X i will have the minimum TR base length, and Bounce will increase the target dimensionality afterward. If the TR is expanded upon a 'success', we need to adjust λ j not to use more than the allocated number of function evaluations in a target space. At each iteration, we therefore set adjustment factor λ j = ( L min /L j ) 1 / ( m i -jB ) . Note that λ j remains unchanged under this rule unless the TR expanded in the previous iteration.

The kernel choice. To harvest the sample efficiency of a low-dimensional target space, we would like to combine categorical variables into a single bin, even if they vary in the number of categories. This is not straightforward. For example, note that the popular one-hot encoding of categorical variables would give rise to multiple binary input dimensions, which would not be compatible with the above strategy of binning variables to form nested subspaces. Bounce overcomes these obstacles and allows variables of the same type to share a representation in the target space. We provide the details in Sect. 3.1.

For the GP model, we use the CoCaBo kernel [67]. In particular, we model the continuous and combinatorial variables with two separate 5 / 2 -Matérn kernels where we use automatic relevance determination (ARD) for the continuous variables and share one length scale for all combinatorial variables. Following Ru et al. [67], we use a mixture of the sum and the product kernel:

where x cnt and x cmb are the continuous and combinatorial variables in x , respectively, and σ 2 f is the signal variance. The trade-off parameter ρ ∈ [0 , 1] is learned jointly with the other hyperparameters via likelihood maximization. See Appendix G.2 for additional details.

Algorithm 1 gives a high-level overview of Bounce . In Appendix A, we prove that Bounce converges to the global optimum under mild assumptions. We now explain the different components of Bounce in detail.

3.1 The subspace embedding of mixed spaces

Bounce supports mixed spaces of four types of input variables: categorical, ordinal, binary, and continuous variables. We discuss binary and categorical variables separately because we model them differently. The proposed embedding maps only variables of a single type to each 'bin', i.e., no target dimension of the embedding has variables of different types. Thus, target dimensions are homogeneous in this regard. Note that the number of target dimensions of each type is implied by the current bin size of the embedding that may grow during the execution. The proposed embedding can handle categorical or ordinal input variables that differ in the number of discrete values they can take.

Continuous variables. As common in BO, we suppose that each continuous variable takes values in a bounded interval. Thus, we may normalize each interval to [ -1 , 1] . The embedding of continuous variables, i.e., input dimensions, follows BAxUS [60]: each input dimension D i is associated with a random sign s i ∈ {-1 , +1 } and one or multiple input dimensions can be mapped to the same target dimension of the low-dimensional embedded subspace. Recall that Bounce works on the low-dimensional subspace and thus decides an assignment v j for every target dimension d j of the embedding. Then, all input variables mapped to this particular target dimension are set to this value v j .

Binary variables. Binary dimensions are represented by the values -1 and +1 . Each input dimension D i is associated with a random sign s i ∈ {-1 , +1 } , and the subspace embedding may map one or more binary input dimensions to the same binary target dimension. While the embedding for binary and continuous dimensions is similar, Bounce handles binary dimensions differently when optimizing the acquisition function.

Categorical and ordinal variables. Categorical variables that differ in the number of categories may be mapped to the same target dimension (bin). Suppose that the categorical variables v 1 , . . . , v ℓ with cardinalities c 1 , . . . , c ℓ are mapped to a single bin that is associated with the target dimension d j of the subspace embedding. Then d j is of categorical type and has max { c i | 1 ≤ i ≤ ℓ } =: c max distinct categories, that is, its cardinality is given by the maximum cardinality of any variable mapped to it. Thus, the bin d j can represent every category of these input variables.

Suppose that Bounce assigns the category k ∈ { 1 , . . . , c max } to the categorical bin (target dimension) d j . Wetransform this label to a categorical assignment to each input variable v 1 , . . . , v ℓ , setting v i = ⌈ k · ( c i /c max ) ⌉ . Recall that Bounce may split up bins, i.e., target dimensions, to increase the dimensionality of its subspace embedding. In such an event, every derived bin inherits the cardinality of the parent bin. This allows us to retain any observations the algorithm has taken up to this point. Analogously to the random sign for binary variables, we randomly shuffle the categories before the embedding. This reduces the risk of Bounce being biased towards a specific structure of the optimizer (see Appendix E).

We treat ordinal variables as categorical variables whose categories correspond to the discrete values the ordinal variable can take. For the sake of simplicity, we suppose here that an ordinal variable v i has range { 1 , 2 , . . . , c i } and c i ≥ 2 for all i ∈ { 1 , . . . , ℓ } . Figure 1 shows examples of the binning of categorical and ordinal variables.

3.2 Maximization of the acquisition function

We use expected improvement (EI) [43] for batches of size B = 1 and qEI [5, 80, 86] for larger batches. We optimize the EI using gradient-based methods for continuous problems and local search for combinatorial problems. We interleave gradient-based optimization and local search for functions defined over a mixed space; see Appendix G.1 for details.

4 Experimental evaluation

We evaluate Bounce empirically on various benchmarks whose inputs are combinatorial, continuous, or mixed spaces. The evaluation comprises the state-of-the-art algorithms BODi [21], Casmopolitan [79], COMBO [56], SMAC [41], and RDUCB [88], using code provided by the authors. We also report Random Search [8] as a baseline. For categorical problems, COMBO 's implementation suffers from a bug explained in Appendix H.2. We report the results for COMBO with the correct benchmark implementation as ' COMBO (fixed)'.

The experimental setup. We initialize every algorithm with five initial points. The plots show the performances of all algorithms averaged over 50 repetitions except BODi , which has 20 repetitions due to resource constraints caused by its high memory demand. The shaded regions give the standard error of the mean. We use common random seeds for all algorithms and for randomizing the

0

200

400

Iteration

4

3

2

1

Objective value

50D-LABS

0

200

400

Iteration

50D-LABS (publ.)

Bounce

Casmopolitan

BODi

Random Search

COMBO

RDUCB

SMAC

0

200

400

Iteration

2.78

2.77

2.76

2.75

2.74

Objective value

1e14

125D-ClusterExpansion

0

200

400

Iteration

125D-ClusterExpansion (publ.)

Bounce

Casmopolitan

BODi

Random Search

COMBO

RDUCB

SMAC

benchmark functions. We run all methods for 200 function evaluations unless stated otherwise. The Labs (Section 4.1) and MaxSat125 (Section 4.2) benchmarks are run for 500 evaluations.

The benchmarks. The evaluation uses seven established benchmarks [21]: 53 D SVM , 50 D LABS , 125 D ClusterExpansion [3, 4], 60 D MaxSAT60 [21, 56], 25 D PestControl , 53 D Ackley53 , and 25 D Contamination [6, 39, 56]. Due to space constraints, we moved the results for the MaxSAT60 , Contamination , and Ackley53 benchmarks to Appendix B.1. For each benchmark, we report results for the originally published formulation and for a modification where we move the optimal point to a random location. The randomization procedure is fixed for each benchmark for all algorithms and repetitions. For binary problems, we flip each input variable independently with probability 0 . 5 . For categorical problems, we randomly permute the order of the categories. We motivate this randomization in Section 4.6.

4.1 50D Low-Autocorrelation Binary Sequences ( LABS )

LABS has n = 50 binary dimensions. It has important applications in communications engineering and mathematics; see [58] for details. LABS is a hard combinatorial problem and currently solved via exhaustive search. The goal is to find a sequence x ∈ {-1 , +1 } n with a maximum merit factor F ( x ) = n 2 2 E ( x ) , where E ( x ) = ∑ n -1 k =1 C 2 k ( x ) and C k ( x ) = ∑ n -k i =1 x i x i + k for k = 0 , . . . , n -1 are the autocorrelations of x [58]. Figure 2 summarizes the performances. We observe that Bounce outperforms all other algorithms on the benchmark's original and randomized versions.

4.2 Industrial Maximum Satisfiability: 125D ClusterExpansion benchmark

MaxSat is a notoriously hard problem for which various approximations and exact (exponential time) algorithms have been developed; see [32, 61] for an overview. We evaluate Bounce and the other algorithms on the 125-dimensional ClusterExpansion benchmark, a real-world MaxSAT instance with many applications in materials science [2]. Unlike the MaxSAT60 benchmark (see Appendix B.1.3), ClusterExpansion is not a crafted benchmark, and its optimum has no synthetic structure [1, 3]. We treat the MaxSat problems as black-box problems; hence, algorithms do not have access to the clauses, and we cannot use the usual algorithms.

Figure 3 shows the total negative weight of the satisfied clauses as a function of evaluations. We cannot plot regret curves since the optimum is unknown [4]. We observe that Bounce finds better solutions than all other algorithms. BODi is the only algorithm for which we observe sensitivity to

0

100

200

Iteration

12

14

16

18

Objective value

25D-PestControl

0

100

200

Iteration

25D-PestControl (publ.)

Bounce

BODi

Random Search

Casmopolitan

COMBO

COMBO (fixed)

RDUCB

SMAC

0

100

200

Iteration

2 ×

10

1

3 ×

10

1

4 ×

10

1

Objective value

53D-SVM

0

100

200

Iteration

53D-SVM (publ.)

Bounce

BODi

Random Search

Casmopolitan

RDUCB

SMAC

the location of the optimal assignment: for the published version of the benchmark, BODi quickly jumps to a moderately good solution but fails to make further progress.

4.3 25D Categorical Pest Control

PestControl is a more complex version of the Contamination benchmark and has 25 categorical variables with five categories each [56]. The task is to select one out of five actions { 1 , 2 , . . . , 5 } at each of 25 stations to minimize the objective function that combines total cost and a measure of the spread of the pest. We note the setting x = (5 , 5 , . . . , 5) achieves a good value of 12 . 57 , while the best value found in our evaluation is 12 . 07 is x = (5 , 5 , . . . , 5 , 1) and thus has a Hamming distance of one. The random seed used in our experiments is zero. Figure 4 summarizes the performances of the algorithms. Bounce is robust to the location of the global optimum and consistently obtains the best solutions. In particular, the performances of COMBO and BODi depend on whether the optimum has a certain structure. We discuss this issue in detail in Appendix H.1.

4.4 SVM - a 53D AutoML task

In the SVM benchmark, we optimize over a mixed space with 50 binary and 3 continuous parameters to tune an ε -support vector regression (SVR) model [72]. The 50 binary parameters determine whether to include or exclude an input feature from the dataset. The 3 continuous parameters correspond to the regularization parameter C , the kernel width γ , and the ε parameter of the ε -SVR model [72]. Its root mean squared error on a held-out dataset gives the function value. Figure 5 summarizes the performances of the algorithms. We observe that Bounce , BODi , and Casmopolitan achieve comparable solutions. BODi performs slightly worse if the ordering of the categories is shuffled and slightly better if the optimal assignment to all binary variables is one. COMBO does not support continuous variables and thus was omitted.

4.5 Bounce 's efficacy for batch acquisition

We study the sample efficiency of Bounce when it selects a batch of B points in each iteration to evaluate in parallel. Figure 6 shows the results for B = 1 , 3 , 5 , 10 , and 20 , where Bounce was run for min(2000 , 200 · B ) function evaluations. We configure Bounce to reach the input dimensionality after 100 evaluations for B = 1 , 3 , 5 and after 25 B for B = 10 , 20 . We observe that Bounce leverages parallel function evaluations effectively: it obtains a comparable function value at a considerably smaller number of iterations, thus saving wall-clock time for applications with time-consuming function evaluations. We also studied batch acquisition for continuous problems

0

100

200

12

14

16

18

Objective value

25D-PestControl

0

100

200

200

150

100

50

60D-MaxSAT60

0

100

200

21.5

22.0

22.5

23.0

25D-Contamination

0

100

200

# batch evaluations

0.2

0.3

0.4

0.5

Objective value

53D-SVM

0

100

200

# batch evaluations

0

1

2

53D-Ackley53

0

100

200

# batch evaluations

4

3

2

1

50D-LABS

Bounce, b=1

Bounce, b=3

Bounce, b=5

Bounce, b=10

Bounce, b=20

and found that Bounce also provides significant speed-up. Due to space constraints, we deferred the discussion to Appendix C.

4.6 The sensitivity of BODi and COMBO to the location of the optima

The empirical evaluation reveals that the performances of BODi [21] and COMBO [56] are sensitive to the location of the optima. Both methods degrade on at least one benchmark when the optimum is moved to a randomly chosen point. This is particularly unexpected for categorical variables where moving the optimum to a random location is equivalent to shuffling the labels of the categories of each variable. Such a change of representation should not affect the performance of an algorithm.

BODi is more susceptible to the location of the optimizer than COMBO . The performance of COMBO degrades only on the categorical PestControl benchmark, whereas BODi degrades on five out of seven benchmarks. Looking closer, we observe that BODi 's performance degradation is particularly large for synthetic benchmarks like Ackley53 and MaxSAT60 , where setting all variables to the same value is optimal. Figure 7 summarizes the effects of moving the optimum on BODi . Due to space constraints, we moved the details and a discussion of categorical variables to the appendix. Similarly, setting all binary variables of the SVM benchmark to one produces a good objective value. It is not surprising, given that the all-one assignment corresponds to including all features previously selected for the benchmark because of their high importance.

We show in Appendix H.1 that BODi adds a point to its dictionary with zero Hamming distance to an all-zero or all-one solution, with a probability that increases with the dictionary size. Deshwal et al. [21, p. 7] reported that BODi 's performance 'tends to improve' with the size of the dictionary. Moreover, BODi samples a new dictionary in each iteration, eventually increasing the chance of having such a point in its dictionary. Thus, we hypothesize that BODi benefits from having a nearoptimal solution in its dictionary, likely for all-zero or all-one solutions. For COMBO , Figure 4 shows that the performance on PestControl degrades substantially if the labels of the categories are shuffled. Then COMBO 's sample-efficiency becomes comparable to Random Search .

5 Discussion

BO in combinatorial spaces has many exciting and impactful applications. Its applicability to realworld problems, such as LABS that defy a closed-form solution, makes it a valuable tool for practitioners. Our empirical evaluation reveals that state-of-the-art methods fail to provide good solutions reliably. In particular, it finds that BODi and COMBO , which performed best in recent publications, are sensitive to the location of the optimizer. We identified design flaws in BODi and an implementation bug in COMBO as the root causes of the performance degradations.

The proposed Bounce algorithm is reliable for high-dimensional black-box optimization in combinatorial, continuous, and mixed spaces. The empirical evaluation demonstrates that Bounce reliably outperforms the state-of-the-art on a diverse set of problems. Using a novel TR management strategy, Bounce leverages parallel evaluations of the objective function to improve its performance. We

0

100

200

14

16

18

Objective value

25D-PestControl

0

100

200

3

2

1

50D-LABS

0

100

200

0.2

0.3

0.4

0.5

53D-SVM

0

100

200

1

2

3

53D-Ackley53

0

100

200

Iteration

200

100

0

Objective value

60D-MaxSAT60

0

100

200

Iteration

21.5

22.0

22.5

23.0

25D-Contamination

0

100

200

Iteration

2.78

2.76

2.74

2.72

2.70

1e14

125D-ClusterExpansion

BODi (rand.)

BODi

anticipate headroom by tailoring the modeling of combinatorial objects, e.g., arising in the search for peptides or materials discovery [28, 34, 37, 59, 75, 78]. Here it seems particularly interesting to incorporate prior belief on the importance of decision variables while maintaining the overall scalability. Moreover, extending the present work to black-box constraints [24, 30], multiple objectives, and multiple information sources [18, 31, 62] will considerably expand the applicable use cases.

Limitations. Bounce is not designed to handle noisy evaluations of the objective function. While it seems straightforward to extend Bounce to handle noisy evaluations, e.g., by using a Gaussian process with a noise term and acquisition functions that account for noise [70], we leave this for future work. Moreover, in applications where the categorical or ordinal variables vary substantially in the number of values they can take, there may be better ways to 'bin' them.

Societal impact. Bayesian optimization has recently gained wide-spread popularity for tasks in drug discovery [53], chemical engineering [15, 33, 38, 68, 71], materials discovery [28, 34, 37, 59, 75, 78], aerospace engineering [6, 45, 49], robotics [16, 17, 48, 50, 64], and many more. This highlights the Bayesian optimization community's progress toward providing a reliable 'off-theshelf optimizer.' However, this promise is not yet fulfilled for the newer domain of mixed-variable Bayesian optimization that allows optimization over hundreds of 'tunable levers', some of which are discrete, while others are continuous. This domain is of particular relevance for the tasks above. Bounce 's ability to incorporate more such levers in the optimization significantly impacts the above practical applications, allowing for more granular control of a chemical reaction or a processing path, to give some examples. The empirical evaluation shows that the performance of state-of-theart methods is highly sensitive to the location of the unknown global optima and often degenerates drastically, thus putting practitioners at risk. The proposed algorithm Bounce , however, achieves robust performance over a broad collection of tasks and thus will become a 'goto' optimizer for practitioners in other fields. Therefore, we open-source the Bounce code. 1

Acknowledgments and Disclosure of Funding

Leonard Papenmeier and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Luigi Nardi was partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348 and by affiliate members and other supporters of the Stanford DAWN project Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at the Chalmers Centre for Computational Science and Engineering (C3SE) and the National Supercomputer Centre at Linköping University, partially funded by the Swedish Research Council through grant agreement no. 2022-06725.

References

Webpage of the Fourth Max-SAT Evaluation. http://www.maxsat.udl.cat/09/index. php?disp=submitted-benchmarks . Last access: 2023-05-02.

MaxSAT Evaluation 2018 : Solver and Benchmark Descriptions. 2018. URL http://hdl. handle.net/10138/237139 . Publisher: Department of Computer Science, University of Helsinki.

André Abramé and Djamal Habet. AHMAXSAT: Description and evaluation of a branch and bound Max-SAT solver. Journal on Satisfiability, Boolean Modeling and Computation , 9(1): 89-128, 2014.

Fahiem Bacchus, Matti Järvisalo, and Ruben Martins. MaxSAT Evaluation 2018: New developments and detailed results. Journal on Satisfiability, Boolean Modeling and Computation , 11(1):99-131, 2019.

Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, 2020.

Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures. In International Conference on Machine Learning , pages 462-471. PMLR, 2018.

Petr Baudiš and Petr Pošík. Online Black-Box Algorithm Portfolios for Continuous Optimization. In Parallel Problem Solving from Nature - PPSN XIII , pages 40-49, Cham, 2014. Springer International Publishing.

James Bergstra and Yoshua Bengio. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research , 13(2), 2012.

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for HyperParameter Optimization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 24, 2011.

Mickael Binois and Nathan Wycoff. A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization , 2(2):1-26, 2022.

Mickaël Binois, David Ginsbourger, and Olivier Roustant. A warped kernel improving robustness in Bayesian optimization via random embeddings. In International Conference on Learning and Intelligent Optimization (LION) , pages 281-286. Springer, 2015.

Mickaël Binois, David Ginsbourger, and Olivier Roustant. On the choice of the lowdimensional domain for global optimization via random embeddings. Journal of global optimization , 76(1):69-90, 2020.

Jim Boelrijk, Bernd Ensing, Patrick Forré, and Bob WJ Pirok. Closed-loop automatic gradient design for liquid chromatography using Bayesian optimization. Analytica Chimica Acta , 1242, 2023.

Mohamed Amine Bouhlel, Nathalie Bartoli, Rommel G. Regis, Abdelkader Otsmane, and Joseph Morlier. Efficient global optimization for high-dimensional constrained problems by using the Kriging models combined with the partial least squares method. Engineering Optimization , 50(12):2038-2053, 2018.

Benjamin Burger, Phillip M Maffettone, Vladimir V Gusev, Catherine M Aitchison, Yang Bai, Xiaoyan Wang, Xiaobo Li, Ben M Alston, Buyi Li, Rob Clowes, et al. A mobile robotic chemist. Nature , 583(7815):237-241, 2020.

Roberto Calandra, Nakul Gopalan, André Seyfarth, Jan Peters, and Marc Peter Deisenroth. Bayesian Gait Optimization for Bipedal Locomotion. In Learning and Intelligent Optimization , pages 274-290. Springer International Publishing, 2014.

Roberto Calandra, André Seyfarth, Jan Peters, and Marc Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence , 76 (1):5-23, 2016.

Zachary Cosenza, Raul Astudillo, Peter Frazier, Keith Baar, and David E Block. MultiInformation Source Bayesian Optimization of Culture Media for Cellular Agriculture. Biotechnology and Bioengineering , 2022.

Samuel Daulton, Xingchen Wan, David Eriksson, Maximilian Balandat, Michael A. Osborne, and Eytan Bakshy. Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, 2022.

Aryan Deshwal, Syrine Belakaria, Janardhan Rao Doppa, and Dae Hyun Kim. Bayesian optimization over permutation spaces. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6515-6523, 2022.

Aryan Deshwal, Sebastian Ament, Maximilian Balandat, Eytan Bakshy, Janardhan Rao Doppa, and David Eriksson. Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings. In International Conference on Artificial Intelligence and Statistics , pages 7021-7039. PMLR, 2023.

Adel Ejjeh, Leon Medvinsky, Aaron Councilman, Hemang Nehra, Suraj Sharma, Vikram Adve, Luigi Nardi, Eriko Nurvitadhi, and Rob A. Rutenbar. HPVM2FPGA: Enabling True Hardware-Agnostic FPGA Programming. In Proceedings of the 33rd IEEE International Conference on Application-specific Systems, Architectures, and Processors , 2022.

David Eriksson and Martin Jankowiak. High-dimensional Bayesian optimization with sparse axis-aligned subspaces. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , volume 161 of Proceedings of Machine Learning Research , pages 493-503. PMLR, 27-30 Jul 2021.

David Eriksson and Matthias Poloczek. Scalable Constrained Bayesian Optimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 730-738. PMLR, 13-15 Apr 2021.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable Global Optimization via Local Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, pages 5496-5507, 2019.

CA Floudas, HK Fung, SR McAllister, M Mönnigmann, and R Rajgaria. Advances in protein structure prediction and de novo protein design: A review. Chemical Engineering Science , 61 (3):966-988, 2006.

Peter I Frazier. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

Peter I. Frazier and Jialei Wang. Bayesian Optimization for Materials Design , pages 45-75. Springer International Publishing, Cham, 2016. ISBN 978-3-319-23871-5.

Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Seyede Fatemeh Ghoreishi and Douglas Allaire. Multi-information source constrained Bayesian optimization. Structural and Multidisciplinary Optimization , 59:977-991, 2019.

Seyede Fatemeh Ghoreishi and Douglas L Allaire. A fusion-based multi-information source optimization approach using knowledge gradient policies. In 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference , page 1159, 2018.

Pierre Hansen and Brigitte Jaumard. Algorithms for the maximum satisfiability problem. Computing , 44(4):279-303, 1990.

Florian Hase, Loïc M Roch, Christoph Kreisbeck, and Alán Aspuru-Guzik. Phoenics: a Bayesian optimizer for chemistry. ACS central science , 4(9):1134-1145, 2018.

Florian Häse, Matteo Aldeghi, Riley J Hickman, Loïc M Roch, and Alán Aspuru-Guzik. Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge. Applied Physics Reviews , 8(3):031406, 2021.

Erik Hellsten, Carl Hvarfner, Leonard Papenmeier, and Luigi Nardi. High-dimensional Bayesian Optimization with Group Testing. arXiv preprint arXiv:2310.03515 , 2023.

Erik Hellsten, Artur Souza, Johannes Lenfers, Rubens Lacouture, Olivia Hsu, Adel Ejjeh, Fredrik Kjolstad, Michel Steuwer, Kunle Olukotun, and Luigi Nardi. BaCO: A Fast and Portable Bayesian Compiler Optimization Framework. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems , 2023.

Henry C Herbol, Weici Hu, Peter Frazier, Paulette Clancy, and Matthias Poloczek. Efficient search of compositional space for hybrid organic-inorganic perovskites via Bayesian optimization. npj Computational Materials , 4(1):1-7, 2018.

José Miguel Hernández-Lobato, James Requeima, Edward O. Pyzer-Knapp, and Alán AspuruGuzik. Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space. In Proceedings of the 34th International Conference on Machine Learning , volume 70, pages 1470-1479. PMLR, 06-11 Aug 2017.

Yingjie Hu, Jian-Qiang Hu, Yifan Xu, Fengchun Wang, and Rong Zeng Cao. Contamination control in food supply chain. In Proceedings of the 2010 Winter Simulation Conference , pages 2678-2681. IEEE, 2010.

Zak E Hughes, Michelle A Nguyen, Jialei Wang, Yang Liu, Mark T Swihart, Matthias Poloczek, Peter I Frazier, Marc R Knecht, and Tiffany R Walsh. Tuning Materials-Binding Peptide Sequences toward Gold-and Silver-Binding Selectivity with Bayesian Optimization. ACS nano , 15(11):18260-18269, 2021.

Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential Model-Based Optimization for General Algorithm Configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization , pages 507-523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-25566-3.

Anant Singh Jain and Sheik Meeran. Deterministic job-shop scheduling: Past, present and future. European journal of operational research , 113(2):390-434, 1999.

Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization , 13(4):455, 1998.

Jungtaek Kim, Seungjin Choi, and Minsu Cho. Combinatorial Bayesian optimization with random mapping functions to convex polytopes. In Uncertainty in Artificial Intelligence , pages 1001-1011. PMLR, 2022.

Rémi Lam, Matthias Poloczek, Peter Frazier, and Karen E Willcox. Advances in Bayesian optimization with applications in aerospace engineering. In 2018 AIAA Non-Deterministic Approaches Conference , page 1656, 2018.

Ben Letham, Roberto Calandra, Akshara Rai, and Eytan Bakshy. Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 1546-1558, 2020.

Chong Liu and Yu-Xiang Wang. Global optimization with parametric function approximation. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 22113-22136, 2023.

Daniel J Lizotte, Tao Wang, Michael H Bowling, Dale Schuurmans, et al. Automatic Gait Optimization With Gaussian Process Regression. In IJCAI , volume 7, pages 944-949, 2007.

Trent W Lukaczyk, Paul Constantine, Francisco Palacios, and Juan J Alonso. Active subspaces for shape optimization. In 10th AIAA multidisciplinary design optimization conference , page 1171, 2014.

Matthias Mayr, Faseeh Ahmad, Konstantinos I. Chatzilygeroudis, Luigi Nardi, and Volker Krüger. Skill-based Multi-objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration. CoRR , abs/2203.10033, 2022.

Luigi Nardi, David Koeplinger, and Kunle Olukotun. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) , pages 347-358, 2019.

Amin Nayebi, Alexander Munteanu, and Matthias Poloczek. A framework for Bayesian Optimization in Embedded Subspaces. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research (PMLR) , pages 4752-4761, 09-15 Jun 2019.

Diana M. Negoescu, Peter I. Frazier, and Warren B. Powell. The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery. INFORMS Journal on Computing , 23 (3):346-363, 2011.

Kai Wang Ng, Guo-Liang Tian, and Man-Lai Tang. Dirichlet and related distributions: Theory, methods and applications. 2011.

Changhai Nie and Hareton Leung. A survey of combinatorial testing. ACMComputing Surveys (CSUR) , 43(2):1-29, 2011.

Changyong Oh, Jakub Tomczak, Efstratios Gavves, and Max Welling. Combinatorial Bayesian Optimization using the Graph Cartesian Product. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019.

Steve O'Hagan, Warwick B Dunn, Marie Brown, Joshua D Knowles, and Douglas B Kell. Closed-loop, multiobjective optimization of analytical instrumentation: gas chromatography/time-of-flight mass spectrometry of the metabolomes of human serum and of yeast fermentations. Analytical Chemistry , 77(1):290-303, 2005.

Tom Packebusch and Stephan Mertens. Low autocorrelation binary sequences. Journal of Physics A: Mathematical and Theoretical , 49(16):165001, 2016.

Daniel Packwood. Bayesian Optimization for Materials Science . Springer, 2017.

Leonard Papenmeier, Luigi Nardi, and Matthias Poloczek. Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, 2022.

Matthias Poloczek and David P Williamson. An experimental evaluation of fast approximation algorithms for the maximum satisfiability problem. Journal of Experimental Algorithmics (JEA) , 22:1-18, 2017.

Matthias Poloczek, Jialei Wang, and Peter Frazier. Multi-information source optimization. Advances in neural information processing systems , 30, 2017.

Warren B Powell. A unified framework for stochastic optimization. European Journal of Operational Research , 275(3):795-821, 2019.

Akshara Rai, Rika Antonova, Seungmoon Song, William Martin, Hartmut Geyer, and Christopher Atkeson. Bayesian optimization using domain knowledge on the ATRIAS biped. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1771-1778, 2018.

Rommel G Regis and Christine A Shoemaker. Combining radial basis function surrogates and dynamic coordinate search in high-dimensional expensive black-box optimization. Engineering Optimization , 45(5):529-555, 2013.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

Binxin Ru, Ahsan Alvi, Vu Nguyen, Michael A Osborne, and Stephen Roberts. Bayesian optimisation over multiple continuous and categorical inputs. In International Conference on Machine Learning , pages 8276-8285. PMLR, 2020.

Artur M Schweidtmann, Adam D Clayton, Nicholas Holmes, Eric Bradford, Richard A Bourne, and Alexei A Lapkin. Machine learning meets continuous flow chemistry: Automated optimization towards the Pareto front of multiple objectives. Chemical Engineering Journal , 352: 277-282, 2018.

Kenan Šehi´ c, Alexandre Gramfort, Joseph Salmon, and Luigi Nardi. LassoBench: A HighDimensional Hyperparameter Optimization Benchmark Suite for Lasso. In First Conference on Automated Machine Learning (Main Track) , 2022.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 104 (1):148-175, 2015.

Benjamin J Shields, Jason Stevens, Jun Li, Marvin Parasram, Farhan Damani, Jesus I Martinez Alvarado, Jacob M Janey, Ryan P Adams, and Abigail G Doyle. Bayesian reaction optimization as a tool for chemical synthesis. Nature , 590(7844):89-96, 2021.

Alex J Smola and Bernhard Schölkopf. A tutorial on support vector regression. Statistics and computing , 14:199-222, 2004.

Francisco J. Solis and Roger J-B. Wets. Minimization by Random Search Techniques. Mathematics of Operations Research , 6(1):19-30, 1981.

Lei Song, Ke Xue, Xiaobin Huang, and Chao Qian. Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization. Advances in Neural Information Processing Systems (NeurIPS) , 35, 2022.

Artur Souza, Leonardo B Oliveira, Sabine Hollatz, Matt Feldman, Kunle Olukotun, James M Holton, Aina E Cohen, and Luigi Nardi. DeepFreak: Learning crystallography diffraction patterns with automated machine learning. arXiv preprint arXiv:1904.11834 , 2019.

Hampus Gummesson Svensson, Esben Jannik Bjerrum, Christian Tyrchan, Ola Engkvist, and Morteza Haghir Chehreghani. Autonomous drug design with multi-armed bandits. In 2022 IEEE International Conference on Big Data (Big Data) , pages 5584-5592. IEEE, 2022.

Alexander Thebelt, Calvin Tsay, Robert Lee, Nathan Sudermann-Merx, David Walz, Behrang Shafei, and Ruth Misener. Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces. Advances in Neural Information Processing Systems , 35: 37401-37415, 2022.

Tsuyoshi Ueno, Trevor David Rhone, Zhufeng Hou, Teruyasu Mizoguchi, and Koji Tsuda. COMBO: An efficient Bayesian optimization library for materials science. Materials Discovery , 4:18-21, 2016.

Xingchen Wan, Vu Nguyen, Huong Ha, Binxin Ru, Cong Lu, and Michael A. Osborne. Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10663-10674. PMLR, 1824 Jul 2021.

Jialei Wang, Scott C Clark, Eric Liu, and Peter I Frazier. Parallel Bayesian global optimization of expensive functions. Operations Research , 68(6):1850-1865, 2020.

Ke Wang and Alexander W Dowling. Bayesian optimization for chemical products and functional materials. Current Opinion in Chemical Engineering , 36:100728, 2022.

Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search. Advances in Neural Information Processing Systems (NeurIPS) , 33:19511-19522, 2020.

Xilu Wang, Yaochu Jin, Sebastian Schmitt, and Markus Olhofer. Recent Advances in Bayesian Optimization. ACM Comput. Surv. , 2023.

Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando de Feitas. Bayesian Optimization in a Billion Dimensions via Random Embeddings. Journal of Artificial Intelligence Research (JAIR) , 55:361-387, 2016.

Christopher K Williams and Carl Edward Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparameterization trick for acquisition functions. NeurIPS Workshop on Bayesian Optimization , 2017.

David P Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends® in Theoretical Computer Science , 10(1-2):1-157, 2014.

Juliusz Krzysztof Ziomek and Haitham Bou Ammar. Are Random Decompositions all we need in High Dimensional Bayesian Optimisation? In International Conference on Machine Learning , pages 43347-43368. PMLR, 2023.

A Consistency of Bounce

In this section, we prove the consistency of the Bounce algorithm. The proof is based on Papenmeier et al. [60] and Eriksson and Poloczek [24].

Theorem 1 ( Bounce consistency) . With the following definitions

Def. 1. ( x k ) ∞ k =1 is a sequence of points of decreasing function values;

Def. 2. x ∗ ∈ arg min x ∈X is a minimizer of f in X ;

and under the following assumptions:

Ass. 1. D is finite;

Ass. 2. f is observed without noise;

Ass. 3. The range of f is bounded in X , i.e., ∃ C ∈ R ++ s.t. | f ( x ) | < C ∀ x ∈ X ;

Ass. 4. For at least one of the minimizers x ∗ i the (partial) assignment corresponding to the continuous variables lies in a (continuous) region with positive measure;

Ass. 5. One Bounce reached the input dimensionality D , the continuous elements of the initial points { x cont i } n init n =1 after each TR restart are chosen

(a) uniformly at random for continuous variables; and

(b) such that every realization of the combinatorial variables has positive probability;

then the Bounce algorithm finds a global optimum with probability 1, as the number of samples N goes to ∞ .

Proof. The range of f is bounded per Assumption 3 , and Bounce only considers a function evaluation a 'success' if the improvement over the current best solution exceeds a certain constant threshold. Bounce can only have a finite number of 'successful' evaluations because the range of f is bounded per Assumption 3 . For the sake of a contradiction, we suppose that Bounce does not obtain an optimal solution as its number of function evaluations N →∞ . Thus, there must be a sequence of failures, such that the TRs in the current target space, i.e., the current subspace, will eventually reach its minimum base length. Recall that in such an event, Bounce increases the target dimension by splitting up the 'bins', thus creating a subspace of ( b + 1) -times higher dimensionality. Then Bounce creates a new TR that again experiences a sequence of failures that lead to another split, and so on. This series of events repeats until the embedded subspace eventually equals the input space and thus has dimensionality D . See lines 12 -16 in Algorithm 1 in Sect. 3.

Still supposing that Bounce does not find an optimum in the input space, there must be a sequence of failures such that the side length of the TR again falls below the set minimum base length, now forcing a restart of Bounce . Recall that at every restart, Bounce samples a fresh set of initial points uniformly at random from the input space; see line 18 in Algorithm 1. Therefore, with probability 1, a random sample will eventually be drawn from any subset Y ⊆ X with positive Lebesgue measure ( ν ( Y ) > 0 ):

where µ is the uniform probability measure of the sampling distribution that Bounce employs for initial data points upon restart [73].

Let

denote the essential infimum of f on X with ν being the Lebesgue measure [73].

Following Solis and Wets [73], we define the optimality region, i.e., the set of points whose function value is larger by at most ε than the essential infimum:

with ε > 0 and M < 0 . Because of Ass. 4, at least one optimal point lies in a region of positive measure that is continuous for the continuous variables. Therefore, we have that α = f ( x ∗ ) . Note that this is also the case if the domain of f only consists of combinatorial variables (Ass. 5). Then, R ε,M = { x ∈ X | f ( x ) < f ( x ∗ ) + ε } .

and,

i.e., x ⋆ k eventually falls into the optimality region [73]. By letting ε → 0 , x ⋆ k converges to the global optimum with probability 1 as k →∞ .

B Additional experiments

We compare Bounce to the other algorithms on three additional benchmark problems: Ackley53 and MaxSAT60 [21]. Moreover, we run two additional studies to investigate the performance of Bounce further. First, we run Bounce on a set of continuous problems from Papenmeier et al. [60] to showcase the performance and scalability of Bounce on purely continuous problems. We then present a 'low-sequency' version of Bounce to showcase how such a version can outperform its competitors on the original benchmarks by introducing a bias towards low-sequency solutions.

B.1 Bounce and other algorithms on additional benchmarks

B.1.1 The synthetic Ackley53 benchmark function

Ackley53 is a 53-dimensional function with 50 binary and three continuous variables. Wan et al. [79] discretized 50 continuous variables of the original Ackley function, requiring these variables to be either zero or one. This benchmark was designed such that the optimal value of 0 . 0 is at the origin x = (0 , . . . , 0) . Here, we perturb the optimal assignment of combinatorial variables by flipping each binary variable with probability 1 / 2 . Figure 8 summarizes the performances of the algorithms. Bounce outperforms all other algorithms and proves to be robust to the location of the optimum point. Casmopolitan is a distanced runner-up. BODi initially outperforms Casmopolitan on the published benchmark version but falls behind later.

B.1.2 Contamination control

The Contamination benchmark models a supply chain with 25 stages [39]. At each stage, a binary decision is made whether to quarantine food that has not yet been contaminated. Each such intervention is costly, and the goal is to minimize the number of contaminated products and prevention cost [6, 56]. Figure 9 shows the performances of the algorithms.

Bounce , Casmopolitan , and BODi all produce solutions of comparable objective value. Bounce and Casmopolitan find better solutions than BODi initially, but after about 100 function evaluations, the solutions obtained by the three algorithms are typically on par.

0

100

200

Iteration

10

1

10 0

Objective value

53D-Ackley53

0

100

200

Iteration

53D-Ackley53 (publ.)

Bounce

BODi

Random Search

Casmopolitan

RDUCB

SMAC

Let ( x ⋆ k ) ∞ k =1 denote the sequence of best points that Bounce discovers with x ⋆ k being the best point up to iteration k . This sequence satisfies Def. 1 by construction. Note that x ⋆ k ∈ R ε,M implies that x ⋆ k ′ ∈ R ε,M for all k ′ ≥ k +1 [73] because observations are noise-free. Then,

0

100

200

Iteration

21.50

21.75

22.00

22.25

22.50

Objective value

25D-Contamination

0

100

200

Iteration

25D-Contamination (publ.)

Bounce

BODi

Random Search

Casmopolitan

RDUCB

SMAC

0

100

200

Iteration

200

150

100

50

Objective value

60D-MaxSAT60

0

100

200

Iteration

60D-MaxSAT60 (publ.)

Bounce

BODi

Random Search

Casmopolitan

COMBO

RDUCB

SMAC

B.1.3 The MaxSAT60 benchmark

MaxSAT60 is a 60-dimensional, weighted instance of the Maximum Satisfiability (MaxSAT) problem. MaxSAT is a notoriously hard combinatorial problem that cannot be solved in polynomial time (unless P = NP ). The goal is to find a binary assignment to the variables that satisfies clauses of maximum total weight. For every i in { 1 , 2 , . . . , d } , this benchmark has one clause of the form x i with a weight of 1 and 638 clauses of the form ¬ x i ∨ ¬ x j with a weight of 61. Following [21, 56, 79], we normalize these weights to have zero mean and unit standard deviation. This normalization causes the one-variable clauses to have a negative weight , i.e., the function value improves if such a clause is not satisfied, which is atypical behavior for a MaxSAT problem. Since the clauses with two variables are satisfied for x i = x j = 0 and the clauses with one variable of negative weights are never satisfied for x i = 0 , the normalized benchmark version has a global optimum at x ∗ = (0 , . . . , 0) by construction. The problem's difficulty is finding an assignment for variables such that all two-variable clauses are satisfied and as many one-variable clauses as possible are not captured by normalized weights.

Figure 10 summarizes the performances of the algorithms. The general version that attains the global optimum for a randomly selected binary assignment is shown on the left. The special case where the global optimum is set to the all-zero assignment is shown on the right.

We observe that Bounce requires the smallest number of samples to find an optimal assignment in general, followed by BODi and Casmopolitan . Only in the special case where the optimum is the all-zero assignment, BODi ranks first, confirming the corresponding result in Deshwal et al. [21].

C An evaluation of Bounce on continuous problems

0

10 0

10 1

Objective value / Regret

500D-Branin2

220

240

260

280

300

124D-Mopta08

0

10 0

500D-Hartmann6

0

200

400

# batch evaluations

0.05

0.10

0.15

0.20

Objective value / Regret

388D-SVM

0

200

400

# batch evaluations

0.285

0.290

0.295

0.300

0.305

0.310

180D-LassoDNA

0

200

400

# batch evaluations

2

4

6

8

10

1000D-LassoHard

BAxUS, b=1

Bounce, b=1

Bounce, b=3

Bounce, b=10

Bounce, b=20

0

1000

2000

0

10 0

Objective value / Regret

500D-Branin2

0

1000

2000

220

240

260

280

300

124D-Mopta08

0

1000

2000

0

10 0

500D-Hartmann6

0

1000

2000

# function evaluations

0.05

0.10

0.15

0.20

Objective value / Regret

388D-SVM

0

1000

2000

# function evaluations

0.29

0.30

0.31

180D-LassoDNA

0

1000

2000

# function evaluations

2

4

6

8

10

1000D-LassoHard

Bounce, b=1

Bounce, b=3

Bounce, b=10

Bounce, b=20

To showcase the performance and scalability of Bounce , we run it on a set of continuous problems from Papenmeier et al. [60]. The 124-dimensional Mopta08 benchmark is a constrained vehicle optimization problem. We adopt the soft-constrained version from Eriksson and Jankowiak [23]. The 388-dimensional SVM problem [23] concerns the classification performance with an SVR on the slice localization dataset. The 180-dimensional LassoDNA benchmark [69] is a sparse regression problem on a real-world dataset, and the 1000-dimensional LassoHard benchmark optimizes over a synthetic dataset. The 500-dimensional Branin2 and Hartmann6 problems are versions of the 2and 6-dimensional benchmark problems where additional dimensions with no effect on the function value were added.

We set the number of function evaluations to max(2000 , 500 B ) for a batch size of B and configure Bounce such that it reaches the input dimensionality after 500 function evaluations. Figures 11 and 12 show the simple regret for the synthetic Branin2 and Hartmann6 problems, and the best function value obtained after a given number of batch (Figure 11) or function (Figure 12) evaluations for the remaining problems: Mopta08 , SVM , LassoDNA , and LassoHard .

We observe that Bounce always benefits from more parallel function evaluations. The difference between smaller batch sizes, such as B = 1 and B = 3 or B = 3 and B = 10 , is more remarkable than between larger batch sizes, like B = 10 and B = 20 . Parallel function evaluations prove

especially effective on SVM and LassoDNA . Here, the optimization performance improves drastically. We conclude that a small number of parallel function evaluations already helps to considerably increase the optimization performance.

On the synthetic Branin2 and Hartmann6 problems, Bounce quickly converges to the global optimum. Here, we see that a larger number of parallel function evaluations also helps in converging to a better solution.

In Figure 11, we also compare to BAxUS by Papenmeier et al. [60], which does not support parallel function evaluations and is therefore run with a batch size of 1. We observe that Bounce with a batch size of 1 outperforms BAxUS on all benchmarks except for Hartmann6 , showcasing Bounce 's overall performance improvements even for continuous problems and single-element batches.

D Batched evaluations on mixed and combinatorial problems

In addition to Figure 6, which shows the best objective value in relation to the number of batches, Figure 13 shows the objective value in terms of function evaluations.

0

1000

2000

12

14

16

18

Objective value

25D-PestControl

0

1000

2000

200

150

100

50

60D-MaxSAT60

0

1000

2000

21.5

22.0

22.5

23.0

25D-Contamination

0

1000

2000

# function evaluations

2 ×

10

1

3 ×

10

1

4 ×

10

1

Objective value

53D-SVM

0

1000

2000

# function evaluations

0

1

2

53D-Ackley53

0

1000

2000

# function evaluations

4

3

2

1

50D-LABS

Bounce, b=1

Bounce, b=3

Bounce, b=5

Bounce, b=10

Bounce, b=20

The figure confirms that Bounce leverages parallel function values efficiently.

E Low-sequency version of Bounce

We show how we can bias Bounce towards low-sequency solutions. A binary vector has low sequency if there are few changes from 0 to 1 or vice versa. Similarly, a solution to a categorical or ordinal problem has low sequency if there are few changes from one category or level to another. We remove the random signs (for binary and continuous variables) and the random offsets (for categorical and ordinal variables) from the Bounce embedding. We conduct this study to show a) that Bounce can outperform BODi on the unmodified versions of the benchmark problems if we introduce a similar bias towards low-sequency solutions and b) that the random signs empirically show to remove biases towards low-sequency solutions. However, we want to emphasize that the results of this section are not representative of the performance of Bounce on arbitrary real-world problems. Nevertheless, if one knows that the problem has a low-sequency structure, then Bounce can be configured to exploit this structure and outperform BODi .

Figure 14 shows the results of the low-sequency version of Bounce on the original benchmarks from Section 4. We observe that Bounce outperforms BODi and the other algorithms on the unmodified versions of the benchmark problems. This shows that Bounce can outperform BODi on the unmodified version of the benchmarks if we introduce a similar bias towards low-sequency solutions.

Figure 15 shows the results of the low-sequency version of Bounce on the flipped benchmarks from Section 4. The low-sequency version of Bounce is robust towards the randomization of the optimal point.

0

100

200

2 ×

10

1

3 ×

10

1

4 ×

10

1

6 ×

10

1

Objective value

53D-SVM

0

100

200

10

1

10 0

53D-Ackley53

0

100

200

200

100

0

60D-MaxSAT60

0

100

200

Iteration

12

14

16

18

Objective value

25D-PestControl

0

100

200

Iteration

21.5

22.0

22.5

23.0

23.5

25D-Contamination

0

100

200

Iteration

4

3

2

1

50D-LABS

Bounce

BODi

Casmopolitan

COMBO

0

100

200

2 ×

10

1

3 ×

10

1

4 ×

10

1

6 ×

10

1

Objective value

53D-SVM

0

100

200

10

1

10 0

53D-Ackley53

0

100

200

200

150

100

50

60D-MaxSAT60

0

100

200

Iteration

12

14

16

18

Objective value

25D-PestControl

0

100

200

Iteration

21.5

22.0

22.5

23.0

23.5

25D-Contamination

0

100

200

Iteration

4

3

2

1

50D-LABS

Bounce

BODi

Casmopolitan

COMBO

F Effect of trust-region management

Bounce uses a novel trust-region management that differs from previous approaches in that it allows arbitrary trust region base lengths in [ L min , L max ] . This strategy allows Bounce to efficiently leverage parallel function evaluations, which we refer to as batch acquisition.

We compare Bounce 's trust-region management strategy with the strategy employed by BAxUS [60] for purely continuous benchmarks and an adapted version of Casmopolitan 's [79] strategy for mixed or discrete-space benchmarks. For the latter, the difference is that in Bounce we reduce the failure tolerance as described by Papenmeier et al. [60]: We first calculate the number of times the TR base length needs to be reduced to reach the minimum TR base length as k = ⌊ log 1 . 5 -1 L min L init ⌋ , and then find the failure tolerance for the i -th target space by τ i fail = max ( 1 , min (⌊ m s i k ⌋)) , where m s i is the budget for the i -th target space [60]. This change in regard to Casmopolitan is necessary because Casmopolitan 's failure tolerance of 40 [79] would cause the algorithm to spend a large part of the evaluation budget in initial target spaces of low dimensionality.

We show the effect of this strategy on discrete and mixed-space benchmarks in Figure 16. Figure 17 shows the effect on continuous benchmarks. All experiments are replicated 50 times. We observe that the proposed trust-region management method not only enables efficient batch parallelism but also improves the performance for single-function evaluations when compared to the respective baseline for the TR management that we stated above. The TR management proposed here usually

12

14

16

18

Objective value / Regret

25D-PestControl

200

150

100

50

60D-MaxSAT60

21.5

22.0

22.5

23.0

25D-Contamination

0

100

200

# batch evaluations

1.6 ×

10

1

1.8 ×

10

1

2 ×

10

1

2.2 ×

10

1

2.4 ×

10

1

2.6 ×

10

1

2.8 ×

10

1

3 ×

10

1

Objective value / Regret

53D-SVM

0

100

200

# batch evaluations

0

1

2

53D-Ackley53

0

100

200

# batch evaluations

4

3

2

1

50D-LABS

Bounce, b=1

Bounce, b=3

Bounce, b=5

Bounce, b=10

Bounce, b=20

prev. TR method

0

10 0

10 1

Objective value / Regret

500D-Branin2

225

250

275

300

325

124D-Mopta08

0

10 0

500D-Hartmann6

0

200

400

# batch evaluations

0.05

0.10

0.15

0.20

0.25

Objective value / Regret

388D-SVM

0

200

400

# batch evaluations

0.29

0.30

0.31

0.32

180D-LassoDNA

0

200

400

# batch evaluations

0.0

2.5

5.0

7.5

10.0

12.5

15.0

1000D-LassoHard

Bounce, b=1

Bounce, b=3

Bounce, b=10

Bounce, b=20

prev. TR method

provides better solutions at the same number of batches than the respective baseline TR management. There are a few exceptions. On Labs , the previous TR management strategy outperforms Bounce 's strategy by a small margin for batch sizes 5, 10, and 20, and on Ackley53 , the previous strategy converges to a better solution for a batch size of three, five, and ten.

On the continuous benchmarks (Figure 17), we observe that the new TR management strategy also improves the performance for single function evaluations with the exception of Branin2 . Similar to the mixed and combinatorial-space benchmarks, large batch sizes (10 and 20) bring little advantage compared to a batch size of 5. For Hartmann6 , the previous strategy with a batch size of 20 performs worse than 3.

G Implementation details

We implement Bounce in Python using the BoTorch [5] and GPyTorch [29] libraries.

We employ a Γ(1 . 5 , 0 . 1) prior on the lengthscales of both kernels and a Γ(1 . 5 , 0 . 5) prior on the signal variance. We further use a Γ(1 . 1 , 0 . 1) prior on the noise variance.

Motivated by Wan et al. [79] and Eriksson et al. [25], we use an initial trust region baselength of 40 for the combinatorial variables, and 0.8 for the continuous variables. We maintain two separate TR shrinkage and expansion parameters ( γ cmb and γ cnt) for the combinatorial and continuous variables, respectively such that each TR base length reaches its respective minimum of 1 and 2 -7 after a given number of function evaluations. When Bounce finds a better or worse solution, we increase or decrease both TR base lengths.

We use the author's implementations for COMBO 2 , BODi 3 , RDUCB 4 , SMAC 5 , and Casmopolitan 6 . We use the same settings as the authors for COMBO , RDUCB , SMAC , and BODi . For Casmopolitan , we use the same settings as the authors for benchmarks reported in Wan et al. [79] and set the initial trust region base length to 40 otherwise.

Due to its high-memory footprint, we ran BODi on NVidia A100 80GB GPUs for 300 GPU/h. We ran Bounce on NVidia A40 GPUs for 2,000 GPU/h. We ran the remaining methods for 20,000 GPU/h on one core of Intel Xeon Gold 6130 CPUs with 60GB of memory.

G.1 Optimization of the acquisition function

We use different strategies to optimize the acquisition function depending on the type of variables present in a problem.

Continuous problems. For purely continuous problems, we follow a similar approach as TuRBO [25]. In particular, we use the lengthscales of the GP posterior to shape the TR. We use gradient descent to optimize the acquisition function within the TR bounds with 10 random restarts and 512 raw samples. For a batch size of 1, we use analytical EI. For larger batch sizes, we use the BoTorch implementation of qEI [5, 66, 86].

Binary problems. Similar to Wan et al. [79], we use discrete TRs centered on the current best solution. A discrete TR describes all solutions with a certain Hamming distance to the current best solution.

We use a local search approach to optimize the acquisition function for all problems with a combinatorial search space of only binary variables: When starting the optimization, we first create a set of min(5000 , max(2000 , 200 · d i )) random solutions. The choice of the number of random solutions is based on TuRBO [25]. For each candidate, we first draw L i indices uniformly at random from { 1 , . . . , d i } without replacement, where L i is the TR length at the i -th iteration. We then sample d i values in { 0 , 1 } and set the candidate at the sampled indices to the sampled values. All other values are set to the values of the current best solution. Note that this construction ensures that each candidate solution lies in the TR bounds of the current best solution. We add all neighbors (i.e., points with a Hamming distance of 1) of the current best solution to the set of candidates. This is inspired by BODi [21]. We find the 20 candidates with the highest acquisition function value and use local search to optimize the acquisition function within the TR bounds: At each local search step, we create all direct neighbors that do not coincide with the current best solution or would violate the TR bounds. We then move the current best solution to the neighbor with the highest acquisition function value. We repeat this process until the acquisition function value does not increase anymore. Finally, we return the best solution found during local search.

Categorical problems. We adopt the approach for binary problems, i.e., we first create a set of random solutions with the same size as for purely binary problems and start the local search on the 20 best initial candidates.

Suppose the number of categorical variables of the problem is smaller or equal to the current TR length. In that case, we sample, for each candidate and each categorical variable, an index uniformly at random from { 1 , 2 , . . . , | v i |} where | v i | is the number of values of the i -th categorical variable. We then set the candidate at the sampled index to 1 and all other values to 0.

If the number of categorical variables of the problem is larger than the current TR length L i , we first sample L i categorical variables uniformly at random from [ d i ] without replacement. For each initial candidate and each sampled categorical variable, we sample an index uniformly at random, for which we set the categorical variable to 1 and all other values to 0. The values for the variables that were not sampled are set to the values of the current best solution.

As for the binary case, we add all neighbors of the current best solution to the set of candidates, and we sample the 20 candidates with the highest acquisition function value.

We then use multi-start local search to optimize the acquisition function within the TR bounds while neighbors are created by changing the index of one categorical variable. Again, we repeat until convergence and return the best solution found during the local search.

Ordinal problems. The construction for ordinal problems is similar to the one for categorical problems.

Suppose the number of ordinal variables of the problem is smaller or equal to the current TR length. In that case, we sample an ordinal value uniformly at random to set the ordinal variable for each candidate and each ordinal variable. Otherwise, we choose as many ordinal variables as each candidate's current TR length and sample an ordinal value uniformly at random to set the ordinal variable. We add all neighbors of the current best solution, all solutions where the distance to the current best solution is 1 for one ordinal variable, to the set of candidates. We then sample the 20 candidates with the highest acquisition function value and use local search to optimize the acquisition function within the TR bounds. In the local search, we increment or decrement the value of a single ordinal variable.

Mixed problems. Mixed problems are effectively handled by treating every variable type separately. Again, we create a set of initial random solutions where the values for the different variable types are sampled according to the abovementioned approaches. This can lead to solutions outside the TR bounds. We remove these solutions and find the 20 best candidates only across the solutions within the TR bounds.

When optimizing the acquisition function, we differentiate between continuous and combinatorial variables. We optimize the continuous variables by gradient descent with the same settings as purely continuous problems. When optimizing, we fix the values for the combinatorial values.

We use local search to optimize the acquisition function for the combinatorial variables. In this step, we fix the values for the continuous variables and only optimize the combinatorial variables. We create the neighbors by creating neighbors within Hamming distance of 1 for each combinatorial variable type and then combining these neighbors. Again, we run a local search until convergence.

Wedofive interleaved steps, starting with the continuous variables and ending with the combinatorial variables.

G.2 Kernel choice

We use the CoCaBo kernel [67] with one global lengthscale for the combinatorial and ARD for the continuous variables:

where x cnt and x cmb are the continuous and combinatorial variables in x , respectively, i.e.,

and ρ ∈ [0 , 1] is a tradeoff parameter learned jointly with the other hyperparameters during the likelihood maximization. Here, ℓ cmb and ℓ cnt ,i are the lengthscale hyperparameters.

H Additional analysis of BODi and COMBO

H.1 Analysis of BODi

Binary problems. BODi by Deshwal et al. [21] uses a dictionary of anchor points A = ( a 1 , . . . , a m ) to encode a candidate point z . In particular, the i -th entry of the m -dimensional embedding ϕ A ( z ) is obtained by computing the Hamming-distance between z and a i . Notably, Deshwal et al. [21] choose the dimensionality of the embedding m as 128 , which is larger than the dimensionality of the objective functions themselves.

The sampling procedure for dictionary elements a i is chosen to yield a wide range of sequencies . The sequency of a binary string is defined as the number of times the string changes from 0 to 1 and vice versa. Deshwal et al. [21] propose two approaches to generate the dictionary elements: (i) using binary wavelets, and (ii) by first drawing a Bernoulli parameter θ i ∼ U (0 , 1) for each i ∈ [ m ] and then drawing a binary string a i from the distribution B ( θ i ) . The latter approach is their default method.

Weprove that BODi generates an all-zeros (or, by a similar symmetry argument, all-ones) representer point with a probability that is significantly higher than 2 -D . Moreover, we claim without proof that a similarly increased probability holds for points with low Hamming distance to all-zeros or all-ones. This is consistent with the intention of BODi to sample points of diverse sequency and not necessarily an issue.

However, we claim without proof that having such points in the dictionary substantially increases the probability that BODi evaluates the all-zeros (and, by symmetry, the all-ones) points. That hypothesis is consistent with our observation in Section 4.6 that BODi has a much higher chance of finding good or optimal solutions when they are near the all-zero point.

Deshwal et al. [21] choose the dictionary to have m = 128 dictionary elements. Given a Bernoulli parameter θ i , the probability that the i -th dictionary point a i is a point of sequency zero is given by θ D i +(1 -θ i ) D :

Then, since θ i follows a uniform distribution, the overall probability for a point of zero sequency is given by

i.e., the probability of at least one dictionary element being of sequency zero is 1 -( 1 -2 D +1 ) m . The probability of BODi 's dictionary to contain a zero-sequency point increases with the number of dictionary elements m and decreases with the function dimensionality D (see Figure 18).

For instance, for the 60-dimensional MaxSAT60 benchmark, the probability that at least one dictionary element is of sequency zero is 1 -( 1 -2 60+1 ) 128 ≈ 0 . 986 (see Figure 18).

Note that at least one point z ∗ has a probability of ≤ 1 / 2 d to be drawn. The probability of the dictionary containing that z ∗ is less than or equal to 1 -( 1 -1 / 2 d ) m which is already less than

0.01 for d = 14 and m = 128 . In Section 4, we show that randomizing the optimal point structure leads to performance degradation for BODi . We hypothesize this is due to the reduced probability of the dictionary containing the optimal point after randomization.

Categorical problems. We calculate the probability that BODi contains a vector in its dictionary where all elements are the same. For categorical problems, BODi first samples a vector θ from the τ max -simplex ∆ τ max for each vector a i in the dictionary, with τ max being the maximum number of categories across all categorical variables of a problem. We assume that all variables have the same number of categories as is the case for the benchmarks in Deshwal et al. [21]. Let τ be the number of categories of the variables. For each element in a i , BODi draws a value from the categorical distribution with probabilities θ . While line 7 in Algorithm 5 in Deshwal et al. [21] might suggest that the elements in θ are shuffled for every element in a i , we observe that θ remains fixed based on the implementation provided by the authors 7 . The random resampling of elements from θ is probably only used for benchmarks where the number of realizations differs between categorical variables.

Then, for a fixed θ , the probability that all D elements in a i for any i are equal to some fixed value t ∈ { 1 , 2 , . . . , τ } is given by θ d t . The probability that, for any of the m dictionary elements, all D elements in a i are equal to some fixed value t ∈ { 1 , 2 , . . . , τ } is given by

Wenote that θ follows a Dirichlet distribution with α = 1 [54]. Then, θ t is marginally Beta (1 , τ -1) -distributed [54]. With that, Eq. (2) becomes

now, by using the equality E [ x D ] = ∏ D -1 r =0 α + r α + β + r for the D -th raw moment of a Beta ( α, β ) distribution [54]

We discussed in Section 4.3 that the PestControl benchmark obtains a good solution at x = 5 . One could assume that BODi performs well on this benchmark because its dictionary will likely contain this point. However, we observe that the probability is effectively zero for τ = 5 , m = 128 , and D = 25 (see Figure 19), which are the choices for the PestControl benchmark in Deshwal et al. [21]. This raises the question of (i) whether our hypothesis is wrong and (ii) what the reason for BODi 's performance degradation on the PestControl benchmark is.

We show that BODi 's reference implementation differs from the algorithmic description in an important detail, causing BODi to be considerably more likely to sample category five on PestControl (or the 'last' category for arbitrary benchmarks) than any other category.

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

number of elements set to value

0

2000

4000

6000

8000

10000

12000

Count

Category 1

Category 2

Category 3

Category 4

Category 5

Figure 20 shows five histograms over the number of dictionary elements set to each category. The values on the x -axis give the number of elements in a 25-dimensional categorical vector being set to a specific category. One would expect that the histograms have a similar shape regardless of the category. However, for category 5, we see that more elements are set to this category than for the other categories: The probability of k elements being set to category 5 is almost twice as high as the probability of being set to another category for k ≥ 3 . In contrast, the probability that no element in the vector belongs to category 5 is virtually zero. This behavior is beneficial for the PestControl benchmark, which obtained the best value found during our experiments for x ∗ = (5 , 5 , . . . , 5 , 1)

(see Section 4). While we see that the probability of each dictionary entry being set to category 5 is very low, we assume that we sample sufficiently many dictionary elements within a small Hamming distance to the optimizer such that BODi 's GP can use this information to find the optimizer.

The reason for oversampling of the last category lies in a rounding issue in sampling dictionary elements. In particular, for a given dictionary element a i and a corresponding vector θ with | θ | = τ , for each i ∈ { 1 , 2 , . . . , τ -1 } , Deshwal et al. [21] set ⌊ D θ i ⌋ elements to category i . The remaining D -∑ τ -1 i =1 ⌊ D θ i ⌋ elements are then set to category τ . This causes the last category to be overrepresented in the dictionary elements. For the choices of the PestControl benchmark, D = 25 and τ = 5 , the first four categories had a probability of ≈ 0 . 1805 . In contrast, the last one had a probability of ≈ 0 . 278 for 10 8 simulations 8 . We assume that the higher probability of the last category is the reason for the performance difference between the modified and the unmodified version of the PestControl benchmark.

H.2 COMBO on categorical problems

On the categorical PestControl benchmark, COMBO [56] behaved similar to BODi [21]

The histogram in Figure 21 shows how many element entries of a 25-dimensional dictionary element are set to each of the five categories. We see that the first and the last categories are over-represented on both benchmark versions. As discussed in Section 4, this benchmark attains the best observed value for x ∗ = (5 , 5 , . . . , 5 , 1) . We observe that on the original and modified version of the benchmark, where the categories are shuffled, COMBO sets disproportionately many elements to categories one and five. Note that for the modified version of the benchmark, all categories are equally likely in the optimal solution.

We argue that this behavior is at least partially caused by implementation error in the construction of the adjacency matrix and the Laplacian for categorical problems 9 . This error causes categorical variables to be modeled like ordinal variables. According to Oh et al. [56], categorical variables are modeled as a complete graph (see Figure 22).

a0

a1

a2

a3

a4

Looking into the source code of COMBO , we find the adjacency matrix for the first category of a categorical variable with five categories is constructed as

which is the adjacency matrix for a five-vertex path graph.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMost real-world optimization problems contain a mix of continuous, categorical, binary, and ordinal variables and might be high-dimensional. Although there are some methods that have investigated this problem, they are not always reliable with regard to finding a satisfactory optimum. In order to tackle this problem, the authors propose Bounce (Bayesian Optimization Using iNcreasingly high-dimensional Combinatorial and continuous Embeddings). This work seems to be a natural extension of BAxUS, which used a similar embedding strategy as for Bounce but only for continuous variables. It uses a novel trust region management system to grow or shrink the trust regions. The proposed method is tested on a representative range of test problems and compared with state-of-the-art benchmarks and shows convincing results.\n\nSTRENGTHS:\n- Most real-world optimization problems have mixed spaces and are high-dimensional for which vanilla Bayesian optimization methods are not suited, this work thus addresses an important problem.\n- The proposed algorithm is tested on a broad range of problems and compared with state-of-the-art benchmarks and has a strong performance.\n- The authors provide proof that the algorithm converges in the limit of infinite iterations, making it a reliable method.\n- The paper is well-written and well-structured.\n\nWEAKNESSES:\n- In general, the paper is very complete and well-written and contains a very complete related work section. However, the paper does expect the reader to have a significant amount of knowledge of previous work. Especially about BAxUS and papers using TR management strategies. Perhaps the authors could provide a little bit more information and illustrations in this work to make it more easily accessible. For instance, provide an illustration of the binning procedures and/or about the TR management.\n\nQUESTIONS:\n- Although I do understand that this is a typical question in BO paper reviews, for most problems, benchmarks are performed for 200 iterations. Would you say that this is enough for the dimensionalities of the benchmark problems? For instance, for the 125-D MaxSAT, you would say you probably need a lot more data points for the surrogate model to model the objective function effectively. What are your thoughts on this and do you think some of the benchmarks can be better than Bounce at a higher budget?\n\n- It is mentioned in Appendix C that the original implementations for COMBO, BODi and Casmopolitan are used. And that you use the same setting as in those works, what kind of settings are these exactly? Do these approaches use the same kernels for the GP models for example? Could this influence the performance on the benchmarks?\n\n- Have the authors contacted Oh et al. regarding the bug in COMBO? If it is possible, it would of course be ideal if results using a bug-less version can be provided.\n\n- I really like the results shown in Figure 5 regarding the efficacy of batch acquisition. In general, I like that the # batch evaluations are used on the x-axis as this directly shows what is good to use for users who are optimizing parallel processes. However, I'm also interested in how these plots look as a function of function evaluations. My intuition would say that b=1 is always better, as in this way the model is as informed as it can for the next iteration. Just out of interest, could you elaborate on this here or could you add a plot to the appendix that shows these results as a function of iterations?\n\n- As a suggestion to your citations regarding chemical engineering and materials discovery. There also is a range of works regarding the optimization of lab equipment for sample analysis. See for instance [Hagan et al.](https://pubs.acs.org/doi/full/10.1021/ac049146x?casa_token=dnIitFW7lO4AAAAA%3Ado8InveDdcPw3TwtOVSQuRvM5NQhSFEo3M1jmpdpEHvRbzA0f1jYTJ2_bloYb7he8-Ofb8u96oMXvCvO), [Boelrijk et al.](https://scholar.google.nl/citations?view_op=view_citation&hl=nl&user=1z-BBwkAAAAJ&citation_for_view=1z-BBwkAAAAJ:9yKSN-GCB0IC). These problems can typically contain mixed spaces and many variables.\n\nSome small textual remarks:\n- Could it be that d_0 and d_{init} are interchangeably used? For instance lines 2 and 3 of Algorithm 1. \n- Line 90, 'sequencies' should be 'sequences' or was this the originally proposed name by the BODi paper?\n- Lines 277-278, as well should be as well as?\n- Typo in Ass. 5. in Appendix Section a. One should be once?\n\nLIMITATIONS:\nThe authors describe the societal impact of their work.\nThe authors do not describe the limitations of their work. Perhaps the authors could dedicate some sentences to this, for example, would the algorithm handle a noisy setting? Or does this violate the binning procedure to some extent?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new BO method, namely Bounce, to tackle the problem of BO with combinatorial and mixed space. The key idea is based on the trust region approach (as with TurBO) and adaptive space (as with BAxUS), but is applied to the combinatorial and mixed variables. The proposed method is also enabled to work with batch. Experiments are conducted on various synthetic and real-world problems to evaluate the efficacy of the proposed method.\n\nSTRENGTHS:\n+ The paper’s writing is generally clear and easy to understand\n+ The paper tackles an interesting problem, which is to solve BO problems with categorical and mixed variables\n+ The methodology developed in the paper seems to be sound to me. \n+ The experiments are conducted with various benchmark optimization problems (synthetic and real-world)\n\nWEAKNESSES:\n+ The proposed method seems to be largely an extension from existing methods: TurBO and BAxUS – normally this is alright if the results are impressive but there are some issues with the experiments which I will describe in more detail in the below bullets.\n+ There is not much insight on why the proposed method, Bounce, works. There is a theoretical analysis in the appendix (Theorem 1) that shows the consistency of the Bounce. However, I think this theorem doesn’t have much meaning. It assumes that the search domain is finite, and the objective function is noiseless, and it states that Bounce will find a global optimum with probability of 1 when the number of samples N goes to infinity. But for a finite domain and noiseless observations, any algorithm will be able to find the global optimum?\n+ One of the contributions of the paper is to conduct an in-depth analysis of two state-of-the-art algorithms for combinatorial BO, COMBO [45] and BODi [17], however, I don’t really find this to be interesting. Unless I missed it, it seems to be discussed in Section 4.6? But this is largely empirical and not much insight.\n+ The experiments seem to lack of some well-known baselines for categorical and mixed search space, for example, SMAC, TPE, and some new BO methods like the work “Bayesian Optimization over Hybrid Spaces” by Deshwal et al (ICML 2021), and the work “Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization” by Daulton et al (NeurIPS 2022).\n+ Also, in the experiments, the number of iterations is quite small very high-dimensional problems. The number of iterations evaluated in all problems is just 200, which is too small.\n\nQUESTIONS:\nPlease address my comments in the previous section (section Weakness)\n\nLIMITATIONS:\nI don’t find any dedicated section describing the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new Bayesian optimization algorithm for combinatorial and mixed search spaces containing input variables of different types (continuous, binary, categorical, ordinal), which promises to be relevant for problems as varied as materials discovery, hardware design, neural architecture search, and portfolio optimization.\nBounce relies on a Gaussian process (GP) model of the objective function, which is based on lower-dimensional subspaces of the original search space, generated by partitioning input variables into “bins”.  Notably, the bins only contain input variables of the same type (e.g. continuous) and all variables in a bin are forced to take the same value during the optimization of the acquisition function, in effect operating in a lower-dimensional subspace. During the course of an optimization run, Bounce splits up bins into smaller ones, enabling the algorithm to propose candidates with increasingly finer structure. Bounce also leverages existing techniques for high-dimensional BO, including the trust-region approach.\n\nSTRENGTHS:\n- Strong empirical results, comparing against published baselines, in addition to ablations on the batch optimization performance.\n- Performance is more robust to shifts of the solution than previously published methods.\n- Sheds light on a non-trivial structural assumption of prior methods.\n- Generally a well written paper.\n\nWEAKNESSES:\n- No theoretical analysis. \n- Without a “modified” or “shifted” label for the left subplots of Figures 1, 2, 3, 4, the presentation is confusing and can easily be misread to suggest that the left subplots report results on the problems as defined in the literature, but they do not. Please add a “modified” or “shifted” label for the left subplots.\n\nQUESTIONS:\n- In the first 60 or so iterations, Bounce is outperformed by BODi on the non-modified 125d MaxSAT problem, as well as the  non-modified PestControl problem. Since sample-efficiency is a chief concern for BO methods, is there a way to leverage  BODi’s inductive biases for Bounce? On a high level, something like this might be possible since both methods leverage embeddings of the input space.\n- Is there something more you can say about any theoretical aspects of the method? Does a global convergence guarantee hold? What can you say about the convergence rate, either local or global? How does the choice of embedding influence the convergence?\n- Are there other embeddings approaches that could be combined with Bounce?\n\nSuggestions:\n\n- The description of the cardinal and ordinal embeddings are a bit repetitive and verbose, especially since they formally use the same approach. Can you unify the presentation, possibly include a single math-mode version of the embedding formula? That would aid the readability of the section.\n- Line 191: “The embedding of ordinal variables follows [that of the] categorical variables”\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of optimizing black-box functions defined over high-dimensional combinatorial and mixed continuous-combinatorial spaces. The key idea is to use the Bayesian optimization framework specialized to increasingly large nested embeddings of input dimensions in order to tackle the high-dimensionality challenge. The paper proposes separate embeddings for different class of input variables i.e. continuous, binary, categorical and ordinal. A trust region based approach is employed to enable parallel candidate evaluations in the proposed approach. Experiments are performed on multiple benchmarks to demonstrate the efficacy of the approach.\n\nSTRENGTHS:\n- The problem space considered in the paper is quite relevant and arises in multiple real-world applications. The paper is written really well describing the scope of the problem and associated applications. \n\n- The proposed idea (although building on BAxUS and TurBO) is principled and explained well. The method clearly outperforms all state-of-the-art baselines on several benchmarks demonstrating its efficacy. \n\n- The related work discussion and the amount of effort to compare with all the baselines in proper way is commendable and deserves credit. \n\nOverall, I think the paper will be useful and practical contribution to this problem space of high dimensional combinatorial spaces.\n\nWEAKNESSES:\nPlease see my question below.\n\nQUESTIONS:\nThe intuition or principle behind the embedding specific to categorical variable is not entirely clear to me. While for continuous variables, the embedding comes from the idea of count-sketches, it is not clear immediately what is captured by assigning categorical variables of eve different cardinalities to the same bin. For example, if there is an outlier variable with very large cardinality or if the range of cardinalities is quite large, the embedding would be very sensitive to the outlier? Please expand the description about the categorical embedding more if possible.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm called Bounce, using nested embedding for mixed and combinatorial search space. Bounce partitions input variables into ‘bins’ and set all variables within the same bin to a single value to reduce the dimension. During the optimization, Bounce splits up bins into smaller bins for more refined optimization.\n\nSTRENGTHS:\n1/ The combinatorial and mixed optimization is significant and has wide range of applications in the real world. \n\n2/ The algorithm is reasonable. Bounce designs a reasonable mapping for binary, categorical and ordinal variables, while a similar algorithm, BAXUS, which also uses nested random subspaces, has shown good performance on continuous optimization.\n\nWEAKNESSES:\nI don't have any major complaints. The proposed algorithm is a natural extension of BAXUS for categorical and mixed space. The experiments show that Bounce is reliable and can achieve good performance.   \n\nHere are some minor comments: \n\n1/ One minor weakness is the the high-dimensional continuous spaces part in Section 2. A recent survey paper [1] categorizes these works into several categories, i.e., low-dimensional embeddings, decomposition, and variable selection. This paper mainly discuss the work in low-dimensional embeddings I think discuss the recent works in decompostion [3-4] and variable selection [5-6] would further strengthen this paper. \n\n[1] A Survey on High-dimensional Gaussian Process Modeling with Application to Bayesian Optimization. ACM TELO, 2022. \n\n[2] High-Dimensional Bayesian Optimization via Tree-Structured Additive Models. AAAI, 2021. \n\n[3] Are Random Decompositions all we need in High Dimensional Bayesian Optimisation? ICML, 2023.\n\n[4] Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization. NeurIPS, 2022.\n\n[5] Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes. AISTATS, 2022.\n\n2/ It would benificial to see the the disscussion (even experimental comparison) with a work with similar topic [6]. \n\n[6] Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces\n\nQUESTIONS:\nI am puzzled about why the performance of BODi and COMBO can be sensitive to the location of the optima, which is equivalent to shuffling the labels of the categories of each variable. BODi uses Hamming distance and COMBO uses the combinatorial graph to model the relation of different labels for each variable. Different labels for the same variables are the same, so shuffling the labels will not impact the performance. Can you provide more explanation?\n\nI think the variable selection methods [4-5] maybe also can be applied in the mixed spaces tasks. I want to see some discussions. \n\nI'm glad to increase my score if you address the proposed isssues.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}