SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\alpha_ {\beta} ^ {(\mathrm {U C B})} (\boldsymbol {x}) = \mu_ {\boldsymbol {\theta} ^ {\star}} (\boldsymbol {x}) + \sqrt {\beta} \sigma_ {\boldsymbol {\theta} ^ {\star}} (\boldsymbol {x}).
$$
Thompson Sampling (TS): The TS [63] acquisition function balances the exploration-exploitation trade-off by sampling function values from the posterior distribution of the black-box function and selecting the next query point based on the sampled values. In each iteration, TS samples potential query points  $x$  from the state space, estimates their mean black-box function values (see Appendix B.1), and selects the query point associated with the best-performing estimated mean value. In our proposed framework, the TS acquisition function is currently only utilized by BO methods relying on the Linear Regression surrogate model [1].
Appendix C Available Benchmarks
To enable the wider adoption and systematic evaluation of MCBO methods, our proposed framework addresses the need for standardized development and benchmarking methodologies. Building on the API for defining optimization problems, we provide a broad suite of real-world and synthetic benchmarks in our open-source software. These benchmarks consist of a diverse spectrum of optimization problems and their associated domains, covering a wide range of dimensionalities and difficulties. We provide the full list of the implemented tasks along with their domains in Table 2.
Table 2: Tasks implemented in the MCBO framework and the optimization domains they support. In this table, we assume that  $\mathbf{x} = [c, i, h]$  is partitioned into continuous variables  $c$ , integer-valued variables  $i$  and categorical variables  $h$ .  $\min_c$  and  $\max_c$  specify arbitrary bounds for continuous variables.  $\{\min_i, \max_j\}$  defines a set containing all integers between  $\min_i$  and  $\max_i$ .  $N_c$ ,  $N_i$  and  $N_h$  specify the number of continuous-valued, integer-valued and combinatorial variables respectively.  $M_{N_i}$  specifies the number of categories the  $i^{\text{th}}$  combinatorial variable can take.
Objective f	Inputs
SFU Test Functions	Arbitrary combination of input types and dimensionalitiesc ∈ [minc, maxc]Nc i ∈ {mini, maxi}Ni h ∈ {h1, ..., hM1} × ... × {hn, ..., hMNh}
Ackley-20D (included in SFU)	11 categories per variableh ∈ {−32.768, −26.214, ...26.214, 32.768}20
Ackley-53D (included in SFU)	c ∈ [−1, 1]3 h ∈ {0, 1}50
Pest Control	Pesticide choice at each stage (or use no pesticide)h = {No pesticide, Pest. 1, Pest. 2, Pest. 3, Pest. 4}25
AIG Flow Tuning	h ∈ {rewrite, rewrite -z, refactor, refactor -z, resub, balance, &blut, &sopb, &dsdb, fraig}20
MIG Flow Tuning	h ∈ {balance, cut rewrite, cut rewrite -z, refactor, refactor -z, resubstitute, functional_reduction}20
Antibody Design	Let AA be the set of all 20 naturally occurring amino acidsh ∈ AA11
RNA Inverse Folding	h ∈ {A, C, G, U}Nh
AIG Flow. and Hyp. Tuning	Sequence:h ∈ {rewrite, rewrite -z, refactor, refactor -z, resub, balance, &blut, &sopb, &dsdb, fraig}20Hyperparameters of the operators:14 Boolean Variables and 22 integer variables
XGBoost - MNIST	c1 is learning rate, c2 is min split loss, c3 is subsample, c4 is reg lambda, i1 is max depth, h1 is booster, h2 is grow policy and h3 is objective.c1 ∈ [10-5, 1]c2 ∈ [0, 10]c3 ∈ [0.001, 1]c4 ∈ [0, 5]i1 ∈ {1, 10}h1 ∈ {gbtree, dart}h2 ∈ {depthwise, lossguide}h3 ∈ {multi:softmax, multi:softprob}
ν-SVR - Slice	c1 is ε, c2 is C, c3 is γ and h concerns featuresc1 ∈ [0.01, 1]c2 ∈ [0.01, 100]c3 ∈ [0.1, 10]h ∈ {exclude, include}50
Synthetic Tasks Our suite of synthetic benchmarks provides a controlled setting for evaluating the performance of MCBO methods, where the true optima are often known, and objective function evaluations are inexpensive, enabling rigorous and scalable benchmarking. The benchmarks include the 21 Simon Fraser University (SFU) test functions described by [33], which are generalized to  $d$ -dimensional domains. These functions cover a range of optimization difficulties such as steep ridges, many local minima, valley-shape functions, and bowl-shaped functions. To increase their versatility, we have extended these functions to handle continuous, discrete, nominal, and ordinal variables, as well as combinations of these variable types. As part of the implementation of the SFU
test functions, we also include the Ackley -53D [5, 6] task, which is a special case of the Ackley function.
In addition, we include the pest control task used in [2, 5, 6], which involves optimizing the use of pesticides in a chain of 25 stations to minimize the number of products with pests while minimizing expenses on pesticide control. This task involves selecting from four different pesticides at each station, with varying prices and effectiveness, and has complex dynamics due to interactions between pesticides and pests. The pest control task provides a challenging optimization problem with high-order interactions.
Real-World Tasks To evaluate the effectiveness of MCBO methods on real-world problems with practical objectives and constraints, our benchmark suite also includes a diverse set of real-world tasks spanning four domains: logic synthesis optimization, antibody design, RNA inverse folding, and hyperparameter tuning of ML models. We have selected these tasks for their relevance to different application domains and their challenging optimization landscapes.
Logic Synthesis: In the logic synthesis domain, we support benchmarks that involve optimizing the gate-level representation of Boolean circuits using a sequence of transformative operations. We consider two standard directed graph representations of Boolean circuits: the And-Inverter Graph (AIG) [71] and the Majority-Inverter Graph (MIG) [81]. We have a separate benchmark for each representation. Furthermore, we include an implementation of the AIG optimization benchmark that optimizes both the sequence of transformative operations and the hyperparameters of the underlying transformative algorithms. The AIG sequence and AIG sequence and hyperparameter optimization benchmark implementations rely on the ABC [71] codebase for applying transformative operations to the AIG representation, while the MIG sequence optimization benchmark relies on Mockturtle [82] library. All three logic synthesis benchmarks can be used to optimize any Boolean circuits, and notably the 20 Boolean functions from the open-source EPFL Combinational Benchmark Suite [83] that we use in our experiments.
Antibody Design: In the antibody design benchmark, we aim to optimize the CDRH3 sequence of an antibody for binding to a specific antigen. To achieve this, we use the Absolut! [84] software as an in silico framework for approximating the binding energy between an antibody and an antigen. Furthermore, we follow the recommendations of Khan et al. [72] for the developability constraint and check whether a CDRH3 sequence has no more than five consecutive amino acids that are identical, whether its net charge is in the interval  $[-2, 2]$ , and whether it is free of undesirable glycosylation motifs [85]. Our antibody design benchmark suite includes a diverse set of 159 antigen-CDRH3 binding tasks, where each task corresponds to a different antigen.
RNA Inverse Folding: In the RNA Inverse Folding task, the goal is to find one or more RNA sequences that fold into a target secondary structure. This task is of utmost importance to the RNA design process as it enables the design of novel RNA molecules with specific functions, such as catalysis and regulation [86-88]. As done in [73], we also on the ViennaRNA folding package [89] to model RNA inverse folding. However, as the ViennaRNA package does not deal with pseudoknots that can naturally occur in RNA structures, we further constrain the secondary structure by ensuring that it cannot have two base pairs  $(i,j)$  and  $(k,l)$  with  $i < k < j < l$ . In our benchmarking suite, we rely on the EteRNA100 dataset [90] to facilitate benchmarking on a collection of one hundred RNA secondary structures.
Hyperparameter Optimization of Machine Learning Models: Hyperparameters are critical tuning parameters that can greatly affect the performance of machine learning (ML) models. Optimizing these hyperparameters can improve the model's accuracy and robustness. In this work, we implement two tasks for hyperparameter optimization: one for the XGBoost [75] model on the MNIST [76] dataset (or any other dataset accessible through the Scikit-Learn [91] API), and the other for the  $\nu$ -Support Vector Regression ( $\nu$ -SVR) [77] model on the UCI slice dataset [78].
The search space of the XGBoost task comprises three categorical and five numerical variables. The categorical variables include the choice of the booster type, the grow policy, and the training objective. The numerical variables include the learning rate, max tree depth, minimum split loss, amount of regularization and the sub-sample magnitude.
For the  $\nu$ -SVR task, we use the Scikit-Learn [91] implementation of the  $\nu$ -SVR model. Similar to [6], we make the search space to include 3 continuous hyperparameters of the SVR,  $C$ ,  $\epsilon$ , and  $\gamma$
Figure 6: Evolution of the average rank of Mixed-variable BO algorithms across four tasks, aggregated by the surrogate model (left), acquisition optimizer (centre), and use of a trust region (right).
as well as 50 Boolean parameters corresponding to feature selection. Originally, the slice dataset comprises 384 features, but we reduce this number to 50 by fitting an XGBoost model and by keeping the 50 most important features according to the model. Then the 50 Boolean variables of the search space corresponds to the inclusion or exclusion of each of the retained features for the fit of a  $\nu$ -SVR model. The task consists in determine which features and values of the hyperparameters allow the minimization the RMSE obtained by the corresponding  $\nu$ -SVR model on a held-out test set.
Appendix D Benchmarking Implemented Mixed-Variable Algorithms
D.1 Analyzing Mixed-variable BO Design Choices
Surrogate model We observe in Fig. 6 (left) that BO methods using the GP (mat. 5/2 - HED) surrogate have a statistically significant lower rank compared to methods that use the GP (mat. 5/2 - O) and GP (mat. 5/2 - TO) surrogate. We hypothesize that this is because the HED kernel was better suited to modelling the interactions between the combinatorial variables for the four considered black-box functions.
Acquisition function optimizers Fig. 6 (centre) reveals that the MAB-GD acquisition optimizer significantly underperformed on average compared to the remaining acquisition optimizers. It also shows that the HC-GD and SA acquisition optimizers are best suited for scenarios with low budgets, while there is no statistically significant difference between their performance and that of the GA acquisition optimizer at high budgets.
Trust region (TR) Finally, Fig. 6 (right) encourages the use of a TR in mixed-variable BO, as methods working with a dynamic TR to fit local models and constraining acquisition optimization provide consistently better suggestions compared to global approaches. This confirms and extends the findings of Wan et al. [5] and is consistent with our results presented in section 5.3.
D.2 Which is the Most Robust Mixed-Variable Algorithm
We now consider each mix-and-match mixed-variable BO algorithm individually (without aggregating ranks by primitives) and compare them to non-BO baselines. We first order all the algorithms based on their average ranks across fifteen seeds and four tasks, and we show on Fig. 7 the performance of the three known BO solvers, the four non-BO baselines and the 2 (out of 21) new mix-and-match methods achieving the lowest rank.
We observe that the new mixed-variable algorithm, GP (mat 5/2 - HED) - GA w/ TR, achieves a statistically significant average
Figure 7: Average rank of known mixed-variable algorithms and the two best mix-and-match BO algorithms.
rank improvement compared to the considered known mixed-variable BO algorithms and non-BO baselines. Notably, this algorithm is novel, and its design is aligned with our conclusions from Appendix D.1.
For each BO optimizer and the three best non-BO baselines, we show on Fig. 8 the evolution of the best objective values attained. We note that on three of the considered tasks (Ackley-53D, AIG Flow. and Hyp. Tuning and XGBoost - MNIST), a BO algorithm outperformed the considered non-BO solvers. However, on the final task,  $\nu$ -SVR - Slice, the HC and SA non-BO baselines attained comparable performance to the best-performing BO solvers, highlighting the need for further development of mixed-variable BO techniques when it comes to solving real-world tasks. Nevertheless, BO solvers are still the best on most tasks, though we observe that no single optimizer achieves the lowest objective value across all tasks.
Appendix E Notes on Compatibility of Implemented MCBO Primitives
The MCBO framework includes three core BO primitives: surrogate models, acquisition functions, and acquisition optimizers. Below we discuss some of the limitations on the mix-and-match compatibilities of the currently implemented MCBO primitives and their compatibility with various optimization domains.
E.1 Surrogate Models
Our implementation features the following pre-implemented surrogate models:
GP with SSK [4, 7]
GP with overlap kernel (O) [3]
GP with transformed-overlap kernel (TO) [5]
GP with diffusion kernel (Diff.) [2]
GP with dictionary-based kernel (HED) [6]
- GP with mixture kernel [5]
- Linear Regression [50] with the Horseshoe prior [67] using maximum likelihood, maximum a posteriori, and Bayes estimation (LHS) [1].
Below we provide some additional details about the compatibility of each of these models.
$GP$  (SSK) is only applicable to combinatorial problems where each of the categorical variables shares the same possible categories.
$GP(O)$  is only applicable to combinatorial problems.
$GP(TO)$  is only applicable to combinatorial problems.
$GP(\text{Diff.})$  is only applicable to combinatorial problems. Also, as the GP (Diff.) model as proposed by Oh et al. [2] is really an ensemble of 10 models, any acquisition function the user wishes to combine with this model must be used to initialize the SingleObjAcqExpectation class, which can be subsequently mixed-and-matched with the model.
Figure 8: Evolution of the best observed value on each mixed-variable task achieved by the known mixed-variable optimizers and the two best mix-and-match BO solvers built with MCBO framework. In all tasks, the aim is to minimize the black-box. Hence, the lower the best observed value the better.
$GP$  (HED) is only applicable to combinatorial problems.
GP (mixture kernel) is compatible with mixed-variable problems. The mixture kernel currently supports the use of the RBF or Mat. 5/2 kernel for numeric variables, and the use of the O, TO or HED kernel for categorical variables, resulting in six different potential GP (mixture kernel) models.
$LR(HS)$  is only applicable to combinatorial problems.
E.2 Acquisition Functions
Our implementation features the following pre-implemented acquisition functions:
Expected Improvement (EI) [59, 60]
- Probability of Improvement (PI)[61]
- Lower Confidence Bound (LCB) [62]
Thompson Sampling (TS) [63]
The  $EI$ ,  $PI$  and  $LCB$  acquisition functions support all surrogate models except for the LR (HS) surrogate model. The reason for this is that the LR (HS) does not maintain an analytical estimate of the posterior but instead uses Gibbs sampling [1] to sample from it, making it impossible to calculate EI, PI and LCB in closed form. In contrast, the TS acquisition function only supports the LR (HS) surrogate model.
E.3 Acquisition Optimizers
Our implementation features the following pre-implemented acquisition optimizers:
- Exhaustive Local Search (LS) [2]
- Genetic Algorithm (GA) [7]
- Simulated Annealing (SA) [1]
- Interleaved Search (Hill Climbing + Gradient Descent) (IS (HC + GD))[5]
- Interleaved Search (Multi-Armed Bandit + Gradient Descent) (IS (MAB + GD))[3]
Below we provide some additional details about the compatibility of each of these optimizers.
$LS$  is only applicable to combinatorial problems.
$GA$  is compatible with combinatorial and mixed-variable problems.
SA is compatible with combinatorial and mixed-variable problems.
$IS(HC + GD)$  is compatible with combinatorial and mixed-variable problems. For combinatorial problems, this acquisition optimizer will simply perform hill climbing, while for purely numeric problems, it will perform gradient descent.
$IS(MAB + GD)$  is compatible with combinatorial and mixed-variable problems. For combinatorial problems, this acquisition optimizer will simply use a MAB to optimize the combinatorial variables, while for purely numeric problems, it will perform gradient descent.
We also note that we have generalized the implementation of these acquisition optimizers so that all of them support TR-constrained acquisition optimization, extending [5, 68], and make them handle simple input domain constraints via rejection sampling and projections.
E.4 Implemented Mix-and-Match Optimizers
Given the compatibilities outlined above, in the MCBO framework, for combinatorial formulations, we support 48 different mix-and-match BO optimizers. This results from combining the six different surrogate models suitable for combinatorial formulations (GP (SSK), GP (O), GP (TO), GP (Diff.), GP (HED) and LR), with the four compatible acquisition optimizers (LS, GA, SA and HC) both with and without a trust region constraint.
Similarly, for mixed-variable formulations, we can instantiate 24 different mix-and-match BO optimizers. This results from combining three compatible surrogate models (GP (Mat.  $5/2 + \mathrm{O}$ ), GP (Mat.  $5/2 + \mathrm{TO}$ ) and GP (Mat.  $5/2 + \mathrm{HED}$ )) with the four compatible acquisition optimizers (GA, SA, IS (HC + GD) and IS (MAB + GD)), both with and without a trust region constraint.
We note that 7 of these optimizers are known combinatorial and mixed-variable BO algorithms. After accounting for the fact that some mix-and-match combinatorial optimizers are special cases of the known BO algorithms, this results in a total of 47 novel mix-and-match BO algorithms.
Appendix F Experimental setup
F.1 Hyperparameters
We share in Table 3 a comprehensive list of the hyperparameters used during training and inference in all experiments. More details can be found in the associated code repository.
Table 3: List of MCBO primitives hyperparameters.
Surrogate model
GP mixed-kernel form	λ(Kcat + Knum) + (1 - λ)(Kcat · Knum)
GP Kernel for numeric variables	Matértn-5/2
GP noise level lower bound	10-5
GP fit negative log-likelihood optimizer	Adam(lr = 0.03) [92] for 100 epochs
Diffusion GP characteristics [2]	n_models = 10, n_burn_init = 100
Bayesian linear regression SH estimator [1]	Order = 2, Threshold = 0.1, a = 2, b = 1, n_gibbs = 10-3
Acquisition function optimizer
LS	n_randomVertices=20000, ngreedy_ascent_init=20, n_spray=10
SA	num_iter=100, n_restarts=3, init_temp=1, tolerance=100
GA	ga_num_iter=500, ga_pop_size=100, cat_ta_num_parents=20, cat_ta_num_elite=10
IS (HC + GD)	n_iter=100, n_restarts=3, max_n_perturb_num=20, num_optimizer=Adam, num_lr=1e-3, nom_tol=100
IS (MAB + GD)	max_n_iter=200, mab_resample_tol=500, n_cand=5000, n_restarts=5, num_optimizer=SGD, cont_lr=3e-3, cont_n_iter=100
Trust region Manager
tr_restart_acq_name	LCB
tr_restart_n_cand	min(100 × num_dims, 5000)
tr_min_num_radius	2-5
tr_max_num_radius	1
tr_init_num_radius	0.8 × tr_max_num_radius
tr_min_nominal_radius	1
tr_max_nominal_radius	num_nominal
tr_init_nominal_radius	round(0.8 × tr_max_nominal_radius)
tr_radius-multiplier	1.5
tr_succ_tol	3
tr_fail_tol	40
Appendix G Implementation Details and Additional Features
G.1 Input and Output Normalization
For stable learning with GPs, we normalize all inputs to the range [0, 1] and standardize all black-box function values using the mean and standard deviation of all previously observed values. We apply this standardization independently during every iteration of BO, prior to fitting a model.
G.2 Input Constraints
The MCBO framework supports cheap-to-compute constraints on the search space via rejection sampling. When defining an optimization problem via the TaskBase class, one can additionally define a list of functions which can each take a sample as input and returns whether the sample is valid or not. Then, we pass this list of functions as an argument when constructing a BO algorithm with the BoBuilder class, as we do for the Antibody design task to restrict the search space to feasible antibody sequences.
G.3 Trust-Region-Based Acquisition Optimization
All implemented acquisition optimizers support trust-region-constrained acquisition optimization. Following [5], the trust region centred around the best input found so far in the current trust region,  $\boldsymbol{x}^{\mathrm{TR}}$ , is defined as
$$
\operatorname {T R} \left(\boldsymbol {x} ^ {\operatorname {T R}}\right) = \left\{\boldsymbol {x} \in \mathcal {X} \text {s . t .} d _ {c} \left(\boldsymbol {x} _ {c} ^ {\operatorname {T R}}, \boldsymbol {x} _ {c}\right) \leq L _ {c} \text {a n d} d _ {L _ {n}} \left(\boldsymbol {x} _ {n} ^ {\operatorname {T R}}, \boldsymbol {x} _ {n}\right) \leq 1 \right\},
$$
where  $\pmb{x}_c^{\mathrm{TR}}$  and  $\pmb{x}_c$  are vectors containing all the combinatorial inputs in  $\pmb{x}^{\mathrm{TR}}$  and  $\pmb{x}$  respectively,  $d_c(\cdot ,\cdot)$  is the Hamming distance, and  $L_{c}$  is the size of the TR for combinatorial variables. Similarly,  $\pmb{x}_n^{\mathrm{TR}}$  and  $\pmb{x}_n$  are vectors containing all the numeric and discrete variables in  $\pmb{x}^{\mathrm{TR}}$  and  $\pmb{x}$  respectively,  $d_{L_n}(\cdot ,\cdot)$  is the maximum of the component-wise distance for numeric and discrete variables normalized by dimensional length scales  $L_{n}\in \mathbb{R}_{+}^{\dim (\pmb{x}_{n})}$ , i.e.
$$
d _ {L _ {n}} (\pmb {x} _ {n} ^ {\mathrm {T R}}, \pmb {x} _ {n}) = \max _ {i \in \dim (\pmb {x} _ {n})} \frac {| \pmb {x} _ {n} ^ {\mathrm {T R}} [ i ] - \pmb {x} _ {n} [ i ] |}{L _ {n} [ i ]}.
$$
The radius of the combinatorial trust region,  $L_{c}$ , and the radii for the numeric variables,  $L_{n}$ , are adjusted dynamically during optimization. They are expanded on successive successes (i.e. when the best function value observed improves) and shrunk otherwise. They also have a predetermined maximum and minimum value. If any of the radii were to be expanded past their maximum allowable value, they would be capped at this value instead. However, if any of the radii were to be shrunk smaller than their minimum value, this would trigger a trust region restart.
During a trust region restart, all the radii are set to their initial value, and a global auxiliary surrogate model is used to determine a centre of a new trust region. Suppose an algorithm restarts its trust region for the  $i^{\text{th}}$  time. First, the global surrogate model is fitted to a subset of data  $D^{*} = \{x_{j}^{\text{TR}}, y_{j}^{\text{TR}}\}_{i=1}^{j-1}$ , where  $x_{j}^{\text{TR}}$  is the local maxima found after the  $j^{\text{th}}$  restart, and  $y_{j}^{\text{TR}}$  is its corresponding black-box function value. This auxiliary surrogate model is then used to define an acquisition function, which is then subsequently optimized to suggest a new trust region centre.
G.4 Batch sampling
All currently implemented acquisition function optimizers support batch sampling using the Kriging Believer strategy [93] to select  $b$  points sequentially. To sample the  $i^{\text{th}}$  point, the Kriging Believer strategy replaces the black-box function evaluation for the  $(i - 1)^{\text{th}}$  point with the GP prediction, hal-lucinating what the black-box function value could have been, and retrans the GP on the aggregated dataset, before optimizing the acquisition function to obtain the  $i^{\text{th}}$  suggestion.