SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.2 Surrogate Models, Acquisition Functions, and Acquisition Optimizers
The MCBO framework includes three core BO primitives: surrogate models, acquisition functions, and acquisition optimizers. 1) Our implementation features several pre-implemented surrogate models, such as a GP with the string subsequence kernel (SSK) [4, 7], overlap kernel (O) [3], transformed-overlap kernel (TO) [5], diffusion kernel (Diff.) [2], dictionary-based kernel (HED) [6], mixture kernel [5], and linear regression [50] with the Horseshoe prior [67] using maximum likelihood, maximum a posteriori, and Bayes estimation (LHS) [1]. 2) On the acquisition function side, we include the widely used expected improvement [59, 60], probability of improvement [61] and lower confidence bound [62] for GPs, and Thompson sampling [63] for LHS. 3) Furthermore, our library includes various acquisition optimizers, such as exhaustive Local Search [2], Genetic Algorithm [7], Simulated Annealing [1], and interleaved search (IS) alternating between Hill Climbing or Multi-armed Bandit for combinatorial variables and gradient-descent steps for numeric variables as developed for CoCaBO [3] and Casmopolitan [5]. Moreover, we generalize the implementation of these acquisition optimizers so that all of them support TR-constrained acquisition optimization, extending [5, 68], and make them handle cheap-to-compute input domain constraints via rejection sampling.
4.3 Defining MCBO Algorithms
With our framework, defining MCBO algorithms is effortless. By specifying the IDs of the surrogate model, acquisition function, acquisition optimizer, and TR manager in the BoBuilder class constructor, the corresponding BO primitives are automatically retrieved. In an optimization loop, the optimizer created with BoBuilder handles model fit, acquisition optimization, and TR adjustments. Thanks to the BoBuilder class, we can easily mix-and-match BO primitives from existing algorithms, allowing the implementation of novel MCBO algorithms with just one line of code per algorithm. We demonstrate the versatility of this approach by implementing 47 novel MCBO algorithms and evaluating them on a set of ten tasks in Section 5. As an example, we showcase on the next page how to use BoBuilder to construct the Casmopolitan algorithm [5] and apply it to optimize a generic black-box function for a specified budget.
4.4Baselines
MCBO baselines We leverage the aforementioned BO primitives to implement seven existing MCBO algorithms: Casmopolitan [5], BOiLS [4], COMBO [2], CoCaBO [3], BOSS [7], BOCS [1], and BODi [6]. Furthermore, we explore the remaining combinations of implemented surrogate models and acquisition optimizers, including trust-region-based optimizers, implementing 47 additional novel MCBO algorithms (See Fig. 1). In Section 5, we benchmark the performance of these 54 MCBO algorithms and a set of five standard non-BO baselines that we describe in the following paragraph.
Black-Box Optimization Baselines To facilitate benchmarking against non-BO optimization methods, we include the following baselines in our library: Random Search (RS) [69], Hill Climbing (HC) [70], GA [25], SA [23, 24], and MAB [27]. We make them inherit from the OptimizerBase class, ensuring a consistent API with MCBO solvers with the use of suggest and observe methods.
bobuilder  $=$  BoBuilder( model_id  $\equiv$  'gp_to', acq_opt_id  $\equiv$  'is', acq_func_id  $\equiv$  'ei', tr_id  $\equiv$  basic', \*\*kwargs   
） # Corresponds to Casmopolitan   
opt  $=$  bo.Builder.build_bo( search_space  $\equiv$  search_space, n_init=20
for_in range(budget): x_next  $=$  opt.suggest() y_next  $=$  black_box(x_next) opt. observe(x_next, y_next)   
print(opt.best_x, opt.best_y)
4.5 Available Benchmarks
We include diverse benchmarks, enabling a systematic evaluation of MCBO methods across various optimization domains, dimensionalities, and difficulties. The benchmarks encompass both synthetic and real-world tasks. While we briefly overview the available benchmarks here, we provide a more detailed description, including specific constraints, dimensionalities, supported domains, and implementation details in Appendix C.
The synthetic benchmark suite offers a controlled environment for evaluating the performance of MCBO methods, featuring the 21 Simon Fraser University (SFU) test functions [33] that generalize to  $d$ -dimensional domains, extended to handle continuous, discrete, and nominal variables, as well as combinations of these variable types. Additionally, we include the pest control task [2] that presents a challenging optimization landscape with high-order interactions.
For real-world tasks, the benchmark suite includes logic synthesis optimization [71], antibody design [72], RNA inverse folding [73], and hyperparameter tuning of ML models. The logic synthesis benchmarks optimize Boolean circuits represented by And-Inverter Graphs (AIG) and Majority-Inverter Graphs (MIG) [74]. The antibody design task focuses on optimizing the CDRH3 region of antibodies for binding to a specific antigen. RNA inverse folding aims to find RNA sequences that fold into a target secondary structure. Hyperparameter optimization tasks involve tuning the parameters of XGBoost [75] on the MNIST dataset [76] and  $\nu$ -Support Vector Regression ( $\nu$ -SVR) [77] on the UCI slice dataset [78] for which we do feature selection along with the hyperparameter tuning as in [6].
To illustrate the capacity of our library, we conduct experiments tackling the following questions:
1. Which implemented surrogate model and acquisition optimizer performs best?
2. Does incorporating a TR constraint improve the performance of MCBO?
3. Which implemented Combinatorial and Mixed-variable algorithm is the most robust?
We exploit the high modularity and flexibility of our framework to easily instantiate a total of 48 combinatorial BO algorithms using the BoBuilder class, and compare their performance on six combinatorial tasks, including Ackley-20D, pest control, sequence of operators tuning for AIG and MIG logic synthesis, antibody design, and RNA inverse fold task. We also run five non-BO baselines - RS, HC, GA, SA, and MAB - on these six combinatorial tasks. Similarly, we use BoBuilder to implement 24 mixed-variable BO methods and evaluate them on four mixed-variable
Figure 2: Evolution of the average rank of combinatorial BO algorithms across six tasks, aggregated by surrogate model (left), acquisition optimizer (center), and use of a trust-region (right).
tasks, including Ackley-53D, XGBoost hyperparameter tuning,  $\nu$ -SVR tuning with feature selection, and logic synthesis flow optimization for AIGs. We benchmark the mixed-variable BO algorithms against four non-BO baselines, RS, HC, GA, and SA. Each experiment is repeated across fifteen random seeds, resulting in over 4,000 individual experiments.
5.1 Experimental Procedure
To ensure a fair and consistent comparison, we follow a standardized experimental design. For a given task and random seed, each BO algorithm suggests the same set of 20 uniformly sampled points and observes the corresponding black-box function values. For the next 180 steps, each algorithm suggests a new point to evaluate based on its surrogate model and acquisition function optimization process. We therefore query the black-box function 200 times per algorithm and per seed. For a given black-box evaluation budget  $i = 1, \dots, 200$ , we denote by  $y_{i}^{*}$  the best value attained so far,  $y_{i}^{*} = \min_{1 \leq j \leq i} y_{j}^{3}$ . As the scales of the black-box values can differ drastically from one task to another, we compare the optimizers' performance across tasks by considering their ranks based on the  $y_{i}^{*}$ s. To analyze the impact of some primitives, We can then aggregate the ranks across tasks, random seeds, and the BO primitives not under investigation. For example, when assessing surrogate models, we average the rank of the optimizers sharing the same surrogate across tasks, random seeds, acquisition functions, and acquisition optimizers. In figures displaying the evolution of ranks, we show the mean rank in solid line, and the standard error with respect to tasks and seeds as a shaded area. We conduct a Friedman test to see if we can reject the hypothesis that all the methods' performances are equal, and if it is not the case, we add black vertical lines to connect the algorithms whose mean rank differences are smaller than the length of the critical interval given by the post-hoc Wilcoxon signed-rank test.
5.2 Hardware
We run our experiments on two machines with 4 Tesla V100-SXM2-16GB GPUs and an Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 88 threads, which allowed us to parallelize over tasks and seeds. It took us approximately two weeks to run the set of experiments described in this paper.
5.3 Analyzing Combinatorial BO Design Choices
Surrogate model We observe in Fig. 2 (left) that BO methods with LSH or GP (Diff.) surrogate significantly underperform compared to those based on GP (HED), GP (O), GP (TO), and GP (SSK). However, from the well-performing surrogate models, no single model outperforms the others significantly, indicating that certain models are better suited for specific types of black-box functions as we will investigate in Section 5.4.
Acquisition function optimizers Fig. 2 (center) shows that exhaustive LS performs best at very low budgets. But as the number of step increases, GA delivers superior mean performance. This suggests that the high exploitation offered by LS is beneficial when few suggestions are allowed, but the less myopic approach of GA achieves a better exploration-exploitation trade-off as budget increases.
3Reporting regret is not possible for the real-world tasks whose minima are unknown.
Trust region (TR) Finally, Fig. 2 (right) shows that, on average, methods working with a dynamic TR to fit local models and constrain acquisition optimization provide consistently better suggestions compared to global approaches. This confirms and extends the findings of Wan et al. [5].
5.4 Which is the Most Robust Combinatorial Algorithm?
In this section, we consider each mix-and-match combinatorial BO algorithm individually (without aggregating ranks by primitives), and compare them to non-BO baselines. We first order all the algorithms based on their average ranks across ten seeds and 6 tasks, and we show on Fig. 3 the performance of the 6 known BO solvers, the 5 non-BO baselines, and the 2 (out of 42) new mix-and-match methods achieving the lowest rank.
Figure 3: Average rank of known combinatorial algorithms and the two best mix-and-match BO.
We observe that only the two new MCBO algorithms achieve a statistically significant average rank improvement compared to the GA,
and SA black-box optimization baselines. These two algorithms utilize a GP (SSK) or GP (TO) surrogate model with a GA and trust region constraint for acquisition optimization. Notably, these algorithms are novel and their design choices align with our conclusions from Section 5.3. However, it is important to highlight that the difference in rank between these algorithms and Casmopolitan [5], BOSS [7], and BOiLS [4] is not statistically significant. This can be attributed to the significant role of model fit (see Section 5.4) in BO and the averaging of results across diverse tasks, where different surrogate models may be optimal.
For each BO optimizer and the three best performing non-BO baselines, we show on Fig. 4 the evolution of the best objective values attained. We note that on two tasks (Antibody design and RNA inverse fold), a non-BO algorithm (SA and GA respectively) outperforms all BO solvers, highlighting the need for further development of combinatorial BO techniques when it comes to solving real-world tasks. Nevertheless, BO solvers are still the best on a majority of tasks, though we observe that no single optimizer achieves the lowest objective value across all tasks. We investigate the variability of BO performance in the next paragraph.
GP (TO) - GA w/ TR
Casmopolitan
BOCS
Figure 4: Evolution of best observed value on each combinatorial task achieved by the known combinatorial optimizers and the two best mix-and-match BO solvers built with MCBO framework. In all tasks, the aim is to minimize the black-box. Hence, the lower the best observed value the better.
Model fit and BO performance As underlined in Section 3.1, the choice of kernel impacts the GP modeling capacity, which could explain the variability of BO performance on the different black-box functions. To assess the relation between kernel choice and BO performance, we measure the Pearson correlation between the quality of a GP model fit, and the quality of the objective value attained after 200 iterations of BO equipped with the same surrogate. We get the quality of a GP fit on a given task and seed by conditioning the GP on the first 150 points  $\{(x_i,y_i)\}_{i = 1}^{150}$  coming from our GA run and computing the log-likelihood of the last 50 black-box values  $\{y_i\}_{i = 150}^{200}$  under the GP prediction at  $\{x_{i}\}_{i = 150}^{200}$ . Fixing the acquisition optimizer and the use of TR, we collect for all tasks and seeds the GP log-likelihood and the BO performance when using SSK, TO, O, HED and diffusion kernel. Given a task and a seed, the set of points coming from the GA run are the same to limit a source of noise in the performance comparison. As expected, Fig. 5 shows a positive correlation between BO performance and the capacity of its surrogate model to fit the black-box. The correlation is weaker for local acquisition optimizers (LS) than more explorative ones (GA).
Figure 5: Pearson correlation between the quality of a surrogate fit, and BO regret, split by acquisition function and use of a TR.
5.5 Mixed-variable optimization
Due to space limitations, we defer the analysis of results obtained on mixed tasks to Appendix D. This analysis reveals that a new mixed-variable BO algorithm, which combines a GP (Matérn-5/2 and HED kernel) surrogate model with a GA acquisition optimizer and a trust region constraint, achieves a statistically significant better rank averaged across all the considered tasks.
This work introduces a modular and flexible framework for MCBO, addressing the need for systematic benchmarking and standardized evaluation in the field. Leveraging this framework, we implement a total of 48 combinatorial and 24 mixed-variable BO algorithms with just a single line of code per algorithm. Benchmarking these algorithms on a set of 10 tasks, which is a subset of the tasks available in our framework, we conduct over 4000 individual experiments. Through these experiments, we provide insights into the choice of surrogate model based on its fit, the selection of acquisition optimizer, and the use of a trust region constraint to constrain acquisition optimization.
It is important to note that these experiments represent only a fraction of the possibilities our framework offers. We aim to demonstrate the ease and potential of our MCBO framework, encouraging further analyzes and the development of new primitives. We also invite MCBO developers to integrate their novel options into our flexible codebase, enabling easy mixing and matching of algorithms and facilitating benchmarking across various synthetic and real-world tasks with minimal effort.
[87] Douglas H. Turner and David H. Mathews. Nndb: the nearest neighbor parameter database for predicting stability of nucleic acid secondary structure. *Nucleic acids research*, 38(Database issue):D280-D282, Jan 2010. ISSN 1362-4962. doi: 10.1093/nar/gkp892. URL https://pubmed.ncbi.nlm.nih.gov/19880381.
[88] Chenhui Hao, Xiang Li, Cheng Tian, Wen Jiang, Guansong Wang, and Chengde Mao. Construction of rna nanocages by re-engineering the packaging rna of phi29 bacteriophage. Nature Communications, 5(1):3890, May 2014. doi: 10.1038/ncomms4890. URL https://doi.org/10.1038/ncomms4890.
[89] Ronny Lorenz, Stephan H. Bernhart, Christian Höner zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F. Stadler, and Ivo L. Hofacker. Viennarna package 2.0. Algorithms for Molecular Biology, 6(1):26, Nov 2011. ISSN 1748-7188. doi: 10.1186/1748-7188-6-26. URL https://doi.org/10.1186/1748-7188-6-26.
[90] Peter Eastman, Jade Shi, Bharath Ramsundar, and Vijay S. Pande. Puzzles in the Eterna100 benchmark., 6 2018. URL https://plos.figshare.com/articles/dataset/Puzzles_in_the_Eterna100_benchmark_/6638837.
[91] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
[92] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
[93] David Ginsbourger, Rodolphe Le Riche, and Laurent Carraro. *Kriging Is Well-Suited to Parallelize Optimization*, pages 131-162. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010. ISBN 978-3-642-10701-6. doi: 10.1007/978-3-642-10701-6_6. URL https://doi.org/10.1007/978-3-642-10701-6_6.
Appendix A Lexicon
In Table A we include a full list of the abbreviations used in this paper, alongside their definitions.
Table 1: A full list of the abbreviations used in this paper alongside their definitions
Abbreviation	Definition
AIG	And-Inverter Graphs
API	Application Programming Interface
ARD	Automatic Relevance Determining
BO	Bayesian Optimization
Diff.	Diffusion kernel
EA	Evolutionary Algorithm
EI	Expected Improvement
GA	Genetic Algorithm
GP	Gaussian Process
HC	Hill Climbing
HED	Hamming embedding via dictionary kernel
IS	Interleaved Search
LCB	Lower Confidence Bound
LHS	Linear Regression with the Horseshoe Prior
LS	Exhaustive Local Search
MAB	Multi-Armed-Bandit
MCBO	Mixed-variable and Combinatorial Bayesian Optimisation
MIG	Majority-Inverter Graphs
ML	Machine Learning
O	Overlap Kernel
PI	Probability of Improvement
RNA	Ribonucleic Acid
RS	Random Search
SA	Simulated Annealing
SFU	Simon Fraser University
SSK	String Subsequence Kernel
TO	Transformed Overlap
TR	Trust Region
TS	Thompson sampling
ν-SVR	ν-Support Vector Regression
Appendix B Bayesian Optimization
B.1 Inference with Gaussian Processes
Given a dataset  $\mathcal{D} = \{\pmb{x}_i, y_i\}_{i=1}^n$ , and assuming Gaussian-corrupted observations  $y_i = f(\pmb{x}_i) + \epsilon_i$  where  $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ , the joint probability distribution over the observed data and an arbitrary test input  $\pmb{x}_{\mathrm{test}}$  can be written as:
$$
\left[ \begin{array}{c} \mathbf {y} _ {1: n} \\ f (\boldsymbol {x} _ {\text {t e s t}}) \end{array} \right] \Bigg | \boldsymbol {\theta} \sim \mathcal {N} \left(\mathbf {0}, \left[ \begin{array}{c c} \mathbf {K} _ {\boldsymbol {\theta}} + \sigma^ {2} \mathbf {I} & \mathbf {k} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {\text {t e s t}}) \\ \mathbf {k} _ {\boldsymbol {\theta}} ^ {\top} (\boldsymbol {x} _ {\text {t e s t}}) & k _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {\text {t e s t}}, \boldsymbol {x} _ {\text {t e s t}}) \end{array} \right]\right). \tag {4}
$$
Here, we assume a zero-mean GP prior and use  $\mathbf{y}_{1:n}$  to represent the vector of all outputs, i.e.,  $\mathbf{y}_{1:n} = [y_1,\dots ,y_n]^\top$ . The block-covariance matrix in the equation above is defined as:
$$
\mathbf {K} _ {\boldsymbol {\theta}} = \mathbf {K} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {1: n}, \boldsymbol {x} _ {1: n}\right) \in \mathbb {R} ^ {n \times n}, \quad \text {s u c h t h a t} \left[ \mathbf {K} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {1: n}, \boldsymbol {x} _ {1: n}\right) \right] _ {k, \ell} = k _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {k}, \boldsymbol {x} _ {\ell}\right) \quad \forall (k, \ell)
$$
$$
\mathbf {k} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {\text {t e s t}}\right) = \mathbf {k} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {1: n}, \boldsymbol {x} _ {\text {t e s t}}\right) \in \mathbb {R} ^ {n \times 1}, \quad \text {s u c h t h a t} \left[ \mathbf {k} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {1: n}, \boldsymbol {x} _ {\text {t e s t}}\right) \right] _ {k} = k _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {k}, \boldsymbol {x} _ {\text {t e s t}}\right) \quad \forall k.
$$
By conditioning the joint distribution, we can derive the posterior distribution for predicting  $f(\pmb{x}_{\mathrm{test}})$  [28], resulting in  $f(\pmb{x}_{\mathrm{test}})|\pmb{x}_{\mathrm{test}}, \mathcal{D}, \pmb{\theta} \sim \mathcal{N}\left(\mu_{\pmb{\theta}}(\pmb{x}_{\mathrm{test}}), \sigma_{\pmb{\theta}}^{2}(\pmb{x}_{\mathrm{test}})\right)$ , where:
$$
\mu_ {\boldsymbol {\theta}} (\boldsymbol {x} _ {\text {t e s t}}) = \mathbf {k} _ {\boldsymbol {\theta}} ^ {\top} (\boldsymbol {x} _ {\text {t e s t}}) (\mathbf {K} _ {\boldsymbol {\theta}} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {y} _ {1: n}
$$
$$
\sigma_ {\boldsymbol {\theta}} ^ {2} (\boldsymbol {x} _ {\mathrm {t e s t}}) = k _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {\mathrm {t e s t}}, \boldsymbol {x} _ {\mathrm {t e s t}}) - \mathbf {k} _ {\boldsymbol {\theta}} ^ {\mathsf {T}} (\boldsymbol {x} _ {\mathrm {t e s t}}) (\mathbf {K} _ {\boldsymbol {\theta}} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {k} _ {\boldsymbol {\theta}} (\boldsymbol {x} _ {\mathrm {t e s t}}).
$$
The optimal kernel hyperparameters  $\theta^{*}$  can be learned by minimizing the negative log-likelihood [28].
B.2 Learning Optimal Kernel Hyperparameters
To infer the optimal kernel hyperparameters  $\theta^{*}$  given  $\mathcal{D}$ , we can minimize the negative log marginal likelihood [28], leading to the following optimization problem:
$$
\min  _ {\boldsymbol {\theta}} \mathcal {F} (\boldsymbol {\theta}) = \frac {1}{2} \det  \left(\mathbf {K} _ {\boldsymbol {\theta}} + \sigma^ {2} \mathbf {I}\right) + \frac {1}{2} \mathbf {y} _ {1: n} ^ {\top} \left(\mathbf {K} _ {\boldsymbol {\theta}} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {y} _ {1: n} + \operatorname {c n s t}. \tag {5}
$$
Since the problem in the equation above is non-convex with respect to  $\theta$ , typical solvers employ off-the-shelf packages that often utilize random restarts to escape local minima [42, 43, 79, 80].
B.3 Acquisition Functions
We now provide additional information about the Probability of Improvement (PI) [61], the Lower Confidence Bound (LCB) [62] and Thompson Sampling (TS) acquisition functions, assuming that the aim is to minimize the black-box function value.
Probability of Improvement: The PI acquisition function is closely related to EI in that it also measures the utility of new query points with respect to the best black-box function value observed so far,  $y^{\star}$ . Contrary to EI, however,  $\alpha^{(\mathrm{PI})}(x)$  judges the probability of acquiring new gains compared to  $y^{\star}$  using the following definition:
$$
\alpha^ {(\mathrm {P I})} (\pmb {x}) = \mathbb {E} _ {f (\pmb {x}) | \mathcal {D} _ {i - 1}, \pmb {\theta} ^ {\star}} \left[ \mathbb {I} \left[ f (\pmb {x}) <   y ^ {\star} \right] \right],
$$
where  $\mathbb{I}$  is the indicator function that evaluates to 1 if  $f(x) < y^{\star}$  and to zero otherwise. Intuitively, the PI acquisition function estimates the probability of the black-box function value at  $x$  being lower than the lowest black-box function value observed so far.
Lower Confidence Bounds (LCB): Compared to the EI [59, 60] and the PI acquisition function, the LCB [62] trades-off the mean and the variance of the posterior distribution through an additional tuneable hyperparameter  $\beta \in \mathbb{R}^{+}$ :
$$
\alpha_ {\beta} ^ {(\mathrm {L C B})} (\boldsymbol {x}) = - \mu_ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x}) - \sqrt {\beta} \sigma_ {\boldsymbol {\theta} ^ {*}} (\boldsymbol {x}).
$$
When minimizing a black-box function  $f$ , LCB estimates the potential minimum value at a given query location  $x$ . Conversely, when maximizing the black-box, the Upper Confidence Bound (UCB) [62] should be used instead of the LCB, which is defined as: