SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kamil Dreczkowski *†
Imperial College of London
krd115@ic.ac.uk
Antoine Grosnit*
Technische Universität Darmstadt
Haitham Bou-Ammar
This paper introduces a modular framework for Mixed-variable and Combinatorial Bayesian Optimization (MCBO) to address the lack of systematic benchmarking and standardized evaluation in the field. Current MCBO papers often introduce non-diverse or non-standard benchmarks to evaluate their methods, impeding the proper assessment of different MCBO primitives and their combinations. Additionally, papers introducing a solution for a single MCBO primitive often omit benchmarking against baselines that utilize the same methods for the remaining primitives [1-4]. This omission is primarily due to the significant implementation overhead involved, resulting in a lack of controlled assessments and an inability to showcase the merits of a contribution effectively. To overcome these challenges, our proposed framework enables an effortless combination of Bayesian Optimization components, and provides a diverse set of synthetic and real-world benchmarking tasks. Leveraging this flexibility, we implement 47 novel MCBO algorithms and benchmark them against seven existing MCBO solvers and five standard black-box optimization algorithms on ten tasks, conducting over 4000 experiments. Our findings reveal a superior combination of MCBO primitives outperforming existing approaches and illustrate the significance of model fit and the use of a trust region. We make our MCBO library available under the MIT license at https://github.com/huawei-noah/HEBO/tree/master/MCBO.
The goal of mixed-variable and combinatorial optimization is to seek optimizers of functions defined over search spaces whose sizes grow exponentially with their dimensions. Applications of this field are ubiquitous, spanning a wide range of domains, such as supply chain optimization [8, 9], vehicle routing [10, 11], machine scheduling in manufacturing systems [12, 13], asset optimization [14, 15], among many others. A general recipe for tackling such tasks involves using the knowledge of domain experts to formalize combinatorial problems within the scope of well-established suited mathematical frameworks, and to apply available heuristic-based
Figure 1: In the MCBO framework, we can build an existing or a new algorithm by choosing a surrogate model and an acquisition function optimizer using a single line of code.
solvers. For instance, making the problem compatible with linear, quadratic, nonlinear or integer pro
$^{\dagger}$ Work done during an internship at Huawei Noah's Ark Lab in London RC.
gramming solvers, or reducing the combinatorial task at hand to a standard problem such as knapsack [16], traveling salesman [17], or network flow problem [18], allows the application of optimized and scalable off-the-shelf software developed by many companies and academic laboratories [19-22].
Despite numerous successes on many combinatorial tasks, standard heuristics like those mentioned above fall short in vital domains that require the optimization of black-box objectives. In those instances, we have, at best, partial a priori knowledge about the characteristics of the optimization objective, making it difficult, if not impossible, to map it to one of the forenamed mathematical frameworks. Although well-established black-box optimization methods such as Simulated Annealing (SA) [23, 24], Genetic Algorithms (GAs) [25], and Evolutionary Algorithms (EAs) [26], as well as online learning methods like Multi-Armed-Bandits (MABs) [27], can still be used to solve such problems, they often fall short at optimizing expensive-to-evaluate objectives due to their high sample complexities. Consequently, addressing problems with expensive-to-evaluate black-box objectives necessitates the development of data-driven and sample-efficient solution methodologies.
One promising strategy to handle such objectives is to 1) efficiently learn a (local) probabilistic model of the black-box function, and 2) balance exploration and exploitation by leveraging the model's uncertainty. This concept lies at the heart of Mixed-Variable and Combinatorial Bayesian Optimization (MCBO), a machine learning (ML) subfield crucial for achieving efficient optimization with immense potential for solving practical mixed-variable and combinatorial optimization problems.
MCBO algorithms generally have three high-level primitives: a probabilistic surrogate model, an acquisition function, and an acquisition optimizer that can operate in a trust region (TR). As shown in Fig. 1, we can frame many published MCBO algorithms as a specific combination of primitives, illustrating the intrinsic modularity of MCBO. For example, BOiLS [4] uses a Gaussian Process (GP) [28] with the string subsequence kernel (SSK) [29, 30] as its surrogate, and optimizes its acquisition function (expected improvement (EI)) via TR-constrained Hill Climbing [5]. BOSS [7] shares two of its primitives with BOiLS but only differs in using a GA to optimize its acquisition function.
Although many existing MCBO algorithms share some of the primitives they use, it remains unclear which solutions are state-of-the-art for each primitive and which combination of primitives constitutes a state-of-the-art MCBO method. This issue arises due to two factors.
Firstly, MCBO papers often introduce non-diverse and/or non-standard benchmarks to evaluate their proposed methods. The lack of a standardized set of diverse benchmarks makes it difficult to assess the relative performance of the different MCBO primitives and their combinations. As observed in other fields, establishing standardized test domains is crucial for accelerating progress in MCBO. For example, the ImageNet dataset [31], ShapeNet, and MuJoCo [32] facilitated rapid advancements in 2D computer vision, 3D computer vision, and robotics research. Therefore, it is essential to establish a procedural generation of test domains in MCBO, enabling a systematic evaluation of existing methods to inspire the development of novel approaches.
Secondly, papers introducing a solution for a single MCBO primitive often forget to benchmark against baselines that use the same methods for the remaining primitives [1-4], failing to fully highlight the merits of their proposed solution in a controlled setting. This is most likely due to the need for time-consuming and tedious efforts to modify and combine MCBO primitives from existing open-source implementations, as no standardized API exists to allow the interactions among them.
To address the aforementioned problems, we propose a flexible and comprehensive Python framework for MCBO. Our library provides a high-level API for all the MCBO primitives, and implementations of the primitives of several key MCBO baselines. Our framework also features a BoBui1der class that enables effortless implementation of existing and novel algorithms by flexibly combining MCBO primitives using a single line of code. In Section 5, we showcase the versatility of our library by implementing seven existing MCBO baselines and 47 novel MCBO algorithms. We evaluate these against five standard black-box optimization baselines on ten tasks, revealing insights into key MCBO design choices that contribute to robust algorithms.
Our library also includes implementations of a wide range of synthetic and real-world mixed-variable and combinatorial benchmarks, covering a broad spectrum of domain dimensionalities and optimization difficulties. These benchmarks include well-known optimization problems, such as the Ackley function [33] and the Pest Control problem [2], and optimization problems that extend
the current application venues of BO, including Antibody Design, RNA inverse folding, and Logic Synthesis optimization.
In the interest of space, throughout this paper we make heavy use of abbreviations. We therefore provide a list of these abbreviations, alongside their definitions, in Table 1 of Appendix A.
Bayesian Optimisation Frameworks Existing popular libraries for Bayesian Optimization (BO), such as Spearmint [34], GPyOpt [35], Cornell-MOE [36], RoBO [37], Emukit [38], Dragonfly [39], ProBO [40], and GPFlowOpt [41], offer diverse capabilities and modeling techniques for various optimization tasks. These libraries excel in different areas, including hyperparameter sampling, input warping and parallel optimization [34, 35], multi-fidelity optimization [36-38], and probabilistic programming [40]. However, in contrast to our work, these libraries primarily focus on continuous and discrete search spaces and lack direct support for combinatorial and mixed-variable domains.
Closest to our work is BoTorch [42], a model-agnostic Python library that is integrated with GPyTorch [43], providing efficient and scalable implementations of GPs in PyTorch [44]. Although BoTorch includes the GP kernel for mixed-variable optimization proposed by Wan et al. [5], it primarily focuses on continuous and discrete search spaces and does not provide direct built-in support for combinatorial formulations. In contrast, our framework is designed to tackle the unique challenges of BO in combinatorial and mixed-variable spaces and extends the capabilities of existing libraries by incorporating surrogate models and acquisition optimization techniques tailored to these domains.
Furthermore, our framework offers an unprecedented level of ease and flexibility in implementing BO algorithms by mixing-and-matching implemented BO primitives. With just a single line of code, users can combine different surrogate models, acquisition functions, and optimization methods. In contrast, other libraries would require time-consuming manual implementation processes to make components match.
Mixed-variable and Combinatorial Optimization Benchmarks Existing popular benchmarks for mixed-variable and combinatorial problems include the 24 synthetic mixed-integer test functions [45] that are part of the COCO [46] benchmarking suite. Our framework extends some of these functions, including the sphere, rotated hyper ellipsoid, Rastrigin, and Rosenbrock functions, to accommodate arbitrary dimensional domains and combinations of variable types. Moreover, we introduce additional synthetic functions that uphold the same generalization principles.
Distinguishing our benchmarking suite from benchmarks such as HPOBench [47], HPO-B [48], and YAHPO Gym [49], which primarily focus on a single domain such as hyperparameter optimization, our framework takes a more comprehensive perspective. Our suite encompasses a broader spectrum of tasks, spanning hyperparameter tuning, antibody design, Electronic Design Automation, RNA inverse folding, along with the classical synthetic tasks. This multi-domain nature underpins our conviction that benchmarking on diverse tasks that cover a broad spectrum of problem types and complexities strengthens the robustness and reliability of findings. We hold firm to the notion that this multifaceted approach leads to more generalisable conclusions than scenarios where only a single family of tasks is used for benchmarking, which can still be useful for important black-box families.
3 Mixed-Variable and Combinatorial Bayesian Optimization
BO is a sequential model-based technique to efficiently optimize a black-box function  $f(\cdot)$  over a search space  $\mathcal{X}$ . Due to the black-box nature of  $f(\cdot)$ , we can only evaluate it at input locations  $\pmb{x} \in \mathcal{X}$  to get (noisy) outputs  $y \in \mathbb{R}$ , such that  $\mathbb{E}[y|f(\pmb{x})] = f(\pmb{x})$ . BO tackles global optimization problems by iteratively repeating two steps. At iteration  $i$ , it first suggests a point  $\pmb{x}_i$  to evaluate, and in a second step, the learner observes  $y_i$ , by evaluating the black-box at  $\pmb{x}_i$ . To enable sample efficiency, the suggestion of  $\pmb{x}_i$  typically involves learning a (local) probabilistic surrogate model of  $f(\cdot)$  from the set of already observed points, and is followed by the optimization of an acquisition function  $\alpha(\cdot)$  trading-off exploration vs exploitation. We highlight both steps in Alg. 1, and depict a generic BO loop operating for a total budget of  $T_{\mathrm{max}}$ .
3.1 The Surrogate Model
Optimizing expensive black-box functions requires modeling the objective to enable efficient search by prioritizing informed decision-making over blind function evaluations [34]. When dealing with mixed-variable and combinatorial formulations, various surrogate models are available, including Bayesian Linear Regression [50], Random Forests [51], TreeStructured Parzen Estimators [52], and Bayesian Neural Networks [53]. Still, GPs [28] remain the most widely adopted models in the literature due to their tractability, sample efficiency, and capacity to maintain calibrated uncertainties [28, 34, 39, 54, 55], and therefore our library mostly focuses on their support.
Algorithm 1: Generic BO Algorithm
Input: Initial dataset
$$
\begin{array}{l} \mathcal {D} _ {n _ {0}} = \{\boldsymbol {x} _ {j}, y _ {j} \} _ {j = 1} ^ {n _ {0}}, \text {b u d g e t} \\ T _ {\max }, \text {b l a c k - b o x} f. \end{array}
$$
for  $i = n_0 + 1,\dots,T_{\mathrm{max}}$  do
$$
\begin{array}{l} \text {L e a r n a s u r r o g a t e m o d e l} \mathcal {M} (\cdot | \mathcal {D} _ {i - 1}) \\ \text {o f} f. \end{array}
$$
$$
\begin{array}{l} \text {U s e} \mathcal {M} (\cdot | \mathcal {D} _ {i - 1}) \text {t o d e f i n e} \\ \text {a c q u i s i t i o n f u n c t i o n} \alpha (\boldsymbol {x} | \mathcal {D} _ {i - 1}). \end{array}
$$
$$
\text {O p t i m i z e} \alpha (\boldsymbol {x} | \mathcal {D} _ {i - 1}) \text {t o g e t} \boldsymbol {x} _ {i}.
$$
$$
\begin{array}{l} \text {G e t} y _ {i} \text {b y e v a l u a t i n g f a t} x _ {i}. \\ \text {S e t} \mathcal {D} _ {i} \leftarrow \mathcal {D} _ {i - 1} \cup \{\boldsymbol {x} _ {i}, y _ {i} \}. \end{array}
$$
Output:  $\pmb{x}^{\star} = \arg \min_{\pmb{x}_i \in \mathcal{D}_{T_{\max}}} y_i$ .
A GP is a non-parametric model that represents the
prior belief about a black-box function as a distribution over functions and updates this distribution as new observations become available, resulting in a posterior distribution. A GP is fully defined by its mean function  $m(\cdot)$  and kernel function,  $k_{\theta}(\cdot, \cdot)$ , where  $\theta$  are the kernel hyperparameters [28, 54]. The mean function captures the overall trend and bias of the modeled function, while the kernel function characterizes the correlation between function values at different input locations. Specifically, the kernel function,  $k_{\theta}(x, x')$ , expresses our assumptions regarding the smoothness and periodicity of the modeled function, as it corresponds to the covariance between pairs of function values  $\operatorname{Cov}(f(x), f(x'))$ .
When modeling black-box functions defined over combinatorial spaces, one commonly used kernel function is the Overlap kernel [3, 56, 57], defined using Kronecker delta function  $\delta(\cdot, \cdot)$  as
$$
k _ {\boldsymbol {\theta}} ^ {\mathrm {O}} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) = \frac {\sigma}{d} \sum_ {p = 1} ^ {d} \lambda_ {p} \delta \left(\boldsymbol {x} [ p ], \boldsymbol {x} ^ {\prime} [ p ]\right), \tag {1}
$$
with  $\pmb{\theta} = (\sigma, \lambda_1, \dots, \lambda_d) \in \mathbb{R}_+^{d + 1}$ , where  $\sigma$  represents the kernel variance,  $d$  is the dimensionality of  $\pmb{x}$ , and  $\lambda_p$  denotes the Automatic Relevance Determining (ARD) [58] length scale for the  $p^{\mathrm{th}}$  variable. In combinatorial problems,  $\pmb{x} \in \{c_1^1, \dots, c_{N_1}^1\} \times \dots \times \{c_1^M, \dots, c_{N_M}^M\}$ , where  $\{c_1^i, \dots, c_{N_i}^i\}$  is the set defining all the possible elements that the  $i^{\mathrm{th}}$  combinatorial variable can assume,  $N_i$  is the cardinality of this set, and  $M$  is the total number of combinatorial variables. In this context, the Overlap kernel measures the extent to which variables in the two input vectors share the same categories.
A related kernel is the Transformed Overlap (TO) kernel [5] applying the exponential function to the output of the Overlap kernel. This transformation enhances the kernel's expressive power, enabling it to model more complex functions [5]. Other kernels tailored for combinatorial inputs include the SSK [29, 30], the Diffusion kernel (Diff.) [2], and the Hamming embedding via dictionary kernel (HED) [6], which are all supported in our framework.
Given a dataset  $\mathcal{D} = \{\pmb{x}_i, y_i\}_{i=1}^n$  with Gaussian-corrupted observations  $y_i = f(\pmb{x}_i) + \epsilon_i$ , with  $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$ , and given a GP prior, the posterior of the black-box function value at a test point  $\pmb{x}_{\mathrm{test}}$  denoted as  $f(\pmb{x}_{\mathrm{test}}) | \mathcal{D}, \pmb{\theta}$ , is also a Gaussian distribution  $\mathcal{N}\left(\mu_{\pmb{\theta}}(\pmb{x}_{\mathrm{test}}), \sigma_{\pmb{\theta}}^2(\pmb{x}_{\mathrm{test}})\right)$ . For brevity, we defer to Appendix B the analytic expressions for  $\mu_{\pmb{\theta}}(\pmb{x}_{\mathrm{test}})$  and  $\sigma_{\pmb{\theta}}^2(\pmb{x}_{\mathrm{test}})$  along with the method to learn the optimal kernel hyperparameters  $\pmb{\theta}^*$  by minimizing the negative log-likelihood.
3.2 The Acquisition Function
The acquisition function plays a crucial role in BO as it approximates the utility of evaluating the black-box function at a specific input  $x \in \mathcal{X}$ . Since the black-box function is unknown, the acquisition function considers both the estimated value of the objective function and its associated uncertainty. This enables the acquisition function to effectively balance exploration of the search space, gathering more information about the underlying objective function and exploiting currently promising regions likely to contain the optimal solution. Ultimately, the acquisition function is a criterion for selecting the next point to evaluate in the optimization process.
The Expected Improvement (EI) [59, 60] acquisition function measures the utility of new query points by evaluating the expected gain compared to the function values observed so far, considering
the uncertainty of the model's posterior. During each iteration  $i$  of the optimization process, let  $y_{1:i-1}^{\star}$  denote the best black-box function value in the dataset  $\mathcal{D}_{i-1}$ . The EI function is defined as:
$$
\alpha^ {(\mathrm {E I})} (\pmb {x}) = \mathbb {E} _ {f (\pmb {x}) | \mathcal {D} _ {i - 1}, \pmb {\theta} ^ {\star}} \left[ \max \{y _ {1: i - 1} ^ {\star} - f (\pmb {x}), 0 \} \right],
$$
where the expectation is taken with respect to the posterior of a trained model. Our framework supports EI as well as other popular acquisitions such as Probability of Improvement (PI) [61], lower confidence bound (LCB) [62], and Thompson sampling (TS) [63], all described in Appendix B.3.
3.3 Acquisition Optimization Methods
The acquisition optimization method plays a vital role in BO as it determines the next evaluation point  $\pmb{x}_i$  by optimizing the acquisition function  $\alpha(\pmb{x}|\mathcal{D}_{i-1})$ . In general, maximizing the acquisition function to find  $\pmb{x}_i$  is an optimization problem defined globally over the entire search space:
$$
\boldsymbol {x} _ {i} = \underset {\boldsymbol {x} \in \mathcal {X}} {\arg \max } \alpha (\boldsymbol {x} | \mathcal {D} _ {i - 1}). \tag {2}
$$
When dealing with a continuous search space, optimizing the acquisition function is relatively straightforward and can be performed by using off-the-shelf first or second-order optimization methods with random restarts (refer to [64] for a comprehensive study on this topic). However, when dealing with combinatorial inputs, the absence of a defined gradient hinders the direct application of gradient-based optimization methods [65]. To address this challenge, various zero-order methods have been proposed for maximizing the acquisition function in combinatorial spaces. These methods encompass a range of global optimization algorithms and heuristics, including Hill Climbing (HC) [5], exhaustive Local Search (LS) [2], SA [1], GA [7], and MAB [3], which we include in our library.
These acquisition optimizers are often applied as global optimization algorithms to solve Equation 2. However, just like in continuous spaces, when the dimensionality of the search space is high, the surrogate may struggle to accurately model the black box over the entire search space. Taking inspiration from related work in BO for continuous spaces [66], some recent combinatorial BO algorithms, including Casmopolitan [5], introduce the notion of a TR to constrain the acquisition optimization procedure in the context of MCBO. In Casmopolitan, the TR around the best input found so far in the current trust region,  $\boldsymbol{x}^{\mathrm{TR}}$ , is defined as
$$
\operatorname {T R} \left(\boldsymbol {x} ^ {\mathrm {T R}}\right) = \left\{\boldsymbol {x} \in \mathcal {X} \text {s . t .} d _ {h} \left(\boldsymbol {x} _ {h} ^ {\mathrm {T R}}, \boldsymbol {x} _ {h}\right) \leq L _ {h} \text {a n d} d _ {L _ {n}} \left(\boldsymbol {x} _ {n} ^ {\mathrm {T R}}, \boldsymbol {x} _ {n}\right) \leq 1 \right\}, \tag {3}
$$
where  $\pmb{x}_h^{\mathrm{TR}}$  and  $\pmb{x}_h$  are vectors containing all the combinatorial inputs in  $\pmb{x}^{\mathrm{TR}}$  and  $\pmb{x}$  respectively,  $d_h(\cdot ,\cdot)$  is the Hamming distance [5] and  $L_{h}$  is the size of the TR for combinatorial variables. Similarly,  $\pmb{x}_n^{\mathrm{TR}}$  and  $\pmb{x}_n$  are vectors containing all the numeric and discrete variables in  $\pmb{x}^{\mathrm{TR}}$  and  $\pmb{x}$ , respectively,  $d_{L_n}(\cdot ,\cdot)$  is the maximum of the component-wise distance for numeric and discrete variables normalized by dimensional length scales  $L_{n}\in \mathbb{R}_{+}^{\dim (x_{n})}$  for the numeric and discrete variables [5].
4 The MCBO Framework
To facilitate code reusability and enable benchmarking on a standardized set of tasks, we introduce the MCBO framework, whose source code is open-source under the MIT license. Our ready-to-use software provides 1) An API for defining BO primitives, 2) Multiple primitives from key existing MCBO baselines, 3) A BO constructor to effortlessly combine primitives into new algorithms, 4) A variety of mixed-type and combinatorial BO and non-BO baselines, 5) A suite of standard and novel mixed-type and combinatorial benchmarks, and 6) An API for defining novel optimization problems.
We structure the MCBO framework in a modular fashion to allow for the rapid prototyping of new solutions by combining existing BO primitives. We build MCBO on top of PyTorch [44] to make it compatible with the rich ecosystem of code developed for training various regression models. The entire library structure naturally revolves around seven Python classes; the SearchSpace, TaskBase, ModelBase, AcqBase, AcqOptimizerBase, OptimizerBase, and the BoBase class.
4.1 Defining Optimization Problems
To frame the optimization of a black-box within our framework, we need to create a task class (inheriting from TaskBase) that implements two methods: get_search_space.params and evaluate. The latter implements the call to the black-box, while the former provides the list of variables that compose the optimization domain of the black-box. We build a search space (an instance of SearchSpace) by providing this list of variables with their names, types, and any additional information needed, such as the categories they can take for categorical variables. As an example, consider a combinatorial optimization problem whose search space contains five categorical variables (with categories "A", "B", "C", and "D") and for which the black-box function is already defined as black_box scripted in a python script. In the MCBO framework, we can create the corresponding task as follows:
class BlackBox(TaskBase): def get_search_space(self) -> List[Dict[str, Any] categories  $=$  ['A', 'B', 'C', 'D'] params  $=$  ['name': f'x{i}', 'type': 'nominal', 'categories': categories} for i in range(5)] return params def evaluate(self, x: pd.DataFrame) -> np.ndarray: return black_box.script(x)   
black_box  $\equiv$  BlackBox()   
search_space  $\equiv$  black_box.get_search_space() # get an instance of SearchSpace