SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ErikOrmHellsten
erik.hellsten@cs.lth.se
Gaussian processes are the model of choice in Bayesian optimization and active learning. Yet, they are highly dependent on cleverly chosen hyperparameters to reach their full potential, and little effort is devoted to finding good hyperparameters in the literature. We demonstrate the impact of selecting good hyperparameters for GPs and present two acquisition functions that explicitly prioritize hyperparameter learning. Statistical distance-based Active Learning (SAL) considers the average disagreement between samples from the posterior, as measured by a statistical distance. SAL outperforms the state-of-the-art in Bayesian active learning on several test functions. We then introduce Self-Correcting Bayesian Optimization (SCoreBO), which extends SAL to perform Bayesian optimization and active learning simultaneously. SCoreBO learns the model hyperparameters at improved rates compared to vanilla BO, while outperforming the latest Bayesian optimization methods on traditional benchmarks. Moreover, we demonstrate the importance of self-correction on atypical Bayesian optimization tasks.
Bayesian Optimization (BO) is a powerful paradigm for black-box optimization problems, i.e., problems that can only be accessed by pointwise queries. Such problems arise in many applications, ranging from including drug discovery [21] to configuration of combinatorial problem solvers [27, 28], hardware design [14, 43], hyperparameter tuning [11, 30, 33, 52], and robotics [4, 9, 40, 41].
Gaussian processes (GPs) are a popular choice as surrogate models in BO applications. Given the data, the model hyperparameters are typically estimated using either Maximum Likelihood or Maximum a Posteriori estimation (MAP) [49]. Alternatively, a fully Bayesian treatment of the hyperparameters [46, 55] removes the need to choose any single set through Monte Carlo integration. This procedure effectively considers all possible hyperparameter values under the current posterior, thereby accounting for hyperparameter uncertainty. However, the relationship between accurate GP hyperparameter estimation and BO performance has received little attention [3, 7, 58, 69, 71], and active reduction of hyperparameter uncertainty is not an integral part of any prevalent BO acquisition function. In contrast, the field of Bayesian Active Learning (BAL) contains multiple acquisition functions based solely on reducing hyperparameter-induced measures of uncertainty [26, 34, 50], and the broader field of Bayesian Experimental Design [1, 10, 48] revolves around acquisition of data to best learns the model parameters.
The importance of the GP hyperparameters in BO is illustrated in Fig. 1, which shows average simple regret over 20 optimization runs of 8-dimensional functions drawn from a Gaussian process prior. The curves correspond to the performance of Expected Improvement with noisy experiments (NEI) [36] acquisition function under a fully Bayesian hyperparameter treatment using NUTS [25]. Two prevalent hyperparameter priors, described in detail in App. B.1, as well as the true model hyperparameters, are used. Clearly, good model hyperparameters have substantial impact on BO performance, and BO methods could greatly benefit from estimating the model hyperparameters as accurately as possible. Furthermore, the hyperparameter estimation task can become daunting under complex problem setups, such as non-stationary objectives (spatially varying lengthscales, heteroskedasticity) [6, 13, 16, 56, 64], high-dimensional search spaces [15, 47], and additively decomposable objectives [19, 32]. The complexity of such problems warrants the use of more complex, task-specific surrogate models. In such settings, the success of the optimization may increasingly hinge on the presumed accuracy of the task-specific surrogate.
We proceed in two steps. We first introduce Statistical distance-based Active Learning (SAL), which improves Bayesian active learning by generalizing previous work [26, 50] and introduces a holistic measure of disagreement between the marginal posterior predictive distribution and each conditional posterior predictive. We consider the hyperparameter-induced disagreement between models in the acquisition function, thereby accelerating the learning of model hyperparameters. We then propose Self-Correcting Bayesian Optimization (SCoreBO), which builds upon SAL by explicitly learning the location of the optimizer in conjunction with model hyperparameters. This achieves accelerated hyperparameter learning and yields improved optimization performance on both conventional and exotic BO tasks. Formally, we make the following contributions:
Figure 1: Simple regret of using true hyperparameters, BoTorch (v.0.8.4 default) and lognormal hyperparameter priors with fully Bayesian hyperparameter treatment. The prior substantially impacts final performance, and correct hyperparameters yield vastly better results.
1. We introduce SAL, a novel and efficient acquisition function for hyperparameter-oriented Bayesian active learning based on statistical distances (Sec. 3.1),
2. We introduce SCoreBO, the first acquisition function for joint BO and hyperparameter learning (Sec. 3.2),
3. We display highly competitive performance on an array of conventional AL (Sec. 4.1) and BO tasks (Sec. 4.2), and demonstrate SCoreBOs, ability to enhance atypical models such as SAASBO [15] and HEBO [13], and identify decompositions in AddGPs [32](Sec. 4.3).
2.1 Gaussian processes
Gaussian processes (GPs) have become the model class of choice in most BO and active learning applications. They provide a distribution over functions  $f \sim \mathcal{GP}(m(\cdot), k(\cdot, \cdot))$  fully defined by the mean function  $m(\cdot)$  and the covariance function  $k(\cdot, \cdot)$ . Under this distribution, the value of the function  $f(x)$ , at a given point  $x$ , is normally distributed with a closed-form solution for the mean and variance. We assume that observations are perturbed by Gaussian noise, such that  $y_{x} = f(x) + \varepsilon$ ,  $\varepsilon \sim N(0, \sigma_{\varepsilon}^{2})$ . We also assume the mean function to be constant, such that the dynamics are fully determined by the covariance function  $k(\cdot, \cdot)$ .
To account for differences in variable importance, each dimension is individually scaled using length-scale hyperparameters  $\ell_i$ . For  $D$ -dimensional inputs  $\mathbf{x}$  and  $\mathbf{x}'$ , the distance  $r(\mathbf{x},\mathbf{x}')$  is subsequently computed as  $r^2 = \sum_{i=1}^{D}(x_i - x_i')^2/\ell_i^2$ . Along with the outputscale  $\sigma_f$ , the set  $\theta = \{\ell, \sigma_\varepsilon, \sigma_f\}$  comprises the set of hyperparameters that are conventionally learned. The likelihood surface for the GP hyperparameters is typically highly multi-modal [49, 70], where different modes represent different bias-variance trade-offs [49, 50]. To avoid having to choose a single mode, one can define a prior  $p(\theta)$  and marginalize with respect to the hyperparameters when performing predictions [35]. We outline fully Bayesian hyperparameter treatment in GPs App. G.1.
Bayesian Optimization (BO) seeks to maximize to a black-box function  $f$  over a compact domain  $\mathcal{X}$ ,
$$
\boldsymbol {x} ^ {*} \in \underset {\boldsymbol {x} \in \mathcal {X}} {\arg \max } f (\boldsymbol {x}), \tag {1}
$$
such that  $f$  can only be sampled point-wise through expensive, noisy evaluations  $y_{\boldsymbol{x}} = f(\boldsymbol{x}) + \varepsilon$ , where  $\varepsilon \sim \mathcal{N}(0, \sigma_{\varepsilon}^{2})$ . New configurations are chosen by optimizing an acquisition function, which uses the surrogate model to quantify the utility of evaluating new points in the search space. Examples of such heuristics are Expected Improvement (NEI) [8, 31] and Upper Confidence Bound (UCB) [3, 57, 60]. More sophisticated look-ahead approaches include Knowledge Gradient (KG) [17, 68] as well as a class of particular importance for our approach - the information-theoretic acquisition function class. These acquisition functions consider a mutual information objective to select the next query,
$$
\alpha_ {\mathrm {M I}} (\boldsymbol {x}) = I \left(y _ {\boldsymbol {x}}; * \mid \mathcal {D} _ {n}\right), \tag {2}
$$
where  $\ast$  can entail either the optimum  $x^{*}$  as in (Predictive) Entropy Search (ES/PES) [23, 24], the optimal value  $f^{*}$  as in Max-value Entropy Search (MES) [42, 59, 65] or the tuple  $(x^{*}, f^{*})$ , used in Joint Entropy Search (JES) [29, 61]. FITBO [51] shares similarities with our work, in that the optimal value is governed by a hyperparameter, in their case of a transformed GP.
Within BO, the fully Bayesian hyperparameter treatment is conventionally extended from the predictive posterior to the acquisition function such that for  $M$  models with hyperparameters  $\theta_{m}, m \in \{1, \dots, M\}$  sampled from the posterior over hyperparameters  $p(\boldsymbol{\theta}|\mathcal{D})$ , the acquisition function  $\alpha$  is computed as an expectation over the hyperparameters [46, 55]
$$
\alpha (\boldsymbol {x} | \mathcal {D}) = \mathbb {E} _ {\boldsymbol {\theta}} [ \alpha (\boldsymbol {x} | \boldsymbol {\theta}, \mathcal {D}) ] \approx \frac {1}{M} \sum_ {m = 1} ^ {M} \alpha (\boldsymbol {x} | \boldsymbol {\theta} _ {m}, \mathcal {D}) \quad \boldsymbol {\theta} _ {m} \sim p (\boldsymbol {\theta} | \mathcal {D}). \tag {3}
$$
This is also the definition of fully Bayesian treatment considered in this work.
2.3 Bayesian Active Learning
In contrast to BO, which aims to find a maximizer to an unknown function, Active Learning (AL) [54] seeks to accurately learn the black-box function globally. Thus, the objective is to minimize the expected prediction loss. AL acquisition functions are classified as either decision-theoretic, which minimize the prediction loss over a validation set, or information-theoretic, which minimize the space of plausible models given the observed data [26, 37].
In the information-theoretic category, Active Learning McKay (ALM) [37] selects the point with the highest Shannon Entropy, which for GPs amounts to selecting the point with the highest variance. Under fully Bayesian hyperparameter treatment, it is referred to as Bayesian ALM (BALM). Bayesian Active Learning by Disagreement (BALD) [26] was among the first Bayesian active learning approaches to explicitly focus on learning the model hyperparameters. It approximates the reduction in entropy over the GP hyperparameters from observing a new data point
$$
\alpha_ {\mathrm {B A L D}} (\boldsymbol {x}) = I \left(y _ {\boldsymbol {x}}; \boldsymbol {\theta} | \mathcal {D}\right) = \mathrm {H} \left(p \left(y _ {\boldsymbol {x}} \mid \mathcal {D}\right)\right) - \mathbb {E} _ {\boldsymbol {\theta}} \left[ \mathrm {H} \left(p \left(y _ {\boldsymbol {x}} \mid \boldsymbol {\theta}, \mathcal {D}\right)\right) \right] \tag {4}
$$
and was later extended to deep Bayesian active learning [34] and active model (kernel) selection [18]. Lastly, Riis et al. [50] propose a Bayesian Query-by-Committee (BQBC) strategy. BQBC queries where the variance  $V$  of the GP mean is the largest, with respect to changing model hyperparameters:
$$
\alpha_ {B Q B C} (\boldsymbol {x}) = V _ {\boldsymbol {\theta}} \left[ \mu_ {\boldsymbol {\theta}} (\boldsymbol {x} | \mathcal {D}) \right] = \mathbb {E} _ {\boldsymbol {\theta}} \left[ \left(\mu_ {\boldsymbol {\theta}} (\boldsymbol {x} | \mathcal {D}) - \mu (\boldsymbol {x} | \mathcal {D})\right) ^ {2} \right], \tag {5}
$$
where  $\mu (\pmb {x})$  is the marginal posterior mean at  $\pmb{x}$ , and  $\mu_{\theta}(\pmb {x})$  is the posterior mean conditioned on  $\pmb{\theta}$ . As such, BQBC queries the location which maximizes the average distance between the marginal posterior and the conditionals according to some distance metric (here, the posterior mean), henceforth referred to as hyperparameter-induced posterior disagreement. However, disagreement in mean alone does not fully capture hyperparameter-induced disagreement. Thus, [50] also presents Query-by-Mixture of Gaussian Processes (QBMGP), that adds the BALM criterion to the BQBC acquisition function.
2.4 Statistical Distances
A statistical distance quantifies the distance between two statistical objects. We focus on three (semimetrics, which have closed forms for Gaussian random variables. The closed forms expressions, as well as additional intuition on their interaction with Gaussian random variables, can be found in App. G.2.
The Hellinger distance is a dissimilarity measure between two probability distributions which has previously been employed in the context of BO-driven automated model selection by Malkomes et al. [39]. For two probability distributions  $p$  and  $q$ , it is defined as
$$
H ^ {2} (p, q) = \frac {1}{2} \int_ {\mathcal {X}} \left(\sqrt {p (x)} - \sqrt {q (x)}\right) ^ {2} \lambda d x, \tag {6}
$$
for some auxiliary measure  $\lambda$  under which both  $p$  and  $q$  are absolutely continuous.
The Wasserstein distance is dissimilarity metric between two distributions describing the average distance one distribution has to be moved to morph into another. The Wasserstein- $k$  distance is defined as
$$
W _ {k} (p, q) = \left(\int_ {0} ^ {1} \left| F _ {q} (x) - F _ {p} (x) \right| ^ {k} d x\right) ^ {1 / k} \tag {7}
$$
where, in this work, we focus on the case where  $k = 2$ .
The KL divergence The KL divergence is a standard asymmetrical measure for dissimilarity between probability distributions. For two probability distributions  $P$  and  $Q$ , it is given by  $\mathcal{D}_{KL}(P\parallel Q) = \int_{\mathcal{X}}P(x)\log (P(x) / Q(x))dx$ . The distances in Eq. (6), Eq. (16) and the KL divergence are used for the acquisition functions presented in Sec. 3.
In Sec. 3.1, we introduce SAL, a novel family of metrics for BAL. In Sec. 3.2, we extend this to SCoreBO, the first acquisition function for joint BO and hyperparameter-oriented active learning, inspired by information-theoretic BO acquisition functions. In Sec. 3.3, we demonstrate how to efficiently approximate different types of statistical distances within the SAL context.
3.1 Statistical distance-based Active Learning
In active learning for GPs, it is important to efficiently learn the correct model hyperparameters. By measuring where the posterior hyperparameter uncertainty causes high disagreement in model output, the search can be focused on where this uncertainty has a high impact. However, considering only the posterior disagreement in mean, as in BQBC, is overly restrictive as it does not fully utilize the available distributions for the hyperparameters. For example, it ignores uncertainty in the outputscale hyperparameter of the Gaussian process, which disincentives exploration. As such, we propose to generalize the acquisition function in Eq. (5) to instead consider the posterior disagreement as measured by any statistical distance. Locations where the posterior distribution changes significantly as a result of model uncertainty are good points to query, in order to quickly learn the model hyperparameters. When an observation at such a location is obtained, hyperparameters which predicted that observation poorly will have a substantially smaller likelihood, which in turn aids hyperparameter convergence. The resulting SAL acquisition function is as follows:
$$
\alpha_ {S A L} (\boldsymbol {x}) = \mathbb {E} _ {\boldsymbol {\theta}} \left[ d \left(p \left(y _ {\boldsymbol {x}} \mid \boldsymbol {\theta}, \mathcal {D}\right), p \left(y _ {\boldsymbol {x}} \mid \mathcal {D}\right)\right) \right] \approx \frac {1}{M} \sum_ {m = 1} ^ {M} d \left(p \left(y _ {\boldsymbol {x}} \mid \boldsymbol {\theta} _ {m}, \mathcal {D}\right), p \left(y _ {\boldsymbol {x}} \mid \mathcal {D}\right)\right), \tag {8}
$$
where  $M$  is the number of hyperparameter samples drawn from its associated posterior,  $\theta_{m} \sim p(\theta | \mathcal{D})$ ,  $\pmb{\theta} = \{\ell, \sigma_{f}, \sigma_{\varepsilon}\}$ , and  $d$  is a statistical distance. Notably, SAL generalizes both BQBC and BALD, which are exactly recovered by choosing the semimetric to the difference in mean or the forward KL divergence, with a short proof for the latter in App. F:
Proposition 1. SAL equipped with the KL-divergence is equivalent to BALD.
Figure 2: Marginal posterior (top left, grey in other plots in top row),  $\alpha_{SAL}$  using the Hellinger distance (bottom left, black), and the three conditional GPs (blue, orange, green) and their marginal contribution to the total acquisition function (bottom row). The large disagreement in noise level and lengthscale, primarily caused by the orange GP (large noise, long lengthscale), makes  $\alpha_{SAL}$  query the lowest-valued point for a second time (selected location as vertical dashed line in the leftmost plot) to determine the mean and variance at that location.
Fig. 2 visualizes the SAL acquisition function. The marginal posterior (left) is made up of three vastly different conditional posteriors with hyperparameters sampled from  $p(\theta | \mathcal{D})$  - one with high outputscale (blue), one with very high noise (orange), and one with short lengthscale (green). For each of the blue, orange and green conditionals, the distance to the marginal posterior is computed. Intuitively, disagreement in noise level  $\sigma_{\varepsilon}$  can cause large posterior disagreement at already queried locations. Similarly, uncertainty in outputscale  $\sigma_f$  between posteriors will yield disagreement in large-variance regions, which will result in global variance reduction. Compared to other active learning acquisition functions, SAL carries distinct advantages: it has incentive to query the same location multiple times to estimate noise levels, and accomplishes the typical active learning objectives of predictive accuracy and global exploration by alleviating uncertainty over the lengthscales and outputscale of the GP. As we show in our experiments (Sec. 4.1, App. D), SAL yields superior predictions and reduces hyperparameter uncertainty at drastically improved rates.
3.2 Self-Correcting Bayesian Optimization
Equipped with the SAL objective from Eq. (8), we have an intuitive measure for the hyperparameter-induced posterior disagreement, which incentivizes hyperparameter learning by querying locations where disagreement is the largest. However, it does not inherently carry an incentive to optimize the function. To inject an optimization objective into Eq. (8), we draw inspiration from information-theoretic BO and further condition on samples of the optimum. Conditioning on potential optima yields an additional source of disagreement reserved for promising regions of the search space.
We consider  $(\pmb{x}^{*},f^{*})$ , representing the global optimum and optimal value considered in JES [29, 61], as hyperparameters. When conditioning on  $(\pmb{x}^{*},f^{*})$ , we condition on an additional observation, which displaces the mean and reduces the variance at  $x^{*}$ . Moreover, the posterior over  $f$  becomes an upper truncated Gaussian, reducing the variance and pushing the mean marginally downwards in uncertain regions far away from the optimum as visualized in Fig. 3. Consequently, sampling and conditioning on  $(\pmb{x}^{*},f^{*})$  introduces an additional source of disagreement between the marginal posterior and the conditionals globally. The optimizer  $(\pmb{x}^{*},f^{*})$  is obtained through posterior sampling [67]. For brevity, we hereafter denote  $(\pmb{x}^{*},f^{*})$  by  $\ast$ . The resulting SCoreBO acquisition function is
$$
\alpha_ {S C} (\boldsymbol {x}) = \mathbb {E} _ {\boldsymbol {\theta}, *} [ d (p (y _ {\boldsymbol {x}} | \mathcal {D}), p (y _ {\boldsymbol {x}} | \boldsymbol {\theta}, *, \mathcal {D})) ]. \tag {9}
$$
The joint posterior  $p(\pmb{\theta}, \ast |\mathcal{D}) = p(\ast |\pmb{\theta}, \mathcal{D})p(\pmb{\theta}|\mathcal{D})$  used for the expectation in Eq. (9) can be approximated by hierarchical sampling. We first draw  $M$  hyperparameters  $\pmb{\theta}$  and thereafter  $N$  optimizers  $\ast |\pmb{\theta}$ . As such, the expression for the SCoreBO acquisition function is:
$$
\alpha (\boldsymbol {x}) \approx \frac {1}{N M} \sum_ {m = 1} ^ {M} \sum_ {n = 1} ^ {N} d \left(p \left(y _ {\boldsymbol {x}} \mid \mathcal {D}\right), p \left(y _ {\boldsymbol {x}} \mid \boldsymbol {\theta} _ {m}, * _ {\boldsymbol {\theta} _ {m, n}}, \mathcal {D}\right)\right), \tag {10}
$$
where  $N$  is the number of optimizers sampled per hyperparameter set. Notably, while the acquisition function in (9) considers the optimizer  $(x^{*},f^{*})$ , SCoreBO is not restricted to employing that quantity alone. Drawing parallels to PES and MES, we can also choose to condition on either  $x^{*}$  or  $f^{*}$  alone in place of  $(x^{*},f^{*})$ . Doing so introduces a smaller disagreement in the posterior at the conditioned
Figure 3: Approximate marginal posterior after having conditioned on  $(\pmb{x}^{*},f^{*})$  (top left),  $\alpha_{SC}$  using the Hellinger distance (bottom left), the three conditional truncated posteriors and their marginal contribution to the total acquisition function for the same iteration as Fig. 2. Conditioning on  $(\pmb{x}^{*},f^{*})$  (marked as  $\star$ , drawn from function samples in dashed) introduces additional disagreement between the marginal posterior and the sampled GPs in promising regions as a result of conditioning. In the figure, we marginalize over  $M = 3$  sets of hyperparameters and  $N = 2$  optimizers per GP, where each optimizer's contribution to the acquisition function is visible under its corresponding GP. Note that, since function draws are noiseless, the conditioned optimum does not need to surpass the best noisy observation in value. This phenomenon is most notable in (orange).
Algorithm 1 SCoreBO iteration
1: Input: Number of hyperparameter sets  $M$ , number of sampled optima  $N$ , current data  $\mathcal{D}$
2: Output: Next query location  $x'$ .
3: for  $m \in \{1, \ldots, M\}$  do
4:  $\pmb{\theta}_{m} \sim p(\pmb{\theta}|\mathcal{D})$
5: for  $n\in \{1,\ldots ,N\}$  do
6:  $\star_{\pmb{\theta}_{m,n}} \gets \max f_{\pmb{\theta}_{m,n}}$ , where  $f_{\pmb{\theta}_{m,n}} \sim p(f|\pmb{\theta}_{m}, \mathcal{D})$  {Draw  $n$  optima for each  $\pmb{\theta}_{m}$ }
7:  $p(y_{\boldsymbol{x}}|\pmb{\theta}_{m},*\pmb{\theta}_{m,n},\mathcal{D})\gets \mathrm{CondGP}(*_{\pmb{\theta}_{m,n}},\pmb{\theta}_{m},\mathcal{D})$  {Condition GPs on each optimum}
8: end for
9: end for
10:  $\pmb{x}^{\prime} = \arg \max \alpha(\pmb{x})$  {Defined in Eq. (10)}
location  $x^{*}$ , thus decreasing the acquisition value there. This will in turn decrease the emphasis that SCoreBO puts on optimization, relative to hyperparameter learning. In Fig. 3, the SCoreBO acquisition function is displayed for the same scenario as in Fig. 2. By conditioning on  $N = 2$  optimizers per GP, we obtain  $N \times M$  posteriors (displaying the posterior for one out of two optimizers, i.e. the left star in (blue), in Fig. 3). The mean is pushed upwards around the extra observation and the posterior predictive distribution over  $f$  is truncated as it is now upper bounded by  $f^{*}$ . While the preferred location under SAL is still attractive, the best location to query is now one that is more likely to be optimal, but still good under SAL.
Algorithm 1 displays how the involved densities are formed for one iteration of SCoreB0. For each hyperparameter set, a number of optima are sampled and individually conditioned on (CondGP) given the current data and hyperparameter set. After this procedure is completed for all hyperparameter sets, the statistical distance between each conditional posterior and the marginal is computed. The conditioning on the fantasized data point involves a rank-1 update of  $\mathcal{O}(n^2)$  of the GP for each draw. As such, the complexity of constructing the acquisition functions is  $\mathcal{O}(MNn^2)$  for  $M$  models,  $N$  optima per model and  $n$  data points. We utilize NUTS [25] for the MCMC involved with the fully Bayesian treatment, at a cost of  $\mathcal{O}(Dn^3)$  per sample.
3.3 Approximation of Statistical Distances