Title: Self-Correcting Bayesian Optimization through Bayesian Active Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian Optimization

Bayesian Active Learning

Gaussian Processes
Abstract: Gaussian processes are the model of choice in Bayesian optimization and active learning. Yet, they are highly dependent on cleverly chosen hyperparameters to reach their full potential, and little effort is devoted to finding good hyperparameters in the literature. We demonstrate the impact of selecting good hyperparameters for GPs and present two acquisition functions that explicitly prioritize hyperparameter learning. Statistical distance-based Active Learning (SAL) considers the average disagreement between samples from the posterior, as measured by a statistical distance. SAL outperforms the state-of-the-art in Bayesian active learning on several test functions. We then introduce Self-Correcting Bayesian Optimization (SCoreBO), which extends SAL to perform Bayesian optimization and active learning simultaneously. SCoreBO learns the model hyperparameters at improved rates compared to vanilla BO, while outperforming the latest Bayesian optimization methods on traditional benchmarks. Moreover, we demonstrate the importance of self-correction on atypical Bayesian optimization tasks.

Self-Correcting Bayesian Optimization through Bayesian Active Learning

Abstract

Gaussian processes are the model of choice in Bayesian optimization and active learning. Yet, they are highly dependent on cleverly chosen hyperparameters to reach their full potential, and little effort is devoted to finding good hyperparameters in the literature. We demonstrate the impact of selecting good hyperparameters for GPs and present two acquisition functions that explicitly prioritize hyperparameter learning. Statistical distance-based Active Learning ( SAL ) considers the average disagreement between samples from the posterior, as measured by a statistical distance. SAL outperforms the state-of-the-art in Bayesian active learning on several test functions. We then introduce Self-Correcting Bayesian Optimization ( SCoreBO ), which extends SAL to perform Bayesian optimization and active learning simultaneously. SCoreBO learns the model hyperparameters at improved rates compared to vanilla BO, while outperforming the latest Bayesian optimization methods on traditional benchmarks. Moreover, we demonstrate the importance of self-correction on atypical Bayesian optimization tasks.

1 Introduction

Bayesian Optimization (BO) is a powerful paradigm for black-box optimization problems, i.e., problems that can only be accessed by pointwise queries. Such problems arise in many applications, ranging from including drug discovery [21] to configuration of combinatorial problem solvers [27, 28], hardware design [14, 43], hyperparameter tuning [11, 30, 33, 52], and robotics [4, 9, 40, 41].

Gaussian processes (GPs) are a popular choice as surrogate models in BO applications. Given the data, the model hyperparameters are typically estimated using either Maximum Likelihood or Maximum a Posteriori estimation (MAP) [49]. Alternatively, a fully Bayesian treatment of the hyperparameters [46, 55] removes the need to choose any single set through Monte Carlo integration. This procedure effectively considers all possible hyperparameter values under the current posterior, thereby accounting for hyperparameter uncertainty. However, the relationship between accurate GP hyperparameter estimation and BO performance has received little attention [3, 7, 58, 69, 71], and active reduction of hyperparameter uncertainty is not an integral part of any prevalent BO acquisition function. In contrast, the field of Bayesian Active Learning (BAL) contains multiple acquisition functions based solely on reducing hyperparameter-induced measures of uncertainty [26, 34, 50], and the broader field of Bayesian Experimental Design [1, 10, 48] revolves around acquisition of data to best learns the model parameters.

The importance of the GP hyperparameters in BO is illustrated in Fig. 1, which shows average simple regret over 20 optimization runs of 8-dimensional functions drawn from a Gaussian process prior. The curves correspond to the performance of Expected Improvement with noisy experiments ( NEI ) [36] acquisition function under a fully Bayesian hyperparameter treatment using NUTS [25]. Two prevalent hyperparameter priors, described in detail in App. B.1, as well as the true model hyperparameters, are used. Clearly, good model hyperparameters have substantial impact on BO performance, and BO methods could greatly benefit from estimating the model hyperparameters as accurately as possible. Furthermore, the hyperparameter estimation task can become daunting under complex problem setups, such as non-stationary objectives (spatially varying lengthscales, heteroskedasticity) [6, 13, 16, 56, 64], high-dimensional search spaces [15, 47], and additively decomposable objectives [19, 32]. The complexity of such problems warrants the use of more complex, task-specific surrogate models. In such settings, the success of the optimization may increasingly hinge on the presumed accuracy of the task-specific surrogate.

We proceed in two steps. We first introduce Statistical distance-based Active Learning ( SAL ), which improves Bayesian active learning by generalizing previous work [26, 50] and introduces a holistic measure of disagreement between the marginal posterior predictive distribution and each conditional posterior predictive. We consider the hyperparameter-induced disagreement between models in the acquisition function, thereby accelerating the learning of model hyperparameters. We then propose Self-Correcting Bayesian Optimization ( SCoreBO ), which builds upon SAL by explicitly learning the location of the optimizer in conjunction with model hyperparameters. This achieves accelerated hyperparameter learning and yields improved optimization performance on both conventional and exotic BO tasks. Formally, we make the following contributions:

0

20

40

60

80

100

120

140

160

Iteration

0

1

2

3

4

Simple Regret

GP-sample (8D)

EI - Wide LogNormal Prior

EI - Default BoTorch Prior

EI - True Hyperparameters

NEI - Wide LogNormal Prior

NEI - Default BoTorch Prior

NEI - True Hyperparameters

We introduce SAL , a novel and efficient acquisition function for hyperparameter-oriented Bayesian active learning based on statistical distances (Sec. 3.1),

We introduce SCoreBO , the first acquisition function for joint BO and hyperparameter learning (Sec. 3.2),

We display highly competitive performance on an array of conventional AL (Sec. 4.1) and BO tasks (Sec. 4.2), and demonstrate SCoreBO s , ability to enhance atypical models such as SAASBO [15] and HEBO [13], and identify decompositions in AddGPs [32](Sec. 4.3).

2 Background

2.1 Gaussian processes

Gaussian processes (GPs) have become the model class of choice in most BO and active learning applications. They provide a distribution over functions f ' GP p m p¨q , k p¨ , ¨qq fully defined by the mean function m p¨q and the covariance function k p¨ , ¨q . Under this distribution, the value of the function f p x q , at a given point x , is normally distributed with a closed-form solution for the mean and variance. We assume that observations are perturbed by Gaussian noise, such that y x ' f p x q ` ε, ε ' N p 0 , σ 2 ε q . We also assume the mean function to be constant, such that the dynamics are fully determined by the covariance function k p¨ , ¨q .

To account for differences in variable importance, each dimension is individually scaled using lengthscale hyperparameters ℓ i . For D -dimensional inputs x and x 1 , the distance r p x , x 1 q is subsequently computed as r 2 ' ř D i ' 1 p x i ´ x 1 i q 2 { ℓ 2 i . Along with the outputscale σ f , the set θ ' t ℓ , σ ε , σ f u comprises the set of hyperparameters that are conventionally learned. The likelihood surface for the GP hyperparameters is typically highly multi-modal [49, 70], where different modes represent different bias-variance trade-offs [49, 50]. To avoid having to choose a single mode, one can define a prior p p θ q and marginalize with respect to the hyperparameters when performing predictions [35]. We outline fully Bayesian hyperparameter treatment in GPs App. G.1.

2.2 Bayesian Optimization

Bayesian Optimization (BO) seeks to maximize to a black-box function f over a compact domain X ,

such that f can only be sampled point-wise through expensive, noisy evaluations y x ' f p x q ` ε , where ε ' N p 0 , σ 2 ε q . New configurations are chosen by optimizing an acquisition function , which uses the surrogate model to quantify the utility of evaluating new points in the search space. Examples of such heuristics are Expected Improvement ( NEI ) [8, 31] and Upper Confidence Bound ( UCB ) [3, 57, 60]. More sophisticated look-ahead approaches include Knowledge Gradient ( KG ) [17, 68] as well as a class of particular importance for our approach - the information-theoretic acquisition function class. These acquisition functions consider a mutual information objective to select the next query,

where ˚ can entail either the optimum x ˚ as in (Predictive) Entropy Search ( ES / PES ) [23, 24], the optimal value f ˚ as in Max-value Entropy Search ( MES ) [42, 59, 65] or the tuple p x ˚ , f ˚ q , used in Joint Entropy Search ( JES ) [29, 61]. FITBO [51] shares similarities with our work, in that the optimal value is governed by a hyperparameter, in their case of a transformed GP.

Within BO, the fully Bayesian hyperparameter treatment is conventionally extended from the predictive posterior to the acquisition function such that for M models with hyperparameters θ m , m P t 1 , . . . , M u sampled from the posterior over hyperparameters p p θ | D q , the acquisition function α is computed as an expectation over the hyperparameters [46, 55]

This is also the definition of fully Bayesian treatment considered in this work.

2.3 Bayesian Active Learning

In contrast to BO, which aims to find a maximizer to an unknown function, Active Learning (AL) [54] seeks to accurately learn the black-box function globally. Thus, the objective is to minimize the expected prediction loss. AL acquisition functions are classified as either decision-theoretic , which minimize the prediction loss over a validation set, or information-theoretic , which minimize the space of plausible models given the observed data [26, 37].

In the information-theoretic category, Active Learning McKay ( ALM ) [37] selects the point with the highest Shannon Entropy, which for GPs amounts to selecting the point with the highest variance. Under fully Bayesian hyperparameter treatment, it is referred to as Bayesian ALM ( BALM ). Bayesian Active Learning by Disagreement ( BALD ) [26] was among the first Bayesian active learning approaches to explicitly focus on learning the model hyperparameters. It approximates the reduction in entropy over the GP hyperparameters from observing a new data point

and was later extended to deep Bayesian active learning [34] and active model (kernel) selection [18]. Lastly, Riis et al. [50] propose a Bayesian Query-by-Committee ( BQBC ) strategy. BQBC queries where the variance V of the GP mean is the largest, with respect to changing model hyperparameters:

where µ p x q is the marginal posterior mean at x , and µ θ p x q is the posterior mean conditioned on θ . As such, BQBC queries the location which maximizes the average distance between the marginal posterior and the conditionals according to some distance metric (here, the posterior mean), henceforth referred to as hyperparameter-induced posterior disagreement . However, disagreement in mean alone does not fully capture hyperparameter-induced disagreement. Thus, [50] also presents Query-by-Mixture of Gaussian Processes ( QBMGP ), that adds the BALM criterion to the BQBC acquisition function.

2.4 Statistical Distances

A statistical distance quantifies the distance between two statistical objects. We focus on three (semi)metrics, which have closed forms for Gaussian random variables. The closed forms expressions, as well as additional intuition on their interaction with Gaussian random variables, can be found in App. G.2.

The Hellinger distance is a dissimilarity measure between two probability distributions which has previously been employed in the context of BO-driven automated model selection by Malkomes et al. [39]. For two probability distributions p and q , it is defined as

for some auxiliary measure λ under which both p and q are absolutely continuous.

The Wasserstein distance is dissimilarity metric between two distributions describing the average distance one distribution has to be moved to morph into another. The Wassersteink distance is defined as

where, in this work, we focus on the case where k ' 2 .

The KL divergence The KL divergence is a standard asymmetrical measure for dissimilarity between probability distributions. For two probability distributions P and Q , it is given by D KL p P || Q q ' ş X P p x q log p P p x q{ Q p x qq dx . The distances in Eq. (6), Eq. (16) and the KL divergence are used for the acquisition functions presented in Sec. 3.

3 Methodology

In Sec. 3.1, we introduce SAL , a novel family of metrics for BAL. In Sec. 3.2, we extend this to SCoreBO , the first acquisition function for joint BO and hyperparameter-oriented active learning, inspired by information-theoretic BO acquisition functions. In Sec. 3.3, we demonstrate how to efficiently approximate different types of statistical distances within the SAL context.

3.1 Statistical distance-based Active Learning

In active learning for GPs, it is important to efficiently learn the correct model hyperparameters. By measuring where the posterior hyperparameter uncertainty causes high disagreement in model output, the search can be focused on where this uncertainty has a high impact. However, considering only the posterior disagreement in mean, as in BQBC , is overly restrictive as it does not fully utilize the available distributions for the hyperparameters. For example, it ignores uncertainty in the outputscale hyperparameter of the Gaussian process, which disincentives exploration. As such, we propose to generalize the acquisition function in Eq. (5) to instead consider the posterior disagreement as measured by any statistical distance. Locations where the posterior distribution changes significantly as a result of model uncertainty are good points to query, in order to quickly learn the model hyperparameters. When an observation at such a location is obtained, hyperparameters which predicted that observation poorly will have a substantially smaller likelihood, which in turn aids hyperparameter convergence. The resulting SAL acquisition function is as follows:

where M is the number of hyperparameter samples drawn from its associated posterior, θ m ' p p θ | D q , θ ' t ℓ , σ f , σ ε u , and d is a statistical distance. Notably, SAL generalizes both BQBC and BALD, which are exactly recovered by choosing the semimetric to the difference in mean or the forward KL divergence, with a short proof for the latter in App. F:

Proposition 1. SAL equipped with the KL-divergence is equivalent to BALD .

0.0

0.2

0.4

0.6

0.8

1.0

2

1

0

1

2

3

Function value

Marginal Posterior

Posterior mean

Posterior uncertainty

Observed data

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

2

1

0

1

2

3

Conditional, large

f

2

1

0

1

2

3

Conditional, large

2

1

0

1

2

3

Conditional, small

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

Acq. value

Fig. 2 visualizes the SAL acquisition function. The marginal posterior (left) is made up of three vastly different conditional posteriors with hyperparameters sampled from p p θ | D q - one with high outputscale (blue), one with very high noise (orange), and one with short lengthscale (green). For each of the blue, orange and green conditionals, the distance to the marginal posterior is computed. Intuitively, disagreement in noise level σ ε can cause large posterior disagreement at already queried locations. Similarly, uncertainty in outputscale σ f between posteriors will yield disagreement in large-variance regions, which will result in global variance reduction. Compared to other active learning acquisition functions, SAL carries distinct advantages: it has incentive to query the same location multiple times to estimate noise levels, and accomplishes the typical active learning objectives of predictive accuracy and global exploration by alleviating uncertainty over the lengthscales and outputscale of the GP. As we show in our experiments (Sec. 4.1, App. D), SAL yields superior predictions and reduces hyperparameter uncertainty at drastically improved rates.

3.2 Self-Correcting Bayesian Optimization

Equipped with the SAL objective from Eq. (8), we have an intuitive measure for the hyperparameterinduced posterior disagreement, which incentivizes hyperparameter learning by querying locations where disagreement is the largest. However, it does not inherently carry an incentive to optimize the function. To inject an optimization objective into Eq. (8), we draw inspiration from informationtheoretic BO and further condition on samples of the optimum. Conditioning on potential optima yields an additional source of disagreement reserved for promising regions of the search space.

We consider p x ˚ , f ˚ q , representing the global optimum and optimal value considered in JES [29, 61], as hyperparameters. When conditioning on p x ˚ , f ˚ q , we condition on an additional observation, which displaces the mean and reduces the variance at x ˚ . Moreover, the posterior over f becomes an upper truncated Gaussian, reducing the variance and pushing the mean marginally downwards in uncertain regions far away from the optimum as visualized in Fig. 3. Consequently, sampling and conditioning on p x ˚ , f ˚ q introduces an additional source of disagreement between the marginal posterior and the conditionals globally . The optimizer p x ˚ , f ˚ q is obtained through posterior sampling [67]. For brevity, we hereafter denote p x ˚ , f ˚ q by ˚ . The resulting SCoreBO acquisition function is

The joint posterior p p θ , ˚ | D q ' p p ˚ | θ , D q p p θ | D q used for the expectation in Eq. (9) can be approximated by hierarchical sampling. We first draw M hyperparameters θ and thereafter N optimizers ˚ | θ . As such, the expression for the SCoreBO acquisition function is:

where N is the number of optimizers sampled per hyperparameter set. Notably, while the acquisition function in (9) considers the optimizer p x ˚ , f ˚ q , SCoreBO is not restricted to employing that quantity alone. Drawing parallels to PES and MES , we can also choose to condition on either x ˚ or f ˚ alone in place of p x ˚ , f ˚ q . Doing so introduces a smaller disagreement in the posterior at the conditioned

0.0

0.2

0.4

0.6

0.8

1.0

2

1

0

1

2

3

Function value

Marginal Posterior

Posterior mean

Posterior uncertainty

Observed data

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

2

1

0

1

2

3

Conditional, large

f

2

1

0

1

2

3

Conditional, large

2

1

0

1

2

3

Conditional, small

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

Acq. value

location x ˚ , thus decreasing the acquisition value there. This will in turn decrease the emphasis that SCoreBO puts on optimization, relative to hyperparameter learning. In Fig. 3, the SCoreBO acquisition function is displayed for the same scenario as in Fig. 2. By conditioning on N ' 2 optimizers per GP, we obtain N ˆ M posteriors (displaying the posterior for one out of two optimizers, i.e. the left star in (blue), in Fig. 3). The mean is pushed upwards around the extra observation and the posterior predictive distribution over f is truncated as it is now upper bounded by f ˚ . While the preferred location under SAL is still attractive, the best location to query is now one that is more likely to be optimal, but still good under SAL .

Algorithm 1 displays how the involved densities are formed for one iteration of SCoreBO . For each hyperparameter set, a number of optima are sampled and individually conditioned on ( CondGP ) given the current data and hyperparameter set. After this procedure is completed for all hyperparameter sets, the statistical distance between each conditional posterior and the marginal is computed. The conditioning on the fantasized data point involves a rank-1 update of O p n 2 q of the GP for each draw. As such, the complexity of constructing the acquisition functions is O p MNn 2 q for M models, N optima per model and n data points. We utilize NUTS [25] for the MCMC involved with the fully Bayesian treatment, at a cost of O p Dn 3 q per sample.

3.3 Approximation of Statistical Distances

We consider two proper statistical distances, Wasserstein distance and Hellinger distance. In contrast to BQBC , the statistical distance between the normally distributed conditionals and the marginal posterior predictive distribution (which is a Gaussian mixture), is not available in closed-form. We propose two approaches: estimating the distances using MC, which we outline for both distances in App E.1, and estimation using moment matching (MM), which we outline below.

Approximation through Moment Matching We propose to fully utilize the closed-form expressions of the involved distances for Gaussians, and approximate the full posterior mixture p p y x | D q with a Gaussian distribution using moment matching (MM) for the first and second moment. While a Gaussian mixture is not generally well approximated by a Normal distribution, we show empirically in App. E that the distance between the conditionals and the approximate posterior is small. In the moment matching approach, the conditional posterior p p y x | θ , ˚ , D q utilizes a lower bound on the change in the posterior induced by conditioning on ˚ , as derived in GIBBON [42], which conveniently involves a second moment matching step of the extended skew Gaussian [45] p p y x | θ , ˚ , D q . This naive approach circumvents a quadratic cost O p N 2 M 2 q in the number of samples of each pass through the acquisition function, and yields comparable performance to the MC estimation procedures proposed in App. E.1. In App. E, we qualitatively assess the accuracy of the MM approach for both distances, and display its ability to preserve the shape of the acquisition function.

4 Experiments

In this section we showcase the performance of the SAL and SCoreBO acquisition functions on a variety of tasks. For active learning, SAL shows state-of-the-art performance on a majority of benchmarks, and is more robust than the baselines. For the optimization tasks, SCoreBO more efficiently learns the model hyperparameters, and outperforms prominent Bayesian optimization acquisition functions on a variety of tasks. All experiments are implemented in BoTorch [2] 1 . We use the same LN p 0 , 3 q 2 hyperparameter priors as Riis et al. [50] unless specified otherwise. SCoreBO and all baselines utilize fully Bayesian treatment of the hyperparameters. The complete experimental setup is presented in detail in Appendix B, and our code is publicly available at https: //github.com/hvarfner/scorebo.git . We utilize the moment matching approximation of the statistical distance. Experiments for the MC variant of SCoreBO are found in App. E.2.

4.1 Active Learning Tasks

To evaluate the performance of SAL , we compare it with BALD , BQBC and QBMGP on the same six functions used by Riis et al. [50]: Gramacy (1D) has a periodicity that is hard to distinguish from noise, Higdon and Gramacy (2D) varies in characteristics in different regions, whereas Branin, Hartmann-6 and Ishigami have a generally nonlinear structure. We display both the Wasserstein and Hellinger distance versions of SAL , denoted as SAL -WS and SAL -HR, respectively. We evaluate each method on their predictive power, measured by the negative Marginal Log Likelihood (MLL) of the model predictions over a large set of validation points. MLL emphasizes calibration (accurate uncertainty estimates) in prediction over an accurate predictive mean. In Fig. 11, we show how the average validation set MLL changes with increasing training data. SAL -HR is the top-performing acquisition function on three out of six tasks, and rivals BALD for stability in predictive performance. This is particularly evident on the Ishigami function, where most methods fluctuate in the quality of their predictions. This can be attributed to emphasis on rapid hyperparameter learning, which is visualized in detail in App. D, Fig. 15. In the rightmost plot, the real-time average per-seed ranking of acquisition function performance is displayed as a function of the fraction of budget expended. SAL -HR performs best, followed by BQBC and BALD . SAL -WS, however, does not display similarly consistent predictive quality as SAL -HR. The ability of SAL -HR to correctly estimate hyperparameters ensures calibrated uncertainty estimates, which makes it the better candidate for BO. In App. C.1, Fig. 11, we show the evolution of the average Root Mean Squared Error (RMSE) of the same tasks, where SAL -WS performs best and SAL -HR lags behind, which demonstrates the viability of various distance metrics on different tasks.

4.2 Bayesian Optimization Tasks

For the BO tasks, we use the Hellinger distance for its proficiency in prediction calibration and hyperparameter learning. We compare against several state-of-the-art baselines from the BO literature: NEI for noisy experiments [36], as well as JES [29], the MES approach GIBBON [42] and PES [24]. As an additional reference, we include NEI for noisy experiments [36] using MAP estimation.

0

50

100

Iteration

0.8

0.6

0.4

0.2

0.0

0.2

Negative MLL

Higdon (1D)

0

50

100

Iteration

1.5

1.0

0.5

0.0

0.5

Gramacy (1D)

0

100

200

Iteration

0.0

0.5

1.0

1.5

2.0

2.5

Gramacy (2D)

0

100

200

Iteration

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

0

100

200

300

Iteration

1

0

1

2

3

0

100

200

300

Iteration

0

1

2

3

4

5

Hartmann (6D)

50%

100%

Percentage of run

2.5

3.0

3.5

4.0

Relative ranking

SAL - HR

SAL - WS

BQBC

BALD

0

50

100

Iteration

2.0

1.5

1.0

0.5

0.0

0.5

1.0

Log Inference Regret

0

50

100

Iteration

1.75

1.50

1.25

1.00

0.75

0.50

0.25

0.00

0.25

Hartmann (3D)

0

50

100

150

Iteration

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

Hartmann (4D)

0

100

200

Iteration

1.0

0.8

0.6

0.4

0.2

0.0

0.2

0.4

Hartmann (6D)

0

50

100

Iteration

1.0

0.5

0.0

0.5

1.0

1.5

Rosenbrock (2D)

0

50

100

150

Iteration

0.5

0.0

0.5

1.0

1.5

2.0

Rosenbrock (4D)

25%

50%

75% 100%

Percentage of run

3.0

3.2

3.4

3.6

3.8

4.0

4.2

4.4

Relative ranking

NEI

NEI - MAP

PES

GIBBON

JES

SCoreBO

Efficiently learning the hyperparameters To showcase SCoreBO 's ability to find the correct model hyperparameters, we run all relevant acquisition functions on samples from the 8-dimensional GP in Fig. 1. We exploit that for GP samples, the objectively true hyperparameters are known (in contrast to typical synthetic test functions). We utilize the same priors as in Fig. 1 on all the hyperparameters and compare SCoreBO to NEI to assess the ability of each acquisition function to work independently of the choice of prior. In Fig. 5, for each acquisition function, we plot the average log regret over 20 dfifferent 8-dimensional instances of this task. The tasks at hand have lengthscales that vary substantially between dimensions, as detailed in App B. The explanation for the good performance of SCoreBO can be see in Fig. 17 in App. D, where SCoreBO converges substantially faster towards the correct hyperparameter values than NEI for both types of priors.

Synthetic test functions We run SCoreBO on a number of commonly used synthetic test functions for 25 | θ | iterations, and present how the log inference regret evolves over the iterations in Fig. 6. All benchmarks are perturbed by Gaussian noise. We evaluate inference regret, i.e., the current best guess of the optimal location arg max x µ p x q , which is conventional for nonmyopic acquisition functions [22, 24, 29]. SCoreBO yields the the best final regret on four of the six tasks. In the relative rankings (rightmost plot), SCoreBO ranks poorly initially, but once hyperparameters are learned approximately halfway through the run, it substantially outperforms the competition. On Rosenbrock (4D), the relatively poor performance can explained by the appar-

20

40

60

80

100

120

140

160

0.0

0.5

1.0

1.5

2.0

2.5

Simple Regret

GP-sample (8D)

NEI - Correct HPs

NEI - LogNormal Prior

NEI - BoTorch Prior

SCoreBO - LogNormal Prior

SCoreBO - BoTorch Prior

Iteration

ent non-stationarity of the task, detailed in Fig. D.3, which makes hyperparameters diverge over time. This exposes a weakness of SCoreBO : When the modeling assumptions (such as stationarity) do not align with the task, optimization performance may suffer due to perpetual disagreement in the posterior. In App. C.2, we display the performance of SCoreBO -KL and SCoreBO -WS on the same set of benchmarks, where both display highly competitive performance.

0

20

40

60

80

100

Iteration

0

5

10

15

20

Min. Observed Value

Ackley-4 (25D)

0

20

40

60

80

100

Iteration

3.0

2.5

2.0

1.5

1.0

0.5

Hartmann-6 (25D)

0

20

40

60

80

100

Iteration

0.32

0.34

0.36

0.38

0.40

0.42

0

20

40

60

80

100

Iteration

0.08

0.10

0.12

0.14

0.16

0.18

0.20

0.22

SVM (388D)

SCoreBO

NEI

GIBBON

JES

PES

2

50

100

2

1

0

50

100

1

0

1

1

50

100

1

0

1

2

50

100

1

0

1

3

50

100

1

0

1

4

50

100

1

0

1

5

50

100

1

0

1

6

NEI

SCoreBO

BALD

4.3 A Practical Need for Self-correction

Lastly, we evaluate the performance of SCoreBO on three atypical tasks with increased emphasis on the surrogate model: (1) high-dimensional BO through sparse adaptive axis-aligned priors (SAASBO) [15], (2) BO with additively decomposable structure (AddGPs) [19, 32] and (3) nonstationary, heteroskedastic modelling with HEBO [13]. Eriksson & Jankowiak [15] consider their proposed method for noiseless tasks, where active variables easily distinguish from their non-active counterparts. However, SAASBO is not restricted to noiseless tasks. For AddGPs, data crosscovariance, and lack thereof, is similarly difficult to infer in the presence of noise.

In Fig. 7, we visualize the performance of SCoreBO and competing acquisition functions with SAASBO priors on two noisy benchmarks, Ackley-4 and Hartmann-6, with dummy dimensions added, as well as two real-world benchmarks: fitting a weighted Lasso model in 180 dimensions [53], and the tuning of all 385 lengthscales and three regularization parameters of an SVM [12], a task also considered by Eriksson & Jankowiak [15]. On these benchmarks, where finding the correct hyperparameters is crucial for performance, SCoreBO clearly outperforms traditional methods. To further exemplify how SCoreBO identifies the relevant dimensions, in Fig. 8, we show how the hyperparameters evolve on the 25D-embedded Ackley (4D) task. SCoreBO quickly finds the correct lengthscales and outputscale with high certainty, whereas NEI remains uncertain of which dimensions are active throughout the optimization procedure. Impressively, SCoreBO finds accurate hyperparameters even faster than BALD , despite the latter being a pure active learning approach.

Secondly, we demonstrate the ability of SCoreBO to self-correct on uncertainty in kernel design , by considering AddGP tasks. We utilize the approach of Gardner et al. [19], where additive decompositions are marginalized over. Ideally, a sufficiently accurate decomposition is found quickly, which rapidly speeds up optimization through accurate cross-correlation of data. Fig. 9 demonstrates SCoreBO 's performance on two GP sample tasks and a real-world task estimating cosmological constants (leftmost 3 plots) and its ability to find the correct additive decompositions (right). We observe that SCoreBO identifies correct decompositions substantially better than NEI . Final performance, however, is only marginally better, as substantial resources are expended finding the right decompositions. Notably, the Cosmological Constants task does not display additive decomposability. As such, SCoreBO unsuccessfully expends resources attempting to reduce disagreement over additive structures, which hampers performance. This demonstrates that while SCoreBO learns the problem structure at increased rates, improved BO performance does not automatically follow.

Lastly, we apply SCoreBO to the HEBO [13] GP model, the winner of the NeurIPS 2020 Black-box optimization challenge [62]. The model employs input [56] and output warpings, the former of

10

20

30

40

50

60

0.5

1.0

1.5

2.0

2.5

Simple Regret

GP-sample (2D+2D+2D)

20

40

60

80

100

0

1

2

3

4

Simple Regret

GP-sample (2D+2D+2D+2D+2D)

0

50

100

150

200

22

24

26

28

30

Min. Observed Value

Cosmological Constants (11D)

NEI

SCoreBO

10

20

30

40

50

0

1

2

Incorrectly split components

NEI

SCoreBO

which are learnable to account for the heteroskedasticity that is prevalent in real-world optimization, and particularly HPO [13, 56], tasks. The complex model provides additional degrees of freedom in learning the objective. We evaluate SCoreBO and all baselines on three 4D deep learning HPO tasks: two involving large language models, and one from computer vision, from the PD1 [66] benchmarking suite. Fig. 10 displays that SCoreBO obtains the best final accuracy on 2 out of 3 tasks, suggesting that self-correction is warranted for optimization of deep learning pipelines.

0

20

40

60

80

100

120

Iteration

29.0%

30.0%

31.0%

32.0%

33.0%

34.0%

Accuracy (%)

PD1-WMT

0

20

40

60

80

100

120

Iteration

18.00%

18.50%

19.00%

19.50%

PD1-CIFAR

0

20

40

60

80

100

120

Iteration

60.25%

60.50%

60.75%

61.00%

61.25%

61.50%

61.75%

62.00%

PD1-LM1B

SCoreBO

JES

GIBBON

EI

PES

0

20

40

60

80

10

120

50%

75%

PD1-WMT

0

20

40

60

80

10

120

25%

50%

75%

PD1-CIFAR

0

20

40

60

80

10

120

70.0%

80.0%

PD1-LM1B

NEI

PES

GIBBON

JES

SCoreBO

5 Conclusion and Future Work

The hyperparameters of Gaussian processes play an integral role in the efficiency of both Bayesian optimization and active learning applications. In this paper, we propose Statistical distance-based Active Learning ( SAL ) and Self-Correcting Bayesian Optimization ( SCoreBO ), two acquisition functions that explicitly consider hyperparameter-induced disagreement in the posterior distribution when selecting which points to query. We achieve high-end performance on both active learning and Bayesian optimization tasks, and successfully learn hyperparameters and kernel designs at improved rates compared to conventional methods. SCoreBO breaks ground for new methods in the space of joint active learning and optimization of black-box functions, which allows it to excel in highdimensional BO, where learning important dimensions are vital. Moreover, the potential downside of self-correction is displayed when the model structure does not support the task at hand, or when self-correction is not required to solve the task. For future work, we will explore additional domains in which SAL and SCoreBO can allow for increased model complexity in BO applications.

6 Limitations

SCoreBO displays the ability to increase optimization efficiency on complex tasks that necessitate accurate modeling. However, SCoreBO 's efficiency is ultimately contingent on the intrinsic ability of the GP to model the task at hand. Appendix 19 demonstrates this issue for the Rosenbrock (4D) function, where SCoreBO performs worse relative to other acquisition functions. There, the hyperparameter values increase over time instead of converge, which suggests that the objective is not part of the class of functions defined by the kernel. Thus, the self-correction effort is less helpful towards optimization. Moreover, increasing the model capacity, such as in Sec. 4.3, comes with increasing resources allocated towards self-correction. In highly constrained-budget applications, such resource allocation may not yield the best result, especially if increased model complexity is unwarranted. This is evident from the synthetic AddGP tasks, where despite accurately identifying the additive components, SCoreBO does not provide substantial performance gains over NEI . Lastly, SCoreBO 's reliance on fully Bayesian hyperparameter treatment makes it more computationally demanding than MAP-based alternatives, limiting its use in high-throughput applications.

References

Atkinson, A. and Donev, A. Optimum Experimental Designs . Oxford science publications. Clarendon Press, 1992. ISBN 9780198522546. URL https://books.google.se/books? id=cmmOA_-M7S0C .

Balandat, M., Karrer, B., Jiang, D. R., Daulton, S., Letham, B., Wilson, A. G., and Bakshy, E. Botorch: A framework for efficient monte-carlo bayesian optimization. In Advances in Neural Information Processing Systems , 2020. URL http://arxiv.org/abs/1910.06403 .

Berkenkamp, F., Schoellig, A. P., and Krause, A. No-regret bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research , 20(50):1-24, 2019. URL http://jmlr.org/papers/v20/18-213.html .

Berkenkamp, F., Krause, A., and Schoellig, A. Bayesian optimization with safety constraints: Safe and automatic parameter tuning in robotics. Machine Learning , 06 2021. doi: 10.1007/ s10994-021-06019-1.

Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N. D. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research , 2018.

Bodin, E., Kaiser, M., Kazlauskaite, I., Dai, Z., Campbell, N., and Ek, C. H. Modulating surrogates for Bayesian optimization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 970-979. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr. press/v119/bodin20a.html .

Bogunovic, I. and Krause, A. Misspecified gaussian process bandit optimization. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems , volume 34, pp. 3004-3015. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 177db6acfe388526a4c7bff88e1feb15-Paper.pdf .

Bull, A. D. Convergence rates of efficient global optimization algorithms. 12:2879-2904, 2011.

Calandra, R., Gopalan, N., Seyfarth, A., Peters, J., and Deisenroth, M. Bayesian gait optimization for bipedal locomotion. In Pardalos, P. and Resende, M. (eds.), Proceedings of the Eighth International Conference on Learning and Intelligent Optimization (LION'14) , 2014.

Chaloner, K. and Verdinelli, I. Bayesian Experimental Design: A Review. Statistical Science , 10(3):273 - 304, 1995. doi: 10.1214/ss/1177009939. URL https://doi.org/10.1214/ss/ 1177009939 .

Chen, Y., Huang, A., Wang, Z., Antonoglou, I., Schrittwieser, J., Silver, D., and de Freitas, N. Bayesian optimization in alphago. CoRR , abs/1812.06855, 2018. URL http://arxiv.org/ abs/1812.06855 .

Cortes, C. and Vapnik, V. Support vector networks. Machine Learning , 20:273-297, 1995.

Cowen-Rivers, A. I., Lyu, W., Wang, Z., Tutunov, R., Hao, J., Wang, J., and Bou-Ammar, H. HEBO: heteroscedastic evolutionary bayesian optimisation. CoRR , abs/2012.03826, 2020. URL https://arxiv.org/abs/2012.03826 .

Ejjeh, A., Medvinsky, L., Councilman, A., Nehra, H., Sharma, S., Adve, V., Nardi, L., Nurvitadhi, E., and Rutenbar, R. A. Hpvm2fpga: Enabling true hardware-agnostic fpga programming. In Proceedings of the 33rd IEEE International Conference on Application-specific Systems, Architectures, and Processors , 2022.

Eriksson, D. and Jankowiak, M. High-dimensional Bayesian optimization with sparse axisaligned subspaces. In de Campos, C. and Maathuis, M. H. (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , volume 161 of Proceedings of Machine Learning Research , pp. 493-503. PMLR, 27-30 Jul 2021. URL https://proceedings.mlr.press/v161/eriksson21a.html .

Eriksson, D., Pearce, M., Gardner, J., Turner, R. D., and Poloczek, M. Scalable global optimization via local bayesian optimization. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alch´ e-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips. cc/paper/2019/file/6c990b7aca7bc7058f5e98ea909e924b-Paper.pdf .

Frazier, P. I. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

Gardner, J., Malkomes, G., Garnett, R., Weinberger, K. Q., Barbour, D., and Cunningham, J. P. Bayesian active model selection with an application to automated audiometry. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_files/paper/2015/file/ d9731321ef4e063ebbee79298fa36f56-Paper.pdf .

Gardner, J., Guo, C., Weinberger, K., Garnett, R., and Grosse, R. Discovering and Exploiting Additive Structure for Bayesian Optimization. In Singh, A. and Zhu, J. (eds.), Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 54, pp. 1311-1319. Proceedings of Machine Learning Research, 2017.

Gardner, J. R., Pleiss, G., Bindel, D., Weinberger, K. Q., and Wilson, A. G. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems , 2018.

Griffiths, R.-R. and Hern´ andez-Lobato, J. M. Constrained bayesian optimization for automatic chemical design. arXiv: Machine Learning , 2017.

Hennig, P. and Schuler, C. Entropy search for information-efficient global optimization. 98888 (1):1809-1837, 2012.

Hennig, P. and Schuler, C. J. Entropy search for information-efficient global optimization. Journal of Machine Learning Research , 13(1):1809-1837, June 2012. ISSN 1532-4435.

Hern´ andez-Lobato, J. M., Hoffman, M. W., and Ghahramani, Z. Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems , 2014. URL https://proceedings.neurips.cc/paper/2014/file/ 069d3bb002acd8d7dd095917f9efe4cb-Paper.pdf .

Hoffman, M. D. and Gelman, A. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research , 15(47):1593-1623, 2014. URL http://jmlr.org/papers/v15/hoffman14a.html .

Houlsby, N., Husz´ ar, F., Ghahramani, Z., and Lengyel, M. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 , 2011.

Hutter, F., Hoos, H., and Leyton-Brown, K. Sequential model-based optimization for general algorithm configuration. In Coello, C. (ed.), Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION'11) , volume 6683, pp. 507-523, 2011.

Hutter, F., Lindauer, M., Balint, A., Bayless, S., Hoos, H., and Leyton-Brown, K. The configurable SAT solver challenge (CSSC). 243:1-25, 2017.

Hvarfner, C., Hutter, F., and Nardi, L. Joint entropy eearch for maximally-informed bayesian optimization. In Proceedings of the 36th International Conference on Neural Information Processing Systems , 2022.

Hvarfner, C., Stoll, D., Souza, A., Lindauer, M., Hutter, F., and Nardi, L. PiBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. In International Conference on Learning Representations , 2022.

Jones, D., Schonlau, M., and Welch, W. Efficient global optimization of expensive black box functions. 13:455-492, 1998.

Kandasamy, K., Schneider, J., and P´ oczos, B. High Dimensional Bayesian Optimisation and Bandits via Additive Models. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML'15) , volume 37, pp. 295-304. Omnipress, 2015.

Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., and Xing, E. P. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems , 31, 2018.

Kirsch, A., Van Amersfoort, J., and Gal, Y. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems , 32, 2019.

Lalchand, V. and Rasmussen, C. E. Approximate inference for fully bayesian gaussian process regression. In Symposium on Advances in Approximate Bayesian Inference , pp. 1-12. PMLR, 2020.

Letham, B., Brian, K., Ottoni, G., and Bakshy, E. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis , 2018.

MacKay, D. J. Information-based objective functions for active data selection. Neural computation , 4(4):590-604, 1992.

Malkomes, G. and Garnett, R. Automating bayesian optimization with bayesian optimization. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/ 2b64c2f19d868305aa8bbc2d72902cc5-Paper.pdf .

Malkomes, G., Schaff, C., and Garnett, R. Bayesian optimization for automated model selection. In Hutter, F., Kotthoff, L., and Vanschoren, J. (eds.), Proceedings of the Workshop on Automatic Machine Learning , volume 64 of Proceedings of Machine Learning Research , pp. 41-47, New York, New York, USA, 24 Jun 2016. PMLR. URL https://proceedings.mlr.press/v64/ malkomes_bayesian_2016.html .

Mayr, M., Ahmad, F., Chatzilygeroudis, K. I., Nardi, L., and Kr¨ uger, V. Skill-based Multiobjective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration. CoRR , abs/2203.10033, 2022. URL https://doi.org/10.48550/arXiv.2203. 10033 .

Mayr, M., Hvarfner, C., Chatzilygeroudis, K., Nardi, L., and Krueger, V. Learning skill-based industrial robot tasks with user priors. IEEE 18th International Conference on Automation Science and Engineering , 2022. URL https://arxiv.org/abs/2208.01605 .

Moss, H. B., Leslie, D. S., Gonzalez, J., and Rayson, P. Gibbon: General-purpose informationbased bayesian optimisation. Journal of Machine Learning Research , 22(235):1-49, 2021. URL http://jmlr.org/papers/v22/21-0120.html .

Nardi, L., Koeplinger, D., and Olukotun, K. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) , pp. 347-358. IEEE, 2019.

Nguyen, Q., Wu, K., Gardner, J., and Garnett, R. Local bayesian optimization via maximizing probability of descent. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems , volume 35, pp. 13190-13202. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/555479a201da27c97aaeed842d16ca49-Paper-Conference.pdf .

Nguyen, Q. P., Low, B. K. H., and Jaillet, P. Rectified max-value entropy search for bayesian optimization, 2022. URL https://arxiv.org/abs/2202.13597 .

Osborne, M. A. Bayesian Gaussian processes for sequential prediction, optimisation and quadrature . PhD thesis, Oxford University, UK, 2010.

Papenmeier, L., Nardi, L., and Poloczek, M. Increasing the scope as you learn: Adaptive bayesian optimization in nested subspaces. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems , 2022. URL https: //openreview.net/forum?id=e4Wf6112DI .

Rainforth, T., Foster, A., Ivanova, D. R., and Smith, F. B. Modern bayesian experimental design, 2023.

Rasmussen, C. and Williams, C. Gaussian Processes for Machine Learning . The MIT Press, 2006.

Riis, C., Antunes, F. N., H¨ uttel, F. B., Azevedo, C. L., and Pereira, F. C. Bayesian active learning with fully bayesian gaussian processes. arXiv preprint arXiv:2205.10186 , 2022.

Ru, B., Osborne, M. A., Mcleod, M., and Granziol, D. Fast information-theoretic Bayesian optimisation. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 4384-4392. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/ru18a.html .

Ru, B., Wan, X., Dong, X., and Osborne, M. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. arXiv preprint arXiv:2006.07556 , 2020.

ˇ Sehi´ c, K., Gramfort, A., Salmon, J., and Nardi, L. LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso. arXiv preprint arXiv:2111.02790 , 2021.

Settles, B. Active learning literature survey. 2009.

Snoek, J., Larochelle, H., and Adams, R. Practical Bayesian optimization of machine learning algorithms. In Bartlett, P., Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS'12) , pp. 2960-2968, 2012.

Snoek, J., Swersky, K., Zemel, R., and Adams, R. Input warping for Bayesian optimization of non-stationary functions. In Xing, E. and Jebara, T. (eds.), Proceedings of the 31th International Conference on Machine Learning, (ICML'14) , pp. 1674-1682. Omnipress, 2014.

Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. W. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory , 58(5):3250-3265, May 2012. ISSN 1557-9654. doi: 10.1109/tit.2011.2182033. URL http://dx.doi.org/10.1109/TIT.2011.2182033 .

Stanton, S., Maddox, W., and Wilson, A. G. Bayesian optimization with conformal prediction sets. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pp. 959-986. PMLR, 25-27 Apr 2023. URL https://proceedings. mlr.press/v206/stanton23a.html .

Takeno, S., Fukuoka, H., Tsukada, Y., Koyama, T., Shiga, M., Takeuchi, I., and Karasuyama, M. Multi-fidelity Bayesian optimization with max-value entropy search and its parallelization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 9334-9345. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/takeno20a.html .

Takeno, S., Inatsu, Y., and Karasuyama, M. Randomized Gaussian process upper confidence bound with tighter Bayesian regret bounds. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pp. 33490-33515. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/takeno23a.html .

Tu, B., Gandy, A., Kantas, N., and Shafei, B. Joint entropy search for multi-objective bayesian optimization. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id= ZChgD8OoGds .

Turner, R., Eriksson, D., McCourt, M., Kiili, J., Laaksonen, E., Xu, Z., and Guyon, I. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In Escalante, H. J. and Hofmann, K. (eds.), Proceedings of the NeurIPS 2020 Competition and Demonstration Track , volume 133 of Proceedings of Machine Learning Research , pp. 3-26. PMLR, 06-12 Dec 2021. URL https: //proceedings.mlr.press/v133/turner21a.html .

Wan, X., Nguyen, V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 10663-10674. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/wan21b.html .

Wan, X., Nguyen, V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. International Conference on Machine Learning (ICML) 38 , 2021.

Wang, Z. and Jegelka, S. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning (ICML) , 2017.

Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Nado, Z., Gilmer, J., Snoek, J., and Ghahramani, Z. Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXiv:2109.08215 , 2023.

Wilson, J. T., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. P. Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning , 2020. URL https://arxiv.org/abs/2002.09309 .

Wu, J., Poloczek, M., Wilson, A. G., and Frazier, P. Bayesian optimization with gradients. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 64a08e5f1e6c39faeb90108c430eb120-Paper.pdf .

Wynne, G., Briol, F.-X., and Girolami, M. Convergence guarantees for gaussian process means with misspecified likelihoods and smoothness. J. Mach. Learn. Res. , 22(1), jan 2021. ISSN 1532-4435.

Yao, Y., Vehtari, A., and Gelman, A. Stacking for non-mixing bayesian computations: The curse and blessing of multimodal posteriors, 2020.

Zhang, B., Cole, D. A., and Gramacy, R. B. Distance-distributed design for gaussian process surrogates, 2019.

A Additional Related Work

We review approaches that address model accuracy in Bayesian optimization.

Expanded Space of Models [38, 39] expands model uncertainty to kernel design in order to find the most accurate model possible within an expanded class of functions, whereas [56] expands model uncertainty to involve warpings of the input space. Since the expanded class of functions offers additional modelling flexibility, BO can ideally be conducted on a more accurate model than under a vanilla setting. As demonstrated with AddGPs in Sec. 4.3, SCoreBO can work in conjunction with kernel search to find such models at an accelerated rate.

Simplified Modelling A contrasting line of work involve methods which restrict modeling by reducing the level of detail [6] or scope of the optimization [16, 44, 63]. This line of work acknowledges that modeling may not, or should not, be globally accurate or granular in order to conduct optimization efficiently within the allocated budget. As such, these lines of work address the issue of model accuracy by diametrically opposed philosophies. SCoreBO offers an orthogonal approach to model accuracy by accelerating the convergence of the model at hand, regardless of its level of complexity.

B Supplementary Material on Experiments

B.1 Experimental Setup

All the relevant methods are implemented as acquisition functions in BoTorch. For both the active learning and BO experiments, we run NUTS [25] in Pyro [5] to draw samples from the GP posterior over hyperparameters. Tab. 3 displays the parameters of the MCMC in detail, as well as other relevant parameters of various MC estimations throughout the article. For the active learning experiments, we mimic the experimental setup used in Riis et al. [50], and put a log-normal distribution LN p 0 , 3 q on the lengthscales, outputscale variance and noise variance. Furthermore, we consider the mean constant c as a learnable parameter in the BO experiments, with a conventional N p 0 , 1 q prior on the standardized inputs. When referring to the BoTorch priors , the priors are Γ p 3 , 6 q , Γ p 2 , 0 . 15 q , and Γ p 1 . 1 , 0 . 05 q for the lengthscales, outputscale, and noise variance, respectively, with the same prior on the learnable constant mean c .

8D Gaussian Process sample task For the 8 D GP sample task, we utilize the lognormal prior from Sec. B on the hyperparameters. The hyperparameters of the sampled objective functions are outlined in Tab. 1. As such, Tab. 1 displays the true hyperparameters of the task we are trying to optimize. These hyperparameters are referenced in Fig. 17 and Fig. 18

SAASBO experiments For the SAASBO experiments, we utilize Ax 3 , which runs the BoTorch 4 implementation of SAASBO with the deafult prior on the hyperparameters as per BoTorch version 0.8.4, which differs slightly from the paper by Eriksson & Jankowiak [15]. The lengthscale parameters are given a hierarchical prior as τ 2 ' HC p α q , κ 2 i ' HC p 1 q and ℓ i ' 1 κ i τ . We retain the default value of α ' 0 . 1 . The noise variance, outputscale and mean constant are given the priors σ 2 ε ' Γ p 0 . 9 , 10 q , σ 2 f ' Γ p 2 , 0 . 15 q and c ' N p 0 , 1 q .

Additive Gaussian Process experiments The Additive GP setup closely resembles that of [19]. An additive partitioning is sampled, and the marginal likelihood of the model is maximized with regard to θ ' t ℓ , σ f , σ ε u . We utilize a slightly adapted proposal distribution, and fix a maximal number of additive partitions g max . Moreover, each dimension d belongs to one distinct additive decomposition

g d , where g d P r 1 , . . . , g max s . At each iteration of the MCMC scheme, two dimensions i, j ' tp i, j q P r 1 , . . . D s , i ‰ j u , are sampled uniformly at random and assigned the same new group index g , where g ' U p 1 , g max q . Setting g i ' g, g j ' g thus proposes that dimensions i and j belong to the same additive decomposition. Under this proposal distribution, the proposed number of additive decompositions can never surpass g max , but a lesser number of additive groups can be proposed. We utilize the same warm-starting mechanism as described in Gardner et al. [19], where at each iteration, the final accepted sample from the previous iteration acts as the initial proposal. The substitution of proposal distribution from the (unavailable) original implementation [19] was made to simplify a batch GP implementation in GPyTorch [20]. We choose to employ a BoTorch prior on the hyperparameters for all Additive GP tasks and use a squared exponential kernel, as described in [19].

HEBO Experiments For the warped GP experiments, we adapt slightly more conservative priors due to the large number of hyperparameters of the model. Specifically, we set LN p 0 , 1 q priors on the on lengthscales, outputscale variance and noise variance. For the input warpings, we employ a Kuwaraswamy distribution, a differentiable-CDF alternative to the input warpings proposed by Snoek et al. [56]. For each input dimension j , the untransformed input x j has a transformation applied as

where z j is the resulting, transformed input. The warping parameters α and β are both given a LN p 0 , 0 . 1 q prior. We note that the HEBO experiments are susceptible to the number of hyperparameter sets. As such, we do not recommend running the experiments than fewer number of hyperparameter sets than that outlined by Tab. 3, as substantially fewer did not yield substantial empirical gains relative to vanilla BO.

Table 3: MCMChyperparameters for all experiments. For the AddGP experiments, each hyperparameter set involves the sampled additive decomposition and its associated MAP-trained hyperparameters. The total number of hyperparameter sets drawn are Warmup + Thinning * No. Hyperparameter sets.

B.2 Benchmarks

For the active learning benchmarks, we follow Riis et al. [50] in the types of benchmarks and noise levels used. Each benchmark, as well as its search space, dimensionality and noise level is described in Tab. 4 and Tab. 5 for AL and BO, respectively. The noise level for all of the BO synthetic test functions were set to σ ε ' 0 . 5 , except the Rosenbrock benchmarks, where the noise standard deviation was set to σ ε ' 2 . 5 due to the extremely large output range of the function. A smaller noise level would consequently bring the signal-to-noisy ratio under the permitted threshold supported by BoTorch.

Compute resources. All experiments are carried out on Intel Xeon Gold 6130 CPUs. Each repetition of the tasks in Sec. 4.1 and Sec. 4.2 are run on 4 cores, and the tasks in Sec. 4.3 are run on 8 cores. Approximately 1 , 000 core hours are used for each of the AL synthetic tasks, 2000 for the BO synthetic tasks, and 5000 for each task in Sec. 4.3.

C Additional Experiments

We display the RMSE performance of each of the SAL variants, a comparison of the MC and MM variants of SAL and SCoreBO .

C.1 AL RMSE Performance

Fig. 11 displays the performance of both SAL variants and benchmark AL acquisition functions for the same set of tasks as in Sec. 4.1. We observe that SAL -WS consistently displays top performance, whereas SAL -HR lags behind substantially. This showcases SAL -HR's emphasis on hyperparameter learning as opposed to global exploration. By accurately assessing hyperparameters while sacrificing global exploration, SAL -HR ensures predictions with calibrated uncertainty , while sacrificing the accuracy in predictive mean that follows from exploring the search space. SAL -WS offers a compromise which performs well under both metrics, sacrificing hyperparameter learning and calibration for accuracy in predictive mean.

0

50

100

Iteration

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

Negative MLL

Higdon (1D)

0

50

100

Iteration

1.5

1.0

0.5

0.0

0.5

Gramacy (1D)

0

100

200

Iteration

0.0

0.5

1.0

1.5

2.0

Gramacy (2D)

0

100

200

Iteration

1.0

0.8

0.6

0.4

0.2

0.0

0

100

200

300

Iteration

1

0

1

2

3

0

100

200

300

Iteration

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Hartmann (6D)

25%

50%

75% 100%

Percentage of run

2.5

3.0

3.5

4.0

Relative ranking

SAL - HR

SAL - WS

BALD

BQBC

C.2 SCoreBO Distance Measure Ablation Analysis

In Fig. 13, We compare the Hellinger and Wasserstein variants of SCoreBO , both utilizing the MC approximation of the statistical distance. Since the MC approximation is asymptotically exact, we can better assess the performance of each distance metric, without having to consider the confounding factor that the MM approximation introduces. We note that SCoreBO -WS outperforms SCoreBO -HR

on two tasks, but SCoreBO -HR is the overall more consistent approach. We hypothesize that the relative failure of SCoreBO -WS on Rosenbrock (4D) is caused by the objective's non-stationarity, which likely causes exceedingly large exploration of the hyperparameter space. This is supported by Fig. D.3, where the hyperparameters diverge over time on the Rosenbrock function.

Furthermore, in Fig. 12, we compare the KL and Hellinger variants of SCoreBO , both utilizing the moment matching estimation of the posterior. Notably, SCoreBO -KL is the natural extension of BALD to the self-correcting framework, in line with Prop. 1. SCoreBO -HR performs marginally better than SCoreBO -KL, winning 4 out of 6 tasks. However, the two variants are relatively close.

0

50

100

2

1

0

1

2

Log Inference Regret

0

50

100

1.5

1.0

0.5

0.0

0.5

Hartmann-3

0

100

1.25

1.00

0.75

0.50

0.25

0.00

0.25

0.50

100

200

1.00

0.75

0.50

0.25

0.00

0.25

Hartmann-6

0

50

100

1.0

0.5

0.0

0.5

1.0

1.5

2.0

Rosenbrock-2

0

100

0.0

0.5

1.0

1.5

2.0

2.5

Rosenbrock-4

SCoreBO - HR

SCoreBO - WS

0

25

50

75

100

Iteration

2.0

1.5

1.0

0.5

0.0

0.5

1.0

Log Inference Regret

0

50

100

Iteration

1.75

1.50

1.25

1.00

0.75

0.50

0.25

0.00

0.25

Hartmann (3D)

0

50

100

150

Iteration

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

Hartmann (4D)

0

50

100

150

200

Iteration

1.0

0.8

0.6

0.4

0.2

0.0

0.2

0.4

Hartmann (6D)

0

25

50

75

100

Iteration

1.0

0.5

0.0

0.5

1.0

1.5

Rosenbrock (2D)

0

50

100

150

Iteration

0.0

0.5

1.0

1.5

2.0

Rosenbrock (4D)

SCoreBO

ScoreBO - KL

C.3 SAASBO Noise Ablations

In Fig. 13, We measure the performance of SCoreBO and NEI on the embedded 25-dimensional Ackley (4D) and Hartmann (6D) tasks with varying noise levels using the SAAS [15] prior. We run three noise levels for each task: Noiseless (solid line), low (dashed line), where the noise standard deviation corresponds to 3% of total output range for Hartmann (6D), and 1.3% for Ackley, and high (dotted line, 13.3% / 4%). With increasing levels of noise, the difficulty of inferring active dimensions is expected to increase substantially, which should in turn hamper BO performance. In Fig. 14, we see that NEI and SCoreBO perform comparably on noiseless tasks (solid line) finding close-to-optimal solutions within 100 iterations. Moreover, for small levels of noise), the performance is still comparable. However, we observe a drastic fall-off in performance for NEI at the highest noise level, whereas SCoreBO 's degrades gracefully. Notably, SCoreBO almost retains the performance of the noiseless at the highest noise level for Ackley.

D Hyperparameter convergence

In Figures 15, 16, 17, 18, and 19 We demonstrate examples of hyperparameter convergence in AL and BO, as well as an example of hyperparameter divergence on the Rosenbrock p 4 D q function, where SCoreBO performs marginally worse.

0

20

40

60

80

100

0

5

10

15

20

Min. Observed Value

Ackley-4 (25D)

NEI

SCoreBO

=0.0

=0.31

=1.0

0

20

40

60

80

100

3.0

2.5

2.0

1.5

1.0

0.5

Min. Observed Value

Hartmann-6 (25D)

Iteration

Iteration

D.1 Active Learning Tasks

We display the hyperparameter convergence of SAL -WS, SAL -HR and the baseline active learning acquisition functions in Fig. 15. Both variants display accelerated hyperparameter learning compared to BQBC . SAL -HR in particular achieves low-variance hyperparameter uncertainty on Ishigami and the higher-dimensional Hartmann-6, where other methods struggle. We obtain approximately correct hyperparameters for these tasks by randomly sampling 300 points on the noiseless benchmark, thereafter performing MCMC and averaging the sampled hyperparameter estimates in logspace. The noise level is known a priori. and estimates the other hyperparameters with substantially greater certainty than other methods. We note that there is a drift in the hyperparameters as the number of observations increase, where output- and lengthscales trade off to reduce model complexity. As such, we provide an approximately stationary alternative in Fig. 16, where the outputscale is removed to avoid drift. In both cases, SAL -HR displays superior hyperparameter convergence, obtaining accurate hyperparameters in far fewer iterations, and with substantially less uncertainty than the alternatives.

0

100

200

300

1

2

3

Log hyperparam. value

2

0

100

200

300

2

1

0

1

2

0

100

200

300

1

0

1

0

100

200

300

1

0

1

2

0

100

200

300

1

0

1

3

SAL - HR

SAL - WS

BALD

BQBC

0

100

200

300

2

1

0

1

Log hyperparam. value

2

0

100

200

300

1.0

0.5

0.0

1

0

100

200

300

1

0

1

2

0

100

200

300

1

0

1

3

SAL - HR

SAL - WS

BALD

BQBC

D.2 GP sample tasks

We display the convergence of SCoreBO and NEI with a wide lognormal and BoTorch prior on the GP sample task. We observe that the lognormal prior is well-aligned for most hyperparameters, whereas BoTorch prior is misaligned. This is evidenced by the unimportant dimensions ℓ 6 , ℓ 7 , and ℓ 8 , which have suggested lengthscales that are incorrect by more than an order of magnitude. Nevertheless, SCoreBO suggests lengthscales that are approximately twice as long p 10 0 . 25 « 1 . 8 q as NEI p 10 ´ 0 . 05 «

0 . 9 q , and thus avoids unneessary exploration along these dimensions. Moreover, SCoreBO correctly identifies the most important dimensions ℓ 1 , ℓ 2 , and ℓ 3 with good accuracy quickly, whereas NEI struggles to identify ℓ 1 . SCoreBO slightly overestimates the importance of dimensions 2 and 3, likely to compensate for the inability to accurately estimate the importance of other hyperparameters.

1

0

1

1

1

0

1

2

1

0

1

3

1

0

1

4

1

0

1

5

0

25

50

75

100

125

150

1

0

1

6

0

25

50

75

100

125

150

1

0

1

7

0

25

50

75

100

125

150

1

0

1

2

8

0

25

50

75

100

125

150

1

0

2

0

25

50

75

100

125

150

1.0

0.5

0.0

0.5

2

Noisy EI

ScoreBO

1.0

0.5

1

0.6

0.4

0.2

2

0.6

0.4

0.2

3

0.6

0.4

0.2

0.0

4

0.50

0.25

0.00

5

0

25

50

75

100

125

150

0

1

6

0

25

50

75

100

125

150

0

1

7

0

25

50

75

100

125

150

0

1

8

0

25

50

75

100

125

150

1

0

2

0

25

50

75

100

125

150

0.0

0.5

2

Noisy EI

ScoreBO

D.3 Hyperparameter Divergence on Synthetic BO tasks

We highlight additional examples on synthetic BO test functions where hyperparameters diverge. Due to the non-stationary structure of Rosenbrock in particular (and to a lesser extent, Branin), hyperparameters values diverge as the number of observations increase. In particular, the extreme steepness along the edges suggests an exceedingly large outputscale. With increasing observations, a lengthscale-outputscale trade-off occurs, where both hyperparameters grow seemingly indefinitely. Notably, this behavior is consistent regardless of the acquisition function (BO, AL, SOBOL). Due to the restricted hyperparameter set employed in the AL tasks, this problem is distinct to the BO tasks.

2

0

50

100

150

6

8

2

0

50

100

150

2

4

0

50

100

150

0.5

0.0

0.5

1

0

50

100

150

0.5

0.0

0.5

2

0

50

100

150

0.5

0.0

0.5

3

0

50

100

150

0

1

4

NEI

SCoreBO

E Approximation Strategies

We display the quality of the moment matching approximation for both the Hellinger and Wasserstein distance. Moreover, we compare the performances of the MM and MC approaches.

E.1 MCapproximation of SAL and SCoreBO

Using Monte Carlo, different distances are most efficiently estimated in different manners. To approximate the Wasserstein distance, we utilize quasi-Monte Carlo. From the definition of the distance in one dimension, we obtain

where u ℓ ' U p 0 , 1 q , and P p x q and Q p x q are the respective cumulative distributions for p p x q and q p x q . To approximate the Hellinger distance, we obtain

where x ℓ ' p p x q is sampled using MC. In SCoreBO , p p x q is the marginal p p y x | D q , and q p x q each of the various conditionals p p y x | ˚ , θ , D q .

E.2 Performance of Monte Carlo

We display the performance of the MC variants of SAL -WS and SCoreBO -HR compared to their MMcounterparts. Overall, performances are comparable, as each variant slightly exceeds the other on a couple of benchmarks. On the most complex benchmarks (Ishigami, Hartmann-4, Hartmann6), the MC variant outperforms MM slightly, which suggests that MC is increasingly justified as disagreement in the posterior gets larger.

20

40

0.8

0.7

0.6

0.5

0.4

0.3

Negative MLL

Higdon (1D)

20

40

1.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

Gramacy (1D)

0

50

100

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Gramacy (2D)

0

50

100

0.9

0.8

0.7

0.6

0.5

0

50

100

150

0.5

0.0

0.5

1.0

1.5

2.0

2.5

0

50

100

150

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Hartmann-6

SAL - HR

SAL - WS

SAL - WS (MC)

SAL - HR (MC)

Iteration

Iteration

Iteration

Iteration

Iteration

Iteration

0

50

100

2.0

1.5

1.0

0.5

0.0

0.5

1.0

Log Inference Regret

0

50

100

1.50

1.25

1.00

0.75

0.50

0.25

0.00

0.25

Hartmann-3

0

50

100

150

1.4

1.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

100

200

1.2

1.0

0.8

0.6

0.4

0.2

0.0

0.2

0.4

Hartmann-6

0

50

100

1.0

0.5

0.0

0.5

1.0

1.5

Rosenbrock-2

0

50

100

150

0.0

0.5

1.0

1.5

2.0

Rosenbrock-4

SCoreBO - MC

SCoreBO

E.3 Hellinger Distance Approximation

We display the accuracy of the moment matching approximation, and the sensitivity of the MC approximation to the number of samples L . In Fig. 24 and Fig. 25, we highlight two examples of the moment matching approximation in comparison to a large-scale, asymptotically exact variant of the MC approximation with 2048 samples. In Fig. 24, the MM approximation struggles to capture the sharp, multimodal surfaces in (blue), and consistently overestimates the distance in (orange). In Fig. 25, the included conditional posteriors are substantially more similar, and as such, the moment matching approximation is more accurate. The shape of the acquisition function is captured almost perfectly, and the magnitude is only marginally overestimated, most prominently in (green). We display the

0.0

0.2

0.4

0.6

0.8

1.0

2

1

0

1

2

Function value

Posterior mean

Posterior uncertainty

Observed data

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.1

0.2

0.3

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0.3

0.4

0.5

0.6

0.7

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

Acq. value

0.0

0.2

0.4

0.6

0.8

1.0

2

1

0

1

2

Function value

Posterior mean

Posterior uncertainty

Observed data

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.1

0.2

0.3

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.3

0.4

0.5

2

1

0

1

2

2

1

0

1

2

2

1

0

1

2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

Acq. value

E.4 Wasserstein Distance Approximation

0.0

0.2

0.4

0.6

0.8

1.0

2

0

2

Function value

Posterior mean

Posterior uncertainty

Observed data

0.0

0.2

0.4

0.6

0.8

1.0

0.6

0.8

1.0

1.2

1.4

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.6

0.8

1.0

1.2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

2.5

2

0

2

4

2

0

2

4

4

2

0

2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

2.5

Acq. value

0.0

0.2

0.4

0.6

0.8

1.0

3

2

1

0

1

2

Function value

Posterior mean

Posterior uncertainty

Observed data

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

2

0

2

3

2

1

0

1

2

3

2

1

0

1

2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

Acq. value

F Equivalence to BALD

We show that SAL, using the KL divergence as the metric d , is equivalent to BALD.

G Additional Background

We provide additional background on fully Bayesian hyperparameter treatment in Gaussian processes and details regarding statistical distance metrics.

G.1 Fully Bayesian Treatment

The posterior probability of observing a value y x for a point x is given as:

where f are the noiseless, latent function values as x and D is the observed data. The inner integral is equal to the GP predictive posterior,

However, the outer integral is intractable and is estimated using Markov Chain Monte Carlo (MCMC) methods. The resulting posterior prediction

is a Gaussian Mixture Model (GMM).

Within BAL and BO, the fully Bayesian treatment is often extended to involve the acquisition function, such that the acquisition function α is computed as an expectation over the hyperparameters [46, 55]

This is also the definition of fully Bayesian treatment considered in this work.

G.2 Statistical Distance Details

The Hellinger distance is a similarity measure between two probability distributions which has previously been employed in the context of BO-driven automated model selection by Malkomes et al. [39]. For two probability distributions p and q , it is defined as

with some auxiliary measure λ with which both p and q are absolutely continuous. Specifically, for two normally distributed variables z 1 ' N p µ 1 , σ 2 1 q , z 2 ' N p µ 2 , σ 2 2 q ,

The Hellinger distance seeks to minimize the ratio between difference in mean and the sum of variances, which punishes outlier predictive distributions of high confidence. Similar to KL, initial queries have a tendency to be axis-aligned to attain selective length scale information.

The Wasserstein distance is the average distance needed to move the probability mass of one distribution to morph into the other. The Wassersteink distance is defined as

For the normal distributions z 1 and z 2 , the Wasserstein-2 distance is defined as

In practice, W 2 places a premium on matching large-variance regions, leading to higher global exploration which can be detrimental for global optimization.

The KL divergence The KL divergence is a standard asymmetrical measure for dissimilarity between probability distributions. For two probability distributions P and Q , it is given by D KL p P || Q q ' ş X P p x q log p P p x q{ Q p x qq dx . For Gaussian variables, it is computed as

The KL divergence mainly prioritizes same order-of-magnitude variances, and will initially query the same location multiple times to assess noise levels. Thereafter, it tends to query in an axis-aligned fashion, close to previous queries, to attain information regarding the length scales, but places a low priority on global exploration.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a novel function SAL used for active learning based on distribution difference. Then the author further incorporated the SAL function into Bayesian optimization (BO) to improve BO performance. Most numerical experiments are conducted on synehtic benchmakr functions. Better performances are achieved by using the proposed method compared to several baseline functions.\n\nSTRENGTHS:\nOverall, the paper is easy to follow despite several confusions.\n\nWEAKNESSES:\nLack of real-world experiments.\n\nI would suggest the authors be precisely clear on their symbols. \n\nI have some questions about the technical details.\n\nQUESTIONS:\nIn general, I grasp the main idea of this paper. However, I suggest that the authors distinguish among three different things: (i) the true underlying function $f(x)$, (ii) the observed noisy function value $f(x)+\\epsilon$, and (iii) the GP model $GP(x)$, which is an approximation to $f(x)$. The authors should carefully examine the usage of each symbol throughout their paper and ensure that they properly distinguish between (i)-(iii). Currently, the authors sometimes misuse these symbols, which hinders the understanding of the paper. \n\nBelow, I have listed several questions and would appreciate it if the authors could provide answers. This will help me gain a better understanding and assessment of the paper.\n\n1. The authors state that the mean function $m(\\cdot)$ is set to a constant. Why is this constant not included in the set of hyperparameters?\n\n2. The numerical experiments primarily focus on benchmark functions, with the exception of DNA classification task and cosmological estimation. It would be beneficial to include more real-world experiments. For example, Bayesian optimization is commonly used for tuning network parameters in automl. It would be valuable to add experiments similar to that.\n\n3. What is the specific form of $p(\\theta|D)$ in step 4 shown in Algorithm 1?\n\n4. What is $p(f|\\theta,D)$ in Step 6? And what does Step 6 do? The symbol $f$ is used to represent the underlying function in equation (1). I am confused about how to sample an $f$ from $p(\\cdot|\\theta,D)$. Since $f(\\cdot)$ represents the underlying function and does not depend on the GP model, I assume that $p(f|\\theta,D)$ means sampling a GP instance given $\\theta$ and $D$. If this is the case, I suggest the authors use another symbol instead of $f$ here.\n\n5. One advantage of Bayesian optimization is that its acquisition function does not require calling the real function. Instead, it only needs predictions based on the GP (e.g., EI = mean of GP - std of GP). Thus, Bayesian optimization can save functional evaluation times. Typically, Bayesian optimization is employed in situations where observing the function value $y_x$ at a specific $x$ is time-consuming or expensive. However, I am confused here. Does $y_x$ represent the observed function value at $x$ according to equation (1)? Does this mean that in order to evaluate equation (9), we actually need to simulate or obtain $y_x$ at a specific $x$? In other words, does the proposed acquisition function (9) require the observation of $y_x$? If so, it seems to me that the method is not valid, as it sacrifices the fundamental benefit of Bayesian optimization.\n\n6. Continuing from the previous question, how do we evaluate the distribution value $p(y_x|\\cdots)$ in Step 7? Moreover, how do we evaluate $p(y_x|D)$ in (9)?\n\n7. What is the difference between $p(y_x|\\theta,D)$ used in equation (7) and $p(f|\\theta,D)$ used in Step 6? Are they different in terms of considering $\\epsilon$ or not (i.e., in equation (1), $y_x=f(x)+\\epsilon$)?\n\n8. I am confused about the condition on $(x^\\star,f^\\star)$. What do the authors mean by $(x^\\star,f^\\star)$? Are they the true optimals related to the underlying function $f$ or are they the optimals related to a GP model?\n\n9. The considered numerical experiments all seem rather low-dimensional.\n\n10. I am concerned about the insufficient technical contribution. It appears to me the main contribution could be summarized in one equation, Eq. (7). Previous similar papers usually also provided theoretical guarantees (e.g., the JES paper).  Solely a new function form seems rather insufficient as a standalone paper. I would appreciate it if the author could justify their contribution. \n\n**Writing suggestions**\n\n1. I assume \"HP\" in Figure 1 means \"hyper-parameter.\" However, at first sight, \"HP\" is confusing, especially since many words have abbreviations as \"HP.\" I suggest that the authors write the full name when it first appears.\n\n2. Line 72 contains a typo: \"$\\epsilon\\sim N(0,\\sigma^2_\\epsilon)$\" instead of \"$\\epsilon^2$.\"\n\n3. Keep the symbol format consistent: In line 76, $\\theta$ is not bold, while in line 80, $\\theta$ is bold.\n\n4. The Hellinger distance has the general form shown in equation (4). Why is there no general form provided for the Wasserstein distance?\n\n5. In line 221, \"robust. than\" needs to be revised for clarity.\n\nLIMITATIONS:\nThe authors provided limitations in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a technique called statistical distance-based active learning (SAL) for the purpose of learning the hyperparameters of a Gaussian process. Additionally, SAL is integrated with information-theoretic Bayesian optimization (BO) in a framework referred to as self-correcting BO (SCoreBO). This integrated approach enables simultaneous learning of GP hyperparameters and optimization of the black box function. The experimental results demonstrate the enhanced performance of SCoreBO compared to traditional benchmarks, including unconventional BO tasks.\n\nSTRENGTHS:\nThe paper introduces new active learning algorithms that leverage the Query-by-Committee strategy and utilize two statistical distances. It also addresses the practical challenge of simultaneously learning hyperparameters while optimizing a black-box function, which holds significant importance in real-world applications.\n\nWEAKNESSES:\nThe main weakness of the paper is inadequate literature review, resulting in major concerns on the novelty of the proposed problem and solution and the experimental results.\n\n1. Although BALD is widely recognized as a highly effective active learning strategy (e.g., employed in PES, MES, JES), it is surprising that BALD is not considered as a baseline in the active learning experiments. Conversely, the other baselines employed in the active learning experiments are relatively weak, such as utilizing only the posterior variance (BALM), the posterior mean (BQBC), or simply adding BALM and BQBC.\n\n2. Equation (7) (the proposed SAL) is essentially the Jensen-Shannon divergence, representing the mutual information between $\\theta$ and $y_x$—in other words, BALD when $d$ corresponds to the KL divergence. Surprisingly, this crucial aspect is not discussed in the paper, raising the question of why the preference is given to the Hellinger distance or Wasserstein distance instead of the KL divergence.\n\n3. The paper introduces the problem of optimizing a black box function with unknown Gaussian Process (GP) hyperparameters, asserting that this aspect has received limited attention, without providing references to existing Bayesian Optimization (BO) works that address unknown GP hyperparameters. However, it is worth noting that in the early work PES, there exists a dedicated section that tackles the issue of unknown GP hyperparameters by learning the posterior distribution of the hyperparameters and averaging PES over samples of GP hyperparameters. This crucial detail is missing from the paper, despite frequent mentions of PES throughout. Furthermore, this implies that other information-theoretic acquisition functions such as MES and JES could also be extended to handle unknown GP hyperparameters using a similar approach. Consequently, the novelty in the problem appears to be overstated and a significant concern arises regarding the absence of several essential baselines in the experiments, including PES, MES, and JES averaged over samples of GP hyperparameters.\n\n4. The motivation behind the proposed SCoreBO in Equation (8) needs further enhancement. Given that the goal of Bayesian Optimization (BO) is to optimize a black-box function, it seems appropriate to exclude $\\theta$ from the second argument in $d$ in Equation (8). This is because it is redundant to learn the hyperparameters of the Gaussian Process (GP) if they do not impact the difference between $f^*$ or $x^*$ under different GP hyperparameter samples. Essentially, the acquisition function should be a JES, with the GP hyperparameters marginalized out. To substantiate this approach, it would be valuable to present compelling experimental results comparing it with such a JES that marginalizes out the GP hyperparameters.\n\nQUESTIONS:\nPlease clarify the above weaknesses.\n\n========================\n\nAfter reading author's response\n\nThe authors have made substantial efforts in providing supplementary experimental outcomes and elaborating the reasoning behind the integration of hyperparameters into the acquisition function. \n\nI believe that these experiments and explanations have the potential to provide stronger motivation for the paper and make the proposed approach's performance more convincing. As a result, I improve my rating accordingly.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents two algorithms/methods, SAL and SCoreBO, for active learning and Bayesian optimization using GPs. Both build on the premise that hyperparameters are critical to effective AL or BO; hence it is important to learn both the function and the hyperparameters through an acquisition function that accounts for both needs.\n\nContributions:\n- The SAL acquisitions function for AL considers the statistical distance between a condition (on the hyperparameters) and the marginal posterior.\n- The SCoreBO algorithm for BO, which uses SAL in combination with Thompson sampling/posterior sampling \n- Empirical evaluation on several relevant benchmark problems \n- Comparison with a set of alternative methods\n\nSTRENGTHS:\n- Well-written and clear (with only a few unclear aspects)\n- Relatively simple but effective approach  (based on various other works in fully Bayesian BO including [42])\n- Thorough evaluation of a suitable set of both AL and BO benchmarks, including non-standard problems\n- Comparison with what appears to be a sensible set of baselines, showing favorable performance on the BO tasks\n\nWEAKNESSES:\nThe following are mostly just questions and comments, not all weaknesses per se:\n\n- Lack of details regarding a fully Bayesian treatment\n  -   The paper mentions that it applies a fully Bayesian treatment, yet I am (as a reader) still a bit unsure about the specific inference being used. I think more details are needed here to make it precise. In particular, discussing $p(\\theta | D)$ would be helpful to ensure the paper presents a self-contained view. \n\n- Justification of distance metrics and alternatives\n  -  I feel the paper lacks justification for the choice of distance measures. I appreciate a choice needs to be made at some point, but alternative statistical distance measures also have closed-form, e.g., (symmetric) KL, and even approximations for mixtures. \n  -  I am slightly concerned about the sensitivity to the distance function for the AL (and BO as shown in the appendix). For the AL task, have the authors performed longer experiments than presented in Figure 6 (i.e. why stop at 100/150/200 iterations)?  \n\n- What's the complexity? It would be helpful with an indication/summary of the complexity of SCoreBO relative to a standard fully Bayesian BO approach. \n\n- (The need for) theoretical bounds: What is the prospect of providing theoretical bounds for the specific algorithm?\n\nMinor:\n- As a sanity check, have the authors compared SCoreBO with a box standard BO approach without a fully Bayesian treatment of hyperparameters (i.e., ML-II)?\n- Figure 1: For clarity, I'd suggest specifying what a \"BoTorch prior\" is\n- Figure 7: There are two dashed lines; perhaps clarify the caption.\n\nQUESTIONS:\nIncluded in the above.\n\nLIMITATIONS:\nIncluded in the above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStandard acquisition functions in Bayesian Optimization (BO) only aim to finding the optimum, but do not directly consider the problem of hyperparameter learning in Gaussian Processes (GP) which has considerable impact on the optimization performance. \nThis paper introduces a new acquisition function for Active Learning (AL) and Bayesian Optimization (BO), whose goal is to learn both hyperparameters and the location of the optimum. Specifically, the acquisition function for AL is based on extending similar previous proposals of Active Learning via disagreement, by using different statistical distances. This new acquisition function is then adapted for the BO task by conditioning on sampled locations/value of the optimum. The new acquisition functions are shown to work (slightly) better at AL and BO, especially with unusual BO tasks.\n\nSTRENGTHS:\n- **Originality:** While natural, the problem of hyperparameter learning in BO has rarely been addressed, so this paper tackles an open and underestimated problem in the field. The proposed solutions are interesting.\n- **Significance:** Given the prominence of BO in machine learning and other fields, this work is potentially very significant.\n- **Quality:** The general quality of the work is good, although there are some open questions (see below).\n- **Clarity:** The paper is generally clear.\n\nAs general comments, Statistical distance-based Active Learning (SAL) is well-motivated and explained, a nice generalization of previous proposals. It is also nice that the paper explores the use of two different distances (Wasserstein and Hellinger), and that it gives approximations on how to compute them. The proposed \"motivated heuristic\" for extending SAL to BO via conditioning over the location-value of the optimum is interesting.\n\n### Post-rebuttal\n\nThanks to the authors for addressing the points I raised. I am glad that they managed to run additional experiments with a more realistic benchmark, showcasing the effectiveness of their method. Overall, I am satisfied with the paper and the score of 6 reflects my current evaluation of the paper (\"Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.\").\n\nWEAKNESSES:\n- The paper showcases many synthetic functions but there are only a couple of real-world applications. Real-world applications are particularly important for this work because they are exactly the situations at risk of breaking active learning (e.g., due to model misspecification). Indeed, the Cosmological Constant task in the paper is the one in which ScoreBO does not show any advantage over vanilla Expected Improvement. For this reason, it would have been nice to show other real examples in which ScoreBO does improve performance.\n- As a minor comment, please double-check the paper and bibliography for typos or errors. At a quick glance I spotted a few mistakes (e.g., the authors of reference [43] \"Fast information-theoretic Bayesian optimisation\" are in a wrong order; occasionally \"Bayesian\" appears lowercase, etc.).\n\nQUESTIONS:\n- This paper would strongly benefit from showing ScoreBO at work on other real-world tasks, to showcase that indeed this new acquisition function and its focus on active learning does not harm when deployed on real problems (e.g.,  RL, hyperparameter tuning, and other typical applications of BO which are conspicuously missing here).\n- Minor: Fix the few typos in the paper and bibliography.\n\nLIMITATIONS:\nThe conclusion quickly states a few limitations but it'd be good to have a separate **Limitations** section which very explicitly mentions them.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}