Self-Correcting Bayesian Optimization through Bayesian Active Learning
Authors: Carl Hvarfner
Erik Orm Hellsten
Frank Hutter
Luigi Nardi
Venue: NeurIPS 2023 poster (2023)
Keywords: Bayesian Optimization
Bayesian Active Learning
Gaussian Processes
TL;DR: Joint Bayesian optimization and active learning of hyperparameters

Abstract: Gaussian processes are the model of choice in Bayesian optimization and active learning. Yet, they are highly dependent on cleverly chosen hyperparameters to reach their full potential, and little effort is devoted to finding good hyperparameters in the literature. We demonstrate the impact of selecting good hyperparameters for GPs and present two acquisition functions that explicitly prioritize hyperparameter learning. Statistical distance-based Active Learning (SAL) considers the average disagreement between samples from the posterior, as measured by a statistical distance. SAL outperforms the state-of-the-art in Bayesian active learning on several test functions. We then introduce Self-Correcting Bayesian Optimization (SCoreBO), which extends SAL to perform Bayesian optimization and active learning simultaneously. SCoreBO learns the model hyperparameters at improved rates compared to vanilla BO, while outperforming the latest Bayesian optimization methods on traditional benchmarks. Moreover, we demonstrate the importance of self-correction on atypical Bayesian optimization tasks.

Self-Correcting Bayesian Optimization through Bayesian Active Learning
Carl Hvarfner
Erik Orm Hellsten erik hellstenccs 1th. se Lund University
carl. carl.hvarfnee www.fferrcs......ommm...owo....e 1th.se Lund University
Frank Hutter
Luigi Nardi
fh@cs. wwwuumiiiiiiuueeeee de University Freiburg
luigi nardi@ca 1th Lund University Stanford University DBtune
Abstract
Gaussian processes are the model of choice in Bayesian optimization and active learning. Yet, they are highly dependent on cleverly chosen hyperparameters to reach their full potential, and little effort is devoted finding good hyperparameters in the literature. We demonstrate the impact of selecting good hyperparameters for GPs and present two acquisition functions that explicitly prioritize hyperparameter learning. Statistical distance-based Active Learning (SAL) considers the average disagreement between samples from the posterior, as measured by statistical distance. SAL outperforms the state-of-the-ard in Bayesian active learning sev- eral test functions. We then introduce Belf-Correcting Bayesian Optimization (BCoreBO), which extends SAL to perform Bayesian optimization and active learn- ing simultancously. SCoreBO learns the model hyperparameters at improved rates compared to vanilla BO, while outperforming the latest Bayesian optimization methods on traditional benchmarks. Moreover, we demonstrate the importance self-correction atypical Bayesian optimization tasks.
Introduction
Bayesian Optimization (BO) is powerful paradigm for black-box optimization problems, i.c., problems that can only be accessed by pointwise queries. Such problems arise in many applications, ranging from including drug discovery (2]] configuration of combinatorial problem solvers [27 28
hardware design 14 hyperparameter tuning 11 30
52 and robotics
Gaussian processes (GPs) are a popular choice as surrogate models in BO applications. Given the data, the model hyperparameters are typically estimated using either Maximum Likelihood or Maximum a Posteriori estimation (MAP) [49. Alternatively, fully Bayesian treatment of the hyperparameters [46 55 removes need to choose any single set through Monte Carlo integration. This procedure effectively considers all possible hyperparameter values under the current posterior, thereby accounting for hyperparameter uncertainty. However, the relationship between accurate GP hyperaarameter estimation and BO performance has received little attention 3 58 69 71]. and active reduction of hyperparameter uncertainty not an integral part any prevalent BO acquisition function. In contrast, the field of Bayesian Active Learning (BAL) contains multiple acquisition functions based solely on reducing eeperparameter-induee the broader field of Bayesian Experimental Design [0] 48 revolves around acquisition of data to best learns the model parameters.
measures of uncertainty 26 34 50] and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The importance of the GP hyperparameters in BO is illustrated in Fig. [ which shows average simple regret over 20 optimization runs of 8-dimensional functions drawn from a Gaussian pro- cess prior. The curves correspond to the performance of Expected Improvement with noisy ex- periments (NEI) [36] acquisition function under a fully Bayesian hyperparameter treatment using NUTS (25). Two prevalent hyperparameter priors, described in detail in App. B. as well as the true model hyperparameters, are used. Clearly, good model hyperparameters have substantial impact on BO performance, and BO methods could greatly benefit from estimating the model hyperparameters as accurately as possible. Furthermore, the hyperparameter estimation task can become daunting under complex problem setups, such non-stationary objectives (spatially vary- 56 64], high- high-dimensional search spaces [I3] 47), and additively decomposable objectives [99 32). The complexity of such problems warrants the use of more complex, task-specific surrogate models. In such settings, the success of the optimization may increasingly hinge on the presumed accuracy of the task-specific surrogate.
ing lengthscales, heteroskedasticity) 66, 13 16
We proceed in two steps. We first introduce Statisti- cal distance-based Active Learning (SAL), which im- proves Bayesian active learning by generalizing previ- ous work [26] [50] and introduces a holistic measure of disagreement between the marginal posterior pre- dictive distribution and each conditional posterior pre- dictive. We consider the syperparameter-indueee dis-
GPssample (8D)
NEI Wide LogNormal Prior
Simple Regret
NEI Default BoTorch Prior
NRI True Hyperparametery
agreement between models in the acquisition function, thereby accelerating the learning of model hyperparam- eters. We then propose Self- Seff-Correc Bayesian Opti- mization (SCoreBO), which builds upon SAL by explicitly learning the location of the optimizer in conjunction with model hyperparameters. This achieves accelerated hy- perparameter learning and yields improved optimization performance on both conventional and exotic BO tasks. Formally, we make the following contributions:
20
4.
60
80 Iteration
1.00
120
140
1.0
Figure 1: Simple regret of using true hyperpa- rameters, BoTorch (v.0.8.4 default) and lognor- mal hyperparameter priors with fully Bayesian hyperparameter treatment. The prior substan- tially impacts final performance, and correct hyperparameters yield vastly better results.
1. We introduce SAL, a novel and efficient acquisition function for hyperparameter-orienttee Bayesian active learning based on statistical distances (Sec. 3.1
2. We introduce SCoreBO, the first acquisition function for joint BO and hyperparameter learning (Sec. 3.2)
3. We display highly competitive performance on an array of conventional AL (Sec. 4.1 and BO tasks (Sec. 4.2 and demonstrate SCoreBOs, ability to enhance atypical models such as
SAASBO and HEBO [13], and identify decompositions in AddGPs (Sec.
2 Background
2.1 Gaussian processes
Gaussian processes (GPs) have become the model class of choice in most BO and active learning applications. They provide a distribution over functions f GP((m()), k(;,)) fully defined by the mean function m(-) and the covariance function k( Under this distribution, the value of the function fx),, at a given point x, is normally distributed with a closed-form solution for the mean and variance. We assume that observations are perturbed by Gaussian noise, such that Ya = f(x) + 8, E ~ NNO, We also assume the mean function to be constant, such that the dynamics fully determined by the covariance function k((,,)).
To account for differences in variable importance, each dimension is individually scaled using length- scale hyperparameters f.. For D- D-dimensional inputs and x', the distance r(æ, x²) is subsequently
computed as 2 [x - / Along with the outputscale Ty. the set 0 {l. oz, oy) comprises the set of hyperparameters that are conventionally learned. The likelihood surface for the GP hyperparameters is typically highly multi-modal
(4 70,, where different modes represent different bias-varianc, trade-offs 49 50) To avoid having to choose single mode, one can define prior p(0) and marginalize with respect to the hyperparameters when performing predictions (35]
We outline fully Bayesian hyperparameter treatment in GPs App. G.I

---

2.2 Bayesian Optimization
Bayesian Optimization (BO) seeks to maximize to a black-box function ff over a compact domain X,
* E arg max f(x);
(1)
such that f can only be sampled point-wise through expensive, noisy evaluations y/2 finction, where E ~ (O, o²). New configurations are chosen by optimizing an acquisition finnisitionnunncionn, which uses the surrogate model to quantify the utility of evaluating new points in the search space. Examples of such heuristics are Expected Improvement (NEI) [ 31 and Upper Confidence Bound (UCB) [ 57 60). More sophisticated look-ahead approaches include Knowledge Gradient (KG) [17 68 as well as a class of particular importance for our approach- - the information-theorriic acquisition function class. These acquisition functions consider mutual information objective select the next query,
f(x)
CinI (>) [(ye: * (Dn),
where * can entail either the optimum x* as in (Predictive) Entropy Search (ES/PES) [23 24 the optimal value f as in Max-value Entropy Search (MES) 42 59 65 or the tuple (æ* f*), used in Joint Entropy Search (JES) [29 61]. FITBO [51] shares similarities with our work, in that the optimal value is governed by hyperparameter, in their case of transformed GP.
Within BO, the fully Bayesian hyperparameter treatment conventionally extended from the predictive posterior to the acquisition function such that for M models with hyperparameters Om m {1..... sampled from the posterior over hyperparameters p(0)D), the acquisition function a is computed as an expectation over the hyperparameters 46 55
M
a<x|D) [e[a(z]0,
1
Σ
a(z|0m, D) Om D(0
(3)
M
This is also the definition of fully Bayesian treatment considered in this work.
2.3 Bayesian Active Learning
In contrast BO, which aims to find maximizer an unknown function, Active Learning (AL) [54] seeks to accurately learn the black-box function globally. Thus, the objective to minimize the expected prediction loss. AL acquisition functions classified either preision-theoretiic, which minimize the prediction loss over a validation set, or information- which minimize the space
of plausible models given the observed data 26 37
In the cnarmation-thhoreiie category, Active Learning McKay (ALM) 37 highest Shannon Entropy, which for GPs amounts to selecting the point with the highest variance. Under fully Bayesian hyperparameter treatment, it is referred to as Bayesian ALM (BALM). Bayesian Active Learning by Disagreement (BALD) (26] was among the first Bayesian active learning approaches to explicitly focus on learning the model hyperparameters. It approximates the reduction in entropy over the GP hyperparameters from observing new data point
selects the point with the
CBALD (z) [yy= D) H(p(y= (D) Eg [p[(H(e( O,D]
and was later extended deep Bayesian active learning 34 and active model (kernel) selection [18] Lastly, Riis et al. (50) propose Bayesian the variance V of the GP mean is the largest, with respect to changing model hyperparameters:
Committeee (BQBC) strategy. BQBC queries where
an) = Voluo(<: (x] Eo[(uo
where juxz) is the marginal posterior mean at æ. and μo (xx) is the posterior mean conditioned on As such, BQBC queries the location which maximizes the average distance between the marginal posterior and the conditionals according to some distance metric (here, the posterior mean), henceforth referred to as hyperparameter-induee
posterior disagreement. However, disagreement in mean alone does not fully capture hyperparameter-induce. disagreement. Thus, (50) also presents Query-by- byxxtuby of Gaussian Processes (QBMGP), that adds the BALM criterion to the BQBC acquisition function.
3

---

2.4 Statistical Distances
statistical distance quantifies the distance between two statistical objects. We focus on three (semi- Jmetrics, which have closed forms for Gaussian random variables. The closed forms expressions, as well as additional intuition on their interaction with Gaussian random variables, can be found in App. G.2
The Hellinger distance is a dissimilarity measure between two probability distributions which has previously been employed in the context of BO-driven automated model selection by Malkomes et al. 39 For two probability distributions p and q, it is defined
1 J (Vole)- ( Adæ, 2
(p. 9)
(6)
for some auxiliary measure À under which both and are absolutely continuous.
The Wasserstein distance is dissimilarity metric between two distributions describing the average distance one distribution has to be moved to morph into another. The Wasserstein-k distance is defined as
(L, Fp()))
1/k
Ww(p, q)
where, in this work, we focus on the case where k: 2.
The KL divergence The KL divergence is a standard asymmetrical measure for dissimilar- ity between probability distributions. For two probability distributions P and Q, is given by DRI(P Q) Sx ((z)))og(((z)/() P(z)/Q(z))ddz The distances in Eq. (6). Eq. (16) and the KI. diver- gence are used for the acquisition functions presented Sec.
Methodology
In Sec. 3.1, introduce SAL, novel family of metrics for BAL. In Sec. 3.2, extend this to SCoreBO, the first acquisition function for joint BO and hyperparameter-oriineee active learning. inspired by information-thentt -hhorretc BO acquisition funetions. In Sec. 3.3, demonstrate how to efficiently approximate different types of statistical distances within the SAL context.
3.1 Statistical distance-based Active Learning
In active learning for GPs, it is important to efficiently learn the correct model hyperparameters. By measuring where the posterior hyperparameter uncertainty causes high disagreement in model output, the search can be focused on where this uncertainty has high impact. However, considering only mean, as in BQBC, is overly restrictive as does not fully utilize the
the posterior disagreement available distributions for the hyperparameters. For example, it ignores uncertainty in the outputscale hyperparameter the Gaussian process, which disincentives exploration. As such, we propose to generalize the acquisition function in Eq. (5) to instead consider the posterior disagreement as measured by any statistical distance. Locations where the posterior distribution changes significantly query, in order quickly learn the model
result of model uncertainty are good points hyperparameters. When an observation at such a location obtained, hyperparameters which predicted that observation poorly will have substantially smaller likelihood. which turn aids hyperparameter convergence. The resulting SAL acquisition function is as follows:
M Σ m==1
O.SAA (x) = Ee[d(p(yz () p(yn D))
d(p(ya|Om P),p(ux())) (D),
(8)
M
where M is the number of hyperparameter samples drawn from its associated posterior, 6n P()
0 a o, oe), and d is statistical distance. Notably, SAL generalizes both BQBC and BALD, which are exactly recovered by choosing the semimetric to the difference mean or the forward KI. divergence, with short proof for the latter in App.
Proposition 1. SAL equipped with the KL-divergence is equivalent to BALD.

---

Marginal Posterior
Conditional large
Conditional. large
Conditional, small
Acg. value Fanction sallue
Paane enat Ocsireed dats
64
04-
34
1.8
13
66
Figure 2: Marginal posterior (top left, grey in other plots in top row), 0s using the Hellinger distance (bottom left. black), and the three conditional GPs (blue, orange. green) and their marginal contribution to the total acquisition function (bottom row). The large disagreement in noise level and lengthscale, primarily caused by the orange GP (large noise, long lengthscale), makes ass query the lowest-valued point for second time (selected location as vertical dashed line in the leftmost plot) to determine the mean and variance at that location.
Fig. visualizes the SAL acquisition function. The marginal posterior (left) is made up of three vastly different conditional posteriors with hyperparameters sampled from p(0)D) one with high outputscale (blue), one with very high noise (orange), and one with short lengthscale (green). For each the blue, orange and green conditionals, the distance to the marginal posterior is computed. Intuitively, disagreement in noise level 0g can cause large posterior disagreement at already queried locations. Similarly, uncertainty in outputscale or between posteriors will yield disagreement in large-variance regions, which will result in global variance reduction. Compared to other active learning acquisition functions, SAL carries distinet advantages: it has incentive to query the same location multiple times to estimate noise levels, and accomplishes the typical active learning objectives of predictive accuracy and global exploration by alleviating uncertainty over the lengthscales and outputscale of the GP. As show in experiments (Sec. 4.1 App. D), SAL yields superior predictions and reduces hyperparameter uncertainty at drastically improved rates.
3.2 Self-Correcting Bayesian Optimization
Equipped with the SAL objective from Eq. (8), we have intuitive measure for the hyperparameter- induced posterior disagreement, which incentivizes hyperparameter learning by querying locations where disagreement is the largest. However, it does not inherently carry incentive to optimize the function. To inject an optimization objective into Eq. (8) we draw inspiration from information- theoretic BO and further condition samples of the optimum. Conditioning potential optima yields an additional source of disagreement reserved for promising regions of the search space.
We consider (x", f representing the global optimum and optimal value considered in JES 29 as hyperparameters. When conditioning on (*) we condition on an additional observation, which displaces the mean and reduces the variance at x Moreover, the posterior over / becomes an upper truncated Gaussian, reducing the variance and pushing the mean marginally downwards in uncertain regions far away from the optimum visualized in Fig. Consequently, sampling and con- ditioning on (x* f*) introduces an additional source of disagreement between the marginal posterior and the conditionals globally. The optimizer (x* f*) is obtained through posterior sampling [67]. by *. The resulting SCoreB0 acquisition function is
For brevity, we hereafter denote
asc az) Eg, [d(p(y= (D), p(y= 00, *, D))].
The joint posterior p(0, * (D) p(* 0,, D) D) used for the expectation in Eq. 9 can be approximated by hierarchical sampling. We first draw M hyperparameters 0 and thereafter N optimizers * 0. As such, the expression for the SCoreBO acquisition function is:
M
axx)=
1
Σ
D d (D, 00m.. m... D)),
(10)
NM
where N the number of optimizers sampled per hyperparameter set. Notably, while the acquisition function in (9) considers the optimizer (æ*, {*), SCoreBO is not restricted to employing that quantity
alone. Drawing parallels to PES and MES, we can also choose condition on either x or f alone
in place of (x* ))) Doing introduces smaller disagreement in the posterior the conditioned
5

---

Marginal Posterior
Conditional large
Conditional. large
Conditional small
Acg. value Panction sallue
Paane enat Ocsireed dats
co
64
04-
34
1.8
10
42
Figure 3: Approximate marginal posterior after having conditioned on (a* f*) (top left), asc using the Hellinger distance (bottom left), the three conditional truncated posteriors and their marginal contribution to the total acquisition function for the same iteration Fig. 2 Conditioning (a= ff)) (marked as drawn from function samples in dashed) inroduces additional disagreement between the marginal posterior and the sampled GPs promising regions as result of conditioning, In the figure, we marginalize over M sets of hyperparameters and N 2 optimizers per GP, where each optimizer's 's contribution to the acquisition function is visible under its corresponding GP. Note that, since function draws are noiseless, the conditioned optimum does not need to surpass the best noisy observation in value. This phenomenon is most notable in (orange).
Algorithm 1 SCoreBO iteration
1: Input: Number of hyperparameter sets M, number of sampled optima N, current data
2: Oufput: Next query location x'.
3: for m E {l..... 3 do
4: On p(0
5: for n (1, (l...., N do
6: *em, max fom.n m: where fem. ~ p(f]mmm p(f(mm m D) {Draw optima for each 0...)
7: p(ye mi *0 nnm. D)
CondGP ( mm D) {Condition GPs on each optimum)
8:
end for
9:
end for
10: x' = arg mnggaaxuu (xx)
{Defined in Eq. 10
location x", thus decreasing the acquisition value there. This will turn decrease the emphasis that SCoreB0 puts optimization, relative to hyperparameter learning. In Fig. 3 the SCoreBO acquisition function is displayed for the same scenario as Fig. 2 optimizers GP, we obtain N M posteriors (displaying the posterior for one out of two optimizers, i.e the left star in (blue), in Fig. 3. The mean pushed upwards around the extra observation and the posterior predictive distribution over f is truncated is now upper bounded by f*. While the preferred location under SAL still attractive, the best location to query now one that is more likely to be optimal, but still good under SAL.
By conditioning on N
Algorithm displays how the involved densities are formed for one iteration of SCoreBO. For each hyperparameter set, a number of optima are sampled and individually conditioned on (CondGP) given the current data and hyperparameter set. After this procedure completed for all hyperparameter sets, the statistical distance between each conditional posterior and the marginal is computed. The conditioning on the fantasized data point involves a rank- update (n²) of the GP for each draw. As such, the complexity of constructing the acquisition functions is O(MNn²) for M models, N optima per model and n data points. We utilize NUTS (25) for the MCMC involved with the fully Bayesian treatment, at cost of ODDr") per sample.
3.3 Approximation of Statistical Distances
We consider two proper statistical distances, Wasserstein distance and Hellinger distance. In contrast to BQBC, the statistical distance between the normally distributed conditionals and the marginal posterior predictive distribution (which is a Gaussian mixture), is not available in closed-form form. We propose two approaches: estimating the distances using MC, which we outline for both distances in App E. and estimation using moment matching (MM), which we outline below.

---

Approximation through Moment Matching We propose to fully utilize the closed-form expres- sions of the involved distances for Gaussians, and approximate the full posterior mixture p(yn (D) with Gaussian distribution using moment matching (MM) for the first and second moment. While Gaussian mixture is not generally well approximated by Normal distribution, we show empirically in App. E that the distance between the conditionals and the approximate posterior is small. In the moment matching approach, the conditional posterior p\ye 0 *, D) utilizes a lower bound on the change in the posterior induced by conditioning on *, as derived in GIBBON [42]. which conve- niently involves a second moment matching step of the extended skew Gaussian 45] p(yn 0 *, D).
This naive approach circumvents a quadratic cost O M²) in the number of samples of each pass through the acquisition function, and yields comparable performance to the MC estimation procedures proposed in App. In App. E we qualitatively assess the accuracy of the MM approach for both distances, and display its ability to preserve the shape of the acquisition function.
4 Experiments
In this section we showcase the performance of the SAL and SCoreBO acquisition functions a variety of tasks. For active learning, SAL shows state-of-the-art performance on a majority of benchmarks, and is more robust than the baselines. For the optimization tasks, SCore more efficiently learns the model hyperparameters, and outperforms prominent Bayesian optimization acquisition functions on variety of tasks. All experiments are implemented in BoTorrc 2 We use the same LO 3 hyperparameter priors as Riis et al. [50] unless specified otherwise. SCoreBO and all baselines utilize fully Bayesian treatment of the hyperparameters. The complete experimental setup is presented in detail in Appendix [ and our code is publicly available at lhttps 7/github. com/hvarfner scorebo. . git We utilize the moment matching approximation of the statistical distance. Experiments for the MC variant of SCoreBO are found in App. E.2
4.1 Active Learning Tasks
To evaluate the performance of SAL, we compare it with BALD, BQBC and QBMGP on the same six functions used by Riis et al. [50]: Gramacy (1D) has a periodicity that is hard to distinguish from noise, Higdon and Gramacy (2D) varies in characteristics in different regions, whereas Branin, Hartmann-6 and Ishigami have generally nonlinear structure. We display both the Wasserstein and Hellinger distance versions of SAL, denoted as SAL-WS and SAL-HR, respectively. We evaluate each method on their predictive power, measured by the negative Marginal Log Likelihood (MIL) of the model predictions over a large set of validation points. MLL emphasizes calibration (accurate uncertainty estimates) in prediction over an accurate predictive mean. In Fig. [T, we show how the average validation set MLL changes with increasing training data. SAL-HR is the top-perforne peppefformmng acquisition function on three out of six tasks, and rivals BALD for stability in predictive performance. This is particularly evident of their predictions. This can be attributed to emphasis on rapid hyperparameter learning, which is visualized in detail in App.DD Fig. [5] In the rightmost plot, the real-time average per-seed ranking of acquisition function performance is displayed as a function of the fraction of budget expended. SAL-HR performs best, followed by BQBC andBALD. SAL-WS, however, does not display similarly consistent predictive quality as SAL-HR. The ability of SAL-HR to correctly estimate hyperparameters ensures calibrated uncertainty estimates, which makes the better candidate for BO. In App. C.I we show the evolution of the average Root Mean Squared Error (RMSE) of the same tasks, where SAL-WSS performs best and SAL-HR lags behind, which demonstrates the viability of various distance metrics on different tasks.
the Ishigami function, where most methods fluctuate in the quality
Fig. I1
4.2 Bayesian Optimization Tasks
For the BO tasks, we use the Hellinger distance for its proficiency in prediction calibration and hyperparameter learning. We compare against several ofate-of-the-arr baselines from the BO literature: NEI for noisy experiments (36), well as JES (29]. the MES approach GIBBON 42 and PES [24]. As an additional reference, we include NEI for noisy experiments (36] using MAP estimation.
httpp://botorcc..or// (v000)
All Normal and LogNormal distributions parametrized by the mean and variance.
7

---

Higdon (1D)
Gromacy (1D)
Gremscy (29)
Brunin 120)
Shuquni (2D)
Hartmunn (6D)
Relative ranking
Mays
0.5
4.
604
-0.3
048
02
-075
1x
383
530
155 300
T6
24
50%
1.00
SAL SAL. WS5
BQBC
QSMG
SALD
Figure 4: Negative Marginal Log Likelihond (MLL) on six active leamnnng functions the (smoothed) relative rankings throughout cach run for QBMGP, EQBC, BALD and SAL using Wasserstein and Hellinger distance. We plot mean and one standard error for 25 repetitions.. SAL-HR is the top performing method, placing first in relative rankings. On Ishigami, only SAL-IIR and BALD produces stable results.
Branin 120)
Hartmsen (SD)
Hartmann (43)
Hartmasn (6D)
Buseebbrock (203)
Reeemboock (4D)
Relative ranking
E23
E31
96
10
277
80
3.22
.10
-12
97 100
50 13
130
20. 19m
Pesccolape
Figure 6: Average log inference regret and (smoothed) relative ranking across 50 repetitions between the acquisition functions for SCoreBO, JES, MES and NEI six synthetic test functions. SCoreB0 produces the best final regret 4 of 6 tasks, and has substantially lower average ranking by the end of each run.
Efficiently learning the hyperparameters To showcase SCoreBO's s ability to find the correct model hyperparameters,
all relevant acquisition functions samples from the 8-dimensional GP in Fig. [ We exploit that for GP samples, the objectively true hyperparameters are known (in contrast typical synthetic test functions). We utilize the same priors as in Fig. on all the hyperparameters and compare SCoreB0 to NEI to assess the ability of each acquisition function to work independently of the choice of prior. In Fig. 5 for each acquisition function, we plot the average log regret over 20 dfifferent 8-dimensional instances of this task. The tasks at hand have lengthscales that vary substantially between dimensions, as detailed in App B The explanation for the good performance of SCoreBO can be see in Fig. [7]in App. D where SCoreBO converges substantially faster towards the correct hyperparameter values than NEI for both types of priors.
GP-sample (8D)
Synthetic test functions We run SCoreBO on a num- ber of commonly used synthetic test functions for 25|0] iterations, and present how the log inference regret
2.5
NEI Correct HPs
Siople Regret
LogNormal Prior
NEI BoTorch Prior
evolves over the iterations in Fig. 6 All benchmarks are perturbed by Gaussian noise. We evaluate infer-
2.5
SCareBO Log Prior
SConeBO BoTorch Prior
ence regret, i.e., the current best guess of the optimal location arg max ((x),, which is conventional for non- myopic acquisition functions 22
0.5
0.0
24 29 SCoreB0 yields the the best final regret on four of the six tasks. In the relative rankings (rightmost plot), SCoreBO ranks poorly initially. but once hyperparameters are learned approximately halfway through the run, substantially outperforms the competition. On Rosenbrock (4D), the relatively poor performance can explained by the appar- ent non-stationarity of the task, detailed in Fig. [D3]] which makes hyperparameters diverge over time. This exposes weakness SCoreB0: When the modeling assumptions (such as stationarity) do not align with the task, optimization performance may suffer due to perpetual disagreement in the posterior. In App. C.2 we display the performance of SCoreBO-KI. and SCoreBO-Wi WS on the same set of benchmarks, where both display highly competitive performance.
120
100
Figure Regret for NEI and SCoreBO on 8-dimensional GP sample for two differ- ent types of hyperparameter priors. Mean and standard deviation are plotted for all hyperpa- rameter samples across 20 repetitions.
8

---

Adkloy (253)
Hartmans- (250)
Losso-DNA 12 BOD)
SVM (389D)
4.2
-16
040
SCoreBO
-20
NEI
GIBBON
CLU4
X13
FES
Figure 7: Final loss using SAASBO priors on the noisy embedded Ackley-4, embedded Hartmann-6 6, the DNA classification and the SVM HPO task, mean and one standard error. SCoreBO identifies the important dimensions rapidly, and successfully optimizes the tasks. The optimal value is marked with dashed line.
-1-
SCorBBO
BALDD
-2
1:00
Figure 8: Hyperparameter convergence on 25DDembedded 4D Ackley function with SAASBO HP prior for SCoreBO, NEI and BALD. Log HP mean and standard deviation is plotted per iteration. SCoreB0 identifies f1...., f as important (short lengthscales, & マ 10"1) ) with low uncertainty and fs..... ..... l25 as dummy dimensions (f, 10' ). NEI fails to identify any important lengthscales. whereas SCoreBO correctly identifies active dimensions with high certainty.. Notably, SCoreB0 finds accurate hyperparameters even faster than BALD, a pure active learning approach. Reference HP values (where available) are marked with a dashed line.
4.3 A Practical Need for Self-correction
Lastly, we evaluate the performance of SCoreBO on three atypical tasks with increased emphasis on the surrogate model: (1) high-dimensional BO through sparse adaptive axis-aligned priors (SAASBO) [5]. (2) BO with additively decomposable structure (AddGPs) [19 32] and (3) non- stationary, heteroskedastic modelling with HEBO (13]. Eriksson & Jankowiak [15 consider their proposed method for noiseless tasks, where active variables easily distinguish from their non-active counterparts. However, SAASBO is not restricted to noiseless tasks. For AddGPs, data cross- covariance, and lack thereof, is similarly difficult to infer in the presence of noise.
In Fig. 7 we visualize the performance of SCoreBO and competing acquisition functions with SAASBO priors on two noisy benchmarks, Ackley-4 and Hartmann-6, with dummy dimensions added, as well as two real- world benchmarks: fitting a weighted Lasso model in 180 dimensions (33. and tuning of all 385 lengthscales and three regularization parameters of an SVM [[2], task also considered by Eriksson & Jankowiak 115. On these benchmarks, where finding the correct hyperparameters crucial for performance, SCoreBO clearly outperforms traditional methods. To further exemplify how SCoreBO identifies the relevant dimensions, in Fig. g show how the hyperparameters evolve on the 25D-embedded Ackley (4D) task. SCoreBO quickly finds the correct lengthscales and outputscale with high certainty, whereas NEI remains uncertain of which dimensions are active throughout the optimization procedure. Impressively, SCoreBO finds accurate hyperaarameters faster than BALD, despite the latter being pure active learning approach.
Secondly, we demonstrate the ability SCoreBO to self-correct on uncertainty in kernel design, by considering AddGP tasks. We utilize the approach of Gardner et al. [19, where additive de- found quickly,
compositions are marginalized over. Ideally, a sufficiently accurate decomposition which rapidly speeds up optimization through accurate cross-correlation of data. Fig. demmnstraaes SCoreBO's 's performance on two GP sample tasks and a worl-world task estimating cosmological constants (leftmost 3 plots) and its ability to find the correct additive decompositions (right). We observe that SCoreBO identifies correct decompositions substantially better than NEI. Final perfor- mance, however, is only marginally better, substantial resources are expended finding the right decompositions. Notably. the Cosmological Constants task does not display additive decomposability. As such, SCoreBO unsuccessfully expends resources attempting reduce disagreement over additive structures, which hampers performance. This demonstrates that while SCoreBO learns the problem structure increased rates, improved BO performance does automatically follow.
Lastly, we apply SCoreBO to the HEBO 13 GP model, the winner of the NeurIPS 2020 Black-box optimization challenge [62]. The model employs input [56) and output warpings, the former of

---

GP-sample (2D+30+2D)
GPsample (DD+2D+DD-2+-20
Cosmological Constants (11
Incorrectly split components
Glavereet Viluo
Nei
Stmppo Bagres.
ShRpp Regnet
SCowRO
XIII
12
SCieeBO
30
30c
04
100
130
15
Figure 9: Final value of using AddGPs on 6D and 10D GP sample functions, fully decomposable in groups of two, and the Cosmological Constants tasks. SCoreBO achieves better final performance (left, middle) with low uncertainty, and successfully finds the additive components of the 6D task (right).
which are learnable account for the heteroskedasticity that is prevalent in real-world optimization, and particularly HPO 13 [66]. tasks. The complex model provides additional degrees of freedom
three 41D deep learning HPO tasks: two involving large language models, and one from computer vision, from the PD1 [66] benchmarking suite. Fig. [i]dillays that SCoreBO obtains the best final accuracy on out of 3 tasks, suggesting that self-correction is warranted for optimization of deep learning pipelines.
in learning the objective. We evaluate SCoreB0 and all baselines
285
2485
StorcHoo
32.55
JES
GIBBON
NEI
31.55
Ieeraion
Ileration
Heeatton
Figure 10: Performance on the PD1 deep learning tasks over 20 repetitions using the warpings from HEBO 13 SCoreBO obtains the best final accuracy 2 out of tasks, placing second the third.
Conclusion and Future Work
The hyperparameters of Gaussian processes play an integral role in the efficiency of both Bayesian optimization and active learning applications. In this paper, we propose Statistical distance- based Active Learning (SAL) and Self- Coff-Correeting Bayesian Optimization (SCoreBO), two acquisition func- tions that explicitly consider hyperparameter-ineee selecting which points to query. We achieve high-end performance on both active learning and Bayesian optimization tasks, and successfully learn hyperparameters and kernel designs at improved rates compared to conventional methods. SCoreBO breaks ground for new methods in the space of joint active learning and optimization of black-box functions, which allows it excel in high- dimensional BO, where learning important dimensions are vital. Moreover, the potential downside of self-correction is displayed when the model structure does not support the task at hand, or when self-correction is not required to solve the task. For future work, we will explore additional domains in which SAL and SCoreBO allow for increased model complexity in BO applications.
disagreement in the posterior distribution when
6 Limitations
SCoreBO displays the ability to increase optimization efficiency on complex tasks that necessitate accurate modeling. However, SCoreBO's efficiency is ultimately contingent the intrinsic ability of the GP to model the task at hand. Appendix 19 demonstrates this issue for the Rosenbrock (4D) function, where SCoreBO performs worse relative other acquisition functions. There, the hyperparameter values increase over time instead of converge, which suggests that the objective is not part the class of functions defined by the kernel. Thus, the self-correction effort is less helpful towards optimization. Moreover, increasing the model capacity, such as in Sec. 4.3 comes with increasing resourees allocated towards self-correction. such resource allocation may not yield the best result, especially if increased model complexity is unwarranted. This is evident from the synthetic AddGP tasks, where despite accurately identifying the additive components, SCoreBO does not provide substantial performance gains over NEI. Lastly, SCoreBO's reliance on fully Bayesian hyperparameter treatment makes more computationally demanding than MAP-based alternatives, limiting its use in high- thgouhhroughput applications.
highly constrained-but budget applications.
10

---

Acknowledgements
We thank the anonymous reviewers for their valuable contributions related SAL-KL and and its relationship with BALD, well as their general feedback on the clarity of the paper and how our method was conveyed. We also thank Eytan Bakshy for their helpful feedback on earlier versions of this paper. Luigi Nardi was supported in part by affiliate members and other supporters of the Stanford DAWN project - Ant Financial, Facebook, Google, Intel, Microsoft, NEC, SAP, Teradata, and VMware. Carl Hvarfner, Erik Hellsten and Luigi Nardi were partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Luigi Nardi partially supported by the Wallenberg Launch Pad (WALP) grant Dnr 2021.0348. Frank Hutter acknowledges support through TAILOR, a project funded by the EU Horizon 2020 research and innovation programme under GA No 952215, by the Deutsche (DFG, German Research Foundation) under grant number 417962828, by the state of Baden-Württend Württemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG, and by the European Research Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765). The computations were also enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at LUNARC partially funded by the Swedish Research Council through grant agreement no. 2018-05973. Funded by the European Union. Views and opinions expressed are however those the authon(s) only and do not necessarily reflect those of the European Union the ERC. Neither the European Union nor the ERC can be held responsible for them.
Forschungsgermiinchhan
Funded
the European Union

---

References
[1] Atkinson, A. and Donev, A. Optimum Experimental Designs. Oxford science publications. Clarendon Press, 1992. ISBN 9780198522546. URL httpp://bbokks: wtw.::eee...ooo........eee// /RL.ttgee//boossggggee...ss
Add=mmUA__-M7S00
[2] Balandat, M., Karrer, B., Jiang, D. R., Daulton, S., Letham, B., Wilson, A. G., and Bakshy, E. Botorch: A framework for efficient monte-carlo bayesian optimization. In Advances in Neural Information Processing Systems, 2020. URL http::ttp/// //rpvv/r/vvvvv/a////v///aa/aiiia httg//bbs1991000 06403
[3] Berkenkamp, F., Schoellig, A. P., and Krause, A. No-regret bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 20(50):1-24, 2019. URL http:// mlr.
213.html
[4] Berkenkamp, F,, Krause, A., and Schoellig, A. Bayesian optimization with safety constraints: Safe and automatic parameter tuning in robotics. Machine Learning, 06 2021. doi: 10. s 009944-021-0019-11
[5] Bingham, E., Chen, J. P., Jankowiak, M.. Obermeyer, F,, Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N. D. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research, 2018.
[6] Bodin, E., Kaiser, M., Kazlauskaite, I., Dai, Z., Campbell, N., and Ek, C. H. Modulating surrogates for Bayesian optimization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 970 979.9999.00 PMLR, 13-18 Jul 2020. URL htt.:ttst///rceeeddiiie mlr press/v119/bodin22) html
[7] Bogunovic, I. and Krause, A. Misspecified gaussian process bandit optimization. In Ran- zato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 3004-3015. Curran Associates, Inc., 2021. URL URL htt.tttsp://rrcceeiinnn www.iinnnnnnnurrreeeeeeeeeeeeree neurips. ct/ccc//pppaao wwcppppr_raaee www.ffflleepppeeeee pttpp/aaapre2////////// 477
pdf
[8] Bull, A. D. Convergence rates of efficient global optimization algorithms. 12:2879-2904, 2011.
[9] Calandra, R., Gopalan, N., Seyfarth, A., Peters, J.. and Deisenroth, M. Bayesian gait optimiza- tion for bipedal locomotion. In Pardalos, P. and Resende, M. (eds.), Proceedings of the Eighth International Conference Learming and Intelligent Optimization (LION' 2014.
[10] Chaloner, K. and Verdinelli, I. Bayesian Experimental Design: A Review. Statistical Science, 10(3):273 304, 1995. doi: 10. 10.1214/ss/ /1tp/s/////77999999 URL /RL/pttts://doi.orgg /2/4//s\/ 1177009939
[11] Chen, Y., Huang, A., Wang, Z., Antonoglou, I,, Schrittwieser, J., Silver, D., and de Freitas, N. Bayesian optimization in alphago. CoRR, abs/18 812.0885, 2018. URL htth:http///xii org/ abs/1812.00 06855
[12] Cortes, C. and Vapnik, V. Support vector networks. Machine Learning, 20:273-297, 1995.
[13] Cowen-Rivers, A. I., Lyu, W., Wang, Z., Tutunov, R., Hao, J., Wang, J., and Bou-Amman H. HEBO: heteroscedastic evolutionary bayesian optimisation. CoRR, abs/2012.03826, 2020. URL hstps://arxiv.orgg// wwtp//////a////////aa /tw./ 03826
[14] Ejjeh, A., Medvinsky, L., Councilman, A., Nehra, H., Sharma, S., Adve, V., Nardi, L., Nurvi- tadhi, E., and Rutenbar, R. A. Hpvm2fpga: Enabling true sardware-agnoste fpga programming. In Proceedings of the 33rd IEEE International Conference on Application -ppecfffc Systems, Architectures, and Processors, 2022.
[15] Eriksson, D. and Jankowiak, M. High-dimensional Bayesian optimization with sparse axis- aligned subspaces. In Campos, C. and Maathuis, M. H. (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Pro- ceedings of Machine Learning Research, pp. 493-503. PMLR, 27-30 Jul 2021. URL https://procest http:///w.....t.p..oooo/////aaaa
mlr, press//v/61/errisson21.
12

---

[16] Eriksson, D., Pearce, M., Gardner, J., Turner, R. D., and Poloczek, M. Scalable global optimization via local bayesian optimization. In Wallach, H., Larochelle, H., Beygelzimer, A.. d'Alché-Buc, F., Fox, E., and Gareett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. / neurips cc
pdf
[17] Frazier, P. I. A tutorial on bayesian optimization. arXiv preprini arXiv: 10XX:v:800..0m 1807.0281111111 2018.
[18] Gardner, J.. Malkomes, G., Garnett, R., Weinberger, K. Q., Barbour, D., and Cun- Bayesian active model selection with an application to automated au- diometry. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances Neural Information Processing Systems, volume 28. Curran Associates. Inc., 2015. URL https: https:://proceeddingg/ hew.eeeeddiinnnneein.s.... wwtrriis.cc/pppeee http:/pper..illeppppprrp//////// 48725391E400033beee9999955668--3
ningham,
pdf
[19] Gardner, J., Guo, C., Weinberger, K. Garnett, R., and Grosse, R. Discovering and Exploiting Additive Structure for Bayesian Optimization. In Singh, A. and Zhu, J. (eds.), Proceedings of the Seventeenth International Conference Artificial Intelligence and Statistics (AISTATS), volume 54, pp. 1311-1319. Proceedings of Machine Learning Research, 2017.
[20] Gardner, J. R., Pleiss, G., Bindel, D., Weinberger, K. Q., and Wilson, A. G. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
[21] Griffiths, R.-R. and Hernández-Lobato.] J. M. Constrained bayesian optimization for automatic chemical design. arXiv: Machine Learning, 2017.
[22] Hennig, and Schuler, C. Entropy search for information--fffccent global optimization. 98888 (1):1809-183,, 2012.
[23] Hennig, P. and Schuler, C. J. Entropy search for information-efficienn global optimization. Journal of Machine Learning Research, 13(1):1809-1837, June 2012. ISSN 1532-4435.
[24]
Hernández-Lobato, J. M., Hoffman, M. W., and Ghahramani, Z. Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems, 2014. URL http:::tpp///ooeeeeeeiiii new.eeeeeddinnnneneneeeeeeee nwwrinu httpscccaaaaer/14/ii22.
[25] Hoffman, M. D. and Gelman, A. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Research, 15(47)) 2014. URL (Rtp::////jmm wwwg/pappgaaa.... wwieeee//v//////e www.1///ohaaannammmm 14a. html
[26] Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M. Bayesian active learning for classification and preference learning. arXiv preprint 112.541 2011.
[27] Hutter, F., Hoos, H., and Leyton-Brown, K. Sequential model-based optimization for general algorithm configuration. In Coello, C. (ed.), Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION' 11), volume 6683, pp. 507-523, 2011.
[28] Hutter, F., Lindauer, M., Balint, A., Bayless, S., Hoos, H., and Leyton-Brown, K. The configurable SAT solver challenge (CSSC). 243:1-25, 2017.
[29] Hvarfner, C., Hutter, F,, and Nardi, L. Joint entropy eearch for maximally-informel bayesian optimization. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
[30] Hvarfner, C., Stoll, D., Souza, A., Lindauer, M., Hutter, F,, and Nardi, L. PiBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization. International Conference on Learning Representations, 2022.
[31] Jones, D., Schonlau, M., and Welch, W. Efficient global optimization of expensive black box functions. 13:455-492, 1998.
13

---

[32] Kandasamy, K., Schneider, J., and Póczos, B. High Dimensional Bayesian Optimisation and Bandits via Additive Models. In Bach, and Blei, D. (eds.). Proceedings of the 32nd International Conference on Machine Learning (ICMI. 15), volume 37, pp. 295-304. Omnipress, 2015.
[33] Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., and Xing, E. P. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems, 31, 2018.
[34] Kirsch, A., Van Amersfoort, J., and Gal, Y. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
[35] Lalchand, V. and Rasmussen, C. E.. Approximate inference for fully bayesian gaussian process regression, In Symposium on Advances in Approximate Bayesian Inference, pp. 1-12. PMLR, 2020.
[36] Letham, B., Brian, K., Ottoni, G., and Bakshy, E. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis, 2018.
[37] MacKay, J. Information-based objective functions for active data selection. Neural compula- fion, 4(4):590-604, 1992.
[38] Malkomes, G. and Gameett, R. Automating bayesian optimization with bayesian optimization. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 http:ttt:: httphtttts//prooeeeding www.ris.....p........... wwwr.ps.cc/paper.....eeeeeeeeeee htce/aper_ffiiie faarrrffllsppppee 0bb42219663000a8bcc270002---
httppppppp..lleee/ppp//////////1 patp://///e/////eeee pdf
[39] Malkomes, G., Schaff, C., and Garnett, R. Bayesian optimization for automated model selection. In Hutter, F., Kotthoff, L.,, and Vanschoren, J. (eds.), Proceedings of the Workshop on Automatic Machine Learning, volume 64 of Proceedings of Machine Learning Research, pp. 41-47, New York, New York, USA, 24 Jun 2016. PMLR. UtRLhhttss//pproeeeee 2016.
mlr press/vv4.
[40] Mayr, M., Ahmad, F., Chatzilygeroudis, K. I., Nardi, L., and Krüger, V. Skill-based Multi- objective Reinforcement Learning of Industrial Robot Tasks with Planning and Knowledge Integration. CoRR, abs/2203.10033. 2022. URLp:tts:://oo: htt////...orr/1....488//////1 2203 10033
[41] Mayr, M., Hvarfner, C., Chatzilygeroudis, K., Nardi, L., and Krueger, V. Learning skill-based industrial robot tasks with user priors. IEEE 18th International Conference Automation Science and Engineering, 2022. URL /ttts:://artiiii org/ abs//////2/22/ 01605
[42] Moss, H. B., Leslie. D. S., Gonzalez, J., and Rayson, P. Gibbon: General information- based bayesian optimisation. Journal of Machine Learning Research, 22(235):1 191 2021. URL http://jmlr: org/ wwtg/papers/v22/2222 21201200000200001 html
[43] Nardi, L., Koeplinger, D., and Olukotun, K. Practical design space exploration. In 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 347-358. IEEE, 2019.
[44] Nguyen, Q. Wu, K., Gardner, J., and Garnett, R. Local bayesian optimization via maximizing probability of descent. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D.,, Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 13190-13202. Curran Associates, Inc., 2022. URL httpttt::///tceeeee
neurips wwwpepprpppppppppcpppppeeee.e..p palplprpeeereee//apppppppppesesr Powfeaaapparrrooeeeeeeeeeeeeeeee .pdf
[45] Nguyen, Q P., Low, B. K. H.. and Jaillet, P. Rectified max-value entropy search for bayesian optimization, 2022. URL https://arxxv... org./ivoorrrreee http://bb/2200.11112 13597
[46] Osborne, M. A. Bayesian Gaussian processes for sequential prediction, optimisation and quadrature. PhD thesis, Oxford University, UK, 2010.
14

---

[47] Papenmeier, L., Nardi. L.. and Poloczek, M. Increasing the scope as you learn: Adaptive bayesian optimization in nested subspaces. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho. K. (eds.), Advances in Neural Information Processing Systems, 2022. URL. https //openreview //pparrevieer.ee//ororddeeeeee wet.:/oommmm//d///f111100
[48] Rainforth, T., Foster, A., Ivanova, D. R., and Smith, F. Modern bayesian experimental design. 2023.
[49] Rasmussen, C. and Williams, C. Gaussian Processes for Machine Learning. The MIT Press, 2006.
[50] Ríis, C.,, Antunes, F. N., Hüttel, F. B., Azevedo, C. L., and Pereira, F. C. Bayesian active learning with fully bayesian gaussian processes. arXiv preprini arXiv:2205. 10186, 2022.
[51] Ru, B., Osborne, M. A., Meleod, M., and Granziol, D. Fast intfrmationn-hhereee Bayesian optimisation. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4392. PMLR, 15 Jul 2018. URL https::/prooeeedingss mml. mlr http:///w....ii..ppee/////////// wtwpppeeee/eeeeeeeeeeee html
[52] Ru, B., Wan, X., Dong, X., and Osborne, M. Interpretable neural architecture search via bayesian optimisation with weisfeiler-lehman kernels. arXiv preprint arXiv:2006.07556, 2020.
[53] Sehić, K., Gramfort, A., Salmon, J., and Nardi, L. LassoBench: A High- High--Dmeensional Hy- perparameter Optimization Benchmark Suite for Lasso. arXiv preprint arXiv:2111.02790, 2021.
[54] Settles, B. Active learning literature survey. 2009.
[55] Snoek, J., Larochelle, H., and Adams, R. Practical Bayesian optimization of machine learning algorithms. In Bartlett, P., Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurlPS' 12), pp. 2960-2968, 2012.
[56] Snock, J., Swersky, K., Zemel, R., and Adams, R. Input warping for Bayesian optimization of non-stationary functions. In Xing, E. and Jebara, T. (eds.), Proceedings of the 31th International Conference Machine Learning, (ICML' 14), pp. 1674 1682. Omnipress, 2014.
[57] Srinivas, N., Krause, A., Kakade, S. M.. and Seeger, M. W. for gaussian process optimization in the bandit setting. IEEE Transactions on Information 525335033, May 2012. ISSN 1557-9654, doi: 10.1 10..10999i.22111.218003 URL attp://dx. dww.doo..o.g........... org/ 10. 1109/TIT. 2011. 2182033
cettmaattionthhooeeee regret bounds
[58] Stanton, S., Maddox, W., and Wilson, A. G. Bayesian optimization with conformal prediction sets. In Ruiz, F., Dy. J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 959-986. PMLR, 25-27 Apr 2023. URL httpp:://proceddieee
mlr. ppwpss/v006/stattonn222222
html
[59] Takeno, S., Fukuoka, H., Tsukada, Y., Koyama, T., Shiga, M., Takeuchi, I., and Karasuyama, M. Multi-fidelity Bayesian optimization with max-value entropy search and its parallelization. In III. H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pp. 9334-9345. PMLR. 13-18 Jul 2020. URL httph: UR/httpss//prooeeeing
mlr wttppppwsee/eeppppppppppp.p wttps/////11///aateeee0000 html
[60] Takeno, S., Inatsu, Y., and Karasuyama, M. Randomized Gaussian process upper confidence bound with tighter Bayesian regret bounds. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 33490-33515, PMLR, 23-29 Jul 2023. URL https: httt:e://proceedings mlr. wttppppwweeeeeeeeepppeeppee wtw.//202//aaeeo22aaaaa httl
[61] Tu, B., Gandy, A., Kantas, N., and Shafei, B. Joint entropy search for multi-objective bayesian optimization. In Oh, A. H., Agarwal, A., Belgrave. D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URI http:///wpp//ee./////// wwt.:orunffffdd ZChgD80oGds
15

---

[62] Turner, R., Eriksson, D., McCourt, M. Kiili, J., Laaksonen, E., Xu, Z., and Guyon, 1. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In Escalante, H. J. and Hofmann, K. (eds.), Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pp- 3-26. PMLR, 06-12 Dec 2021. URLhttps //proceedings. mlr. press/ vttes/vv33////// preess/133/burrer22 wtt.////uueeeaaa.aa html
[63] Wan, X., Nguyen. V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisation over high- nigh-dimensional categorical and mixed search spaces. In Mcila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10663-10674, PMLR, 18-24 Jul 2021. htL.httss://proeeee
mlr wwwpppesssvvviv aaaass/v139//an//// wan21b.... hhmmmmllmmmmmm...m
[64] Wan, X., Nguyen, V., Ha, H., Ru, B., Lu, C., and Osborne, M. A. Think global and act local: Bayesian optimisation over high- dimeesiinniii categorical and mixed search spaces. International Conference on Machine Learning (ICML) 38, 2021.
[65] Wang, Z. and Jegelka, S. Max-value entropy search for efficient bayesian optimization. In International Conference on Machine Learning (ICMI.), 2017.
[66] Wang, Z., Dahl, G. E., Swersky, K., Lee, C., Nado, Z., Gilmer, J., Snoek, J., and Ghahramani, Pre-trained Gaussian processes for Bayesian optimization. arXiv preprint arXtv:2109.08215. 2023.
[67] Wilson, J. T., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. P. Efficiently sampling functions from gaussian process posteriors. In International Conference on Machine Learning, 2020. URL //axiivv////x.v../oo//v wtg/ 09309
[68] Wu, J., Poloczek, M., Wilson, A. G., and Frazier, P. Bayesian optimization with gradients. In Guyon, I., Luxburg. U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates,
Inc., 2017. URL http:/tttt::/preeeeeineeee mew.rrooeeeeiiinnnn.n.........p neurips. cc/ S3000eef1.60SSfaa00106eeeee22223
[69] Wynne, G., Briol, F--.., and Girolami, M. Convergence guarantees for gaussian process means with misspecified likelihoods and smoothness. J. Mach. Learn. Res., 22(1). jan 2021. ISSN 1532-4435.
[70] Yao, Y., Vehtari, A., and Gelman, A. Stacking for non-mixing bayesian computations: The curse and blessing of multimodal posteriors, 2020.
[71] Zhang, B., Cole, D. A., and Gramacy, R. B. Distance-distriutte design for gaussian process surrogates, 2019.
16

---

A Additional Related Work
We review approaches that address model accuracy in Bayesian optimization.
Expanded Space of Models
expands model uncertainty to kernel design in order to find the most accurate model possible within expanded class of functions, whereas [56] expands model uncertainty to involve warpings of the input space. Since the expanded class of functions offers additional modelling flexibility, BO can ideally be conducted vanilla setting, As demonstrated with AddGPs in Sec. [4.3] SCoreBO can work in conjunction with kernel search find such models an accelerated rate.
38
39
accurate model than under
Simplified Modelling A contrasting line of work involve methods which restrict modeling by reducing the level of detail [6] scope of the optimization [16 44 63.. This line of work acknowl- edges that modeling may not, or should not, be globally accurate or granular in order to conduct optimization efficiently within the allocated budget. As such, these lines of work address the issue of model accuracy by diametrically opposed philosophies. SCoreBO offers an orthogonal approach to model accuracy by accelerating the convergence of the model at hand, regardless of its level of complexity.
B
Supplementary Material on Experiments
B.1
Experimental Setup
All the relevant methods are implemented as acquisition functions in BoTorch. For both the active learning and BO experiments, we run NUTS [25] Pyro [5] draw samples from the GP posterior over hyperparameters. Tab, displays the parameters of the MCMC in detail, as well as other relevant parameters of various MC estimations throughout the article. For the active learning experiments, we mimic the experimental setup used in Riis et al. [50]. and put a log-normal distribution C.(0,, 3) on the lengthscales, outputscale variance and noise variance. Furthermore, we consider the mean constant
learnable parameter in the BO experiments, with conventional ( 1) prior on the standardized inputs. When referring to the BoTorch priors, the priors are I(3, 6). [(2, 0.15), and 1(1.1, 0.05) for the lengthscales, outputscale, and noise variance, respectively, with the same prior on the learnable constant mcan c.
8D Gaussian Process sample task For the 8D GP sample task, we utilize the lognormal prior from Sec. B on the hyperparameters. The hyperparameters of the sampled objective functions are outlined in Tab. [ As such, Tab. displays the true hyperparameters of the task we are trying to optimize. These hyperparameters are referenced Fig. 17 and Fig.
Task
0²
l
Kerne Matérn
GP sample- 8D
0.1 exp 10 0.5, 0.5, 0, 0, 1.5, 1.5, 1.5]
Table I: Hyperparameters of the 8D GP sample task.
SAASBO experiments For the SAASBO experiments, we utilize Ax 3l which runs the BoTorch implementation of SAASBO with the deafult prior the hyperparameters 0.8.4, which differs slightly from the paper by Eriksson & Jankowiak (15] The lengthscale parameters are given hierarchical prior as 72 HC(a), x² HC(1) and
per BoTorch version
We retain the default value
RFF
of a = 0.1. The noise variance, outputscale and mean constant are given the priors o? T(0.9, 10), o3 [(2, and N 1).
Additive Gaussian Process experiments The Additive GP setup closely resembles that of 19. An additive partitioning is sampled, and the marginal likelihood of the model is maximized with regard to 0 = (e, ao,y,, j.). We utilize slightly adapted proposal distribution, and fix maximal number of additive partitions gmaz- Moreover, each dimension df belongs to one distinct additive decomposition
17

---

where 9a E [1..... [..z)) At each iteration of the MCMC scheme, two dimensions i.j = [(i.j) [I....Di, 2 # j]. are sampled uniformly at random and assigned the same new group index g. where g ~ 21(1, Imax). Setting Di 9: 9j g thus proposes that dimensions and belong to the same additive decomposition. Under this proposal distribution, the proposed number of additive decompositions can never surpass Imaz. but lesser number of additive groups can be proposed. We utilize the same warm -tarting mechanism as described in Gardner et al. [1]] where at each iteration, the final accepted sample from the previous iteration acts as the initial proposal. The substitution proposal distribution from the (unavailable) original implementation [9] was made to simplify batch GP implementation in GPyTorch [20]. We choose to employ BoTorrh prior the hyperparameters for Additive GP tasks and use a squared exponential kernel, as described in
Task
Imaz
GP sample- - 2 GP sample 2 2)
3
5
Cosmological constants
4
Table 2: Additive tasks and their respective maximal number of maximal additive decompositions.
HEBO Experiments For the warped GP experiments, we adapt slightly more conservative priors due the large number of hyperparameters of the model. Specifically, we set C. 1) priors on the on lengthscales, outputscale variance and noise variance, For the input warpings, we employ Kuwaraswamy distribution, differentiable-CDD alternative the input warpings proposed by Snoek et al. [56]. For each input dimension j. the untransformed input 2, has a transformation applied as
Zj = 29)8
(11)
where 2j is the resulting, transformed input. The warping parameters a and 3 are both given CN 0.1) prior. We note that the HEBO experiments are susceptible to the number of hyper- parameter sets. As such, we do not recommend running the experiments than fewer number of hyperparameter sets than that outlined by Tab. B as substantially fewer did not yield substantial empirical gains relative to vanilla BO.
Task
Warmup
Thinning
No. hyperparameter sets
No. optima
No. RFFs
Active Learning
256
16
16
N/A
N/A
BO Synthetic
256
16
16
8192
BO - GP samples
256
16
16
8
2048
BO SAASBO
128
8
16
8192
BO - Additive GPs
32
4
12
2048
BO Warped GPs
64
32
2048
Table 3: MCMC hyperparameters for all experiments. For the AddGP experiments, cach hyperparameter set
involves the sampled additive decomposition and its associated MAP-trained hyperparameters. The total number of hyperparameter sets drawn are Warmup Thinning * No. Hyperparameter
B.2 Benchmarks
For the active learning benchmarks, we follow Riis al. [50] in the types benchmarks and noise levels used. Each benchmark, well as its search space, dimensionality and noise level is described in Tab. and Tab. 5 for AL and BO, respectively. The noise level for all of the BO synthetic test functions were set to ae 0.5, except the Rosenbrock benchmarks, where the noise standard deviation was set to o, = 2.5 due to the extremely large output range of the function. A smaller noise level would consequently bring the signal-to-noisy ratio under the permitted threshold supported by BoTorch.
Compute resources. All experiments are carried out on Intel Xeon Gold 6130 CPUs. Each repetition of the tasks in Sec. Sec. cores. Approximately 1, 000 core hours are used for each of the AL synthetic tasks, 2000 for the BO synthetic tasks, and 5000 for each task in Sec. 4.3
Sec. 4.2 are run on 4 cores, and the tasks in Sec. 4.3 are run on
18

---

Task
Dimensionality
O,
Search space
Gramacy
0.1
[0.5, 2.5]
Higdon
1
0.1
[0, 20
Gramacy
2
0.05
[-2, 6)
Branin
2
11.32
[-5, 10 x [0, 15]
Ishigami
3
0.187
n. π
Hartmann-6 6
0.0192
[0. 1j
Table 4: Benchmarks used for the active learning experiments.
Task
Dimensionality
0x
Search space
Branin
2
0.5
-5,, 10] x 10, 15
Rosenbrock-2
2
2.5
[-1.5, 1.5)
Hartmann-3
0.5
[0, 1]
Rosenbrock-4
4
2.5
[-1.5, 1.5]
Hartmann-4
4
0.5
[0, 1j
Hartmann-6
6
0.5
0, 10
Table 5: Benchmarks used for the Bayesian optimization experiments.
C Additional Experiments
We display the RMSE performance of each the SAL variants, a comparison of the MC and MM variants of SAL and SCoreB0.
C.1 AL RMSE Performance
Fig. II displays the performance of both SAL variants and benchmark AL acquisition functions for the same set of tasks as in Sec. [.]] We observe that SAL-WS consistently displays top performance, whereas SAL-HR lags behind substantially. This showcases SAL-HR's emphasis hyperparameter learning as opposed to global exploration. By accurately assessing hyperparameters while sacrificing global exploration, SAL-HR ensures predictions with calibrated uncertainty, while sacrificing the accuracy in predictive mean that follows from exploring the search space. SAL-WS offers compromise which performs well under both metries, sacrificing hyperparameter learning and calibration for accuracy in predictive mean.
Le
350
300 360
35% 50%
130%
SAL- 111
SAN WC5
GUMCG
TALL
DGBC
Figure 11: Root Mean Squared Error (RMSE) six active learning functions and the (smoothed) relative rankings throughout each run for QBıGP, EQBC, BALD and SAL using Wasserstein and Hellinger distance. plot mean and one standard error for 25 repetitions.. SAL-WS is the top performing method, placing first in relative rankings. On Ishigami, only SAL-HR and BALD produces stable results.
C.2 SCoreBO Distance Measure Ablation Analysis
We compare the Hellinger and Wasserstein variants of SCoreBO, both utilizing the MC approximation of the statistical distance. Since the MC approximation is asymptotically exact, we can better assess the performance of each distance metric, without having to consider the confounding factor that the MM approximation introduces. We note that SCoreB0-Wi -S outperforms SCoreBD-HR
In Fig. 13
19

---

on two tasks, but SCoreBO-HR is the overall more consistent approach. We hypothesize that the relative failure of SCoreBO-WS on Rosenbrock (4D) caused by the objective's non-stationarity. which likely causes exceedingly large exploration of the hyperparameter space. This is supported by where the hyperparameters diverge over time on the Rosenbrock function.
Fig. D.3
Furthermore, in Fig. 12 we compare the KL and Hellinger variants of SCoreBO, both utilizing the moment matching estimation of the posterior. Notably, SCoreBO-KL is the natural extension of BALD to the self-correcting framework, in line with Prop. 1. SCoreBO-HR performs marginally better than SCoreBO-Ke KL, winning out of tasks. However, the two variants are relatively close.
Branin
Hartmann 3
Hartmann 4
Hartmans. 6
Rasenbrock 2
Rosenbrocke4
0.5
0.50
0.25
0.25
SCone50 WS
SComedd IIIl
0.0
0.00
0.00
2.0
Log inforenes
0.25
-025
40.75
0.500
0.5
0.0
1.0
-1 1.00
-0.75
-0.5
-1.
1.25
-1.0
0.0
100 50
1.0
100
1000
100
1000
Figure 12: Log Regret of the Hellinger and Wasserstein MC-variants of SCoreBO. Both variants are competitive on all benchmarks, except for Wasserstein Rosenbrock (4D) which lags behind slightly. Overall, Hellinger constistent, and wins 4 out of henchmarks.
Pranin (20)
Hartmann (3D]
Hartmann (4D)
Hartmenn (8D)
Resenbrack 12
Rewenbrakk (4D)
SCoraB0
ScorcBO KL
-356
244
-122
-08
4.5
0L5
-33
-1.
-1.
150
150
R4 Iteatton
150
30
100
100
155
tteration
Iecation
Figure 13: Log Regret Hellinger moment matching variants SCoreBO. Both variants are competitive on all benchmarks. Overall, Hellinger is marginally better, and wins 3 out of 6 benchmarks with an approximate tie on Branin.
C.3 SAASBO Noise Ablations
In Fig. 13 We measure the performance of SCoreBD and NEI on the embedded 25-dimensional Ackley (4D) and Hartmann (6D) tasks with varying noise levels using the SAAS [15] prior. We run three noise levels for each task: Noiseless (solid line), low (dashed line), where the noise standard deviation corresponds to 3% of total output range for Hartmann (6D), and 1.3% for Ackley, and high (dotted line, 13.3% 4%). With increasing levels of noise, the difficulty of inferring active dimensions is expected to increase substantially, which should turn hamper BO performance. In that NEI and SCoreBO perform comparably on noiseless tasks (solid line) finding
Fig. [4] close-to-optimal solutions within 100 iterations. Moreover, for small levels of noise), the performance still comparable. However, we observe drastic fall-off in performance for NEI the highest noise level, whereas SCoreBO's degrades gracefully. Notably, SCoreBO almost retains the performance of the noiseless at the highest noise level for Ackley.
D Hyperparameter convergence
and 19 We demonstrate examples of hyperparameter convergence in AL and BO, as well as an example of hyperparameter divergence on the Rosenbrock (4D) function, where SCoreBO performs marginally worse.
In Figures 15 16
20

---

4 (25D)
Hartmann-6 (25D)
-05
Min. Ohhorrved Uniun
Olbverved
-13
10-
NEI
-20
SCoreBO
0.0
02 0.31
0, 1.0
Iterationn
Iteration
Figure 14: Best observed value for varying levels of ohservation noise for NEI and SCoreBO using SAAS priors on the 25D-embedded lower-dinensional test functions. For Low levels of noise, the performance of NEI and SCoreBO are comparable, but SCoreBO retains performance substantially better for higher levels for noise.
D.1 Active Learning Tasks
We display the hyperparameter convergence of SAL-WS, SAL-HR and the baseline active learning acquisition functions in Fig. [5] Both variants display accelerated hyperparameter learning compared BQBC. SAL-HR particular achieves low-variance hyperparameter uncertainty on Ishigami and the nigher-dimensional Hartmann-6 6, where other methods struggle. We obtain approximately correct hyperparameters for these tasks by randomly sampling 300 points on the noiseless benchmark, thereafter performing MCMC and averaging the sampled hyperparameter estimates in logspace. The noise level known priori. and estimates the other hyperparameters with substantially greater certainty than other methods. We note that there is drift in the hyperparameters as the number of observations increase, where output- and lengthscales trade off to reduce model complexity. As such, we provide an approximately stationary alternative in Fig. [6] where the outputscale is removed to avoid drift. both cases, SAL-HR displays superior hyperparameter convergence, obtaining accurate hyperparameters in far fewer iterations, and with substantially less uncertainty than the alternatives.
4
22
Log laperpanae. valoe
SAL-
BALD
Q JRMGP
100
201
302
1.00
1.0
230
300
100
3010
100
200
300
Figure 15: Hyperparameter convergence on the Ishigami test function with outputscalc. While no acquisition
converges, SAL-HI and BALD display substantially more stable hyperparameters than other approaches.
b
f
SAL
Log Ioperpueun.
DALD
3.5
BQBC
QBMGP
302
1.00
100
220
200
Figure 16: Hyperparameter convergence on the Ishigami test function without outputscale. SAL-HR and BALD display stable hyperparameter convergence, and are the only acquisition function accurately estimate all parameters.
D.2 GP sample tasks
We display the convergence of SCoreBO and NEI with a wide lognormal and BoTorch prior on the GP sample task. We observe that lognormal prior is well-aligned for most hyperparameters, whereas BoTorrh prior is misaligned. This is evidenced by the unimportant dimensions fa, l, and fg. which have suggested lengthscales that are incorrect by more than an order of magnitude. Nevertheless, SCoreBO suggests lengthscales that approximately twice as long (105:2 1.8) as NEI 10 10000.05
21

---

0.9), and thus avoids unneessary exploration along these dimensions. Moreover, SCoreB0 correctly identifies the most important dimensions l2, and E with good accuracy quickly, whereas NEI struggles to identify &, SCoreB0 slightly overestimates the importance of dimensions 2 and 3, likely to compensate for the inability to accurately estimate the importance of other hyperparameters.
Nutay
ScomD0
Figure 17: Hyperparameter convergence on the 8-dimensional GP sample for the broad log-normal prior. The black dashed line indicates true lyperparameter values. Mean and standard deviation are plotted across 20 repe- titions, and 3 iteration moving average of the plotted moments is applied increase readability. Lengthscales ordered smallest (most important) to largest (least important), SCoreB0 finds accurate hyperparameters faster, has the most accurate values for all hyperparameters, and has substantially lower variance for all important (i.c. not 6,, lT, and fs) hyperparamcters except for the noise variance.
Nutoy
ScamBO
Figure 18: Hyperparameter convergence on the s-dimensional GP sample for the BoTorch priors. The black dashed line indicates hyperparameter values. Mean and standard deviation plotted across 20 repetitions, iteration moving average
plotted moments is applied increase readability. Lengthscales la ordered smallest (most important) to largest (least important). SCoreBO finds accurate hyperparameters faster, has the most accurate values for all hyperparameters, and has substantially lower variance for all important (i.c. not &a, l,, and fa) hyperparameters except for the noise variance.
D.3 Hyperparameter Divergence on Synthetic BO tasks
We highlight additional examples on synthetic BO test functions where hyperparameters diverge. Due to the non-stationary structure of Rosenbrock in particular (and a lesser extent, Branin), hyperparameters values diverge as the number of observations increase. In particular, the extreme steepness along the edges suggests an exccedingly large outputscale. With increasing observations, a lengthscale-outputsee. trade-off occurs, where both hyperparameters grow seemingly indefinitely. Notably, this behavior consistent regardless of the acquisition function (BO, AL, SOBOL). Due to the restricted hyperparameter set employed in the AL tasks, this problem is distinct the BO tasks.
o²
f
0,5
32
NBN
50mB0
110
130
181
100
181
120
32
150
190
180
Figure 19: Hyperparameter divergence for SCoreBO and NEI Rosenbrock (4D), The outputscale grows larger with increasing iterations, and the lengthscales grow similarly large countermeasure.
22

---

E
Approximation Strategies
We display the quality of the moment matching approximation for both the Hellinger and Wasserstein distance. Moreover, we compare the performances of the MM and MC approaches.
E.1
MC approximation of SAL and SCoreB0
Using Monte Carlo, different distances are most efficiently estimated in different manners. To approximate the Wasserstein distance, we utilize quasi-Monte Carlo. From the definition of the distance in one dimension, we obtain
L Σ (Q(up) - P(u))
( 2) = [Q(u) du P(u)ddd =
(12)
where us ~ 2(0 1), and P(x) and Q(x) are the respective cumulative distributions for p(æ) and q(x). To approximate the Hellinger distance, we obtain
(p. -
g(x) p(x)
pla)d
Σ
q(xe)
(13)
p(xz)
where ~ p(z) is sampled using MC. In SCoreBO, p(z) is the marginal p(ya) (D), and ((x) each of the various conditionals p(yal *. 0, D).
E,2 Performance of Monte Carlo
We display the performance of the MC variants of SAL-WS and SCoreBO-HR compared to their MM counterparts. Overall, performances are comparable, as each variant slightly exceeds the other on couple of benchmarks. On the most complex benchmarks (Ishigami, Hartmann-4, Hartmann- 6), the MC variant outperforms MM slightly, which suggests that MC is increasingly justified as disagreement in the posterior gets larger.
Higdon
Gramacy (1D)
Gramacy
Ishigami
SAL. 118
SAL WS
SAL- WSS 2ND (I.00C)
15
1.0
.07
00
Figure 20: Negative marginal log likelihood (MLL) of the SAL MC (blue) and MM (red) variants the active learning benchmarks. Overall performance comparable, with three effectively tied benchmarks. MC outperforms slightly Hartmann-4 Hartmann-6, and MM on Hartmann-3.
Branin
Hartmann-3
Hartmonn 4
Hartmann
Rosembracks
Rosembrocis 4
Inforence Sogres
ScCoebil
-325
1.3-
-350
-6.
-8.
00
455
8.5
-3.2
+30
-34
Figure Log regret of SCoreB0-MM and SCoreBO-MC on the synthetic BO benchmarks. Overall perfor- mance is comparable, with MM outperforming marginally on 4 out of 6 tasks. MC notably outperforms slightly on the difficult Ishigami test function.
23

---

E.3 Hellinger Distance Approximation
We display the accuracy of the moment matching approximation, and the sensitivity of the MC approximation to the number of samples L. In Fig. 24d Fig. 25 we highlight two examples of the moment matching approximation in comparison to large-scale, asymptotically exact variant of the MC approximation with 2048 samples. In Fig. 24 the MM approximation struggles to capture the sharp, multimodal surfaces in (blue), and consistently overestimates the distance in (orange). In Fig. [55 the included conditional posteriors are substantially more similar, and as such, the moment matching approximation is more accurate. The shape of the acquisition function is captured almost perfectly, and the magnitude is only marginally overestimated, most prominently in (green), We
display the
Funccton valoe
Danadie thartilry
winn
32
Figure 22: Example of the per-sample Hellinger distance computation using moment matching (solid lines) and large-scale, asymptotically exact quasi-MC with 2048 samples. The moment matching approximation mostly retains shape asymptotically exact variant. However, does not perfectly capture the multi modality in (blue), and overestimates the distance in the low-variance region at the right edge of (orange). The acquisition function y-axis is scaled individually per model to better highlight the difference in acquisition function value.
Ponetar
Coverred cam
0.
3.0
012
Figure Example of the per-sample Hellinger distance computation using moment matching (solid lines) and large-scale, asymptotically exact quasi-MC with 2048 samples. The moment matching approximation captures the shape of the asymptotically exact variant well, but overestimates the distance slightly in (green). The acquisition function y-axis is scaled individually per model to better highlight the difference in acquisition function value.
E.4 Wasserstein Distance Approximation
24

---

THE Oesemed dib
261
Figure 24: Example of the per-sample Wasserstein distance computation using moment matching (solid lines) and large-scale. asymptotically exact quasi-MC with 2048 samples. The moment matching approximation mostly retains the shape of the asymptotically exact variant. The shape of the acquisition function is generally well captured, but high-variance regions have their distance underestimated by the moment matching approach, and low-variance regions have their distance over-cstimated, leading a biased approximation. The acquisition function y-axis is scaled individually per model to better highlight the difference in acquisition function value.
Funccios
1.
Poderre
14.
Figure 25: Example of the per-sample Wasserstein distance computation using moment matching (solid lines) and large-scale, asymptotically exact quasi-MC with 2048 samples. The moment matching approximation captures the shape of asymptotically exact variant well, only marginally over- underestimates distance. The acquisition function y-axis is scaled individually per model better highlight the difference in acquisition function value.
F Equivalence to BALD
We show that SAL, using the KI. divergence as the metric d, is equivalent to BALD.
BALD- 12200) H(p(v))
(o
p(0)p(y) loglp(v)]dOdy + p(0) I plyl0))og[p(vl00]d
ply) 0) p(y)
p(0)
plv ooog
dyd0 p(0) (p)
SAL.KL
G Additional Background
We provide additional background on fully Bayesian hyperparameter treatment in Gaussian processes and details regarding statistical distance metries.
G.1 Fully Bayesian Treatment
The posterior probability of observing value ya for point is given as:
25

---

p(un (D) = J p 110
where f are the noiseless, latent function values as and D is the observed data. The inner integral is equal to the GP predictive posterior.
J, pluaa O)p(f D)df p(ya) D, 0).
However, the outer integral is intractable and is estimated using Markov Chain Monte Carlo (MCMC) methods. The resulting posterior prediction
p(y= (D) = [ p(yel ()
M p(y= 0,0;), 0, ~ p(ODD)
Σ
x
M
is a Gaussian Mixture Model (GMM).
Within BAL and BO, the fully Bayesian treatment is often extended to involve the acquisition function, such that the acquisition function a is computed as an expectation over the hyperparameters [46 55
a( a2 D) Ez[a(<10,
a(x10,, D) 0₃ ()
M
This is also the definition of fully Bayesian treatment considered in this work.
G.2
Statistical Distance Details
The Hellinger distance is a similarity measure between two probability distributions which has previously been employed in the context of BO-driven automated model selection by Malkomes et al. For two probability distributions p and g, it is defined
39
H² 9) 2 ] (Vplø) v g(z)
Adx,
(14)
with some auxiliary measure À with which both and are absolutely continuous. Specifically, for two normally distributed variables 21 NNH,, of), 22 N oz),
20102
(M) 122 of oz
(2) =1
1 of
exp
(15)
The Hellinger distance seeks minimize the ratio between difference in mean and the sum of variances, which punishes outlier predictive distributions of high confidence. Similar to KL, initial queries have a tendency to be axis-aligned to attain selective length scale information.
The Wasserstein distance is the average distance needed to move the probability mass of one distribution morph into the other. The Wasserstein-k distance defined as
Wx(p.q) (L. [F,(z) -[1,,())--P(()) P2(x))*
(16)
For the normal distributions 21 and 22, the Wasserstein-2 distance is defined as
W2(21, v(mi 22m² 2 + (a) o2)2.
(17)
In practice, W2 places premium on matching variance regions, leading to higher global exploration which can be detrimental for global optimization.
26

---

The KL divergence The KL divergence is standard asymmetrical measure for dissimilar- ity between probability distributions. For two probability distributions P and Q, is given by (KI(P Q) Sx P(x))oog( (())/((z))) For Gaussian variables, it is computed as
LL log of (m m2
(18)
of
The KL divergence mainly prioritizes same order-of-mannitude variances, and will initially query the same location multiple times to assess noise levels. Thereafter, it tends to query in an axis-aligned fashion, close to previous queries, to attain information regarding the length scales, but places low priority on global exploration.
27


Reviews:
[{"review_id": "H54cjDEAo4", "reviewer": "Reviewer_ta8V", "summary": "Standard acquisition functions in Bayesian Optimization (BO) only aim to finding the optimum, but do not directly consider the problem of hyperparameter learning in Gaussian Processes (GP) which has considerable impact on the optimization performance. \nThis paper introduces a new acquisition function for Active Learning (AL) and Bayesian Optimization (BO), whose goal is to learn both hyperparameters and the location of the optimum. Specifically, the acquisition function for AL is based on extending similar previous proposals of Active Learning via disagreement, by using different statistical distances. This new acquisition function is then adapted for the BO task by conditioning on sampled locations/value of the optimum. The new acquisition functions are shown to work (slightly) better at AL and BO, especially with unusual BO tasks.", "questions": "- This paper would strongly benefit from showing ScoreBO at work on other real-world tasks, to showcase that indeed this new acquisition function and its focus on active learning does not harm when deployed on real problems (e.g.,  RL, hyperparameter tuning, and other typical applications of BO which are conspicuously missing here).\n- Minor: Fix the few typos in the paper and bibliography.", "limitations": "The conclusion quickly states a few limitations but it'd be good to have a separate **Limitations** section which very explicitly mentions them.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Originality:** While natural, the problem of hyperparameter learning in BO has rarely been addressed, so this paper tackles an open and underestimated problem in the field. The proposed solutions are interesting.\n- **Significance:** Given the prominence of BO in machine learning and other fields, this work is potentially very significant.\n- **Quality:** The general quality of the work is good, although there are some open questions (see below).\n- **Clarity:** The paper is generally clear.\n\nAs general comments, Statistical distance-based Active Learning (SAL) is well-motivated and explained, a nice generalization of previous proposals. It is also nice that the paper explores the use of two different distances (Wasserstein and Hellinger), and that it gives approximations on how to compute them. The proposed \"motivated heuristic\" for extending SAL to BO via conditioning over the location-value of the optimum is interesting.\n\n### Post-rebuttal\n\nThanks to the authors for addressing the points I raised. I am glad that they managed to run additional experiments with a more realistic benchmark, showcasing the effectiveness of their method. Overall, I am satisfied with the paper and the score of 6 reflects my current evaluation of the paper (\"Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.\").", "weaknesses": "- The paper showcases many synthetic functions but there are only a couple of real-world applications. Real-world applications are particularly important for this work because they are exactly the situations at risk of breaking active learning (e.g., due to model misspecification). Indeed, the Cosmological Constant task in the paper is the one in which ScoreBO does not show any advantage over vanilla Expected Improvement. For this reason, it would have been nice to show other real examples in which ScoreBO does improve performance.\n- As a minor comment, please double-check the paper and bibliography for typos or errors. At a quick glance I spotted a few mistakes (e.g., the authors of reference [43] \"Fast information-theoretic Bayesian optimisation\" are in a wrong order; occasionally \"Bayesian\" appears lowercase, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We are pleased to see that the reviewer recognizes the significance of the joint BO/hyperparameter learning problem setting and our proposed approach.\n\n\n---\n> __1.__ _This paper would strongly benefit from showing ScoreBO at work on other real-world tasks, to showcase that indeed this new acquisition function and its focus on active learning does not harm when deployed on real problems (e.g., RL, hyperparameter tuning, and other typical applications of BO which are conspicuously missing here)._\n---\n\nThree (4D) hyperparameter tuning tasks from the PD1 benchmarking suite have been added in the rebuttal PDF, Fig. 4 — Two involving large language models, and one from computer vision. The surrogate model from HEBO [Cowen-Rivers et. al. 2020], which employs input and output warpings [Snoek et, al 2014], is used to account for the heteroskedasticity prevalent in HPO tasks. SCoreBO outperforms the other methods on 2 out of 3 tasks, placing second on the third.\n\nThe three additional real-world benchmarks hopefully depict a clearer picture of the performance of SCoreBO, together with the other two real-world benchmarks included in the original submission, namely Lasso-DNA and Cosmological Constants.\n\n\n---\n> 2. _The conclusion quickly states a few limitations but it’d be good to have a separate \nLimitations section which very explicitly mentions them._\n---\n\nThe authors would like to thank the reviewer for this suggestion. A dedicated Limitations section will be added in the CR. We intend to discuss the potential pitfalls of utilizing SCoreBO with misspecified models. This is partly addressed in Appendix C in relation to the Rosenbrock functions, where SCoreBO performs worse than on other tasks, relative to other acquisition functions. On Rosenbrock, the hyperparameter values increase over time instead of converge, which suggests that the latent function is not part of the model class. Thus, the self-correction effort of SCoreBO is less rewarding.\n\nThe authors believe that these limitations, as well as the broader subject of model misspecification in BO, necessitate further research. Encouragingly, the reviewer's perspective appears to align with this viewpoint.\n\n\nWe thank the reviewer for pointing out the typos, which will be addressed in the CR.\n\n_______\n\nHopefully, the additional real-world applications introduced in this rebuttal showcase the potential and usefulness of SCoreBO. We would be happy to address additional questions that the reviewer might have.\n"}, {"review_id": "IA1a0n9sbP", "reviewer": "Reviewer_fp2C", "summary": "The paper presents two algorithms/methods, SAL and SCoreBO, for active learning and Bayesian optimization using GPs. Both build on the premise that hyperparameters are critical to effective AL or BO; hence it is important to learn both the function and the hyperparameters through an acquisition function that accounts for both needs.\n\nContributions:\n- The SAL acquisitions function for AL considers the statistical distance between a condition (on the hyperparameters) and the marginal posterior.\n- The SCoreBO algorithm for BO, which uses SAL in combination with Thompson sampling/posterior sampling \n- Empirical evaluation on several relevant benchmark problems \n- Comparison with a set of alternative methods \n\n", "questions": "Included in the above.", "limitations": "Included in the above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-written and clear (with only a few unclear aspects)\n- Relatively simple but effective approach  (based on various other works in fully Bayesian BO including [42])\n- Thorough evaluation of a suitable set of both AL and BO benchmarks, including non-standard problems\n- Comparison with what appears to be a sensible set of baselines, showing favorable performance on the BO tasks", "weaknesses": "\nThe following are mostly just questions and comments, not all weaknesses per se:\n\n- Lack of details regarding a fully Bayesian treatment\n  -   The paper mentions that it applies a fully Bayesian treatment, yet I am (as a reader) still a bit unsure about the specific inference being used. I think more details are needed here to make it precise. In particular, discussing $p(\\theta | D)$ would be helpful to ensure the paper presents a self-contained view. \n\n- Justification of distance metrics and alternatives\n  -  I feel the paper lacks justification for the choice of distance measures. I appreciate a choice needs to be made at some point, but alternative statistical distance measures also have closed-form, e.g., (symmetric) KL, and even approximations for mixtures. \n  -  I am slightly concerned about the sensitivity to the distance function for the AL (and BO as shown in the appendix). For the AL task, have the authors performed longer experiments than presented in Figure 6 (i.e. why stop at 100/150/200 iterations)?  \n\n- What's the complexity? It would be helpful with an indication/summary of the complexity of SCoreBO relative to a standard fully Bayesian BO approach. \n\n- (The need for) theoretical bounds: What is the prospect of providing theoretical bounds for the specific algorithm?\n\nMinor:\n- As a sanity check, have the authors compared SCoreBO with a box standard BO approach without a fully Bayesian treatment of hyperparameters (i.e., ML-II)?\n- Figure 1: For clarity, I'd suggest specifying what a \"BoTorch prior\" is\n- Figure 7: There are two dashed lines; perhaps clarify the caption.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and are glad to see that the reviewer appreciates both the importance of the problem setting as well as the proposed approach. \n\n---\n> __1)__ _Lack of details regarding a fully Bayesian treatment_\n---\nWe agree with the reviewer that the paper would benefit from more clearly describing the sampling procedure when marginalizing over the hyperparameters. In the CR, we will extend the description of this in Appendix A1 as well as add a shorter description in the main text.\nBriefly, SCoreBO uses No U-Turn Sampling (NUTS) MCMC sampling (the Pyro implementation) and marginalizes over 16 models. \n\nWe will allocate a section of the background to fully Bayesian hyperparameter treatment in the CR.\n\n---\n> __2)__ _Justification of distance metrics and alternatives_\n---\n\nThe requested Jensen-Shannon (JS, symmetric KL) divergence has been added to the set of AL and BO experiments (Fig. 2, 3 in the attached rebuttal PDF) and __all AL experiments have been doubled in length__. The rank of the different methods is roughly the same as in the original manuscript — We find that __the JS variant is occasionally a powerful alternative but less consistent__ than the Hellinger distance for both AL and BO.\n\nAll distance measures have their specific strengths and weaknesses. However, the Hellinger distance has been the most empirically consistent choice overall. This may be due to the intuition below which aligns well with the global optimization goal under limited evaluation budget:\n- The Jensen-Shannon divergence prioritizes __same order-of-magnitude variances__. As such, it is inclined to repetitively query the same location to correctly estimate noise levels.\n- Wasserstein (earth-mover) distance seeks to minimize the difference in __first and second moments__.  In practice, this places a premium on matching large-variance regions, leading to higher global exploration which can be detrimental for global optimization.\n- Hellinger distance seeks to minimize the __ratio between difference in mean and the sum of variances__, which punishes outlier predictive distributions of high confidence. This turns out to be the most practical metric for posterior convergence. \n\nWe will clarify these points which characterize the different distance metrics in the CR.\n\n\n---\n> __3)__ _What’s the complexity? It would be helpful with an indication/summary of the complexity of SCoreBO relative to a standard fully Bayesian BO approach._ \n---\n\nThe conditioning on Thompson samples involves a rank-1 update of $\\mathcal{O}(n^2)$ of the GP for each Thompson sample draw. As such, the complexity of constructing the acquisition functions is $\\mathcal{O}(MNn^2)$ for $M$ models, $N$ optima per model and $n$ data points. The MCMC involved with the fully Bayesian treatment is $\\mathcal{O}(|\\theta|n^3)$ per sample. \n\nThe complexity of the forward pass is the same as (fully Bayesian) JES, namely $\\mathcal{O}(MNn^2)$. As such, these methods are roughly identical in terms of runtime. For reference, EI has a forward pass complexity of $\\mathcal{O}(Mn^2)$ and no setup. For all acquisition functions, the NUTS sampling accounts for a large majority of the total runtime.\n\nWe will extend the complexity section for the CR. \n\n---\n> __4)__ _(The need for) theoretical bounds: What is the prospect of providing theoretical bounds for the specific algorithm?_\n---\nTheoretical bounds for SCoreBO are likely tied to bounds for BO algorithms with fully Bayesian treatment, which, to the best of the authors' knowledge, there are none to date. Furthermore, bounds on SCoreBO are likely tied to bounds on to information-theoretic acquisition functions. Among these acquisition functions, PES/JES have no convergence bounds, and the bounds proposed in MES are not believed to be correct (Takeno et. al 2022, App. H). Thus, while theoretical bounds for SCoreBO are an interesting direction for future research, they appear challenging at present.\n\n---\n> __5)__ _The SCoreBO algorithm for BO, which uses SAL in combination with Thompson sampling/posterior sampling._ \n---\nTo ensure clarity: SCoreBO employs JES-like conditioning, which uses Thompson sampling (TS) as an intermediate step, to build the acquisition function. SCoreBO does not, though, use TS itself as the acquisition function. \n\n____\n\nWe hope that the additional results and answers have addressed the reviewer's concerns and improved the reviewer’s perception of our work. We would be happy to address any additional questions that may arise.\n\n#### References\nShion Takeno, Tomoyuki Tamura, Kazuki Shitara, Masayuki Karasuyama. Sequential and Parallel Constrained Max-value Entropy Search via Information Lower Bound. Proceedings of the 39th International Conference on Machine Learning, PMLR 162:20960-20986, 2022. "}, {"review_id": "4BQrr1U8VC", "reviewer": "Reviewer_LR3N", "summary": "The paper introduces a technique called statistical distance-based active learning (SAL) for the purpose of learning the hyperparameters of a Gaussian process. Additionally, SAL is integrated with information-theoretic Bayesian optimization (BO) in a framework referred to as self-correcting BO (SCoreBO). This integrated approach enables simultaneous learning of GP hyperparameters and optimization of the black box function. The experimental results demonstrate the enhanced performance of SCoreBO compared to traditional benchmarks, including unconventional BO tasks.", "questions": "Please clarify the above weaknesses.\n\n========================\n\nAfter reading author's response\n\nThe authors have made substantial efforts in providing supplementary experimental outcomes and elaborating the reasoning behind the integration of hyperparameters into the acquisition function. \n\nI believe that these experiments and explanations have the potential to provide stronger motivation for the paper and make the proposed approach's performance more convincing. As a result, I improve my rating accordingly.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper introduces new active learning algorithms that leverage the Query-by-Committee strategy and utilize two statistical distances. It also addresses the practical challenge of simultaneously learning hyperparameters while optimizing a black-box function, which holds significant importance in real-world applications.", "weaknesses": "The main weakness of the paper is inadequate literature review, resulting in major concerns on the novelty of the proposed problem and solution and the experimental results.\n\n1. Although BALD is widely recognized as a highly effective active learning strategy (e.g., employed in PES, MES, JES), it is surprising that BALD is not considered as a baseline in the active learning experiments. Conversely, the other baselines employed in the active learning experiments are relatively weak, such as utilizing only the posterior variance (BALM), the posterior mean (BQBC), or simply adding BALM and BQBC.\n\n2. Equation (7) (the proposed SAL) is essentially the Jensen-Shannon divergence, representing the mutual information between $\\theta$ and $y_x$—in other words, BALD when $d$ corresponds to the KL divergence. Surprisingly, this crucial aspect is not discussed in the paper, raising the question of why the preference is given to the Hellinger distance or Wasserstein distance instead of the KL divergence.\n\n3. The paper introduces the problem of optimizing a black box function with unknown Gaussian Process (GP) hyperparameters, asserting that this aspect has received limited attention, without providing references to existing Bayesian Optimization (BO) works that address unknown GP hyperparameters. However, it is worth noting that in the early work PES, there exists a dedicated section that tackles the issue of unknown GP hyperparameters by learning the posterior distribution of the hyperparameters and averaging PES over samples of GP hyperparameters. This crucial detail is missing from the paper, despite frequent mentions of PES throughout. Furthermore, this implies that other information-theoretic acquisition functions such as MES and JES could also be extended to handle unknown GP hyperparameters using a similar approach. Consequently, the novelty in the problem appears to be overstated and a significant concern arises regarding the absence of several essential baselines in the experiments, including PES, MES, and JES averaged over samples of GP hyperparameters.\n\n4. The motivation behind the proposed SCoreBO in Equation (8) needs further enhancement. Given that the goal of Bayesian Optimization (BO) is to optimize a black-box function, it seems appropriate to exclude $\\theta$ from the second argument in $d$ in Equation (8). This is because it is redundant to learn the hyperparameters of the Gaussian Process (GP) if they do not impact the difference between $f^*$ or $x^*$ under different GP hyperparameter samples. Essentially, the acquisition function should be a JES, with the GP hyperparameters marginalized out. To substantiate this approach, it would be valuable to present compelling experimental results comparing it with such a JES that marginalizes out the GP hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their through review. There are, however, crucial misunderstandings regarding our work: __all baselines use fully Bayesian hyperparameter treatment__ and the active learning presented in this paper __goes beyond conventional hyperparameter marginalization__. The feedback related to these issues is addressed in __1)__.\n\nWe will further clarify the interplay between fully Bayesian hyperparameter treatment and active learning and address the reviewer's remaining concerns. We hope that this gives a fresh perspective on the novelty of the work as perceived by the reviewer.\n\n---\n> __1)__ The paper __(a)__ lacks _“several essential baselines in the experiments, including PES, MES, and JES averaged over samples of GP hyperparameters”_ as well as __(b)__ lacks references to the _“dedicated section [in PES] that tackles the issue of unknown GP hyperparameters by learning the posterior distribution of the hyperparameters and averaging PES over samples of GP hyperparameters.”_\n---\n\n__a) All baselines marginalize over the hyperparameters__ in the original manuscript as per Line 224-225:\n\n#### \"ScoreBO _and all baselines_ use fully Bayesian hyperparameter treatment.\" \n\nWe will ensure that this is clearer in the CR.\n\n__b)__ The aforementioned Section 2.3 - _Hyperparameter Learning_ in PES simply describes a fully Bayesian treatment of the hyperparameters, i.e. an extension of the work by Osborne et. al. (2010) and Snoek et. al. (2012). These seminal works are referenced on line 27.\n\nHowever, the aforementioned __fully Bayesian hyperparameter treatment must be distinguished from actively learning the hyperparameters__, i.e. selecting queries to find __more accurate hyperparameters__. Fully Bayesian hyperparameter treatment is a __prerequisite__ for (Bayesian) active learning to take place. Moreover, the active learning component differentiates SCoreBO from PES/JES/MES — SCoreBO learns __more accurate__ hyperparameters (in the spirit of BALD) while optimizing the objective, whereas ES methods - _even their fully Bayesian variants_ - only do the latter.\n\n\n—\n> __2)__: __(a)__ _“[I]t is redundant to learn the hyperparameters of the Gaussian Process (GP) if they do not impact the difference between or under different GP hyperparameter samples”_ and __(b)__ _“Given that the goal of Bayesian Optimization (BO) is to optimize a black-box function, it seems appropriate to exclude theta from the second argument in d in Equation (8)”_. Lastly, __(c)__ _“it would be valuable to present compelling experimental results comparing it with such a JES that marginalizes out the GP hyperparameters.”_\n\n__a.__ Having an __accurate__ sense of the HPs is often crucial for the predictive performance of the GP model and, hence, for the efficiency of the BO algorithm.  To exemplify this, the reviewer is referred to Fig. 1 in the rebuttal PDF, where the impact of the active learning on hyperparameter convergence is visualized for the 25D Ackley task. __Fully Bayesian MES and JES never find the active dimensions__ 1-4, whereas __SCoreBO rapidly finds them__ and outclasses MES/JES/EI as a result (Fig. 7 in main paper). SCoreBO’s joint objective of active hyperparameter learning and optimization makes it __generate more accurate hyperparameters__ by reducing hyperparameter- _and_ optima-induced disagreement. Thus, it quickly obtains hyperparameters which suggests the right dimensions 1-4 as active. \n\nThis should clarify that active learning is _not_ redundant in a BO context, as the investment in active learning can produce substantially more accurate hyperparameters, which in turn can yield improved optimization efficiency.\n\n__b.__ The $\\theta$ parameter in the second argument of Eq. (8) __enables the active learning component of the joint BO/AL objective - the paper’s main contribution__. For this reason, it is paramount to keep $\\theta$ as described. Without $\\theta$, SCoreBO would be similar in spirit to fully Bayesian JES.\n\n__c.__ As established previously, the JES in the experiments employs hyperparameter marginalization.\n\n\n---\n> __3)__ BALD is not considered as a baseline in the active learning experiments.\n---\nWe initially decided to exclude BALD for its lackluster performance in Riis et. al. (2022). However, we agree with the reviewer and added it in the rebuttal PDF in Fig. 2. It performs very well, only marginally worse than SAL-HR on average. We thank the reviewer for pointing this out. \n\n---\n> __4)__: SAL is essentially the Jensen-Shannon divergence (...)\n---\nEquation (7) proposes a general instance where the specific distance metric can be instantiated. We have added the Jensen-Shannon (JS) divergence to both the AL and BO experiments in Fig. 1 and Fig. 2 of the rebuttal PDF. It performs very well on some tasks, but does not generally achieve the desired consistency.\n\nThe KL/JS divergence is indeed related, but not equivalent, to BALD. Both SAL-KL and SAL-JS minimize the _relative entropy_ between distributions. BALD, which computes the _differential entropy_ minimization, _would_ be equivalent only if the reference distribution was uniform. There is an analogous discussion in Henning and Schuler (2012, p.13).\n\n____\nWe hope that we have clarified the role of the joint optimization and HP learning objective, and the distinct differences between this and previous work. With these misunderstandings out of the way, we would appreciate if the reviewer reconsidered their score. Moreover, we would be happy to address additional questions.\n\n#### References\n1. Christoffer Riis, Francisco Antunes, Frederik Boe Hüttel, Carlos Lima Azevedo, Francisco Câmara Pereira. Bayesian Active Learning with Fully Bayesian Gaussian Processes. NeurIPS, 2022.\n2. Phillip Henning and Christian Schuler. Entropy Search for Information-Efficient Global Optimization. JMLR, 2012.\n"}, {"review_id": "42xs20dH9x", "reviewer": "Reviewer_BCnP", "summary": "The paper proposed a novel function SAL used for active learning based on distribution difference. Then the author further incorporated the SAL function into Bayesian optimization (BO) to improve BO performance. Most numerical experiments are conducted on synehtic benchmakr functions. Better performances are achieved by using the proposed method compared to several baseline functions.", "questions": "In general, I grasp the main idea of this paper. However, I suggest that the authors distinguish among three different things: (i) the true underlying function $f(x)$, (ii) the observed noisy function value $f(x)+\\epsilon$, and (iii) the GP model $GP(x)$, which is an approximation to $f(x)$. The authors should carefully examine the usage of each symbol throughout their paper and ensure that they properly distinguish between (i)-(iii). Currently, the authors sometimes misuse these symbols, which hinders the understanding of the paper. \n\nBelow, I have listed several questions and would appreciate it if the authors could provide answers. This will help me gain a better understanding and assessment of the paper.\n\n1. The authors state that the mean function $m(\\cdot)$ is set to a constant. Why is this constant not included in the set of hyperparameters?\n\n2. The numerical experiments primarily focus on benchmark functions, with the exception of DNA classification task and cosmological estimation. It would be beneficial to include more real-world experiments. For example, Bayesian optimization is commonly used for tuning network parameters in automl. It would be valuable to add experiments similar to that.\n\n3. What is the specific form of $p(\\theta|D)$ in step 4 shown in Algorithm 1?\n\n4. What is $p(f|\\theta,D)$ in Step 6? And what does Step 6 do? The symbol $f$ is used to represent the underlying function in equation (1). I am confused about how to sample an $f$ from $p(\\cdot|\\theta,D)$. Since $f(\\cdot)$ represents the underlying function and does not depend on the GP model, I assume that $p(f|\\theta,D)$ means sampling a GP instance given $\\theta$ and $D$. If this is the case, I suggest the authors use another symbol instead of $f$ here.\n\n5. One advantage of Bayesian optimization is that its acquisition function does not require calling the real function. Instead, it only needs predictions based on the GP (e.g., EI = mean of GP - std of GP). Thus, Bayesian optimization can save functional evaluation times. Typically, Bayesian optimization is employed in situations where observing the function value $y_x$ at a specific $x$ is time-consuming or expensive. However, I am confused here. Does $y_x$ represent the observed function value at $x$ according to equation (1)? Does this mean that in order to evaluate equation (9), we actually need to simulate or obtain $y_x$ at a specific $x$? In other words, does the proposed acquisition function (9) require the observation of $y_x$? If so, it seems to me that the method is not valid, as it sacrifices the fundamental benefit of Bayesian optimization.\n\n6. Continuing from the previous question, how do we evaluate the distribution value $p(y_x|\\cdots)$ in Step 7? Moreover, how do we evaluate $p(y_x|D)$ in (9)?\n\n7. What is the difference between $p(y_x|\\theta,D)$ used in equation (7) and $p(f|\\theta,D)$ used in Step 6? Are they different in terms of considering $\\epsilon$ or not (i.e., in equation (1), $y_x=f(x)+\\epsilon$)?\n\n8. I am confused about the condition on $(x^\\star,f^\\star)$. What do the authors mean by $(x^\\star,f^\\star)$? Are they the true optimals related to the underlying function $f$ or are they the optimals related to a GP model?\n\n9. The considered numerical experiments all seem rather low-dimensional.\n\n10. I am concerned about the insufficient technical contribution. It appears to me the main contribution could be summarized in one equation, Eq. (7). Previous similar papers usually also provided theoretical guarantees (e.g., the JES paper).  Solely a new function form seems rather insufficient as a standalone paper. I would appreciate it if the author could justify their contribution. \n\n**Writing suggestions**\n\n1. I assume \"HP\" in Figure 1 means \"hyper-parameter.\" However, at first sight, \"HP\" is confusing, especially since many words have abbreviations as \"HP.\" I suggest that the authors write the full name when it first appears.\n\n2. Line 72 contains a typo: \"$\\epsilon\\sim N(0,\\sigma^2_\\epsilon)$\" instead of \"$\\epsilon^2$.\"\n\n3. Keep the symbol format consistent: In line 76, $\\theta$ is not bold, while in line 80, $\\theta$ is bold.\n\n4. The Hellinger distance has the general form shown in equation (4). Why is there no general form provided for the Wasserstein distance?\n\n5. In line 221, \"robust. than\" needs to be revised for clarity.", "limitations": "The authors provided limitations in the last section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Overall, the paper is easy to follow despite several confusions.", "weaknesses": "Lack of real-world experiments.\n\nI would suggest the authors be precisely clear on their symbols. \n\nI have some questions about the technical details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough work reviewing the paper. We have added additional experiments, and hope to clarify the notation-induced confusion. \n\n\n__0)__ _Distinguish among three different things: $f(x), f(x) + ε, GP(x)$_\n\n\nUsing the notation $f(x)$ to denote both the true black-box function _and_ a random variable modeled by the GP can be confusing. __This is, however, standard practice in BO literature__ [Hernandez-Lobato et.al, 2014, Wang and Jegelka, 2017, Hvarfner et. al. 2022, Takeno et. al. 2022] and we are hesitant to break this convention. \n\nTo clarify:\n- $p(f |D)$ is the distribution over functions that the GP induces, and a sample draw from this distribution is a function $f_i \\sim p(f |D)$ (as in Alg. 1 step 6).\n- $y_x$ (reads $y$ at $x$) is a random variable representing the noisy output at $x, y_x = f (x) + \\epsilon$. Its __predictive posterior under the GP__ is $p(y_x|D)$.\n\nHopefully, this clarifies our notation to the reviewer. \n\n__1)__ \nThis is well spotted by the reviewer. We do indeed marginalize over the mean (Appendix A), but we should have declared it as a HP in Section 2.1. This is part of the HPs that SAL and SCoreBO learn. We will update the CR.\n\n__2)__ \nWe have added __three (4D) HPO tuning tasks__ from the PD1 benchmarking suite — Two involving large language models, and one from computer vision. We adopt the same setting as in HEBO [Cowen-Rivers et. al. 2020], where input and output warpings [Snoek et, al 2014] are used to account for the heteroskedasticity prevalent in real HPO tasks. We report the results in the PDF material submitted with this rebuttal and observe that __SCoreBO outperforms the other methods on 2 out of 3 tasks__, placing second on the third.\n\n__3)__\n$p(\\theta|D)$ is the posterior over the HPs given the data. We can only sample from it through MCMC since it does not have a closed-form expression.\n\n\n__4)__\n$p(f |\\theta, D)$ is the distribution over functions that the GP with HPs \\theta induces. Step 6 draws a sample function $f_i$ from $p(f |\\theta, D)$, then maximizes it to obtain a __simulated optimal location__ and value $(x^*, f^*)$. To obtain $f_i \\sim  p(f |\\theta, D)$, we use the decoupled sampling approach of Wilson et. al. (2020). When that sample function $f_i$ has been obtained, we use gradient-based optimization to find $({x_i}^*, {f_i}^*)$. The explanation in line 173 will be expanded in the CR.\n\n\n__5)__\nThroughout the paper, the observations are denoted by D = $(x_i, y_{x_i})_{i=1}^N$. The random variable $y_x$ denotes a not-yet-observed random variable (modeled by the GP) at the input location $x$.\nSAL and SCoreBO __do not need to query the black-box function to evaluate the acquisition function__ (that would indeed invalidate all that Bayesian optimization stands for). As mentioned previously, $p(y_x|D)$ is the distribution of the function values at $x$, so (9) evaluates the disagreement between the distributions of the output at $x$. \n\n__6)__\nAs $y_x$ is the random variable representing the output at $x$, $p(y_x|D)$ is the predictive posterior distribution of the GP at the input location $x$.\n\n__7)__\nThe random variable $f$ is noiseless, whereas $y_x$ is noisy. However, more importantly $p(f|D, \\theta)$ is the GP-induced distribution over _functions_, whereas $p(y_x|D, \\theta)$ is the predictive distribution of the noisy output _at one specific location_ $x$.\n\n__8)__\n$(x^*, f^*)$ are random variables denoting the optimal value and its location, using the same notation as Hvarfner et. al. (2022). These are samples as described in (4.).\n\n__9)__\nThe general trend in high-dimensional tasks is that the model is tailored to take into account the high-dimensional setting, such as in the authors’ AddGPs and SAASBO experiments, but also in methods such as REMBO, ALEBO and BaXuS. In this context, SCoreBO experiments span a comprehensive range of dimensionalities, including 8D, 11D, 25D and 180D.\n\n__10)__\nWhile the extension of B-QBC into SAL, which is the essence of Eq. (7), in itself is a significant contribution, it is only a building block in the overall body of work. The paper further __presents the first strategy for joint optimization and active learning of HPs__. AL and BO have historically been considered as two different domains, and there has historically been little effort in integrating elements of active learning into BO. SCoreBO proposes a novel method to achieve this by incorporating Entropy Search concepts into the SAL objective. The estimation schemes conditioned on the HPs and the optimizers should also be considered a technical contribution.\n\nThe need for the aforementioned joint objective is made clear from the results in Fig. 7 and Fig. 8, where active dimensions of high-dimensional problems must be learned for the optimization to be effective. In Fig. 1 of the rebuttal PDF, we show the HP convergence of SCoreBO and JES for 25D Ackley with 4 active dimensions. Clearly, JES fails to identify the active dimensions, whereas SCoreBO rapidly succeeds. The SCoreBO strategy is fundamentally different from that of JES because it __trades HP learning and optimization in a principled manner__. JES, however, only does the latter.\n\nOn the lack of theoretical guarantees, the authors would like to point out that __the JES papers do not provide any convergence guarantees__ (Hvarfner et. al., Tu et. al., 2022), unfortunately. Furthermore, the MES proof (Wang and Jegelka, 2017) is highly contested (Takeno et. al., 2022).\n\n\n____\n\nLastly, We thank the reviewer for the suggestions which will be added to the CR. The Wasserstein distance does have a closed form for Gaussians, which we will add as well as suggested.\n\nWe hope that the clarifications regarding the contributions and the requested supplementary experiments have enhanced the reviewer's perception of our work. If that is the case, we would appreciate if the reviewer increases their rating. If any ambiguities persist, we are happy to clarify further.\n\n"}]

Author rebuttal:
The authors would like to thank all reviewers for their effort. As per the reviewers' requests, we have added 4 additional plots to the rebuttal PDF.

__Fig. 1:__ Hyperparameter convergence of fully Bayesian JES, MES and SCoreBO on the SAASBO task, the noisy 25-D Ackley function. We see that the fully Bayesian information-theoretic acquisition functions fail to find any of the active dimensions of the task, while SCoreBO finds all of them with low uncertainty. Thus, SCoreBO successfully optimizes the task, whereas JES and MES do not. 

__Fig. 2:__ Prolonged active learning results with double the iteration budget of the initial submission. We include SAL using Jensen-Shannon (JS) divergence as the distance metric, and BALD as an additional baseline. BALD is a highly competitive baseline. SAL-JS performs well on many tasks, but is inconsistent.

__Fig. 3:__ BO synthetic experiments with SCoreBO using Jensen-Shannon divergence as well as non-fully Bayesian EI (MAP). SCoreBO-JS performs well on many tasks, but is inconsistent. EI-MAP performs well on some tasks, but lags behind on some tasks, most prominently on Hartmann (6D).

__Fig. 4:__ Three (4D) hyperparameter tuning tasks from the PD1 benchmarking suite - Two involving large language models, and one from computer vision. SCoreBO outperforms the other methods on 2 out of 3 tasks, placing second on the third.

_________

The authors look forward to a productive discussion with the reviewers!

Meta-review: This paper proposes new acquisition functions for Bayesian active learning and Bayesian optimization based on disagreement over posterior hyperparameter samples.  Comprehensive experimental results are shown, where the proposed methods perform favorably against baselines.

Most reviewers acknowledged significant contributions but originally raised major concerns:

- Inadequate literature review
- Novelty overstated
- Lack of baseline experiments
- Unclear motivation of ScoreBO.  Advantage over JES?
- Not many real-world data experiments

Through the extensive discussion between the reviewers and the authors, the major concerns have been (not fully but) mostly addressed.  The authors are expected to revise the paper as they promised, which will significantly improve the paper.
Decision: Accept (poster)