Bayesian Active Learning with Fully Bayesian Gaussian Processes
Authors: Christoffer Riis
Francisco Antunes
Frederik Boe Hüttel
Carlos Lima Azevedo
Francisco C. Pereira
Venue: ICLR 2022 Submitted (2022)

Abstract: The bias-variance trade-off is a well-known problem in machine learning that only gets more pronounced the less available data there is. When data is scarce, such as in metamodeling, active learning, and Bayesian optimization, neglecting this trade-off can cause inefficient and non-optimal querying, leading to unnecessary data labeling. In this paper, we focus on metamodeling with active learning and the canonical Gaussian Process (GP). We recognize that, for the GP, the bias-variance trade-off regulation is made by optimization of the two hyperparameters: the length scale and noise-term. Considering that the optimal mode of the joint posterior of the hyperparameters is equivalent to the optimal bias-variance trade-off, we approximate this joint posterior and utilize it to design two new acquisition functions. The first one is a mode-seeking Bayesian variant of Query-by-Committee (B-QBC), and the second is simultaneously mode-seeking and minimizing the predictive variance through a Query by Mixture Gaussian Processes (QB-MGP) formulation. Across seven simulators, we empirically show that B-QBC outperforms the benchmark functions, whereas QB-MGP is the most robust acquisition function and achieves the best accuracy with the fewest iterations. We generally show that incorporating the bias-variance trade-off in the acquisition functions mitigates unnecessary and expensive data labeling.

Under review as a conference paper at ICLR 2022
BAYESIAN ACTIVE LEARNING WITH FULLY BAYESIAN GAUSSIAN PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
The bias variance trade-off is a well-known problem in machine learning that only gets more pronounced the less available data there is. When data is scarce, such as in metamodeling, active learning, and Bayesian optimization, neglecting this trade-off can cause inefficient and non-optimal querying, leading to unnecessary data labeling. In this paper, we focus metamodeling with active learning and the canonical Gaussian Process (GP). We recognize that, for the GP, the bias- variance trade-off regulation is made by optimization of the two hyperparameters: the length scale and noise-term. Considering that the optimal mode of the joint posterior of the hyperparameters is equivalent the optimal off, we approximate this joint posterior and utilize it to design two new acqui- sition functions. The first one mode-seeking Bayesian variant of Query-by- Committee (B-QBC), and the second is simultaneously mode-secking and min- imizing the predictive variance through a Query by Mixture Gaussian Processes (QB-MGP) formulation. Across seven simulators, we empirically show that B- QBC outperforms the benchmark functions, whereas QB-MGP is the most robust acquisition function and achieves the best accuracy with the fewest iterations. We generally show that incorporating the birs-variance trade-off in the acquisition functions mitigates unnecessary and expensive data labeling.
trade-
INTRODUCTION
Gaussian Processes (GPs) are the canonical models to use for Bayesian optimization and metamod- eling (Snoek et al. 2012 Gramacy| 2020). GPs are well-known for their ability to deal with small to medium size data sets as well as balancing complexity and regularization together with their inherent ability to handling uncertainties this makes them ideal in such applications (Williams & Rasmussen 2006). In both cases, often only limited data is accessible so the natural balance between complexity and regularization helps prevent severe overfitting, additionally making the model flex- ible enough to model nonlinear functions. Likewise, the quantification of uncertainty is commonly used in the acquisition function to guide the Bayesian optimization algorithms and the active learn- ing schemes that are almost inevitable efficiently build metamodel.
On the other hand, it is not a flawless procedure to use GPs to guide the two schemes. The same GP used both predictor and guide, and thus the choice of predictor will affect the guide, and vice versa. Trivially, the more data there is available for modeling, the less pronounced this problem will be. However, in the context of both Bayesian optimization and active learning, where the data sets tend to be rather small, wrong predictor can result in misguidance, thus hindering the performance and efficiency. In this paper, we mitigate this problem by not only focusing on single predictive model but consider many predictive models through multiple model hypotheses at once.
GP typically fitted through evaluation of the marginal likelihood, which automatically incorpo- rates a trade-off between complexity and regularization (Williams & Rasmussen] (2066). However, when the data is scarce, is more challenging to choose the appropriate trade-off, and different configurations of the hyperparameters of the GP can give rise to distinct fits. This is illustrated in Figure where two seemingly reasonable fits might guide the schemes quite differently. The prob- lem directly related to the well-known bias-variance variance trade-off, although we reformulate it balance between modeling the data as signal or noise. For the common stationary covariance func- tions of GPs, e.g. the radial-basis function or the Matérn class of functions, together with a Gaussian

---

Under review as a conference paper at ICLR 2022
High noise regime
Low noise regime
Length scale
mW
1.0
0.5
X
0.0
2
3
Noise
Figure Two GPs with different hyperpara- meters. The left plot shows GP with high noise and long length scale, and the right plot shows GP with low noise and short length scale.
Figure 2: Joint posterior the two hyperpara- meters length scale and noise. The posterior is multimodal with a low noise, high signal and a high noise, low signal mode.
likelihood, this is directly reflected in the two hyperparameters length scale and noise o². the data is modeled primarily noise, both of and & are large, whereas if the data is modeled primarily as a signal, both of and are small. In the case of limited data, the joint posterior distribution of the two hyperparameters is likely be characterized by two modes, illustrated in Figure 2 for the data in Figure I] In that case, it is difficult to tell whether it best to model the data as noise or signal, and a wrong choice of mode will imply non-optimal guidance from the acquisition function.
The literature suggests handling this problem with clever initializations of the hyperparameters Williams & Rasmussen (1996) or by favoring small and of by either always initializing hyper- parameters in the low noise regime or by applying strong priors (Gramacy) 2020). However, none of these approaches directly address the core problem: which mode choose? Ideally, this should be answered with prior information about the problem, although typically that is not available, making these approaches less practical.
In this paper, we follow general approach and assume no prior knowledge about the kernel or hyperparameters. We consider multiple model hypotheses by replacing the fitting procedure of the marginal likelihood with Markov Chain Monte Carlo (MCMC) sampling. Lalchand & Rasmussen (2020) show that with fixed, medium-sizee data sets and carefully chosen kernels, it is beneficial to fit GPs with MCMC instead of by maximizing the marginal likelihood. We show that the same true for very small data sets and with a general kernel. Our main contribution is the proposal of new acquisition functions for active learning that utilizes the extra information from the hyper- paramters' posteriors estimated by MCMC to seek the most reasonable mode alongside minimizing the predictive variance. We show that the two acquisition functions than other common functions across multiple benchmark simulators used in the related literature.
accurate and robust
RELATED WORK
The two proposed acquisition functions are specifically designed for active learning schemes for regression tasks. In this section, we lay out the related work, although the specific acquisition func- tions used for comparison purposes in the experiments are described in detail in secciion 4] Further, in this section, we cover the essentials of Query-by- we--byyy---m-eee and Gaussian Mixture Models, as these constitute the backbone of the proposed acquisition functions.
Active Learning The main idea of active learning (AL) is actively choose a new data point to label and add to the current training data set (Settles) 2009). In the context of metamodeling. new data often added sequentially, i.e., one data point at time (Gramacy 2020). but other applications, can be beneficial to query batches of data instead (Kirsch et al. 2019).
The acquisition functions can divided into model-based and model-free functions, where the former utilize information from the model and the latter do not (0 (eile et al. 2017). Both types of functions seek to minimize the expected predictive loss of the model. Another approach to minimize the number of possible models. Houlsby et al. (2011) divide the active learning acquisition functions into being based on either decision or information theory. Decision-basee functions seek to minimize the expected predictive loss of the model in the hope of maximizing the performance
2

---

Under review as a conference paper at ICLR 2022
on the test set. -otpmmatioon-hhorrti-----ee- functions instead try to reduce the number of possible models, for example, through the KL-divergence or Shannon entropy (Houlsby et al.) 2011).
It is not straightforward to use information-theoree access to the posterior of the model's parameters, [Houlsby et al. (201I) have derived the algorithm Bayesian Active Learning by Disagreement (BALD), which can be applied in general. Generally, BALD seeks the data point that maximizes the decrease in the expected posterior entropy of the parameters.
acquisition functions. However, if you have
(QBC) is a specific acquisition function that was originally proposed for classification tasks (Seung et al. T992). It aims to maximize the disagree- ment among the committee to get the highest information gain and minimize the version space, which is the set of model hypotheses aligned with the training data. The construction of the com- mittee is the core component of QBC since it is the committee's 's ability to accurately and diversely represent the version space that gives rise to informative disagreement criteria (Settles) 2009).
The
QBC can also be applied for regression problems. Krogh & Vedelsby (1955) construct the members of the committee by random initializations of the weights in the neural networks. [RayChaudhuri & Hamey (1995) apply bagging and train the members on different subsets of the data set. In general. QBC constructed by bagging has been used as a benchmark with mixed results (Cai et al.) 2013] Wu 2018 Wu et al. (2019). Burbidge et al. (2077) show that the less noise there is in the output, the better QBC is compared to random querying. They also highlight the fact that with a misspecified model, QBC might perform worse than random querying. None of these approaches explore the usage of MCMC samples of the posterior to construct a committee. To the best of our knowledge, are the first propose QBC based on model hypotheses drawn from the posterior.
Gaussian Process as a Gaussian Mixture Model Mixture models have recently been applied in active learning for classification tasks. [swanto T02TT) propose use Gaussian Mixture Models (GMMs) with active learning, where they design a specific acquisition function that queries the data point that maximizes the expected likelihood of the model. Zhao et al. (2020) use a mixture of GPs in active learning, where each component is fitted to a subset of the training set.
The combination of GMMs and GPs have previously been explored for static data sets. Chen & Ren (2009) investigate regression tasks and apply bagging, where they repeatedly randomly sample data points from the training set to construct new subsets to get GPs fitted different data. Among other combination rules, they combine the predictive posteriors of these GPs into GMMs. They obtain even better performance by weighting the models by the predictive uncertainty such that models with high uncertainty given smaller weights, and vice versa. Instead of using bagging, we construct the multiple GPs by using the MCMC samples of the hyperparameters' joint posterior, and then obtain natural weighting of the GPs: the GMM will consist of more GPs with hyperparameters close to the modes than hyperparameters far away. The formal procedure is given in /ecctinn 4] To the best of our knowledge, are the first combine GMMs and GPs in this manner.
3 GAUSSIAN PROCESSES
The Gaussian Processes (GPs) are the central models in this work. In this section, we give a brief overview of GPs before covering the Fully Bayesian GPs. For a thorough description of GPs, we refer to Williams & Rasmussen 2006)..
Gaussian Processes A Gaussian Process (GP) is stochastic function fully defined by mean function m²-) and a covariance function (often called a kernel) k(,,-). Given the data (X,y) v) = {24, 34 h=1 where y is the corrupted observations of some latent function values f with Gaussian noise e, i.e., yi fi + Ei, & E (0 of), a GP is typically denoted as gP(m, ky (2t, x²)). It is common practice to set the mean function equal to the zero-value vector and thus, the GP is fully determined by the kernel x'),, For short, we will denote the kernel Ke. which explicitly states that the kernel is parameterized with some hyperparameters 0. Given the optimal hyperparameters, the predictive posterior for unknown test inputs x² is given by 1(
xx) = ~N(,,, ("",E") with
n¹ K2 (Ko + o -1 y and Σ =K K2 (Ko + T K&T
(1)
where K3" denotes the covariance matrix between the test inputs, and Kğ denotes the covariance matrix between the test inputs and training inputs.
3

---

Under review as a conference paper at ICLR 2022
Covariance matrix We use the canonical kernel automatic relevance determination (ARD) Radial-basis function (RBF) given by & (2,z²) exp (-1lz x'112/22² where & is a vector of length scales E,, La one for each input dimension. Often the kernel is scaled by output vari- ance but here we fix it to one and solely focus on the two other hyperparameters: length scale and noise-term. The noise-term of is integrated into the kernel with an indicator variable by adding the term (-==))) to the current kernel.
Fully Bayesian Gaussian Processes (FBGP) A FBGP extends a GP by putting a prior over the hyperparameters ~ p(0) and approximate their full posteriors. The joint posterior is then given by
p(f. X) x x))
and the predictive posterior for the test inputs X is
p(f"ly)= - 1f
(3)
where the conditioning on X and X have been omitted for brevity. The inner integral reduces to the predictive posterior given by normal GP. However, the outer integral remains intractable and is approximated with MCMC inference as
M Ex(MMu:))) O2 ~ p(0v))
p (( = p (MMM).0)((000)
(4)
M
As well known in machine learning modeling, there is no free lunch. Adapting the hyperparameters of FBGP computationally expensive compared the approach with GPs and maximum likeli- hood estimates. However in Bayesian optimization and active learning, the computational burden for querying a new data point will often be of magnitudes higher.
Evaluation and Fitting of GPs A GP is typically fitted by maximizing the marginal likelihood with gradient descent, where the marginal likelihood can be computed analytically. For the FBGP, the inner integral is intractable and is therefore optimized with approximate inference using the self- tuning method of Monte Carlo Markov Chain (MCMC) called the Ko-U-Tum-SSapler (NUTS) (Hoffman & Gelman 2014). The MCMC inference makes the full posterior of the hyperparameters available, making it possible to extend the point estimates to a joint distribution. In the experiment, apply acquisition functions that use the full posterior distribution; however, since the experiments are designed to benchmark the acquisition functions, we always use the mode of the joint posterior for prediction and evaluation.
4 ACTIVE LEARNING
In this section, we lay out the most common acquisition functions and then propose first a Bayesian variant of
and secondly an extension motivated by Gaussian Mixture Models, which seek to minimize both the predictive variance and the number of model hypotheses. All the acquisition functions choose data point among the possible data points in the unlabeled pool U.
4.1 CLASSIC ACTIVE LEARNING
Many classic active learning acquisition functions are based on the model's uncertainty and entropy (Settles) 2009) Gramacy 2020). We use the most common acquisition function based on the predic- tive entropy and denoted Active Learning MacKay (ALM) (MacKay 1992).
Entropy (ALM) For a Gaussian distribution. the Shannon entropy H|] is proportional to the predictive variance o² (2x),, i.e. H[x] In In(((mo² so a new data point Fnew is queried
Inew arg max o (I:)
(S)
Intuitively, ALM queries the data point, where the uncertainty of the prediction is the highest.
4

---

Under review as a conference paper at ICLR 2022
4.2 BAYESIAN ACTIVE LEARNING
If we have access to the posterior of the model's parameters, we can utilize acquisition functions with extra Bayesian level, giving rise to Bayesian active learning. The posterior of the hyper- parameters of an FBGP can be estimated with MCMC such that GPs with different pairs of the kernel parameters, length scale, and noise can be drawn, i... L, o2 p(oly, X). The following four acquisition functions all utilize this information and are approximated using the samples from the MCMC, where we compute 1( y, X, X*) using equation
Entropy (B-ALM) In the Bayesian setting, we can use the information from the full posteriors instead of only using the point estimates. The Bayesian variant of ALM (B-ALM) is then given as
Knew arg max (0)0(()))))0(())10))
Bayesian Active learning by Disagreement (BALD) Houlsby et al. (2011) propose to find the data point that maximizes the decreases in the expected posterior entropy of the hyperparameters. They rewrite the objective from computing entropies in the parameter space to the output space by observing that is equivalent to maximizing the conditional mutual information between the parameters and unknown output 0 y jr D), where (ar, y) the data point and ' is the training data. The acquisition function denoted Bayesian Active Learning by Disagreement (BALD) and is given by:
Enew
arg max T10, y|x, Dj = arg max Hlyla, D] ()> [y 0]]
BALD was originally derived for non-parametric discriminative models but has recently been ex- tended to batches and deep learning with great success (Kirsch et al. 2019). Despite the success, show that BALD is not the optimal way to query new data points when considering GPs for regression tasks.
Bayesian Query-by byyCommittteeeeeee (B-QBC) Motivated by reducing the number of possible models and with the multimodal posterior of the parameters in mind, we propose a new acquisition function that seeks the correct mode. More precisely, we propose use the joint posterior of the parame- ters obtained through MCMC to draw multiple models and then query a new data point where the mean predictions (x) of these models disagree the most. Since the models are drawn from the parameters' posterior, the collection of models is dominated by models near the posterior modes. Querying the data point that maximizes the disagreement between the models' predictions corre- sponds to querying the data point providing the highest information about the true mode. Thus, this can be seen as mode-seeking Bayesian Query-by-Comen Committee (B-QBC). Given that μ(xr) the average mean function, B- B-QBC is given as
Fnew arg max p D) ((o(()) - ((
Query by Mixture of Gaussian Processes (QB-MGP) We extent the mode-seeking B-QBC to consider the predictive variance as well. We denote the new acquisition function Query by Mixture of Gaussian Processes (QB-MGP) since it can be motivated by the fact that each prediction of the FBGP can be written as a Gaussian Mixture Model (GMM). More formally, we propose to use the MCMC samples to make each prediction into GMM, yielding the predictive posterior given
M
( = pf
Σ (f M 1=1
O, ~ p(llv)
(9)
This hierarchical predictive posterior is a mixture of M Gaussians with mean PGMM and covariance matrix KAMM As defined as (Lalchand Rasmussen 2020):
1 M
me, and KGMM 1
PGMM
Ke,
(pe, 2
(10)
M
M
Finding the data point that maximizes the variance of the GMM is now equivalent to simultancously considering the B-ALM and B- B-QBC:
Knew = arg max Bo + Eooppt01)[((o00)> (1)))))
(11)
a
5

---

Under review as a conference paper at ICLR 2022
Table I: Stochastic simulators used in the experiments.
Simulator
d
Noise dz
Input space
Previously used in
Sine: (6x)
0.2
[0.5]
Gramacyld ld
1
0.1
2.55
Gramacy Lee
2012
Higdon ld
0.1
[0, 20]
Gramacy Lee
2009
Gramacy 2020
Motorcycle
Her"
[0, 60]
Silverman 1985
Gramacy Lec 2008
Gramacy
2020
Gramacy2d
0.05
[_2.
Gramacy Loc
2012 Sauer ct al.
2020
Branin
11.32
[-5, [-5,10] 10 [0, 15]
Keane el. (2008
Picheny el al 2013
Cole ct 2021
Ishigami
0.187
(- T.
Marrel et al. (2009
Cole et 2021
Hartmann
0.0192
[0, 1]
Picheny et al. 2013
Cole ct al. 2021
Heteroscedastic noise
MM
Sine
Gramacyld
Higdon
Motorcycle
Gramacy2d
Branin
www
Figure 3: Visualization of the simulators (excl. Ishigami and Hartmann due to the dimensionality).
EXPERIMENTS
In this section, start by introducing simple simulator denoted Sine (see Figure 3). which to show that with few data points and no prior knowledge, it can be catastrophic to use a GP fitted by MAP since it does not find the optimal mode. We show that the FBGP partly solves the problem by utilizing the information from the full posterior. Additionally, we show that both B-QCB and QB-MGP outperform the other acquisition functions. Both results are shown in Figure
We benchmark the performance of the FBGP and the two proposed acquisition functions with the standard fitting procedures of maximum posteriori (MAP) and the standard acquisition functions based on the entropy, i.c., ALM. B-ALM, and BALD, on various classic simulators used in recent literature on GPs and active learning. They are all listed in Table and those with less than three inputs are shown in Figure 3D The motorcycle simulator is created by fitting a variational GP (Hens- man et al.) 2015) the motorcycle accident data set (Silverman) 1985). The evaluation metrics are the negative log marginal likelihood (NLML) and root mean square error (RMSE). The performance for each the active learning iteration is shown, but for a comparable overview, we focus on a ref- erence iteration that follows convergence criterion: when the best acquisition function (based on NLML) has reached convergence, we label that iteration as reference (for that specific simulator). no convergence is reached, we iteration number 100 as reference. For these evaluations, the in- troduced toy simulator is excluded, and we only consider simulators used the literature previously.
In the experiments, we use a zero-mean GP with an ARD RBF kernel. In each iteration of the active learning loop, the inputs are rescaled to the unit cube [0. 1jd. and the outputs standardized to have zero mean and unit variance. Following Lalchand & Rasmussen 22020), we give all the hyperparameters relatively uninformative (0, 3) priors in log space. The initial data sets consist of three data points chosen by maximin Latin Hypercube Sampling, and in each iteration, one data point is queried. The unlabeled pool U consists of the input space discretized into 100 equidistant points along each dimension. If U contains more than 10, 000 data points, we randomly sample subset of 10, 000 data points in each iteration and use that as the new pool. For the MAP estimation, estimated by gradient descent using Adam (Kingma & Ba, 2015))
the optimal hyperparameters with a learning rate 0.1. The inference in FBGP is carried out using NUTS (Hoffman & Gelman 2014) in Pyro (Bingham et al. 2018) with five chains and 500 samples, including warm-up period with 200 samples. The remaining 1500 samples are all used for the Bayesian acquisition functions. All experiments are repeated ten times with different initial data sets. The models implemented in GoycTorch (Gardner et al. 2018), and all code can be found at www avai l Tables www.l/lle.. e/later com
'll of them except number and can he found htthhttppp.. wwwp///www... 8 http://wwwwfff....cs///aaa//

---

Under review as a conference paper at ICLR 2022
Sine
Gramacy ld
1.00
1.00
NUML
RM4GE
0.73
MLML
0.0
0.75
-0.5
0.50
0.25
-1.0
0
-1.5
0.00
0
10
20
30
40
0
10
20
30
40
75
0
23
50
75
HT iigdon
Motorcycle
0.5
1.0
1.00
NUML
RMGE
0.4
NELML
0.75
2
0.500
0.0
0.2
0
40
60
0
40
60
20
30
0
30
G Gramacy2d
Branin
0.12
1.5
70
1.0
NUML
RMGE
0.10
NLMLL
RMSSE
0.5
2
0.08
0.0
-0.5
0
0
30
0
30
Ishiqami
Hartmann
1.3
NUML
RANSE
NLML
RMSSE
1.3
0
0.9
75
75
75
GP (MAP): ALM
FBGP: ALM
FBGP: B-ALM
FBGP: BALD
FBGP: B B-QBC
FBGP: QB-MGP
Figure 4: Performance across the 10 runs 11 standard deviation. The a-i represents the number of iterations, and the vertical dotted lines show the reference iterations (also listed in Table 2).
5.1 EXPERIMENTS WITH 8 SIMULATORS
We benchmark the acquisition functions the Sine simulator and seven classic simulators. We divide the simulators into subgroups and describe how the functions works in different noise settings, complexities, and with multiple inputs, effectively encompassing five distinct modeling scenarios.
Simple simulator The simulator Sine is constructed to show the advantage FBGPs compared to GPs. It also shows the benefit of applying B-QBC and QB-MGP compared to the other acquisition functions. Considering the NLML, B-QBC is slightly better than the other functions, although when look the RMSE, QB-MGP is the overall best choice. This shows the gain of reducing the predictive variance alongside the number of model hypotheses.
Noise or signal This simulator Gramacyld has previously been used study the effect of the noise-term in GPs (Gramacy & Lee. 2012). The simulator has a periodic signal that is hard to reveal if the data points are not queried cleverly. Both B-QBC and QB-MGP reach convergence simultaneously. while the other acquisition functions struggle in distinguishing noise from the signal.
Linear and non-linear output regions The two simulators Higdon and Gramacy2d have been used to illustrate cases where GPs struggle in modeling the data due to the output signal having both linear and non-linear regions (Gramacy Leel 2009). Our experiments on these simulators show the performance of the acquisition functions when the GP is an inadequate choice of model. Querying data points in the linear and non-linear regions will yield GP with a longer and shorter length scale, respectively. On Higdon, it is seen that both B-QBC and QB-MGP balance the sampling since the corresponding NLML and RMSE are low. On Gramacy2d, QB-MGP achieving the highest RMSE due to favoring to capture the small region with non-linear signal compared to the vast linear region. Overall, these results show that when the GP is inadequate to model the data, both B-QBC and QB-MGP perform better.
7

---

Under review as a conference paper at ICLR 2022
Table 2: The likelihood-ratii and the relative change in RMSE (%) compared the best performing function averaged over 10 runs.
Gramacyld ld
Higdon
Motorcycle
Gramacy2d 2d
Branin
Ishigami
Hartmann
Dimensions
Overall performance
Ieration
30
20
100
100
Meun
Median
Min
Likeliheod-ratios
GP (MAP): ALM
-1.79
00.29
-0.44
-3.14
-0.11
-2.96
-0.64
-1.34
-0.64
-3.14
FBGP: ALM
-1.82
-0.57
-0.04
-1.62
-0.28
-2888
-1.36
-1.22
-1.36
-2.88
TBGP: B.ALM
-1.80
-0.60
-0.07
-0.69
-0.45
-2,20
-2.01
-1.12
-0.69
-2.20
FBGP: BALD
-2.04
-0.7
-0.88
-144
-1.66
-1.46
-2.04
FIGGP: B- B-QBC [Ours]
-0.04
0
-182
-1228
-0.45
0
-0.51
-0.4
-1.82
QD-MGP (Ours)
-0.25
Relative change in RMSB (9)
Mean
Median
Max
GP (MAP)
366
8
61
65.7
366
FBGP: ALM
359
35
25
66.9
359
TBGP: B-ALM
352
32
14
19
22
63.3
19
352
FRGP: RALD)
387
299
63
108.7
12
387
TISGP: B. B-QBCC (Ours)
266
48.4
12"
286
FBGP: QB-MGP (Ours?
3
Heteroscedastic noise The experiments the Motorcycle simulator explore how the active learn- ing acquisition functions perform when the simulator has heteroscedastic noise, but we model it with homoscedastic GP. Most conspicuous is the poor performance of B-QBC, indicating that this is misled by the heteroscedastic noise. However, the combination B-ALM and B-QBC in QB-MGP gives the best performance. This agrees with our expectation that QB-MGP seems be more robust than B-QBC to different types of noise.
Multiple inputs To evaluate the performance higher dimensions, we consider the smooth 2d Branin simulator, the strongly non-linear 3d Ishigami simulator, and the 6d Hartmann simulator with six local minima. BALD underperforms on Branin, but the other acquisition functions have similar performance. Looking at all the active learning iterations, the GP fitted with MAP seems to give the best trade-off between the NLML and RMSE, indicating that the regular GP with ALM is the best for smooth problems. For Ishigami and Hartmann, 100 iterations are not enough for convergence, although B-QBC achieves the lowest NLML, whereas BALD reaches the lowest RMSE.
5.2 THE OVERALL PERFORMANCE
We now provide performance overview evaluating the acquisition functions the iteration in which the best one has converged in terms of NLML. We both evaluate the NLML and RMSE, but clearly see the difference performance, we compute the likelihood-ratios for the NLML and the relative change in RMSE to the best performing acquisition function. In that way, we can summarize the performance across the different simulators. The likelihood-ratios and relative changes RMSE are listed in Table 2 and the actual values are found in Appendix, Table 3
Considering the likelihood-ratios, B-QBC and QB-MGP are the two best acquisition functions for six out of the seven simulators. The median performance of B-QBC is slightly better than QB- MGP, but both the mean and the minimum value of QB-MGP better, indicating that QB-MGP more robust across the simulators. Regarding the relative changes in RMSE, BALD and QB-MGP are the best performing functions. Overall, QB-MGP achieves both the lowest mean, median, and maximum value, clearly stating that QB-MGP is the most accurate and robust acquisition function across the simulators. For a visual representation, the box plots of the results are shown in InFFiguee 5) B-QBC and QB-MGP are the two best acquisition functions based on the NLML. Regarding the RMSE, the B-QCB is lacking behind QB-MGP, and we see that QB-MGP benefits for taking the predictive uncertainty into account.
5.3 SCOPE & LIMITATIONS
In applications with active learning, it is not only the performance at convergence that is of in- terest since we often have limited computational budget. However, in general, we observe that the relative performances between the acquisition functions do not vary significantly doing the
8

---

Under review as a conference paper at ICLR 2022
Negative marginal log likelihood
Root mean square emrrr
FBGP:
FBGP: B-QRC
FBGP: BALD
FBGP: B-ALN
FBGP: ALM
GP (MAP): ALM
-3
Likelhood-rotoe
Relative change (S)
200
400
Figure 5: Visualization of likelihood-ratios and relative change in RMSE (%).
tive learning iterations, and thus we believe that the evaluations at the reference points provide an accurate evaluation.
The Bayesian acquisition functions utilize the full estimated joint posterior to choose a new data point. Similarly, we could also use the posterior for prediction instead of only using the mode. Then the predictive posterior is given as for the mixtures of Gaussian in equation 10 However, since we focus on benchmarking the acquisition functions and not the models, we believe that using the mode is more accurate when comparing it to the regular GP since we only change the fitting procedure.
This paper is based on empirical results that are dependent on the specific simulators. would be infeasible to apply these acquisition functions to all previously used simulators, but we hope that our choice diverse and distinct classic simulators representative of the engineering problems occurring in the real world. Nonetheless, we encourage the reader to test other simulators by using our code at www. www.waaailallle.......a wwweeelaerr.cc wat.aaabeeeaeee.oommm
CONCLUSION
In this paper. we propose two active learning acquisition functions: Bayesian Enery-by-Comminne (B- BBCC)) and Query by a Mixture of Gaussian Processes (QB-MGP), both of which are suited for fully Bayesian GPs. They are designed to explicitly handle the well-known by optimization of the GP's two hyperparameters, length scale and noise-term. We empirically show that they query new data points more efficiently than previously used acquisition functions. Across seven classic simulators, which cover different types of noise, complexity and number of inputs, we show that QB OBMMGPi is the most robust function and achieves the best accuracy with fewest iterations.
trade-off
The incorporation of domain knowledge and expert guidance regarding the simulator under study be decisive factor in a successful metamodeling strategy. However, in many practical situ- ations, such a priori domain expertise may not be readily accessible or even translatable into the functional structure of the metamodel useful modeling information. On these occasions, generic tools that are robust enough to handle a plethora of diverse simulation output behaviors are pru- dently advisable. On the one hand, we aim maintaining an economical approach the problem of exploring the simulation input space. On the other hand, the final fitting performance of the metamodel must be equally taken into utmost account. Simply discarding one of these two aspects might well end up rendering the active learning- based metamodeling strategy not only computation- ally unsustainable but also effectively counterproductive. To this end, we believe that the proposed acquisition functions properly address both concerns, while being entirely independent of any prior understanding of the underlying output distributions of the simulator.
In terms of future work, we plan to develop new acquisition functions, which constitute, in the short-term, variations of those explored herein. Then, we aim to apply the proposed approach in the context of larger and realistic simulation models addressing real-world problems. Eventually, our ultimate goal is to provide practitioners with an auxiliary tool to map the simulators' output behaviors in more efficient manner.
ACKNOWLEDGMENTS
This work supported by NOSTROMO, framed in the scope of the SESAR 2020 Exploratory Research topic ----2--R4-2-22- funded by SESAR Joint Undertakingthrough the European Union's Horizon 2020 research and innovation programme under grant agreement No 892517.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
In this work, we strove to make the results as reproducible as possible. To this end, besides using open and free source software (Python packages) implement all the mentioned models and al- gorithms, the simulators/functions used to generate the studied data are of the public domain and referenced accordingly. Alongside thorough and clear paragraph on the experimental settings, we also attach compressed file containing the source code used to reproduce the presented experi- ments. In the end, we commit to provide non-anonymized dynamic link the repository with the most up-to-date version of the code soon.
REFERENCES
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep Univer- sal Probabilistic Programming. Journal of Machine Learning Reseurch, 2018.
Robert Burbidge, Jem J. Rowland, and Ross D. King. Active learning for regression based on query by committee. Lecture Notes Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 4881 LNCS:209 2007. ISSN 16113349. doi: 3-0-778-------77222223 22.
Wenbin Cai, Ya Zhang, and Jun Zhou. Maximizing expected model change for active learning in regression. Proceedings IEEE International Conference on Data Mining, ICDM, (December 2013):51-60, 2013. ISSN 15504786. doi: 10. 109//CDDM.21331044
Tao Chen and Jianghong Ren. Bagging for Gaussian process regression. Neurocomputing. 72(7-9): 1605-1610, 2009. ISSN 09252312. doi: 10. http::/..eccom.200009.90000
D. Austin Cole, Robert B. Gramacy, James Warner, Geoffrey F Bomarito, Patrick E. Leser, and William P Leser. Entropy-based adaptive design for contour finding and estimating reliability. arXiv preprint 1eXXi:2105.1133: 2021.
Jacob R. Gardner, Geoff Pleiss, David Bindel, Kilian Q. Weinberger, and Andrew Gordon Wilson Gpytorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. Advances in Neural Information Processing Systems, 201 BDDDcmmberreeeurP777777----6 2018. ISSN 10495258.
Robert B. Gramacy. Surrogates. doi: 10. 101120197803378115499
Chapman and Hall/CRC, mar 2020. ISBN 9780367815493.
URL https: htpp:://www.c
9781000766202
Robert B. Gramacy and Herbert K.H. Lee. Bayesian treed gaussian process models with an appli- cation to computer modeling- Journal of the American Statistical Association, 103, 2008 ISSN 01621459. doi: 10.
Robert Gramacy and Herbert K.H. Lee. Adaptive design and analysis of supercomputer experi- ments. Technometrics. 5102):130-145, 2009. ISSN 00401706. doi: 10. 10.1198/TECH..200000155
URL htt.:tss//aii.oo//11...o http://...111///11 1ww.11.1/........0000 2009. 0015
Robert B. Gramacy and Herbert K.H. Lee. Cases for the nugget in modeling computer experiments.
Statistics and Computing, 22, 2012. ISSN 09603174. doi:
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable variational gaussian pro- cess classification. In Artificial Intelligence and Statistics, pp. 351-360. PMLR, 2015.
Matthew D Hoffman and Andrew Gelman. The no-u-tumn sampler: adaptively setting path lengths in hamiltonian monte carlo. Mach. Learn. Res., 15(1):1593-1623 2014.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning. pp. 1-17, 2011. RRLbttp: / /arxivvv.vv.. org/ abs/ 1112. 5745
10

---

Under review as a conference paper at ICLR 2022
B. H. Iswanto. Active learning by increasing model likelihood for Gaussian mixture mod- els based classifiers. 10P Conference Series: Materials Science and Engineering, 1098(3): 032036, mar 2021. ISSN 1757-8981. doi: i op. art
URL https:
Andy Keane, Alexander Forrester, and Andras Sobester. Engineering Design via Surrogate Mod- elling: A Practical Guide. 2008. doi: 10.2514/4.4799 4/4.447577
Diederik Kingma and Jimmy Lei Ba. Adam: method for stochastic optimization. 3rd Interna- tional Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, pp- 1-15, 2015.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. BatchBALD: Efficient and diverse batch ac- quisition for deep Bayesian active learning. Advances in Neural Information Processing Systems, 32(NeurIPS), 2019. ISSN 10495258.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances neural information processing systems 7, pp. 231-238, 1995.
Vidhi Lalchand and Carl Edward Rasmussen. Approximate inference for fully bayesian gaussian process regression. In Symposium on Advances in Approximate Bayesian Inference, pp. 1-12. PMLR, 2020.
David J. C. MacKay. Information-Basee Objective Functions for Active Data Selection. Neural Computation, 4(4):590-66) 1992. ISSN 0899-7667. doi: 10.1162/meco..1922.4442
Amandine Marrel, Bertrand looss, Béatrice Laurent, and Olivier Roustant. Calculations of sobol indices for the gaussian process metamodel. Reliability Engineering and System Safety, 94, 2009. ISSN 09510933 09518320. doi: 10. r0.10066j.erss.2008.. 07.008.
Jack O'Neill, Sarah Jane Delany, and Brian MacNamee. Model-Free and Model-Based Ac- tive Learning for Regression. In Advances in computational intelligence systems, pp. 375-
386. Springer, 2017. ISBN 9783319465623. doi: 10.1007/978-33319946662 24. URL
httt:://link. www.spriieeee com 10000....099770 11188-33319-11 46562 1
Victor Picheny, Tobias Wagner, and David Ginsbourger. A benchmark of kriging- infill criteria for noisy optimization. Structural and Multidisciplinary Optimization, 48, 2013. ISSN 16151488. doi: 00.1077/s0015-001--9909
Tirthankar RayChaudhuri and Leonard G.C. Hamey. Minimization of data collection by active learning. IEEE International Conference on Neural Networks Conference Proceedings, 3: 3:1338- 1341, 1995. doi: 10.1 10.1110/.. 1995.mm79999999483
Annie Sauer, Robert B. Gramacy, and David Higdon. Active learning for deep gaussian process surrogates. arXiv preprint arXiv:2012.0800 2020.
Burr Settles. Active learning literature survey. 2009.
H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory- COLT '92, pp. 287-2944 New York, New York, USA, 1992. ACM Press. ISBN 089791497X. doi: 1335111333.11..177.... ARLhttp: 130417
B. W. Silverman. Some aspects of the spline smoothing approach to non-parametric regression curve fitting. Journal the Royal Statistical Sociery: Series (Methodological), 47, 1985. ISSN 0035-9246. doi: 10.111 2511111/2111116666 198.1.11...911111313331
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. volume 4, 2012.
Christopher K. Williams and Carl Edward Rasmussen, Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for regression. 1996.

---

Under review as a conference paper at ICLR 2022
Dongrui Wu. Pool-Based Sequential Active Learning for Regression. TRANSACTIONS ON NEU- RAL NETWORKS AND LEARNING SYSTEMS, 30(5):1348-13 1359, 2018. ISSN 23318422.
Dongrui Wu, Chin Teng Lin, and Jian Huang. Active learning for regression using greedy sampling.
Information Sciences, 474:90-105, 2019. ISSN 00200255. doi: 10.1016/j.ins. 201..001009 URL
httpss//ddo www.gg/......... 00166 ins 2018 09. 0600
Jing Zhao, Shiliang Sun, Huijuan Wang. and Zehui Cao. Promoting active learning with mixtures of Gaussian processes. Knowledge-Baud -arige Systems, 188:105044, 2020. ISSN 09507051. doi: 10.101 10.10166j.knosyy.201111 knnosys.2019.100044 URL fttps://doi.ooo 101.00.111.000000 5 105044
2019,
A APPENDIX
Table 3:
Benchmark of acquisition functions test sets. The table shows the mean and standard
deviation over runs.
GP (MAP)
FBGP (MCMC)
Simulator
ALM
ALM
B-ALM
B-QBC
QB-MGP
BALD
Sine
30
NLML
1.34 0.39
028
0.91
0,21 0.09
0.26 0.14
0.40
RMSE (P)
0.69 + 0.17
0.15 + 0.03
0.15 + 0.02
0,23 0.08
0.16 + 0.02
0.34++0.22
Gramacyld
80
NLME
-0225
1:003
-0.23:0005
5024
-C12
0.08
20.99:0..4
RMSE (p)
0.01
0.00
0.32
0.08
0.02
0.07 - 0.01
0.37
HigdonId
50
NLML
0.11
4002 0.09
1001 0.133
-0.31 + 0.05
0.19 + 0.17
0.13
RMSE (P)
0.09::0.03
0.10
0.10
0.88
0.01
0.08 0.03
0.30
Motöržyele
20
NCME
0.32
0..1
0.65
1.00
1.75
0.63
0.08
RMSE (p)
0.26 +: 0.06
0,28 + 0.07
0.28 + 0.07
0,90 0.26
0.24 + 0.05
0.38 + 0.22
Gramacy24
30
NLML
0.77
1.5 0.61
1.03
0.69 0.33
1.79 1
RMSE (9)
0.09
0.00
0.10
0.10
0.09 0.01
Braninza
30
NCML
20.33
-0.24
"0,4430055
10.990.111
2.22
0.36
RMSE (P)
42.16 2.14
42.29 1.37
+
42.61
1.32
41.48
1.26
46.58 6.94
Isligamiid
100
NLME
0.98 1026
1022
0.60
0027
022
0.36
0.32
0.50
RMSE (p)
7.24 0.98
0.80
0.50
0.19
5.34
4.50 0.34
Hartmann6d
100
NLML
1.80
1.41
-1122:0.9
2.12
1.36
-1.99 0.28
RMSE (2)
1.01
1181001
117
121
1.01
Best
1/6
7/16
1/16
1/16
3/6
3/6
12


Reviews:
[{"review_id": "o1RCOkKscQ", "reviewer": "Reviewer_uC7q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- the contributions of this paper are interesting and appear to be original\n- the approaches are based on sound probabilistic foundations\n- the experiments on synthetic data provide intuitive insights into the relative merits of different approaches.\n\n**Weaknesses**\n\n- connections and comparison to prior work is incomplete\n- experimental results are not very conclusive; no experiments on real data\n- parts of the paper are unnecessarily convoluted and unclear\n\nThe problem of overfitting to the marginal likelihood problem is well-known in the GP community. In the context of Bayesian optimization for example, some authors have proposed a similar method that takes advantage of a mixture of GPs [1]. How does this paper's approach differ?\n\nThe experimental results on synthetic data are quite comprehensive and insightful. One relevant baseline that appears to be missing is Zhao et al (2020). How does their approach compare empirically?\n\nIt is difficult to reconcile the results on the negative log-marginal likelihood (NLML) with the RMSE. The squared error is identical to the negative log-likelihood (under a Gaussian model) and should thus closely match the NLML results - how do you explain the difference in relative ranking and the fact that many of the RMSE curves slope upwards? How is the RMSE in fact evaluated—is there a hold-out set?\n\nAdditional experiments on real-world datasets & finding one realistic application where the proposed approach is clearly superior would significantly strengthen the paper.\n\nFinally, I encourage the authors to improve the clarity of the paper: introducing all concepts progressively, and making statements precise. Examples:\n\n- canonical Gaussian Process: to the best of my knowledge, this is not a widely used terminology for the RBF kernel\n- metamodeling is never formally introduced\n- \"given the data (x, y), ..., a GP is typically denoted as GP(...)\" -> the data does not appear to be relevant to the way the GP is denoted.\n- the definition of the RBF kernel is incorrect (there is a scalar quantity, the squared norm, that is divided by a vector)\n- I don't think the reference to the \"no free lunch\" theorem is appropriate in the context of the discussion on page 4\n- \"the inner integral is intractable and is therefore optimized with approximate inference\": what exactly is optimized?\n- section 5: how is convergence declared?\n\n[1]: An Empirical Bayes Approach to Optimizing Machine Learning Algorithms, J. McInerney, NIPS 2017", "summary_of_the_paper": "This paper introduces two new active-learning strategies for Gaussian process regression, based on a fully-Bayesian treatment of GPs. Instead of using fixed hyperparamers obtained by maximizing the marginal log-likelihood of the GP model, the authors suggest learning a posterior belief over GP hyperparameters, and make use of the full distribution to design more effective active-learning methods.\n\nIn practice, this results in two concrete proposals:\n\n1) a strategy that is similar to query-by-committee, where the committee is formed of multiple GPs sampled from the hyper-posterior (B-QBC).\n2) a variant that combines a notion of disagreement with the predictive uncertainty (QB-MGP)\n\nThese methods are then compared to others on 8 synthetic datasets.", "summary_of_the_review": "I think there is potential in this paper: the problem is important and the method is sound. Improving the clarity of the paper, comparing against McInerney (2017) and Zhao et al. (2020) and applying the method on real-world data would make this paper significantly stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YkxrVVHzBVF", "reviewer": "Reviewer_hDWW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "####################################\n\nStrengths:\n- The themes discussed in this paper, including the full posterior having modes corresponding to either signal or noise which should be discriminated between with active learning, are very interesting. In particular, Figures 1 and 2 were interesting and illustrative.\n- The related work section and description of Gaussian Processes is mostly satisfactory\n- The experiments compare against a diverse set of active learning acquisition functions and experimental simulators\n\n####################################\n\nWeaknesses:\n\nWhile this paper presents some interesting ideas, overall I feel that certain major elements need to be significantly improved before it is ready for publication.\n\nWhen I first read the abstract, I was intrigued by the idea of the bias-variance tradeoff being neglected in active learning, how in GP this tradeoff corresponds to the length scale and noise term, how “the optimal mode of the joint posterior of the hyperparameters is equivalent to the optimal bias-variance tradeoff”, and how B-QBC and QB-MGP directly incorporate this tradeoff in their acquisition functions to mitigate bias-variance issues. However, in my opinion, these points are not strongly reflected in the paper itself:\n\n*** Introduction ***\n\nStarting in the introduction, I had expected a cohesive theme illustrating the issues presented in the abstract: how the bias-variance tradeoff is neglected in active learning, a strong argument for bias-variance correspondence to length scale and noise term, and strong connections with the hyperparameters posterior. Instead, I found the topics in the introduction only very loosely coupled: specifically, I think the connections between the predictor vs guide, bias vs variance, and signal vs noise in GPs need to be strengthened and more direct arguments made. For instance, statements such as “The problem is directly related to the well-known bias-variance trade-off, although we reformulate it as a balance between modeling the data as signal or noise” or “However, none of\nthese approaches directly address the core problem: which mode to choose? Ideally, this should be\nanswered with prior information about the problem, although typically that is not available, making\nthese approaches less practical.” seem extremely important, but most of the time I found such critical statements vague and unsupported. While Figure 1-2 were interesting, I did not find myself convinced that the bimodal behavior in Figure 2 is a general phenomenon and found myself unsure how it connected to bias and variance. Is this multimodal behavior between signal and noise an established fact in the literature, or is this perspective a major contribution of this paper? Either way, I believe statements around this point need to be strengthened. Overall, after reading the introduction I found myself unconvinced about the strong points presented in the abstract. I also expected that I would find more details supporting these points in Section 3, which seem to be missing there (e.g., establishing connections between bias-variance and specific GP hyperparameters).\n\n*** Sections 2-3 ***\n\nOtherwise, I was mostly happy with Sections 2-3. One point in Section 3 I’d like to bring up is that the authors claim earlier in the paper that “In this paper, we follow a general approach and assume no prior knowledge about the kernel or hyperparameters.” However, the authors state in Section 3 that they use an RBF kernel. Can the authors comment about this? Is this specific choice not in fact an assumption about the kernel?\n\n*** Section 4 ***\n\nIn Section 4, B-QBC seems reasonable, but I was not convinced that it somehow addressed the key bias-variance tradeoff and/or mode selection problem in GP active learning described earlier in the paper. The idea of discriminating between modes seems reasonable, but I believe that in order to motivate B-QBC, a much stronger argument (stronger than the illustrative example in Figures 1-2) needs to be presented about why the hyperparameter posterior is multimodal, and how B-QBC will specifically select the correct one. In terms of QB-MGP, I found myself confused by its presentation. I was not convinced that equation (11) is maximizing the predictive variance of the mixture model --- I can imagine that it comes directly from a variance decomposition, but I believe the derivation of this fact should be presented. Moreover, as with B-QBC, why should maximizing the predictive variance of this mixture model solve the mode-selection problem and tradeoff between bias and variance? Furthermore, how do the authors expect B-QBC and QB-MGP to compare qualitatively? Since QB-MGP is equivalent to B-QBC plus a variance term, how do the authors expect this variance term to impact the performance of the acquisition function? Overall, B-QBC and QB-MGP seem to me to be the two fundamental contributions of the paper, and I believe more discussion is needed here to derive and motivate their use and strongly connect them to resolving bias-variance/signal-noise/mode selection tradeoffs.\n\n*** Section 5 ***\n\nWhile I appreciate the multitude of simulators tested in the experiments, I am not convinced by the experimental results. The performance curves seem to have high variance, and I am not convinced by the summary statistics and takeaways presented. It is not at all clear to me from Figure 4-5 and Tables 2-3 that “B-QBC outperforms the benchmark functions, whereas QB-MGP is the most robust acquisition function and achieves the best accuracy with the fewest iterations.” \n\nSpecifically, analyzing performance at a single iteration number based on a converged reference curve only captures a small part of the overall picture. I believe it would be better to examine summary statistics that better capture the overall performance of each method, such as evaluating multiple iteration numbers or looking at summary statistics such as AUC. The authors make a comment in section 5.3 attempting to justify this single iteration point evaluation, but I did not find it convincing. I am also unsure if the likelihood ratio and error quotients in Table 2 provides an adequate summary of the data. When using the raw results in Table 3, any sort of benefits for B-QBC and QB-MGP in Figure 5 become much less clear to me when looking at raw data instead of quotients. Can the authors comment about this choice of presenting summary statistics in terms of relative performance rather than raw values? Regardless, given the not insignificant error bars, I would be more convinced by these results if they were accompanied by statistical testing against either the best performing method or random sampling (see other active learning meta-analyses such as “A benchmark and comparison of active learning for logistic regression”, Yang and Loog 2018).\n\nIn Figure 4, it seems difficult to take away anything very useful from these experiments, due to the overlapping error bars in each trial and the fact that in some experiments and methods (including QB-MGP and B-QBC) the error is in fact increasing. It seems that the performance differences can be made more clear by plotting error bars for summary statistics (e.g., standard error or confidence intervals) and increasing the number of trials beyond only 10. I do not look at Figure 4 and have confidence that I should use QB-MGP and B-QBC over the other methods. In my opinion, the takeaways stated in the text are overstated based on these curves, such as QB-MGP being the “best choice” from the Sine RMSE plot, when it seems to be that multiple methods tie QB-MGP in performance, or that B-QBC could even be considered better than the other methods, given the overlapping error bars. Furthermore, there are analytical takeaways that to me seem vague and unsupported, such as “This agrees with our expectation that QB-MGP seems to be more robust than B-QBC to different types of noise.” Why is this an expectation? Statements such as “QB-MGP is achieving the highest RMSE due to favoring to capture the small region with a non-linear signal compared to the vast linear region” or “Looking at all the active learning iterations, the GP fitted with MAP seems to give the best trade-off between the NLML and RMSE, indicating that the regular GP with ALM is the best for smooth problems” are in my opinion speculative takeaways that seem unsupported. Could QB-MGP favoring the non-linear portions be shown empirically, for instance?\n\nAlthough it’s possible that I missed these details in my reading (although I went back again to check), I have some reproducibility concerns. It is unclear how “convergence” is calculated for selecting each plot reference as well as the examined iteration number. How exactly are NLML and RMSE calculated, do they utilize the full posterior, and which data are they calculated with respect to --- the sampled data points, or a held-out set of randomly sampled data? Are the distributions in Figure 5 at the simulator level, or is all data pooled together from all simulators? In Table 2, if different trials' references converged at different times, how was a single iteration value selected? Low-level details such as these are important and seem to be missing (or should at least be included in an appendix), which prevents a full understanding of the experimental results.\n\nFurthermore, I believe additional experiments should be run to support the main claims made in the paper. A major claim is that the full posterior is multi-modal, motivating the proposed active learning experiments. I think an experiment that shows this empirically through some metric would strengthen this major claim. I also think that additional experiments are needed to characterize the performance of the proposed schemes beyond performance alone. Specifically, my reading of the paper is that the proposed methods are designed to select the correct full posterior mode as quickly as possible. It would be useful to show during trials of B-QBC and/or QB-MGP how a single (correct) mode of the full posterior emerges during MCMC sampling, and the others are suppressed (and possibly comparing this metric against baseline methods).\n\nIn general, I did not find myself convinced by Section 5.3 and found it lacking. It discusses two important details --- why a single iteration slice is sufficient for performance analysis, and where specifically in the acquisition and model evaluation pipeline the full posterior was used rather than the hyperparameter MAP (and why this choice was made for every step) --- but I found the first discussion unconvincing and the second discussion a bit  confusing. Specifically, I found this important sentence confusing “However, since we focus on benchmarking the acquisition functions and not the models, we believe that using the mode is more accurate when comparing it to the regular GP since we only change the fitting procedure.”\n\n######################################\n\nOther points *not* factoring into decision, but serving as feedback for the authors:\n\nIt has been pointed out before (Settles, B. (2012). Active learning: Synthesis lectures on artificial intelligence and machine learning. Long Island, NY: Morgan &  Clay Pool. --- Section 3.5) that information gain maximization can be interpreted as a type of query by committee, where KL divergence on the label is used as a notion of disagreement. Specifically, if the BALD acquisition function is equivalently written as $\\mathbb{E}_{\\theta} [\\mathrm{KL}(p(Y \\mid \\theta) \\Vert p(Y))]$, then this is comparable to B-QBC, which uses a mean discrepancy instead of a KL-divergence. I think this is an interesting connection that would strengthen the work in terms of why B-QBC might perform better than BALD. What is mean discrepancy doing well that KL divergence isn't?\n\nTo me, the use of \"GMM\" seems unusual here to describe a mixture of Gaussian Processes. I checked the literature cited by the authors, and I did not see “Gaussian Mixture Model” or GMM being used as the authors do here to describe a mixture of Gaussian Processes (as opposed to the classical GMM notion of a mixture of multivariate Gaussians in a vector space). Personally, I think using the phrase \"GMM\" runs the risk of confusing the model with a classical GMM framework (as opposed to a Gaussian Process). However, this may be a matter of personal opinion.\n\nThe first paragraph of Section 5 seems out of place to me. It discusses important points, but I think Section 5 should start with a more general description of the experiments instead of just the Sine simulator. Also, I think the experiments could be strengthened by comparison against random (passive) sampling.\n\nMinor points:\n- in the abstract, Query by Mixture Gaussian Processes -> Query by Mixture *of* Gaussian Processes\n- the sentence “The literature suggests handling this problem with clever initializations...either always initializing…” seems redundant to me\n- the location of Gramacy2d and Motorcycle should probably be switched in Figure 4 since the text seems to follow a row-first description of Figure 4\n- I think the right subplot of Figure 5 should show a zoomed-in region to better visualize the methods besides BALD\n- Personally, I don’t think the future work paragraph adds much to the paper, and I don’t understand what this sentence proposes: “Eventually, our ultimate goal is to provide practitioners with an auxiliary tool to map the simulators’ output behaviors in a more efficient manner.”\n", "summary_of_the_paper": "This paper takes a fully Bayesian approach to Gaussian Process (GP) active learning by using MCMC sampling to consider multiple model hypotheses from a full posterior, from which it selects examples for GP regression using two new active learning strategies --- Bayesian Query-by-Committee (B-QBC) and Query by Mixture of Gaussian Processes (QB-MGP). B-QBC queries the data point that maximizes the disagreement between the sampled models’ mean values. The idea is that this should provide the highest information about the optimal posterior mode. QB-MGP uses a combination of B-QBC with entropy sampling. The authors evaluate these strategies against several baseline active learning methods on several simulators.", "summary_of_the_review": "In my opinion, this paper is ultimately making strides at an interesting and important topic in GP active learning --- what role does the full posterior have in managing the bias-variance tradeoff, and how can these insights be used to design better active learning acquisition functions? I think the proposition of QB-MGP and B-QBC is an important stride in this direction. However, I believe the paper is not ready for publication. In terms of presentation, I found the arguments made about bias-variance in its relation to the full posterior and the motivation for QB-MGP and B-QBC unconvincing. I also found some statements made in the paper to be confusing or unsupported. In terms of experiments, I found myself unconvinced by the presented analysis (specifically due to the high variance in the results and the lack of more general summary statistics such as AUC rather than a single iteration slice) and am concerned with some aspects of reproducibility. Therefore, I would recommend **rejection** for this paper and encourage the authors to revise the writing to strengthen the presented arguments, better motivate their active learning strategies, improve reproducibility, and conduct a stronger analysis and visualization of the experimental results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RKRlsfREzii", "reviewer": "Reviewer_ssFN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper seems to be interesting. \nThe part that I most enjoy to read is Section 2 of \"related works\" and Section 4.1, which are also the clearer parts of the paper. \n\n- The rest of the paper, including abstract and introduction, is quite confusing in my opinion. The concepts of \"bias-variance trade-off\" and \"active learning\" are often referred but it is very difficult to understand the connection.\n\n- Some sentences like:\n\n\"In this paper, we follow a general approach and assume no prior knowledge about the kernel or\nhyperparameters. We consider multiple model hypotheses by replacing the fitting procedure of the\nmarginal likelihood with Markov Chain Monte Carlo (MCMC) sampling. Lalchand & Rasmussen\n(2020) show that with fixed, medium-sized data sets and carefully chosen kernels, it is beneficial\nto fit GPs with MCMC instead of by maximizing the marginal likelihood.\" \n\nare not very precise. I assume that you compare different kernels approximating the marginal likelihood ? \nOr maybe you refers considering a mixture of models with different kernels. In any case, we are assuming some kind of kernel functions and you have to learn their parameters. Please, improve your explanations in all the work.\n\n- Regarding the state of the art discussion in Sections 2 and 4.1 about active learning. Novel more robust (with less numerical problems) and more advanced (considering the gradient of the solutions) acquisition functions has been considered in the literature\n\nD. H. Svendsen et al, Active Emulation of Computer Codes with Gaussian Processes - Application to Remote Sensing, Pattern Recognition Volume 100, 2020, \n\nF. Llorente et al., \"Adaptive quadrature schemes for Bayesian inference via active learning\", IEEE Access, Volume 8, 2020,\n\nM. Kanagawa and P. Hennig, “Convergence Guarantees for Adaptive Bayesian Quadrature Methods,” in Advances in Neural Information Processing Systems, 2019, pp. 6234–6245.\n\nThey have been successfully applied in regression and to build adaptive quadrature schemes (all of them in the context of active learning). Please, include these kind of acquisition functions in your discussion.\n\n", "summary_of_the_paper": "It seems that the authors introduces two novel acquisitions functions for Gaussian Process models:\nthe first one is a mode-seeking Bayesian variant of Query-by- Committee (B-QBC), and the second is simultaneously mode-seeking and minimizing the predictive variance through a Query by Mixture Gaussian Processes (QB-MGP) formulation.  The authors also present several simulations.", "summary_of_the_review": "The paper contains interesting material, but specially abstract and introduction are very confused.\nThe state-of-the-art discussion must be also improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mjb6yavGDB", "reviewer": "Reviewer_H5XZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This work is interesting and well-motivated. Bayesian Gaussian process-based active learning has practical benefits as demonstrated in numerical tests.\n\nWeaknesses:\n1. It is well known that active learning tends to explore the function space. The proposed B-QBC seems to exploit more of the available information. Is there any intuition why it works well in practice?\n2. It is not clear what the performance metrics (i.e., NLML and NMSE) are in the experiments. How is the experimental validation different from Bayesian optimization?\n3. The experiments are based on synthetic functions. It would be better if practical problems are considered.", "summary_of_the_paper": "This paper considers Bayesian treatment of the hyperparameters in Gaussian process based active learning, based on whichtwo novel acquisition functions are proposed.", "summary_of_the_review": "This is an interesting work that considered Bayesian Gaussian processes for active learning. But the merits of the paper can be better enhanced by addressing the aforementioned comments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studied Bayesian active regression with Gaussian processes, and proposed two intuitive algorithms inspired by the classical disagreement-based and uncertainty sampling criteria. The reviewers appreciate the motivation and overall idea of taking a fully Bayesian approach by utilizing the joint posterior of the hyperparameters for active learning. However, there are shared concerns among the reviewers in the clarify and consistency of several key technical components, including discussion around bias-variance tradeoff and its connection to the fully Bayesian approach, as well as in the experimental details, which make the current package insufficient for publication. 

Reviewers provide very useful feedback (in particular with a very extensive review by Reviewer hDWW) for improving the current work. The authors acknowledge in their responses that these are valid concerns and they would address these issues in a further version of this work.
Decision: Reject