Efficient Active Learning for Gaussian Process Classification by Error Reduction
Authors: Guang Zhao
Edward Dougherty
Byung-Jun Yoon
Francis Alexander
Xiaoning Qian
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian Active Learning
Gaussian Process Classification
Expected Error Reduction
Query Synthesis

Abstract: Active learning sequentially selects the best instance for labeling by optimizing an acquisition function to enhance data/label efficiency. The selection can be either from a discrete instance set (pool-based scenario) or a continuous instance space (query synthesis scenario). In this work, we study both active learning scenarios for Gaussian Process Classification (GPC). The existing active learning strategies that  maximize the Estimated Error Reduction (EER) aim at reducing the classification error after training with the new acquired instance in a one-step-look-ahead manner. The computation of EER-based acquisition functions is typically prohibitive as it requires retraining the GPC with every new query. Moreover, as the EER is not smooth, it can not be combined with gradient-based optimization techniques to efficiently explore the continuous instance space for query synthesis. To overcome these critical limitations, we develop computationally efficient algorithms for EER-based active learning with GPC. We derive the joint predictive distribution of label pairs as a one-dimensional integral, as a result of which the computation of the  acquisition function avoids retraining the GPC for each query, remarkably reducing the computational overhead. We also derive the gradient chain rule to efficiently calculate the gradient of the acquisition function, which leads to the first query synthesis active learning algorithm implementing EER-based strategies. Our experiments clearly demonstrate the computational efficiency of the proposed algorithms. We also benchmark our algorithms on both synthetic and real-world datasets, which show superior performance in terms of sampling efficiency compared to the existing state-of-the-art algorithms. 

Efficient Active Learning for Gaussian Process Classification by Error Reduction
Guang Zhao', Edward R. Dougherty', Byung-Jun Yoon Francis Alexander & Xiaoning Qian 1,2.3 guangzhao@t edu, falexander@bnl, (edward, bjyoon, xqian tamu.edu edu
"Department of Electrical & Computer Engineering, Department of Computer Science & Engineering,
*Computational Science Initiative, Brookhaven National Laborator Upton, NY 11973, USA
Texas A&M University
College Station, TX 77843, USA
Abstract
Active learning sequentially selects the best instance for labeling by optimizing an acquisition function to enhance data/label efficiency. The selection can be either from a discrete instance set (pool-based scenario) or continuous instance this work, we study both active learning scenarios for Gaussian Process Classification (GPC). The existing active learning strategics that maximize the Estimated Error Reduction (EER) aim at reducing the classification error after training with the new acquired instance in step-look-abes -hccd manner. The computation of EER- based acquisition functions query.
space (query synthesis scenario).
is typically prohibitive Moreover, as the EER not smooth, can not be combined with gradient based optimization techniques to efficiently explore the continuous instance space for query synthesís. To overcome these critical limitations, we develop computationally efficient algorithms for -EERd active learning with GPC. We derive the joint predictive distribution of label pairs as a one-dimensional integral, as a result of which the computation of the acquisition function avoids retraining the GPC for each query. remarkably reducing the computational overhead. We also derive the gradient chain rule efficiently calculate the gradient of the acquisition function, which leads to the first query synthesis active learning algorithm implementing EER-based strategies. Our experiments clearly demonstrate the computational efficiency of the proposed algorithms. We also benchmark our algorithms on both synthetic and real-world datasets, which show superior performance in terms of sampling efficiency compared to the existing state-of-the-an algorithms.
it requires retraining the GPC with every
Introduction
Compared to traditional passive learning with randomly sampled training instances, active learning aims "optimally" querying instances for labeling to achieve label efficiency when training machine learning models, especially when labeling is difficult or costly. There are two fundamental scenarios of active learning discussed in the literature: query synthesis and pool-based sampling (14). In query synthesis, the leaner can request labels for any instance generated from continuous feature space while pool-based sampling selects the instance from finite set. Query synthesis more challenging due to the infinite search space and inherent higher label uncertainty. Recent research on query synthesis with deep generative models has shown promising potential [18 [I]. However, in
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

n- 18
C2
32
-44
Figure 1: Phase diagram identification by active learning with Gaussian Process Classification (GPC).
many science and engineering applications, acquiring the label for even one instance is prohibitively resource-demanding and therefore active learning with deep models may not be practical.
For example, one of critical materials science research questions is to identify phase transition diagrams, where the phase transition response surface can be complex [8] Identifying phase transitions can be formulated as finding the optimal classification boundaries between different phases in the enormous materials design space. However, precisely knowing the phase of each design with the corresponding input features requires costly and time-consuming materials synthesis and profiling experiments or running complex simulation models. Furthermore, there may exist significant uncertainty in acquired phase labels due to technical limitations. Hence, active learning for optimal Bayesian classifiers is natural solution to help identify the phase diagram with few as possible synthesized or simulated materials under such uncertainty and complexity. Gaussian Process Classification (GPC) as a popular and powerful Bayesian classifier with the flexibility of adopting different kernels 15). is suitable for solving this problem with appropriate active learning strategies. Fig D shows example of phase identification in two-dimensional design space by active learning with GPC (Details in the Appendix). In the figure, the black solid line indicates the phase transition response surface, the crosses and dots indicate the queries of different phases, and the colorbar indicates the predictive distribution of GPC. From the figure we can see, the predictive distribution identifies the phase transition boundary with few samples guided by active learning using GPC surrogate models, which has significant cost and time savings compared to the traditional trial-and-errol phase diagram identification paradigm in the current materials science practice.
Many active learning strategies have been proposed for GPC. For example. Bayesian Active Learning by Disagreement (BALD) selects the instance with the maximum mutual information between the observation and the derived uncertain model [5]. There also have been strategies targeting at reducing classification error directly or indirectly. Estimated Error Reduction (EER) strategies optimize for the eтror reduction after training with the new queries [13 6). We note here that based active learning has been studied for both Gaussian Process Regression (GPR) and GPC [13 6
For GPR, the regression eтror can be represented by the posterior predictive variance with the analytical expression and the acquisition function is easy to calculate for efficient active learning. contrast, for GPC, there is no analytical expression for the posterior predictive. The model updates need approximate inference, such Expectation Propagation (EP), an algorithm iteratively approximating the GPC posterior 77. Moreover, the classification етrоr computation requires the new query labels; so the caleulation of the corresponding EER- -baed acquisition function requires incrementally retraining the model for each possible query label to calculate the expected future error as the utility to guide
active learning. The complexity of training GPC with EP approximation is ( where n is the number of observed data. This has been the main hurdle to develop active learning for GPC models.
To reduce the calculation cost EER, the paper (6] proposed to use a non-iterative but less accurate method Assumed Density Filtering (ADF) approximating the new posterior when calculating the acquisition functions. The paper proposed novel approximated error reduction (AER) criterion, in which the error reduction of a candidate is estimated based on the impact over its nearby instances. The approximated estimation avoids re-inferring the labels of massive instances. These methods are relatively efficient in computation. But besides EP approximation, they need additional approxima- tions in calculating acquisition functions so the acquisition functions are not precisely calculated, which may degrade the desired data efficiency.
2

---

In this paper, within the BER-based active learning framework for GPC, we develop computationally efficient algorithms to compute EER to guide both query synthesis and pool-based active learning. In particular, we consider EER as the reduction of the Mean Objective Cost of Uncertainty (MOCU) [16] since the learning objective of GPC, in particular for identifying phase diagram, is to reduce the classification error. By deriving the joint distribution of queries as one -iedimensio integral, we avoid retraining the GPC for each query when calculating the EER/MOCU- We further leverage smooth approximation of MOCU, Soft MOCU (SMOCU) [17), to enable efficient gradient computation of the SMOCU reduction by deriving the corresponding chain rule for efficient query synthesis with GPC. To the best of our knowledge, this is the first algorithm for query synthesis active learning based on EER strategies. We show in our experiments that our algorithms accelerate the computation of the acquisition functions. Compared with other existing algorithms, we demonstrate consistent data efficiency of our algorithms with both synthetic and real-world datasets.
acquisition function.
2
Problem Setting and Background
2.1 Gaussian Process Classification (GPC)
Consider binary classification problem with the instance space X' and binary label set y - {0, 1). we aim to train a classifier &b X + y to predict labels for unobserved instances th [{. x, E X. The Gaussian Process classification (GPC) framework connects the instance and the label by a latent function f. which is a random process depending on æ. Assume that f follows a Gaussian Process (GP) prior f GP(m(-), -)), where u(()) is a mean function and :(: is covariance kernel function 99. GP is popular and powerful model for both regression and classification. A good property of GP is that given any finite number of instances 2;, the joint distribution of fax; is still Gaussian. For classification problems, given f. the label y takes a Bernoulli distribution with probability p(y = 1 1 f) = (f(()), where is the Gaussian cumulative distribution function.
Given a sequence of observations D = {x, Y) with X = {0r, 22,..... .n and Y {y1,
}, the class labels are conditionally independent given the latent function. Therefore, the joint likelihood can be factorized as: p( p(Y | f) = II p(gn (z,, f). Since the likelihood proba- bility is non- Con-aausiian, the posterior π [f]x, Y) x (((())p(())) [f,, X) can not be computed analytically and approximation are often adopted. The Expectation Propagation (EP) algorithm approximates o (f|X, Y) with Gaussian approximation q(flx, Y) = N(((() :)) by iteratively moment matching marginal posteriors (9.
With the approximated posterior, the marginal distribution of f. = f(.) is still Gaussian. De- note the mean and variance as jl, and o.. respectively, then there is closed-form expression for p(y. I.,, X, Y) as:
p(y. - 11x,,,,,,,,,)). = (()))
14.
VT
VT+... o..
Given the predictive probability, we assume the prediction of the instance is the most probable label arg max,, p(y. XX, Y. x.). which known as the Optimal Bayesian Classifier (OBC) I
2.2 Bayesian Active Learning Based on Estimated Error Reduction (EER)
In the Bayesian active learning procedure, we consider a general case that the label given by the annotator is not deterministic, but the label follows an unknown distribution instead. Given the GPC model and observations, active learning sequentially chooses a query from a discrete instance set or continuous instance space, to efficiently improve the prediction performance of GPC. In each iteration, the query is chosen by optimizing an acquisition function. The acquisition function reflects the anticipated benefit of the queries, and it usually depends on the model posterior (f] Y). After observing the new data (I., the model is updated by Bayes' rule to get the posterior (J|X, Y, y.). In the following content, we'll Il discuss the iteration of active learning after observing training data (X, Y), for the sake of clarity, we omit X, Y from the notations of posterior and posterior predictive distributions as x (s) and p(y]z)).
Assume the distribution of instances over X' is vs ~ p(æ.), denote the optimal Bayesian classi- fier (OBC) (.). Based on GPC, the expected classification error of the OBC is:
Ez, (1 p(u+ aa(n) (2x,) [.,)} (x.)) E.. {1 max p(v= [x.)).
(2)
8a
3

---

In the GPC setting, if the latent function is known, we can accordingly label each instance with the most probable label as ( arg max, p(y x. f), where x denotes the optimal classifier. The classification error of y is Ex, {1 max, ply f)}. Due to the model uncertainty, the classification error of OBC should be larger than that of the optimal classifier. The Mean Objective Cost of Uncertainty (MOCU) measures the classification error increase due to the model uncertainty [16.. It is defined as the OBC classification error minus the expected classification error of the optimal classifier under model uncertainty:
M(*(f)) = Ex, {1 max p(ys (B maxp f)l}. N.
As MOCU captures the direct influence of the model uncertainty on the classification error, we can take the MOCU reduction in one sneessteploook-bbile setting as the acquisition function:
U (ax;; a (f)) -M((f)) - Ey. lr. [M( x y+))]
(4)
The second term is the MOCU after observing (æ.,, My), and averages over p(y- (x.) since we do not know the label of yy before observation. It is easy to verify that in (4), the maxy, p(ya 2 f) terms in Mfx(f)) and MMa[ffxx y are cancelled and the acquisition function can be proved to be equivalent to the Expected Error Reduction (EER) of OBC [7:
(a.)) En, {1 max ply xt x.)) Ey. (B [1 - max pe,, &., 9,
9o = BE, (Ey. [max p (2,, x, v)] max p(y.(=,))
(5)
Although MOCU reduction or EER is optimal for single queries, with noisy observation labels, it may get stuck before converging to the optimal classifier as discussed in [16]. To address this issue, the paper (17) proposed smooth concave approximation of MOCU, called Soft-MOCU (SMOCU), which replaces the max,, p(ys (x.) term in (3) with LogSumExp term:
(x)x(f)) = Em, (1 LogSumExx((k p(y|=.)) - Em()) (E,, max p(y, f)1).
(6)
k
where & is parameter controlling the approximation to MOCU. The resulting acquisition function based on the SMOCU reduction can be defined as:
S (x.) = Ez, (Ey m. LogSumExp(A (k: p(ys (x., Ix,, y.))] LogSumExp(k: p(uslx,))) (7)
k
which guarantees the convergence to the optimal classifier 17 The resulting acquisition function
smooth function of p(y, x., æ. y) and has no maximization operators. We will leverage this smooth acquisition function to derive the first efficient gradient- query synthesis active learning algorithm for GPC.
3 Efficient Active Learning for GPC
In this section, we present our ZEERbbssed active learning algorithms for GPC based on the acquisition functions defined by the MOCU and SMOCU reduction. At each iteration, with the updated GPC given previous observations, the acquisition function (5) or 7 can be optimized to guide the selection of the next query for active learning. We first present straightforward algorithm for both the discrete instance set (pool-based sampling) and continuous instance space (query synthesis) scenarios, where the acquisition function is optimized by random optimization. Random optimization first collects random sample set X C. X of size M1. calculates the acquisition function for each sample in the set and then takes the instance with the maximum acquisition function value the query.
To calculate (5) or (7) the integral over space is not analytical. Hence we need calculate the integral by Monte Carlo sampling with M2 samples of I, E X. Define (x,; and (zx; v.)) such that UM (x.))
Ez, (xx;; (xx);==))) Let g( x.) denote either g (xx;; ) or g (xxx ). For each x., the calculation of g( gx x.) requires deriving the probability distribution of p(ys (a..) and p(ya (x, v. y.), Vy. E ). Here, p(ys|ar (ax.) can be calculated directly from (1) while updating p(,./a,,, F. v.) needs incremental training of GPC with observations (X,Y Y.a.. .. y.) based on the EP approximation, and then calculating (1). The whole
Ez, (xx xx)) and US
procedure of optimizing the acquisition function at the n-th iteration need to retrain GPC for each

---

possible pair (Ix, v-), therefore, the EP approximation needs to be performed 2 x M1 times. The whole procedure is illustrated as the pseudocode Algorithm in the Appendix.
There are three issues of the acquisition function calculation in the straight forward algorithm. First, need large number of samples M2 have reliable estimation of the integral in (5) or (7) Second, the calculation of p(ya 2., y-) requires incremental retraining of the GPC model for each pair of (x+: )/ with computational complexity ((M)n Third, even though U ) is differentiable function of p(ya Ø., æ. W+), in the algorithm we actually use the EP approximation (.. y/-) calculate U (x.). ). and it is impossible to calculate the gradient Vg(ff z.. y=) during the EP procedure.
GER-based active learning algorithms to address these three presented challenges: 1) By importance sampling leveraging inhere GPC assumptions, we reduce the required number of samples for estimating acquisition functions; 2) We compute p(ys|as. :. y-) by deriving ana- lytic solution to marginalize the joint distribution p(us, Ye xt, æ.,) from the approximated posterior q(]|X, Y). which avoids the retraining with EP approximation: 3) More critically, we derive the first gradient-based query synthesis algorithm when using the SMOCU-based acquisition function by deriving the gradient Vq(f(x,, v/) for efficient active learning.
We develop
3.1 Importance Sampling
Regarding the issue in requiring the high sampling number of I, for reliable estimation of the acquisition function, we notice that most of the kernels applied in GPC assume that the observed data only have influence on neighhoring regions. Hence, we only need to account for the samples near æ. to estimate its influence on the classification error, thereafter the acquisition function for each new query. More specifically, we can use importance sampling to reduce the required sampling number. Reformulate the expectation (x.) = Ee, [9(2;;; æ.)] over an assistant distribution p(x.; x.) as: Uxx)) = E.,.aiim.......... [p(x; (gllz..g.ge)) (g(;); x..) p(z;: x.)).
The variance of the expectation minimized when p(z,; x.) is proportional to p(æ.)g(<s x.). But this requires knowledge of the value of (zx)), which is the acquisition function that we try to estimate. In practice, we can choose p(x,;; x 2. x. )p(x)) instead, as k( x.) reflects the non-zero region of g( x.): k(x,, x.) 0 means (2., and (x., y.) are independent, thus p(y, E., x. y.) = p(y, (x.)) and g( x.) Ro O. For example, if p(xx)) is uniformly distributed within finite region, k(x;, A(Z,, x.) is a square exponential kernel, then p(a;;; xx.) can be chosen truncated Gaussian distribution. Note that importance sampling and random optimization is only suitable for continuous instance space, or discrete set with large cardinality, For small instance set, can traverse all the elements for calculating the expectation and optimizing the acquisition function.
3.2 Joint Distribution Calculation
To avoid the retraining of GPC for each (x., we can calculate the posterior predictive by p(ya|at (2., w., y.) p(ys, wx, a.)/p(y, (x.). which requires computing the joint distribution of p(ys, ys lx, ). We remind the reader all the probabilities are conditioned on {X, Y} and we omit them for the seek of brevity. Denote f. = f(x,) and fa = f(x.). Now we show how to simplify the calculation of p(ye. (x., J. In the Gaussian approximation of the posterior q(ff.X. Y), the joint distribution of fs: still Gaussian. Since ys and y. are conditionally independent given fs and f., the joint distribution can be expressed as:
// fal Dox) dff..
ply. 1, y, = 1| x.))
(8)
where Pa. and Σg, are the marginal mean and covariance matrix of fa and f.. This integral can be simplified as a one-dimensional integral. Denote p.a (( Ps and Σ.. Oss Tex μr
Tex oue
We can decompose the joint Gaussian distribution as the marginal distribution of Sa times the conditional distribution of fo given far ie. &(fer fo Hon, E.e) o(f. 6(falme, (f Ji, (fa), ō..).
5

---

where (f.) = 2r + (fo 20)000/
30/0000000000//0//0/03 /... and o.. = J.z -32//Os5 Then, 8
can be transformed as:
//
ply. = 1, Vv = 1| x.) =
falMss Ege )df.dffff..
= |f @ (f.),
/ Φ( (Ja) (fffa)
(9)
The last line is based the integral equation introduced in [9] The above equation (9) calculates the joint distribution with the 1-d integral in constant time. With the joint distribution p(ys = easily obtain the joint distribution of ys. V. with other label pairs, and
y. 1 ).,,, finally we can obtain the posterior predictive p(y. 2,. 2., yy. without retraining the GPC with EP approximation. Based on this solution with importance sampling, we can develop efficient algorithms estimating MOCU/SMOCU reduction. Combining with Random Optimization, we develop the active learning algorithm with MOCU reduction named Retrainini MOCU reduction with Random Optimization (NR-MOCU-RO) and similarly for SMOCU reduction NR-SM AR-SMOCU-RO The pseudocode of MOCU-RO can be found in the Appendix (Algorithm 2).
3.3 Gradient Calculation
With the introduced marginalization strategy and importance sampling, significantly improve the computational efficiency for pool-based active learning with GPC. However, in the query synthesis problems, we would like to optimize the acquisition functions with gradient-basee algorithms. Usually the acquisition functions are multi-modal in the feature space, so the common practice is to perform random optimization first, and then take the optimal point as the initial point perform the gradient- based algorithms [4]
Here consider the gradient calculation of the acquisition function based the SMOCU reduction (US(x,) for query synthesis active learning as U (x.) is a smooth function whose gradients exist everywhere. For Gaussian Process, the gradients of its mean and covariance functions have closed- form expressions of the gradients of its adopted kernel function. Here we assume that given the EP approximation q( Y) and any pair of points (ar,, ). the gradients Vpe, Vo.. and Va with respect to x, are known. Calculation of these gradients is detailed in Appendix E. With this assumption, we can use the chain rule finally express the gradients U (x.) in the form of Vp.. Vo,, and Va
In 7 the second term is unrelated I.,, the gradient of the SMOCU reduction is:
(U)()=) = Ez, Vgg (x,, x.)) = VE p(y x.) LogSumExp(k p(ys ( 2 y+)1}
V.
Σ Vp(y,/xx)) Ex,, LogSumExpl is ply, x.. x x., y-)1 N-
Σ p(v.kw.) Ez, (V LogSumExp/H /k: p(y, ( V., 2.,
(10)
In the first term, Vp(y- xx)) can be calculated with V and Vo.. based on (I.. For the second term, we can use the chain rule to compute VLogSumExple [k p(ya (x,, x. y-)] = 91 92, where:
91 HLoSSumExp l: p(ys [x,, F., y+)
J2 Fp(y,/x,, x.. yz).
(11)
0p(Ue (2,, x., M/))
Since p(ya x., x.,, y.) = p(yer /1 only need to calculate Vp(us, y/ |2,,, xx.) based
[., x. 3/v(y (x.) and we have calculated Vp(y, (x.), now we
1, y 1/ xx, = / ((f.))(felH) (La(fa)
Vply. = 1,
Vo.. +1 )df
-f f (all)) Vo.o
(Ts) V(
(Sa(s))
Von +I ydfs, (12)

---

which is again a 1-d integral. The gradient of V R.(Sa) connected the calculation of V ja. (fa) and Vō..:
can be again calculated by chain rule, and
vo.++
Me Vo
2
Vja+(f)) Vu,
Vo Vo
(13)
Jss
3ss
Therefore, 4
(=(f=) is linear function of fa. and we can use numerical integral methods to calculate 12 The query synthesis algorithm with the integral computation is summarized in the
vo.++I
pseudocode:
NR-SMOCU with Stochastic Gradient Descent (NR-SMOCU-SGD),
In summary, to reduce the number of samples æ, for reliable estimation of acquisition functions, our algorithm utilizes an importance sampling with an assistant distribution chosen according to the kernel function. By calculating the posterior predictive directly from the joint distribution, the algorithm avoids retraining GPC with EP approximation. The introduction of the joint distribution also enables the efficient calculation of the gradient of the smooth acquisition function, with which we develop an efficient active learning algorithm for query synthesis.
NR-SMOCU-SGD: n-th iteration
1: function GR GRADIENTOPT((x))) ((X XX, Y))
2: Obtain initial point æ, from RANDOMOPTO(()) Y))
3: while not converge do
4: Sample M2 samples of a. ~ p(xx;; Calculate p(y. (x.) and Vp(y- x.)) by
5:
6:
for each x, do
7:
for y in {0, 1] do Calculate p(ye, yr v.). end for Calculate Vgg( end for
8:
Vp(ys, yn Ie xx)) and plys (x,) by
9 12
9:
10:
(x. x.) by
10
12
11:
12:
(US(2.) = 1 Ex, ((x,) V (x, (/)aaaa)))
Me
13:
Update with VU (I.) end while
14:
15:
return v.
16:
end function
4 Experiments
In this section we demonstrate the efficiency of our active learning algorithms combined with either random optimization (NR-MOCU-RO, NR-SMOCU-RO) Adagrad (NR-SMOCU-SGD) in the following sets of experiments. In the first set of experiments, we analyze and benchmark the running time of our algorithm by comparing to the naive computation of the MOCU/SMOCU reduction. Then we benchmark our algorithms with other active learning algorithms for both query synthesis on synthetic benchmark datasets, and pool-based active learning on real-world datasets. The competing algorithms include random sampling, Maximum Entropy Search (MES) 12]. Bayesian Active Learning by Disagreement (BALD) [5]; several MOCU-based active learning algorithms are also included to better show the accuracy and computation efficiency of our proposed algorithms, including the original MOCU and SMOCU
algorithms as described in the beginning of Section 3. In addition, an ADF-MOCU algorithm is also included for comparison, which uses Assumed Density Filtering (ADF) to retrain the GPC (6) and importance sampling (IS) to calculate the expectation. In our experiments,
use GP prior for ʃ with the squared-exponentis kernels k(x, x') >2xxp( | æ - x' /122, where {y, |} are model hyperparameters. The label probability modeled with the probit function p( y) x. f) 4 (f(()))) The code for our experiments is made available at hhttt:::: //github. com/ www//ianm..aeeeoooe aabaaaaaaaaaaaaaaaaaaaaaaaaaaaaa NR_ SMOCRSSSSSUSSCCER SRROCCU_SDDGGPPPP
4.1 Estimation Accuracy And Running Time Comparison
We first evaluate the effect of using the joint distribution integral in calculating the acquisition (1) functions. We compare the estimation of (!) (yal E. xr y+) through the joint distribution Integral
7

---

Table 1: Running time (in seconds) and estimation accuracy.
Algorithm
n 10
n= 100
=500
1000
naive
0.125
0.675
64.2
846
Time (s)
IS
0.126
0.674
64.2
848
IS+JDI
0.568
0.591
0.598
0.643
naive
0.149
0.207
0.195
0.27
Relative Error
IS
0.013
0.029
0.027
0.026
IS+JDI
0.013
0.032
0.027
0.022
( with yl (ys [x,, x2 y/) estimated by Retraining GPC with EP approximation. For this set of experiments, generate the initial data points using latent function f sampled from the GP prior. The instance space X' = [-4, 4]. and the hyperparameters 7² = 0.5, 1 = 1. We initially sample 100 data points to train GPC, then we compare the values g( = LogSumExpl (:: -p"") (ys (2,, x, y)) and g = LogSumExp p( (y,, xz, æ. y))/k, since these are related to the calculation of the SMOCU reduction. With 1000 pairs of (æ., v.) randomly sampled from X, we calculate the relative error as (g(*) - 1/ The average relative error is 1.8e-5 and the maximum relative error is 2.4e-3. We also compare the values g( maxy, ( (y.la., x, y) and g g( maxy, p! (yslaz (y,..,, Z, y), which is used to calculate the MOCU reduction. The average relative error is 1.3e-5 and the maximum relative error is 2.2e-3. These results validate that using (9) can provide accurate estimates of the acquisition functions.
Next, we compare the running time and accuracy of estimating acquisition functions by three algorithms: 1) a naive algorithm calculating p(y, [x5, Re,yo) v-) with GPC retraining (naive), 2) sampling a, by Importance Sampling and calculating p(y, (22, R., with GPC retraining (IS), 3) sampling v, by Importance Sampling and calculating p(y. [2,, L., yo) with the Joint Distribution Integral (IS+JDI). The algorithms are implemented in Python 3.7 on personal computer with Intel i5- 10400 2.9 GHz CPU and 16G RAM. We set the initial datasets of size n = 10, 100, 500 and 1000, respectively. Note that in all three algorithms, inferring the predictive distribution for each æ, also takes a considerable part of computations, especially when n is small, so we separate out the computation of the predictive distributions and directly compare the running time between retraining GPC (in the naive and IS algorithm) and calculating the joint distribution integral (in the IS+JDI algorithm). For all three competing algorithms, we sample 1000 æc,' s, and calculate (T. We benchmark them with a "ground truth" algorithm: calculating (x. by the naive algorithm with /eó x, samples. We perform this comparison for 100 a. ss. The average running time for a single æ. and relative error are shown in Table [ From the table, the relative error of the naive algorithm with 1000 samples is much larger than the other two as expected since the other two algorithms use importance sampling with smaller estimation variance. As n increase, the running time of both naive and IS algorithms increases fast since GPC training has a complexity of O(n²), ). while the running time of IS+JDI does not change much because the joint distribution integral is calculated in almost constant time. The results show that importance sampling does not impose much extra computation load in active learning, while it significantly increases the estimation accuracy. When is large, the joint distribution integral faster than retraining GPC with EP approximation.
4.2
Query Synthesis
We now benchmark our proposed algorithms with a challenging synthetic dataset checkerboard 4 x 4, similar the one tested in (5). which emulates the setup of phase diagram identification in materials science. Fig. 2a illustrates the classification boundaries. We perform all the competing active learning algorithms on this dataset. MES, BALD BALD.RO-MOCU, RO-SMOCU, ADF-MOCU, NR-MOCU-RO and NR-SMOCU-RO are all optimized by random optimization with Mi = 1000. In NR-SMOCU- SGD, we first perform random optimization with M₁ 800 and set the best point as the initial point for Adagrad so that WR-SMOCU-SGGI has similar running time compared with NR- ER-SMOCU-RO at their corresponding setups for fair comparison. Algorithm performance is measured in terms of the eтror regret defined as the OBC error at n-th iteration minus the optimal classifier error of the simulated ground truth. To show the influence of the observation error on the performance of different algorithms, we further assume that there is a constant flip error rate on the observing labels. We report the results with different error rates equal to 0 and 0.2. In the experiments, we initially draw 100 samples for labeling to estimate the hyperparameters {7, (]. Then we perform different
8

---

algorithms to collect new data. We repeat the procedure for 150 runs and plot the average error regret with standard deviations in Figs. 2p-c. for the flip error rate 0.2, the running time of each algorithm in each iteration is also shown in Fig. 21. The algorithm performance with the flip emor rate of 0. is shown in the Appendix.
We can observe from these figures that MES does not performs well, which is because MES tends query the points close to the decision boundary, while this problem has multiple intertwined boundaries and MES cannot differentiate different boundaries. OR-MOCU and OR-SMOCU perform worse than our proposed NR-MOCU-RO and UR-SMOCU-RO with respect to both computation and sample efficiency. That is because when compared with IS, the naive Monte Carlo used in DR-MOCU/OR-SMO -MOCU with GPC cannot accurately estimate the acquisition function values. On the other hand, the running time of OR-MOCU/OO SU/OO-SMMCUU SMOCU increases much faster than NR-MOCU- CONNR-SMOCU-RO since they retrain the model for every query samples. ADF-MOCU runs faster than our proposed algorithms, but it performs poorly as the ADF approximation is not accurate enough. We also observe that as the error rate increases, the difference between NR-MOCU and NR-SMOCU also increases, that is because as the error rate increases, MOCU reduction tends to ignore the long-term effect a query, leading to the degraded long-term performance, also discussed in [16.. Among these algorithms, NR-SMOCU-SGD performs better than other proposed algorithms by leveraging the gradient information in the optimization procedure.
4.3 Pool-Based Active Learning with Real- World Datasets
We also compare algorithms on the UCI datasets
pool-based active learning. NR-SMOCU-
SGD is not included as it is designed to search the continuous space. For each dataset, we split it into training and testing datasets. We take the training dataset as the sampling pool for active learning, initially we randomly choose two samples from each class for labelling, and use them to estimate the GPC hyperparameters. Then we apply the competing active learning algorithms to sequentially select the query from the training dataset and estimate the OBC eπror with the testing dataset after each iteration. Details of the tested UCI datasets are provided in the Appendix.
Running time 14
0.14
0.13
BALD
OR-SMOCU
MOCU
0.10
MR-MCCU-RO
NR. RR-SMOCUIRO
WR-SMOCU-SGD
Iteration number
(a) Checkerboard 4 x problem
(b) Flip error rate
120
2. x
2x
100
Eroo rogres
1.9
80-
18x10-1
BALD
MES
Runningg
00
BALD
MES
OR/MCU
1.7x10
ADF-NOCU
16x
NI--OCIU-RO
WR.SMOCU.RO
20
NR- NR-SMOCURRO
NI--MCCU-RO
WR-SMOCLL-5GD
20
20
Number lakeled
(c) Flip error rate 0.2
(d) Running time comparison for flip error rate = 0.2
Figure 2: Comparison of expected OBC error regret and running time on the 4 checkerboard problem.

---

0.16
0.300
0.275
rate
0.10
random
0.225
random
MES
BALD
0,200
ADF- MO-MM
ADF-MOCU
NR. SMOCUROO
0.150
NR-SMOCU-RO
10
15
20
10
20
Iteration
Iteration number
(a) WDBC
(b) Ionosphere
0.12
ente rate
0.22
0.09
0.20
MES
MES
BALD
0.06
BALDD
0.18
0.16
NR-MOCILRO
NR MR-MOCI-RO
NR-SMOCU-RO
NR-SMOCU-RD
Iteration number
Iteration number
(c) Vehicle
(d) Wine
Figure 3: Classification error rate comparison on UCI datasets.
We repeat the active learning procedures for 100 runs and plot the average OBC error with standard deviation values for each algorithm for performance comparison shown in Fig- In pool-based active learning, the expectation in MOCU/SMOCU is calculated by averaging over the whole instance pool, and there no need for sampling, so the acquisition functions calculated OR-MOCU/OR SMOCU are almost the same as the ones calculated by performances of OR-MOCU/OR-SMEE We notice that ADF-MOCU also performs similar to NR- IR-MOCU-RO, though ADF provides rough approximations. That is because for the discrete space and small number of iterations, the small deviation due to approximation will not change the ranking of the instances in the pool. Overall, ADF-MOCU, NR-MOCU-RO and NR-SMOCU-RO are better than MES and BALD, which validates that MOCU-based algorithms achieve better sample, in particular label efficiency. Note that in the Vehicle and Wine datasets, MES performs closely to MOCU-based algorithms. When checking the converging GPC model, the classification boundary is relatively simple on these datasets with relatively -ow-dimensional feature space (d < 18). The WDBC and lonosphere data are in higher dimensions (d > 30), for which our proposed algorithms perform significantly better.
-OS-ROONNSSOO---ROOOOO Therefore, are not provided in the figure for the sake of conciseness.
5 Conclusions
We have proposed efficient EER-based active learning algorithms with GPC, which estimate the error reduction by querying instances based on the joint distribution of label pairs. We have derived the joint distribution as one-dimensional integral with constant computation cost and calculate the predictive posterior based on Together with importance sampling, the acquisition function can be estimated efficiently by the 1-d integral of the joint distribution without incrementally retraining GPC with EP approximation, which has computation complexity of O(n²). Without the need for EP approximation, we can further derive the chain rule to calculate the gradient of the SMOCU reduction, which provides us the first efficient gradient-based query synthesis active learning algorithm. Our experiments have demonstrated not only the accuracy and the running speed of our algorithms but also consistently better data/label efficiency compared with competing algorithms for both query synthesis and pool-based active learning.
10

---

Acknowledgments X. Qian was supported in part by the National Science Foundation (NSF) Awards 1553281, 1812641, 1835690, 1934904, and 2119103. B.-J. Yoon was supported in part by the NSF Award 1835690. The work of E. R. Dougherty and F. J. Alexander was supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Mathematical Multifaceted Integrated Capability Centers program under Award DE-SC0019300
References
[1] Lori A Dalton and Edward R Dougherty. Optimal classifiers with minimum expected error within Bayesian framework -aat Discrete and Gaussian models. Pattern Recognition, (6(5):131--1314, 2013.
[2] Dheeru Dua and Casey Graff. UCI machine learning repository. 2017.
[3] Weijie Fu, Meng Wang. Shijie Hao, and Xindong Wu. Scalable active learning by approximated error reduction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1396-1405, 2018.
[4] José Miguel Hernández-Lobato Matthew W Hoffman, and Zoubin Ghahramani. Predic- tive entropy search for efficient global optimization of black-box functions. arXiv preprint 1rXiv:1406.2541, 2014.
[5] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:11 2011.
[6] Ashish Kapoor, Eric Horvitz, and Sumit Basu. Selective supervision: Guiding supervised learning with decision-theoretie active learning. In IJCAI, volume 7, pages 877-882, 2007.
[7] Malte Kuss, Carl Edward Rasmussen, and Ralf Herbrich. Assessing approximate inference for binary Gaussian process classification. Journal of machine learning research, 6(10), 2005.
[8] GE Poirier, WP Fitts, and JM White. Two-dimensional phase diagram of decanethiol on au (111). Langmuir, 17(4): 176-11887, 2001.
[9] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pages 63-71. Springer, 2003.
[10] Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the 18th International Conference on Machine Learning, ICML '01, pages 441-448, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc.
[11] Raphael Schumann and Ines Rehbein. Active learning via membership query synthesis for semi- supervised sentence classification. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 472-481 2019.
[12] Paola Sebastiani and Henry P Wynn. Maximum entropy sampling and optimal Bayesian experimental design. Journal of the Royal Statistical Society: Series (Statistical Methodology), 62(1):145-157, 2000.
[13] Sambu Seo, Marko Wallat, Thore Graepel, and Klaus Obermayer. Gaussian process regression: Active data selection and test point rejection. In Mustererkemmang 2000, pages 27-34. Springer, 2000.
[14] Burr Settles, Active learning literature survey. Computer Science Technical Report 1648, University of Wisconsin-Madisene 2009.
[15] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378. PMLR, 2016.
[16] Guang Zhao, E Dougherty, Byung-Jun Yoon, F Alexander, and Xiaoning Qian. Uncertainty- aware active learning for optimal Bayesian classifier. In Proc. 9th Int. Conf. Learn. Repre- sent,(ICRR), pages 1-20, 2021.

---

[17] Guang Zhao. Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xiaoning Qian. Bayesian active learning by soft mean objective cost of uncertainty. International Conference on Artificial Intelligence and Statistics, pages 3970-3978. PMLR, 2021.
[18] Jia-Jie Zhu and José Bento. Generative adversarial active learning. arXiv preprint 1rxi::100..07966, 2017.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Sections & 5. In addition to deriving label- efficient EER-based active learning for GPC with improved computational efficiency, we have derived the first gradient-based query synthesis algorithm for GPC.
(b) Did you describe the limitations of your work? [Yes] See the discussions in Sections
2 & 3. Active learning algorithms this work are derived for GPC using probit likelihood, which may have its inherent limitation in nigh-dimeensione
feature space.
(c) Did you discuss any potential negative societal impacts of your work? [No] The paper focuses on active learning algorithm development, which helps reduce the cost and time of machine learning and experimental design. We do not foresee immediate negative societal impact if not implemented for malicious machine learning applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The settings for deriving the joint distribution and gradient calculation based on basic calculus as presented in Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] The derivation of the joint distribution and gradient calculation detailed in Section
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL.)? [Yes] Included in the supplementary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the corresponding descriptions in all the experiments in Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] For all the experimental results, we have shown the error bars with standard deviation values.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] All the experiments are implemented PC described in Section 4.1.
12

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In Section 4, we have explicitly stated and cited the corresponding programming environment and competing algorithms for performance comparison.
(b) Did you mention the license of the assets? [No] The licenses of the assets can be viewed in their respective documentation.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We have included our code in the supplementary.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "hAj_WKdZUkq", "reviewer": "Reviewer_6XPv", "summary": "The manuscript addressed some detailed problems when applying existing strategies to pool-based active learning and query synthesis-based active learning. They use importance sampling to reduce the sample numbers, derive an analytic expression to calculate the prediction distribution, and employed SGD to optimize the query samples.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe novelty of this manuscript is limited, which does not bring breakthroughs for active learning methods based on Gaussian processes. This manuscript did not propose any novel selection strategy (one of the cores in the field of active learning), nor did it make substantial improvements to the learner. They applied the existing selection strategies MOCU and SMOCU to pool-based and query synthesis-based active learning using some conventional technology, e.g., importance sampling and Adagrade to solve efficiency and gradient related problems.\n\nQuality: \nIn Introduction, it is described that the complexity of training GPC with EP approximation is O(n^3). As far as I know, since the sparse GP was proposed the complexity has been greatly reduced, even down to linear with training size when using a stochastic EP or variational auto-encoder. The authors never mentioned these works.\n\nIt’s good to use analytical inference to obtain the prediction distribution expression as introduced in Section 3.2, but with the increase of newly coming data, the model parameters had better be learned.\n\nThe discussion of the related work is insufficient missing some recent work, and the contrast experiments are crude which just compared with MES (Sebastiani et al., 2000) and BALD (Houlsby et al., 2011). The comparisons among NR-MOCU-RO, NR-SMOCU-RO, and NR-SMOCU-SGD did not contribute to the verification of the advantages of the proposed method, which just compared the performance of some existing technologies such as LogSumExp and Adagrad. The authors have mentioned that deep generative models show promising potential with related work, but why not compare with those novel deep models on higher-dimensional datasets.\n\nClarity:\nThis manuscript seems to be finished in a hurry, and a lot of writing is not standard. For example, in Eq. (5) and many other places, y and ys are confused.\n\nSignificance: \nThe experiments are not convincing. In Table 1, the efficiency improvement is not significant. The results have not been thoroughly analyzed, e.g., why IS and IS+JDI has lower error rate than naïve method？What does the random optimization in NR-MOCU-RO mean?  Why does MES perform closely to their algorithms in the Vehicle dataset?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should give the limitations and potential negative social impacts of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "CrI39fraJTl", "reviewer": "Reviewer_anoP", "summary": "This study gives an efficient solution for Bayesian active learning based on Estimated Error Reduction (EER) with the Gaussian process classification model. Specifically, the recently proposed MOCU-based Bayesian active learning for EER is used for representing the joint distribution of queries as a one-dimensional integral and avoid relearning the GPC every time when computing the acquisition function. Thanks to the smoothness of the SMOCU-based acquisition, not only the pool-based but also a query synthesis active learning is efficiently implemented.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I acknowledge the importance of this work, which uses the recently proposed MOCU and SMOCU to improve the efficiency of Bayesian active learning based on GPC.\nThe formulation that attributes the simultaneous posterior distribution of labels to a one-dimensional integral seems to be reasonable, and the fact that membership synthesis is possible is also commendable.\nThere is no novelty in speeding up GP(C) with Gaussian approximation, marginalization, and importance sampling, and the main novelty of this research is the use of SMOCU to make the acquisition function differentiable and to enable membership synthesis.\n\nIn my opinion, although I acknowledge the early adaptation of recently developed technology has its own importance to some degree, the contribution of the paper is not so large, particularly without any theoretical evaluation of the estimation accuracy of p(y|x_s,x_*,y_*) by the proposed method.\n\nMissing reference:\nActive learning is already successfully applied to phase diagram identification problem. I recommend citing the following article:\n\"Efficient construction method for phase diagrams using uncertainty sampling\", K. Terayama et al., Phys. Rev. Materials 3, 2019.\n\n\n--- After Reading authors' response\nI acknowledge that there are some novelty in this submission, but I still think the contribution of this paper is not enough to clear the bar of NeurIPS.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the applicability issue to high-dimensional datasets is acknowledged.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "4eheCgyNUVA", "reviewer": "Reviewer_zuL2", "summary": "This paper addresses active selection of queries via expected error reduction (EER) for Gaussian process classification. The authors describe Gaussian process classification and EER, and then summarize the MOCU acquisition function (equivalent to EER) and SMOCU acquisition function, which is an approximation to MOCU and therefore an approximation to EER; both MOCU and SMOCU are existing metrics in the literature. The main contribution of this work is an efficient method to estimate the MOCU and SMOCU metrics for pool-based active learning, and a gradient calculation for SMOCU, leading to SMOCU query synthesis (and hence, an approximation to EER query synthesis, which the authors claim to be the first attempt at query synthesis for an EER-based strategy). Specifically, the authors implement importance sampling, a calculation of critical probabilistic quantities that do not require expectation propagation (as would be required otherwise), and derive an acquisition function gradient calculation. The authors empirically evaluate the computational benefits of their methods, and test empirical generalization accuracy on real and synthetic datasets against several baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The contributions made in this paper seem to be natural applications of existing active learning acquisition functions to the setting of Gaussian process classification. However, the means by which this is accomplished involved creative mathematical manipulations to make acquisition function optimization more computationally tractable. It is nontrivial to arrive at the appropriate insights needed to develop such approximations.\n\n**Quality**: The introduction, background, and mathematical developments seem to be technically sound, and the illustrative examples presented were convincing. My main concerns involve the extent of the experiments performed. In my opinion, critical experimental results are missing. Specifically, I was surprised that a comparison was not made against an EER baseline in either the synthetic or real-world datasets, and additionally think that the computational times of each comparison method should be evaluated and presented. I appreciate the effort to benchmark computational performance in Table 1 to illustrate the tradeoff between estimation accuracy and computational budget, but I think such an analysis should also be present in the full active learning experiments.\n\nMy reason for these concerns is from the advertised main contribution of the paper, which from my understanding is to improve upon EER selection in a computationally efficient manner. To me, whichever method is developed should then be empirically tested against EER, and the tradeoff between generalization error and computational budget fully evaluated. It seems to me that EER could have been tested with expectation propagation being used in the acquisition function, even if this would require a large computational budget. I welcome the authors to comment about these concerns about lack of comparison against EER (at least in the pool-based experiments) and lack of computational expense results (across all experiments).\n\nThe authors comment in line 268 “Note that in all three algorithms, inferring the predictive distribution for each xs also takes a considerable part of computations, especially when n is small.” The results in Table 1 could be strengthened by explicitly separating out this computational cost in the analysis.\n\nI also felt some aspects of the discussion were lacking. It doesn’t seem that the performance of BALD was discussed in Section 4.2. This seems important since BALD seems to perform just as well as the proposed methods.\n\nAs I illustrate below in “Details”, I am concerned about the lack of detail in how the experiments were performed, which could make reproducibility harder.\n\n**Clarity**: The paper was mostly clear. The inclusion of Figure 1 as an illustrative example was effective. The authors did a good job overall of describing Gaussian process classification and their solution approaches. However, I think insufficient experimental details are included, which could make reproducibility more difficult. I’ve detailed these in “Details” below.\n\n**Significance**: The problem of efficient EER-based active learning strategies for pool-based or query synthesis Gaussian process classification is a solid technical contribution in the realm of Gaussian process classification active learning. Except for my concerns about the experiments above, I believe this work constitutes a significant advancement of efficient active learning for Gaussian process classification.\n\n**Details**: \n- line 158: why is the gradient impossible to compute during EP? This should be elaborated on in detail, especially since as advertised in line 163, it seems from the writing that this gradient is in fact derived. I found these two statements confusing, and it seems that the points made here should be understandable and clear since this discussion constitutes the motivation for Section 3.3. Related to this, I am confused by this comment: “Without the need for EP approximation, we can further derive the chain rule to calculate the gradient of the SMOCU reduction, which provides us the first efficient gradient-based query synthesis active learning algorithm.” My understanding from the paper is that it is the smoothness of SMOCU that enables the use of a gradient, not the avoidance of EP.\n- what is the choice of p(x) in the experiments\n- I found it confusing to use the terminology absolute error ratio instead of “relative error”. It left me wondering if “absolute error rate” used in Table 1 was a normalized version or not\n- in line 260, how were the candidate values of $x_*$ sampled or selected?\n- I think the acquisition functions for MES and BALD should be included at least in the supplement\n- I was very confused by the description of the injected bit flip error (description starting 282). Is this a bit flip of a noiseless label, or an additional bit flip on a label already sampled according to the probit function?\nin line 300, using only 2 labeled points seems very low to me to estimate hyperparameters. Can the authors comment about this?\n- what is the test/train split for the UCI datasets? Are these splits randomized across runs, or fixed a priori for all runs?\n- in line 307: “When checking the converging GPC model, the classification boundary is relatively simple on the the Vehicle dataset with relatively low-dimensional feature space (d = 18)” How was this ‘simple boundary’ checked?\n\n**Minor points**:\n- I was very confused by the use of the notation g, starting in line 144. This should be defined explicitly, instead of implicitly as was done in line 145\n- line 151: $M_1$ is undefined\n- overall, I found the paragraph between lines 143-152 a bit difficult to follow and understand\n- line 221: relable -> reliable\n- line 242: form -> from\n- line 248: what is the max taken over? I assume $y_s$\n- does the performance metric in line 281 refer to equation (3)? If so, this should be referenced\n- supplement line 15: strait-forward -> straightforward\n- I think g(...) should be defined explicitly in the presentation of Algorithm 1\n\n**Overall**: Overall I enjoyed reading the paper. If my concerns above were addressed --- in particular my concerns about the extent of the experiments --- I would consider increasing my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mostly addressed the limitations and societal impact of their work. I do think that the following comment should be elaborated on “Active learning algorithms in this work are derived for GPC using probit likelihood, which may have its inherent limitation in high-dimensional feature space.” Dimensionality of the space is an important factor, and I think the paper would be strengthened by additional consideration to the problem dimensionality. This could be accomplished through additional synthetic experiments evaluating performance as a function of dimension, or a discussion of why specifically dimensionality should change aspects of the solution or computational cost. Similarly, how do these results scale with larger values of n? Stopping after on the order of 50-100 queries seems low to me.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledge that this paper has the contribution that it makes acquisition function optimization in Gaussian Process Classification (GPC) more computationally tractable; however, they also expressed the following concerns in the discussion:
- this work is an improvement of a very new active learning method in a specific setting (GPC), and whether the new active learning method has high impact on the community is still unclear;
- lack of comprehensive comparison with the important baselines, such as original EER/MOCU/SMOCU methods (although the reviewers appreciate the additional experimental results on the checkboard problem in the rebuttal).

We encourage the authors to incorporate the reviewers' suggestions to strengthen the paper in the next iteration. 
Decision: Accept (Poster)