Title: Efficient Active Learning for Gaussian Process Classification by Error Reduction
Authors: Guang Zhao
Edward Dougherty
Byung-Jun Yoon
Francis Alexander
Xiaoning Qian
Venue: NeurIPS 2021
Keywords: Bayesian Active Learning
Gaussian Process Classification
Expected Error Reduction
Query Synthesis

Abstract: Active learning sequentially selects the best instance for labeling by optimizing an acquisition function to enhance data/label efficiency. The selection can be either from a discrete instance set (pool-based scenario) or a continuous instance space (query synthesis scenario). In this work, we study both active learning scenarios for Gaussian Process Classification (GPC). The existing active learning strategies that  maximize the Estimated Error Reduction (EER) aim at reducing the classification error after training with the new acquired instance in a one-step-look-ahead manner. The computation of EER-based acquisition functions is typically prohibitive as it requires retraining the GPC with every new query. Moreover, as the EER is not smooth, it can not be combined with gradient-based optimization techniques to efficiently explore the continuous instance space for query synthesis. To overcome these critical limitations, we develop computationally efficient algorithms for EER-based active learning with GPC. We derive the joint predictive distribution of label pairs as a one-dimensional integral, as a result of which the computation of the  acquisition function avoids retraining the GPC for each query, remarkably reducing the computational overhead. We also derive the gradient chain rule to efficiently calculate the gradient of the acquisition function, which leads to the first query synthesis active learning algorithm implementing EER-based strategies. Our experiments clearly demonstrate the computational efficiency of the proposed algorithms. We also benchmark our algorithms on both synthetic and real-world datasets, which show superior performance in terms of sampling efficiency compared to the existing state-of-the-art algorithms. 


Review discussion:
I acknowledge the importance of this work, which uses the recently proposed MOCU and SMOCU to improve the efficiency of Bayesian active learning based on GPC.
The formulation that attributes the simultaneous posterior distribution of labels to a one-dimensional integral seems to be reasonable, and the fact that membership synthesis is possible is also commendable.
There is no novelty in speeding up GP(C) with Gaussian approximation, marginalization, and importance sampling, and the main novelty of this research is the use of SMOCU to make the acquisition function differentiable and to enable membership synthesis.
In my opinion, although I acknowledge the early adaptation of recently developed technology has its own importance to some degree, the contribution of the paper is not so large, particularly without any theoretical evaluation of the estimation accuracy of p(y|x_s,x_*,y_*) by the proposed method.
Missing reference:
Active learning is already successfully applied to phase diagram identification problem. I recommend citing the following article:
"Efficient construction method for phase diagrams using uncertainty sampling", K. Terayama et al., Phys. Rev. Materials 3, 2019.
--- After Reading authors' response
I acknowledge that there are some novelty in this submission, but I still think the contribution of this paper is not enough to clear the bar of NeurIPS.
No
1. Our main contributions include proposing new sample- and computation-efficient algorithms for active learning with GPC, based on MOCU/SMOCU acquisition functions, which have theoretical convergence guarantees and were shown to perform better than uncertainty sampling and other information theoretic active learning methods as MOCU/SMOCU directly reduces uncertainty affecting operational objectives. We will better articulate the contributions.
2. We empirically demonstrated the sample- and computation-efficiency by comparing prediction performances with the increasing number of queries through experiments. In active learning, we care more about the active learning performance, regarding sample- and computation-efficiency. It is difficult to directly connect the estimation accuracy of $ p(y|x_s,x_*,y_*) $  to the active learning performance in theory, due to the potential complexity involving iterative optimization during active learning.  Usually, the effect of estimation accuracy on active learning is just verified empirically [a].
Therefore, we will empirically verify the effect of the estimation accuracy by comparing the original MOCU/SMOCU algorithm with our proposed algorithms.
We will proofread the paper and make all the efforts to improve the presentation and add the missing references.
[a] Roy, Nicholas, and Andrew McCallum. "Toward optimal active learning through monte carlo estimation of error reduction." ICML, Williamstown 2 (2001): 441-448.
**Originality**: The contributions made in this paper seem to be natural applications of existing active learning acquisition functions to the setting of Gaussian process classification.
However, the means by which this is accomplished involved creative mathematical manipulations to make acquisition function optimization more computationally tractable. It is nontrivial to arrive at the appropriate insights needed to develop such approximations.
**Quality**: The introduction, background, and mathematical developments seem to be technically sound, and the illustrative examples presented were convincing. My main concerns involve the extent of the experiments performed. In my opinion, critical experimental results are missing.
Specifically, I was surprised that a comparison was not made against an EER baseline in either the synthetic or real-world datasets, and additionally think that the computational times of each comparison method should be evaluated and presented. I appreciate the effort to benchmark computational performance in Table 1 to illustrate the tradeoff between estimation accuracy and computational budget, but I think such an analysis should also be present in the full active learning experiments.
My reason for these concerns is from the advertised main contribution of the paper, which from my understanding is to improve upon EER selection in a computationally efficient manner. To me, whichever method is developed should then be empirically tested against EER, and the tradeoff between generalization error and computational budget fully evaluated. It seems to me that EER could have been tested with expectation propagation being used in the acquisition function, even if this would require a large computational budget. I welcome the authors to comment about these concerns about lack of comparison against EER (at least in the pool-based experiments) and lack of computational expense results (across all experiments).
The authors comment in line 268 “Note that in all three algorithms, inferring the predictive distribution for each xs also takes a considerable part of computations, especially when n is small.” The results in Table 1 could be strengthened by explicitly separating out this computational cost in the analysis.
I also felt some aspects of the discussion were lacking. It doesn’t seem that the performance of BALD was discussed in Section 4.2. This seems important since BALD seems to perform just as well as the proposed methods.
As I illustrate below in “Details”, I am concerned about the lack of detail in how the experiments were performed, which could make reproducibility harder.
**Clarity**: The paper was mostly clear. The inclusion of Figure 1 as an illustrative example was effective. The authors did a good job overall of describing Gaussian process classification and their solution approaches.
However, I think insufficient experimental details are included, which could make reproducibility more difficult. I’ve detailed these in “Details” below.
**Significance**: The problem of efficient EER-based active learning strategies for pool-based or query synthesis Gaussian process classification is a solid technical contribution in the realm of Gaussian process classification active learning.
Except for my concerns about the experiments above, I believe this work constitutes a significant advancement of efficient active learning for Gaussian process classification.
**Details**:
- line 158: why is the gradient impossible to compute during EP? This should be elaborated on in detail, especially since as advertised in line 163, it seems from the writing that this gradient is in fact derived. I found these two statements confusing, and it seems that the points made here should be understandable and clear since this discussion constitutes the motivation for Section 3.3. Related to this, I am confused by this comment: “Without the need for EP approximation, we can further derive the chain rule to calculate the gradient of the SMOCU reduction, which provides us the first efficient gradient-based query synthesis active learning algorithm.” My understanding from the paper is that it is the smoothness of SMOCU that enables the use of a gradient, not the avoidance of EP.
- what is the choice of p(x) in the experiments
- I found it confusing to use the terminology absolute error ratio instead of “relative error”. It left me wondering if “absolute error rate” used in Table 1 was a normalized version or not
- in line 260, how were the candidate values of $x_*$ sampled or selected?
- I think the acquisition functions for MES and BALD should be included at least in the supplement
- I was very confused by the description of the injected bit flip error (description starting 282). Is this a bit flip of a noiseless label, or an additional bit flip on a label already sampled according to the probit function?
in line 300, using only 2 labeled points seems very low to me to estimate hyperparameters. Can the authors comment about this?
- what is the test/train split for the UCI datasets? Are these splits randomized across runs, or fixed a priori for all runs?
- in line 307: “When checking the converging GPC model, the classification boundary is relatively simple on the the Vehicle dataset with relatively low-dimensional feature space (d = 18)” How was this ‘simple boundary’ checked?
**Minor points**:
- I was very confused by the use of the notation g, starting in line 144. This should be defined explicitly, instead of implicitly as was done in line 145
- line 151: $M_1$ is undefined
- overall, I found the paragraph between lines 143-152 a bit difficult to follow and understand
- line 221: relable -> reliable
- line 242: form -> from
- line 248: what is the max taken over? I assume $y_s$
- does the performance metric in line 281 refer to equation (3)? If so, this should be referenced
- supplement line 15: strait-forward -> straightforward
- I think g(...) should be defined explicitly in the presentation of Algorithm 1
**Overall**: Overall I enjoyed reading the paper. If my concerns above were addressed --- in particular my concerns about the extent of the experiments --- I would consider increasing my score.
No
We truly appreciate the reviewer’s detailed and constructive comments. We here provide responses to the main critiques by the reviewer.
1. Comparison with the EER baseline and show computational times
We will add the original EER and SMOCU algorithms into our current query synthesis and pool-based experiments for comparison on both the error rate and running time among these comparing algorithms. Due to the limited time to post the rebuttal, we will report the numbers during the discussion period and add them into the final paper.
2. Separating out the computational cost in Table 1
We thank the reviewer for the useful suggestion. As the reviewer suggested, we have run an additional experiment with $n$ up to 1000, in which we separate out the computation of the predictive distributions and directly compare the running time between retraining GPC (in the naive algorithm) and calculating the joint distribution integral (in the IS+JDI algorithm) when calculating the acquisition function $U^S(x_*)$. The running time with different $n$ is shown in the following table:
|               | n=10| n=100|n = 500|n=1000|
| ----------- | ----------- |------|---|-------|
| retraining GPC| 0.125	|0.675|64.2|846|
|  joint distribution integral| 0.568|0.591|0.598|0.643|
It is obvious to see that with increasing $n$, the running time of retraining GPC increases as expected based on the cubic time complexity, while the joint distribution integral can be calculated in constant time. We will add these results with discussions and revise Table 1 in our final paper accordingly.
3. Discussion on the BALD performance in Figure 2
From Figure 2 we can see that BALD usually performs bad at the beginning. This is because BALD aims to reduce the model uncertainty but not all the model uncertainty is related to the classification performance.
On the other hand, BALD performs reasonably well in the long run as the model uncertainty is reduced to certain degree to help classification. We will make more analyses and add detailed discussions on the experiment results.
4. Why is the gradient impossible to compute during EP?
For query synthesis scenarios, the gradients of the acquisition function involve computing the gradients of the updated posterior predictive distribution, which are intractable with EP approximation due to the fact that the chain rule computing the gradients is not feasible to the numerical procedure of moment matching in EP approximation. We detail the explanation in the following:
First we provide more details of the EP approximation. Given observations {$X, Y$}, EP approximates the non-Gaussian likelihood $ p(Y|X, f)$ with a Gaussian function as: $p(Y|X, f)\approx C \mathcal{N}(f|\hat{\mu}, \hat{\Sigma})$. Here $C$ is a constant and $\hat{\mu}$ and $\hat{\Sigma}$ are calculated by iterative moment match.
Given a testing instance $x_*$, the latent variable $f_* = f(x_*) \sim \mathcal{N}(\mu_*, \sigma_{**})$ where:
$ \mu_* = \mathcal{K}(x_*, X) (\mathcal{K}(X, X)+\hat{\Sigma})^{-1} \hat{\mu} $,
$\sigma_{**} = \mathcal{K}(x_*, x_*) - \mathcal{K}(x_*, X)(\mathcal{K}(X, X)+\hat{\Sigma})^{-1}\mathcal{K}(X, x_*)$,
where $\mathcal{K}$ is the kernel function.
Then we show how we calculate  $\nabla_{x_*} p(y_s, y_*|x_s, x_*, X, Y)$.
Since $\hat{\mu}$ and $\hat{\Sigma}$ are unrelated to $x_*$, the calculation of $\nabla \mu_* $ and $\nabla \sigma_{**}$ turn into the calculation of $\nabla \mathcal{K}(x_*, X)$ and $\nabla \mathcal{K}(x_*, x_*)$, which can be done as long as the kernel function is differentiable.
Similarly, we can calculate $\nabla \sigma_{s*}$. Given these gradient, we can calculate $\nabla_{x_*} p(y_s, y_*|x_s, x_*, X, Y)$ with equations (12) and (13).
Finally we explain why we cannot calculate the gradient if we retrain the GPC with EP approximation for observations {$X, Y, x_*, y_*$}. We assume EP approximates $ p(Y, y_*|X, x_*, f)$ with Gaussian function $C_* \mathcal{N}(f|\hat{\mu_*}, \hat{\Sigma_*})$. Now since  $\hat{\mu_*}$ and $\hat{\Sigma_*}$ are related to $x_*$, we should also calculate the gradients of them.
However, $\hat{\mu_*}$ and $\hat{\Sigma_*}$ are calculated numerically by iterative moment match, so the gradients of them are intractable.
We'll include all the details of the calculation in the appendix of the final paper.
5.  Choice of $p(x)$.
In the paper, the choice of $p(x)$ is the uniform distribution over the instance space.
6. The “absolute error rate” in table 1 is a typo, we actually mean the absolute error ratio of $U^S(x_*)$ computing by the three algorithms, comparing with the “ground truth”, we will use the more common term “relative error” in the paper.
7. In line 260, $x_*$ is just uniformly sampled from the instance space.
8. We’ll include the acquisition functions for MES and BALD in the appendix
9.  The bit flip error is injected to the noiseless labels defined by the checkerboard function.
We used four labeled points (two from each class) for hyperparameter estimation.  For the UCI datasets, we first normalized all the features to zero mean and unit variance, and we used the simple square exponential kernel with only two hyperparameters, under this case, four labeled points are enough to learn reasonable hyperparameters.
10. We split the test/train randomly across runs.
11. We tested the Vehicle dataset with the simple classifier, logistic regression, and the error rate is low, so we think the optimal classification boundary is close to a linear boundary.
12. We will fix typos and notation problems.
13. We will also revise the comments about limitations and potential societal impacts as advised by the reviewer.
The acquisition function is defined as the expectation over the instance space, and we use Monte Carlo to calculate the expectation. In high-dimensional instance space (dimension larger than 100), finding a proper proposal distribution for importance sampling is difficult, so the calculation may require a huge number of samples.
Regarding the number of queries, we will discuss based on different applications. In material phase diagram identification, which differs significantly from computer vision or image analysis problems, it is much more expensive to synthesize and profile the corresponding materials under different conditions, it is indeed already too much to ask to get more than 100 profiled materials.
Thank you for your reply. I was wondering if there was a status update on these experiments: "We will add the original EER and SMOCU algorithms into our current query synthesis and pool-based experiments for comparison on both the error rate and running time among these comparing algorithms. Due to the limited time to post the rebuttal, we will report the numbers during the discussion period and add them into the final paper."
Thank you for checking. We have just finished another set of  new experiments for comparison with existing active learning algorithms including the original EER/MOCU and SMOCU on both classification performance and running time with different numbers of active learning iterations. We have posted a general response to all the reviewers. Based on the additional comparison results, we confirmed that our proposed algorithms outperform the original EER/MOCU and SMOCU algorithms for Gaussian Process classifiers (GPC) and have better computational efficiency with the increasing number of active learning iterations.
Dear authors,
Thank you for your thorough rebuttal and for posting additional experimental results. After reading the other reviews, I will maintain my score at a 5. Since this work is an extension of a relatively new method that has been untested for GPC previously, I am concerned that the extensions have essentially skipped a research step in properly benchmarking the original EER/MOCU/SMOCU methods on GPC, which forms a basis of comparison for the new methods. Thank you for making strides in this direction in the rebuttal. I believe a more thorough analysis is needed on the full set of datasets against these baselines in order to increase the score.
We appreciate the reviewer's comments. We will include all the baseline results in our experiments with different datasets. Is there anything still requiring clarification or additional experiments for the reviewer to reconsider the score?
Originality:
The novelty of this manuscript is limited, which does not bring breakthroughs for active learning methods based on Gaussian processes. This manuscript did not propose any novel selection strategy (one of the cores in the field of active learning), nor did it make substantial improvements to the learner. They applied the existing selection strategies MOCU and SMOCU to pool-based and query synthesis-based active learning using some conventional technology, e.g., importance sampling and Adagrade to solve efficiency and gradient related problems.
Quality:
In Introduction, it is described that the complexity of training GPC with EP approximation is O(n^3). As far as I know, since the sparse GP was proposed the complexity has been greatly reduced, even down to linear with training size when using a stochastic EP or variational auto-encoder. The authors never mentioned these works.
It’s good to use analytical inference to obtain the prediction distribution expression as introduced in Section 3.2, but with the increase of newly coming data, the model parameters had better be learned.
The discussion of the related work is insufficient missing some recent work, and the contrast experiments are crude which just compared with MES (Sebastiani et al., 2000) and BALD (Houlsby et al., 2011). The comparisons among NR-MOCU-RO, NR-SMOCU-RO, and NR-SMOCU-SGD did not contribute to the verification of the advantages of the proposed method, which just compared the performance of some existing technologies such as LogSumExp and Adagrad. The authors have mentioned that deep generative models show promising potential with related work, but why not compare with those novel deep models on higher-dimensional datasets.
Clarity:
This manuscript seems to be finished in a hurry, and a lot of writing is not standard.
For example, in Eq. (5) and many other places, y and ys are confused.
Significance:
The experiments are not convincing. In Table 1, the efficiency improvement is not significant. The results have not been thoroughly analyzed, e.g., why IS and IS+JDI has lower error rate than naïve method？What does the random optimization in NR-MOCU-RO mean?  Why does MES perform closely to their algorithms in the Vehicle dataset?
No
We thank the reviewer’s for the constructive comments. We here provide responses to the main critiques by the reviewer.
1. Sparse GP
Compared with naive GP, sparse GP trades off the precision for efficient training. In the cases when $n$ is not too large, training with complexity $O(n^3)$ is acceptable, but optimizing the acquisition function requires a complexity of $O(M_1n^3)$, which can cause the scalability concerns in computation. Our proposed algorithm can calculate the precise acquisition function value with a complexity of $O(M_1)$ in calculating the joint distribution integral.
2. Learning hyperparameters
In the convergence proof of the active learning methods (such as SMOCU and BALD), the hyperparameters are assumed to be fixed. Updating hyperparameters in each iteration can make the active learning methods not perform as expected and may even make the active learning methods perform worse than random sampling, if they are sensitive to hyperparameter setups.
3. Only two benchmark algorithms are included, no comparison with original MOCU/SMOCU
In Bayesian active learning, BALD is among the state-of-the-art methods [5] and MES (uncertainty sampling) is the simplest and most commonly used method. We will run additional experiments to include more benchmark algorithms into comparison, including original MOCU/SMOCU and the Approximated Error Reduction (AER) algorithm [3] in the experiments that they are applicable.  Our paper focuses on the active learning problem with the GPC model, in particular with the query synthesis algorithms designed for GPC. Comparing the performance with deep models is out of the scope of this paper.
4. Table 1 is not convincing.
We have run an additional experiment with $n$ up to 1000, in which we directly compare the running time between retraining GPC (in the naive algorithm) and calculating the joint distribution integral (in the IS+JDI algorithm) when calculating the acquisition function $U^S(x_*)$. The running time with different $n$ is shown in the following table:
|               | n=10| n=100|n=500|n=1000|
| ----------- | ----------- |------|---|-------|
| retraining GPC| 0.125	|0.675|64.2|846|
|  joint distribution integral| 0.568|0.591|0.598|0.643|
It is obvious to see that with increasing $n$, the running time of retraining GPC increases in the cubic order, while the joint distribution integral can be calculated in constant time. We will add these results with discussions and revise Table 1 in our final paper accordingly.
5. IS and IS+JDI use importance sampling, so the variance of the estimation is smaller than naive Monte Carlo, and therefore the calculation of the acquisition function is more accurate than the naive method.
6. Random optimization first collects a random sample set, and then returns the instance in the set with the maximum acquisition function value. In pool-based active learning, we just use exhaustive search as typically done instead.
7. For active learning based on GPC, MES performing well indicates the classification boundary is relatively simple. We will make more analysis and add discussions on the experiment results.
In response to more comprehensive comparisons to existing methods with additional running time comparison, we have finally (we have to program some of the other competing methods ourselves for comparison) obtained additional comparison results on the checkboard $4 \times 4$ problem with the flip error rate = 0.2, comparing both the classification performance and running time.
Three more benchmark methods are included in the experiments. The first two are original EER/MOCU (OR-MOCU) and SMOCU (OR-SMOCU) methods, with their pseudo code provided in Algorithm 1 in the appendix.
The third method ADF-MOCU is based on an approximate inference technique called Assumed Density Filtering (ADF) [6]. ADF only performs moment matching for the new query to update the surrogate GPC model, in contrast to EP that performs iterative moment matching for all labeled data. ADF is faster but less accurate than EP, while the joint distribution integral (JDI) used in our proposed methods has almost the same accuracy as EP (shown in Sec. 4.1 in the original paper). The authors of [6] suggested to use ADF to approximately retrain the model when calculating the acquisition function values for each potential query.
Therefore, ADF-MOCU leads to worse sample efficiency than our proposed methods, due to its lower accuracy when computing the acquisition function values. In our implementation of ADF-MOCU, we used ADF to retrain the GPC, and still used importance sampling (IS) to calculate the expectation.
The additional performance comparison results are shown below:
**Error regret:**
|               | n=10| n=20|n=40|n = 60|
| ----------- | ----------- |------|---|-------|
|random|$0.2003\pm0.0019$|$0.1949\pm0.0019$|$0.1832\pm0.0019$|$0.1724\pm0.0018$|
|MES|$0.1998\pm0.0021$|$0.1899\pm0.0021$|$0.1751\pm0.0022$|$0.1641\pm0.0022$|
|BALD|$0.1965\pm0.0019$|$0.1864\pm0.0020$|$0.1726\pm0.0020$|$0.1609\pm0.0020$|
|OR-MOCU|$0.1943\pm0.0021$|$0.1844\pm0.0021$|$0.1702\pm0.0021$|$0.1595\pm0.0021$|
|OR-SMOCU|$0.1954\pm0.0020$|$0.1862\pm0.0020$|$0.1707\pm0.0020$|$0.1591\pm0.0020$|
|ADF-MOCU|$0.1956\pm0.0020$|$0.1857\pm0.0020$|$0.1703\pm0.0019$|$0.1582\pm0.0020$|
|NR-MOCU-RO|$0.1934\pm0.0020$|$0.1842\pm0.0020$|$0.1682\pm0.0020$|$0.1566\pm0.0021$|
|NR-SMOCU-RO|$0.1929\pm0.0019$|$0.1822\pm0.0019$|$0.1668\pm0.0019$|$0.1559\pm0.0019$|
|NR-SMOCU-SGD|$0.1913\pm0.0017$|$0.1810\pm0.0018$|$0.1671\pm0.0017$|$0.1552\pm0.0017$|
**Running time for each iteration (in seconds):**
|               | n=10| n=20|n = 40|n=60|
| ----------- | ----------- |------|---|-------|
|random|$0.0\pm0.0$|$0.0\pm0.0$|$0.0\pm0.0$|$0.0\pm0.0$|
|MES|$0.7245\pm0.02$|$0.7174\pm0.02$|$0.7174\pm0.02$|$0.7243\pm0.02$|
|BALD|$1.384\pm0.04$|$1.384\pm0.04$|$1.346\pm0.04$|$1.322\pm0.03$|
|OR-MOCU|$57.29\pm1.60$|$64.47\pm1.85$|$78.07\pm2.01$|$116.2\pm3.19$|
|OR-SMOCU|$60.49\pm1.61$|$65.47\pm1.68$|$84.14\pm2.33$|$119.8\pm3.22$|
|ADF-MOCU|$16.8\pm0.45$|$17.9\pm0.49$|$20.5\pm0.59$|$25.27\pm0.72$|
|NR-MOCU-RO|$31.8\pm1.22$|$33.36\pm1.33$|$35.83\pm1.48$|$43.07\pm2.23$|
|NR-SMOCU-RO|$32.96\pm1.02$|$34.27\pm1.20$|$39.31\pm1.68$|$46.38\pm2.33$|
|NR-SMOCU-SGD|$31.84\pm1.08$|$33.4\pm1.23$|$35.81\pm1.54$|$43.21\pm2.46$|
Based on these new results, we can see clearly that OR-MOCU and OR-SMOCU perform worse than our proposed NR-MOCU-RO and NR-SMOCU-RO for both computation and sample efficiency. That's because when compared with IS, the naive Monte Carlo used in OR-MOCU/OR-SMOCU with GPC cannot accurately estimate the acquisition function values.
On the other hand, the running time of OR-MOCU/OR-SMOCU increases faster than NR-MOCU-RO/NR-SMOCU-RO since they retrain the model for every query samples, which requires $O(M_2 n^3)$ computation complexity. ADF-MOCU runs faster than our proposed algorithms, but it performs poorly as the ADF approximation is not accurate enough. Our NR-SMOCU-SGD again performs better than the competing algorithms as it utilizes the gradient information in the optimization procedure. We will add these additional results and provide more details of the experiments.
Dear reviewers, we appreciate your time and efforts providing critiques and constructive review comments. We believe that we have tried to address most of the raised concerns in our responses to all the reviewers. As after our rebuttal responses, we have not received any feedback from reviewer **6XPv** and only one statement from reviewer **anoP** added in the original review post, simply stating that ". there are some novelty in this submission, but I still think the contribution of this paper is not enough to clear the bar of NeurIPS," we would like to humbly ask the reviewers to share more detailed feedback to our point-by-point response.
We understand that this requires additional time and efforts but this period of the reviewing process is exactly for interactions with effective communication. We again believe that we have addressed most of the concerns and tried our best to add requested experimental results in our response to the reviewers. We would greatly appreciate your detailed reasons or constructive suggestions if more experiments or explanations would be needed for the reviewers to reconsider their scores.

Meta-review:
role: PC
data: comment: The reviewers acknowledge that this paper has the contribution that it makes acquisition function optimization in Gaussian Process Classification (GPC) more computationally tractable; however, they also expressed the following concerns in the discussion:
- this work is an improvement of a very new active learning method in a specific setting (GPC), and whether the new active learning method has high impact on the community is still unclear;
- lack of comprehensive comparison with the important baselines, such as original EER/MOCU/SMOCU methods (although the reviewers appreciate the additional experimental results on the checkboard problem in the rebuttal).

We encourage the authors to incorporate the reviewers' suggestions to strengthen the paper in the next iteration. 

Decision: Accept (Poster)