Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The manuscript addressed some detailed problems when applying existing strategies to pool-based active learning and query synthesis-based active learning. They use importance sampling to reduce the sample numbers, derive an analytic expression to calculate the prediction distribution, and employed SGD to optimize the query samples.
main_review: Originality:
The novelty of this manuscript is limited, which does not bring breakthroughs for active learning methods based on Gaussian processes. This manuscript did not propose any novel selection strategy (one of the cores in the field of active learning), nor did it make substantial improvements to the learner. They applied the existing selection strategies MOCU and SMOCU to pool-based and query synthesis-based active learning using some conventional technology, e.g., importance sampling and Adagrade to solve efficiency and gradient related problems.

Quality: 
In Introduction, it is described that the complexity of training GPC with EP approximation is O(n^3). As far as I know, since the sparse GP was proposed the complexity has been greatly reduced, even down to linear with training size when using a stochastic EP or variational auto-encoder. The authors never mentioned these works.

It’s good to use analytical inference to obtain the prediction distribution expression as introduced in Section 3.2, but with the increase of newly coming data, the model parameters had better be learned.

The discussion of the related work is insufficient missing some recent work, and the contrast experiments are crude which just compared with MES (Sebastiani et al., 2000) and BALD (Houlsby et al., 2011). The comparisons among NR-MOCU-RO, NR-SMOCU-RO, and NR-SMOCU-SGD did not contribute to the verification of the advantages of the proposed method, which just compared the performance of some existing technologies such as LogSumExp and Adagrad. The authors have mentioned that deep generative models show promising potential with related work, but why not compare with those novel deep models on higher-dimensional datasets.

Clarity:
This manuscript seems to be finished in a hurry, and a lot of writing is not standard. For example, in Eq. (5) and many other places, y and ys are confused.

Significance: 
The experiments are not convincing. In Table 1, the efficiency improvement is not significant. The results have not been thoroughly analyzed, e.g., why IS and IS+JDI has lower error rate than naïve method？What does the random optimization in NR-MOCU-RO mean?  Why does MES perform closely to their algorithms in the Vehicle dataset?


limitations_and_societal_impact: The authors should give the limitations and potential negative social impacts of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses active selection of queries via expected error reduction (EER) for Gaussian process classification. The authors describe Gaussian process classification and EER, and then summarize the MOCU acquisition function (equivalent to EER) and SMOCU acquisition function, which is an approximation to MOCU and therefore an approximation to EER; both MOCU and SMOCU are existing metrics in the literature. The main contribution of this work is an efficient method to estimate the MOCU and SMOCU metrics for pool-based active learning, and a gradient calculation for SMOCU, leading to SMOCU query synthesis (and hence, an approximation to EER query synthesis, which the authors claim to be the first attempt at query synthesis for an EER-based strategy). Specifically, the authors implement importance sampling, a calculation of critical probabilistic quantities that do not require expectation propagation (as would be required otherwise), and derive an acquisition function gradient calculation. The authors empirically evaluate the computational benefits of their methods, and test empirical generalization accuracy on real and synthetic datasets against several baselines.
main_review: **Originality**: The contributions made in this paper seem to be natural applications of existing active learning acquisition functions to the setting of Gaussian process classification. However, the means by which this is accomplished involved creative mathematical manipulations to make acquisition function optimization more computationally tractable. It is nontrivial to arrive at the appropriate insights needed to develop such approximations.

**Quality**: The introduction, background, and mathematical developments seem to be technically sound, and the illustrative examples presented were convincing. My main concerns involve the extent of the experiments performed. In my opinion, critical experimental results are missing. Specifically, I was surprised that a comparison was not made against an EER baseline in either the synthetic or real-world datasets, and additionally think that the computational times of each comparison method should be evaluated and presented. I appreciate the effort to benchmark computational performance in Table 1 to illustrate the tradeoff between estimation accuracy and computational budget, but I think such an analysis should also be present in the full active learning experiments.

My reason for these concerns is from the advertised main contribution of the paper, which from my understanding is to improve upon EER selection in a computationally efficient manner. To me, whichever method is developed should then be empirically tested against EER, and the tradeoff between generalization error and computational budget fully evaluated. It seems to me that EER could have been tested with expectation propagation being used in the acquisition function, even if this would require a large computational budget. I welcome the authors to comment about these concerns about lack of comparison against EER (at least in the pool-based experiments) and lack of computational expense results (across all experiments).

The authors comment in line 268 “Note that in all three algorithms, inferring the predictive distribution for each xs also takes a considerable part of computations, especially when n is small.” The results in Table 1 could be strengthened by explicitly separating out this computational cost in the analysis.

I also felt some aspects of the discussion were lacking. It doesn’t seem that the performance of BALD was discussed in Section 4.2. This seems important since BALD seems to perform just as well as the proposed methods.

As I illustrate below in “Details”, I am concerned about the lack of detail in how the experiments were performed, which could make reproducibility harder.

**Clarity**: The paper was mostly clear. The inclusion of Figure 1 as an illustrative example was effective. The authors did a good job overall of describing Gaussian process classification and their solution approaches. However, I think insufficient experimental details are included, which could make reproducibility more difficult. I’ve detailed these in “Details” below.

**Significance**: The problem of efficient EER-based active learning strategies for pool-based or query synthesis Gaussian process classification is a solid technical contribution in the realm of Gaussian process classification active learning. Except for my concerns about the experiments above, I believe this work constitutes a significant advancement of efficient active learning for Gaussian process classification.

**Details**: 
- line 158: why is the gradient impossible to compute during EP? This should be elaborated on in detail, especially since as advertised in line 163, it seems from the writing that this gradient is in fact derived. I found these two statements confusing, and it seems that the points made here should be understandable and clear since this discussion constitutes the motivation for Section 3.3. Related to this, I am confused by this comment: “Without the need for EP approximation, we can further derive the chain rule to calculate the gradient of the SMOCU reduction, which provides us the first efficient gradient-based query synthesis active learning algorithm.” My understanding from the paper is that it is the smoothness of SMOCU that enables the use of a gradient, not the avoidance of EP.
- what is the choice of p(x) in the experiments
- I found it confusing to use the terminology absolute error ratio instead of “relative error”. It left me wondering if “absolute error rate” used in Table 1 was a normalized version or not
- in line 260, how were the candidate values of $x_*$ sampled or selected?
- I think the acquisition functions for MES and BALD should be included at least in the supplement
- I was very confused by the description of the injected bit flip error (description starting 282). Is this a bit flip of a noiseless label, or an additional bit flip on a label already sampled according to the probit function?
in line 300, using only 2 labeled points seems very low to me to estimate hyperparameters. Can the authors comment about this?
- what is the test/train split for the UCI datasets? Are these splits randomized across runs, or fixed a priori for all runs?
- in line 307: “When checking the converging GPC model, the classification boundary is relatively simple on the the Vehicle dataset with relatively low-dimensional feature space (d = 18)” How was this ‘simple boundary’ checked?

**Minor points**:
- I was very confused by the use of the notation g, starting in line 144. This should be defined explicitly, instead of implicitly as was done in line 145
- line 151: $M_1$ is undefined
- overall, I found the paragraph between lines 143-152 a bit difficult to follow and understand
- line 221: relable -> reliable
- line 242: form -> from
- line 248: what is the max taken over? I assume $y_s$
- does the performance metric in line 281 refer to equation (3)? If so, this should be referenced
- supplement line 15: strait-forward -> straightforward
- I think g(...) should be defined explicitly in the presentation of Algorithm 1

**Overall**: Overall I enjoyed reading the paper. If my concerns above were addressed --- in particular my concerns about the extent of the experiments --- I would consider increasing my score.
limitations_and_societal_impact: The authors mostly addressed the limitations and societal impact of their work. I do think that the following comment should be elaborated on “Active learning algorithms in this work are derived for GPC using probit likelihood, which may have its inherent limitation in high-dimensional feature space.” Dimensionality of the space is an important factor, and I think the paper would be strengthened by additional consideration to the problem dimensionality. This could be accomplished through additional synthetic experiments evaluating performance as a function of dimension, or a discussion of why specifically dimensionality should change aspects of the solution or computational cost. Similarly, how do these results scale with larger values of n? Stopping after on the order of 50-100 queries seems low to me.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This study gives an efficient solution for Bayesian active learning based on Estimated Error Reduction (EER) with the Gaussian process classification model. Specifically, the recently proposed MOCU-based Bayesian active learning for EER is used for representing the joint distribution of queries as a one-dimensional integral and avoid relearning the GPC every time when computing the acquisition function. Thanks to the smoothness of the SMOCU-based acquisition, not only the pool-based but also a query synthesis active learning is efficiently implemented.
main_review: I acknowledge the importance of this work, which uses the recently proposed MOCU and SMOCU to improve the efficiency of Bayesian active learning based on GPC.
The formulation that attributes the simultaneous posterior distribution of labels to a one-dimensional integral seems to be reasonable, and the fact that membership synthesis is possible is also commendable.
There is no novelty in speeding up GP(C) with Gaussian approximation, marginalization, and importance sampling, and the main novelty of this research is the use of SMOCU to make the acquisition function differentiable and to enable membership synthesis.

In my opinion, although I acknowledge the early adaptation of recently developed technology has its own importance to some degree, the contribution of the paper is not so large, particularly without any theoretical evaluation of the estimation accuracy of p(y|x_s,x_*,y_*) by the proposed method.

Missing reference:
Active learning is already successfully applied to phase diagram identification problem. I recommend citing the following article:
"Efficient construction method for phase diagrams using uncertainty sampling", K. Terayama et al., Phys. Rev. Materials 3, 2019.


--- After Reading authors' response
I acknowledge that there are some novelty in this submission, but I still think the contribution of this paper is not enough to clear the bar of NeurIPS.

limitations_and_societal_impact: Yes, the applicability issue to high-dimensional datasets is acknowledged.
review_title: null

Meta-review: The reviewers acknowledge that this paper has the contribution that it makes acquisition function optimization in Gaussian Process Classification (GPC) more computationally tractable; however, they also expressed the following concerns in the discussion:
- this work is an improvement of a very new active learning method in a specific setting (GPC), and whether the new active learning method has high impact on the community is still unclear;
- lack of comprehensive comparison with the important baselines, such as original EER/MOCU/SMOCU methods (although the reviewers appreciate the additional experimental results on the checkboard problem in the rebuttal).

We encourage the authors to incorporate the reviewers' suggestions to strengthen the paper in the next iteration. 
Decision: Accept (Poster)