Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel active learning method - called BEMPS (Bayesian Estimate of Mean Proper Scores) - in the context of text classification.
Based on the ELR (expected loss reduction) approach, the authors propose to estimate the increase in the so-called "strictly proper scores"  such as the log probability or negative mean square error.
They show that the proposed method, BEMPS, makes the classifier asymptotically converge to the optimal classifier as the active learning process continues.
For proof, it builds upon techniques that were recently utilized for proving the convergence of the active learning scheme based on weighted MOCU (mean objective cost of uncertainty).
Based on extensive evaluations on various benchmarks, this study shows that BEMPS generally outperforms other state-of-the-art active learning methods.


main_review: GENERAL COMMENTS

Overall, this is an interesting and well-written paper that improves the current state-of-the-art in active learning (AL) - especially in the context of text classification.
The paper presents strong performance evaluation results that clearly demonstrate the advantages of the proposed method, BEMPS, while also presenting interesting convergence proofs of the proposed active learner.
Although the convergence proofs are based on techniques that were recently used to show the convergence of the WMOCU (weighted MOCU)-based active learning scheme, the authors show that the proposed approach often yields favorable learning outcomes compared to the WMOCU-based active learning scheme as well as several other AL algorithms that were tested.


DETAILED COMMENTS

1. What is the main impact of changing the ELR scheme to utilize proper scoring rules or the Bregman divergence instead of the classification error?
Considering that the main goal of the active learning process is to enhance the performance of the classifier (therefore reducing its classification accuracy), would this change lead to any "semantic" differences of the learning process compared to other ELR methods?
Although convergence to the optimal classifier is shown in the paper, what would be the main benefits of changing the scoring scheme?
Providing further insights would be useful.
Especially, considering that the original WMOCU-based scheme, to which BEMPS is compared, takes an "objective-based" approach, which focuses on the expected reduction of the classification error, it would be meaningful to compare BEMPS with WMOCU from this perspective.

2. Based on the presentation in Sec. 3.1., what is the relation between  BEMPS and WMOCU?
Does the WMOCU scheme become a special case of BEMPS (or vice versa) under certain conditions?

3. What do"fully conditional model" and "full conditionality" mean?
Please provide a clear definition.

4. It is unclear how MOCU was utilized in the active learning framework.
For example, in order to evaluate the expected gain of a new data point (i.e., label) acquisition, expected conditional MOCU given the new data should be evaluated.
But as the current paper doesn't provide the relevant details, it is not clear how MOCU was utilized.

5. Further discussion is needed regarding the relationship between the batch size and  the overall performance (e.g., lines 266-267)
For example, why would the no-batch case perform worse compared to using small batches?
And would a larger batch size lead to better performance as the number of acquired labels increase outperforming small batch cases? (Although not initially, when the number of newly acquired data points is small?)

6. Please carefully proofread the manuscript and the supplementary material to correct any typos/errors.
- a prior -> a priori (line 110)
- equation # missing in the supplementary material (line 68)

-----------

Note:

I acknowledge that the authors' response to the original review comments has been reviewed and considered in the rating.


limitations_and_societal_impact: Some limitations of the proposed method are discussed in the conclusion.
Potential negative societal impact is not discussed in detail.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper analyzes possible acquisition functions for active learning in the field of text classification. The authors introduce BEMPS (Bayesian Estimate of Mean Proper Scores), which leverages the existing Expected Loss Reduction method to estimate the increase in scores like log probability and negative mean square.
main_review: The authors propose a novel approach to an important problem: increasing the diversity of the examples selected for labeling during active learning. The paper is reasonable well written, though the prose is awkward at times (e.g., lines 33 and 44-48) and makes a compelling case for the proposed approach. The contributions appear to be original. 

One way to improve the paper's readability would be to add an intuitive running example before you get to Section 3, which is quite dry and dense on the allotted two pages. You can probably do this in less than one page, based on on figure and a few paragraphs. Illustrating your three main contributions (lines 35-48) in an intuitive manner will dramatically improve the paper's readability for a general audience. You are trying to do this in section 3.2 in a more formal manner.

 In Fig 1:
- is the dashed line (in the image on the left) the performance if training on the entire dataset?
- could you please discuss why RAND outperforms BALD & MOCU  (and ties with Max Ent) in this scenario?

OTHER COMMENTS:
- given that the work on WAAL, VAAL, and MAL is on image datasets, you do not have to make three times the argument on why you are not comparing with them (one time is enough :)
- however, the idea of combining active & semi-supervised learning is not new - see [McCallum & Nigam, 1998] or [Muslea at at, 2002]
- line 32: newer --> recent
- line 44:  wth --> with             (please spell-check!)

McCallum, A., & Nigam, K. (1998). Employing EM in pool-based active learning for text classification. In ICML (pp. 359–367)

Muslea, I., Minton, S., & Knoblock, C. A. (2002, July). Active+ semi-supervised learning= robust multi-view learning. In ICML (Vol. 2, pp. 435-442).
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the problem of active learning by querying examples that will lead to the most reduction in classification
error, which is a fairly common idea. It proposes to place this in the
context of strictly proper scoring rules in order to derive new
example acquisition functions and generalize existing
ones. Experiments on multiple datasets illustrate this for two common
scoring rules, with extensive analysis, showing that the proposed
methods perform well.

main_review: Plus:
+ clean framework with some theoretical guarantees
+ experimental comparison investigates many aspects of the method
+ consistent performance improvement over alternatives

Minus:
- semi-supervised learning is specifically excluded
- novelty seems limited
- many moving parts 

The first part of the paper is a quick (and steep) intro of the theory behind the method. It is very formal and could benefit from more actual examples of instantiations of the concepts early on, but this is a matter of taste. This is followed by a sizeable experimental section which covers a number of questions and analyzes the impact of various aspects of the method (batch size and diversity, dynamic validation set). The results are convincing, with learning curves (with error bars) and statistical testing of improved performance.

The paper seemed technically correct, building upon previous work in machine (active) learning and statistics to place the method within a clean theoretical framework using strictly proper scoring rules and extending some existing methods in the process. Some of the contributions seem modest, for example the "dynamic validation" is essentially randomly splitting training and validation sets during ensemble training -- producing ensembles by samping train/val set has been used extensively and it is unclear whether there is something clearly novel in the particular way it is done here. There are many moving parts and hyper-parameters in the method. Some of them are tested in the ablation study in Sec. 4.2. Results suggest that the proposed method functions well, but for a fairly specific choice of these settings. One may wonder how these settings would generalize to other domains or data sets, for example with much larger datasets.

The main limitation of the paper imho is that it specifically excludes semi-supervised learning from consideration and comparisons (the "fully conditional" assumption). SSL and AL go hand-in-hand since at least co-training. Even though the nice theoretical framework does not accomodate SSL, it is hard not to at least compare with real state-of-the-art in AL.
[**EDIT** The authors' point is fair that one has to start somewhere]

Clarifications:

Is the 'running time per single acquisition' normalized for batch size, or for one batch (i.e. one run of Alg.3)? I would guess the latter, but 'single acquisition' introduces a bit of ambiguity.

It could help to place the sample sizes l^k_i,j,d on Fig 1 (left).

Minor points:

The pairwise comparison matrix is a poster case for correcting for multiple comparisons.

In the conclusion BEMPS seems to turn into BERPS.

Typo: generated (l.92)

**EDIT** Thanks for the detailed reply. Review is updated accordingly.
limitations_and_societal_impact: Yes this is mentioned in the checklist, although not really relevant for this paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes new acquisition functions for Bayesian active learning, which replaces the classification errors in MOCU with strictly proper scoring rules. The paper proves the convergence of the new acquisition functions and demonstrates the AL performance with text classification benchmark datasets. 
main_review:  The paper proposes a class of acquisition functions based on strictly proper scoring rules, which are guaranteed to converge to the true model. The batch AL algorithm is concise and general. The experiments do demonstrate the effectiveness of the proposed methods.
limitations_and_societal_impact: I think the paper lacks analysis and discussions of the proposed methods. The paper do prove the convergence of the acquisition functions based on strictly proper scoring rules, but there are no further discussions on the sampling efficiency, and the guideline for choosing the strictly proper scoring rules (or the G functions). Although the paper provides sufficient experiments, the paper doesn't provide enough discussions on the experiment results. 

I personally want to see the discussions on questions such as: since BALD/WMOCU are guaranteed to converge, why we still need to propose new methods? Why do we choose logarithmic scoring rule and squared error scoring rule? And based on the properties of the different scoring rules, in what conditions (large or small uncertainty, different sampling budgets) they can be more efficient than other competing methods?
review_title: null

Meta-review: The paper develops a new acquisition function for active learning, which is referred to as Bayesian Estimate of Mean Proper Scores or BEMPS for short. BEMPS generalizes weighted Mean Objective Cost of Uncertainty or WMOCU, a recently developed active learning acquisition function based on expected loss reduction (ELR) by using the strictly proper scoring rules to replace the classification errors. This generalization offers a more natural and principled way to avoid the acquisition function from getting stuck before reaching the optimal classifier, while also ensuring a broader applicability where errors are less informative. 

All the reviewers agree that this is a well-written paper, which tackles an important and interesting problem. They also acknowledge the theoretical contributions, including the proof of convergence to the true model parameter (but guarantee on sample efficiency is not provided, which is important for active learning). Evaluation on text classification tasks shows convincing results as compared with some recent baselines. 

Most questions raised by reviewers have been adequately addressed by the authors during the discussion phase. Given the similarity between the proposed BEMPS and existing ELR based acquisition functions (some of which are also proved to converge to the optimal classifier), it is important for the authors to include some additional insights on the advantage of using the strictly proper scoring rules, for example, on what specific conditions that BEMPS can bring additional benefit to the active sampling process, in the final version of the paper. 

Decision: Accept (Poster)