Diversity Enhanced Active Learning with Strictly Proper Scoring Rules
Authors: Wei Tan
Lan Du
Wray Buntine
Venue: NeurIPS 2021 Poster (2021)
Keywords: active learning
machine learning
deep active learning
acquisition function
uncertainty based method
diversity base method

Abstract: We study acquisition functions for active learning (AL) for text classification. The Expected Loss Reduction (ELR) method focuses on a Bayesian estimate of the reduction in classification error, recently updated with Mean Objective Cost of Uncertainty (MOCU).  We convert the ELR framework to estimate the increase in (strictly proper) scores like log probability or negative mean square error, which we call Bayesian Estimate of Mean Proper Scores (BEMPS). We also prove convergence results borrowing techniques used with MOCU. In order to allow better experimentation with the new acquisition functions,  we develop a complementary batch AL algorithm, which encourages diversity in the vector of expected changes in scores for unlabelled data. To allow high performance text classifiers, we combine ensembling and dynamic validation set construction on pretrained language models.  Extensive experimental evaluation then explores how these different acquisition functions perform. The results show that the use of mean square error and log probability with BEMPS yields robust acquisition functions, which consistently outperform the others tested.

Diversity Enhanced Active Learning with Strictly Proper Scoring Rules
Wei Tan Monash University wei, tan2@monash. edu
Lan Du Monash University lan. du@monash edu
Wray Buntine Monash University
wray. bennybbunninemonnaee edu
Abstract
We study acquisition functions for active learning (AL) for text classification. The Expected Loss Reduction (ELR) method focuses on a Bayesian estimate of the reduction in classification error, recently updated with Mean Objective Cost of Uncertainty (MOCU). We convert the ELR framework to estimate the increase in (strictly proper) scores like log probability or negative mean square error, which we call Bayesian Estimate of Mean Proper Scores (BEMPS). We also prove convergence results borrowing techniques used with MOCU. In order allow better experimentation with the acquisition functions, complementary batch AL algorithm, which encourages diversity in the vector expected changes in scores for unlabelled data. To allow high performance text classifiers, we combine ensembling and dynamic validation set construction on pretrained language models. Extensive experimental evaluation then explores how these different acquisition functions perform. The results show that the use of mean square error and log probability with BEMPS yields robust acquísition functions, which consistently outperform the others tested.
develop
Introduction
Classification has extensive uses and deep learning has substantially improved its performance, but a major hurdle for its use is the paucity of labelled or annotated data. The data labelling process performed by domain experts is expensive and tedious produce, especially in the medical field, where due to lack of expertise and privacy issues, annotation costs are time-consuming and expensive [Л Active Learning (AL) is an approach to speeding up learning by judiciously selecting data be annotated [27]. AL is perhaps the simplest of all numan-in-the-loon learning approaches, because the only interaction is the (expert) human providing a class label, yet for even in the simplest of tasks, classification, general theory of AL is not agreed on.
Moreover, in practical situations, with deep learning: the expense to retrain deep neural networks and the use of validation data for training (11, transformer language models (16), batch mode AL with diversity [22], and consideration of expert capabilities and costs 8 43) Also important in experimental work is the need for realistic labelling sizes, with practitioners work with saying expert annotation may allow budget of up 1000 data points, rarely more. Using large batch sizes (c.g., 1000 in i 20) can thus be impractical.
needs to consider many issues when designing an AL system
Our fundamental research contribution is to suggest what makes good acquisition function for the uncertainty component, without any batching. While there are many recent methods looking at the uncertainty diversity trade-off for batch AL 22,, few recently have focused understanding the uncertainty side alone. A substantial advance is a recent theoretical framework, Mean Objective
"Corresponding author
Our implementation of BEMPS can be downloaded from htttt//://gggct...../////r/ttttt
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Cost of Uncertainty (MOCU) 442 that provides a convergence proof. As our first contribution, we convert their expected loss reduction framework to an AL model using strictly proper scoring rules or Bregman divergences p instead of classification errors. MOCU required manipulations
of the expected error, resulting in weighted-MOCU (WMOCU), in order to achieve convergence and avoid getting stuck in error bands. Strictly proper scoring rules naturally avoid this problem by generalising expected errors to expected scores. Using strictly proper scoring rules means better calibrated classifiers are rewarded. The scoring rules go beyond simple minimum errors of WMOCU and can be adapted to different kinds of inference tasks (e.g., different utilities, precision-recall trade-offs, etc.). This property is preferable and beneficial for applications such medical domains where actual errors become less relevant for an inference task.
In order to evaluate the new acquisition functions we use text classification, which is our target application domain. For realistic evaluation, we want to use near state of the art systems, which means using pretrained language models with validation sets (24), and neural network ensembles [3] Ensembling also doubles as heuristic technique to yield estimates of model uncertainty and posterior probabilities. Coming up with a simple approach to combine ensembling and validations sets is our second research contribution. The importance of these combinations for AL has been noted [40] 25). For further batch comparisons, we then bring back diversity into the research, suggesting a way to naturally complement our new family of acquisition functions with a method to achieve diversity, our third research contribution. Extensive experiments with comprehensive set of ablation studies on four text classification datasets show that our BEMPS -bseed AL model consistently outperforms recent techniques like WMOCU and BADGE, although explicitly exclude recent semi-supervvsed AL methods because they represent an unfair comparison against strictly supervised learning.
2 Related work
The proposed BEMPS, general Bayesian model for acquisition functions, quantifies the model uncertainty using the theory of (strictly) proper scoring rules for categorical variables [9]. Thus, we first review existing acquisition functions (aka query strategies) proposed in some recent AL models that are most related to ours, and refer interested readers to [22] for a comprehensive discussion of recent strategies. One common and simple heuristic often used is maximum entropy 45 34 77, where one chooses samples that maximise the predictive entropy. Bayesian AL by disagreement (BALD) 10 and its batch version (i.e., BatchBALD) [12] instead compute the mutual information between the model predictions and the model parameters, which indeed chooses samples that maximise the decrease in expected entropy (20. Recently, inspired by the one-step-look look-ahead strategy of ELR 223,, Zhao et al. [42] extended MOCU [39 to WMOCU with a theoretical guarantee of convergence. But WMOCU only applies errors which are less relevant for some inference tasks. BEMPS naturally extends those methods, focusing on Bayesian estimation of uncertainty related to the model's expected proper score.
Ensemble methods are used to obtain better uncertainty estimates with deep learning, including Monte- Carlo dropout (MC-dropout) 12 21 and deep ensembles [33. More sophisticated techniques being developed, for instance, MCMC, hybrid and deterministic approaches [36 2). However, plain ensembling remains competitive and simple method for deep learning (2).
Query strategies considering just uncertainty do not always work well in batch setting, due to the highly similar samples acquired in batch [35] 22 17] To overcome this problem, there have been many AL methods that achieve batch diversity by acquiring samples that are both informative and diverse, such as [19 38 14 1 40 43 29]. BADGE ( and ALPS [40 are the two recent AL methods focusing on batch diversity. BADGE uses gradient embeddings of unlabelled samples as inputs of K-MEANS++ to select a set of diverse samples, which relies on fine-tuning pretrained language models. Instead of using gradient embeddings, ALPS uses surprisal embeddings computed from the predictive word probabilities generated by masked language model for cold-start AL. Whereas our BEMPS computes for each unlabelled sample a vector of expected change in the proper scores, relating directly to the performance.
use semi-supervised learning, for instance WAAL's models are trained on the feature space obtained with the help of the unlabelled data. We exclude these methods from our comparison since they are semi-supervised learning algorithms. Both Core-set (26) and WAAL [30] also set up sophisticated non-Bayesian cost functions using
Other recent batch AL approaches, WAAL 30 and VAAL 32

---

bounds to deal with the many unknown probabilities when trying to optimise for batch, whereas the Bayesian formula of BEMPS can be directly estimated.
In experimental work, the use of a validation set to train deep learning models in active learning is not uncommon, the goal of which is to avoid over-fitting and achieve early-stopping Some existing methods assume that there is a large validation set available a prior [2] /zi. but means the cost of labelling the set not factored into the labelling budget. We argue that the availability of separate validation set is impractical in real world AL scenarios. Although Yuan et al. 40] use fixed epochs to train the classifier without a validation set to save the cost, the classifier could either be under- -it or over-fit. We instead use a dynamic approach to generate alternative validation sets from the ever increasing labelled pool after each iteration.
3 Bayesian estimate of mean proper scores
We first review the general Bayesian model for acquisition functions that includes ELR, MOCU and BALD. We then develop Bayesian Estimate of Mcan Proper Scores (BEMPS), a new uncertainty quantification framework with theoretical foundation based on strictly proper scoring rules ( It naturally extends ELR and BALD, focusing on the Bayesian estimation of uncertainty related to the models expected performance.
Suppose models of our interest are parameterised by parameters E O, L indicates labelled data, probability of label y for data x is given by Pr(y 2), and Pr(- 0,2z) presents a vector of label probabilities. With a fully conditional model, the posterior of 0 is unaffected by unlabelled data, which means Pr(0) L,U) = Pr(O | L) for any unlabelled data U. Moreover, we assume without loss of generality that this model family is well behaved in a statistical sense, so the model is identifiable. The "cost" of the posterior Pr(01 L) can be measured by some functional (Prr(( L)), denoted Q(L) for short, where Q(L)= 2 0 and QL) 0 when some convergence objective has been achieved. For our model this is when Pr(0 L)) has converged to a point mass at single model. A suitable
objective function is to measure the expected decrease in Q--) due to acquiring the label for data point z. The corresponding acquisition function for AL is formulated (TO] Equation (1)) as
AQ(= Q(L) Z..) [Q(LU U {(x, v)3]]
whereas for ELR the expression is split over an inequality sign 23 Equation (2)). It estimates how much the cost is expected to reduce when a new data point x is acquired. Since the true label for the new data x is unknown a prior, we have to use expected posterior proportions from our model, Pr(y z) to estimate the likely label. For BALD [TO] using Shannon's 's entropy, QL( = ((r(( I.)), which measures uncertainty in the parameter space and thus has no strong relationship to actual errors 42) MOCU and ELR use Bayesian regret given by the expected loss difference between the optimal Bayesian classifier and the optimal classifier:
Qmocu(T) Epr(w)) min(1 Pr(y/ [,,')) - EPr(O))) min(1 Pr(y/ | 10, x'))
(2)
WMOCU uses weighting function defined by Eq (11) in 42 to have more amenable definition of AQ(x L) than the MOCU method. Although WMOCU guarantees AQ(=[ converging to the optimal classifier (under minimum errors) according the QLL) with the strictly concave function by Eq (15) in [42], the optimal approximation of the convergences can only be solved by controlling hyperparameter of the weighting function manually. To allow theoretical guarantees of convergence under more general loss functions, we propose a different definition for (L)) based on strictly proper scoring rules.
3.1 Strietly Proper Scoring Rules for Active Learning
A scoring rule assesses the quality of probabilistic prediction of categorical variables, and is often used in training a classification algorithm. For a model Pr(y 0, x)) with input data x and if one observes label y. the score is given by function S(Pr(- 0, v). A strictly proper scoring rule has the behaviour that in the limit of infinite labelled data T.p, as n -> oo, the average score Ex.wee, S(Pr(- 0 2), y) has unique maximum for 0 at the "truc" model (for our identifiable
3

---

model family). The Savage representation [9] states that strictly proper scoring rule for categorical variables takes the form S(q(.). v) G(q(-)) (6) - --))
for a strictly convex function G(.) with subgradient d.)... Note that the expectation of scoring rule according to the supplied probability takes simple form Eq(n) [S(g(-),)) v) = E, ())) y) = G(g(-)).
With strictly proper scoring rules, we develop a generalised class of acquisition functions built using the posterior (i.e., w.r.t. Pr(O) | L)) expected difference between the score for the Bayes optimal classifier and the score for the "true" model. This inherently Bayesian due to the use of Pr(O L).
(s(L) = P((1L) (Pecvle [S(Pr(- 0)))).) S(Pr(. [,
(3)
(P((2))((01))) B(Pr(- L,,),, [,,))
(4)
Epi(2)
(P)(1L)) [G(Pr(- [,,)]] G(Pr(. [,,)
(5)
AQs(r[L)
Epi(2) (PrtylE.s)) [((P((- | L, (1, (), x()) G(Pr( I,,"'))
(6)
The (L) has three equivalent variations, one for an arbitrary strictly proper scoring rule (()()) (Eq (3), one for a corresponding Bregman divergence --)-)) (Eq and the third for an arbitrary
strictly convex function G(.) (Eq (5) Their connections are given in
It is noteworthy that the acquisition function [2)( defined in Eq (6) is in a general form, applicable to any strictly proper scoring function for categorical variables. For instance. using a logarithmic scoring rule, popular for deep neural networks, we have (tog((()) v) = log (yy) and Glog(g (g))) (c)))) Using the squared error scoring rule, known as a Brier score, we have
SMSE(1(-).)) -g (a())-1)=)) b) 2
and G = E,, a(v)² 1. Combining these G(-)
66) yields two acquisition functions for the different scoring rules. These as well as the corresponding for BALD have some elegant properties, with proof' given in Appendix A. Lemma (Properties of scoring). In the context of a fully conditional classification model Pr(y| 0, 2), the (T.). Qs( [Q)(x)T), (Qs(z]LT defined above are all non-negative.
functions with Equation
Moreover, they guarantee learning will converge to the "truth".
Theorem 1 (Convergence of active learning). We have fully conditional classification model Pr(y 8 x). for 8 E θ with finite discrete classes y and input features x. Moreover, there is a unique "true" model parameter 0, with which the data is generated, the prior distribution p(0) satisfies p(0,)= > O, and the model is identifiabl. Then for the Al. defined by the acquisition functions defined above of (2x or (Qs(x)]T after being applied for n steps gives labelled data Len then 1im [Q I,,) = O and likewise for (s(.). Moreover, lim,- Pr(O I,) is a delta function at f = 0, for data acquired by both AQr L)) or (1)(x
Finiteness and discreteness æ is used to adapt results from 42 to show for all x that [Q( + 0 - 0o, not issue since real data is always finite. Interestingly [Qr(=)L), i.c., BALD, achieves convergence too, which occurs because the model is identifiable and fully conditional, during AL we free choose values that would distinguish different parameter values 8. Full conditionality also supports BEMPS because it means any inherent bias in the AL selection is nullified with the use of the data distribution Pr(z). But it also means that the theory has not been shown to hold for semi-supervised learning algorithms, where full conditionality does not apply.
Compared with BALD, MOCU and WMOCU, the advantage of using strictly proper scoring rules in BEMPS is that they generalise expected errors to expected scores, which can be tailored for different inference tasks. BALD uses mutual information to score samples based on how their labels could inform the true model parameter distribution, which will be problematic if the uncertainty of model parameters has reduced relationship to the classification performance. This is reflected by its poor performance in our experiments. MOCU however has convergence issues as ELR, as pointed out by Zhao et al. 42 Even though WMOCU can overcome the convergence issues, it is limited to the minimisation of expected errors.
In contrast, measuring the quality of predictive distributions via rewarding calibrated predictive distributions, scoring rules are in favour of adaptability. For instance, scoring rules can be developed [9] for some different inference tasks, including Brier score, logarithmic score, etc. Many loss

---

Algorithm 1 Estimating point- wise ( x²) with Equation (6)
Algorithm 2 Estimate of argmax (z
Require: unlabelled pool U, estimation pool X 1: for 3 E do Q = 0
Require: unlabelled data point existing lahelled data I. estimation point x'
2:
3:
for x' X X do
Require: model/nctwork ensemble @ -(01 .. 8=} built from labelled data L.
4:
Q= +- =Q[[),,))
5: return arganax Q=
Require: strictly convex function G() taking as input
a probability density over
Algorithm 3 Finding diverse batch
1: Q =0
Require: unlabelled pool U, batch size B Require: estimation pool X, top fraction T
2: qu(()= Eeeg Pr(0 L)) 0,
1: Vreu =0
3: for y
2: for x U X do
4:
(()) = Exsen Pr(01 L, (I) Pr(- 0,z²)
3: Qr Q= DeC2.' AQ(=)L,²))
5: Q += (z(v)(((()))
4: V + +topk((, (UT)
5: batch =0
6: g() = Eoeg Pr(0 Pr 00,z)
6: centroids k-Means centers (VeCZEV, B)
7: G(q(-))
7: for E centroids do
8: return Q
8: batch U= {argmin |cc vec= ||}
9: return batch
functions used by neural networks, like cross-entropy loss, are indeed proper scoring rules In most cases we can create a particular model to match just about any Bregman divergence (e.g.. minimum squared errors is a Gaussian). In practice we can use robust models (e.g.. Dirichlet- multinomial rather than a multinomial, a negative binomial rather than a Poisson, Cauchy rather than Gaussian) in our log probability. For the particular cases we used, minimising Brier score gets the probability right in a least squares sense, i.e., minimising the squared error between the predictive probability and the one-shot label representation, which pays less attention to very low probability events. Meanwhile, log probability gets the probability scales right, paying attention to all events.
3.2 Acquisition Algorithms with Enhanced Batch Diversity
Algorithm 2 gives an implementation of BEMPS for an arbitrary strictly convex function G(-). returning the data point with the best estimated measure. To work with a Bregman divergence or proper score, the corresponding strictly convex function G(-) should first be derived. When G(-) is
negative entropy, we call this Corel Log and when G(-) is the sum of squares we call this CoreMSE, corresponding to the log probability or Brier scoring rules respectively. Algorithm [calls Algorithm 1 to get the estimation at test point æ', which implements the function inside Epi(2") [-] in Equation (6)
Note Pr(0) L, (Ir, y)) is computed from Pr(@ L) using Bayes theorem. Both Algorithms 2 and use fixed estimation pool, X, a fixed random subset of the initial unlabelled data used to estimate expected values Epr(z') [.]. Algorithm 3 returns B data points representing a batch with enhanced diversity: first calls Algorithm|| to get, for each data point x the unlabelled pool, the vector of expected changes in score values over the estimation pool. Thus, this vector conveys information about uncertainty directly related to the change in score due to the addition of x. While the gradient embedding used in [ represents a sample's 's impact on the model, our vector represents sample's 's impact on the mean proper score. Concurrently Algorithm 3 computes the estimate of (Q L) for these same xS. The top 7% of scoring data are then clustered with k-Means and representative of each cluster closest to the cluster mean is returned. This k-Means selection process tends to generate diverse batch of high-scoring samples. The intuition is that 1) only higher scoring data x should appear in a batch; 2) those clusters capture the pattern of expected changes in score values deduced by samples in the unlabelled pool, where the samples with a similar mean change in score values are grouped together; 3) samples in the same cluster can affect the learning similarly, so should not co-occur in batch.
4 Experiments
We demonstrate the efficacy of BEMPS via evaluating the performance of CoreMSE and Corel.gg as its two examples on various benchmark datasets for text classification. These two acquisition functions were compared with recent techniques for AL.
5

---

Table
Datasets and the used language model
Dataset
Unlabelled/Test sizes
# Classes
Lang. Model
Initial labelled size
AG NEWS
120,000 7,660
4
DistiBBERT
26
PUBMED 20K RCT
15,000 / 2,500
5
DistiBBERT
26
IMDB SST-5
25,000 / 25,000
2
DistilBERT
26
8544 2210
5
DistiBBERT
Table 2: The average running time (in seconds) per single acquisition iteration with unlabelled pools of varied sizes, which were generated from AG NEWS.
Unlabelled
CoreMSE
CoreLog
Max-Ent
BALD
MOCU
WMOCU
BADGE
ALPS
Rand
10k
65
65
61
61
65
65
76
62
<1
25k
163
163
159
159
163
163
323
283
<1
50k
326
326
322
322
326
327
884
573
<1
100k
654
6554
649
650
658
659
2914
1299
<1
Datasets We four benchmark text datasets for three different classification tasks: topic classifi- cation, sentence classification, and sentiment analysis, as shown in Table The AG NEWS for topic classification contains 120K texts of four balanced classes [41]. The PUBMED 20k used for sentence classification 3] which contains about 20K medical abstracts with five categories. This dataset is imbalanced. For sentiment analysis, we used both the SST-5 and the IMDB datasets. SST-5 contains 1 11K sentences extracted from movie reviews with five imbalanced sentiment labels [33] and IMDB contains 50K movie reviews with balanced classes [8]
Baselines We considered Max-Entropy [37], BALD [TO. MOCU [42), WMOCU [42). BADGE and ALPS [40] together with random baseline. Max- Max-Entropy and BALD are earlier uncertainty- based methods. MOCU, WMOCU, BADGE and ALPS are the four recent AL methods. Following their originally published algorithms, we re-implemented them for text classification tasks, with the same backbone classifier for fair comparisons. Note that the hypothetical labels used in BADGE were computed with ensembles.
Experiment setup We used a small and fast pretrained language model, DistilBERT 24 as the backbone classifier in our experiments. We fine-tuned DistilBERT on each dataset after each AL iteration with a random re-initialization (5), proven to improve the model performance over the use of incremental fine-tuning with the newly acquired samples 771. The maximum sequence length was set to 128, and a maximum of 30 epochs was used fine-tuning DistilBERT with early stopping 4.. We used AdamW (15) as the optimizer with learning rate 2e-5 and betas 0.9/0.999. Mcanwhile, the initial training and validation split contain only 20 and 6 samples respectively.
Each AL method was run for five times with different random number seeds on each dataset. The batch size B was set to (1, 5, 10, 50, 100). To compute the predictive distributions Pr(- L, and Pr(- L, (I, y), 2'), we borrowed the idea of deep ensembles DistiBBRRSs with randomly generated aain-validation splits (i.e., aaain/vaidatioon=0/33399 based on incrementally augmented labelled pool. Thus each member of the ensemble has different mein/validation set, helping to diversify the ensemble. We call this split process dynamic validation set (aka Dynamic VS), With it, our implementation of CoreMSE and CoreLog does not rely separate large validation set. More detailed experimental settings are given in Appendix B.
13] where we trained five
All experiments were run on 8 Tesla 16GB V100 GPUs. For each AL method, we computed the average running time per one acquisition iteration as the total running time divided by the total example the running times of all the AL methods on AG NEWS with unlabelled pools of different size. The batch size B was set to 50, top fraction 7 to 50% and the size of X to 600. Except for BADGE and the random baseline, the runtimes of the other methods increase nearly linearly with the size of the unlabelled pool.
number of iterations (i.e., 10). Table 2 summarizes
Comparative performance metries We followed Ash et al. [ to compute a pairwise comparison matrix but use counting-bass algorithm 28], as shown in the left of Figure I] The rows and columns of the matrix correspond to the AL methods used in our experiments. Each entry represents the outcome of the comparison between method and method over all datasets (D). Let Ci,j.d = when method beats method on dataset d, and 0 otherwise. Each cell value of the matrix is them computed C2, Σd C.j.d- To determine the value of Cs,..d used two-side paired t-test

---

Leaming curve
Painwise companison
0.95
VINOOU
O
2
0.25
Max Er1
0
2
0.20
50
100
150
203
290
300
Gorettte
Cooday
VIADEU
Mox
84L0
MaGU
Tola
#Labels acquired
CorcMSE
CoreLog
WMOCU
Rand
Max- Ent
BALD
MOCU
Figure I: Performance on SST-5 dataset. The left half illustrates the learning curve, while the right half illustrates the matrix of paired comparisons. The dashline represents the performance of the backbone classifier trained on the entire dataset.
to compare their performance for 5 weighted F1 scores (or accuracy) at maximally spaced labelled sample sizes {" 12, ".j.d"... 15 a. from the learning curve. We compute the t-score as t = /5p/e o where jú and are the usual sample mean and std.dev. In Figure 1, we selected five samples at the different iterations according to the step size 50 in the experiment. For example, the first sample ij is chosen at the 50th iteration, and the second sample i.j a is chosen at the 100 iteration, etc.
== = ((hgd) = V (tja-p2) The C.j.dd is assigned to if method beats method with (-score > 2.776 (p-value < <0.05). We accumulate the outcome of each pair comparison into the total quantity of each strategy (i.e., the "Total" column in the matrix). The highest total quantity gives a ranking over the AL methods. We also report the learning curves of all the AL methods with both weighted Fl score and accuracy.
4.1 Model performance: learning curves and comparative comparisons
Active learning with batch size one We first compared our CoreMSE and CoreLog based on Algorithm [ to the baselines on the PUBMED and the SST-5 datasets to demonstrate how those methods perform particularly on hard classification setting where classes are imbalanced. The learning curve sitting the left of Figure shows CoreMSE, Corelog and WMOCU outperform all the other methods considered, we attribute their estimation of uncertainty being better related to classification accuracy. Among these three methods, our CoreMSE performs the best in term of Fl score. The matrix at the right Figure then presents statistical summary of comparative performance. CoreMSE has the highest total quantity which further confirms its effectiveness in acquiring informative samples in AL. More results SST-5 and PUBMED, including both learning curves and comparative matrices are reported in Appendix C.
Batch active learning We compared batch CoreMSE and batch CoreLog implemented based on Algorithm with BADGE, ALPS and batch WMOCU on the four datasets listed in Table I We extended WMOCU with our Algorithm to build its batch counterpart. Specifically, we generated vec, using its point-wise error estimates, ie.. Eq (10) in 42 The random bascline selected B unlabelled samples randomly. Here we present the results derived with B 50 as example. More comprehensive results with different batch sizes, including accuracy, can be found in Appendix C.
The learning curves in Figure 2 show that batch CoreMSE and CoreLog almost always outperform the other AL methods as the number acquired samples increases. Batch WMOCU devised with our batch algorithm compare favourably with BADGE and ALPS that use gradient/surpriss embeddings to increase batch diversity. These results suggest that selecting the representative samples from clusters learned with vectors of expected change in scores can better improve batch diversity, leading to an improved AL performance. Comparing CoreMSE/CCrel..o with WMOCU further shows the advantage of BEMPS. Moreover, the performance differences between our methods and others

---

PUBMED,
AGNEWS,
0.55
030
0.75
0.75
0.26
0.60
0.70
0.52
0.55
0.60
0.50
0.00
0.75
300
acquired
acquired
#Labels acquired
acquired
CoreMSE
CoreLog
WMOCU
Rand
BADGE
ALPS
Figure 2: Learning curves of batch size 50 for PUBMED, IMDB, SST-5 and AG NEWS, The dashline represents the performance the backbone classifier trained on the entire dataset.
CoreNSS
16
11
10
48
3
15
14
12
53
-50
-40
CoreLog
3
13
11
11
47
15
12
52
-40
-32
WMOCU
0
0
4
2
9
-24
Rend
2
3
Rand
16
BADGE
7
3
10
BAUGEE
13
ALPS
6
ALPS
0
5
9
CovMAGE
Carolog
WMOOCU
Rand
BADGE
AUPS
Total
Comeltt
(a) Fl-based pairwise comparison
(b) Accuracy-baaac based pairwise comparison
Figure 3: Pairwise comparison matrices of batch active learning strategies.
on PUBMED and SST-5 indicate that batch CoreMSE and CoreLog can still achieve good results when the annotation budget limited in those imbalanced datasets. We also created four pairwise comparison matrices for different batch sizes using either F1 score or accuracy. Figure 3) show the sum of the four matrices, summarizing the comparative performance maximum cell value is now x 4 = 16. In other words, if a method beats another on all the four datasets across the four different batch sizes, the corresponding cell value will be 16. Both matrices computed with Fl score and accuracy respectively show both CoreMSE and CoreLog are ranked higher than the other methods. The observations discussed above also consistent across different batch sizes.
the four datasets. The
4.2 Ablation Studies
Batch size Figure 4 we plotted the learning curves of batch CoreMSE with different batch sizes (i.e., B e {1, 5, 10, 50, 100})) on PUBMED and SST-5 as example. The curves demonstrate that the performance of smaller batch sizes (5 or 10) is superior to that of large batch sizes (50 or 100), especially in the early rounds of training, but surprisingly also superior that of the no-batch case. The no-batch case need to perform multiple one- mee-tee-lookk-aeee
optimising acquisitions (A (Algorithm 1 and Algorithm 2) sequentially in order to acquire B samples whereas the batch case only perform the acquisition at once, heuristically, incorporating a form of diversity based directly on the error surface (Algorithm and Algorithm 3). This phenomena also seen with BatchBALD [12] see Figure 4). Thus we can conclude that the tee-step-lookkaheeeee of Equation (T is only greedy, and not optimal. More results of Batch CoreMSE and Corel log on PUBMED, SST-5, AG NEWS and IMDB can be found in Appendix B. In general, all the results shown in the learning curves prove that

---

PUBMED
SSTS
0.00
0.75
0.55
D.70
2.30
0166
0.00
0.06
320
0.6
#Labels acquired
#Labels acquired
Batch size
Batch size
Batch size 10
Batch size 50
Batch size 100
Figure 4: Learning curves of batch size 1, 5, 10, 50 and 100 for CoreMSE
PUBMED, Batch
Batch size:
Batch 50
AGNEWS, Batch
0.00
0.50
0.50
0.35
0.70
0.75
0.69
11
0.55
0.20
0.00
0.75
0.40
0.50
#Labels acquired
#Labels acquired
#Labels acquired
Alaaell acquired
CoreMSE
CoreLog
Pretrained
CoreMSE-topK
CoreLog-topk
Figure 5: Learning curves of the model training with diversity. The dashline represents the perfor- mance of the backbone classifier trained on the entire dataset.
batch algorithms improve AL performance over the no-batch ones, and smaller batch sizes are more preferable to the large ones.
Batch diversity To further study the effectiveness of Algorithm we considered the following variants: 1) Pretrained LM: Instead of using the expected changes of scores represent each unlabelled sample æ, we used the embedding generated by the last layer of DistilBERT in k-Means clustering, which is similar to the BERT-KM in [40 : and 2) CoreMSE_ peresss_ttppp and CoreLog_topp We
simply chose the top-B samples ranked by Qz. Figure 5 shows the results for all these variants. Our batch CoreMSE and CoreLog perform much better than the corresponding CoreMSE top and CoreLog top, which showcases Algorithm can promote batch diversity that benefits AL for text classification. The performance difference between Pretrained _L and batch CoreMSE/CCrell.oo indicates that representing each unlabelled sample as vector of expected changes in scores (i.e., Equation 6 effective capturing the information to be used for diversification.
Dynamic VS We studied how the dynamic VS impacts the ensemble model training by comparing batch CoreMSE with dynamic VS its following variations: 1) 3/30 epochs without VS: ensemble model training without VS and each model was trained for or 30 epochs 4, 2) Fixed length (# (#1000) VS: pre-fixed validation set with 1000 labelled samples separate from the labelled pool, used in some existing AL empirical work: 3) Constant VS: a variant of dynamic VS where one random split was generated after each iteration and then shared by all ensemble models. Figure 6 shows the learning curves of those compared AL methods. Dynamic VS gains an advantage after the third acquisition iteration on PUBMED and SST-5, the first iteration on AG NEWS. not surprising to see that 30 epochs without VS and Fixed length VS performs better in the early acquisition iterations,

---

PUBMED,
AGNEWS,
0.00
0.50
0.80
0.77
0.35
0.70
0.00
0.30
0.70
0.25
0.20
0.40
0.50
300
acquired
#Labels acquired
#Labels acquired
acquired
Dynamic VS
Constant
Fixed epachs (83) without VS
Fixed opochs (#30) without VS
Fixed length (#1000) vs
Figure 6: Learning curves the model training with a dynamic validation set, constant validation set, fixed # epochs without validation set, fixed length # labels validation set for CoreMSE. The dashline represents the performance the backbone classifier trained on the entire dataset.
since they used the whole augmented labelled pool in training DistilBERT, whereas CoreMSE used 70%. But choosing the number of epochs without validation set is simply heuristic otherwise. Also Fixed length VS is midway between Constant VS and Dynamic VS, indicating the variability in ensembles inherent in the dynamic training sets a source of improvement. More ablation results are reported in Appendix D.
Conclusion
We developed the BEMPS framework for acquisition functions for AL based around strictly proper scoring rules [9, or alternatively, Bregman divergences. These are fundamental scores in statistical learning theory and almost universally used when training neural networks, unlike simple errors, thus make solid basis. experiments we used mean squared error and log probability, making new acquisition functions CoreMSE and CoreLog respectively. For this, developed convergence theory, borrowing techniques from 42), that we also extended to the earlier BALD acquisition function. A primary limitation of the theory is that it does not support the of unlabelled data within the learning algorithm, for instance, as would be done by state of the art semi-supervised learning methods. Given that semi-superviser learning and AL really address common problem, this represents an area for future work for our techniques. But note recent high performing batch AL semi-supervised learning, thus excluded them from comparison.
For more efficient and effective evaluation of the new BEMPS based acquisition functions, we developed two techniques to improve performance, batch AL strategy naturally complement to our BEMPS algorithms, and dynamic validation cenettionn-nsmmmblnng hybrid that generates high scoring ensembles but does not require separate large labelled dataset. While this ensembling increases computational demand, it is only done for small datasets. These were both tested in isolation and shown to perform well. Though, interestingly, batch BEMPS works better than no-batch BEMPS. also seen for BatchBALD. This suggests that the mee-ste--look-hhee theory is needed. Finally, followed some of the strong evaluation standards set in earlier research, but we kept to small labelled set sizes in keeping with text annotation practice, testing our approach against WMOCU, BADGE, BALD and baselines. Using mean squared error and log probability as the scoring rules yielded consistently high-performing AI. on a variety of data sets.
can be improved, and additional
Acknowledgements
The work has been supported by the Tides Foundation through Grant 1904-57761, as part of the Google AT Impact Challenge, with Turning Point. Wray Buntine's work was also partly supported by DARPA's Learning with Less Labelling (LwLL) program under agreement FA8750-19-2-08
10

---

References
[1] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020. URL /ttllttss////peereeei
nwt fortt//ommmm wor.mrddd-rggg..BKE..
[2] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry P. Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020. URL httppa://openeviee..ee netfformmidd BJxI6gHKDr
[3] Franck Dernoncourt and Young Lec. PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), pages 308-313, Taipei, Taiwan, November 2017. Asian Federation
of Natural Language Processing. httL:ttt///www.... www.wa.cweb.coo www.:/ebbrrreeeee http://wwwww.....////////o////// wwagg/atthloogg/11-220002 117-2222001111222222
[4] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, A. Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. ArXiv, abs/2000.066355, 2020.
[5] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse. trainable neural networks. In International Conference Learning Representations, 2019. http:tts////pppeeeiieeee
met/forum?id-rJ1. b3RcF7
[6] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[7] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In International Conference un Machine Learning. pages 1183-1192. PMLR, 2017. URL http:// proceedings mlr.
http://////aaaa///aa//aaaa////// pdf
Ruijiang Gao and Maytal Saar- Tsechansccn, Cost-accuracy aware adaptive labeling for active learning. In Proceedings of the AÁAI Conference on Artificial Intelligence, pages 2569-2576, 2020.
[9] Tilmana Gueiting and Adrian Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-333 -388 2007. URL htttp:://di..rgg/ 1198/
450500000011
[10] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. stat, 1050:24, 2011.
[11] Amelia Jiménez-Sánchez, Shadi Alharqouni, and Diana Mateus. Capsule networks against medical imaging data challenges. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, pages 150-160. Springer, 2018.
[12] Andreas Kirsch, Joost Amersfoort, and Yarin Gal. BatchBALD: Efficient and diverse batch acquisition for deep Bayesian active leariig.. Advances Neural Information Processing Systems, 2019.
[13] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information I Processing Systems, NIPS' 17, pages 6405 6406-6616, Red Hook, NY, USA, 2017. Curran Associates Inc.
[14] Peng Liu, Hui Zhang, and Kie B Bom. Active deep learning for classification of hyperspectral images. IEEE journal of selected topics in applied earth observations and remote sensing, 10(2): 12-2772,-724, 2017. ISSN 1909.
[15] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URI. wtt.hhtt:///oeeeeieeeeeeee wtt.:e//ooommmieeeeeee......... [tt.
[16] Jinghui Lu and Brian MacNamee, Investigating the effectiveness of representations based pretrained raanfoomer-bbsed language models active learning for labelling text datasets. CoRR, 101s/2000413138, 13138
2020. URL TtRLhttts::// httpp:://arxv..or//abaaaaae/
[17] Shuang Ma. Zhaoyang Zeng. Daniel McDuff, and Yale Song. Active contrastive learning of audio- visual video representations. In International Conference on Learning Representations, 2021. URL
https://openreviov
htt.://awwrrr.tteeeeee/ro/raaaaa net//orrumiid--Mizttuu.

---

[18] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings the 49th annual meeting the association for computational linguistics: Human language technologies, pages 142-150, 2011. URL https:/ //ww aclveb.
11P1--1111,,.1111111111111111111 pdf
[19] Hieu T. Nguyen and Arold Smeulders. Active learning using pre-clustering. In Proceedings of the Twenty-Firg International Conference on Machine Learning, ICML '04, page 79, New York. NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi: 00.1145/001533000155554 URL
1111111111111330 1015349
[20] Robert Pinsler, Jonathan Gordon, Eric Nalismick, and José Miguel Hernández-Lbaat Bayesian batch active learning as sparse subset approximation. In H. Wallach, Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL URL_httss: //tp::ss//rccedinggs.. weurips.cc/paperr// 2019/ Baa/////000000077/0///0444444000 Paa0e44000---prr.... pdf
[21] Remus Pop and Patric Fulop. Deep ensemble bayesian active learning. In Bayesian Deep Learning Workshop at NeurlPS, 2020.
[22] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. survey of deep active learning. CoRR, 000/2, 2020. URL https:s/://aaxi org/ abs/2009 00236
[23] Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, pages 441-448, San Francisco, CA, USA, 2001.
[24] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, ahs/1910.01 108, 2019. http: waap////b/aa//// 1910
01108
[25] Christopher Schröder and Andrcas Nickler. A survey of active learning for text classificattin using deep neural networks. CoRR, abs/2008 072677, 2020.
07267
[26] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks:: International Conference on Learning Representations, 2018. [KL.pttraa///ppenerrieee.eeeeee
networss: A wpp.roa In
id=HiaTuk-B RW
[27] Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of -isconsin-Madison, 2009. URL BtLpttt::///xoo cs. byu.edu/ www.lldddddddmamii Saarrtinez/classs////ee settles. activelearning pdf
[28] Nihar B Shah and Martin J Wainwright. Simple, robust and optimal ranking from pairwise comparisons. The Journal of Machine Learning Research, (8(1):7246-7283 2017.
[29] Tianze Shi, Adrian Benton, Igor Malioutov, and Ozan Irsoy. Diversity-aware batch active learning for dependency parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2616 2626, Online, June 2021. Association for Computational Linguistics. URLhtttps://www.acleeb aclweb. org/ wwwpg//nnhooggy/// 2021 naacl main. 207
[30] Changjian Shui, Fan Zhou, Christian Gagné, and Boyu Wang. Deep active learning: Unified and principled method for query and training. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 1308-1318, 26-28 Aug 2020.
[31] Aditya Siddhant and Zachary C. Lipton. Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Longuage Processing, pages 2904-2909, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. URL http:: //ttp:://www... aclweb. org/ http///ttooooog/o---oaoe 13188
[32] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[33] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
12

---

[34] Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference neural networks (IJCNN), pages 112-119. IEEE, 2014. URL htt.:aa///eeeeeep..rrr ieee org/stamp/stamm http:///saaaapppppppppaaamammaaa japttpsaarnmbber-885557777
[35] Zengmao Wang. Bo Du, Lefei Zhang, and Liangpei Zhang. A batch-mode active learning framework by querying discriminative and representative samples for hyperspectral image classification. Neurocomputing (Amsterlam), 179:88-100 100, 2016. ISSN 0925-2312.
[36] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspec- tive of generalization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4697 4708. Curran Associates, Inc.. 2020. URL https::/prooeeednggg mew.riiinnnnnuuippppeee cctpappe//02//f1/// 12226448faaadd3355faadda-333---3
pdf
[37] Yazhou Yang and Marco Loog. Active learning using uncertainty information. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages
2016. doi: 10. 10.1109//CR.2216. 7.9004444
[38] Changchang Yin, Buyue Qian, Shilei Cao, Xiaoyu Li, Jishang Wei, Qinghua Zheng, and Jan Davidson. Deep similarity- based batch mode active learning with sxploration-xxplotaatoo In 2017 IEEE International Conference on Data Mining (ICDM), pages 575-584, IEEE, 2017.
[39] Byung- Juung-Jun Yoon, Xiaoning Qian, and Edward Dougherty. Quantifying objective cost of uncertainty in complex dynamical systems. IEEE Transactions on Signal Processing, 61(9):256622666, 2013.
[40] Michelle Yuan, Hsuan- Tien Lin, and Jordan Boyd- Graber. Cold-start active learning through self-supervised language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7935-7948, Online, November 2020. Association for Computational Linguis-
tics. URL httpp://www/ aclweb. org/ hwtgg//nthooogyy//// 2020 wttp:////g..20....mmmmm...... 637 637
[41] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-leve convolutional networks for text classification. In Proceedings of the 28th International Conference Neural Information Processing Systems Volume 1. NIPS' 15, page 649-657, Cambridge, MA, USA, 2015. MIT Press.
[42] Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xiaoning Qian. Uncertainty- aware active learning for optimal Bayesian classifier. In International Conference on Learning Represento- tions, ICLR 2021, 2021.
[43] Fedor Zhdanov. Diverse mini-batch active learning. CoRR, abs/1901.05954, 2019. URL httt::t/ta/tttti brg/abs//te 1901 05954
[44] Jingho Zhu and Matthew Ma. Uncertainty-basee active learning with instability estimation for text. classification. ACM Transactions on Speech and Language Processing (TSLP), 8(4):1221, 2012.
[45] Jingbo Zhu, Huizhen Wang, Benjamin K Tsou, and Matthew Ma. Active learning with sampling by uncertainty and density data annotations. IEEE Transactions on audio, speech, and language processing. 1323:1133) 2009.
13


Reviews:
[{"review_id": "mpEMxWxjo5W", "reviewer": "Reviewer_eduq", "summary": "The paper analyzes possible acquisition functions for active learning in the field of text classification. The authors introduce BEMPS (Bayesian Estimate of Mean Proper Scores), which leverages the existing Expected Loss Reduction method to estimate the increase in scores like log probability and negative mean square.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a novel approach to an important problem: increasing the diversity of the examples selected for labeling during active learning. The paper is reasonable well written, though the prose is awkward at times (e.g., lines 33 and 44-48) and makes a compelling case for the proposed approach. The contributions appear to be original. \n\nOne way to improve the paper's readability would be to add an intuitive running example before you get to Section 3, which is quite dry and dense on the allotted two pages. You can probably do this in less than one page, based on on figure and a few paragraphs. Illustrating your three main contributions (lines 35-48) in an intuitive manner will dramatically improve the paper's readability for a general audience. You are trying to do this in section 3.2 in a more formal manner.\n\n In Fig 1:\n- is the dashed line (in the image on the left) the performance if training on the entire dataset?\n- could you please discuss why RAND outperforms BALD & MOCU  (and ties with Max Ent) in this scenario?\n\nOTHER COMMENTS:\n- given that the work on WAAL, VAAL, and MAL is on image datasets, you do not have to make three times the argument on why you are not comparing with them (one time is enough :)\n- however, the idea of combining active & semi-supervised learning is not new - see [McCallum & Nigam, 1998] or [Muslea at at, 2002]\n- line 32: newer --> recent\n- line 44:  wth --> with             (please spell-check!)\n\nMcCallum, A., & Nigam, K. (1998). Employing EM in pool-based active learning for text classification. In ICML (pp. 359–367)\n\nMuslea, I., Minton, S., & Knoblock, C. A. (2002, July). Active+ semi-supervised learning= robust multi-view learning. In ICML (Vol. 2, pp. 435-442).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "ZyBOkiHr0Jq", "reviewer": "Reviewer_MYxD", "summary": "The paper proposes new acquisition functions for Bayesian active learning, which replaces the classification errors in MOCU with strictly proper scoring rules. The paper proves the convergence of the new acquisition functions and demonstrates the AL performance with text classification benchmark datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " The paper proposes a class of acquisition functions based on strictly proper scoring rules, which are guaranteed to converge to the true model. The batch AL algorithm is concise and general. The experiments do demonstrate the effectiveness of the proposed methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the paper lacks analysis and discussions of the proposed methods. The paper do prove the convergence of the acquisition functions based on strictly proper scoring rules, but there are no further discussions on the sampling efficiency, and the guideline for choosing the strictly proper scoring rules (or the G functions). Although the paper provides sufficient experiments, the paper doesn't provide enough discussions on the experiment results. \n\nI personally want to see the discussions on questions such as: since BALD/WMOCU are guaranteed to converge, why we still need to propose new methods? Why do we choose logarithmic scoring rule and squared error scoring rule? And based on the properties of the different scoring rules, in what conditions (large or small uncertainty, different sampling budgets) they can be more efficient than other competing methods?\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RcKZsaOSxUt", "reviewer": "Reviewer_P5cX", "summary": "This paper proposes a novel active learning method - called BEMPS (Bayesian Estimate of Mean Proper Scores) - in the context of text classification.\nBased on the ELR (expected loss reduction) approach, the authors propose to estimate the increase in the so-called \"strictly proper scores\"  such as the log probability or negative mean square error.\nThey show that the proposed method, BEMPS, makes the classifier asymptotically converge to the optimal classifier as the active learning process continues.\nFor proof, it builds upon techniques that were recently utilized for proving the convergence of the active learning scheme based on weighted MOCU (mean objective cost of uncertainty).\nBased on extensive evaluations on various benchmarks, this study shows that BEMPS generally outperforms other state-of-the-art active learning methods.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "GENERAL COMMENTS\n\nOverall, this is an interesting and well-written paper that improves the current state-of-the-art in active learning (AL) - especially in the context of text classification.\nThe paper presents strong performance evaluation results that clearly demonstrate the advantages of the proposed method, BEMPS, while also presenting interesting convergence proofs of the proposed active learner.\nAlthough the convergence proofs are based on techniques that were recently used to show the convergence of the WMOCU (weighted MOCU)-based active learning scheme, the authors show that the proposed approach often yields favorable learning outcomes compared to the WMOCU-based active learning scheme as well as several other AL algorithms that were tested.\n\n\nDETAILED COMMENTS\n\n1. What is the main impact of changing the ELR scheme to utilize proper scoring rules or the Bregman divergence instead of the classification error?\nConsidering that the main goal of the active learning process is to enhance the performance of the classifier (therefore reducing its classification accuracy), would this change lead to any \"semantic\" differences of the learning process compared to other ELR methods?\nAlthough convergence to the optimal classifier is shown in the paper, what would be the main benefits of changing the scoring scheme?\nProviding further insights would be useful.\nEspecially, considering that the original WMOCU-based scheme, to which BEMPS is compared, takes an \"objective-based\" approach, which focuses on the expected reduction of the classification error, it would be meaningful to compare BEMPS with WMOCU from this perspective.\n\n2. Based on the presentation in Sec. 3.1., what is the relation between  BEMPS and WMOCU?\nDoes the WMOCU scheme become a special case of BEMPS (or vice versa) under certain conditions?\n\n3. What do\"fully conditional model\" and \"full conditionality\" mean?\nPlease provide a clear definition.\n\n4. It is unclear how MOCU was utilized in the active learning framework.\nFor example, in order to evaluate the expected gain of a new data point (i.e., label) acquisition, expected conditional MOCU given the new data should be evaluated.\nBut as the current paper doesn't provide the relevant details, it is not clear how MOCU was utilized.\n\n5. Further discussion is needed regarding the relationship between the batch size and  the overall performance (e.g., lines 266-267)\nFor example, why would the no-batch case perform worse compared to using small batches?\nAnd would a larger batch size lead to better performance as the number of acquired labels increase outperforming small batch cases? (Although not initially, when the number of newly acquired data points is small?)\n\n6. Please carefully proofread the manuscript and the supplementary material to correct any typos/errors.\n- a prior -> a priori (line 110)\n- equation # missing in the supplementary material (line 68)\n\n-----------\n\nNote:\n\nI acknowledge that the authors' response to the original review comments has been reviewed and considered in the rating.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations of the proposed method are discussed in the conclusion.\nPotential negative societal impact is not discussed in detail.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "BrPvuXipYLn", "reviewer": "Reviewer_eoGS", "summary": "The paper addresses the problem of active learning by querying examples that will lead to the most reduction in classification\nerror, which is a fairly common idea. It proposes to place this in the\ncontext of strictly proper scoring rules in order to derive new\nexample acquisition functions and generalize existing\nones. Experiments on multiple datasets illustrate this for two common\nscoring rules, with extensive analysis, showing that the proposed\nmethods perform well.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Plus:\n+ clean framework with some theoretical guarantees\n+ experimental comparison investigates many aspects of the method\n+ consistent performance improvement over alternatives\n\nMinus:\n- semi-supervised learning is specifically excluded\n- novelty seems limited\n- many moving parts \n\nThe first part of the paper is a quick (and steep) intro of the theory behind the method. It is very formal and could benefit from more actual examples of instantiations of the concepts early on, but this is a matter of taste. This is followed by a sizeable experimental section which covers a number of questions and analyzes the impact of various aspects of the method (batch size and diversity, dynamic validation set). The results are convincing, with learning curves (with error bars) and statistical testing of improved performance.\n\nThe paper seemed technically correct, building upon previous work in machine (active) learning and statistics to place the method within a clean theoretical framework using strictly proper scoring rules and extending some existing methods in the process. Some of the contributions seem modest, for example the \"dynamic validation\" is essentially randomly splitting training and validation sets during ensemble training -- producing ensembles by samping train/val set has been used extensively and it is unclear whether there is something clearly novel in the particular way it is done here. There are many moving parts and hyper-parameters in the method. Some of them are tested in the ablation study in Sec. 4.2. Results suggest that the proposed method functions well, but for a fairly specific choice of these settings. One may wonder how these settings would generalize to other domains or data sets, for example with much larger datasets.\n\nThe main limitation of the paper imho is that it specifically excludes semi-supervised learning from consideration and comparisons (the \"fully conditional\" assumption). SSL and AL go hand-in-hand since at least co-training. Even though the nice theoretical framework does not accomodate SSL, it is hard not to at least compare with real state-of-the-art in AL.\n[**EDIT** The authors' point is fair that one has to start somewhere]\n\nClarifications:\n\nIs the 'running time per single acquisition' normalized for batch size, or for one batch (i.e. one run of Alg.3)? I would guess the latter, but 'single acquisition' introduces a bit of ambiguity.\n\nIt could help to place the sample sizes l^k_i,j,d on Fig 1 (left).\n\nMinor points:\n\nThe pairwise comparison matrix is a poster case for correcting for multiple comparisons.\n\nIn the conclusion BEMPS seems to turn into BERPS.\n\nTypo: generated (l.92)\n\n**EDIT** Thanks for the detailed reply. Review is updated accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes this is mentioned in the checklist, although not really relevant for this paper.\n", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops a new acquisition function for active learning, which is referred to as Bayesian Estimate of Mean Proper Scores or BEMPS for short. BEMPS generalizes weighted Mean Objective Cost of Uncertainty or WMOCU, a recently developed active learning acquisition function based on expected loss reduction (ELR) by using the strictly proper scoring rules to replace the classification errors. This generalization offers a more natural and principled way to avoid the acquisition function from getting stuck before reaching the optimal classifier, while also ensuring a broader applicability where errors are less informative. 

All the reviewers agree that this is a well-written paper, which tackles an important and interesting problem. They also acknowledge the theoretical contributions, including the proof of convergence to the true model parameter (but guarantee on sample efficiency is not provided, which is important for active learning). Evaluation on text classification tasks shows convincing results as compared with some recent baselines. 

Most questions raised by reviewers have been adequately addressed by the authors during the discussion phase. Given the similarity between the proposed BEMPS and existing ELR based acquisition functions (some of which are also proved to converge to the optimal classifier), it is important for the authors to include some additional insights on the advantage of using the strictly proper scoring rules, for example, on what specific conditions that BEMPS can bring additional benefit to the active sampling process, in the final version of the paper. 

Decision: Accept (Poster)