Diversity Enhanced Active Learning with Strictly Proper Scoring Rules

1 Introduction . Classification has extensive uses and deep learning has substantially improved its performance , but a major hurdle for its use is the paucity of labelled or annotated data . The data labelling process performed by domain experts is expensive and tedious to produce , especially in the medical field , where due to lack of expertise and privacy issues , annotation costs are time-consuming and expensive [ 11 ] . Active Learning ( AL ) is an approach to speeding up learning by judiciously selecting data to be annotated [ 27 ] . AL is perhaps the simplest of all human-in-the-loop learning approaches , because the only interaction is the ( expert ) human providing a class label , yet for even in the simplest of tasks , classification , a general theory of AL is not agreed on . Moreover , in practical situations , one needs to consider many issues when designing an AL system with deep learning : the expense to retrain deep neural networks and the use of validation data for training [ 31 ] , transformer language models [ 16 ] , batch mode AL with diversity [ 22 ] , and consideration of expert capabilities and costs [ 8 , 43 ] . Also important in experimental work is the need for realistic labelling sizes , with practitioners we work with saying expert annotation may allow a budget of up to 1000 data points , rarely more . Using large batch sizes ( e.g. , 1000 in [ 1 , 20 ] ) can thus be impractical . Our fundamental research contribution is to suggest what makes a good acquisition function for the uncertainty component , without any batching . While there are many recent methods looking at the uncertainty diversity trade-off for batch AL [ 22 ] , few recently have focused on understanding the uncertainty side alone . A substantial advance is a recent theoretical framework , Mean Objective ∗Corresponding author 2Our implementation of BEMPS can be downloaded from https : //github.com/davidtw999/BEMPS . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Cost of Uncertainty ( MOCU ) [ 42 ] that provides a convergence proof . As our first contribution , we convert their expected loss reduction framework to an AL model using strictly proper scoring rules or Bregman divergences [ 9 ] instead of classification errors . MOCU required manipulations of the expected error , resulting in weighted-MOCU ( WMOCU ) , in order to achieve convergence and avoid getting stuck in error bands . Strictly proper scoring rules naturally avoid this problem by generalising expected errors to expected scores . Using strictly proper scoring rules means better calibrated classifiers are rewarded . The scoring rules go beyond simple minimum errors of WMOCU and can be adapted to different kinds of inference tasks ( e.g. , different utilities , precision-recall trade-offs , etc. ) . This property is preferable and beneficial for applications such as medical domains where actual errors become less relevant for an inference task . In order to evaluate the new acquisition functions we use text classification , which is our target application domain . For realistic evaluation , we want to use near state of the art systems , which means using pretrained language models with validation sets [ 24 ] , and neural network ensembles [ 13 ] . Ensembling also doubles as a heuristic technique to yield estimates of model uncertainty and posterior probabilities . Coming up with a simple approach to combine ensembling and validations sets is our second research contribution . The importance of these combinations for AL has been noted [ 40 , 25 ] . For further batch comparisons , we then bring back diversity into the research , suggesting a way to naturally complement our new family of acquisition functions with a method to achieve diversity , our third research contribution . Extensive experiments with a comprehensive set of ablation studies on four text classification datasets show that our BEMPS-based AL model consistently outperforms recent techniques like WMOCU and BADGE , although we explicitly exclude recent semi-supervised AL methods because they represent an unfair comparison against strictly supervised learning . 2 Related work . The proposed BEMPS , as a general Bayesian model for acquisition functions , quantifies the model uncertainty using the theory of ( strictly ) proper scoring rules for categorical variables [ 9 ] . Thus , we first review existing acquisition functions ( aka query strategies ) proposed in some recent AL models that are most related to ours , and refer interested readers to [ 22 ] for a comprehensive discussion of recent strategies . One common and simple heuristic often used is maximum entropy [ 45 , 44 , 34 , 7 ] , where one chooses samples that maximise the predictive entropy . Bayesian AL by disagreement ( BALD ) [ 10 ] and its batch version ( i.e. , BatchBALD ) [ 12 ] instead compute the mutual information between the model predictions and the model parameters , which indeed chooses samples that maximise the decrease in expected entropy [ 20 ] . Recently , inspired by the one-step-look-ahead strategy of ELR [ 23 ] , Zhao et al . [ 42 ] extended MOCU [ 39 ] to WMOCU with a theoretical guarantee of convergence . But WMOCU only applies to errors which are less relevant for some inference tasks . BEMPS naturally extends those methods , focusing on Bayesian estimation of uncertainty related to the model ’ s expected proper score . Ensemble methods are used to obtain better uncertainty estimates with deep learning , including MonteCarlo dropout ( MC-dropout ) [ 6 , 12 , 21 ] and deep ensembles [ 13 ] . More sophisticated techniques are being developed , for instance , MCMC , hybrid and deterministic approaches [ 36 , 2 ] . However , plain ensembling remains a competitive and simple method for deep learning [ 2 ] . Query strategies considering just uncertainty do not always work well in a batch setting , due to the highly similar samples acquired in a batch [ 35 , 22 , 17 ] . To overcome this problem , there have been many AL methods that achieve batch diversity by acquiring samples that are both informative and diverse , such as [ 19 , 38 , 14 , 1 , 40 , 43 , 29 ] . BADGE [ 1 ] and ALPS [ 40 ] are the two recent AL methods focusing on batch diversity . BADGE uses gradient embeddings of unlabelled samples as inputs of k-MEANS++ to select a set of diverse samples , which relies on fine-tuning pretrained language models . Instead of using gradient embeddings , ALPS uses surprisal embeddings computed from the predictive word probabilities generated by a masked language model for cold-start AL . Whereas our BEMPS computes for each unlabelled sample a vector of expected change in the proper scores , relating directly to the performance . Other recent batch AL approaches , WAAL [ 30 ] and VAAL [ 32 ] use semi-supervised learning , for instance WAAL ’ s models are trained on the feature space obtained with the help of the unlabelled data . We exclude these methods from our comparison since they are semi-supervised learning algorithms . Both Core-set [ 26 ] and WAAL [ 30 ] also set up sophisticated non-Bayesian cost functions using bounds to deal with the many unknown probabilities when trying to optimise for a batch , whereas the Bayesian formula of BEMPS can be directly estimated . In experimental work , the use of a validation set to train deep learning models in active learning is not uncommon , the goal of which is to avoid over-fitting and achieve early-stopping . Some existing methods assume that there is a large validation set available a prior [ 1 , 12 , 7 ] , but means the cost of labelling the set is not factored into the labelling budget . We argue that the availability of a separate validation set is impractical in real world AL scenarios . Although Yuan et al . [ 40 ] use fixed epochs to train the classifier without a validation set to save the cost , the classifier could either be under-fit or over-fit . We instead use a dynamic approach to generate alternative validation sets from the ever increasing labelled pool after each iteration . 3 Bayesian estimate of mean proper scores . We first review the general Bayesian model for acquisition functions that includes ELR , MOCU and BALD . We then develop Bayesian Estimate of Mean Proper Scores ( BEMPS ) , a new uncertainty quantification framework with a theoretical foundation based on strictly proper scoring rules [ 9 ] . It naturally extends ELR and BALD , focusing on the Bayesian estimation of uncertainty related to the models expected performance . Suppose models of our interest are parameterised by parameters θ ∈ Θ , L indicates labelled data , probability of label y for data x is given by Pr ( y | θ , x ) , and Pr ( · | θ , x ) presents a vector of label probabilities . With a fully conditional model , the posterior of θ is unaffected by unlabelled data , which means Pr ( θ | L , U ) = Pr ( θ | L ) for any unlabelled data U . Moreover , we assume without loss of generality that this model family is well behaved in a statistical sense , so the model is identifiable . The “ cost '' of the posterior Pr ( θ | L ) can be measured by some functional Q ( Pr ( θ | L ) ) , denoted Q ( L ) for short , whereQ ( L ) ≥ 0 andQ ( L ) = 0 when some convergence objective has been achieved . For our model this is when Pr ( θ | L ) has converged to a point mass at a single model . A suitable objective function is to measure the expected decrease in Q ( · ) due to acquiring the label for a data point x . The corresponding acquisition function for AL is formulated [ 10 , Equation ( 1 ) ] as ∆Q ( x|L ) = Q ( L ) − EPr ( y|L , x ) [ Q ( L ∪ { ( x , y ) } ) ] , ( 1 ) whereas for ELR the expression is split over an inequality sign [ 23 , Equation ( 2 ) ] . It estimates how much the cost is expected to reduce when a new data point x is acquired . Since the true label for the new data x is unknown a prior , we have to use expected posterior proportions from our model , Pr ( y | L , x ) to estimate the likely label . For BALD [ 10 ] using Shannon ’ s entropy , QI ( L ) = I ( Pr ( θ | L ) ) , which measures uncertainty in the parameter space and thus has no strong relationship to actual errors [ 42 ] . MOCU and ELR use a Bayesian regret given by the expected loss difference between the optimal Bayesian classifier and the optimal classifier : QMOCU ( L ) = EPr ( x′ ) min y′ ( 1− Pr ( y′ | L , x′ ) ) − EPr ( θ|L ) [ min y′ ( 1− Pr ( y′ | θ , x′ ) ) ] . ( 2 ) WMOCU uses a weighting function defined by Eq ( 11 ) in [ 42 ] to have a more amenable definition of ∆Q ( x|L ) than the MOCU method . Although WMOCU guarantees ∆Q ( x|L ) converging to the optimal classifier ( under minimum errors ) according to the Q ( L ) with the strictly concave function by Eq ( 15 ) in [ 42 ] , the optimal approximation of the convergences can only be solved by controlling a hyperparameter of the weighting function manually . To allow theoretical guarantees of convergence under more general loss functions , we propose a different definition for Q ( L ) based on strictly proper scoring rules .

Summary: This paper proposes a novel active learning method - called BEMPS (Bayesian Estimate of Mean Proper Scores) - in the context of text classification. Based on the ELR (expected loss reduction) approach, the authors propose to estimate the increase in the so-called "strictly proper scores"  such as the log probability or negative mean square error. They show that the proposed method, BEMPS, makes the classifier asymptotically converge to the optimal classifier as the active learning process continues. For proof, it builds upon techniques that were recently utilized for proving the convergence of the active learning scheme based on weighted MOCU (mean objective cost of uncertainty). Based on extensive evaluations on various benchmarks, this study shows that BEMPS generally outperforms other state-of-the-art active learning methods.