Learning in Confusion: Batch Active Learning with Noisy Oracle

1 INTRODUCTION . Supervised learning is the most widely used machine learning method , but it requires labelled data for training . It is time-consuming and labor-intensive to annotate a large dataset for complex supervised machine learning models . For example , ImageNet ( Russakovsky et al. , 2015 ) reported the time taken to annotate one object to be roughly 55 seconds . Hence an active learning approach which selects the most relevant samples for annotation to incrementally train machine learning models is a very attractive avenue , especially for training deep networks for newer problems that have littel annotated data . Classical active learning appends the training dataset with a single sample-label pair at a time . Given the increasing complexity of machine learning models , it is natural to expand active learning procedures to append a batch of samples at each iteration instead of just one . Keeping such training overhead in mind , a few batch active learning procedures have been developed in the literature ( Wei et al. , 2015 ; Sener & Savarese , 2018 ; Sinha et al. , 2019 ) . When initializing the model with a very small seed dataset , active learning suffers from the coldstart problem : at the very beginning of active learning procedures , the model is far from being accurate and hence the inferred output of the model is incorrect/uncertain . Since active learning relies on output of the current model to select next samples , a poor initial model leads to uncertain estimation of selection criteria and selection of wrong samples . Prior art on batch active learning suffers performance degradation due to this cold-start problem . Most active learning procedures assume the oracle to be perfect , i.e. , it can always annotate samples correctly . However , in real-world scenarios and given the increasing usage of crowd sourcing , for example Amazon Mechanical Turk ( AMT ) , for labelling data , most oracles are noisy . The noise induced by the oracle in many scenarios is resolute . Having multiple annotations on the same sample can not guarantee noise-free labels due to the presence of systematic bias in the setup and leads to consistent mistakes . To validate this point , we ran a crowd annotation experiment on ESC50 dataset ( Piczak , 2015 ) : each sample is annotated by 5 crowdworkers on AMT and the majority vote of the 5 annotations is considered the label . It turned out for some classes , 10 % of the samples are annotated wrong , even with 5 annotators . Details of the experiment can be found in Appendix A . Under such noisy oracle scenarios , classical active learning algorithms such as ( Chen et al. , 2015a ) under-perform as shown in Figure 1 . Motivating from these observations , we fashion a batch active learning strategy to be robust to noisy oracles . The main contributions of this work are as follows : ( 1 ) we propose a batch sample selection method based on importance sampling and clustering which caters to drawing a batch which is simultaneously diverse and important to the model ; ( 2 ) we incorporate model uncertainty into the sampling probability to compensate poor estimation of the importance scores when the training data is too small to build a meaningful model ; ( 3 ) we introduce a denoising layer to deep networks to robustify active learning to noisy oracles . Main results , as shown in Fig . 3 demonstrate that in noise-free scenario , our method performs as the best over the whole active learning procedure , and in noisy scenario , our method outperforms significantly over state-of-the-art methods . 2 RELATED WORK . Active Learning : Active learning ( Tong , 2001 ) is a well-studied problem and has gain interest in deep learning as well . A survey summarizes various existing approaches in ( Settles , 2009 ) . In a nutshell , two key and diverse ways to tackle this problem in the literature are discrimination and representation . The representation line of work focuses on selecting samples that can represent the whole unlabelled training set while the discrimination line of work aims at selecting ‘ tough ’ examples from the pool set , for example , using information theoretic scores in ( MacKay , 1992 ) , entropy as uncertainty in ( Wang & Shang , 2014 ) . Along the lines of ensemble methods we have works , for example , ( Beluch et al. , 2018 ; Freund et al. , 1997 ; Lakshminarayanan et al. , 2016 ) . A recent work of discrimination-based active learning ( Houlsby & Ghahramani , 2011 ) uses mutual information , Bayesian Active Learning by Disagreement ( BALD ) , as discriminating criteria . In ( Gal et al. , 2017 ) the authors used dropout approximation to compute the BALD scores for modern Convolutional Neural Networks ( CNNs ) . However , these approaches do not consider batch acquisition and hence lack of diversity in selected batch samples causing performance lag . Batch Active Learning : Active learning in the batch acquisition manner has been studied from the perspective of set selection and using submodularity or its variants in a variety of works . The authors in ( Wei et al. , 2015 ) utilize submodularity for naive Bayes and nearest neighbor . The concept of adaptive submodularity is related to active learning as well . The problem solves adaptive greedy optimization with sequential decision making ( Golovin & Krause , 2011 ) . Using this concept , ( Chen & Krause , 2013 ) considers pool-based Bayesian active learning with a finite set of candidate hypotheses . A pool-based active learning is also discussed in ( Ganti & Gray , 2011 ) which considered risk minimization under given hypothesis space . The work in ( Wang & Ye , 2013 ) uses both discriminative and representative samples to select a batch . The authors in ( Sener & Savarese , 2018 ) use coreset approach to select representative points of the pool set . Recently , an adversarial learning of variational auto-encoders is used for batch active learning in ( Sinha et al. , 2019 ) . The work make a representation of the training and pool , and adversarially select the pool representatives . Model Uncertainty : The uncertainty for deep learning models , especially CNNs , was first addressed in ( Gal & Ghahramani , 2016 ; Gal , 2016 ) using dropout as Bayesian approximation . Model uncertainty approximation using Batch Normalization ( BN ) has been shown in ( Teye et al. , 2018 ) . Both of these approaches in some sense exploit the stochastic layers ( Dropout , BN ) to extract model uncertainty . The importance of model uncertainty is also emphasized in the work of ( Kendall & Gal , 2017 ) . The work witnesses model as well as label uncertainty which they termed as epistemic and aleatoric uncertainty , respectively . We also address both of these uncertainties in this work . Noisy Oracle : The importance of noisy labels from oracle has been realized in the works like ( Golovin et al. , 2010 ; Chen et al. , 2015b ; Chen & Krause , 2013 ) which utilized the concept of adap- tive submodularity for providing theoretical guarantees . ( Chen et al. , 2017 ) studies the same problem but with correlated noisy tests . Active learning with noisy oracles is also studied in ( Naghshvar et al. , 2012 ; Yan et al. , 2016 ) . However , these work do not consider deep learning setup . A binary classification task with the noisy oracle is considered in ( Du & Ling , 2010 ) . The authors in ( Khetan et al. , 2018 ) used a variation of Expectation Maximization algorithm to estimate the correct labels as well as annotating workers quality . The closest work to us in the noisy oracle setting for deep learning models are ( Jindal et al. , 2019 ; 2016 ) . The authors also propose to augment the model with an extra full-connected dense layer . However , the denoising layer does not follow any probability simplex constraint , and they use modified loss function for the noise accountability along with dropout regularization . 3 PROBLEM FORMULATION . In this section , we introduce the notations used throughout the paper . We then formally define the problem of batch active learning with noisy oracles . Notations : The ith ( jth ) row ( column ) of a matrix X is denoted as Xi , . ( X. , j ) . ∆K−1 is the probability simplex of dimension K , where ∆K−1 = { ( p1 , p2 , . . . , pK ) ∈ RK | ∑K i=1 pi = 1 ∧ pi ≥ 0 ∀i } . For a probability vector p ∈ ∆K−1 , the Shannon entropy is defined as : H ( p ) = −∑Ki=1 pi log ( pi ) , and for p , q ∈ ∆K−1 the Kullback-Leibler ( KL ) divergence is defined as KL ( p||q ) = ∑Ki=1 pi log ( pi/qi ) . The KL-divergence is always non-negative and is 0 if and only if p = q . The expectation operator is taken as E. We are concerned with a K class classification problem with a sample space X and label space Y = { 1 , 2 , . . . , K } . The classification model M is taken to be gθ : X → Y parameterized with θ . The softmax output of the model is given by p = softmax ( gθ ( x ) ) ∈ ∆K−1 . The batch active learning setup starts with a set of labeled samples Dtr = { ( xi , yi ) } and unlabeled samples P = { ( xj ) } . With a query budget of b , we select a batch of unlabeled samples B as , B = ALG ( Dtr , M , b , P ) , |B| ≤ b , where ALG is the selection procedure conditioned on the current state of active learning ( Dtr , M , b , P ) . ALG is designed with the aim of maximizing the prediction accuracy EpX×Y [ ( hθ ( x ) = y ) ] . Henceforth , these samples which can potentially maximize the prediction accuracy are termed as important samples . After each acquisition iteration , the training dataset is updated as Dtr = Dtr ∪ { ( B , yB ) } where yB are the labels of B from an oracle routine . The oracle takes an input x ∈ X and outputs the ground truth label y ∈ Y . This is referred to as ‘ Ideal Oracle ’ and the mapping from x to y is deterministic . A ‘ Noisy Oracle ’ flips the true output y to y′ which is what we receive upon querying x . Similar to ( Chen et al. , 2015a ) , we assume that the label flipping is independent of the input x and thus can be characterized by the conditional probability p ( y′ = i|y = j ) , where i , j ∈ Y . We also refer this conditional distribution as the noisy-channel , and hence the ideal oracle has noisy channel value of 1 for i = j and 0 otherwise . For rest of the paper , we use the noise channel as aK-symmetric channel ( SC ) , see Figure 2b , which is a generalization of the binary symmetric channel . The K-SC is defined as follows p ( y′ = i|y = j ) = { 1− ε for i = j , ε/ ( K − 1 ) for i 6= j } ( 1 ) where ε is the probability of a label flip , i.e. , p ( y′ 6= y ) = ε . We resort to the usage ofK-SC because of its simplicity , and in addition , it abstracts the oracle noise strength with a single parameter ε . Therefore , in noisy active learning , after the selection of required subset B , the training dataset ( and then the model ) is updated as Dtr = Dtr ∪ { ( B , y′B ) } . Next , in Section 4 , we discuss the proposed solution to noisy batch active learning .

Summary: The topic handled in this paper is very important (hot topic), in my opinion. The authors tackled is the problem of training machine learning models incrementally using active learning with the oracle is noisy. Multiple samples are selected instead of a unique  sample as in the classical framework. The paper seems technically sound.