Active Learning in CNNs via Expected Improvement Maximization

1 INTRODUCTION . Deep learning models ( LeCun et al. , 2015 ) have achieved remarkable performance on many challenging prediction tasks , with applications spanning computer vision ( Voulodimos et al. , 2018 ) , computational biology ( Angermueller et al. , 2016 ) , and natural language processing ( Socher et al. , 2012 ) . However , training effective deep learning models often requires a large dataset , and assembling such a dataset may be difficult given limited resources and time . Active learning ( AL ) addresses this issue by providing a framework in which training can begin with a small initial dataset and , based on an objective function known as an acquisition function , choosing what data would be the most useful to have labelled ( Settles , 2009 ) . AL has successfully streamlined and economized data collection across many disciplines ( Warmuth et al. , 2003 ; Tong & Koller , 2001 ; Danziger et al. , 2009 ; Tuia et al. , 2009 ; Hoi et al. , 2006 ; Thompson et al. , 1999 ) . In particular , pool-based AL selects points from a given set of unlabeled pool points for labelling by an external oracle ( e.g . a human expert or biological experiment ) . The resulting labeled points are then added to the training set , and can be leveraged to improve the model and potentially query additional pool points ( Settles , 2011 ) . Until recently , few AL approaches have been formulated for deep neural networks such as CNNs due to their lack of efficient methods for computing predictive uncertainty . Most acquisition functions used in AL require reliable estimates of model uncertainty in order to make informed decisions about which data labels to request . However , recent developments have led to the possibility of computationally tractable predictive uncertainty estimation in deep neural networks . In particular , a framework for deep learning models has been developed viewing dropout ( Srivastava et al. , 2014 ) as an approximation to Bayesian variational inference that enables efficient estimation of predictive uncertainty ( Gal & Ghahramani , 2016 ) . Our approach , which we call “ Dropout-based Expected IMprOvementS ” ( DEIMOS ) , builds upon prior work aiming to make statistically optimal AL queries by selecting those points that minimize expected test error ( Cohn et al. , 1996 ; Gorodetsky & Marzouk , 2016 ; Binois et al. , 2019 ; Roy & McCallum , 2001 ) . We extend such approaches to CNNs through a flexible and computationally efficient algorithm that is primarily motivated by the regression setting , for which relatively few AL methods have been proposed , and extends to classification . Many AL approaches query the single point in the pool that optimizes a certain acquisition function . However , querying points one at a time necessitates model retraining after every acquisition , which can be computationally-expensive , and can lead to time-consuming data collection ( Chen & Krause , 2013 ) . Simply greedily selecting a certain number of points with the best acquisition function values typically reduces performance due to querying similar points ( Sener & Savarese , 2018 ) . Here we leverage uncertainty estimates provided by dropout in CNNs to create a dynamic representation of predictive uncertainty across a large , representative sample of points . Importantly , we consider the full joint covariance rather than just point-wise variances . DEIMOS acquires the point that maximizes the expected reduction in predictive uncertainty across all points , which we show is equivalent to maximizing the expected improvement ( EI ) . DEIMOS extends to batch-mode AL , where batches are assembled sequentially by dynamically updating a representation of predictive uncertainty such that each queried point is expected to result in a significant , non-redundant reduction in predictive uncertainty . We evaluate DEIMOS and find strong performance compared to existing benchmarks in several AL experiments spanning handwritten digit recognition , alternative splicing prediction , and face age prediction . 2 RELATED WORK . AL is often formulated using information theory ( MacKay , 1992b ) . Such approaches include querying the maximally informative batch of points as measured using Fisher information in logistic regression ( Hoi et al. , 2006 ) , and Bayesian AL by Disagreement ( BALD ) ( Houlsby et al. , 2011 ) , which acquires the point that maximizes the mutual information between the unknown output and the model parameters . Many AL algorithms have been developed based on uncertainty sampling , where the model queries points about which it is most uncertain ( Lewis & Catlett , 1994 ; Juszczak & Duin , 2003 ) . AL via uncertainty sampling has been applied to SVMs using margin-based uncertainty measures ( Joshi et al. , 2009 ) . AL has also been cast as an uncertainty sampling problem with explicit diversity maximization ( Yang et al. , 2015 ) to avoid querying correlated points . EI has been used as an acquisition function in Bayesian optimization for hyperparameter tuning ( Eggensperger et al. , 2013 ) . Other AL objectives similar to ours have been explored in ( Cohn et al. , 1996 ; Gorodetsky & Marzouk , 2016 ; Binois et al. , 2019 ; Roy & McCallum , 2001 ) , making statistically optimal queries that minimize expected prediction error , which often reduces to querying the point that is expected to minimize the learner ’ s variance integrated over possible inputs . DEIMOS extends EI and integrated variance approaches , which have traditionally been applied to Gaussian Processes , mixtures of Gaussians , and locally weighted regression , to deep neural networks . Until recently , few AL approaches have proven effective in deep learning models such as CNNs , largely due to difficulties in uncertainty estimation . Although Bayesian frameworks for neural networks ( MacKay , 1992a ; Neal , 1995 ) have been widely studied , these methods have not seen widespread adoption due to increased computational cost . However , theoretical advances have shown that dropout , a common regularization technique , can be viewed as performing approximate variational inference and enables estimation of model and predictive uncertainty ( Gal & Ghahramani , 2016 ) . Simple dropout-based AL objectives in CNNs have shown promising results in computer vision classification applications ( Gal et al. , 2017 ) . Several new algorithms show promising results for batch-mode AL on complex datasets . Batchbald ( Kirsch et al. , 2019 ) extends BALD to batch-mode while avoiding redundancy by greedily constructing a query batch that maximizes the mutual information between the joint distribution over the unknown outputs and the model parameters . Batch-mode AL in CNN classification has also been formulated as a core-set selection problem ( Sener & Savarese , 2018 ) with data points represented ( embedded ) using the activations of the model ’ s penultimate fully-connected layer . The queried batch of points then corresponds to the centers optimizing a robust ( i.e . outlier-tolerant ) k-Center objective for these embeddings . In spite of these recent advances , there are no frameworks for batch-mode AL in CNNs that model the full joint ( rather than point-wise ) uncertainty and do not require a vector space embedding of the data . Additionally , few AL approaches for CNNs have been assessed ( or even proposed ) for regression as compared to classification . 3 ACTIVE LEARNING VIA EXPECTED IMPROVEMENT ( EI ) MAXIMIZATION . 3.1 MOTIVATION . Our pool-based AL via EI maximization framework , DEIMOS , aims to maximally reduce expected ( squared ) prediction error on a large , representative sample of points by querying as few points from the pool as possible . Let Dsamp = ( Xsamp , Ysamp ) be an sufficiently large random sample from the training and pool points that it can be assumed representative of the dataset as a whole . Let Dpool = ( Xpool , Ypool ) denote the available pool of unlabelled data , ( xnew , ynew ) ∈ Dpool a candidate point for acquisition , θ the model parameters with approximate posterior q ( θ ) , and ŷi ( θ ) the model prediction for an input xi . Note that Ypool is not known and Ysamp is only partially known in AL as the pool consists of unlabeled points . For brevity , conditioning on model input is omitted in subsequent equations ( e.g . Eq [ ŷi ( θ ) | xnew , xi ] will be written as Eq [ ŷi ( θ ) | xnew ] ) . We seek to maximize EI by acquiring the pool point that minimizes expected prediction error on Dsamp . The expected prediction error on Dsamp conditioned on some xnew ∈ Xpool is , 1 |Ysamp| ∑ ( xi , yi ) ∈Dsamp Eq , n [ ( yi − ŷi ( θ ) ) 2 | xnew ] = 1 |Ysamp| ∑ ( xi , yi ) ∈Dsamp Eq [ ( ŷi ( θ ) − E [ ŷi ( θ ) ] ) 2 | xnew ] + ( Eq [ ŷi ( θ ) | xnew ] − En [ yi ] ) 2 + En [ ( yi − En [ yi ] ) 2 ] ( 1 ) where Eq and En denote expectation over q ( θ ) and observation noise , respectively , and Eq , n denotes joint expectation over q ( θ ) and observation noise . Here we see the terms contributing to the model ’ s expected prediction error on Dsamp are ( from left to right ) : 1 ) the trace of the predictive variance matrix , 2 ) the sum of the predictive biases squared , and 3 ) the sum of the noise variances across all observations ( a constant in xnew ) ( Cohn et al. , 1996 ) . For a purely supervised model , expected predictions remain the same unless the training set of input-output pairs is modified . Therefore , expected model predictions are unchanged conditioned on any unlabeled point xnew : Eq [ ŷi ( θ ) | xnew ] = Eq [ ŷi ( θ ) ] , and the predictive bias squared for any point ( xi , yi ) stays the same conditioned on xnew : ( Eq [ ŷi ( θ ) | xnew ] −En [ yi ] ) 2 = ( Eq [ ŷi ( θ ) ] −En [ yi ] ) 2 . Substituting accordingly in equation 1 , the pool point that would minimize expected prediction error on Dsamp if queried is : x∗ = arg min xnew∈Xpool 1 |Ysamp| tr ( Varq ( Ŷsamp ( θ ) | xnew ) ) ( 2 ) where Ŷsamp ( θ ) denotes the model predictions for input Xsamp and fixed parameters θ and Varq denotes variance over the approximate posterior . Therefore , the xnew that minimizes expected prediction error upon being queried is that expected to minimize average prediction variance ( specifically the trace of the predictive covariance matrix ) . Under our assumptions , knowing xnew is sufficient to calculate the predictive variance contribution to expected prediction error conditioned on ( xnew , ynew ) , even when ynew is unknown . Importantly , it follows that , even though Ypool is not known in AL , DEIMOS is still able to identify and query the pool point xnew ∈ Xpool that minimizes expected prediction error across Dsamp after ( xnew , ynew ) is incorporated into the training set . 3.2 MC DROPOUT VARIANCE ESTIMATION . The proposed approach can be implemented with any uncertainty estimation method that enables calculation of the variance and covariance of model predictions . In our experiments , we use MC dropout ( Gal & Ghahramani , 2016 ) to obtain estimates of model uncertainty in CNNs . Models are trained with dropout preceding all fully-connected layers and dropout is used at test time to generate T approximate samples from the posterior predictive distribution . The predictive mean is obtained as the average of these samples . In order to estimate the predictive covariance matrix for all sample points , J dropout masks are randomly generated for all dropout layers in the neural network . Crucially , the same J dropout masks are used to make test-time predictions across all sample points to enable estimating correlation between them . From the dropout sample covariance matrix Varq ( Ŷsamp , dropout ( θ ) ) capturing the sample variances and covariances of the J dropout predictions for each sample point , one can estimate the prediction covariance matrix ( Gal & Ghahramani , 2016 ) , Varq ( Ŷsamp ( θ ) ) = τ−1I + Varq ( Ŷsamp , dropout ( θ ) ) . ( 3 ) Here τ = ( 1−p ) l 2 2Nλ represents the model precision in regression tasks , where p is the dropout probability , l is a prior length-scale parameter , N is the number of training points , and λ represents the weight decay ( Gal & Ghahramani , 2016 ) . In the classification setting , τ−1 = 0 .

Summary: The paper considers the problem of active learning for training convolutional neural networks (CNN) in a sample-efficient manner. The proposed approach is built upon the existing idea of selecting points that maximally reduce expected mean squared error (MSE) on a large representative sample of points. MC-dropout is used for obtaining the estimates of model uncertainty. This idea is used for active learning in regression and classification problems with CNNs. A greedy method is proposed to select a batch of points by maximizing the acquisition function score sequentially obtained by updating the covariance matrix on previous points selected in the batch. Experiments are performed on two regression and one classification task.