Selection via Proxy: Efficient Data Selection for Deep Learning

1 INTRODUCTION . Data selection methods , such as active learning and core-set selection , improve the data efficiency of machine learning by identifying the most informative training examples . To quantify informativeness , these methods depend on semantically meaningful features or a trained model to calculate uncertainty . Concretely , active learning selects points to label from a large pool of unlabeled data by repeatedly training a model on a small pool of labeled data and selecting additional examples to label based on the model ’ s uncertainty ( e.g. , the entropy of predicted class probabilities ) or other heuristics ( Lewis & Gale , 1994 ; Rosenberg et al. , 2005 ; Settles , 2011 ; 2012 ) . Conversely , core-set selection techniques start with a large labeled or unlabeled dataset and aim to find a small subset that accurately approximates the full dataset by selecting representative examples ( Har-Peled & Kushal , 2007 ; Tsang et al. , 2005 ; Huggins et al. , 2016 ; Campbell & Broderick , 2017 ; 2018 ; Sener & Savarese , 2018 ) . Unfortunately , classical data selection methods are often prohibitively expensive to apply in deep learning ( Shen et al. , 2017 ; Sener & Savarese , 2018 ; Kirsch et al. , 2019 ) . Deep learning models learn complex internal semantic representations ( hidden layers ) from raw inputs ( e.g. , pixels or characters ) that enable them to achieve state-of-the-art performance but result in substantial training times . Many core-set selection and active learning techniques require some feature representation before they can accurately identify informative points either to take diversity into account or as part of a trained model to quantify uncertainty . As a result , new deep active learning methods request labels in large batches to avoid retraining the model too many times ( Shen et al. , 2017 ; Sener & Savarese , 2018 ; Kirsch et al. , 2019 ) . However , batch active learning still requires training a full deep model for every batch , which is costly for large models ( He et al. , 2016b ; Jozefowicz et al. , 2016 ; Vaswani et al. , 2017 ) . Similarly , core-set selection applications mitigate the training time of deep learning models by using ∗Correspondence : cody @ cs.stanford.edu bespoke combinations of hand-engineered features and simple models ( e.g. , hidden Markov models ) pretrained on auxiliary tasks ( Wei et al. , 2013 ; 2014 ; Tschiatschek et al. , 2014 ; Ni et al. , 2015 ) . In this paper , we propose selection via proxy ( SVP ) as a way to make existing data selection methods more computationally efficient for deep learning . SVP uses the feature representation from a separate , less computationally intensive proxy model in place of the representation from the much larger and more accurate target model we aim to train . SVP builds on the idea of heterogeneous uncertainty sampling from Lewis & Catlett ( 1994 ) , which showed that an inexpensive classifier ( e.g. , naïve Bayes ) can select points to label for a much more computationally expensive classifier ( e.g. , decision tree ) . In our work , we show that small deep learning models can similarly serve as an inexpensive proxy for data selection in deep learning , significantly accelerating both active learning and core-set selection across a range of datasets and selection methods . To create these cheap proxy models , we can scale down deep learning models by removing layers , using smaller model architectures , and training them for fewer epochs . While these scaled-down models achieve significantly lower accuracy than larger models , we surprisingly find that they still provide useful representations to rank and select points . Specifically , we observe high Spearman ’ s and Pearson ’ s correlations between the rankings from small proxy models and the larger , more accurate target models on metrics including uncertainty ( Settles , 2012 ) , forgetting events ( Toneva et al. , 2019 ) , and submodular algorithms such as greedy k-centers ( Wolf , 2011 ) . Because these proxy models are quick to train ( often 10× faster ) , we can identify which points to select nearly as well as the larger target model but significantly faster . We empirically evaluated SVP for active learning and core-set selection on five datasets : CIFAR10 , CIFAR100 ( Krizhevsky & Hinton , 2009 ) , ImageNet ( Russakovsky et al. , 2015 ) , Amazon Review Polarity , and Amazon Review Full ( Zhang et al. , 2015 ) . For active learning , we considered both least confidence uncertainty sampling ( Settles , 2012 ; Shen et al. , 2017 ; Gal et al. , 2017 ) and the greedy k-centers approach from Sener & Savarese ( 2018 ) with a variety of proxies . Across all datasets , we found that SVP matches the accuracy of the traditional approach of using the same large model for both selecting points and the final prediction task . Depending on the proxy , SVP yielded up to a 7× speed-up on CIFAR10 and CIFAR100 , 41.9× speed-up on Amazon Review Polarity and Full , and 2.9× speed-up on ImageNet in data selection runtime ( i.e. , the time it takes to repeatedly train and select points ) . For core-set selection , we tried three methods to identify a subset of points : max entropy uncertainty sampling ( Settles , 2012 ) , greedy k-centers as a submodular approach ( Wolf , 2011 ) , and the recent approach of forgetting events ( Toneva et al. , 2019 ) . For each method , we found that smaller proxy models have high Spearman ’ s rank-order correlations with models that are 10× larger and performed as well as these large models at identifying subsets of points to train on that yield high test accuracy . On CIFAR10 , SVP applied to forgetting events removed 50 % of the data without impacting the accuracy of ResNet164 with pre-activation ( He et al. , 2016b ) , using a 10× faster model than ResNet164 to make the selection . This substitution yielded an end-to-end training time improvement of about 1.6× for ResNet164 ( including the time to train and use the proxy ) . Taken together , these results demonstrate that SVP is a promising , yet simple approach to make data selection methods computationally feasible for deep learning . While we focus on active learning and core-set selection , SVP is widely applicable to methods that depend on learned representations . 2 METHODS . In this section , we describe SVP and show how it can be incorporated into active learning and core-set selection . Figure 1 shows an overview of SVP : in active learning , we retrain a proxy model APk in place of the target model ATk after each batch is selected , and in core-set selection , we train the proxy AP [ n ] rather than the target A T [ n ] over all the data to learn a feature representation and select points . 2.1 ACTIVE LEARNING . Pool-based active learning starts with a large pool of unlabeled data U = { xi } i∈ [ n ] where [ n ] = { 1 , . . . , n } . Each example is from the space X with an unknown label from the label space Y and is sampled i.i.d . over the space Z = X × Y as ( xi , yi ) ∼ pZ . Initially , methods label a small pool of points s0 = { s0j ∈ [ n ] } j∈ [ m ] chosen uniformly at random . Given U , a loss function ` , and the labels { ys0j } j∈ [ m ] for the initial random subset , the goal of active learning is to select up to a budget of b points s = s0 ∪ { sj ∈ [ n ] \ s0 } j∈ [ b−m ] to label that produces a model As with low error . Baseline . In this paper , we apply SVP to least confidence uncertainty sampling ( Settles , 2012 ; Shen et al. , 2017 ; Gal et al. , 2017 ) and the recent greedy k-centers approach from Sener & Savarese ( 2018 ) . Like recent work for deep active learning ( Shen et al. , 2017 ; Sener & Savarese , 2018 ; Kirsch et al. , 2019 ) , we consider a batch setting with K rounds where we select bK points in every round . Following Gal et al . ( 2017 ) ; Sener & Savarese ( 2018 ) ; Kirsch et al . ( 2019 ) , we reinitialize the target model and retrain on all of the labeled data from the previous k rounds to avoid any correlation between selections ( Frankle & Carbin , 2018 ; Kirsch et al. , 2019 ) . We denote this trained model as ATs0∪ ... ∪sk or just ATk for simplicity . Then using A T k , we either calculate the model ’ s confidence as : fconfidence ( x ; A T k ) = 1−max ŷ P ( ŷ|x ; ATk ) and select the examples with the lowest confidence or extract a feature representation from the model ’ s final hidden layer and compute the distance between examples ( i.e. , ∆ ( xi , xj ; ATk ) ) to select points according to the greedy k-centers method from Wolf ( 2011 ) ; Sener & Savarese ( 2018 ) ( Algorithm 1 ) . The same model is trained on the final b labeled points to yield the final model , ATK , which is then tested on a held-out set to evaluate error and quantify the quality of the selected data . Although other selection approaches exist , least confidence uncertainty sampling and greedy k-centers cover the spectrum of uncertainty-based and representativeness-based approaches for deep active learning . Other uncertainty metrics such as entropy or margin were highly correlated with confidence when using the same trained model ( i.e. , above a 0.96 Spearman ’ s correlation in our experiments on CIFAR ) . Query-by-committee ( Seung et al. , 1992 ) can be prohibitively expensive in deep learning , where training a single model is already costly . BALD ( Houlsby et al. , 2011 ) has seen success in deep learning ( Gal et al. , 2017 ; Shen et al. , 2017 ) but is restricted to Bayesian neural networks or networks with dropout ( Srivastava et al. , 2014 ) as an approximation ( Gal & Ghahramani , 2016 ) . Algorithm 1 GREEDY K-CENTERS ( WOLF , 2011 ; SENER & SAVARESE , 2018 ) Input : data xi , existing pool s0 , trained model AT0 , and a budget b 1 : Initialize s = s0 2 : repeat 3 : u = arg maxi∈ [ n ] \s minj∈s ∆ ( xi , xj ; A T 0 ) 4 : s = s ∪ { u } 5 : until |s| = b + |s0| 6 : return s \ s0 Algorithm 2 FORGETTING EVENTS ( TONEVA ET AL. , 2019 ) 1 : Initialize prev_acci = 0 , i ∈ [ n ] 2 : Initialize forgetting_eventsi = 0 , i ∈ [ n ] 3 : while training is not done do 4 : Sample mini-batch B from L 5 : for example i ∈ B do 6 : Compute acci 7 : if prev_acci > acci then 8 : forgetting_eventsi += 1 9 : prev_acci = acci 10 : Gradient update classifier on B 11 : return forgetting_events

Summary: This paper presents a method to speed up the data selection in active learning and core-set learning. The authors present a simple idea: instead of using the full model to select data points, they use a smaller model with fewer layers, potentially trained for fewer iterations. The authors show that this simple approach is able to speed up the data selection portion of both processes significantly with minimal loss in performance, and also results in significant speedup of the entire pipeline (data selection + training).