Selection via Proxy: Efficient Data Selection for Deep Learning

1 INTRODUCTION . Data selection methods , such as active learning and core-set selection , improve the data efficiency of machine learning by identifying the most informative training examples . To quantify informativeness , these methods depend on semantically meaningful features or a trained model to calculate uncertainty . Concretely , active learning selects points to label from a large pool of unlabeled data by repeatedly training a model on a small pool of labeled data and selecting additional examples to label based on the model ’ s uncertainty ( e.g. , the entropy of predicted class probabilities ) or other heuristics ( Lewis & Gale , 1994 ; Rosenberg et al. , 2005 ; Settles , 2011 ; 2012 ) . Conversely , core-set selection techniques start with a large labeled or unlabeled dataset and aim to find a small subset that accurately approximates the full dataset by selecting representative examples ( Har-Peled & Kushal , 2007 ; Tsang et al. , 2005 ; Huggins et al. , 2016 ; Campbell & Broderick , 2017 ; 2018 ; Sener & Savarese , 2018 ) . Unfortunately , classical data selection methods are often prohibitively expensive to apply in deep learning ( Shen et al. , 2017 ; Sener & Savarese , 2018 ; Kirsch et al. , 2019 ) . Deep learning models learn complex internal semantic representations ( hidden layers ) from raw inputs ( e.g. , pixels or characters ) that enable them to achieve state-of-the-art performance but result in substantial training times . Many core-set selection and active learning techniques require some feature representation before they can accurately identify informative points either to take diversity into account or as part of a trained model to quantify uncertainty . As a result , new deep active learning methods request labels in large batches to avoid retraining the model too many times ( Shen et al. , 2017 ; Sener & Savarese , 2018 ; Kirsch et al. , 2019 ) . However , batch active learning still requires training a full deep model for every batch , which is costly for large models ( He et al. , 2016b ; Jozefowicz et al. , 2016 ; Vaswani et al. , 2017 ) . Similarly , core-set selection applications mitigate the training time of deep learning models by using ∗Correspondence : cody @ cs.stanford.edu bespoke combinations of hand-engineered features and simple models ( e.g. , hidden Markov models ) pretrained on auxiliary tasks ( Wei et al. , 2013 ; 2014 ; Tschiatschek et al. , 2014 ; Ni et al. , 2015 ) . In this paper , we propose selection via proxy ( SVP ) as a way to make existing data selection methods more computationally efficient for deep learning . SVP uses the feature representation from a separate , less computationally intensive proxy model in place of the representation from the much larger and more accurate target model we aim to train . SVP builds on the idea of heterogeneous uncertainty sampling from Lewis & Catlett ( 1994 ) , which showed that an inexpensive classifier ( e.g. , naïve Bayes ) can select points to label for a much more computationally expensive classifier ( e.g. , decision tree ) . In our work , we show that small deep learning models can similarly serve as an inexpensive proxy for data selection in deep learning , significantly accelerating both active learning and core-set selection across a range of datasets and selection methods . To create these cheap proxy models , we can scale down deep learning models by removing layers , using smaller model architectures , and training them for fewer epochs . While these scaled-down models achieve significantly lower accuracy than larger models , we surprisingly find that they still provide useful representations to rank and select points . Specifically , we observe high Spearman ’ s and Pearson ’ s correlations between the rankings from small proxy models and the larger , more accurate target models on metrics including uncertainty ( Settles , 2012 ) , forgetting events ( Toneva et al. , 2019 ) , and submodular algorithms such as greedy k-centers ( Wolf , 2011 ) . Because these proxy models are quick to train ( often 10× faster ) , we can identify which points to select nearly as well as the larger target model but significantly faster . We empirically evaluated SVP for active learning and core-set selection on five datasets : CIFAR10 , CIFAR100 ( Krizhevsky & Hinton , 2009 ) , ImageNet ( Russakovsky et al. , 2015 ) , Amazon Review Polarity , and Amazon Review Full ( Zhang et al. , 2015 ) . For active learning , we considered both least confidence uncertainty sampling ( Settles , 2012 ; Shen et al. , 2017 ; Gal et al. , 2017 ) and the greedy k-centers approach from Sener & Savarese ( 2018 ) with a variety of proxies . Across all datasets , we found that SVP matches the accuracy of the traditional approach of using the same large model for both selecting points and the final prediction task . Depending on the proxy , SVP yielded up to a 7× speed-up on CIFAR10 and CIFAR100 , 41.9× speed-up on Amazon Review Polarity and Full , and 2.9× speed-up on ImageNet in data selection runtime ( i.e. , the time it takes to repeatedly train and select points ) . For core-set selection , we tried three methods to identify a subset of points : max entropy uncertainty sampling ( Settles , 2012 ) , greedy k-centers as a submodular approach ( Wolf , 2011 ) , and the recent approach of forgetting events ( Toneva et al. , 2019 ) . For each method , we found that smaller proxy models have high Spearman ’ s rank-order correlations with models that are 10× larger and performed as well as these large models at identifying subsets of points to train on that yield high test accuracy . On CIFAR10 , SVP applied to forgetting events removed 50 % of the data without impacting the accuracy of ResNet164 with pre-activation ( He et al. , 2016b ) , using a 10× faster model than ResNet164 to make the selection . This substitution yielded an end-to-end training time improvement of about 1.6× for ResNet164 ( including the time to train and use the proxy ) . Taken together , these results demonstrate that SVP is a promising , yet simple approach to make data selection methods computationally feasible for deep learning . While we focus on active learning and core-set selection , SVP is widely applicable to methods that depend on learned representations . 2 METHODS . In this section , we describe SVP and show how it can be incorporated into active learning and core-set selection . Figure 1 shows an overview of SVP : in active learning , we retrain a proxy model APk in place of the target model ATk after each batch is selected , and in core-set selection , we train the proxy AP [ n ] rather than the target A T [ n ] over all the data to learn a feature representation and select points . 2.1 ACTIVE LEARNING . Pool-based active learning starts with a large pool of unlabeled data U = { xi } i∈ [ n ] where [ n ] = { 1 , . . . , n } . Each example is from the space X with an unknown label from the label space Y and is sampled i.i.d . over the space Z = X × Y as ( xi , yi ) ∼ pZ . Initially , methods label a small pool of points s0 = { s0j ∈ [ n ] } j∈ [ m ] chosen uniformly at random . Given U , a loss function ` , and the labels { ys0j } j∈ [ m ] for the initial random subset , the goal of active learning is to select up to a budget of b points s = s0 ∪ { sj ∈ [ n ] \ s0 } j∈ [ b−m ] to label that produces a model As with low error . Baseline . In this paper , we apply SVP to least confidence uncertainty sampling ( Settles , 2012 ; Shen et al. , 2017 ; Gal et al. , 2017 ) and the recent greedy k-centers approach from Sener & Savarese ( 2018 ) . Like recent work for deep active learning ( Shen et al. , 2017 ; Sener & Savarese , 2018 ; Kirsch et al. , 2019 ) , we consider a batch setting with K rounds where we select bK points in every round . Following Gal et al . ( 2017 ) ; Sener & Savarese ( 2018 ) ; Kirsch et al . ( 2019 ) , we reinitialize the target model and retrain on all of the labeled data from the previous k rounds to avoid any correlation between selections ( Frankle & Carbin , 2018 ; Kirsch et al. , 2019 ) . We denote this trained model as ATs0∪ ... ∪sk or just ATk for simplicity . Then using A T k , we either calculate the model ’ s confidence as : fconfidence ( x ; A T k ) = 1−max ŷ P ( ŷ|x ; ATk ) and select the examples with the lowest confidence or extract a feature representation from the model ’ s final hidden layer and compute the distance between examples ( i.e. , ∆ ( xi , xj ; ATk ) ) to select points according to the greedy k-centers method from Wolf ( 2011 ) ; Sener & Savarese ( 2018 ) ( Algorithm 1 ) . The same model is trained on the final b labeled points to yield the final model , ATK , which is then tested on a held-out set to evaluate error and quantify the quality of the selected data . Although other selection approaches exist , least confidence uncertainty sampling and greedy k-centers cover the spectrum of uncertainty-based and representativeness-based approaches for deep active learning . Other uncertainty metrics such as entropy or margin were highly correlated with confidence when using the same trained model ( i.e. , above a 0.96 Spearman ’ s correlation in our experiments on CIFAR ) . Query-by-committee ( Seung et al. , 1992 ) can be prohibitively expensive in deep learning , where training a single model is already costly . BALD ( Houlsby et al. , 2011 ) has seen success in deep learning ( Gal et al. , 2017 ; Shen et al. , 2017 ) but is restricted to Bayesian neural networks or networks with dropout ( Srivastava et al. , 2014 ) as an approximation ( Gal & Ghahramani , 2016 ) . Algorithm 1 GREEDY K-CENTERS ( WOLF , 2011 ; SENER & SAVARESE , 2018 ) Input : data xi , existing pool s0 , trained model AT0 , and a budget b 1 : Initialize s = s0 2 : repeat 3 : u = arg maxi∈ [ n ] \s minj∈s ∆ ( xi , xj ; A T 0 ) 4 : s = s ∪ { u } 5 : until |s| = b + |s0| 6 : return s \ s0 Algorithm 2 FORGETTING EVENTS ( TONEVA ET AL. , 2019 ) 1 : Initialize prev_acci = 0 , i ∈ [ n ] 2 : Initialize forgetting_eventsi = 0 , i ∈ [ n ] 3 : while training is not done do 4 : Sample mini-batch B from L 5 : for example i ∈ B do 6 : Compute acci 7 : if prev_acci > acci then 8 : forgetting_eventsi += 1 9 : prev_acci = acci 10 : Gradient update classifier on B 11 : return forgetting_events

Summary: The paper proposes a method for selecting a subset of a large dataset to reduce the computational costs of deep neural netwoks. The main idea is to train a proxy model, a smaller version of the full neural network, to choose important data points for active learning or core-set selection. Experiments on standard classification tasks demonstrate that this approach can yield substantial computational savings with only a small drop in accuracy.