Select Via Proxy: Efficient Data Selection For Training Deep Networks

1 INTRODUCTION . Large-scale labeled data has been critical to the recent success of deep learning ( Halevy et al. , 2009 ; Sun et al. , 2017 ; Hestness et al. , 2017 ) and there are a variety of domains and settings where labeled data is plentiful , such as predicting the next word or character in language modeling and predicting user-provided image tags for image classification . However , for these large datasets , training deep networks can incur prohibitively long training times , measured in days , weeks , or even months ( Sun et al. , 2017 ) . This overhead impedes the development of new machine learning models and uses large amounts of computational resources ( Amodei & Hernandez , 2018 ) . Subsampling training data is a common solution to this problem , but naive , uniform subsampling can miss important rare examples . For instance , in many web applications , data is abundant for a small subset of core users/content but is scarce for new users/content . Similarly , when training autonomous vehicles , yellow lights occur less frequently than green and red lights but are equally important ( Karpathy , 2018 ) . While more sophisticated methods such as core-set selection techniques can select a representative subset , these methods are either algorithm-specific ( Har-Peled & Kushal , 2007 ; Tsang et al. , 2005 ; Huggins et al. , 2016 ; Campbell & Broderick , 2017 ; 2018 ) and don ’ t apply to deep learning , or require a meaningful feature representation ( Wei et al. , 2013 ; 2014 ; Tschiatschek et al. , 2014 ; Ni et al. , 2015 ) that must either be hand-designed or learned for unstructured data , which requires substantial computational resources . In this paper , we introduce a new method called Select Via Proxy ( SVP ) that provides a computationally efficient way of training large deep learning models while empirically maintaining model quality/performance . Our key idea is to quickly select a subset of the most informative data and then train a large model on this subset . Training with less data leads to faster training times , while training on the most informative data allows the model to learn as well as if it were trained on the full data . To identify informative data points we first quickly train a small proxy model , and then use this proxy to select data for training the large full model . The proxy model is a simple and fast-to-train model—for example , a small model architecture ( e.g. , ResNet20 instead of ResNet164 ) trained for a few epochs ( e.g. , 25 % of the desired epochs ) . We subsequently select a training set for the larger model based on the predictions generated by this smaller proxy model . This selection process can be thought of as a preprocessing step that filters data before feeding it into an existing training pipeline , as depicted in Figure 1 . To select the points with highest informativeness , we leverage uncertainty sampling ( Lewis & Gale , 1994 ) from active learning , where uncertainty can be measured by metrics such as entropy of the output probabilities . However , in active learning a model is generally trained to select the next point ( or batch ) ( Settles , 2012 ) , which is efficient in terms of labels , but often computationally expensive . While this can be effective when deciding which data to acquire labels for from an expensive labeler ( e.g . a human ) , the computational cost is too high to accelerate training over an existing large labeled dataset . Using a proxy reduces the cost of selection by up to a 100× . The main benefits of our proposed method are ( i ) it can be easily added to a training pipeline without modifying the training procedure of the target model and ( ii ) the proxy is very fast to train and can substantially improve the training time of large deep models while maintaining the predictive performance . Extensive experimental evaluation shows the effectiveness of our approach . In general , the proxy is very fast to train . As a result , including the time to train the proxy , select points using the trained proxy , and train the large target model on the selected points , we find that SVP can substantially reduce training time compared to training over all the data while maintaining the same predictive performance . SVP leads to a 1.6× and 1.8× speed-up on CIFAR10 ( Krizhevsky & Hinton , 2009 ) and SVHN ( Netzer et al. , 2011 ) by selecting 60 % and 50 % subsets of the data respectively while maintaining predictive performance . To explore the robustness of our proposed framework to the choice of proxy model architecture , we combine two approaches for sentiment analysis that represent separate extremes in speed and accuracy . Our experiments show that SVP with fastText ( Joulin et al. , 2016 ) as a proxy model can be used to remove 20 % of the data from Amazon Review Polarity ( Zhang et al. , 2015 ; He & McAuley , 2016 ) for the target model VDCNN29 ( Conneau et al. , 2017 ) , a fundamentally different architecture from the proxy that is over 100× slower to train ( e.g. , 17 hours instead of 10 minutes ) . This provides a simple but effective means of decreasing training time without increasing error . 2 RELATED WORK . SVP builds upon a large body of related work on sampling and training data selection . Core-set selection . Core-set selection attempts to find a representative subset of points to speed up learning or clustering ; such as k-means and k-medians ( Har-Peled & Kushal , 2007 ) , SVM ( Tsang et al. , 2005 ) , Bayesian logistic regression ( Huggins et al. , 2016 ) , and Bayesian inference ( Campbell & Broderick , 2017 ; 2018 ) . However , these examples are algorithm-specific and do not directly apply to deep neural networks . Very recently , Sener & Savarese ( 2018 ) used core-set selection for active learning with convolutional neural networks . While this technique reduces sample complexity , the proposed technique is computationally intensive and does not save on training time . There is a body of work on data summarization based on submodular maximization ( Wei et al. , 2013 ; 2014 ; Tschiatschek et al. , 2014 ; Ni et al. , 2015 ) . However , the techniques here are rather different as they do not incorporate a model prediction . Subset selection to increase accuracy . Recently , Chang et al . ( 2017 ) proposed to choose data points whose predictions have changed most over the previous epochs as a lightweight estimate of uncertainty . From the machine teaching literature , Fan et al . ( 2018 ) demonstrated that data selection can be learned through reinforcement learning . By repeatedly training a student model over the dataset or similar dataset , they train a teacher model to filter points for each update of the student model using the student ’ s final accuracy and training time as a reward . In comparison to our method , both of these approaches have substantial burn in periods ( i.e. , building a history of predictions or training the teacher ) before data can be effectively selected . In addition , Wang et al . ( 2018b ) trains a model , selects a subset , and then retrains the same model on a subset of the data where a small fraction of unfavorable training examples ( < 5 % ) are removed to give higher accuracy . The goal of the above work is to increase the accuracy of training rather than speed up training . However , these results make it less surprising that we can achieve the same accuracy with a subset of the full dataset even though we are able to remove up to 12× more points . Heterogeneous active learning . In the active learning literature , there are examples of using one model to select points for a different , more expensive model . For instance , ( Lewis & Catlett , 1994 ) uses a probabilistic classifier to select points to label for a decision tree target model . Tomanek et al . ( 2007 ) uses a committee-based active learning algorithm for an NLP task and notes that the set of selected points are “ reusable ” across different models ( maximum entropy , conditional random field , naive Bayes ) . In light of this work , we demonstrate this phenomena generalizes to modern deep learning models where model capacity and training time can be substantially reduced to effectively select informative examples . Optimization and Importance Sampling . There is a large literature on weighting or sampling datapoints in the optimization procedure in order to speed up training or achieve higher accuracy . The most common paradigm is to perform importance sampling on training data points based on the gradient norm ( Alain et al. , 2015 ) , loss ( Loshchilov & Hutter , 2015 ) , bound on the gradient norm ( Katharopoulos & Fleuret , 2017 ) , or approximate loss ( Katharopoulos & Fleuret , 2018 ) . There has also been work on learning the weights or sampling probabilities to minimize the variance of the stochastic gradient ( Bouchard et al. , 2015 ; Borsos et al. , 2018 ; Gopal , 2016 ) . In the theoretical optimization literature , there is work on importance sampling data points for faster convergence ( Needell et al. , 2014 ; Zhao & Zhang , 2015 ; Allen-Zhu et al. , 2016 ) . There has also been a line of work in the Neural Machine Translation literature focused on reducing training time by focusing more on “ hard ” examples in later epochs ( Zhang et al. , 2017 ; van der Wees et al. , 2017 ; Wang et al. , 2018a ; Kocmi & Bojar , 2017 ) . In comparison , our approach does not modify the training procedure of the target model , making it an easy addition to an existing training pipeline . We note that our proposed method is orthogonal to the above . SVP could be used in conjunction with the algorithms above to further decrease training time or reduce computational complexity in the case of active learning and core-set selection , but we leave this for future work . 3 SELECT VIA PROXY . We now present our main contribution , Select via Proxy ( SVP ) , for efficiently training expensive deep learning models . Our approach consists of three steps : 1 ) Create a proxy model that is fast to train and to provide us with an approximate decision boundary , 2 ) use the proxy model to select a subset of uncertain data points around the decision boundary , and 3 ) train the large target model on the selected subset via proxy to refine the decision boundary and get the final accurate model . In the following section , we discuss each step in detail . 3.1 CREATING A PROXY MODEL . The key idea behind the proxy model is to create a small model that is fast to train and can provide a good approximation of the decision boundary of the large target model . In order to create such a model we rely on the following observations . Creating a proxy by scaling down the target model . It has been observed that for deep models with many layers , reducing the dimension ( narrowing ) or number of hidden layers ( shortening ) leads to a considerably reduced training times with only a small drop in accuracy . For example , in image classification , the accuracy of deep networks with residual connections ( e.g. , ResNet164 and ResNet110 ) only slightly diminishes as layers are dropped from the network ( He et al. , 2016b ; a ) . As Figure 2a shows , a model with 20 layers achieves an accuracy of 92.1 % in 22 minutes , while a larger model with 164 layers only improves performance by 2.4 % , but takes 3 hours and 40 minutes to train . Similar results have been shown for scaling down networks with a variety of model architectures ( Huang et al. , 2016 ; Xie et al. , 2017 ; Huang et al. , 2017 ) and a number of other tasks including language modeling , neural machine translation , text classification , and recommendation ( Conneau et al. , 2016 ; He et al. , 2017 ; Jozefowicz et al. , 2016 ; Dauphin et al. , 2017 ; Vaswani et al. , 2017 ) . We exploit the diminishing returns property between training time and reductions in error to scale down a given target model to a small proxy that can be trained quickly but still provides a good approximation of the decision boundary of the target model . Training for a smaller number of epochs . As shown in Figure 2b , a significant amount of training is spent to obtain a relatively small reduction in error . While training ResNet20 , almost half of the training time ( i.e. , 10 minutes out of 22 minutes ) is spent on a 1.3 % improvement in error . Based on the above observation , we can train the proxy model for a smaller number of epochs and still get a good approximation from the decision boundary of the target model . Boosting the performance by ensembling small models . Boosting is a common approach to get a strong learner by ensembling a set of weak learners . Similarly , a stronger proxy can be obtained by training a set of small models and combining their predictions . The stronger proxy can provide a better approximation from the decision boundary of the target model . Considering that the much smaller size of the proxy model compared to the size of the target model , multiple proxy models can be trained with little or no additional resources in parallel , with no increase to overall training time . Each learner within the ensemble has its own noisy approximation of the decision boundary . Hence , the small learners can be combined together through alternating selection or rank combination ( Settles , 2011 ; Reichart et al. , 2008 ) . Figure 3 shows that ensembling improves the already high correlation between the proxy model ’ s ranking of points based on entropy and the same ranking produced by the model trained on the whole data .

Summary: This paper studies a very simple and intuitive method to boost the training speed of deep neural networks. The authors first train some light weighted proxy models, using these models to rank the data according to its uncertainty, and then pick the most uncertain subset to train the final model. Experiments on CIFAR10/SVHN/Amazon Review Polarity demonstrates the effectiveness.