How Important is Importance Sampling for Deep Budgeted Training?

1 INTRODUCTION . The availability of vast amounts of labeled data is crucial in training deep neural networks ( DNNs ) ( Mahajan et al. , 2018 ; Xie et al. , 2020 ) . Despite prompting considerable advances in many computer vision tasks ( Yao et al. , 2018 ; Sun et al. , 2019a ) , this dependence poses two challenges : the generation of the datasets and the large computation requirements that arise as a result . Research addressing the former has experienced great progress in recent years via novel techniques that reduce the strong supervision required to achieve top results ( Tan & Le , 2019 ; Touvron et al. , 2019 ) by , e.g . improving semi-supervised learning ( Berthelot et al. , 2019 ; Arazo et al. , 2020 ) , fewshot learning ( Zhang et al. , 2018b ; Sun et al. , 2019b ) , self-supervised learning ( He et al. , 2020 ; Misra & Maaten , 2020 ) , or training with noisy web labels ( Arazo et al. , 2019 ; Li et al. , 2020a ) . The latter challenge has also experienced many advances from the side of network efficiency via DNN compression ( Dai et al. , 2018 ; Lin et al. , 2019 ) or , neural architecture search ( Tan & Le , 2019 ; Cai et al. , 2019 ) ; and optimization efficiency by better exploiting the embedding space ( Khosla et al. , 2020 ; Kim et al. , 2020 ) . All these approaches are designed under a common constraint : the large dataset size needed to achieve top results ( Xie et al. , 2020 ) , which conditions the success of the training process on computational resources . Conversely , a smart reduction of the amount of samples used during training can alleviate this constraint ( Katharopoulos & Fleuret , 2018 ; Mirzasoleiman et al. , 2020 ) . The selection of samples plays an important role in the optimization of DNN parameters during training , where Stochastic Gradient Descent ( SGD ) ( Dean et al. , 2012 ; Bottou et al. , 2018 ) is often used . SGD guides the parameter updates using the estimation of model error gradients over sets of samples ( mini-batches ) that are uniformly randomly selected in an iterative fashion . This strategy assumes equal importance across samples , whereas other works suggest that alternative strategies for revisiting samples are more effective in achieving better performance ( Chang et al. , 2017 ; Kawaguchi & Lu , 2020 ) and faster convergence ( Katharopoulos & Fleuret , 2018 ; Jiang et al. , 2019 ) . Similarly , the selection of a unique and informative subset of samples ( core-set ) ( Toneva et al. , 2018 ; Coleman et al. , 2020 ) can alleviate the computation requirements during training , while reducing the performance drop with respect to training on all data . However , while removing data samples speeds-up the training , a precise sample selection often requires a pretraining stage that hinders the ability to reduce computation ( Mirzasoleiman et al. , 2020 ; Sener & Savarese , 2018 ) . A possible solution to this limitation might be to dynamically change the important subset during training as done by importance sampling methods ( Amiri et al. , 2017 ; Zhang et al. , 2019b ) , which select the samples based on a sampling probability distribution that evolves with the model and often changes based on the loss or network logits ( Loshchilov & Hutter , 2015 ; Johnson & Guestrin , 2018 ) . An up-to-date importance estimation is key for current methods to succeed but , in practice , is infeasible to compute ( Katharopoulos & Fleuret , 2018 ) . The real importance of a sample changes after every iteration and estimations become out-dated , yielding considerable drops in performance ( Chang et al. , 2017 ; Zhang et al. , 2019b ) . Importance sampling methods , then , focus on selecting samples and achieve a speed-up during training as a side effect . They do not , however , strictly study possible benefits on DNN training when restricting the number of iterations used for training , i.e . the budget . Budgeted training ( Nan & Saligrama , 2017 ; Kachuee et al. , 2019 ; Li et al. , 2020b ) imposes an additional constraint on the optimization of a DNN : a maximum number of iterations . Defining this budget provides a concise notion of the limited training resources . Li et al . ( 2020b ) propose to address the budget limitation using specific learning rate schedules that better suit this scenario . Despite the standardized scenario that budgeted training poses to evaluate methods when reducing the computation requirements , there are few works to date in this direction ( Li et al. , 2020b ; Katharopoulos & Fleuret , 2018 ) . As mentioned , importance sampling methods are closely related , but the avoidance of budget restrictions makes it difficult to understand their utility given the sensitivity to hyperparamenters that they often exhibit ( Chang et al. , 2017 ; Loshchilov & Hutter , 2015 ) . In this paper , we overcome the limitations outlined above by analyzing the effectiveness of importance sampling methods when a budget restriction is imposed ( Li et al. , 2020b ) . Given a budget restriction , we study synergies among important sampling , and data augmentation ( Takahashi et al. , 2018 ; Cubuk et al. , 2020 ; Zhang et al. , 2018a ) . We find the improvements of importance sampling approaches over uniform random sampling are not always consistent across budgets and datasets . We argue and experimentally confirm ( see Section 4.4 ) that when using certain data augmentation ( Takahashi et al. , 2018 ; Cubuk et al. , 2020 ; Zhang et al. , 2018a ) , existing importance sampling techniques do not provide further benefits , making data augmentation the most effective strategy to exploit a given budget . 2 RELATED WORK . Few works exploit a budgeted training paradigm ( Li et al. , 2020b ) . Instead , many approaches aim to speed up the training convergence to a given performance by computing a better sampling strategy or carefully organizing the samples to allow the CNN to learn faster and generalize better . Other works , however , explore how to improve model performance by labeling the most important samples from an unlabeled set of data ( Yoo & Kweon , 2019 ; Ash et al. , 2020 ; Ren et al. , 2020 ) or how to better train DNNs when a limited number of samples per class is available ( Chen et al. , 2019 ; Zhou et al. , 2020 ; Albert et al. , 2020 ) . This section reviews relevant works aiming to improve the efficiency of the DNN training . Self-paced learning ( SPL ) and curriculum learning ( CL ) aim to optimize the training process and improve model performance by ordering the samples from easy to difficult ( Weinshall et al. , 2018 ; Bengio et al. , 2009 ; Hacohen & Weinshall , 2019 ; Cheng et al. , 2019 ) . For instance , CL manages to speed the convergence of the training at the initial stages due to focusing on samples whose gradients are better estimations of the real gradient ( Weinshall et al. , 2018 ) . The main drawback of these methods is that , in most of the cases , the order of the samples ( curriculum ) has to be defined before training , which is already a costly task that requires manually assessing the sample difficulty , transferring knowledge from a fully trained model , or pre-training the model on the given dataset . Some approaches remedy this drawback with a simple curriculum ( Lin et al. , 2017 ) or by learning the curriculum during the training ( Jiang et al. , 2018 ) ; these methods , however , do not aim to speed up the training by ordering the samples , but to improve network convergence by weighting the sample contribution to the loss . Core-set selection approaches aim to find the subset of samples that is most useful ( Toneva et al. , 2018 ; Coleman et al. , 2020 ; Mirzasoleiman et al. , 2020 ) . By identifying the most useful samples from a dataset , these methods aim at maintaining accuracy despite training in a subset of the data . The ability of these methods to reduce the training cost is very limited , since they require pre-training the model . However , these methods demonstrate that DNNs only need a portion of the samples to achieve peak performance . For example , Toneva et al . ( 2018 ) define “ forgetting events ” as the count of times that samples are miss-classified after being correctly predicted during training . They show that higher forgetting and importance are related , as removing samples with lower forgetting events damages the model less than removing the more forgotten ones . Mirzasoleiman et al . ( 2020 ) build clusters with the features from the model and use the centroids as the most informative samples . Coleman et al . ( 2020 ) demonstrate that the difficulty of a sample is invariant to the model capacity and show that they can speed up several sample selection tasks by reducing the size of the model . Importance sampling approaches lie in the middle ground between the previous two : they aim to speed up training convergence by leveraging the most useful samples at every training stage ( Katharopoulos & Fleuret , 2018 ; Jiang et al. , 2019 ; Zhang et al. , 2019b ) – which correspond to sample losses with highest gradient magnitude ( Needell et al. , 2014 ; Zhao & Zhang , 2015 ; Alain et al. , 2016 ) . More recently , Johnson & Guestrin ( 2018 ) has shown that the last layer gradients are a good approximation and are easier to obtain in deep learning frameworks . Alternative importance measures often used include the loss ( Jiang et al. , 2019 ) , the probability predicted for the true class ( Chang et al. , 2017 ) , or the ranking order of these probabilities ( Loshchilov & Hutter , 2015 ) . The approximation of the optimal distribution by importance sampling approaches avoids the cost of computing each sample importance at every iteration . However , they face one main challenge : the optimal sampling distribution changes very rapidly between iterations , leading to outdated estimations . Initial attempts on addressing this challenge included several hyper-parameters to smooth the estimated distribution ( Chang et al. , 2017 ) , more frequent distribution updates via additional forward passes ( Loshchilov & Hutter , 2015 ) , or different alternative measures to estimate the sampling distribution ( Amiri et al. , 2017 ) . Several works added complex support techniques to the training that aimed to estimate a better distribution : using robust optimization ( Johnson & Guestrin , 2018 ) , introducing repulsive point techniques ( Zhang et al. , 2019a ) , or adding a second network to be trained in parallel with the main model Zhang et al . ( 2019b ) . More recent methods leverage the random-then-greedy technique ( Lu & Mazumder , 2018 ) , where a random initial batch of samples is selected and then the probabilities of those samples are computed and used to select a secondary batch that is used for training . Within this scheme , ( Katharopoulos & Fleuret , 2018 ) define a theoretical bound for the magnitude of the gradients that allows for faster computation of the sampling probabilities and ( Jiang et al. , 2019 ) and ( Ioannou et al. , 2019 ) use the loss as a measure of sample importance to keep the sampling distribution updated through the training . Finally , ( Kawaguchi & Lu , 2020 ) introduces the top-k loss from ( Fan et al. , 2017 ) to perform the back-propagation step using the samples with highest losses only . Note that none of these methods avoids doing a full forward pass every epoch to update the sampling probabilities . Learning rate schedules have proven to be useful alternatives for faster convergence . The authors in ( Smith & Topin , 2019 ; Smith , 2017 ) propose a cyclic learning rate schedule to reach faster convergence by using larger learning rates at intermediate training stages and very low rates at the end . Li et al . ( Li et al. , 2020b ) also study the importance of the learning rate schedules to accelerate the training of DNNs . In particular , they explore budgeted training and propose a linearly decaying learning rate schedule that approaches zero at the end of the training , which without additional hyper-parameters , improves the standard learning rate schedules . Data augmentation techniques , generally , aim to increase the variance of the data to achieve better generalization . Recent approaches , however , go a step further and target specific weaknesses from CNNs : cutout ( DeVries & Taylor , 2017 ) drops contiguous patches of data from the input to force the network to spread its attention over the entire object , mixup ( Zhang et al. , 2018a ) proposes to train using convex combinations of images and label pairs which smooth class boundaries and improve model calibration ( Thulasidasan et al. , 2019 ) , and RICAP ( Takahashi et al. , 2018 ) combines the advantages of the two previous techniques by training on images generated from joining multiple patches and doing the corresponding convex combination of labels . More generally , RandAugment ( Cubuk et al. , 2020 ) randomly combines commonly used data augmentation techniques as a reduction of the search space of the recently proposed methods that find automated augmentation policies ( Ho et al. , 2019 ; Cubuk et al. , 2019 ) .

Summary: This paper investigates the use of importance sampling in budgeted training.  Four importance sampling techniques from prior works are applied within the context of fixed training budgets, and compared under different conditions of training set selection, learning rate schedule and data augmentations.  Each aims to sample more useful examples more frequently, by using the loss or gradient magnitude as an importance measure.  Uniform sampling with and without replacement are used as baselines, and experiments are performed on cifar-10 and cifar-100.  The final conclusion is that importance sampling with budgets as low as 20% the original training schedule offer little if any improvement over uniform sampling, while additional data augmentations work well to make up lost validation accuracy.