Prioritized training on points that are learnable, worth learning, and not yet learned

1 INTRODUCTION . State-of-the-art models such as GPT-3 ( Brown et al. , 2020 ) , CLIP ( Radford et al. , 2021 ) , and ViT ( Dosovitskiy et al. , 2021 ) achieve remarkable results by training on vast amounts of web scraped data . Especially for large models , training on such datasets requires excessive training times that are measured in weeks or months rather than hours ( Radford et al. , 2021 ; He et al. , 2020 ) . Even practitioners who work with modestly sized models face long development cycles , due to numerous iterations of algorithm design and hyperparameter selection . These considerations highlight that the total time required for training is a core constraint in the development of such deep learning models . To speed up training , practitioners with sufficient resources use many more machines and much larger batches . However , there are rapidly diminishing returns to using larger batches and distributing them across more machines ( LeCun et al. , 2012 ; Keskar et al. , 2016 ) , to a point where adding machines does not reduce training time ( Anil et al. , 2018 ) . At the same time , practitioners realized that not all real-world data samples are equally useful . Many samples are noisy , i.e . mislabelled or inherently ambiguous . For example , the text associated with a web scraped image is rarely an accurate description of the image . Other samples are redundant , for example due to the typical overrepresentation of certain object classes in web scraped data ( Tian et al. , 2021 ) . Such redundant samples can often be left out without losing performance . In many cases , data is so abundant that state-of-the-art performance is reached in less than a single epoch ( Brown et al. , 2020 ; Kaplan et al. , 2020 ) . With such abundance , it can easily be afforded to skip less useful points . There are different approaches to selecting the most useful datapoints for training . Some bodies of literature , including curriculum learning , suggest prioritizing easy points with low label noise before training on all points equally ( Bengio et al. , 2009 ) . While this can improve convergence and generalization , it has no mechanism to avoid points that are already learned ( redundant ) . Other works instead suggest to train on points that are hard for the current model . In the optimization literature , online batch selection methods ( Loshchilov & Hutter , 2015 ) do this by selecting points with high loss or high gradient norm . Such online selection can successfully skip points that are already learned and focus on the few points where it is possible to make the greatest learning progress ( Figure 1 , left ) . However , we show that prioritising hard examples can be detrimental on noisy data . In real-world datasets , high loss examples may be mislabelled or inherently ambiguous . Indeed , in controlled experiments , with just 10 % uniform label noise , points selected by these methods are overwhelmingly those where the label is corrupted ( Figure 1 , right ) . Our results show that prioritising them degrades performance severely . Additionally , some samples are hard because they are outliers with unusual features , far from the other training samples . This type of difficult point can also lie far from the test data , which makes it less relevant for reducing the test loss . To overcome these limitations , we introduce reducible held-out loss selection ( RHOLS ) . We introduce a selection function—Predictive Information Content ( PIC ) —that quantifies the information each point confers about the labels of a holdout set . Approximating this , we derive an intuitive , tractable , and easy-to-implement novel selection function : reducible held-out ( cross-entropy ) loss . By maximizing the information gained about the labels of a held-out set , our method chooses non-redundant , relevant points with low noise—points that are “ just right ” . We explore the effects of reducible loss selection on five datasets , with and without additional label noise . We evaluate the reduction in required training steps compared to conventional uniform sampling and recent selection methods . Our method reaches target accuracy in 1.76x-3.82x fewer steps than conventional uniform sampling on CIFAR-10 , CIFAR-100 , and CINIC-10 . In the presence of just 10 % label noise , this speedup increases further to 2.34x-5.15x while the speed of state-of-the art prior methods degrades severely . On Clothing-1M , a large-scale dataset with noisy , web-scraped labels , our method reaches target accuracy in 12.5x fewer steps compared to uniform sampling and a 8 % higher final accuracy . 2 BACKGROUND : ONLINE BATCH SELECTION . Consider a model p ( y | x , θ ) trained on data D = { ( xi , yi ) } ni=1 using stochastic gradient descent . At each training step t , we load a batch bt of size nb from D. In online batch selection ( Loshchilov & Hutter , 2015 ) , we pre-sample a larger batch Bt of size nB > nb . Then , we construct a smaller batch bt that consists of the top-ranking nb points in Bt ranked by a label-aware selection function S ( xi , yi ) . We perform one gradient step to minimize a mini-batch loss ∑ i∈bt L ( yi , p ( yi | xi , θ ) ) . The following large batch Bt+1 is then pre-sampled from D without replacement of the previously sampled points ( points are replaced only at the start of the next epoch ) . We work with this approach for its simplicity , convergence guarantees , and strong performance in recent work ( Kawaguchi & Lu , 2020 ) . Further , it could be extended to incorporate stochastic data selection ( Jiang et al. , 2019 ) with importance weights ( Schaul et al. , 2015 ; Katharopoulos & Fleuret , 2018 ) or to include earlier points xi in Bt whose selection score S ( xi , yi ) has been previously computed ( Loshchilov & Hutter , 2015 ) . Without importance weights , the simple approach above biases the location of the minimum of the loss . However , this biased selection can improve test performance both in theory and practice ( Farquhar et al. , 2021 ; Kawaguchi & Lu , 2020 ) . Notation . For a holdout set { yhoi } i , { xhoi } i we will write yho , xho for brevity . Lower-case variables like y are outcomes and upper-case variables like Y are random variables . 3 REDUCIBLE HELD-OUT LOSS—A MEASURE OF PREDICTIVE INFORMATION CONTENT . Online batch selection methods construct batches at training time using points that have high loss or high gradient norm ( Loshchilov & Hutter , 2015 ; Katharopoulos & Fleuret , 2018 ) . In this section , we derive and discuss an alternative selection function , based on an information-theoretic perspective . Predictive Information Content ( PIC ) . Our overall objective is to select the point ( x , y ) ∈ D for training that minimizes the expected loss Eptrue ( x , y ) h [ y | x , Dt ∪ ( x , y ) ] of the model , where h [ y | x , Dt ] : = − log p ( y | x , Dt ) ( 1 ) denotes the cross-entropy loss on ( x , y ) of a model trained on Dt . As a proxy , we aim to maximize the reduction in the Monte Carlo expectation of the cross-entropy loss on a held-out dataset Dho , 1 |Dho| ∑ i∈Dho h [ yhoi | xhoi , Dt ] − h [ yhoi | xhoi , Dt ∪ ( x , y ) ] . ( 2 ) For a deterministic model ( i.e . with fixed parameters ) , this held-out loss ( to a constant factor ) equals h [ yho | xho , Dt ] − h [ yho | xho , Dt ∪ ( x , y ) ] . ( PIC-1 ) ( 3 ) We term this the Predictive Information Content ( PIC ) . It quantifies predictive information : the part of the information gained by observing ( x , y ) that is applicable to predicting the holdout labels . It also relates to the Expected Predictive Information Gain ( EPIG ) recently proposed for active learning ( Kirsch et al. , 2021 ) . In active learning , one selects points without access to their labels , and therefore Kirsch et al . ( 2021 ) must compute the expectation of ( 3 ) over a posterior predictive p ( Yho | xho , Dt ) . However , here we quantify not the expected but the actual information gained when observing ( x , y ) . We therefore follow naming conventions and term ( 3 ) a ( predictive ) information content . See Appendix B for more details on the relationship to active learning and EPIG . Deriving a tractable selection function : the irreducible and reducible losses . The two terms in ( 3 ) can not be computed efficiently ; they require first training the model on each potential point ( x , y ) and then performing a forward pass with the model on the full holdout set . We derive an alternative , more tractable expression . To make our claims precise and our assumptions transparent , we use the language of Bayesian probability theory to formalise our claims . In the Bayesian context , we treat model parameters as a random variable with prior p ( θ ) and infer a posterior p ( θ|Dt ) using the sequence of already-seen training data Dt = b1 : t−1 . The model has a predictive distribution p ( y|x , Dt ) = ∫ θ p ( y|x , θ ) p ( θ|Dt ) dθ . When using a point estimate of θ ( such as an MLE or MAP ) , the predictive distribution can still be written as an integral , but with respect to a Dirac delta at the point estimate as an approximation of the true posterior . Applying Bayes rule to p ( yho | y , xho , x , Dt ) and using the conditional independence assumption p ( yi | xi , xj , Dt ) = p ( yi | xi , Dt ) , we can reformulate PIC as h [ y | x , Dt ] − h [ y | x , yho , xho , Dt ] , ( PIC-2 ) ( 4 ) moving the holdout set to the conditional . We show this derivation in Appendix A . The first term is simply the cross-entropy loss on the potential point ( x , y ) of a model trained on Dt . The second term—the cross-entropy loss of a model trained on Dt and the held-out set—adds computation as it requires constant updating with increasing Dt . We can approximate this second term by training a model only on the held-out dataset , and refer to it as the irreducible held-out loss : h [ y | x , yho , xho ] . ( irreducible held-out loss ) ( 5 ) This approximation may be reasonable when |Dt| |Dho| ( e.g . at the start of training ) or when a small model p ( y | yho , xho ) trained on Dho has limited capacity to improve further by training on Dt . Empirically , we find this approximation does not affect performance ( Appendix D ) . We refer to Eq . ( 5 ) as the irreducible held-out loss since it is the portion of the loss that could not be reduced by training on the held-out set . Plugging this approximation into Eq . ( 4 ) , we get the reducible held-out loss , h [ y | x , Dt ] − h [ y | x , yho , xho ] . ( reducible held-out loss ) ( 6 ) This provides an easy-to-compute selection function that approximates the Predictive Information Content between Dho and the datapoint ( x , y ) for a model trained on Dt . Understanding reducible loss . We now provide intuition on why the reducible held-out loss selects non-redundant , non-noisy points . The reducible held-out loss contains two terms . The first term , the loss , is high on points difficult for the current model i.e . points that are not yet learned and thus not redundant . Indeed , existing online batch selection methods use loss for selection ( Loshchilov & Hutter , 2015 ) . However , not all high loss points are informative—some may be noisy or mislabelled . Crucially , these points can not be predicted using the holdout set and thus have high irreducible loss . Therefore , these non-information points have low reducible loss . Indeed , we show that the cross-entropy loss disproportionally selects such noisy points in Figure 1 whereas selecting points with high reducible loss ( or low irreducible loss ) avoids noisy points . In short , RHOLS prioritises points that are not redundant ( high loss ) and not noisy ( low irreducible loss ) . Loss based selection has additional pitfalls . The loss is likely higher for outliers—values of x far from the training data , such as those in a region with low input density ptrue ( x ) . Since our aim is to reduce the expected loss Eptrue ( x , y ) [ h [ y | x , Dt ] ] under ptrue ( x , y ) , we do not want to prioritize less relevant outliers . Since such points are also less likely in the held-out data , the irreducible loss can also be expected to be higher . Thus , reducible held-out loss should select points that are more ‘ relevant ’ to the holdout set . We find empirical evidence for this below . Selecting whole batches . We developed the theory assuming the selection of a single point ( x , y ) . When selecting entire batches , we select the points with the top-nb scores from a randomly sampled set Bt of candidate examples . This assumes that each point has little effect on the scores of other points , and is often used as a simplifying assumption for selection in the context of active learning ( Kirsch et al. , 2019 ) . This is a much more reasonable assumption in our case than in active learning , because model predictions are not changed much by a single gradient step on one mini-batch . Algorithm 1 shows the implementation of batch selection using the reducible held-out loss . Note we precompute and store the irreducible losses before training . This way , we avoid recomputing them in every epoch and across different training runs . Algorithm 1 Reducible held-out loss selection ( RHOLS ) 1 : Input : Learning rate η , model p ( y | x , Dho ) trained on a holdout set Dho , batch size nb , large batch size nB > nb . 2 : for ( xi , yi ) in training set do 3 : IrreducibleLoss [ i ] ← h [ yi | xi , Dho ] ) 4 : end for Initialize parameters θ0 and t = 0 5 : for t = 1 , 2 , . . . do 6 : Randomly select a large batch Bt of size nB . 7 : ∀i ∈ Bt , compute Loss [ i ] , the loss of point i given parameters θt 8 : ∀i ∈ Bt , compute ReducibleLoss [ i ] ← Loss [ i ] − IrreducibleLoss [ i ] 9 : bt ← top-nb samples in Bt in terms of ReducibleLoss . 10 : gt ← mini-batch gradient on bt using parameters θt 11 : θt+1 ← θt − ηgt Simple parallelized selection . For large-scale neural network training , practitioners with sufficient resources would use many more machines if it further sped up training ( Anil et al. , 2018 ) . However , as more workers are added in synchronous or asynchronous gradient descent , the returns diminish to a point where adding more workers does not further improve wall clock time ( Anil et al. , 2018 ) . For example , there are rapidly diminishing returns for using larger batch sizes or distributing a given batch across more workers ( LeCun et al. , 2012 ; Keskar et al. , 2016 ) . The same holds for distributing the model across more workers along its width or depth ( Rasley et al. , 2020 ; Shoeybi et al. , 2019 ; Huang et al. , 2019 ) . However , we can circumvent these diminishing returns by adding a new dimension of parallelization , namely , for data selection . Since parallel forward passes do not suffer from such diminishing returns , one can easily use extra workers for them . The theoretical runtime speedup can be calculated as follows . The computational cost of selection is nb3nB times as much as the cost of a training step on bt . This is because forward passes require 3x less computation than a full forward-backward pass ( Jouppi et al. , 2017 ) . One can accelerate this computation almost arbitrarily by adding more workers that compute losses using a copy of the model being trained . The limit is reached when the time for selection is dominated by the cost of communicating parameter updates to workers . More sophisticated parallelization strategies allow reducing the time overhead even further , see Section 4 . To avoid assumptions about the particular strategy used , we report experiment results in terms of the required number of steps .

Summary: The authors propose a reducible held-out loss selection method for faster model training. Their method selects a sequence of training points that are referred to as "just right" for training.  The main methodological contribution of the paper is an information-theoretic selection criterion Predictive Information Content (PIC) to choose training points. Authors emprically show efficacy of their selection criterion over previously proposed baselines.