Better Generalization with On-the-fly Dataset Denoising

1 INTRODUCTION . Over-parametrized deep neural networks have remarkable generalization properties while achieving near-zero training error ( Zhang et al. , 2016 ) . However , the ability to fit the entire training set is highly undesirable , as a small portion of mislabeled examples in the dataset could severely hurt generalization ( Zhang et al. , 2016 ; Arpit et al. , 2017 ) . Meanwhile , an exponential growth in training data size is required to linearly improve generalization in vision tasks ( Sun et al. , 2017 ) ; this progress could be hindered if there are mislabeled examples within the dataset . Mislabeled examples are to be expected in large datasets that contain millions of examples . Webbased supervision produces noisy labels ( Li et al. , 2017a ; Mahajan et al. , 2018 ) ; whereas human labeled datasets sacrifice accuracy for scalability ( Krishna et al. , 2016 ) . Therefore , algorithms that are robust to various levels of mislabeled examples are warranted in order to further improve generalization for very large labeled datasets . In this paper , we propose On-the-fly Data Denoising ( ODD ) , a simple and robust method for training with noisy examples based on the implicit regularization effect of stochastic gradient descent . First , we train residual networks with large learning rate schedules and use the resulting losses to separate clean examples from mislabeled ones . This is done by identifying examples whose losses exceed a certain threshold . Reasonable thresholds can be derived from the loss distribution for uniform label noise which does not depend on the amount of mislabeled examples in the dataset . Finally , we remove these examples from the dataset and continue training until convergence . Empirically , ODD performs significantly better than previous methods in datasets containing artificial noise ( Sections 4.1 and 4.2 ) or real-world mislabeled examples ( Section 4.3 ) , while achieving equal or better accuracy than the state-of-the-art on clean datasets ( Sections 4.1 and 4.2 ) . We further conduct ablation studies to demonstrate that ODD is robust w.r.t hyperparameters and artificial noise levels ( Section 4.4 ) . Our method is also able to detect mislabeled examples in the CIFAR-100 dataset without any additional supervision ( Figure 1 ) . 2 DENOISING DATASETS ON-THE-FLY . The goal of supervised learning is to find a function f ∈ F that describes the probability of a random label vector Y ∈ Y given a random input vector X ∈ X , which has underlying joint distribution P ( X , Y ) . Given a loss function ` ( y , ŷ ) , one could minimize the average of ` over P : R ( f ) = ∫ ` ( y , f ( x ) ) dP ( x , y ) The joint distribution P ( X , Y ) is usually unknown , but we could gain access to its samples via a potentially noisy labeling process , such as crowdsourcing ( Krishna et al. , 2016 ) or web queries ( Li et al. , 2017a ) . We denote the training dataset with N examples as D = ( xi , yi ) i∈ [ N ] = G ∪ B. G represents correctly labeled ( clean ) examples sampled from P ( X , Y ) . B represents mislabeled examples that are not sampled from P ( X , Y ) , but from another distribution Q ( X , Y ) ; G ∩ B = ∅ . We aim to learn the function f from D without knowledge about B , G or their statistics ( e.g . |B| ) . A typical approach is to assume that B = ∅ , i.e . all examples are i.i.d . from P ( X , Y ) , and minimizing the following objective : R̂ ( f ) = − 1 N N∑ i=1 ` ( y , f ( x ) ) If B = ∅ is indeed true , then R̂ ( f ) → R ( f ) as N → ∞ . However , this is not true if B 6= ∅ since D is no longer an unbiased population of P ( X , Y ) . Moreover , when F is the space of large neural networks with parameters exceeding N , f could fit the entire training dataset ( Zhang et al. , 2016 ) , including the mislabeled examples . This results in undesired behavior of f on inputs of the mislabeled set , let alone outside the training data . To illustrate the harm of mislabeled examples to generalization , we consider training on CIFAR-10 where some examples are mislabeled uniformly at random . Compared with training on D , training only on G could decrease validation error from 11.53 to 4.25 if there are 20 % mislabeled examples , and from 15.57 to 5.06 if there are 40 % mislabeled examples1 . Therefore , if we are able to identify examples that belong to G , we could vastly improve generalization on P ( X , Y ) . 2.1 SEPARATING MISLABELED EXAMPLES VIA IMPLICIT REGULARIZATION . Fortunately , in the case of classification with deep residual networks ( He et al. , 2015a ) , the implicit generalization of stochastic gradient descent ( SGD ) with large learning rates ( e.g . 0.1 ) can separate examples from G and examples from B via the loss statistics . We demonstrate this in Figure 2 , where we train deep residual networks on CIFAR-100 and ImageNet with different percentages of uniform label noise . In early stages of training , the loss distributions of clean examples and mislabeled ones have notable statistical distance . The network starts to fit mislabeled examples when learning rate starts to decrease , which is also crucial for achieving better generalization on clean datasets . The working of the implicit regularization of gradient descent is by and large an open question that attracts much recent attentions ( Neyshabur , 2017 ; Li et al. , 2017b ; Du et al. , 2018 ) . Empirically , it has been observed that large learning rates are beneficial for generalization ( Kleinberg et al. , 2018 ) . Recent work has shown that the stationary distribution of SGD iterates corresponds to an Ornstein-Uhlenbeck process ( Uhlenbeck & Ornstein , 1930 ) with noise proportional to the learning rate ( Mandt et al. , 2017 ) . Training with large learning rates would then encourage solutions that are more robust to large random perturbations in the parameter space and less likely to overfit to mislabeled examples . Therefore , given these empirical and theoretical evidence on large learning rate helps generalization , we propose to classify correct and mislabeled examples through the loss statistics , and achieve better generalization by removing the examples that are potentially mislabeled and training on clean examples only . 1Validation error is 3.73 for 0 % mislabeled case . Additional details for CIFAR-100 in Table 1 . 2.2 THRESHOLDS THAT CLASSIFY MISLABELED EXAMPLES . To improve generalization in practice , one critical problem is to select a reasonable threshold for classification . High thresholds could include too many examples from B , whereas low thresholds could prune too many examples from G ; reasonable thresholds should also adapt to different unknown ratios of mislabeled examples . Let us first consider the case where Q ( Y |X ) has the highest entropy , which is the uniform distribution over labels . From Figure 2 , the loss distribution for B is relatively stable with different ratios of |B|/|D| ; examples in B are making little progress when learning rate is large . We propose to characterize the ( negative log-likelihood ) loss distribution of uniform label noise pn ( l ) via the following generative procedure : l = −ỹk + log ∑ i∈ [ N ] exp ( ỹi ) ≡ − log softmax ( ỹ ) [ k ] ( 1 ) ỹ = fc ( relu ( x̃ ) ) , x̃ ∼ N ( 0 , I ) , k ∼ Uniform { 0 , . . . , K } where fc ( · ) is the final ( fully connected ) layer of the network , relu ( x̃ ) = max ( x̃,0 ) is the Rectified Linear Unit , and k represents a random label from K classes . This represents the case where the model ’ s prediction is uncorrelated with the labels . The actual noise distribution could skew to the left if the model overfits to the noise , and skew to the right if the model predicts a label different from the noisy one . We find that an identity covariance matrix for x̃ is able to explain the noise distribution ; this could result from well-conditioned objectives defined via deep residual networks ( He et al. , 2015a ) and careful initialization ( He et al. , 2015b ) . We qualitatively demonstrate the validity of our characterization on CIFAR-100 and ImageNet datasets in Figure 2 . Therefore , we could define a threshold via the p-th percentile of pn ( l ) ; it relates to approximately how much examples in B we would retain if Q ( Y |X ) is uniform . In Section 4.4 , we show that this method is able to identify different percentages of uniform label noise with high precision . 2.3 A PRACTICAL ALGORITHM FOR ROBUST TRAINING . We can utilize this implicit regularization effect to remove examples that might harm generalization , leading to On-the-fly Data Denoising ( ODD ) , a simple algorithm robust to mislabeled examples : 1 . Train all examples with large learning rates for E epochs . 2 . Compute the p-th percentile of the distribution in Eq . ( 1 ) , denoted as Tp . 3 . Remove examples whose average loss of the past h epochs exceeds T from the dataset . 4 . Continue training the remaining examples from epoch E + 1 . ODD introduces three hyperparameters : E determines the amount of training that separates clean examples from noisy ones ; p determines Tp that specifies the trade-off between less noisy examples and more clean examples ; h determines the window of averaged loss statistics to reduce variance from data augmentation . We do not explicitly estimate the portion of noise in the dataset , nor do we assume any specific noise model . In fact , the threshold Tp could be used to accurately predict the portion of uniform noise in the dataset , and works quite well even on other types of label noise ; we will demonstrate this in Section 4 . Moreover , ODD is compatible with existing practices for learning rate schedules , such as stepwise ( He et al. , 2015a ) or cosine ( Loshchilov & Hutter , 2016 ) . 3 RELATED WORK . Implicit Regularization of SGD The generalization of neural networks trained with SGD depend heavily on learning rate schedules ( Loshchilov & Hutter , 2016 ) . It has been proposed that wide local minima2 could result in better generalization ( Hochreiter & Schmidhuber , 1995 ; Chaudhari et al. , 2016 ; Keskar et al. , 2016 ) . Several factors could contribute to wider local optima and better generalization , such as smaller minibatch sizes ( Keskar et al. , 2016 ) , reasonable learning rates ( Kleinberg et al. , 2018 ) , and longer training time ( Hoffer et al. , 2017 ) . Moreover , solutions that are further away from the initialization may lead to wider local minima and better generalization ( Hoffer et al. , 2017 ) . In the presence of mislabeled examples , changes in optimization landscape ( Arpit et al. , 2017 ) could result in bad local minima ( Zhang et al. , 2016 ) , although it is argued that larger batch sizes could mitigate this effect ( Rolnick et al. , 2017 ) . Training with Mislabeled Examples One paradigm to robust training with noisy labels involves estimating the noise distribution ( Liu & Tao , 2014 ) or confusion matrix ( Sukhbaatar et al. , 2014 ) . Another line of methods propose to identify and clean the noisy examples through predictions of auxillary networks ( Veit et al. , 2017 ) or via binary predictions ( Northcutt et al. , 2017 ) ; the noisy labels are either pruned ( Brodley et al. , 1996 ) or replaced with model predictions ( Reed et al. , 2014 ) . Our method is comparable to these approaches , but the key difference is that we leverage the implicit regularization effect of SGD to identify noisy examples . Other approaches propose to reweigh the examples via a pretrained network ( Jiang et al. , 2017 ) , meta learning ( Ren et al. , 2018 ) , or surrogate loss functions ( Ghosh et al. , 2017 ; Zhang & Sabuncu , 2018 ) . Some methods require a set of trusted examples ( Xiao et al. , 2015 ; Hendrycks et al. , 2018 ) . ODD has several appealing properties compared to existing methods . First , the thresholds for classifying mislabeled examples from ODD do not rely on estimations of the noise confusion matrix . Next , ODD does not require additional trusted examples . Finally , ODD removes potentially noisy examples on-the-fly ; it has little computational overhead compared to standard SGD training .

Summary: This paper presents ODD, a method that rejects incorrectly labeled / noisy examples from training on the fly. The motivation is sound, that with the capacity of modern neural networks, it's easy to memorize the mislabeled data and thus hurt generalization. If we could reject such mislabeled data, we may be able to get a more generalizable model. The authors made an observation that when training with large learning rate, examples with correct labeling and incorrect labeling exhibits different loss distributions. The authors further noticed that the loss distribution of incorrectly labeled examples can be simulated using eq.(1). Therefore, by setting a threshold that corresponds to a percentile of the incorrectly labeled loss distribution, the authors are able to reject incorrect examples.