Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee

Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data . Such over-fitting ability hinders generalization when mislabeled training examples are present . On the other hand , simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios , a phenomenon not theoretically understood . This paper proposes and analyzes two simple and intuitive regularization methods : ( i ) regularization by the distance between the network parameters to initialization , and ( ii ) adding a trainable auxiliary variable to the network output for each training example . Theoretically , we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels . Our generalization analysis relies on the connection between wide neural network and neural tangent kernel ( NTK ) . The generalization bound is independent of the network size , and is comparable to the bound one can get when there is no label noise . Experimental results verify the effectiveness of these methods on noisily labeled datasets . 1 INTRODUCTION . Modern deep neural networks are trained in a highly over-parameterized regime , with many more trainable parameters than training examples . It is well-known that these networks trained with simple first-order methods can fit any labels , even completely random ones ( Zhang et al. , 2017 ) . Although training on properly labeled data usually leads to good generalization performance , the ability to over-fit the entire training dataset is undesirable for generalization when noisy labels are present . Therefore preventing over-fitting is crucial for robust performance since mislabeled data are ubiquitous in very large datasets ( Krishna et al. , 2016 ) . In order to prevent over-fitting to mislabeled data , some form of regularization is necessary . A simple such example is early stopping , which has been observed to be surprisingly effective for this purpose ( Rolnick et al. , 2017 ; Guan et al. , 2018 ; Li et al. , 2019 ) . For instance , training ResNet-34 with early stopping can achieve 84 % test accuracy on CIFAR-10 even when 60 % of the training labels are corrupted ( Table 1 ) . This is nontrivial since the test error is much smaller than the error rate in training data . How to explain such generalization phenomenon is an intriguing theoretical question . As a step towards a theoretical understanding of the generalization phenomenon for overparameterized neural networks when noisy labels are present , this paper proposes and analyzes two simple regularization methods as alternatives of early stopping : 1 . Regularization by distance to initialization . Denote by θ the network parameters and by θp0q its random initialization . This method adds a regularizer λ } θ ´ θp0q } 2 to the training objective . 2 . Adding an auxiliary variable for each training example . Let xi be the i-th training example and fpθ , ¨q represent the neural net . This method adds a trainable variable bi and tries to fit the i-th label using fpθ , xiq ` λbi . At test time , only the neural net fpθ , ¨q is used and the auxiliary variables are discarded . These two choices of regularization are well motivated with clear intuitions . First , distance to initialization has been observed to be very related to generalization in deep learning ( Neyshabur et al. , 2019 ; Nagarajan and Kolter , 2019 ) , so regularizing by distance to initialization can potentially help generalization . Second , the effectiveness of early stopping indicates that clean labels are somewhat easier to fit than wrong labels ; therefore , adding an auxiliary variable could help “ absorb ” the noise in the labels , thus making the neural net itself not over-fitting . We provide theoretical analysis of the above two regularization methods for a class of sufficiently wide neural networks by proving a generalization bound for the trained network on clean data distribution when the training dataset contains noisy labels . Our generalization bound depends on the ( unobserved ) clean labels , and is comparable to the bound one can get when there is no label noise , therefore indicating that the proposed regularization methods are robust to noisy labels . Our theoretical analysis is based on the recently established connection between wide neural net and neural tangent kernel ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019a ) . In this line of work , parameters in a wide neural net are shown to stay close to their initialization during gradient descent training , and as a consequence , the neural net can be effectively approximated by its first-order Taylor expansion with respect to its parameters at initialization . This leads to tractable linear dynamics under ` 2 loss , and the final solution can be characterized by kernel regression using a particular kernel named neural tangent kernel ( NTK ) . In fact , we show that for wide neural nets , both of our regularization methods , when trained with gradient descent to convergence , correspond to kernel ridge regression using the NTK , which is often regarded as an alternative to early stopping in kernel literature . This viewpoint makes explicit the connection between our methods and early stopping . The effectiveness of these two regularization methods is verified empirically – on MNIST and CIFAR-10 , they are able to achieve highly nontrivial test accuracy , on a par with or even better than early stopping . Furthermore , with our regularization , the validation accuracy is almost monotone increasing throughout the entire training process , indicating their resistance to over-fitting . 2 RELATED WORK . Neural tangent kernel was first explicitly studied and named by Jacot et al . ( 2018 ) , with several further refinements and extensions by Lee et al . ( 2019 ) ; Yang ( 2019 ) ; Arora et al . ( 2019a ) . Using the similar idea that weights stay close to initialization and that the neural network is approximated by a linear model , a series of theoretical papers studied the optimization and generalization issues of very wide deep neural nets trained by ( stochastic ) gradient descent ( Du et al. , 2019b ; 2018b ; Li and Liang , 2018 ; Allen-Zhu et al. , 2018a ; b ; Zou et al. , 2018 ; Arora et al. , 2019b ; Cao and Gu , 2019 ) . Empirically , variants of NTK on convolutional neural nets and graph neural nets exhibit strong practical performance ( Arora et al. , 2019a ; Du et al. , 2019a ) , thus suggesting that ultra-wide ( or infinitely wide ) neural nets are at least not irrelevant . Our methods are closely related to kernel ridge regression , which is one of the most common kernel methods and has been widely studied . It was shown to perform comparably to early-stopped gradient descent ( Bauer et al. , 2007 ; Gerfo et al. , 2008 ; Raskutti et al. , 2014 ; Wei et al. , 2017 ) . Accordingly , we indeed observe in our experiments that our regularization methods perform similarly to gradient descent with early stopping in neural net training . In another theoretical study relevant to ours , Li et al . ( 2019 ) proved that gradient descent with early stopping is robust to label noise for an over-parameterized two-layer neural net . Under a clustering assumption on data , they showed that gradient descent fits the correct labels before starting to over-fit wrong labels . Their result is different from ours from several aspects : they only considered two-layer nets while we allow arbitrarily deep nets ; they required a clustering assumption on data while our generalization bound is general and data-dependent ; furthermore , they did not address the question of generalization , but only provided guarantees on the training data . A large body of work proposed various methods for training with mislabeled examples , such as estimating noise distribution ( Liu and Tao , 2015 ) or confusion matrix ( Sukhbaatar et al. , 2014 ) , using surrogate loss functions ( Ghosh et al. , 2017 ; Zhang and Sabuncu , 2018 ) , meta-learning ( Ren et al. , 2018 ) , using a pre-trained network ( Jiang et al. , 2017 ) , and training two networks simultaneously ( Malach and Shalev-Shwartz , 2017 ; Han et al. , 2018 ; Yu et al. , 2019 ) . While our methods are not necessarily superior to these methods in terms of performance , our methods are arguably simpler ( with minimal change to normal training procedure ) and come with formal generalization guarantee . 3 PRELIMINARIES . Notation . We use bold-faced letters for vectors and matrices . We use } ¨ } to denote the Euclidean norm of a vector or the spectral norm of a matrix , and } ¨ } F to denote the Frobenius norm of a matrix . x¨ , ¨y represents the standard inner product . Let I be the identity matrix of appropriate dimension . Let rns “ t1 , 2 , . . . , nu . Let IrAs be the indicator of event A . 3.1 SETTING : LEARNING FROM NOISILY LABELED DATA . Now we formally describe the setting considered in this paper . We first describe the binary classification setting as a warm-up , and then describe the more general setting of multi-class classification . Binary classification . Suppose that there is an underlying data distribution D over Rd ˆ t˘1u , where 1 and ´1 are labels corresponding to two classes . However , we only have access to samples from a noisily labeled version of D. Formally , the data generation process is : draw px , yq „ D , and flip the sign of label y with probability p ( 0 ď p ă 12 ) ; let ỹ P t˘1u be the resulting noisy label . Let tpxi , ỹiquni “ 1 be i.i.d . samples generated from the above process . Although we only have access to these noisily labeled data , the goal is still to learn a function ( in the form of a neural net ) that can predict the true label well on the clean distribution D. For binary classification , it suffices to learn a single-output function f : Rd Ñ R whose sign is used to predict the class , and thus the classification error of f on D is defined as Prpx , yq „ Drsgnpfpxqq “ ys . Multi-class classification . When there are K classes ( K ą 2 ) , let the underlying data distribution D be over Rd ˆ rKs . We describe the noise generation process as a matrix P P RKˆK , whose entry pc1 , c is the probability that the label c is transformed into c1 ( @ c , c1 P rKs ) . Therefore the data generation process is : draw px , cq „ D , and replace the label c with c̃ from the distribution Prrc̃ “ c1|cs “ pc1 , c ( @ c1 P rKs ) . Let tpxi , c̃iquni “ 1 be i.i.d . samples from the above process . Again we would like to learn a neural net with low classification error on the clean distribution D. For K-way classification , it is common to use a neural net with K outputs , and the index of the maximum output is used to predict the class . Thus for f : Rd Ñ RK , its ( top-1 ) classification error on D is Prpx , cq „ Drc R argmaxhPrKsf phqpxqs , where f phq : Rd Ñ R is the function computed by the h-th output of f . As standard practice , a class label c P rKs is also treated as its one-hot encoding epcq “ p0 , 0 , ¨ ¨ ¨ , 0 , 1 , 0 , ¨ ¨ ¨ , 0q P RK ( the c-th coordinate being 1 ) , which can be paired with the K outputs of the network and fed into a loss function during training . Note that it is necessary to assume pc , c ą pc1 , c for all c “ c1 , i.e. , the probability that a class label c is transformed into another particular label must be smaller than the label c being correct – otherwise it is impossible to identify class c correctly from noisily labeled data .

Summary: This paper proposes two regularization methods for learning on noisily labeled data: the first penalizes the distance w.r.t. Euclidean norm from an initial point and the second uses an additional auxiliary variable for each example to learn a noise. In the theoretical part, the paper shows that an original clean dataset can be learned from a noisily labeled dataset based on NTK-theory. Finally, the effectiveness of proposed regularization methods is well verified empirically for 2-layer NN, CNN, and ResNet on image classification tasks (MNIST, CIFAR-10).