Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee

Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data . Such over-fitting ability hinders generalization when mislabeled training examples are present . On the other hand , simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios , a phenomenon not theoretically understood . This paper proposes and analyzes two simple and intuitive regularization methods : ( i ) regularization by the distance between the network parameters to initialization , and ( ii ) adding a trainable auxiliary variable to the network output for each training example . Theoretically , we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels . Our generalization analysis relies on the connection between wide neural network and neural tangent kernel ( NTK ) . The generalization bound is independent of the network size , and is comparable to the bound one can get when there is no label noise . Experimental results verify the effectiveness of these methods on noisily labeled datasets . 1 INTRODUCTION . Modern deep neural networks are trained in a highly over-parameterized regime , with many more trainable parameters than training examples . It is well-known that these networks trained with simple first-order methods can fit any labels , even completely random ones ( Zhang et al. , 2017 ) . Although training on properly labeled data usually leads to good generalization performance , the ability to over-fit the entire training dataset is undesirable for generalization when noisy labels are present . Therefore preventing over-fitting is crucial for robust performance since mislabeled data are ubiquitous in very large datasets ( Krishna et al. , 2016 ) . In order to prevent over-fitting to mislabeled data , some form of regularization is necessary . A simple such example is early stopping , which has been observed to be surprisingly effective for this purpose ( Rolnick et al. , 2017 ; Guan et al. , 2018 ; Li et al. , 2019 ) . For instance , training ResNet-34 with early stopping can achieve 84 % test accuracy on CIFAR-10 even when 60 % of the training labels are corrupted ( Table 1 ) . This is nontrivial since the test error is much smaller than the error rate in training data . How to explain such generalization phenomenon is an intriguing theoretical question . As a step towards a theoretical understanding of the generalization phenomenon for overparameterized neural networks when noisy labels are present , this paper proposes and analyzes two simple regularization methods as alternatives of early stopping : 1 . Regularization by distance to initialization . Denote by θ the network parameters and by θp0q its random initialization . This method adds a regularizer λ } θ ´ θp0q } 2 to the training objective . 2 . Adding an auxiliary variable for each training example . Let xi be the i-th training example and fpθ , ¨q represent the neural net . This method adds a trainable variable bi and tries to fit the i-th label using fpθ , xiq ` λbi . At test time , only the neural net fpθ , ¨q is used and the auxiliary variables are discarded . These two choices of regularization are well motivated with clear intuitions . First , distance to initialization has been observed to be very related to generalization in deep learning ( Neyshabur et al. , 2019 ; Nagarajan and Kolter , 2019 ) , so regularizing by distance to initialization can potentially help generalization . Second , the effectiveness of early stopping indicates that clean labels are somewhat easier to fit than wrong labels ; therefore , adding an auxiliary variable could help “ absorb ” the noise in the labels , thus making the neural net itself not over-fitting . We provide theoretical analysis of the above two regularization methods for a class of sufficiently wide neural networks by proving a generalization bound for the trained network on clean data distribution when the training dataset contains noisy labels . Our generalization bound depends on the ( unobserved ) clean labels , and is comparable to the bound one can get when there is no label noise , therefore indicating that the proposed regularization methods are robust to noisy labels . Our theoretical analysis is based on the recently established connection between wide neural net and neural tangent kernel ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019a ) . In this line of work , parameters in a wide neural net are shown to stay close to their initialization during gradient descent training , and as a consequence , the neural net can be effectively approximated by its first-order Taylor expansion with respect to its parameters at initialization . This leads to tractable linear dynamics under ` 2 loss , and the final solution can be characterized by kernel regression using a particular kernel named neural tangent kernel ( NTK ) . In fact , we show that for wide neural nets , both of our regularization methods , when trained with gradient descent to convergence , correspond to kernel ridge regression using the NTK , which is often regarded as an alternative to early stopping in kernel literature . This viewpoint makes explicit the connection between our methods and early stopping . The effectiveness of these two regularization methods is verified empirically – on MNIST and CIFAR-10 , they are able to achieve highly nontrivial test accuracy , on a par with or even better than early stopping . Furthermore , with our regularization , the validation accuracy is almost monotone increasing throughout the entire training process , indicating their resistance to over-fitting . 2 RELATED WORK . Neural tangent kernel was first explicitly studied and named by Jacot et al . ( 2018 ) , with several further refinements and extensions by Lee et al . ( 2019 ) ; Yang ( 2019 ) ; Arora et al . ( 2019a ) . Using the similar idea that weights stay close to initialization and that the neural network is approximated by a linear model , a series of theoretical papers studied the optimization and generalization issues of very wide deep neural nets trained by ( stochastic ) gradient descent ( Du et al. , 2019b ; 2018b ; Li and Liang , 2018 ; Allen-Zhu et al. , 2018a ; b ; Zou et al. , 2018 ; Arora et al. , 2019b ; Cao and Gu , 2019 ) . Empirically , variants of NTK on convolutional neural nets and graph neural nets exhibit strong practical performance ( Arora et al. , 2019a ; Du et al. , 2019a ) , thus suggesting that ultra-wide ( or infinitely wide ) neural nets are at least not irrelevant . Our methods are closely related to kernel ridge regression , which is one of the most common kernel methods and has been widely studied . It was shown to perform comparably to early-stopped gradient descent ( Bauer et al. , 2007 ; Gerfo et al. , 2008 ; Raskutti et al. , 2014 ; Wei et al. , 2017 ) . Accordingly , we indeed observe in our experiments that our regularization methods perform similarly to gradient descent with early stopping in neural net training . In another theoretical study relevant to ours , Li et al . ( 2019 ) proved that gradient descent with early stopping is robust to label noise for an over-parameterized two-layer neural net . Under a clustering assumption on data , they showed that gradient descent fits the correct labels before starting to over-fit wrong labels . Their result is different from ours from several aspects : they only considered two-layer nets while we allow arbitrarily deep nets ; they required a clustering assumption on data while our generalization bound is general and data-dependent ; furthermore , they did not address the question of generalization , but only provided guarantees on the training data . A large body of work proposed various methods for training with mislabeled examples , such as estimating noise distribution ( Liu and Tao , 2015 ) or confusion matrix ( Sukhbaatar et al. , 2014 ) , using surrogate loss functions ( Ghosh et al. , 2017 ; Zhang and Sabuncu , 2018 ) , meta-learning ( Ren et al. , 2018 ) , using a pre-trained network ( Jiang et al. , 2017 ) , and training two networks simultaneously ( Malach and Shalev-Shwartz , 2017 ; Han et al. , 2018 ; Yu et al. , 2019 ) . While our methods are not necessarily superior to these methods in terms of performance , our methods are arguably simpler ( with minimal change to normal training procedure ) and come with formal generalization guarantee . 3 PRELIMINARIES . Notation . We use bold-faced letters for vectors and matrices . We use } ¨ } to denote the Euclidean norm of a vector or the spectral norm of a matrix , and } ¨ } F to denote the Frobenius norm of a matrix . x¨ , ¨y represents the standard inner product . Let I be the identity matrix of appropriate dimension . Let rns “ t1 , 2 , . . . , nu . Let IrAs be the indicator of event A . 3.1 SETTING : LEARNING FROM NOISILY LABELED DATA . Now we formally describe the setting considered in this paper . We first describe the binary classification setting as a warm-up , and then describe the more general setting of multi-class classification . Binary classification . Suppose that there is an underlying data distribution D over Rd ˆ t˘1u , where 1 and ´1 are labels corresponding to two classes . However , we only have access to samples from a noisily labeled version of D. Formally , the data generation process is : draw px , yq „ D , and flip the sign of label y with probability p ( 0 ď p ă 12 ) ; let ỹ P t˘1u be the resulting noisy label . Let tpxi , ỹiquni “ 1 be i.i.d . samples generated from the above process . Although we only have access to these noisily labeled data , the goal is still to learn a function ( in the form of a neural net ) that can predict the true label well on the clean distribution D. For binary classification , it suffices to learn a single-output function f : Rd Ñ R whose sign is used to predict the class , and thus the classification error of f on D is defined as Prpx , yq „ Drsgnpfpxqq “ ys . Multi-class classification . When there are K classes ( K ą 2 ) , let the underlying data distribution D be over Rd ˆ rKs . We describe the noise generation process as a matrix P P RKˆK , whose entry pc1 , c is the probability that the label c is transformed into c1 ( @ c , c1 P rKs ) . Therefore the data generation process is : draw px , cq „ D , and replace the label c with c̃ from the distribution Prrc̃ “ c1|cs “ pc1 , c ( @ c1 P rKs ) . Let tpxi , c̃iquni “ 1 be i.i.d . samples from the above process . Again we would like to learn a neural net with low classification error on the clean distribution D. For K-way classification , it is common to use a neural net with K outputs , and the index of the maximum output is used to predict the class . Thus for f : Rd Ñ RK , its ( top-1 ) classification error on D is Prpx , cq „ Drc R argmaxhPrKsf phqpxqs , where f phq : Rd Ñ R is the function computed by the h-th output of f . As standard practice , a class label c P rKs is also treated as its one-hot encoding epcq “ p0 , 0 , ¨ ¨ ¨ , 0 , 1 , 0 , ¨ ¨ ¨ , 0q P RK ( the c-th coordinate being 1 ) , which can be paired with the K outputs of the network and fed into a loss function during training . Note that it is necessary to assume pc , c ą pc1 , c for all c “ c1 , i.e. , the probability that a class label c is transformed into another particular label must be smaller than the label c being correct – otherwise it is impossible to identify class c correctly from noisily labeled data .

Summary: In this paper, based on the effectiveness of early stopping in the training of noisily labeled samples, the authors proposed two intuitive (and novel) regularization methods: (1) regularizing using distance to initialization (2) adding an auxiliary variable b_i for every input x_i during training. In terms of theory, the authors showed that in the NTK regime, both regularization methods trained with gradient descent are equivalent to kernel ridge regression.  Moreover, the authors also provided a generalization bound of the solution on the clean data distribution when trained with noisy label, which was not addressed in previous research.