Title: SIMPLE AND EFFECTIVE REGULARIZATION METHODS FOR TRAINING ON NOISILY LABELED DATA WITH GENERALIZATION GUARANTEE

Abstract: Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.

1. INTRODUCTION:

Modern deep neural networks are trained in a highly over-parameterized regime, with many more trainable parameters than training examples. It is well-known that these networks trained with simple first-order methods can fit any labels, even completely random ones (Zhang et al., 2017) .

Although training on properly labeled data usually leads to good generalization performance, the ability to over-fit the entire training dataset is undesirable for generalization when noisy labels are present. Therefore preventing over-fitting is crucial for robust performance since mislabeled data are ubiquitous in very large datasets (Krishna et al., 2016) .

In order to prevent over-fitting to mislabeled data, some form of regularization is necessary. A simple such example is early stopping, which has been observed to be surprisingly effective for this purpose (Rolnick et al., 2017; Guan et al., 2018; Li et al., 2019) . For instance, training ResNet-34 with early stopping can achieve 84% test accuracy on CIFAR-10 even when 60% of the training labels are corrupted (Table 1 ). This is nontrivial since the test error is much smaller than the error rate in training data. How to explain such generalization phenomenon is an intriguing theoretical question.

As a step towards a theoretical understanding of the generalization phenomenon for overparameterized neural networks when noisy labels are present, this paper proposes and analyzes two simple regularization methods as alternatives of early stopping:

1. Regularization by distance to initialization. Denote by θ the network parameters and by θp0q its random initialization. This method adds a regularizer λ }θ ´θp0q} 2 to the training objective.

2. Adding an auxiliary variable for each training example. Let x i be the i-th training example and f pθ, ¨q represent the neural net. This method adds a trainable variable b i and tries to fit the i-th label using f pθ, x i q `λb i . At test time, only the neural net f pθ, ¨q is used and the auxiliary variables are discarded.

These two choices of regularization are well motivated with clear intuitions. First, distance to initialization has been observed to be very related to generalization in deep learning (Neyshabur et al., 2019; Nagarajan and Kolter, 2019) , so regularizing by distance to initialization can potentially help generalization. Second, the effectiveness of early stopping indicates that clean labels are somewhat easier to fit than wrong labels; therefore, adding an auxiliary variable could help "absorb" the noise in the labels, thus making the neural net itself not over-fitting.

We provide theoretical analysis of the above two regularization methods for a class of sufficiently wide neural networks by proving a generalization bound for the trained network on clean data distribution when the training dataset contains noisy labels. Our generalization bound depends on the (unobserved) clean labels, and is comparable to the bound one can get when there is no label noise, therefore indicating that the proposed regularization methods are robust to noisy labels.

Our theoretical analysis is based on the recently established connection between wide neural net and neural tangent kernel (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019a) . In this line of work, parameters in a wide neural net are shown to stay close to their initialization during gradient descent training, and as a consequence, the neural net can be effectively approximated by its first-order Taylor expansion with respect to its parameters at initialization. This leads to tractable linear dynamics under 2 loss, and the final solution can be characterized by kernel regression using a particular kernel named neural tangent kernel (NTK). In fact, we show that for wide neural nets, both of our regularization methods, when trained with gradient descent to convergence, correspond to kernel ridge regression using the NTK, which is often regarded as an alternative to early stopping in kernel literature. This viewpoint makes explicit the connection between our methods and early stopping.

The effectiveness of these two regularization methods is verified empirically -on MNIST and CIFAR-10, they are able to achieve highly nontrivial test accuracy, on a par with or even better than early stopping. Furthermore, with our regularization, the validation accuracy is almost monotone increasing throughout the entire training process, indicating their resistance to over-fitting.

2. RELATED WORK:

Neural tangent kernel was first explicitly studied and named by Jacot et al. (2018) , with several further refinements and extensions by Lee et al. (2019) ; Yang (2019) ; Arora et al. (2019a) . Using the similar idea that weights stay close to initialization and that the neural network is approximated by a linear model, a series of theoretical papers studied the optimization and generalization issues of very wide deep neural nets trained by (stochastic) gradient descent (Du et al., 2019b; 2018b; Li and Liang, 2018; Allen-Zhu et al., 2018a; b; Zou et al., 2018; Arora et al., 2019b; Cao and Gu, 2019) . Empirically, variants of NTK on convolutional neural nets and graph neural nets exhibit strong practical performance (Arora et al., 2019a; Du et al., 2019a) , thus suggesting that ultra-wide (or infinitely wide) neural nets are at least not irrelevant.

Our methods are closely related to kernel ridge regression, which is one of the most common kernel methods and has been widely studied. It was shown to perform comparably to early-stopped gradient descent (Bauer et al., 2007; Gerfo et al., 2008; Raskutti et al., 2014; Wei et al., 2017) . Accordingly, we indeed observe in our experiments that our regularization methods perform similarly to gradient descent with early stopping in neural net training.

In another theoretical study relevant to ours, Li et al. (2019) proved that gradient descent with early stopping is robust to label noise for an over-parameterized two-layer neural net. Under a clustering assumption on data, they showed that gradient descent fits the correct labels before starting to over-fit wrong labels. Their result is different from ours from several aspects: they only considered two-layer nets while we allow arbitrarily deep nets; they required a clustering assumption on data while our generalization bound is general and data-dependent; furthermore, they did not address the question of generalization, but only provided guarantees on the training data.

A large body of work proposed various methods for training with mislabeled examples, such as estimating noise distribution (Liu and Tao, 2015) or confusion matrix (Sukhbaatar et al., 2014) , using surrogate loss functions (Ghosh et al., 2017; Zhang and Sabuncu, 2018) , meta-learning (Ren et al., 2018) , using a pre-trained network (Jiang et al., 2017) , and training two networks simultaneously (Malach and Shalev-Shwartz, 2017; Han et al., 2018; Yu et al., 2019) . While our methods are not necessarily superior to these methods in terms of performance, our methods are arguably simpler (with minimal change to normal training procedure) and come with formal generalization guarantee.

3. PRELIMINARIES:

Notation. We use bold-faced letters for vectors and matrices. We use }¨} to denote the Euclidean norm of a vector or the spectral norm of a matrix, and }¨} F to denote the Frobenius norm of a matrix. x¨, ¨y represents the standard inner product. Let I be the identity matrix of appropriate dimension. Let rns " t1, 2, . . . , nu. Let IrAs be the indicator of event A.

3.1 SETTING: LEARNING FROM NOISILY LABELED DATA Now we formally describe the setting considered in this paper. We first describe the binary classification setting as a warm-up, and then describe the more general setting of multi-class classification.

Binary classification. Suppose that there is an underlying data distribution D over R d ˆt˘1u, where 1 and ´1 are labels corresponding to two classes. However, we only have access to samples from a noisily labeled version of D. Formally, the data generation process is: draw px, yq " D, and flip the sign of label y with probability p (0 ď p ă1 2 ); let ỹ P t˘1u be the resulting noisy label. Let tpx i , ỹi qu n i"1 be i.i.d. samples generated from the above process. Although we only have access to these noisily labeled data, the goal is still to learn a function (in the form of a neural net) that can predict the true label well on the clean distribution D. For binary classification, it suffices to learn a single-output function f : R d Ñ R whose sign is used to predict the class, and thus the classification error of f on D is defined as Pr px,yq"D rsgnpf pxqq " ys.

Multi-class classification. When there are K classes (K ą 2), let the underlying data distribution D be over R d ˆrKs. We describe the noise generation process as a matrix P P R KˆK , whose entry p c 1 ,c is the probability that the label c is transformed into c 1 (@c, c 1 P rKs). Therefore the data generation process is: draw px, cq " D, and replace the label c with c from the distribution Prrc " c 1 |cs " p c 1 ,c (@c 1 P rKs).

Let tpx i , ci qu n i"1 be i.i.d. samples from the above process. Again we would like to learn a neural net with low classification error on the clean distribution D. For K-way classification, it is common to use a neural net with K outputs, and the index of the maximum output is used to predict the class. Thus for f : R d Ñ R K , its (top-1) classification error on D is Pr px,cq"D rc R argmax hPrKs f phq pxqs, where f phq : R d Ñ R is the function computed by the h-th output of f . As standard practice, a class label c P rKs is also treated as its one-hot encoding e pcq " p0, 0, ¨¨¨, 0, 1, 0, ¨¨¨, 0q P R K (the c-th coordinate being 1), which can be paired with the K outputs of the network and fed into a loss function during training.

Note that it is necessary to assume p c,c ą p c 1 ,c for all c " c 1 , i.e., the probability that a class label c is transformed into another particular label must be smaller than the label c being correct -otherwise it is impossible to identify class c correctly from noisily labeled data.

3.2 RECAP OF NEURAL TANGENT KERNEL Now we briefly and informally recap the theory of neural tangent kernel (NTK) (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019a) , which establishes the equivalence between training a wide neural net and a kernel method.

We first consider a neural net with a scalar output, defined as f pθ, xq P R, where θ P R N is all the parameters in the net and x P R d is the input. Suppose that the net is trained by minimizing the 2 loss over a training dataset tpx i , y i qu n i"1 Ă R d ˆR: Lpθq " 1 2 ř n i"1 pf pθ, x i q ´yi q 2 . Let the random initial parameters be θp0q, and the parameters be updated according to gradient descent on Lpθq. It is shown that if the network is sufficiently wide 1 , the parameters θ will stay close to the initialization θp0q during training so that the following first-order approximation is accurate:

EQUATION

)

This approximation is exact in the infinite width limit, but can also be shown when the width is finite but sufficiently large. When approximation (1) holds, we say that we are in the NTK regime.

Define φpxq " ∇ θ f pθp0q, xq for any x P R d . The right hand side in (1) is linear in θ. As a consequence, training on the 2 loss with gradient descent leads to the kernel regression solution with respect to the kernel induced by (random) features φpxq, which is defined as kpx, x 1 q " xφpxq, φpx 1 qy for x, x 1 P R d . This kernel was named the neural tangent kernel (NTK) by Jacot et al. (2018) . Although this kernel is random, it is shown that when the network is sufficiently wide, this random kernel converges to a deterministic limit in probability (Arora et al., 2019a) . If we additionally let the neural net and its initialization be defined so that the initial output is small, i.e., f pθp0q, xq « 0,2 then the network at the end of training approximately computes the following function:

x Þ Ñ kpx, Xq J pkpX, Xqq ´1 y, (2) where X " px 1 , . . . , x n q is the training inputs, y " py 1 , . . . , y n q J is the training targets, kpx, Xq " pkpx, x 1 q, . . . , kpx, x n qq J P R n , and kpX, Xq P R nˆn with pi, jq-th entry being kpx i , x j q.

Multiple outputs. The NTK theory above can also be generalized straightforwardly to the case of multiple outputs (Jacot et al., 2018; Lee et al., 2019) . Suppose we train a neural net with K outputs, f pθ, xq, by minimizing the 2 loss over a training dataset tpx i ,

y i qu n i"1 Ă R d ˆRK : Lpθq " 1 2 ř n i"1 }f pθ, x i q ´yi } 2 .

When the hidden layers are sufficiently wide such that we are in the NTK regime, at the end of gradient descent, each output of f also attains the kernel regression solution with respect to the same NTK as before, using the corresponding dimension in the training targets y i . Namely, the h-th output of the network computes the function f phq pxq " kpx, Xq J pkpX, Xqq ´1 y phq , where y phq P R n whose i-th coordinate is the h-th coordinate of y i .

4. REGULARIZATION METHODS:

In this section we describe two simple regularization methods for training with noisy labels, and show that if the network is sufficiently wide, both methods lead to kernel ridge regression using the NTK. 3We first consider the case of scalar target and single-output network. The generalization to multiple outputs is straightforward and is treated at the end of this section. Given a noisily labeled training dataset tpx i , ỹi qu n i"1 Ă R d ˆR, let f pθ, ¨q be a neural net to be trained. A direct, unregularized training method would involve minimizing an objective function like Lpθq " 1 2 ř n i"1 pf pθ, x i q ´ỹ i q 2 . To prevent over-fitting, we suggest the following simple regularization methods that slightly modify this objective:

• Method 1: Regularization using Distance to Initialization (RDI). We let the initial parameters θp0q be randomly generated, and minimize the following regularized objective:

EQUATION

• Method 2: adding an AUXiliary variable for each training example (AUX). We add an auxiliary trainable parameter b i P R for each i P rns, and minimize the following objective:

EQUATION

where b " pb 1 , . . . , b n q J P R n is initialized to be 0.

Equivalence to kernel ridge regression in wide neural nets. Now we assume that we are in the NTK regime described in Section 3.2, where the neural net architecture is sufficiently wide so that the first-order approximation (1) is accurate during gradient descent: f pθ, xq « f pθp0q, xq φpxq J pθ ´θp0qq. Recall that we have φpxq " ∇ θ f pθp0q, xq which induces the NTK kpx, x 1 q " xφpxq, φpx 1 qy. Also recall that we can assume near-zero initial output: f pθp0q, xq « 0 (see Footnote 2). Therefore we have the approximation: f pθ, xq « φpxq J pθ ´θp0qq.

(5)

Under the approximation (5), it suffices to consider gradient descent on the objectives (3) and ( 4) using the linearized model instead:

LRDI λ pθq " 1 2 n ÿ i"1 ´φpxiq J pθ ´θp0qq ´ỹi ¯2 `λ2 2 }θ ´θp0q} 2 , LAUX λ pθ, bq " 1 2 n ÿ i"1

´φpxiq J pθ ´θp0qq `λbi ´ỹi ¯2 .

The following theorem shows that in either case, gradient descent leads to the same dynamics and converges to the kernel ridge regression solution using the NTK. Then we must have θptq " θptq for all t. Furthermore, if the learning rate satisfies η ď 1 }kpX,Xq}`λ 2 , then tθptqu converges linearly to a limit solution θ ˚such that:

φpxq J pθ ˚´θp0qq " kpx, Xq J `kpX, Xq `λ2 I ˘´1 ỹ, @x,

where ỹ " pỹ 1 , . . . , ỹn q J P R n .

Proof sketch. The proof is given in Appendix B. A key step is to observe θptq " θp0q`ř n i"1 1 λ b i ptqφ px i q, from which we can show that tθptqu and t θptqu follow the same update rule. Theorem 4.1 indicates that gradient descent on the regularized objectives (3) and ( 4) both learn approximately the following function at the end of training when the neural net is sufficiently wide: f ˚pxq " kpx, Xq J `kpX, Xq `λ2 I ˘´1 ỹ.

(8) If no regularization were used, the labels ỹ would be fitted perfectly and the learned function would be kpx, Xq J pkpX, Xqq ´1 ỹ (c.f. (2)). Therefore the effect of regularization is to add λ 2 I to the kernel matrix, and ( 8) is known as the solution to kernel ridge regression in kernel literature. In Section 5, we give a generalization bound of this solution on the clean data distribution, which is comparable to the bound one can obtain even when clean labels are used in training.

Extension to multiple outputs. Suppose that the training dataset is tpx i , ỹi qu n i"1 Ă R d ˆRK , and the neural net f pθ, xq has K outputs. On top of the vanilla loss Lpθq " 1 2 ř n i"1 }f pθ, x i q ´ỹ i } 2 , the two regularization methods RDI and AUX give the following objectives similar to (3) and (4):

L RDI λ pθq " 1 2 n ÿ i"1 }f pθ, xiq ´ỹi} 2 `λ2 2 }θ ´θp0q} 2 , L AUX λ pθ, Bq " 1 2 n ÿ i"1 }f pθ, xiq `λbi ´ỹi} 2 , B " pb1, . . . , bnq P R Kˆn .

In the NTK regime, both methods lead to the kernel ridge regression solution at each output. Namely, letting Ỹ " p ỹ1 , . . . , ỹn q P R Kˆn be the training target matrix and ỹphq P R n be the h-th row of Ỹ , at the end of training the h-th output of the network learns the following function: f phq pxq " kpx, Xq J `kpX, Xq `λ2 I ˘´1 ỹphq , h P rKs.

(9)

5. GENERALIZATION GUARANTEE ON CLEAN DATA DISTRIBUTION:

We show that gradient descent training on noisily labeled data with our regularization methods RDI or AUX leads to a generalization guarantee on the clean data distribution. As in Section 4, we consider the NTK regime and let kp¨, ¨q be the NTK corresponding to the neural net. It suffices to analyze the kernel ridge regression predictor, i.e., (8) for single output and (9) for multiple outputs.

We start with a regression setting where labels are real numbers and the noisy label is the true label plus an additive noise (Theorem 5.1). Built on this result, we then provide generalization bounds for the classification settings described in Section 3.1. Omitted proofs in this section are in Appendix C.

Theorem 5.1 (Additive label noise). Let D be a distribution over R d ˆr´1, 1s. Consider the following data generation process: (i) draw px, yq " D, (ii) conditioned on px, yq, let ε be drawn from a noise distribution E x,y over R that may depend on x and y, and (iii) let ỹ " y `ε. Suppose that E x,y has mean 0 and is subgaussian with parameter σ ą 0, for any px, yq.

Let tpx i , y i , ỹi qu n i"1 be i.i.d. samples from the above process. Denote X " px 1 , . . . , x n q, y " py 1 , . . . , y n q J and ỹ " pỹ 1 , . . . , ỹn q J . Consider the kernel ridge regression solution in (8): f ˚pxq " kpx, Xq J `kpX, Xq `λ2 I ˘´1 ỹ. Suppose that the kernel matrix satisfies trrkpX, Xqs " Opnq. Then for any loss function : R ˆR Ñ r0, 1s that is 1-Lipschitz in the first argument such that py, yq " 0, with probability at least 1 ´δ we have

EQUATION

where

∆ " O ˆσb logp1{δq n `σ λ b logp1{δq n `b log n δλ n ˙.

Remark 5.1. As the number of samples n Ñ 8, we have ∆ Ñ 0. In order for the second term Op σ λ q in (10) to go to 0, we need to choose λ to grow with n, e.g., λ " n c for some small constant c ą 0. Then, the only remaining term in (10) to worry about is λ 2 b y J pkpX,Xqq ´1y n

. Notice that it depends on the (unobserved) clean labels y, instead of the noisy labels ỹ. By a very similar proof, one can show that training on the clean labels y (without regularization) leads to a population loss bound O ´b y J pkpX,Xqq ´1y n ¯. In comparison, we can see that even when there is label noise, we only lose a factor of Opλq in the population loss on the clean distribution, which can be chosen as any slow-growing function of n. If y J pkpX, Xqq ´1y grows much slower than n, by choosing an appropriate λ, our result indicates that the underlying distribution is learnable in presence of additive label noise. See Remark 5.2 for an example. Remark 5.2. Arora et al. (2019b) proved that two-layer ReLU neural nets trained with gradient descent can learn a class of smooth functions on the unit sphere. Their proof is by showing y J pkpX, Xqq ´1y " Op1q if y i " gpx i q p@i P rnsq for certain function g, where kp¨, ¨q is the NTK corresponding to two-layer ReLU nets. Combined with their result, Theorem 5.1 implies that the same class of functions can be learned by the same network even if the labels are noisy.

Next we use Theorem 5.1 to provide generalization bounds for the classification settings described in Section 3.1. For binary classification, we treat the labels as ˘1 and consider a single-output neural net; for K-class classification, we treat the labels as their one-hot encodings (which are K standard unit vectors in R K ) and consider a K-output neural net. Again, we use 2 loss and wide neural nets so that it suffices to consider the kernel ridge regression solution ((8) or ( 9)). Theorem 5.2 (Binary classification). Consider the binary classification setting stated in Section 3.1. Let tpx i , y i , ỹi qu n i"1 Ă R d ˆt˘1u ˆt˘1u be i.i.d. samples from that process. Recall that Prrỹ i " y i |y i s " p (0 ď p ă 1 2 ). Denote X " px 1 , . . . , x n q, y " py 1 , . . . , y n q J and ỹ " pỹ 1 , . . . , ỹn q J . Consider the kernel ridge regression solution in (8): f ˚pxq " kpx, Xq J `kpX, Xq `λ2 I ˘´1 ỹ. Suppose that the kernel matrix satisfies trrkpX, Xqs " Opnq. Then with probability at least 1 ´δ, the classification error of f ˚on the clean distribution D satisfies

Pr px,yq"D rsgnpf ˚pxqq ‰ ys ď λ `Op1q 2 c y J pkpX, Xqq ´1y n `1 1 ´2p O ¨?p λ `d p log 1 δ n `c log n δλ n ‚.

Theorem 5.3 (Multi-class classification). Consider the K-class classification setting stated in Section 3.1. Let tpx i , c i , ci qu n i"1 Ă R d ˆrKs ˆrKs be i.i.d. samples from that process. Recall that Prrc i " c 1 |c i " cs " p c 1 ,c (@c, c 1 P rKs), where the transition probabilities form a matrix P P R KˆK . Let gap " min c,c 1 PrKs,c "c 1 pp c,c ´pc 1 ,c q.

Let X " px 1 , . . . , x n q, and let y i " e pciq P R K , ỹi " e pciq P R K be one-hot label encodings. Denote Y " py 1 , . . . , y n q P R Kˆn , Ỹ " p ỹ1 , . . . , ỹn q P R Kˆn , and let ỹphq P R n be the h-th row of Ỹ . Define a matrix Q " P ¨Y P R Kˆn , and let q phq P R n be the h-th row of Q.

Consider the kernel ridge regression solution in (9): f phq pxq " kpx, Xq J `kpX, Xq `λ2 I ˘´1 ỹphq . Suppose that the kernel matrix satisfies trrkpX, Xqs " Opnq. Then with probability at least 1 ´δ, the classification error of f " pf phq q K k"1 on the clean data distribution D is bounded as Figure 1 : Performance on binary classification using 2 loss. Setting 1: MNIST; Setting 2: CIFAR.

Pr px,cq"D " c R argmax hPrKs f phq pxq ı ď 1 gap ¨λ `Op1q 2 K ÿ h"1 c pq phq q J pkpX, Xqq ´1q phq n `K ¨O ¨1 λ `d log 1 δ 1 n `c log n δ 1 λ n ‚‚ .

Note that the bounds in Theorems 5.2 and 5.3 only depend on the clean labels instead of the noisy labels, similar to Theorem 5.1.

6. EXPERIMENTS:

In this section, we empirically verify the effectiveness of our regularization methods RDI and AUX, and compare them against gradient descent or stochastic gradient descent (GD/SGD) with or without early stopping. We experiment with three settings of increasing complexities:

Setting 1: Binary classification on MNIST ("5" vs. "8") using a two-layer wide fully-connected net.

Setting 2: Binary classification on CIFAR ("airplanes" vs. "automobiles") using a 11-layer convolutional neural net (CNN).

Setting 3: CIFAR-10 classification (10 classes) using standard ResNet-34.

For detailed description see Appendix D. We obtain noisy labels by randomly corrupting correct labels, where noise rate/level is the fraction of corrupted labels (for CIFAR-10, a corrupted label is chosen uniformly from the other 9 classes.)

6.1. PERFORMANCE OF REGULARIZATION METHODS:

For Setting 1 (binary MNIST), we plot the test errors of different methods under different noise rates in Figure 1a . We observe that both methods GD+AUX and GD+RDI consistently achieve much lower test error than vanilla GD which over-fits the noisy dataset, and they achieve similar test error to GD with early stopping. We see that GD+AUX and GD+RDI have essentially the same performance, which verifies our theory of their equivalence in wide networks (Theorem 4.1).

For Setting 2 (binary CIFAR), Figure 1b shows the learning curves (training and test errors) of SGD, SGD+AUX and SGD+RDI for noise rate 20% and λ " 4. Additional figures for other choices of λ are in Figure 5 . We again observe that both SGD+AUX and SGD+RDI outperform vanilla SGD and are comparable to SGD with early stopping. We also observe a discrepancy between SGD+AUX and SGD+RDI, possibly due to the noise in SGD or the finite width.

Finally, for Setting 3 (CIFAR-10), architecture . Furthermore, AUX does not over-fit (the last epoch performs similarly to the best epoch). In addition, we find that in this setting classification performance is insensitive of whether MSE or CCE is used as the loss function.

6.2. DISTANCE OF WEIGHTS TO INITIALIZATION, VERIFICATION OF THE NTK REGIME:

We also track how much the weights move during training as a way to see whether the neural net is in the NTK regime. For Settings 1 and 2, we find that the neural nets are likely in or close to the NTK regime because the weight movements are small during training. Figure 3 shows in Setting 2 how much the 4-th layer weights move during training. Additional figures are provided as Figures 6 to 8 .

Table 2 summarizes the relationship between the distance to initialization and other hyper-parameters that we observe from various experiments. Note that the weights tend to move more with larger noise level, and AUX and RDI can reduce the moving distance as expected (as shown in Figure 3 ).

The ResNet-34 in Setting 3 is likely not operating in the NTK regime, so its effectiveness cannot yet be explained by our theory. This is an intriguing direction of future work.

# samples noise level width regularization strength λ learning rate Distance Õ Õ -OE -Table 2 : Relationship between distance to initialization at convergence and other hyper-parameters. "Õ": positive correlation; "OE": negative correlation; '-': no correlation as long as width is sufficiently large and learning rate is sufficiently small.

7. CONCLUSION:

Towards understanding generalization of deep neural networks in presence of noisy labels, this paper presents two simple regularization methods and shows that they are theoretically and empirically effective. The theoretical analysis relies on the correspondence between neural networks and NTKs. We believe that a better understanding of such correspondence could help the design of other principled methods in practice. We also observe that our methods can be effective outside the NTK regime. Explaining this theoretically is left for future work.

A DIFFERENCE TRICK:

We give a quick analysis of the "difference trick" described in Footnote 2, i.e., let f pθ, xq "

? 2 2 gpθ 1 , xq ´?2 2 gpθ 2 , xq where θ " pθ 1 , θ 2 q, and initialize θ 1 and θ 2 to be the same (and still random). The following lemma implies that the NTKs for f and g are the same.

Lemma A.1. If θ 1 p0q " θ 2 p0q, then (i) f pθp0q, xq " 0, @x;

(ii) x∇ θ f pθp0q, xq, ∇ θ f pθp0q, x 1 qy " x∇ θ1 gpθ 1 p0q, xq, ∇ θ1 gpθ 1 p0q, x 1 qy, @x, x 1 .

Proof. (i) holds by definition. For (ii), we can calculate that

@ ∇ θ f pθp0q, xq, ∇ θ f pθp0q, x 1 q D " @ ∇ θ1 f pθp0q, xq, ∇ θ1 f pθp0q, x 1 q D `@∇ θ2 f pθp0q, xq, ∇ θ2 f pθp0q, x 1 q D " 1 2 x∇ θ1 gpθ 1 p0q, xq, ∇ θ1 gpθ 1 p0q, xqy `1 2 @ ∇ θ2 gpθ 2 p0q, xq, ∇ θ2 gpθ 2 p0q, x 1 q D " @ ∇ θ1 gpθ 1 p0q, xq, ∇ θ1 gpθ 1 p0q, x 1 q D .

The above lemma allows us to ensure zero output at initialization while preserving NTK. As a comparison, Chizat and Bach (2018) proposed the following "doubling trick": neurons in the last layer are duplicated, with the new neurons having the same input weights and opposite output weights. This satisfies zero output at initialization, but destroys the NTK. To see why, note that with the "doubling trick", the network will output 0 at initialization no matter what the input to its second to last layer is. Thus the gradients with respect to all parameters that are not in the last two layers are 0.

In our experiments, we observe that the performance of the neural net improves with the "difference trick." See Figure 4 . This intuitively makes sense, since the initial network output is independent of the label (only depends on the input) and thus can be viewed as noise. When the width of the neural net is infinity, the initial network output is actually a zero-mean Gaussian process, whose covariance matrix is equal to the NTK contributed by the gradients of parameters in its last layer. Therefore, learning an infinitely wide neural network with nonzero initial output is equivalent to doing kernel regression with an additive correlated Gaussian noise on training and testing labels. 

GD+early stop GD:

Figure 4 : Plot of test error for fully connected two-layer network on MNIST (binary classification between "5" and "8") with difference trick and different mixing coefficients α, where f pθ 1 , θ 2 , xq " a α 2 gpθ 1 , xq ´b 1´α 2 gpθ 1 , xq. Note that this parametrization preserves the NTK. The network has 10,000 hidden neurons and we train both layers with gradient descent with fixed learning rate for 5,000 steps. The training loss is less than 0.0001 at the time of stopping. We observe that when α increases, the test error drops because the scale of the initial output of the network goes down.

B MISSING PROOF IN SECTION 4:

Proof of Theorem 4.1. The gradient of LAUX λ pθ, bq can be written as

∇ θ LAUX λ pθ, bq " ÿ n i"1 a i φpx i q, ∇ bi LAUX λ pθ, bq " λa i , i " 1, . . . , n,

where a i " φpx i q J pθ ´θp0qq `λb i ´ỹ i pi P rnsq. Therefore we have ∇ θ LAUX λ pθ, bq "

ř n i"1 1 λ ∇ bi LAUX λ

pθ, bq ¨φpxq. Then, according to the gradient descent update rule (7), we know that θptq and bptq can always be related by θptq " θp0q `řn

i"1 1 λ b i ptq ¨φpx i q. It follows that θpt `1q " θptq ´η ÿ n i"1

`φpx i q J p θptq ´θp0qq `λb i ptq ´ỹ i ˘φpx i q " θptq ´η ÿ n i"1

`φpx i q J p θptq ´θp0qq ´ỹ i ˘φpx i q ´ηλ

ÿ n i"1 b i ptqφpx i q " θptq ´η ÿ n i"1

`φpx i q J p θptq ´θp0qq ´ỹ i ˘φpx i q ´ηλ 2 p θptq ´θp0qq.

On the other hand, from (6) we have θpt `1q " θptq ´η ÿ n i"1

`φpx i q J pθptq ´θp0qq ´ỹ i ˘φpx i q ´ηλ 2 pθptq ´θp0qq.

Comparing the above two equations, we find that tθptqu and t θptqu have the same update rule. Since θp0q " θp0q, this proves θptq " θptq for all t.

Now we prove the second part of the theorem. Notice that LRDI λ pθq is a strongly convex quadratic function with Hessian ∇ 2 θ LRDI λ pθq "

ř n i"1 φpx i qφpx i q J `λ2 I " ZZ J `λ2 I, where Z " pφpx 1 q, . . . , φpx n qq. From the classical convex optimization theory, as long as η ď

1 }ZZ J `λ2 I} " 1 }Z J Z`λ 2 I} "

1 }kpX,Xq}`λ 2 , gradient descent converges linearly to the unique optimum θ ˚of LRDI λ pθq, which can be easily obtained:

θ ˚" θp0q `ZpkpX, Xq `λ2 Iq ´1 ỹ.

Then we have φpxq J pθ ˚´θp0qq " φpxq J ZpkpX, Xq `λ2 Iq ´1 ỹ " kpx, Xq J `kpX, Xq `λ2 I ˘´1 ỹ, finishing the proof.

C MISSING PROOFS IN SECTION 5:

C.1 PROOF OF THEOREM 5.1

Define ε i " ỹi ´yi (i P rns), and ε " pε 1 , . . . , ε n q J " ỹ ´y.

We first prove two lemmas.

Lemma C.1. With probability at least 1 ´δ, we have

g f f e n ÿ i"1 pf ˚px i q ´yi q 2 ď λ 2 b y J pkpX, Xqq ´1y `σ 2λ a trrkpX, Xqs `σa 2 logp1{δq.

Proof. In this proof we are conditioned on X and y, and only consider the randomness in ỹ given X and y.

First of all, we can write pf ˚px 1 q, . . . , f ˚px n qq J " kpX, Xq `kpX, Xq `λ2 I ˘´1 ỹ, so we can write the training loss on true labels y as

EQUATION

Next, since ε, conditioned on X and y, has independent and subgaussian entries (with parameter σ), by (Hsu et al., 2012) , for any symmetric matrix A, with probability at least 1 ´δ,

EQUATION

)

Let A " kpX, Xq `kpX, Xq `λ2 I ˘´1 and let λ 1 , . . . , λ n ą 0 be the eigenvalues of kpX, Xq. We have Finally, since `kpX, Xq `λ2 I ˘´2 ĺ 1 4λ 2 pkpX, Xqq ´1 (note pλ i `λ2 q 2 ě 4λ i ¨λ2 ), we have

EQUATION

EQUATION

)

The proof is finished by combining (11), ( 13) and ( 14).

Let H be the reproducing kernel Hilbert space (RKHS) corresponding to the kernel kp¨, ¨q. Recall that the RKHS norm of a function f pxq "

α J kpx, Xq is }f } H " b α J kpX, Xqα.

Lemma C.2. With probability at least 1 ´δ, we have

EQUATION

Proof.

In this proof we are still conditioned on X and y, and only consider the randomness in ỹ given X and y. Note that f ˚pxq " α J kpx, Xq with α " `kpX, Xq `λ2 Since ε has independent and subgaussian (with parameter σ) coordinates, using (12) with A " I, with probability at least 1 ´δ we have ? ε J ε ď σ ´?n `a2 logp1{δq

¯.

Now we prove Theorem 5.1.

Proof of Theorem 5.1. First, by Lemma C.1, with probability 1 ´δ{3,

g f f e n ÿ i"1 pf ˚px i q ´yi q 2 ď λ 2 b y J pkpX, Xqq ´1y `σ 2λ a trrkpX, Xqs `σa 2 logp3{δq,

which implies that the training error on the true labels under loss function is bounded as

1 n n ÿ i"1 pf ˚px i q, y i q " 1 n n ÿ i"1 p pf ˚px i q, y i q ´ py i , y i qq ď 1 n n ÿ i"1 |f ˚px i q ´yi | ď 1 ? n g f f e n ÿ i"1 |f ˚px i q ´yi | 2 ď λ 2 c y J pkpX, Xqq ´1y n `σ 2λ c trrkpX, Xqs n `σc 2 logp3{δq n .

Next, for function class F B " tf pxq " α J kpx, Xq : }f } H ď Bu, Bartlett and Mendelson (2002) showed that its empirical Rademacher complexity can be bounded as

RS pF B q fi 1 n E γ"t˘1u n « sup f PF B n ÿ i"1 f px i qγ i ff ď B a trrkpX, Xqs n .

By Lemma C.2, with probability at least 1 ´δ{3 we have

}f ˚}H ď b y J pkpX, Xq `λ2 Iq ´1 y `σ λ ´?n `a2 logp3{δq ¯fi B 1 .

We also recall the standard generalization bound from Rademacher complexity (see e.g. (Mohri et al., 2012) ): with probability at least 1 ´δ, we have

EQUATION

)

Then it is tempting to apply the above bound on the function class F B 1 which contains f ˚. However, we are not yet able to do so, because B 1 depends on the data X and y. To deal with this, we use a standard -net argument on the interval that B 1 must lie in: (note that }y} " Op ? nq)

B 1 P " σ λ ´?n `a2 logp3{δq ¯, σ λ ´?n `a2 logp3{δq ¯`O ´n λ ¯ı .

The above interval has length O `n λ ˘, so its -net N has size O `n λ ˘. Using a union bound, we apply the generalization bound (15) simultaneously on F B for all B P N : with probability at least 1 ´δ{3 we have

sup f B PF # E px,yq"D r pf pxq, yqs ´1 n n ÿ i"1 pf px i q, y i q + ď 2 RS pF B q `O ˜c log n δλ n ¸, @B P N .

By definition there exists B P N such that B 1 ď B ď B 1 ` . Then we also have f ˚P F B . Using the above bound on this particular B, and putting all parts together, we know that with probability at least 1 ´δ, E px,yq"D r pf ˚pxq, yqs

ď 1 n n ÿ i"1 pf ˚px i q, y i q `2 RS pF B q `O ˜c log n δλ n ḑ λ 2 c y J pkpX, Xqq ´1y n `σ 2λ c trrkpX, Xqs n `σc 2 logp3{δq n `2pB 1 ` q a trrkpX, Xqs n `O ˜c log n δλ n ḑ λ 2 c y J pkpX, Xqq ´1y n `σ 2λ c trrkpX, Xqs n `σc 2 logp3{δq n `2a trrkpX, Xqs n ˆby J pkpX, Xqq ´1 y `σ λ ´?n `a2 logp3{δq ¯` ˙`O ˜c log n δλ n " λ 2 c y J pkpX, Xqq ´1y n `5σ 2λ c trrkpX, Xqs n `σc 2 logp3{δq n `2a trrkpX, Xqs n ˆby J pkpX, Xqq ´1 y `σ λ a 2 logp3{δq ` ˙`O ˜c log n δλ n ¸.

Then, using trrkpX, Xqs " Opnq and choosing " 1, we obtain C.2 PROOF OF THEOREM 5.2

Proof of Theorem 5.2. We will apply Theorem 5.1 in this proof. Note that we cannot directly apply it on tpx i , y i , ỹi qu because the mean of ỹi ´yi is non-zero (conditioned on px i , y i q). Nevertheless, this issue can be resolved by considering tpx i , p1 ´2pqy i , ỹi qu instead. 4 Then we can easily check that conditioned on y i , ỹi ´p1 ´2pqy i has mean 0 and is subgaussian with parameter σ " Op ? pq.

With this change, we are ready to apply Theorem 5.1.

Define the following ramp loss for u P R, ȳ P t˘p1 ´2pqu:

ramp pu, ȳq "

$ & % p1 ´2pq, uȳ ď 0, p1 ´2pq ´1 1´2p uȳ, 0 ă uȳ ă p1 ´2pq 2 , 0, uȳ ě p1 ´2pq 2 .

It is easy to see that ramp pu, ȳq is 1-Lipschitz in u for ȳ P t˘p1 ´2pqu, and satisfies ramp pȳ, ȳq " 0 for ȳ P t˘p1 ´2pqu. Then by Theorem 5.1, with probability at least 1 ´δ, E px,yq"D r ramp pf ˚pxq, p1 ´2pqyqs 

:

"Width" refers to number of nodes in a fully connected layer or number of channels in a convolutional layer.

We can ensure small or even zero output at initialization by either multiplying a small factor(Arora et al., 2019a;b), or using the following "difference trick": define the network to be the difference between two networks with the same architecture, i.e., f pθ, xq " ? 2 2 gpθ1, xq ´?2 2 gpθ2, xq; then initialize θ1 and θ2 to be the same (and still random); this ensures f pθp0q, xq " 0 p@xq at initialization, while keeping the same value of xφpxq, φpx 1 qy for both f and g. See more details in Appendix A.

For the theoretical results in this paper we use 2 loss. In Section 6 we will present experimental results of both 2 loss and cross-entropy loss for classification.

For binary classification, only the sign of the label matters, so we can imagine that the true labels are from t˘p1 ´2pqu instead of t˘1u, without changing the classification problem.

This is because the NTK scaling at initialization balances the norm of the gradient in each layer, and the first and the last layer weights have smaller norm than intermediate layers. Since different weights are expected to move the same amount during training(Du et al., 2018a), the first and last layers move relatively further from their initializations. By freezing them during training, we make the network closer to the NTK regime.

Figures and Tables:

Figure 2: Test accuracy during CIFAR-10 training (noise 0.4).

px,yq"D r pf ˚pxq,

observe little over-fitting to noise in the first stage of training (before learning rate decay), we restrict the regularization power of AUX by applying weight decay on auxiliary variables, and dividing their weight decay factor by 10 after each learning rate decay.See Figures 5 to 8 for additional results for Setting 2 with different λ's.

Figure 5: Training (dashed) & test (solid) errors vs. epoch for Setting 2. Noise rate " 20%, λ P t0.25, 0.5, 1, 2, 4, 8u. Training error of AUX is measured with auxiliary variables.

Theorem 4.1. Fix a learning rate η ą 0. Consider gradient descent on LRDI bq with initialization θp0q and bp0q " 0:θp0q " θp0q, θpt `1q " θptq ´η∇ θ LAUX λ p θptq, bptqq, t " 0, 1, 2, . . . bp0q " 0, bpt `1q " bptq ´η∇ b LAUX λ p θptq, bptqq, t " 0, 1, 2, . . .

Table content: <table><tr><td/><td colspan="2">λ with initialization θp0q:</td></tr><tr><td>θpt `1q " θptq ´η∇ θ</td><td>LRDI λ pθptqq, t " 0, 1, 2, . . .</td><td>(6)</td></tr><tr><td>and gradient descent on LAUX</td><td/><td/></tr><tr><td/><td/><td>(7)</td></tr></table>

Table 1 shows the test accuracies of training with and without AUX.

Table content: <table><tr><td colspan="3">Published as a conference paper at ICLR 2020</td><td/><td/></tr><tr><td>Noise rate</td><td>0</td><td>0.2</td><td>0.4</td><td>0.6</td></tr><tr><td>Normal CCE (early stop)</td><td>94.05˘0.07</td><td>89.73˘0.43</td><td>86.35˘0.47</td><td>79.13˘0.41</td></tr><tr><td>Normal MSE (early stop)</td><td>93.88˘0.37</td><td>89.96˘0.13</td><td>85.92˘0.32</td><td>78.68˘0.56</td></tr><tr><td>CCE+AUX (last)</td><td>94.22˘0.10</td><td>92.07˘0.10</td><td>87.81˘0.37</td><td>82.60˘0.29</td></tr><tr><td>CCE+AUX (best)</td><td>94.30˘0.09</td><td>92.16˘0.08</td><td>88.61˘0.14</td><td>82.91˘0.22</td></tr><tr><td>MSE+AUX (last)</td><td>94.25˘0.10</td><td>92.31˘0.18</td><td>88.92˘0.30</td><td>83.90˘0.30</td></tr><tr><td>MSE+AUX (best)</td><td>94.32˘0.06</td><td>92.40˘0.18</td><td>88.95˘0.31</td><td>83.95˘0.30</td></tr><tr><td>(Zhang and Sabuncu, 2018)</td><td>-</td><td>89.83˘0.20</td><td>87.62˘0.26</td><td>82.70˘0.23</td></tr><tr><td colspan="5">Table 1: CIFAR-10 test accuracies of different methods under</td></tr><tr><td>different noise rates.</td><td/><td/><td/><td/></tr></table>

Xq `λ2 Iq ´1 y `bε J pkpX, Xq `λ2 Iq ´1 ε

Table content: <table><tr><td>ď</td><td>b y J pkpX, Xq `λ2 Iq ´1 y</td><td>`?ε J ε λ</td><td>.</td></tr></table>

Peer review (iclr2020):
In this paper, based on the effectiveness of early stopping in the training of noisily labeled samples, the authors proposed two intuitive (and novel) regularization methods: (1) regularizing using distance to initialization (2) adding an auxiliary variable b_i for every input x_i during training. In terms of theory, the authors showed that in the NTK regime, both regularization methods trained with gradient descent are equivalent to kernel ridge regression. Moreover, the authors also provided a generalization bound of the solution on the clean data distribution when trained with noisy label, which was not addressed in previous research. Overall, the paper is very well-organized and well-written. The contribution of the paper is significant, and numerical results also vindicate the theory developed in the manuscript. I recommend accepting the paper. Two minor questions that are not going to affect my rating: 1. The theory developed in the paper depends highly on NTK. What if the network is not sufficiently wide (which is usually the case in practice), and the loss function is not L2? 2. In the second method using the auxiliary variable, it seems that every training sample x_i needs a variable b_i. Is this going to cause any problem in practice if data augmentation is used?