Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized NN
Authors: Bin Dong
Jikai Hou
Yiping Lu
Zhihua Zhang
Venue:  (2020)
Keywords: Distillation
Learning Thoery
Corrupted Label

Abstract: Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, Anisotropic In- formation Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later. Motivated by the recent development on theoretically analyzing overparame- terized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK). AIR facilities a new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation al- gorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoret- ically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of l2 distance, while the previous result was on convergence in 0-1 loss. The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.


Under review as a conference paper at ICLR 2020
DISTILLATION ~ EARLY STOPPING? HARVESTING DARK KNOWLEDGE UTILIZING ANISOTROPIC INFORMATION RETRIEVAL FOR OVERPARAMETERIZED NEURAL NETWORK
Anonymous authors authorrs Paper under double-blind review
ABSTRACT
Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized. we gue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, Anisotropic Information Retrieval (AIR), which means that the neural network tends to fit the informative information first and the -nnirrmrmtttiee information (including noise) later. Mo- tivated by the recent development on theoretically analyzing overparameterized neural networks, we can characterize AIR by the eigenspace the Neural Tangent (NTKe((NTK) AIR facilities new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation algo- rithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoret- ically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of l2 distance, while the previous result was on convergence in 0-1 loss. The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.
INTRODUCTION
Deep learning achieves state-of-thh-ar results many tasks in computer vision and natural language processing LeCun al. (2015). Among these tasks, image classification is considered the fundamental tasks since classification networks are commonly used as base networks for other problems. In order to achieve higher accuracy using network with similar complexity the base network, distillation has been proposed, which aims to utilize the prediction of one (teacher) network guide the training of another (student) network. In Hinton et al. (2013), the authors suggested to generate soft target by a heavy-duty teacher network to guide the training of a light-weighten student network. More interestingly. [Furlanello al. (2018); Bagherinezhad al. (2018) proposed to train student network parameterized identically the teacher network. Surprisingly, the student network significantly outperforms the teacher network. Later, was suggested by Zagoruyko Komodakis 62016a); Huang & Wang (217)); Czarnecki al. (2017) to transfer knowledge of representations, such as attention maps and gradients of the classifier, to help with the training of the student network. In this work, we focus the distillation utilizing the network outputs Hinton al. (2015) Furlanello et al. (2018); Yang et al. (2018a); BBaheeinezzad et al. (2018); Yang et al. (0018)).
of
To explain the effectiveness of distillation, |Hinton al. (2015) suggested that instead of the hard labels (i.e one-hot vectors), the soft labels generated by the pre- trainee teacher network provide extra information, which is called the "Dark Knowledge". The "Dark knowledge" is the knowledge

---

Under review as a conference paper at ICLR 2020
encoded by the relative probabilities of the incorrect outputs. Hinton et al. (2013) Furlanello let al. (2018); Yang et al. (2018a), the authors pointed out that secondary information, i. the semantic similarity between different classes, is part of the "Dark Knowledge", and Bagherinezhad al. (2018) observed that the "Dark Knowledge" can help to refine noisy labels. In this paper, we would like to answer the following question: we theoretically explain how neural networks learn the Dark Knowledge? Answering this question will help us to understand the regularization effect of distillation.
In this work, we assume that the teacher network is overparameterized, which means that it can memorize all the labels via gradient descent training Du et al. (2018b a): Oymak Soltanokotabi (2018); Alleen-lennna et al. (2018). In this case, if we train the overparametenized teacher network until convergence, the network's 's output coincides exactly with the ground truth hard labels. This because the logits corresponding to the incorrect classes are all zero, and hence no "Dark knowledge" can be extracted. Thus, we claim that the core factor that enables an overparameterized network to learn "Dark knowledge" is early stopping.
What's more, Arpit et al. (2017) Rahaman et al. (2018); Xu et al. (2019) observed that "Dark knowledge" represents the discrepancy of convergence speed of different types of information during the training of the neural network. Neural network tends to fit informative information, such as simple pattern, faster than -on-informative and unwanted information such noise. Similar phenomenon was observed in the inverse scale space theory for image restoration Scherzer & Groetsch (200T); Burger et al. (2006); Xu & Osher (2007); [Shi & Osher (2008). In our paper, call this effect Anisotropic Information Retrieval (AIR).
With the aforementioned interpretation of distillation, We further utilize AIR refine noisy labels by introducing a new self-distillation algorithm. To extract anisotropic information, we sequentially extract knowledge from the output the network the previous epoch to supervise the training in the next epoch. By dynamically adjusting the strength of the supervision, we can theoretically prove that the proposed self-distillation algorithm can recover the correct labels, and empirically the algorithm achieves the oftte-of-the-ane results on Fashion MNIST and CIFAR10. The benefit brought by our theoretical study twofold. Firstly, the existing approach using large networks [Li al. (2019); ZZang & Sabuncu (2018) often requires validation set to early terminate the network training. However, our analysis shows that our algorithm can sustain long training without overfitting the noise which makes the proposed algorithm more mser-friendly. Secondly, analysis is based on an fo-loss of the clean labels which enables the algorithm to generate a trained network with a bigger margin and hence generalize better.
1.1 CONTRIBUTIONS
We summarize our contributions as follows
This paper aims to understand distillation theoretically (i.e. understand the regularization effect of distillation). Distillation works due to the soft targets generated by the teacher network. Based on the observation that the overparameterized network can exactly fit the one-hot labels which contain no dark knowledge, we theoretically justify that early stopping is essential for an overparameterized teacher network to extract dark knowledge from the hard labels. This provides a new understanding of the regularization effect of distillation.
This is the first attempt to theoretically understand the role of distillation in noisy label refinery using overparameterized neural networks. Inspired by Li et al. (2019). we utilize distillation to propose a self-distillation algorithm to train a neural network under label corruption. The algorithm is theoretically guaranteed to recover the unknown correct labels in terms of the f2-loss rather than the previous 0-1-loss. This enables the algorithm to generate trained network whose output has bigger margin and hence generalizes better. Furthermore, our algorithm does not need a validation set to early stop during training, which makes it more hyperparameter friendly. The theoretical understanding of the overparameterized networks encourages us to use large models which empirically produce better results.
2

---

Under review as a conference paper at ICLR 2020
2 DISTILLATION 2 EARLY STOPPING?
2.1 No EARLY STOPPING, No DARK KNOWLEDGE
As mentioned in the introduction, an overparameterrized teacher network is able to extract dark knowledge from the on-hot hard labels because of early stopping. In this section we present an experiment to verify this effect of early stopping, where we use big model as a teacher to teach smaller model as the student.
In this experiment, we train a WRN-28 Zagoruyko & Komodakis (2016b) on CIFAR Krizhevsky & Hinton (2009) as the teacher model, and a 5-layers CNN the student. The experimental details are in the supplementary material. The teacher model was trained by 40, 80, 120, 160 and 200 epochs respectively. The results of knowledge distillation by the student model is shown in Fig.
63.8
Teacher
Teacher Testing
90
TAPE tecch
Stuter:tcc
80
10
12.
2.0
360
3.0
200
Epach
Figure 1: A good teacher may not be able to produce good student. To maximize the effectiveness of distillation, you should early stop your epoch at a proper time.
As we can see, the teacher model does not suffer from overfitting during the training, while it does not always educate a good student model either. As suggested by Yang et al. (2018a), a more tolerant teacher educates better students. The teacher model trained with 80 epochs produces the best result which indicates that early stopping of the bigger model can extract more informative information for distillation.
2.2 ANISOTROPIC INFORMATION RETRIEVAL.
In this paper, introduce concept called Anisotropic Information Retrieval (AIR), which means to exploit the discrepancy of the convergence speed of different types of information during the training of an overparameterized neural network. An important observation of AIR is that informative information tends to converge faster than non-informative and unwanted information such as noise.
Selective bias of an iterative algorithm approximate function has long been discovered in different areas. For example, Xu (1992) observed that iterative linear equation solver fits the low frequency component first, and they proposed a multigrid algorithm to exploit this property. In image processing, Scherzer & Groetsch (200T); Burger et al. (006 Xu & Osher (2007); Shi & Osher (2008) proposed inverse scale space methods that recover image features earlier the iteration and noise comes back later. In kernel learning, early stopping of gradient descent is equivalent to the ridge regression [mmale] Zhou (007): Yao al. (2007): Vito et al. (2005), which means that bias from the eigenspace corresponding to the larger cigenvalues is reduced quicker by the gradient descent.
Under the neural network setting, Rahaman et al. (2018) 8); Xu et al. (2099) observed that neural networks find low frequency patterns more easily. Cicek al. (2018) discovered that noisy labels can slow down the training. Arpit et al. (2017); Rolnick et al. (2017) studied the memorization effects of the deep networks, and revealed that, during training, neural networks first memorize the data with clean labels and later with the wrong labels. In the next subsection, we will characterize AIR of overparametrized neural networks using the Neural Tangent Kernel Jacot Jacot al. (2018).
3

---

Under review as a conference paper at ICLR 2020
2.3 AIR AND NEURAL TANGENT KERNEL
In Jacot et al. 2018), the authors introduced the Neural Tangeni Kernel to characterize the trajectory of gradient descent algorithm learning infinitely wide neural networks. Denote f(0 R the output of neural network with 0 E R" being the trainable parameter and x E RP the input. Consider training the neural network using the E-lloss on the dataset {(1), y1))= E R R:
{(0)
Σ (f(0, 24) -.)
0, v)((() to train the neural network. Let ue 2)))(() E IR" be the network outputs on all the data {x;} at iteration and (y))ke was shown by et al. (2018b); Oymak & Soltanolkotabi (2018): Li al. (2019) that the evolution of the error u4 can be formulated in a quasi-linear form,
We use the gradient descent pt+1
(ut y) (I - mlle (u+ v).
where
lle ((/ ( a(0++1 0z), da, Of(O,, zz,)
00
00
i
is known that H. - H, = o(1) with respect to d, where H, = (( @f(0
05
is
09
an positive semi-definite Gram matrix and is the width of the neural network Oymak & Soltanolkotabi (018):: Li et al. (2019). This means that H, = H, when the neural network is infinitely wide. It was further shown by Oymak & Soltanolkotabi (2018); Li et al. (019); Du et al. (2118a) that when the neural network is infinitely wide, the Gram matrix is static, i.c. H, H The static Gram matrix H is referred to as the Neural Tangent KernelNNTK et al. 2018).
Note that H" is symmetric positive semi-definite matrix. Assume that A eigenvalues and 61, E2. ... Cn are the corresponding eigenvectors. The eigenvectors are orthogonal e, Gj >= 0 and H* Σ Xeeee Consider the evolution of the projection of the loss function in different eigenspaces
An are its
((w, 10ve-1)0 e), = ((I nHH (m: - v), = ((w,-()))
- y), (I=("()))"))))
(1 m).:)) ((u, - y), es).
We can see that the component lies the eigenspace with a larger eigenvalue converges faster. Therefore, AIR describes the phenomenon that the gradient descent algorithm searches for information components corresponding to different eigenspaces at different rates. Rahaman et al. (2018); Arpit et al. (2017): Arora et al. 2019); Zhang & Sabuncu (2018) has shown that that one possible reason of neural network's good generalization property is that neural network fits useful information faster. Thus in our paper, regard informative information the eigenspaces associated with the largest few eigenvalues of NTK.
0.50
0.45
Decreasing
0.30
0.25
10
20
50
Noise Level
Figure 2: Components of label noise in the largest five eigensapces of NTK are decreasing.
We denote the projection of supervision signal to the eigenspace with a larger eigenvalue as useful information. In Figure We calculate the ratio of the 172norm of the label vector provided by the
4

---

Under review as a conference paper at ICLR 2020
saff-distillation algorithm lies in the top-5 eigenspace as a representative of informative information. We can see that the informative information decreases when the noise level increases. This motivates us to further explore how "Dark Knowledge" helps with label refinery.
NOISY LABEL REFINERY
Supervised learning requires high quality labels. However, due to noisy crowd-sourcing platforms Khetan al. (2017) and data augmentation Dageline[BaherineeZleee et al. (2018), it hard to acquire entirlly clean labels for training. On the other hand, successive deep models often have huge capacities with millions or even billions of parameters. Such huge capacity enables the network to memorize all the labels, right or wrong Zhang et al.] (2016). which makes learning deep neural networks with noisy labels a challenging task. |Apit et al. (2017); Rolnick et al. (2017) HHan et al.) (2018) pointed out that neural networks often fit the clean labels before the noisy ones during training. Recently, Li et al. (2019) theoretically showed that early stopping can clean up label noise with overparametrized neural networks. This, together with our understanding of distillation with AIR, inspired us to use distillation for noisy label refinery. We shall introduce a new self-distillation algorithm with theoretically guaranteed recovery of the clean labels under suitable assumptions.
3.1 RELATED WORKS
Training deep models on datasets with label corruption is an important and challenging problem that has attracted much attention lately. In|Ma et al. (2018), the authors proposed to regularize the Local Intrinsic Dimensionality of deep representations to detect label noise. [Tanaka et al. (2018) proposed a joint optimization framework simultancously optimize the network parameters and output labels. Zhang & Sabuncu (2018) introduced loss function generalizing the cross entropy for robust learning. The approach that most relevant to ours is called learning with mentor. For example, Diang et al. (2017) used a mentor network to learn the curriculum for the student network. Han et al. (2018) ; Yu et al. (2019) introduced two networks that teach each other reject wrong labels. Hu et al. (201)) designed a new regularizer to restrict every weight vector to be close to its initialization for all iterations thus enforcing the same regularization effect early stopping.
In the literature of distillation, [Bagherinezhad et al. (2018) first utilized distillation to refine noisy labels of ImageNet, while Yang al. 2b)88) introduced a distillation method to complete teacher- student training in one generation. The latter is most related to our proposed self-distillation algorithm. However, the difference that their model aims to ensemble diverse models one training trajectory but ours aims to utilize AIR to refine noisy labels during the training.
3.2 THE SELF- DISTILLATION ALGORITHM
It is known that the label noise lies in the eigenspaces associated to small eigenvalues Arora et al.) (2019): Li et al. (2019). Thus, Li et al. (2019) used early stopping to remove label noise. However, early stopping hard to tune and sometimes leads to unsatisfactory results. In this section, proposed a self-distillation algorithm with an excellent empirical performance and a theoretical guarantee to recover the correct labels under certain conditions but without the requirement of early stopping.
From the perspective of AIR, we observe that the knowledge learned in early epochs informative information (i.e. the eigenspaces associated with the largest few eigenvalues of NTK) and can be used to refine the training for later epochs. In other words, the algorithm distills knowledge sequentially to guide the training of the model in later epochs by the knowledge distilled by the model from earlier epochs. The informative information learned during early epochs is, some sense, "low frequency information", which is the core factor to enable the model generalize well. A nice property of the self distilitilaaoonn algorithm is that it generates the final model in one generation (i.e. single-round training). which has almost additional computational cost compared to normal training. The proposed aif-distillation algorithm is given by Algorithm[] []]
Here, the function h(.)) in the algorithm is the label function. It can either be a hard label function such as hardmax, or a soft label function such as softmax with a certain temperature. The choice of hi()) and interpolation coefficient o, depends on the usage of the self-distillation algorithm. If
5

---

Under review as a conference paper at ICLR 2020
Student
Mentor
Epoch
Epoch n
Epoch
MentorNet
Co-teaching
Self-Distill
Comparison of error flow among MentorNet Jiang et al. (017)) Co-teaching Han et al. (2018) and our algorithm. Our algorithm does not require another teacher network and hence does not impose any additional computation burden.
Figure 3:
Algorithm Delf-Distillation
Randomly initialize the network. O repeat
Fetch data (x1, 31).. (In, 3n) from training set.
Set the label Ui.t = a.3i + (1 a, (NV))4((((,,))))
Detach Js from the computational graph
Update We+11 wi > Vwl(Wx (t.)).
1= L
until training converged =0
we want to clean up label noise, normally choose ht-) to be hardmax softmax with low temperature. The weight a₂ chosen be adaptively decreasing corresponding to increase of confidence the learned model at current epoch. The introduction of h(-) helps to boost AIR and the information gained from the previous epoch.
3.3 THEORETICAL FOUNDATION OF SILL-DIITILLATION
In this section, provide theoretical justification of the performance of the self-distillation algorithm with overparameterized neural networks. Here, we only consider binary classification task with label E {1).
Definition 1. (Noisy Clusterable Dataset DescriptionsLi al, (009
We consider dataset with data: (1ri yi, (z)) Rd (-1 (-1,+1) (I1, Wi) are the input data and its associated label seen by the model while ÿi is the unobserved ground truth label. The pair (x4, U1) with yi -js is called clean data, otherwise it is called corrupted data.
We assume that (Iihig contains points with unit Euclidean norm and has clusters. The data is randomly i..dd sampled from distribution P. Let be the number of points in the Ith cluster. In our analysis, we ssume that number of data each cluster balanced in the sense that n4 2 Clow for constant Clow 0 and all the data is bounded, i.e. [n).
| 13 < R, Vi

---

Under review as a conference paper at ICLR 2020
For each of the K clusters, assume that all the input data lie within the Euclidean ball B(c,, c), where a is the center with unit Euclidean norm and e is the radius.
Assume that the data in the same cluster has the same ground truth label ÿ. For the Ith cluster, denote pt the proportion of the data with wrong labels. Let p max(p: i [K]} and assume that p < 2-
A dataset satisfying the above assumptions is called an (c, dataset.
The above definition of dataset follows that of the previous work Li et al. (2019 Li & Liang (2018). is reasonable to assume that < 2 in order to ensure the correct labels dominate each cluster. In this work, we consider two-layers neural networks. For input data E R., the output of the neural network f is:
f(W, x) = v² ((Ww).
where W E Rexd is the weight matrix and & is the activation function applied to Wx entry We suppose l is even and fix the output layer by assigning half of the entries Vk and the others vi
Given data matrix = [2], F2, ... Inl", we simply denote the output vector on the data matrix as S(W,)) = lo" ((Wx (f(W,z)), (WW a(W) E R.
Following the previous work on training overparameterized neural network Du et al. (2018b), we consider the MSE loss
X(W,X)= = 2 (l,, - -yl1
Definition 2. For data matrix eaxd we denote MDD the small eigenvalue of the neural network covariance matrix
Σ(D) = (DD") @ Eg- (Dg) (1.)"].
The above definition reveals the matching score of the model and data. We denote C = ce,,, the matrix composed by the center of the cluster. We denote A min(A(C), X(X))
for simplification.
We are now ready to present our main theorem that establishes the convergence of the proposed self-distillation algorithm to the ground truth labels under certain conditions.
Theorem 1. Assume that (o(0)), (o)(-) and (Ø"(.) are bounded with upper bound I 1. We fix learning rate 2 for the gradient descent. Assume that the sequence au monotonically decreases to Furthermore, we have two slow-decay conditions on an
max2 0141) S KTT,
Sto MC) 5
(1 2p).
max max2/n((a. ax+1) S 82T²
ClawA (1 2p).
5
OT, 2 max (1-
Ctow A(C) (1 - 2p); 1281"K K
3 3-201
2 2-2p
where Ti = sor K [ log( I 332n A(C
and T2 = inf(t: as < 24/n
We choose the following
1-2p
label function h(..)
1
2p x,
| <
4
M(x)=
1- 1-2p 2p
sgn
(z),
[
4
For the self-distillation algorithm, if the following two conditions for the radius e and the width are satisfied
c-0
VdnIT (V log
(1 (1-2p)² 2p)²
1: = Ω2 max I
K3 log 1 nTnK 3T
1 n log
log n
)
ci A3
8' Clow (1 2p) 8' A
8
then for random initialization Wo 1)kxx with probability 1 6, we have: lim I,, ~l1 =0 0,
where We is the parameter generated by the full- fall-batch self-distillattin algorithm iteration I.
7

---

Under review as a conference paper at ICLR 2020
Theorem 2. Combining Theorem 1
and Neyshabur et al. (2018), with failure probability 5 E (0. 1), using the self- self-distillation algorithm and neural network described in Theorem and under the same condition, we have
KT2 Clow
log 1
B(aa)==ll lim f(W 2112 - O(11 + E-000
Compared to previous result on noisy label Li et al. 2019). our results made the following improve- ments. Firstly, algorithm fits the ground truth labels without the help of early stopping long as a mild condition on O2 is satisfied. Secondly, while previous work only ensures the algorithm to yield correct class labels training set, our results state the & convergence of the outputs the ground truth labels. As result, the solution that our algorithm finds tends to have larger margin which leads better generalization Mohri al. (2018). This is shown in Theorem 2. and will be supported by our empirical studies in the next subsection.
3.4 NOISY LABEL REFINERY
In this section, we conduct experiments on the self-distillation algorithm. In the experiments, we applied our algorithm on corrupted Fashion MNIST and CI- FARIO. At noise level p. every data in the original training dataset chosen and assigned symmetric noisy label with probability p. We test our algo- rithm for 0.2, 0.4, 0.6 and 0.8. The test accuracy is calculated with respect to the ground truth labels.
0.9
SGD Test
Test Accurary
0.7
We adopted the shake-shake network of Gastaldi (217)) with 32 channels and cross entropy loss. We trained the net- work by momentum stochastic gradient descent (SGD) with batch size 128, mo- mentum 0.9 and weight decay of le-4. We schedule the learning rate following cosine learning rate Loshchilov & Hut- ter (217) with maximum learning rate 0.2 and minimum learning 0. der to ensure convergence, we trained the models for 600 epochs. Following [eee et al. (2015). mean subtraction, hor-
0.8
S0D Training
Fpoch
Figure 4: Training on CIFAR10 with 40% noise injection. The normal training suffers from werrfitting, while self- distillation does not. (Note that are conducting cosine learning rate scheduling. The learning rate is extremely small at the end of learning.)
izontal random flip, 32 32 random crops after padding with 4 pixels on each side from the padded the data augmentation process. For testing. We only do evaluation original 32 x 32 image. In self-distillation, we adaptively adjust a, by setting - 0t = À * accuracyy where accuracy is the accuracy caleulated the current batch. The direct ratio À is the only tuning parameter. For CIFARIO, We simply set À as 1 when the noisy level is low, such as = O, 0.2 and 0.4. When the noisy level p 0.6 and 0.8, we take À 1.5. For Fashion MNIST, We simply set as 0.6, 1, 1, 1.4, 1.6 respectively when = 0, 0.2, 0.4, 0.6, 0.8. We report the average final test accuracy of 3 runs for Fashion MNIST and CIFAR 10 in Table T
image is performed
the
8

---

Under review as a conference paper at ICLR 2020
CIFARIO
Fashion MNIST
Noise Rate
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
HoTecchiggHan et al. (2018)
90.12
86.19
80.87
94.28
91.24
86.83
Ma et al. 2018)
91.29
86.64
73.12
94.47
89.12
78,98
Damodaran al. (2019)
91.88
89.12
84.55
94.70
93.37
90.41
GCEZhang & Sabuncu (0188)
89.7
87.62
82.70
67.92
93.21
92.60
91.56
88.33
Ours
94.31
93.75
90.82
86.72
73.91
95.21
94.58
93.78
92.63
89.21
Table 1: Results of Self-distillation.
3.5 DISTILLATION 2 EARLY STOPPING
Distillation can benefit from more than just early stopping. Distillation has the abil- ity to enhance the AIR and thus can ex- tract more dark knowledge from the data via early stopping. For example, Hinton eet al. (2015) enhanced AIR by adjusting the temperature in the softmax layer. In self-distillation, the label function hi(-) is proposed to amplify the information gained in the carlier epochs so that the knowledge gained in the earlier epochs is preserved. Thus, self-distillation does not require early stopping which makes the algorithm more user-friendly. On the other hand, the self- distillation algorithm dynamically enhances AIR which enables it to achieve l conver- gence and thus better generalization. Again, we use the ratio of the norm of the label vector which lies in the top-5 eigenspace as a representative of informative information. We call the subtraction of informative information corresponding the label vector provided by self-distillation algorithm and original label vector as information gain. From 5 we can see that the information gain is mostly larger than zero during the training of self-distillation algorithm. This phenomenon indicates that the supervision signal self-distillation algorithm gains more information than directly using the noisy label. To sum up, well well-designed distillation algorithm can enjoy regularization effect beyond early stopping and is able to gain more knowledge from the data.
0.5
information
0.4
0.3
0.2
0.1
0.0
0
200
400
600
800
1000
1200
1400
Iteration
Figure 5: Self-distillation always gains information.
4 CONCLUSION AND DISCUSSION
This paper provided an understanding of distillation using overparameterized neural networks. We observed that such neural networks posses the property of Anisotropic Information Retrieval (AIR), which means the neural network tends to fit the infomrative information (i.e. the eigenspaces associated with the largest few eigenvalues of NTK) first and the non-informative information later. Through AIR, we further observed that distillation of the Dark Knowledge is mainly due to early stopping. Based on this new understanding, we proposed new self-distillation algorithm for noisy label refinery. Both theoretical and empirical justifications of the performance of the new algorithm were provided.
Our analysis is based on the assumption that the teacher neural network is overparameterized. When the teacher network is not overparameterized, the network will be biased towards the label even without early stopping. It is still an interesting and unclear problem that whether the bias can provide us with more information. For label refinery, our analysis is mostly based on the symmetric noise setting. We are interested in extending our analysis to the asymmetric setting.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. arXiv preprint arXiv: 1..81.00166... 2018.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fiin-ggrinn analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.0888, 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks, In Proceedings of the 34th International Conference on Machine Learning-Volinn Volume 70, pp. 233-242. JMLR. org, 2017.
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint arXiv: 180.064,,,,, 2018.
Martin Burger, Guy Gilboa, Stanley Osher, Jinjun Xu, et al. Nonlinear inverse scale space methods. Communications Mathematical Sciences, 4(1):179-2122 2006.
Safa Cicek, Alhussein Fawzi, and Stefano Soatto. Saas: Speed as a supervisor for semi-supervised learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 149-163, 2018.
Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. In Advances Neural Information Processing Systems, pp. 0378-4287, 2017.
Bharath Bhushan Damodaran, Kilian Fatras, Sylvain Lobry, Rémi Flamary, Devis Tuia, and Nicolas Courty. Pushing the right boundaries matters! wasserstein adversarial training for label noise. arXiv preprint arXiv: arXiv:1944.00 1904.03399 2019.
Simon Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint 1811:0081100004 2018a.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint 18Xiv:1810.020552 2018b.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. arXiv preprint arXiv: 18055:80000047700 2018.
Xavier Gastaldi. Shake regularization. CoRR, abs/1 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems. pp. 8527-8537, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025:: 2015.
Wei Hu, Zhiyuan Li, and Dingli Yu. Understanding generalization of deep neural networks trained with noisy labels. arXiv preprint 19Xiv:1905./1366 2019.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: /007:0/290 2017.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization neural networks. In Advances neural information processing systems, pp- 8571-8580, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv: 1712 05055, 2017.
10

---

Under review as a conference paper at ICLR 2020
Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. arXiv preprint arXiv: 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):466, 2015.
Chen-Yu Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In Artificial Intelligence and Statistics, pp- 562-570, 2015.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping provably robust to label noise for overparameterized neural networks. CoRR, abs/1903 1680, 2019.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent structured data. In NeurlPS, 2018.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey, Dimensionality-driven learning with noisy labels. arXiv preprint arXiv:1806.0260 02612, 2018.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks. arXiv preprint arXiv: 12076 2018
Samet Oymak and Mahdi Soltanolkotabi. Overparameterizee nonlinear learning: Gradient descent takes the shortest path? arXiv preprint arXi:: 2018.
Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of deep neural networks. arXiv preprint arXiv:1806.00 08734, 180.008844,
2018.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning robust to massive label noise. arXiv preprint arXiv:1705. 10694, 2017.
Otmar Scherzer and Chuck Groetsch. Inverse scale space theory for inverse problems. In International Conference on Scale-Space Theories in Computer Vision, pp. 317-325. Springer, 2001.
Jianing Shi and Stanley Osher. A nonlinear inverse scale space method for a convex multiplicative noise model. SIAM Journal on imaging sciences, 1()):244321, 2008
Steve Smale and Ding-Xuar Xuan Zhou. Learning theory estimates via integral operators and their approximations. Constructive approximation, 26(2):153 172,
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-556666 2018.
Emesto De Vito, Lorenzo Rosasco, Andrea Caponnetto, Umberto De Giovannini, and Francesca Odone. Learning from examples an inverse problem. Journal of Machine Learning Research, 6 (May):883-904, 2005.
Jinchao Xu. Iterative methods by space decomposition and subspace correction. SIAM review, 34(4): 581-613, 1992.
Jinjun Xu and Stanley Osher. Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising. IEEF Transactions on Image Processing, 16(2):534-55) 544, 2007.

---

Under review as a conference paper at ICLR 2020
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle:
Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv: 9900..65.3,....... 2019.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan Yuille. Knowledge distillation in generations: More tolerant teachers educate better students. arXiv preprint
2018a.
Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student
optimization in one generation. arXiv preprint arXiv: 2018b.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-33 2007.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement benefit co-teaching? arXiv preprint arXiv: /ttt::990..000 (90100/,0, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 161..098, 2016a.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016b.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1rXiv:1611.03530 2016.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, pp. 8778-8788 2018.
12

---

Under review as a conference paper at ICLR 2020
A PROOF DETAILS
A.1 NEURAL NETWORK PROPERTIES
As preliminaries, we first discuss some properties of the neural network. We begin with the jacobian
of the layer neural network v oTW,,, the Jacobian matrix with respect to W takes the form
(W) (diag(()d ( XT
Thus
J(WJJ(W)) (XW (XWT)) o (XXT)
First borrow Lemma 6.6, 6.7, 6.8 from Oomak Soltanolkotabi (2018) and Theorem 6.7, 6.8 from Li et al. (2019).
Lemma 1. Let X = [1, Un] be a data matrix made up of data with unit Euclidean norm. Assuming that X(X)= O, the following properties hold.
X) J(W X)I
T VR
W
WILF.
J(W,X)) S LTv.
As long k 201 log 3 random Gaussian initialization Wo N(O,, 1) with x(X)
probability at least 1 ò, we have
A(X)
Omin (J(W,,
Lemma 2. Let X [21,22....,, be the data matrix of a e-clusterable dataset. Set X [21, T2,,..., ,,,," in which corresponds to the center of cluster including Es. What's 's more, we define the matrix of cluster center C [2,,C2,... C2...., cx)"" IT. Assuming that X(C) > 0, the following properties hold.
X)
D Peann VT
IW - W- S P W W
(J(W,)) < T Cagli VV,
As long k 20r log + at random Gaussian initialization Wo No 1)kxd, with
probability at least 1 X(C) 8, we have
(lown))(C)
Tmin (J X), 2
(2)
2K
range(J(W,X)) c S+ for any parameter matrix W.
Then, we gives out the perturbation analysis of the Jacobian matrix.
Lemma 3. Let X be e-clusterable data matrix with its center matrix X. For parameter matrices W. W, we have
Ivn Vi
JJW,X) - I(W,X)-J(W,
(w W|r || vke)
(3)
Proof. We bound |J(W,X) - J(W, by
J[W, X) J(W, X < JJW, - J(W, ()II (UW,X) - J(W, &)1l
The first term is bounded by Lemma T
As the second term, bound it by
13

---

Under review as a conference paper at ICLR 2020
[J(W,X) J(W,X)I
vk 1 ( - I11
vb (Il'((X')) X - ((WX) * x11
+ + * x - I1I
S VI ( - ((WXT)
+ + * (X - IX)
(TIIIIII - XII +Il++VAIJI -XXI
XII
Vnle Vk
(I]WI V
Combining the inequality above, we get
T (I - W|F + [[W]e vke)
J(W,,) - J(W, x)
l
Lemma 4. Let X be a e-clusterable data matrix with its center matrix X. We assume [|al 1|W have a upper bound cv k. Then for parameter matrices W1. W2, W1, W2. we have
[][W,, W2, X) J(W, W2, x:)1 Tvn W Will= + IW2 WallF
(c + 1)c) (4)
2 k
Proof. By the definition of average Jacobian, we have
[J W2, X) J(W, We, XIIl
[J(W, + a(W, - W2), x) - J(W, + +o - W2). X)]lda
Vv (lla(W - W1) + (1 a) W2) + |W + a(W; -W2)le + vkejda
<
Vi
1 r L
W1 Will + (1 o)]lW
a)llWa l)) vke)da
Vk
Vnl || WillF
+ +| W211 + +(c+11) 1)c)
<T
2
A.2 PROVE OF THE THEOREM
First, introduce the proof idea of our theorem. Our proof of the theorem divides the learning process into two stages. During the first stage, we aim to prove that the neural network will give out the right classification, i.e. the 0-1-loss converges O. The proof in this part is modified from Li et al. (219)). Furthermore, we proved that training 0-1-loss will keep 0 until the second stage starts and the margin at the first stage will larger than I-2e During the second stage, we prove that the 2
neural networks start further enlarge the margin and finally the loss starts converge to zero.
Following Oymak & Soltanolkotabi (2018): y Li al. 2019): Du et al. (2018b). we directly analysis the dynamics of each individual prediction JWW 2. for 1,2, n1. Oymak & Soltanolkotabi (2018): Li et al. (2019) has shown that this dynamic can be illustrated by the average Jacobian.
Definition 3. We define the average Jacobian for two parameters W, and We and data matrix X as
J(W, W2, X)= = f(
S/ J(W, a(W, - W2), X)da.
(5)
14

---

Tymn Under review as a conference paper at ICLR 2020
Lemma 5. (Liet al. 2019) Lemma 6.2) Given gradient descent iterate 0 0 1)V define
(( - J(0,
The residual == f(0) y, f(0) obey the following equation
f = ((())
our proof, we project the residual to the following subspace
c-clusterable dataset and { be the associated cluster centers, that is, E. iff 2; is from Ith cluster. We define the support subspace S+ as subspace of dimension K, dictated by the cluster membership as follows. Let A₁ C f1.2.... , be the set of coordinates i such that Then S+ is characterized by
Definition 4. Let {arli=
E
Vt, Vi1,2 E A., 1 S.
Definition 5. We define the minimum eigenvalue of matrix on a subspace
miin (B,S)
min -,,-==,
UTUT' | UFBBl2 B|2,,
where Ps is the projection the space S.
Recall the generation process of the dataset Definition 6. (Clusterable Dataset Descriptions)
We assume that {x, [ii] contains points with unit Euclidean norm and has clusters. Let my be the number of points in the lth cluster. Assume that number of data in each cluster is balanced in the sense that ny Z Clow K for constant Clow > 0.
For each of the K clusters, we assume that all the input data lie within the Euclidean ball B(c,, e), where q is the center with unit Euclidean norm and is the radius.
A dataset satisfying the above assumptions is called an e-clusterable dataset.
A.2.1 THE FIRST STAGE: FITTING THE LABEI.
First, we reduct the dataset to its cluster center, 0 for e-clusterable dataset. Lemma 6. We fix the label function
[ < - 2p
1 2p x
4
h(x)
1- 2p
sgn (x)
x
4
Let (x:) be a e-clusterable dataset and {5: be the associated cluster centers, that J. a
iff x; is from lth cluster. We denote the data matrix X and X. We denote a = =V Stove CA(C) In
SK
and I. norm during the optimization. We suppose along the optimization path have S [Jull S B 1-2p
V - We set the learning rale n min( 23² L.5).. where θ is maximum of the residual
for all E S4. We set Ti [log,- na2
fo 1, where Ps. (f X) yo) is the projected
residual on the space S4. Then Vt 2 Tı, we have
The neural network can learn the true label
sgn (J) (W,, X) .j.
The weight vector will be close to its initialization for all iterations
W Welle < n/
| 1| < _ (4]1Fol2 + Syn).
1=0
15

---

Under review as a conference paper at ICLR 2020
Proof. We denote C₁
max max22/n(aa a441). We denote J. J(W, X). Follow the step of (20
gradient descent, we have
n/lJTrella [ W = = rella S v/lJ? Fella S 1/
(6)
We denote G, = J(W++1, W,, X), x)- Then the dynamic of gradient descent can be written by
S(W-- X) r(W,, X) r/Grre.
We consider the dynamic of the residual Tt
Tt+1 = (1 - 17. + + -V++1
We project the residual _
Fet1 = (I-rGG)) + Dt B(+1-
Thus, the norm of the residual can be bounded by
|ett l2 =|((I -
(1 ( (L
Mf((W+1.
(8)
+ (a. 0 1ll -
X))112
(9)
< <| - G
a.) || +
(10)
Utilizing the following lemma
Lemma 7. (Claim2. et al 20191) Let Ps+, be the projection matrix S4. then the following inequality holds
BPss = 2G. - J. = a² ss
on the condition that 1) Lpp|rell
Therefore, as long as 1) S Lpliralla
holds, we have
1/22
||141l12 S (1 ][Fell + 1 2/T((a= a1t1). (11)
When 04 2 1 a²
have
T/
1/22
F 2 (1 ][Fell2 + C1.
By simple calculation, we have
n0? |elll S (1
4
> +
C₁.
na22
After Ti = log,_ na² Folll
1-2p
iterations, we have
2p
F < 4
long C1 < yo² (1 (p)).. On the condition that ar 1 - --30
have
2-2p
3 (1 2p)
Mn (1)
i == 1, .n.
4
As result, all the data have been classified correctly at iteration Ti and the following inequality holds
1 2p
f(Wr, (1) > 2
i= 1.2..... n..
For the next step, induction prove that
2p
h(fa) = 8. 11eel2 5
2 Tı
(12)
4
16

---

Under review as a conference paper at ICLR 2020
Fort t = Ti, it has already been proved. We assume the claim is correct for arbitrary establish the induction for t + 1. By applying projection to S+ on equation 7
we have
1(W,+1, X) :1 =(1 -)ii)
By applying lemma
we have
1 2p
[I(( Bell2 3 ||Fa112 _
4
Given that h(ft) = y. we have
21 21- 2p, i= 1.2,...... n.
Thus, for fi+1 we have
f(W 1+1, 2,),,,)000 S 3 (1- i= 1........, n.
By the definition of h(-), we deduce that h(fe+1) ÿ. Back to equation g
we have
| le <lI(1 - G + (1 az) (f)aaaa)l((((.,.))) -((()))))(((+
(13)
+ (a+
0++1 1| - h(
(14)
n
<(1
)[Fr|l2 22/n((a ac+1)
(15)
2
ga²
5(1
)[Feell + Ci
(16)
2
n02 2
2p
na²
<(1
+
(1 2p)
(17)
2
4
32
2p
(18)
4
Finally, we estimate the total variation of the parameter. Combining equation 1 and 12 we can conclude that inequality
|r+el S (1 na²
]1aall2 + 22/T(a; - a2+1)
holds for all 2 0. After taking sum on both sides for L = 1,1,2..... we have
7a2
|idll s (1-
|elll + 2/n.
t=
By simple calculation, we get
411 1 + 8
|elle <
na²
t=0
Combining equation 6
we have
B
We+1
WillF 5
Fell
(4]1oolla 8v7).
t=0
A.2.2 SECOND STAGE: ENLARGE THE MARGIN
Then we further to adapt the above theorem the e-clusterable dataset by pertubation analysis. Since we have a simple inequality a, 2 ag in the following discussion, we simply replace the a, in the previous conclusion by 0g the conclusion also holds.
17

---

Under review as a conference paper at ICLR 2020
Lemma 8. Let (as)i=1 be a e-clusterable dataset and { be the associated cluster centers, that is, F. a iff s; is from lth cluster. We denote the data matrix X and X. For the same initialization Wo = Wo. we run the self-distillation algorithm on X and t.X respectively. We denote the parameter
matrix We and Wi for O. We denote a== / /tounn nA 8K ,, = Inn and L = T// x/
We set the learning
rate Lj min( 25 T. where θ is maximum of the residual norm during the optimization. We
denote cvlc: the upper bound of the Frobenius norm of the parameter matrxi and we set M = c 1.
We set T- = in 0₁ 24/7a) Then if the following conditions hold
1 2p
<
2 02222202((23 + LT + 1)
we have
II((W, -(W,, 8)112 < 4n veel
[W, Welle S 22 1)el
for all t T2
Proof. We introduce the following notations.
Tt = X(W,,X ye. Tr, = f(W,,)) Dt
J. = J., x), = J. 8)
J1+1,1 J(WT+1, W., X), = J(WT+1, We, X)
d= W - Wrlrr, (,, X) - f(W,, Xl2
We can conclude the following inequalities from lemma 3 lemma
J - Joll < Ld, + MTv
|J+1. - Je+1,ll S L de + d++1
+ MT Vne.
2
Thus the parameters are updated by gradient descent, we have
d++1 ||W+11 W4ille S I|, Welle + IlnJ re nJ. Tre llz
S di + n/lle - Jelll Tr, ll2 + n/IJe - Tr, l2
S de + mLLdd. + MTO.Te + 3(1 + Ln)Pt)
Also, we have
P++1 = =II((++1,,)) X) - -(W,+1, X1)11
<I)(W,, f(W,, X) nJ Jt (r. TT:)lla
+n][JJ+11, - -J JJ rell2 7/ Je J. Drall
<11/(W, S(W,, - (rr -rr))lk
3d, + dr+1 + 2Mee
+ 1111rre1/2
2
<1(11 nJ+t1.
J. (J(WW, X) S(W,, 8))1 Il (ve Gc)ll2
3d, + det1
+ 11111rell2T..
2nc)
2
na²
S(1
Pr + n1,32 - (1=)))l((((W,..)))
X)) ((((W,
2
3d, + de+1
+ n)81111l12(1
+ 2MT
2
18

---

Under review as a conference paper at ICLR 2020
If Ps Σ 1-30 holds for I T2, then for T1 3 1 T2, have f(W,, X) y. Under such
4
circumstance, we have | X)) h(f(W, X) = 0. For 1 < T1, we have
na²
(1 JP + - (1-aa)ll(((WW. - ((](W,))))
2
70² 2
S(1
Pt + n13 - Lap)
2
na²
a
<(1
Pr n132
Lipt
2
4 Ln.
Sp.
To sum up, we can guarantee that pt I 1-2e holds for S T2, we have
P++1 S Pr + n 1 (rella(L 3d, d2+1 + 2MT Vnc).
For [1rel22. we have
||rel2 S Tr, ll + l|re Tre llz
< Tr, 1l + +1+ [f), - f(W,,
< Tr, 12 (1 Ln)Pe
5 + (1 + Ln)pt-
Thus we have the following inequality for Pt
3d, + dd+1
Pi+1 P: n + (1+ (L
+ 2MI Vnc).
We claim that if the following conditions for and k hold
2p
$ 4Mp + L., /nT + ()
k: 2 (2 Ln) + 1)
one can show that
Pt < 4m Ln)
d S MT (2 2 1)ct
for S T2 by induction. It is obviously when t = 0. We further suppose the inequalities hold for an arbitrary f satisfying < T2, we have
d+1 < d, + mLOd, + MTO 3(1 + L.)p.)
< d, + ((2MOO 42 L.) vTe L.))
< d, + (2MTON/e 42 M.M
< 22, L.2 1))(++1)
because of the condition on ensures that LOd, S MTO When it comes to pr+1, we have
P++1 S.p P. (1+ 3d. + de+1 + 2MT nC)
2
P. 100 L,) TT + 1)c(t 1) + 2MT VTi))
P: (1(0+((+ 2.T
$ P: 77(2+ L.)0((2TTT 2MT Vnc)
$ P:
< (2 + Ln) +
because of the conditions on and ensure that Ldr, & MT /ne and pt S O.
19

---

Under review as a conference paper at ICLR 2020
Now are ready to finalize the proof of our main theorem.
Proof. We denote C2 - max 2/n(a, 0s41)). We take θ (C3T log _h
1)yn. C3 is the
=2
constant in Hoeffding's 's inequality. By lemma have @ 2 max
|Trell2 with probability 1 4
Combining lemma 6 and 8 we have f(W,, X = ÿ for Ti S L < 72 and
rr, lla I X) yrall2
- 1W, X) - f(W,, X)12 + II x) - 0r,ll2 112 + 11ūr, yr, l2
< [[ X) S(WT,, X) [rT,, |22 + ar, 11ū yl12
2p
1 2p
- 4
+
4 24
Similar to the proof in 6 we consider the gradient descent on original dataset X after T2. We proof the following claim by induction
5 (1- 8
1
M((W.), U. |lrlllll2
2p)
T2.
24'
For T2, it has already been proved. We assume the claim is correct for arbitrary we establish the induction for + By equation
we have
S(Ws+I, X) yo = -r/Gs)rs
By applying lemma we have
II Toll2 S 1|ral12 I (1 (1- 12pppp + 1
24
Given that ((,(W,,))) y. we have
ys (i))(()) 2 22T,, i= 1, 1,2, ,
Thus, for f(W we have
5
1
1 - 2p
(We+1,x,)ū(0) 2 1 - 20rr
8 22p)
24 4
As result, we have f(W X) ÿ. Furthermore, we can bound ||r by
||r+11l22 S|[(1 - G 1 2| + (1 - a.)))M((( - (((()
(a,
0z || h(f
<(1
|r:s ||2 + C2
<(1 na 2
(1 2p) + //1 + qa²
(1 2p)
8
24
32
5
1
(1
2p)
To sum up, have the following inequality holds for all s T2
na22
|rst S (1 2
||rrl11 + 2/T((a, as+1).
After taking sum on both sides for s 2 T2, we have
2
21
laa
+
5 2p
Irall <
<
na²
n122
Furthermore, we have
80 WollF n5 Σ
3 3.5
Σ W
||r l|2 <
- 2p).
a²
s=
20

---

Under review as a conference paper at ICLR 2020
Finally, we check all the conditions for and 17.
2n(Cas /og +1) 5
For 77, we require 7 - 2
and ry
S
min
| (... For l
LB
220,877
Gown
O nk log h have (("Rneaa
a
min L 120,22T:
| l1
1| 2,3²
On such condition, can choose matrix and the initial parameter matrix can be bounded by
श. The distance between the intermediate parameter
8 a²
8
== max (IIW,- Wolle, [|,, Wolle) < (411Follz + +V/)) +1, + 120,020
a² ( 2p)
(19)
8KT
(4Ca1)
log 8 + 13) + dT2
(20)
Clww
By lemma and lemma 2 as long as A 2 201 n log om
with probability 1 - 2. we have
miin (JW,, X)), mmin (JW,, S), 2 2nx
To ensure a lower bounding the eigenvalue of the gram matrix, we need to verify that
RL = R S a
That is to say
8KT2R²
k k
(21)
Cow
Another condition related to R is the condition on M. We require
(M 1)Vk > [Wol| R.
By Bernstein's inequality, have
Woll S V Ca log 8 5 SVk
with probability 6/4. On the condition that l 2 (acutuually one can show that 8KT we can choose M Va C44 log 2
Cow 2 R2),
We take these constants to the lemma 8 Firstly, for e we have
1 2p
c 4Mp3(2 L TT + +
1 1p2p
2( d dCC, + 22 + (1 + (C2I' log K + 1))
0 (1 (1-2p)²
Vndl's log 1
For we have
k 2 (2 + 1)
4 2T + 1)
200
1- 2p
n²TT log
(1 2p)²
21

---

Under review as a conference paper at ICLR 2020
We point out that Secondly we bound dr, by
0 n T2 log 1
nk log
(1 22)) 2 0
Clow
drg S
T2 1)672
(22)
(1 -22 T22 + 1)T2
(23)
42 + L.) VnTz(1 + +))
1
$ (1 - (3) + L.)T2 + 23 + Lm)
(24)
(VViT2))
(25)
We combine equation 20
21 and 25
we deduce the last condition for ls
K
0" Clow A }
nKT22
k: = Ω
max
Gow
A3
log
To sum up, we require l: to satisfy
K
1 nKTz
T²
1 n
n2 g log
})
k: == Ω max
log
log
A3
Clow A (1 2p)
5 A
Tow
and require to satisfy
c=o
(1 (111-111)00
2p) (1-2p)²
VndTT log 1
We substitute C1 and C2 by the value of a in lemma 6 and lemma respectively and then get the condition for at
max 2 0.+1) < K<7
Stw M()) 5121"K
(1 2p),
max 2/n(a, a.+1) S 22T,
Gran (1 2p).
5121"K
on 2 max(1
Slow X(C) (1 2p),
-
128T
2-2p
Also, we substitute Tj by the value of a in lemma and combine the fact that have
na² a² 1 we
4 8/52
S
1 2p
Ti = log
no
8 |Fo
log
8 Ng la
2p
log
1- n0
5
log Foll2
n12
2p
8 K
r /v32n log
(C) log(
17.
1 1 2p
22

---

Under review as a conference paper at ICLR 2020
B EXPERIMENT DETIALS
Experiment In Section2.1
The network structure of the small student network is demonstrated in
Table
2
Type
Kernel
Dilation
Stride
Outputs
Remark
conv.
3x3
1x1
32
bn
maxpool.
2x2
conv.
3x3
1x1
64
bn
maxpool.
2x2
conv.
3x3
1x1
128
bn
maxpool.
2x2
Flatten
Ic.
512
bn
fc.
10
dropout
Table 2: Architecture of the student network. After each convolution layer, there is Rectified Linear Unit(ReLU) layer.
Experiment In Section2.3 and Section3.5 For these two experiments, we modify CIFARTO to binary classification task. We choose class 2, 7 to be the positive class and the others to be negative. We train resnet56 with MSE loss. We set batch size 128, momentum 0.9, weight decay 5e 4 and learning rate 0.1.
In the experiment section2.3, we fetch a batch of data (batch size=128) from the testset and randomly corrupted the label by noise level 0.1, 0.2, 0.3, 0.4, 0.5. We plot the ratio of the norm of the label vector which lies in the subspaces corresponding to top-5 eigenvalues of NTK.
In the experiment in section2.3, calculate the ratio of the norm of the label vector which lies in the subspaces corresponding to top-5 eigenvalues NTK firstly. We calculate the ratio of the norm of the label vector provided by the self-distillation algorithm lies in the top-5 eigenspace. We calculate the difference of the latter ratio and the former ratio and called it information gain. We plot the information gain of the first 1500 iterations.
23


Reviews:
[{"review_id": "rJgoFcQoKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a theoretical framework to understand the regularization effect of distillation. Based on the observation that a overparameterized NN has the ability to memorize all the data, thus early stopping is essential for an overparameterized teacher network to extract dark knowledge from the hard labels, the author starts to analysis distillation from an early stopping view point.\n\nThen the author consider to use distillation to learning with corrupted labels inspired by the previous works using early stopping to clean label noise. The author also tries to formulate the “informative information” via the NTK theory. Theoretically, the author provide a proof of convergence to the ground truth labels in terms of l2 distance. While the previous result was on convergence in 0-1 loss, which means the distillation can enlarge the margin the classifier. From this proof, aurthors also make an interesting discussion to show how distillation introduces further information rather than just early stopping. Authors also demonstrate their result on Fashion MNIST and CIFAR-10 to convince the reader the benefit of the algorithm.\n\nThe paper is interesting and brings new theoretically thoughts to understand the distillation method. The relationship between early stopping and distillation can inspire the machine learning researchers to explore more about distillation both empirically and theoretically.\n\nMinor Questions:\n1. The analysis is based on the assumption that the teacher is overparameterized. What will happen if the teacher network is not overparameterized?\n2. Does the assumption of dataset is too strong in the theorem?\n3. Some notation is not clear, e.x. in  Algorithm1 what is $\\mathcal{N}(x_i, w_t)$ and what is base network and mother network? These should be instead by student network and teacher network.\n4. Author claims that early stopping is hard to tune while introducing extra hyper-parameters in the self-distillation algorithm.  Would the extra hyper-parameter makes the algorithm might be even harder to tune?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1xggkhG9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a self-distillation algorithm for training an over-parameterized neural network with noisily labelled data. It is shown that for a binary classification task on clustered data (same as [Li et al. 2019]), even if the labels are corrupted, the self-distillation algorithm applied on a sufficiently wide two-layer network can recover the correct labels (in l_2 loss). Experiments on CIFAR-10 and Fashion MNIST are provided, which show that the self-distillation algorithm is effective for label noise with relatively little tuning.\n\nAlthough the theoretical part of the paper has a large overlap with [Li et al. 2019], I find the self-distillation algorithm very interesting and it's nice that it can achieve zero l_2 loss w.r.t the correct labels. However, I think the paper could still use some improvement.\n\n1. The theorem in Section 3.3 is only for binary classification. Can it be generalized to multi-class classification?\n\n2. The theoretical guarantee is only for training data. Is it possible to prove a generalization bound? There's a remark on top of page 8 about margin. It would be nice to elaborate on this and maybe make it formal.\n\n3. Section 2 is not very satisfying. I don't quite see the point of this section. In particular, the concept of AIR is nothing new and its connection to NTK top eigenspace has already been written in previous work (e.g. [Arora et al. 2019]). I'd suggest to not have this section and to make Section 3 the main contribution of this paper.\n\n\n-------\nupdate:\nThanks to the authors for the response and for adding a generalization bound.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxfNIDYqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new perspective on understanding knowledge distillation as a transfer of information defined with respect to the neural tangent kernel. Additionally, a new framework for learning the classifier on a noisy labeled dataset is proposed based on the knowledge transfer framework. \n\nOverall, I think the paper lacks justification (and explanation) for its main statement on how knowledge distillation is related to the early stopping of the teacher network. Especially, it is confusing since Section 2 and 3 make different statements. Specifically, Section 2 shows that early stopping \"helps\" knowledge distillation while Section 3 shows that knowledge distillation can \"replace\" early stopping. The former observation implies that early stopping is complementary to knowledge distillation, while the latter implies otherwise.\n\nFurthermore, Section 2 mainly explains why the eigenspaces associated with the largest eigenvalue of the neural tangent kernel is \"informative information\". However, there is no elaboration on how the knowledge distillation process leads to the transfer of such information, i.e., there is no connection between the neural tangent kernel and the knowledge distillation process. Although Figure 1. suggests that early stopping indeed improves the knowledge distillation process, they are not enough to support the statement convincingly enough. \n\nWithout proper support on the main statement of this paper, the paper looses much of its claimed contributions. The label refinery algorithm for the noisy labeled dataset is interesting, but it is not evaluated thoroughly enough to demonstrate its superiority over existing algorithms. It also does not have much originality when compared to similar algorithms [1, 2]. Bagherinezhad  et al., [1] also tried to remove \"noisy supervisions\" that were generated by harsh augmentation on images. Han et al., [2] and Li et al., [3] also use distillation-like processes to learning noisy datasets.\n\n[1] Label Refinery: Improving ImageNet Classification through Label Progression, Bagherinezhad  et al., 2018\n[2] Co-teaching: Robust Training of Deep NeuralNetworks with Extremely Noisy Labels, Han et al., 2018 \n[3] Learning from Noisy Labels with Distillation, Li et al., 2017", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tries to bridge early stopping and distillation.

1) In Section 2, the authors empirically show more distillation effect when early stopping.
2) In Section 3, the authors propose a new provable algorithm for training noisy labels.

In the discussion phase, all reviewers discussed a lot. In particular, a reviewer highlights the importance of Section 3. On the other hand, other reviewers pointed out "what is the role of Section 2", as the abstract/intro tends to emphasize the content of Section 2.

I mostly agree all pros and cons pointed out by reviewers. I agree that the paper proposed an interesting idea for refining noisy labels with theoretical guarantees. However, the major reason for my reject decision is that the current write-up is a bit below the borderline to be accepted considering the high standard of ICLR, e.g., many typos (what is the172norm in page 4?) and misleading intro/abstract/organization. In overall, it was also hard for me to read the paper. I do believe that the paper should be much improved if the authors make more significant editorial efforts considering a more broad range of readers. 

I have additional suggestions for improving the paper, which I hope are useful.

* Put Section 3 earlier (i.e., put Section 2 later) and revise intro/abstract so that the reader can clearly understand what is the main contribution. 
* Section 2.1 is weak to claim more distillation effect when early stopping. More experimental or theoretical study are necessary, e.g., you can control temperature parameter T of knowledge distillation to provide the "early stopping" effect without actual "early stopping" (the choice of T is not mentioned in the draft as it is the important hyper-parameter).
* More experimental supports for your algorithm should be desirable, e.g., consider more datasets, state-of-the-art baselines, noisy types, and neural architectures (e.g., NLP models).
* Softening some sentences for avoiding some potential over-claims to some readers.

Decision: Reject