Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized NN

1 INTRODUCTION . Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing LeCun et al . ( 2015 ) . Among these tasks , image classification is considered as one of the fundamental tasks since classification networks are commonly used as base networks for other problems . In order to achieve higher accuracy using a network with similar complexity as the base network , distillation has been proposed , which aims to utilize the prediction of one ( teacher ) network to guide the training of another ( student ) network . In Hinton et al . ( 2015 ) , the authors suggested to generate a soft target by a heavy-duty teacher network to guide the training of a light-weighted student network . More interestingly , Furlanello et al . ( 2018 ) ; Bagherinezhad et al . ( 2018 ) proposed to train a student network parameterized identically as the teacher network . Surprisingly , the student network significantly outperforms the teacher network . Later , it was suggested by Zagoruyko & Komodakis ( 2016a ) ; Huang & Wang ( 2017 ) ; Czarnecki et al . ( 2017 ) to transfer knowledge of representations , such as attention maps and gradients of the classifier , to help with the training of the student network . In this work , we focus on the distillation utilizing the network outputs Hinton et al . ( 2015 ) ; Furlanello et al . ( 2018 ) ; Yang et al . ( 2018a ) ; Bagherinezhad et al . ( 2018 ) ; Yang et al . ( 2018b ) . To explain the effectiveness of distillation , Hinton et al . ( 2015 ) suggested that instead of the hard labels ( i.e one-hot vectors ) , the soft labels generated by the pre-trained teacher network provide extra information , which is called the `` Dark Knowledge '' . The `` Dark knowledge '' is the knowledge encoded by the relative probabilities of the incorrect outputs . In Hinton et al . ( 2015 ) ; Furlanello et al . ( 2018 ) ; Yang et al . ( 2018a ) , the authors pointed out that secondary information , i.e the semantic similarity between different classes , is part of the `` Dark Knowledge '' , and Bagherinezhad et al . ( 2018 ) observed that the `` Dark Knowledge '' can help to refine noisy labels . In this paper , we would like to answer the following question : can we theoretically explain how neural networks learn the Dark Knowledge ? Answering this question will help us to understand the regularization effect of distillation . In this work , we assume that the teacher network is overparameterized , which means that it can memorize all the labels via gradient descent training Du et al . ( 2018b ; a ) ; Oymak & Soltanolkotabi ( 2018 ) ; Allen-Zhu et al . ( 2018 ) . In this case , if we train the overparameterized teacher network until convergence , the network ’ s output coincides exactly with the ground truth hard labels . This is because the logits corresponding to the incorrect classes are all zero , and hence no `` Dark knowledge '' can be extracted . Thus , we claim that the core factor that enables an overparameterized network to learn `` Dark knowledge '' is early stopping . What ’ s more , Arpit et al . ( 2017 ) ; Rahaman et al . ( 2018 ) ; Xu et al . ( 2019 ) observed that `` Dark knowledge '' represents the discrepancy of convergence speed of different types of information during the training of the neural network . Neural network tends to fit informative information , such as simple pattern , faster than non-informative and unwanted information such as noise . Similar phenomenon was observed in the inverse scale space theory for image restoration Scherzer & Groetsch ( 2001 ) ; Burger et al . ( 2006 ) ; Xu & Osher ( 2007 ) ; Shi & Osher ( 2008 ) . In our paper , we call this effect Anisotropic Information Retrieval ( AIR ) . With the aforementioned interpretation of distillation , We further utilize AIR to refine noisy labels by introducing a new self-distillation algorithm . To extract anisotropic information , we sequentially extract knowledge from the output of the network in the previous epoch to supervise the training in the next epoch . By dynamically adjusting the strength of the supervision , we can theoretically prove that the proposed self-distillation algorithm can recover the correct labels , and empirically the algorithm achieves the state-of-the-art results on Fashion MNIST and CIFAR10 . The benefit brought by our theoretical study is twofold . Firstly , the existing approach using large networks Li et al . ( 2019 ) ; Zhang & Sabuncu ( 2018 ) often requires a validation set to early terminate the network training . However , our analysis shows that our algorithm can sustain long training without overfitting the noise which makes the proposed algorithm more user-friendly . Secondly , our analysis is based on an ` 2-loss of the clean labels which enables the algorithm to generate a trained network with a bigger margin and hence generalize better . 1.1 CONTRIBUTIONS . We summarize our contributions as follows • This paper aims to understand distillation theoretically ( i.e . understand the regularization effect of distillation ) . Distillation works due to the soft targets generated by the teacher network . Based on the observation that the overparameterized network can exactly fit the one-hot labels which contain no dark knowledge , we theoretically justify that early stopping is essential for an overparameterized teacher network to extract dark knowledge from the hard labels . This provides a new understanding of the regularization effect of distillation . • This is the first attempt to theoretically understand the role of distillation in noisy label refinery using overparameterized neural networks . Inspired by Li et al . ( 2019 ) , we utilize distillation to propose a self-distillation algorithm to train a neural network under label corruption . The algorithm is theoretically guaranteed to recover the unknown correct labels in terms of the ` 2-loss rather than the previous 0-1-loss . This enables the algorithm to generate a trained network whose output has a bigger margin and hence generalizes better . Furthermore , our algorithm does not need a validation set to early stop during training , which makes it more hyperparameter friendly . The theoretical understanding of the overparameterized networks encourages us to use large models which empirically produce better results . 2 DISTILLATION ≈ EARLY STOPPING ? . 2.1 NO EARLY STOPPING , NO DARK KNOWLEDGE . As mentioned in the introduction , an overparameterized teacher network is able to extract dark knowledge from the on-hot hard labels because of early stopping . In this section we present an experiment to verify this effect of early stopping , where we use a big model as a teacher to teach a smaller model as the student . In this experiment , we train a WRN-28 Zagoruyko & Komodakis ( 2016b ) on CIFAR100 Krizhevsky & Hinton ( 2009 ) as the teacher model , and a 5-layers CNN as the student . The experimental details are in the supplementary material . The teacher model was trained by 40 , 80 , 120 , 160 and 200 epochs respectively . The results of knowledge distillation by the student model is shown in Fig . 1 . As we can see , the teacher model does not suffer from overfitting during the training , while it does not always educate a good student model either . As suggested by Yang et al . ( 2018a ) , a more tolerant teacher educates better students . The teacher model trained with 80 epochs produces the best result which indicates that early stopping of the bigger model can extract more informative information for distillation . 2.2 ANISOTROPIC INFORMATION RETRIEVAL . In this paper , we introduce a new concept called Anisotropic Information Retrieval ( AIR ) , which means to exploit the discrepancy of the convergence speed of different types of information during the training of an overparameterized neural network . An important observation of AIR is that informative information tends to converge faster than non-informative and unwanted information such as noise . Selective bias of an iterative algorithm to approximate a function has long been discovered in different areas . For example , Xu ( 1992 ) observed that iterative linear equation solver fits the low frequency component first , and they proposed a multigrid algorithm to exploit this property . In image processing , Scherzer & Groetsch ( 2001 ) ; Burger et al . ( 2006 ) ; Xu & Osher ( 2007 ) ; Shi & Osher ( 2008 ) proposed inverse scale space methods that recover image features earlier in the iteration and noise comes back later . In kernel learning , early stopping of gradient descent is equivalent to the ridge regression Smale & Zhou ( 2007 ) ; Yao et al . ( 2007 ) ; Vito et al . ( 2005 ) , which means that bias from the eigenspace corresponding to the larger eigenvalues is reduced quicker by the gradient descent . Under the neural network setting , Rahaman et al . ( 2018 ) ; Xu et al . ( 2019 ) observed that neural networks find low frequency patterns more easily . Cicek et al . ( 2018 ) discovered that noisy labels can slow down the training . Arpit et al . ( 2017 ) ; Rolnick et al . ( 2017 ) studied the memorization effects of the deep networks , and revealed that , during training , neural networks first memorize the data with clean labels and later with the wrong labels . In the next subsection , we will characterize AIR of overparametrized neural networks using the Neural Tangent Kernel Jacot et al . ( 2018 ) . 2.3 AIR AND NEURAL TANGENT KERNEL . In Jacot et al . ( 2018 ) , the authors introduced the Neural Tangent Kernel to characterize the trajectory of gradient descent algorithm learning infinitely wide neural networks . Denote f ( θ , x ) ∈ R as the output of neural network with θ ∈ Rn being the trainable parameter and x ∈ Rp the input . Consider training the neural network using the ` 2-loss on the dataset { ( xi , yi ) } ni=1 ∈ Rp × R : ` ( θ ) = 1 2 n∑ i=1 ( f ( θ , xi ) − yi ) 2 . We use the gradient descent θt+1 = θt − η∇l ( θt ) to train the neural network . Let ut = ( f ( θt , xi ) ) i∈ [ n ] ∈ Rn be the network outputs on all the data { xi } at iteration t and y = ( yi ) i∈ [ n ] . It was shown by Du et al . ( 2018b ) ; Oymak & Soltanolkotabi ( 2018 ) ; Li et al . ( 2019 ) that the evolution of the error ut − y can be formulated in a quasi-linear form , ( ut − y ) = ( I − ηĤt ) · ( ut − y ) , where Ĥt = ( 〈∫ 1 0 ∂f ( θt + α ( θt+1 − θt ) , xi ) ∂θ dα , ∂f ( θt , xj ) ∂θ 〉 ) i , j . It is known that Ĥt − Ht = o ( 1 ) with respect to d , where Ht = ( 〈 ∂f ( θt , xi ) ∂θ , ∂f ( θt , xj ) ∂θ 〉 ) i , j is an n × n positive semi-definite Gram matrix and d is the width of the neural network Oymak & Soltanolkotabi ( 2018 ) ; Li et al . ( 2019 ) . This means that Ĥt = Ht when the neural network is infinitely wide . It was further shown by Oymak & Soltanolkotabi ( 2018 ) ; Li et al . ( 2019 ) ; Du et al . ( 2018a ) that when the neural network is infinitely wide , the Gram matrix is static , i.e . Ht = H∗ . The static Gram matrix H∗ is referred to as the Neural Tangent Kernel ( NTK ) Jacot et al . ( 2018 ) . Note that H∗ is a symmetric positive semi-definite matrix . Assume that λ1 > · · · > λn ≥ 0 are its n eigenvalues and e1 , e2 , · · · , en are the corresponding eigenvectors . The eigenvectors are orthogonal < ei , ej > = 0 and H∗ = ∑n i=1 λieie T i . Consider the evolution of the projection of the loss function in different eigenspaces 〈 ( ut − y ) , ei〉 = 〈 ( I − ηH∗ ) ( ut − y ) , ei〉 = 〈 ( ut − y ) , ( I − ηH∗ ) ei〉 = ( 1− ηλi ) 〈 ( ut − y ) , ei〉 . We can see that the component lies in the eigenspace with a larger eigenvalue converges faster . Therefore , AIR describes the phenomenon that the gradient descent algorithm searches for information components corresponding to different eigenspaces at different rates . Rahaman et al . ( 2018 ) ; Arpit et al . ( 2017 ) ; Arora et al . ( 2019 ) ; Zhang & Sabuncu ( 2018 ) has shown that that one possible reason of neural network ’ s good generalization property is that neural network fits useful information faster . Thus in our paper , we regard informative information as the eigenspaces associated with the largest few eigenvalues of NTK . We denote the projection of supervision signal to the eigenspace with a larger eigenvalue as useful information . In Figure 2 , We calculate the ratio of the172norm of the label vector provided by the self-distillation algorithm lies in the top-5 eigenspace as a representative of informative information . We can see that the informative information decreases when the noise level increases . This motivates us to further explore how `` Dark Knowledge '' helps with label refinery .

Summary: This paper proposes a self-distillation algorithm for training an over-parameterized neural network with noisily labelled data. It is shown that for a binary classification task on clustered data (same as [Li et al. 2019]), even if the labels are corrupted, the self-distillation algorithm applied on a sufficiently wide two-layer network can recover the correct labels (in l_2 loss). Experiments on CIFAR-10 and Fashion MNIST are provided, which show that the self-distillation algorithm is effective for label noise with relatively little tuning.