Prestopping: How Does Early Stopping Help Generalization Against Label Noise?

1 INTRODUCTION . By virtue of massive labeled data , deep neural networks ( DNNs ) have achieved a remarkable success in numerous machine learning tasks , such as image classification ( Krizhevsky et al. , 2012 ) and object detection ( Redmon et al. , 2016 ) . However , owing to their high capacity to memorize any label noise , the generalization performance of DNNs drastically falls down when noisy labels are contained in the training data ( Jiang et al. , 2018 ; Han et al. , 2018 ; Song et al. , 2019 ) . In particular , Zhang et al . ( 2017 ) have shown that a standard convolutional neural network ( CNN ) can easily fit the entire training data with any ratio of noisy labels and eventually leads to very poor generalization on the test data . Thus , it is challenging to train a DNN robustly even when noisy labels exist in the training data . A popular approach to dealing with noisy labels is “ sample selection ” that selects true-labeled samples from the noisy training data ( Jiang et al. , 2018 ; Ren et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ; Song et al. , 2019 ) . Here , ( 1−τ ) ×100 % of small-loss training samples are treated as true-labeled ones and then used to update a DNN robustly , where τ ∈ [ 0 , 1 ] is a noise rate . This loss-based separation is well known to be justified by the memorization effect ( Arpit et al. , 2017 ) that DNNs tend to learn easy patterns first and then gradually memorize all samples . In practice , Han et al . ( 2018 ) empirically proved that training on such small-loss samples yields a much better generalization performance under artificial noise scenarios . Despite its great success , Song et al . ( 2019 ) have recently argued that the performance of the lossbased separation becomes considerably worse depending on the type of label noise . For instance , in symmetric noise ( Figure 1 ( a ) ) , the loss-based approach well separates true-labeled samples from false-labeled ones because many true-labeled ones exhibit smaller loss than false-labeled ones . On the other hand , in pair and real-world noises ( Figures 1 ( b ) and 1 ( c ) ) , many false-labeled samples are misclassified as true-labeled ones because the two distributions overlap closely ; this overlap confirms that the loss-based separation still accumulates severe label noise from many misclassified cases , especially in real-world noise or pair noise which is regarded as more realistic than symmetric noise ( Ren et al. , 2018 ; Yu et al. , 2019 ) . This limitation definitely calls for a new approach that supports any type of label noise for practical use . In this regard , as shown in Figure 2 ( a ) , we thoroughly investigated the memorization effect of a DNN on the two types of noises and found two interesting properties as follows : • A noise type affects the memorization rate for false-labeled samples : The memorization rate for false-labeled samples is faster with pair noise than with symmetric noise . That is , the red portion in Figure 2 ( a ) starts to appear earlier in pair noise than in symmetric noise . This observation supports the significant overlap of true-labeled and false-labeled samples in Figure 1 ( b ) . Thus , the loss-based separation performs well only if the false-labeled samples are scarcely learned at an early stage of training , as in symmetric noise . • There is a period where the network accumulates the label noise severely : Regardless of the noise type , the memorization of false-labeled samples significantly increases at a late stage of training . That is , the red portion in Figure 2 ( a ) increases rapidly after the dashed line , in which we call the error-prone period . ( See Section 3.2.1 for the details of estimating the error-prone period ) . We note that the training in that period brings no benefit . The generalization performance of “ Default ” deteriorates sharply , as shown in Figure 2 ( c ) . Based on these findings , we contend that eliminating this error-prone period should make a profound impact on robust optimization . In this paper , we propose a novel approach , called Prestopping , that achieves noise-free training based on the early stopping mechanism . Because there is no benefit from the error-prone period , Prestopping early stops training before that period begins . This early stopping effectively prevents a network from overfitting to false-labeled samples , and the samples memorized until that point are added to a maximal safe set because they are true-labeled ( i.e. , blue in Figure 2 ( a ) ) with high precision . Then , Prestopping resumes training the early stopped network only using the maximal safe set in support of noise-free training . Notably , our proposed merger of “ early stopping ” and “ learning from the maximal safe set ” indeed eliminates the error-prone period from the training process , as shown in Figure 2 ( b ) . As a result , the generalization performance of a DNN remarkably improves in both noise types , as shown in Figure 2 ( c ) . 1The learning rate , as usual , was decayed at 50 % and 75 % of the total number of training epochs . To validate the superiority of Prestopping , DenseNet ( Huang et al. , 2017 ) and VGG-19 ( Simonyan & Zisserman , 2015 ) were trained on both simulated and real-world noisy data sets , including CIFAR-10 , CIFAR-100 , ANIMAL-10N , and Food-101N . Compared with four state-of-the-art methods , Prestopping significantly improved test error by up to 18.1pp2 in a wide range of noise rates . 2 RELATED WORK . Numerous studies have been conducted to address the problem of learning from noisy labels . A typical method is using “ loss correction ” that estimates the label transition matrix and corrects the loss of the samples in the mini-batch . Bootstrap ( Reed et al. , 2015 ) updates the network based on their own reconstruction-based objective with the notion of perceptual consistency . F-correction ( Patrini et al. , 2017 ) reweights the forward or backward loss of the training samples based on the label transition matrix estimated by a pre-trained normal network . R3 : D2L ( Ma et al. , 2018 ) employs a simple measure called local intrinsic dimensionality and then uses it to modify the forward loss in order to reduce the effects of false-labeled samples in learning . Active Bias ( Chang et al. , 2017 ) heuristically evaluates uncertain samples with high prediction variances and then gives higher weights to their backward losses . Ren et al . ( 2018 ) include small clean validation data into the training data and re-weight the backward loss of the mini-batch samples such that the updated gradient minimizes the loss of those validation data . However , this family of methods is known to accumulate severe noise from the false correction , especially when the number of classes or the number of false-labeled samples is large ( Han et al. , 2018 ; Yu et al. , 2019 ) . To be free from the false correction , many recent researches have adopted “ sample selection ” that trains the network on selected samples . These methods attempt to select true-labeled samples from the noisy training data for use in updating the network . Decouple ( Malach & Shalev-Shwartz , 2017 ) maintains two networks simultaneously and updates the models only using the samples that have different label predictions from these two networks . R3 : Wang et al . ( 2018 ) proposed an iterative learning framework that learns deep discriminative features from well-classified noisy samples based on the local outlier factor algorithm ( Breunig et al. , 2000 ) . MentorNet ( Jiang et al. , 2018 ) introduces a collaborative learning paradigm that a pre-trained mentor network guides the training of a student network . Based on the small-loss criteria , the mentor network provides the student network with the samples whose label is probably correct . Co-teaching ( Han et al. , 2018 ) and Co-teaching+ ( Yu et al. , 2019 ) also maintain two networks , but each network selects a certain number of small-loss samples and feeds them to its peer network for further training . Compared with Co-teaching , Co-teaching+ further employs the disagreement strategy of Decouple . R1 : ITLM ( Shen & Sanghavi , 2019 ) iteratively minimizes the trimmed loss by alternating between selecting a fraction of small-loss samples at current moment and retraining the network using them . INCV ( Chen et al. , 2019 ) randomly divides the noisy training data and then utilizes cross-validation to classify true-labeled samples with removing large-loss samples at each iteration . However , their general philosophy of selecting small-loss samples works well only in some cases such as symmetric noise . RA : Differently to this family , we exploit the maximal safe set initially derived from the memorized samples at the early stop point . Most recently , a hybrid of “ loss correction ” and “ sample selection ” approaches was proposed by Song et al . ( 2019 ) . Their algorithm called SELFIE trains the network on selectively refurbished false-labeled samples together with small-loss samples . SELFIE not only minimizes the number of falsely corrected samples but also exploits full exploration of the entire training data . Its component for sample selection can be replaced by our method to further improve the performance . RA : For the completeness of the survey , we mention the work that provides an empirical or a theoretical analysis on why early stopping helps learn with the label noise . Oymak et al . ( 2019 ) and Hendrycks et al . ( 2019 ) have argued that early stopping is a suitable strategy because the network eventually begins to memorize all noisy samples if it is trained too long . Li et al . ( 2019 ) have theoretically proved that the network memorizes false-labeled samples at a later stage of training , and thus claimed that the early stopped network is fairly robust to the label noise . However , they did not mention how to take advantage of the early stopped network for further training . Please note that Prestopping adopts early stopping to derive a seed for the maximal safe set , which is exploited to achieve noise-free training during the remaining learning period . Thus , our novelty lies in the “ merger ” of early stopping and learning from the maximal safe set . 2A pp is the abbreviation of a percentage point . 3 ROBUST TRAINING VIA Prestopping 3.1 PRELIMINARIES . A k-class classification problem requires the training data D = { xi , y∗i } Ni=1 , where xi is a sample and y∗i ∈ { 1 , 2 , . . . , k } is its true label . Following the label noise scenario , let ’ s consider the noisy training data D̃ = { xi , ỹi } Ni=1 , where ỹi ∈ { 1 , 2 , . . . , k } is a noisy label which may not be true . Then , in conventional training , when a mini-batch Bt = { xi , ỹi } bi=1 consists of b samples randomly drawn from the noisy training data D̃ at time t , the network parameter θt is updated in the descent direction of the expected loss on the mini-batch Bt as in Eq . ( 1 ) , where α is a learning rate and L is a loss function . θt+1 = θt − α∇ ( 1 |Bt| ∑ x∈Bt L ( x , ỹ ; θt ) ) ( 1 ) As for the notion of network memorization , a sample x is defined to be memorized by a network if the majority of its recent predictions at time t coincide with the given label , as in Definition 3.1 . Definition 3.1 . ( Memorized Sample ) Let ŷt = Φ ( x|θt ) be the predicted label of a sample x at time t and Htx ( q ) = { ŷt1 , ŷt2 , . . . , ŷtq } be the history of the sample x that stores the predicted labels of the recent q times , where Φ is a neural network . Next , P ( y|x , t ; q ) is formulated such that it provides the probability of the label y ∈ { 1 , 2 , ... , k } estimated as the label of the sample x based on Htx as in Eq . ( 2 ) , where [ · ] is the Iverson bracket3 . P ( y|x , t ; q ) = ∑ ŷ∈Htx ( q ) [ ŷ = y ] |Htx ( q ) | ( 2 ) Then , the sample x with its noisy label ỹ is a memorized sample of the network with the parameter θt at time t if the condition in Eq . ( 3 ) holds . argmaxyP ( y|x , t ; q ) = ỹ ( 3 )

Summary: The paper proposes to study how early stopping in optimization helps find confident examples. Overall, the paper is well-organized and easy to read. Although there is some parallel study regarding the theoretical aspect of how early stopping help finds confident examples (i.e., Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks, which has unfortunately not been cited), the paper focuses on the empirical perspective. A thorough empirical study illustrating how early stop works would interest the label noise community.