Prestopping: How Does Early Stopping Help Generalization Against Label Noise?

1 INTRODUCTION . By virtue of massive labeled data , deep neural networks ( DNNs ) have achieved a remarkable success in numerous machine learning tasks , such as image classification ( Krizhevsky et al. , 2012 ) and object detection ( Redmon et al. , 2016 ) . However , owing to their high capacity to memorize any label noise , the generalization performance of DNNs drastically falls down when noisy labels are contained in the training data ( Jiang et al. , 2018 ; Han et al. , 2018 ; Song et al. , 2019 ) . In particular , Zhang et al . ( 2017 ) have shown that a standard convolutional neural network ( CNN ) can easily fit the entire training data with any ratio of noisy labels and eventually leads to very poor generalization on the test data . Thus , it is challenging to train a DNN robustly even when noisy labels exist in the training data . A popular approach to dealing with noisy labels is “ sample selection ” that selects true-labeled samples from the noisy training data ( Jiang et al. , 2018 ; Ren et al. , 2018 ; Han et al. , 2018 ; Yu et al. , 2019 ; Song et al. , 2019 ) . Here , ( 1−τ ) ×100 % of small-loss training samples are treated as true-labeled ones and then used to update a DNN robustly , where τ ∈ [ 0 , 1 ] is a noise rate . This loss-based separation is well known to be justified by the memorization effect ( Arpit et al. , 2017 ) that DNNs tend to learn easy patterns first and then gradually memorize all samples . In practice , Han et al . ( 2018 ) empirically proved that training on such small-loss samples yields a much better generalization performance under artificial noise scenarios . Despite its great success , Song et al . ( 2019 ) have recently argued that the performance of the lossbased separation becomes considerably worse depending on the type of label noise . For instance , in symmetric noise ( Figure 1 ( a ) ) , the loss-based approach well separates true-labeled samples from false-labeled ones because many true-labeled ones exhibit smaller loss than false-labeled ones . On the other hand , in pair and real-world noises ( Figures 1 ( b ) and 1 ( c ) ) , many false-labeled samples are misclassified as true-labeled ones because the two distributions overlap closely ; this overlap confirms that the loss-based separation still accumulates severe label noise from many misclassified cases , especially in real-world noise or pair noise which is regarded as more realistic than symmetric noise ( Ren et al. , 2018 ; Yu et al. , 2019 ) . This limitation definitely calls for a new approach that supports any type of label noise for practical use . In this regard , as shown in Figure 2 ( a ) , we thoroughly investigated the memorization effect of a DNN on the two types of noises and found two interesting properties as follows : • A noise type affects the memorization rate for false-labeled samples : The memorization rate for false-labeled samples is faster with pair noise than with symmetric noise . That is , the red portion in Figure 2 ( a ) starts to appear earlier in pair noise than in symmetric noise . This observation supports the significant overlap of true-labeled and false-labeled samples in Figure 1 ( b ) . Thus , the loss-based separation performs well only if the false-labeled samples are scarcely learned at an early stage of training , as in symmetric noise . • There is a period where the network accumulates the label noise severely : Regardless of the noise type , the memorization of false-labeled samples significantly increases at a late stage of training . That is , the red portion in Figure 2 ( a ) increases rapidly after the dashed line , in which we call the error-prone period . ( See Section 3.2.1 for the details of estimating the error-prone period ) . We note that the training in that period brings no benefit . The generalization performance of “ Default ” deteriorates sharply , as shown in Figure 2 ( c ) . Based on these findings , we contend that eliminating this error-prone period should make a profound impact on robust optimization . In this paper , we propose a novel approach , called Prestopping , that achieves noise-free training based on the early stopping mechanism . Because there is no benefit from the error-prone period , Prestopping early stops training before that period begins . This early stopping effectively prevents a network from overfitting to false-labeled samples , and the samples memorized until that point are added to a maximal safe set because they are true-labeled ( i.e. , blue in Figure 2 ( a ) ) with high precision . Then , Prestopping resumes training the early stopped network only using the maximal safe set in support of noise-free training . Notably , our proposed merger of “ early stopping ” and “ learning from the maximal safe set ” indeed eliminates the error-prone period from the training process , as shown in Figure 2 ( b ) . As a result , the generalization performance of a DNN remarkably improves in both noise types , as shown in Figure 2 ( c ) . 1The learning rate , as usual , was decayed at 50 % and 75 % of the total number of training epochs . To validate the superiority of Prestopping , DenseNet ( Huang et al. , 2017 ) and VGG-19 ( Simonyan & Zisserman , 2015 ) were trained on both simulated and real-world noisy data sets , including CIFAR-10 , CIFAR-100 , ANIMAL-10N , and Food-101N . Compared with four state-of-the-art methods , Prestopping significantly improved test error by up to 18.1pp2 in a wide range of noise rates . 2 RELATED WORK . Numerous studies have been conducted to address the problem of learning from noisy labels . A typical method is using “ loss correction ” that estimates the label transition matrix and corrects the loss of the samples in the mini-batch . Bootstrap ( Reed et al. , 2015 ) updates the network based on their own reconstruction-based objective with the notion of perceptual consistency . F-correction ( Patrini et al. , 2017 ) reweights the forward or backward loss of the training samples based on the label transition matrix estimated by a pre-trained normal network . R3 : D2L ( Ma et al. , 2018 ) employs a simple measure called local intrinsic dimensionality and then uses it to modify the forward loss in order to reduce the effects of false-labeled samples in learning . Active Bias ( Chang et al. , 2017 ) heuristically evaluates uncertain samples with high prediction variances and then gives higher weights to their backward losses . Ren et al . ( 2018 ) include small clean validation data into the training data and re-weight the backward loss of the mini-batch samples such that the updated gradient minimizes the loss of those validation data . However , this family of methods is known to accumulate severe noise from the false correction , especially when the number of classes or the number of false-labeled samples is large ( Han et al. , 2018 ; Yu et al. , 2019 ) . To be free from the false correction , many recent researches have adopted “ sample selection ” that trains the network on selected samples . These methods attempt to select true-labeled samples from the noisy training data for use in updating the network . Decouple ( Malach & Shalev-Shwartz , 2017 ) maintains two networks simultaneously and updates the models only using the samples that have different label predictions from these two networks . R3 : Wang et al . ( 2018 ) proposed an iterative learning framework that learns deep discriminative features from well-classified noisy samples based on the local outlier factor algorithm ( Breunig et al. , 2000 ) . MentorNet ( Jiang et al. , 2018 ) introduces a collaborative learning paradigm that a pre-trained mentor network guides the training of a student network . Based on the small-loss criteria , the mentor network provides the student network with the samples whose label is probably correct . Co-teaching ( Han et al. , 2018 ) and Co-teaching+ ( Yu et al. , 2019 ) also maintain two networks , but each network selects a certain number of small-loss samples and feeds them to its peer network for further training . Compared with Co-teaching , Co-teaching+ further employs the disagreement strategy of Decouple . R1 : ITLM ( Shen & Sanghavi , 2019 ) iteratively minimizes the trimmed loss by alternating between selecting a fraction of small-loss samples at current moment and retraining the network using them . INCV ( Chen et al. , 2019 ) randomly divides the noisy training data and then utilizes cross-validation to classify true-labeled samples with removing large-loss samples at each iteration . However , their general philosophy of selecting small-loss samples works well only in some cases such as symmetric noise . RA : Differently to this family , we exploit the maximal safe set initially derived from the memorized samples at the early stop point . Most recently , a hybrid of “ loss correction ” and “ sample selection ” approaches was proposed by Song et al . ( 2019 ) . Their algorithm called SELFIE trains the network on selectively refurbished false-labeled samples together with small-loss samples . SELFIE not only minimizes the number of falsely corrected samples but also exploits full exploration of the entire training data . Its component for sample selection can be replaced by our method to further improve the performance . RA : For the completeness of the survey , we mention the work that provides an empirical or a theoretical analysis on why early stopping helps learn with the label noise . Oymak et al . ( 2019 ) and Hendrycks et al . ( 2019 ) have argued that early stopping is a suitable strategy because the network eventually begins to memorize all noisy samples if it is trained too long . Li et al . ( 2019 ) have theoretically proved that the network memorizes false-labeled samples at a later stage of training , and thus claimed that the early stopped network is fairly robust to the label noise . However , they did not mention how to take advantage of the early stopped network for further training . Please note that Prestopping adopts early stopping to derive a seed for the maximal safe set , which is exploited to achieve noise-free training during the remaining learning period . Thus , our novelty lies in the “ merger ” of early stopping and learning from the maximal safe set . 2A pp is the abbreviation of a percentage point . 3 ROBUST TRAINING VIA Prestopping 3.1 PRELIMINARIES . A k-class classification problem requires the training data D = { xi , y∗i } Ni=1 , where xi is a sample and y∗i ∈ { 1 , 2 , . . . , k } is its true label . Following the label noise scenario , let ’ s consider the noisy training data D̃ = { xi , ỹi } Ni=1 , where ỹi ∈ { 1 , 2 , . . . , k } is a noisy label which may not be true . Then , in conventional training , when a mini-batch Bt = { xi , ỹi } bi=1 consists of b samples randomly drawn from the noisy training data D̃ at time t , the network parameter θt is updated in the descent direction of the expected loss on the mini-batch Bt as in Eq . ( 1 ) , where α is a learning rate and L is a loss function . θt+1 = θt − α∇ ( 1 |Bt| ∑ x∈Bt L ( x , ỹ ; θt ) ) ( 1 ) As for the notion of network memorization , a sample x is defined to be memorized by a network if the majority of its recent predictions at time t coincide with the given label , as in Definition 3.1 . Definition 3.1 . ( Memorized Sample ) Let ŷt = Φ ( x|θt ) be the predicted label of a sample x at time t and Htx ( q ) = { ŷt1 , ŷt2 , . . . , ŷtq } be the history of the sample x that stores the predicted labels of the recent q times , where Φ is a neural network . Next , P ( y|x , t ; q ) is formulated such that it provides the probability of the label y ∈ { 1 , 2 , ... , k } estimated as the label of the sample x based on Htx as in Eq . ( 2 ) , where [ · ] is the Iverson bracket3 . P ( y|x , t ; q ) = ∑ ŷ∈Htx ( q ) [ ŷ = y ] |Htx ( q ) | ( 2 ) Then , the sample x with its noisy label ỹ is a memorized sample of the network with the parameter θt at time t if the condition in Eq . ( 3 ) holds . argmaxyP ( y|x , t ; q ) = ỹ ( 3 )

Summary: This paper presents a training approach on label noise datasets and outperforms state-of-art methods. It defines the samples whose average probability on assigned label in recent q iterations is largest among all labels as memorized samples, in the sense of the network memorize these samples. Then authors proposed two stage method which firstly early-stops at minimum validation error (or $\tau$ memorized rate), and then trains on maximal safe set that gathers memorized samples. The experiments compared several state-of-art approaches and showed that the proposed method benefits from early-stopping and safe set. Authors also showed that the prestopping idea can also be used to improve other approaches.