Understanding and Improving Early Stopping for Learning with Noisy Labels

1 Introduction . Deep networks have revolutionized a wide variety of tasks , such as image processing , speech recognition , and language modeling [ 7 ] , However , this highly relies on the availability of large annotated data , which may not be feasible in practice . Instead , many large datasets with lower quality annotations are collected from online queries [ 5 ] or social-network tagging [ 18 ] . Such annotations inevitably contain mistakes or label noise . As deep networks have large model capacities , they can easily memorize and eventually overfit the noisy labels , leading to poor generalization performance [ 36 ] . Therefore , it is of great importance to develop a methodology that is robust to noisy annotations . Existing methods on learning with noisy labels ( LNL ) can be mainly categorized into two groups : model-based and model-free algorithms . Methods in the first category mainly model noisy labels with the noise transition matrix [ 24 , 34 , 33 , 30 ] . With perfectly estimated noise transition matrix , models trained with corrected losses can approximate to the models trained with clean labels . However , ∗co-first author †Correspondence to Tongliang Liu ( tongliang.liu @ sydney.edu.au ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . current methods are usually fragile to estimate the noise transition matrix for heavy noisy data and are also hard to handle a large number of classes [ 9 ] . The second type explores the dynamic process of optimization policies , which relates to the memorization effect−deep neural networks tend to first memorize and fit majority ( clean ) patterns and then overfit minority ( noisy ) patterns [ 2 ] . Recently , based on this phenomenon , many methods [ 9 , 26 , 15 , 16 , 29 ] have been proposed and achieved promising performance . To exploit the memorization effect , when the double descent phenomenon [ 3 , 22 , 11 ] can not be guaranteed to occur , a core issue is to study when to stop the optimization of the network . While stopping the training for too few epochs can avoid overfitting to noisy labels , it can also make the network underfit to clean labels . Current methods [ 25 , 23 ] usually adopt an early stopping strategy , which decides the stopping point by considering the network as a whole . However , since DNNs are usually optimized with stochastic gradient descent ( SGD ) with backpropagation , supervisory signals will gradually propagate through the whole network from latter layers ( i.e. , layers that are closer to output layers ) to former layers ( i.e. , layers that are closer to input layers ) . Noting that the output layer is followed by the empirical risk in the optimization procedure . We hypothesize that noisy labels may have more severe impacts for the latter layers , which is different from current methods [ 9 , 15 ] that usually stop the training of the whole network at once . To empirically verify the above hypothesis , we analyze the impact of noisy labels on representations from different layers with different training epochs . To quantitatively measure the impact of noisy labels from intermediate layers , we first train the whole network on noisy data with different training epochs and fix the parameters for the selected layer and its previous layers . We then reinitialize and optimize the rest layers with clean data , and the final classification performance is adopted to evaluate the impact of noisy labels . For the final layer , we directly report the overall classification performance . As illustrated in Figure 1 , we can see that latter layers always achieve the best performance at relatively smaller epoch numbers and then exhibit stronger performance drops with additional training epochs , which verifies the hypothesis that noisy data may have more severe impacts for latter layers . With this understanding , we can infer that the early stopping , which optimizes the network all at once , may fail to fully exploit the memorization effect and induce sub-optimal performance . To address the above problem , we propose to optimize a DNN by considering it as a composition of several DNN parts and present a novel progressive early stopping ( PES ) method . Specifically , we initially train former DNN layers by optimizing them with a relatively large number of epochs . Then , to alleviate the impact of noisy labels for latter layers , we reinitialize and progressively train latter DNN layers by using smaller numbers of epochs with preceding DNN layers fixed . Since different layers are progressively trained with different early stopping epochs , we term the proposed method as progressive early stopping ( PES ) . Despite its simplicity , compared with normal early stopping trick , PES can help to better exploit the memorization effect and obtain more promising and stable results . Moreover , since the model size and training epochs are gradually reduced during the optimization procedure , the training time of PES is only slightly greater than that of the normal early stopping . Finally , by combining PES with existing approaches on noisy label training tasks , we establish new state-of-the-art ( SOTA ) results on CIFAR-10 and CIFAR-100 with synthetic noise . We also achieve competitive results on one dataset with real-world noise : Clothing-1M [ 32 ] . The rest of the paper is organized as follows . In Section 2 , we first introduce the proposed progressive early stopping and then present the details of the proposed algorithm by combining our method with existing approaches on noisy label training tasks . Section 3 shows the experimental results of our proposed method . Related works are briefly reviewed in Section 4 . Finally , concluding remarks are given in Section 5 . 2 Proposed Method . Let D be the distribution of a pair of random variables ( X , Y ) ∈ X × { 1 , ... K } , where X indicates the variable of instances , Y is the variable of labels , X denotes the feature space , andK is the number of classes . In many real-world problems , examples independently drawn from the distribution D are unavailable . Before being observed , the clean labels are usually randomly corrupted into noisy labels . Let D̃ be the distribution of the noisy example ( X , Ỹ ) , where Ỹ indicates the variable of noisy labels . For label-noise learning , we can only access a sample set { xi , ỹi ) } ni=1 independently drawn from D̃ . The aim is to learn a robust classifier from the noisy sample set that can classify test instances accurately . In the following , we first elaborate on the proposed progressive early stopping ( PES ) . Then , based on PES , we provide a learning algorithm that learns with confident examples and semi-supervised learning techniques . 2.1 Progressive Early Stopping . When trained with noisy labels , if clean labels are of majority within each noisy class , deep networks tend to first fit clean labels during an early learning stage before eventually memorizing the wrong labels , which can be explained by the memorization effect . Many current methods utilize this property to counteract the influence of noisy labels by stopping the optimization at an early learning phase . Specifically , a deep classifier can be obtained by optimizing the following objective function with a relatively small epoch number T : min Θ 1 n n∑ i=1 L ( f ( xi ; Θ ) , ỹi ) , ( 1 ) where f ( · ; Θ ) is a deep classifier with model parameters Θ and L is the cross-entropy loss . When trained with noisy data , early learning regularization ( ELR ) [ 16 ] reveals that , for the most commonly used cross-entropy loss , the gradient is well correlated with the correct direction at the early learning phase . Therefore , with a properly defined small epoch number T , the classifier can have higher accuracy than at initialization . While , if we continue to optimize the deep model after T epochs , the classifier will be able to memorize more noise labels . Therefore , it is critical to select a proper epoch number T to utilize the memorization effect and alleviate the influence of noisy labels . Current methods typically select the epoch number T by considering the network as a whole . However , as Figure 1 makes clear , the impact of noisy labels on different DNN layers are different , which implies that the traditional early stopping trick , which optimizes the whole network all at once , may make different DNN layers to be antagonistically affected by each other , thus degrading the final model performance . To this end , we propose to separate a DNN into different parts and progressively train layers in different parts with different training epochs . Specifically , assume that the whole network f ( · ; Θ ) can be constituted with L DNN parts z1 = f1 ( x ; Θ1 ) , zl = fl ( zl−1 ; Θl ) , l = 2 , . . . , L ( 2 ) where fl ( · ; Θl ) is the l-th DNN part and zl is the corresponding output . The output of the last part zL is the prediction . The network f ( · ; Θ ) can also be represented as f ( · ; Θ1 , ... ΘL ) . To counteract the impact of noisy labels , We initially optimize the parameter Θ1 for the first part by training the whole network for T1 epochs with the following objective min Θ1 ... Θk 1 n n∑ i=1 L ( f ( xi ; Θ1 , . . . , ΘL ) , ỹi ) . ( 3 ) Then , we keep the obtained parameter Θ∗1 fixed , reinitialize and progressively learn the l-th ( l = 2 , . . . , L ) DNN part with the parameters for preceding DNN parts fixed . The training procedure is conducted with Tl epochs by optimizing the following objective min Θl ... Θk 1 n n∑ i=1 L ( f ( xi ; Θ∗1 , . . . , Θ∗l−1 , Θl , . . . , ΘL ) , ỹi ) , l = 2 . . . L ( 4 ) We gradually optimize the ( l+1 ) -th DNN part with the obtained parameter Θ∗l fixed , the optimization is continued until all the parameters have been optimized . As elaborated above , latter DNN parts are more sensitive to noisy labels than their former counterparts . Therefore , for the above initializing optimization in Eq . ( 3 ) and the following L− 1 steps of optimization in Eq . ( 4 ) , we gradually reduce the training epochs ( i.e . T1 ≥ T2 ≥ · · · ≥ TL ) to better exploit the memorization effect . After optimization , we can obtain the final network as f ( · , Θ ) = f ( · ; Θ∗1 , . . . , Θ∗L ) . Since this model is obtained by progressively exploiting the early stopping strategies for different DNN parts , we term the proposed method as progressive early stopping ( PES ) . To explicitly verify the effectiveness of the proposed PES method , we conduct several pilot experiments , which compare the traditional early stopping and PES with label noise from different types and different levels . The results are illustrated in Figure 2 , from which we can see that , compared with models trained with traditional early stopping , models trained with PES can achieve superior classification accuracy with smaller variations in all the cases . Current state-of-the-art methods [ 15 ] usually adopt models with the traditional early stopping as base models to distill confident examples and then utilize semi-supervised learning techniques by considering confident examples as labeled data and other noisy examples as unlabeled data to further improve the results . The final performance still heavily relies on the base model trained with noisy labels . By improving the performance of the base model , our method combined with semi-supervised learning techniques is able to establish new state-of-the-art results . In the following subsections , we will elaborate on how to utilize PES to distill confident examples and further combine it with semi-supervised learning techniques .

Summary: The paper introduces a novel training scheme (progressive early stopping, PES) aimed at improving the performance of convolutional networks for image classification in presence of noisy labels. The strategy is motivated by the observation that the layers close to the output tend to overfit noisy targets more than early layers. PES thus proposes to differentiate the number of training epochs for different groups of layers, training early groups for more epochs than final ones. In detail PES divides the networks in N blocks and trains for T_1 epochs the whole network. It then freezes the learned weights of the first block and  reinitializes  and trains the layers of the remaining N-1 blocks for T_2 epochs. Crucially, T_2 < T_1. The same policy is then progressively repeated for all the remaining N-2 blocks with T_n <T_n-1 for n in  {1, …, N}.  PES is also combined with a semi-supervised scheme (MixMatch) to reach SOTA performances on CIFAR10 and CIFAR100 datasets.