Understanding and Improving Early Stopping for Learning with Noisy Labels
Authors: Yingbin Bai
Erkun Yang
Bo Han
Yanhua Yang
Jiatong Li
Yinian Mao
Gang Niu
Tongliang Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: learning with noisy labels
weekly supervised learning

Abstract: The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods.  To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering a DNN as a whole. However, a DNN can be considered as a composition of a series of layers, and we find that the latter layers in a DNN are much more sensitive to label noise, while their former counterparts are quite robust. Therefore, selecting a stopping point for the whole network may make different DNN layers antagonistically affect each other, thus degrading the final performance. In this paper, we propose to separate a DNN into different parts and progressively train them to address this problem. Instead of the early stopping which trains a whole DNN all at once, we initially train former DNN layers by optimizing the DNN with a relatively large number of epochs. During training, we progressively train the latter DNN layers by using a smaller number of epochs with the preceding layers fixed to counteract the impact of noisy labels. We term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with the traditional early stopping, PES can help to obtain more promising and stable results. Furthermore, by combining PES with existing approaches on noisy label training, we achieve state-of-the-art performance on image classification benchmarks. The code is made public at https://github.com/tmllab/PES.

Understanding and Improving Early Stopping for Learning with Noisy Labels
Yingbin Bai
Erkun Yang2*
Bo Han
Yanhua Yang²
Jiatong Li
Yinian Mao
Gang Niu
Tongliang Liu"
'ML Lab, University of Sydney; Xidian University; Hong Kong Baptist University;
"Meituan Group; SRIKEN AIP
Abstract
The memorization effect of deep neural network (DNN) plays pivotal role in many state-of-the-are label-noise learning methods. To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering DNN as whole. However, a DNN can be considered as a composition series of layers, and we find that the latter layers a DNN are much more sensitive to label noise, while their former counterparts are quite robust. Therefore, selecting a stopping point for the whole network may make different DNN layers antagonistically affect each other, thus degrading the final performance. In this paper, we propose to separate DNN into different parts and progressively train them to address this problem. Instead of the early stopping which trains a whole DNN all at once, we initially train former DNN layers by optimizing the DNN with a relatively large number of epochs. During training, we progressively train the latter DNN layers by using smaller number of epochs with the preceding layers fixed counteract the impact of noisy labels. We term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with the traditional early stopping, PES can help to obtain more promising and stable results. Furthermore, by combining PES with existing approaches on noisy label training, we achieve state-of- the art performance on image classification benchmarks. The code is made public at //ttts://ggiiieii com/ ttmttm/mlabbba//
Introduction
Deep networks have revolutionized wide variety of tasks, such image processing, speech recognition, and language modeling (7). However, this highly relies on the availability of large annotated data, which may not be feasible in practice. Instead, many large datasets with lower quality annotations are collected from online queries (5) or social-network tagging [18]. Such annotations inevitably contain mistakes or label noise. As deep networks have large model capacities, they can easily memorize and eventually overfit the noisy labels, leading to poor generalization performance [36]. Therefore, it is of great importance to develop a methodology that is robust to noisy annotations.
Existing methods on learning with noisy labels (LNL) be mainly categorized into two groups: model-based and model-free algorithms. Methods in the first category mainly model noisy labels with With perfectly estimated noise transition matrix, models trained with corrected losses can approximate to the models trained with clean labels. However,
the noise transition matrix [24 34 33 30
"co-first author
"Conespondence Tongliang Liu (tongliang liu @llluyyyyydcd.uuuu
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

00
60
To exploit the memorization effect, when the double descent phenomenon 3 guaranteed to occur, core issue is to study when to stop the optimization of the network. While stopping the training for too few epochs can avoid overfitting to noisy labels, it can also make the usually adopt an early stopping strategy, which decides the stopping point by considering the network as a whole. However, since DNNs are usually optimized with stochastic gradient descent (SGD) with backpropagation, supervisory signals will gradually propagate through the whole network from latter layers (i.e.. layers that are closer to output layers) to former layers (i.e., layers that are closer to input layers). Noting that the output layer is followed by the empirical risk in the optimization procedure. We hypothesize that noisy labels may that Test accuacy
scuray
Test -3
3
3.1 Juyer
layor
inger Fnnel layse
luyar
peral layer
20
Epocho
Epochs
Epochs
(a) Symmetric 50%
(b) Pairflip 45%
(c) Instance 40%
We train a ResNet-18 18 model on CIFAR-10 with three types of noisy labels and evaluate the impact of noisy labels the representations from the 9-th layer, the 17-th layer, and the final layer. The X-axis is the number of epochs for the first block of the network. The curves present the mean of five runs and the best performances are indicated with dotted vertical lines.
Figure I:
current methods are usually fragile to estimate the noise transition matrix for heavy noisy data and
also hard to handle a large number of classes (9) The second type explores the dynamic process of optimization policies, which relates to the memorization effect-deep neural networks tend to first
memorize and fit majority (clean) patterns and then overfit minority (noisy) patterns [2]. Recently,
based on this phenomenon, many methods [9] 26 15 promising performance.
16 29 have been proposed and achieved
cannot be
network underfit to clean labels. Current methods 25 23
have more severe impacts for the latter layers, which is different from current methods usually stop the training of the whole network at once.
To empirically verify the above hypothesis, analyze the impact of noisy labels representations from different layers with different training epochs. To quantitatively measure the impact of noisy labels from intermediate layers, first train the whole network on noisy data with different training epochs and fix the parameters for the selected layer and its previous layers. We then reinitialize and optimize the rest layers with clean data, and the final classification performance is adopted to evaluate the impact of noisy labels. For the final layer, we directly report the overall classification performance. As illustrated in Figure we can see that latter layers always achieve the best performance at relatively smaller epoch numbers and then exhibit stronger performance drops with additional training epochs, which verifies the hypothesis that noisy data may have more severe impacts for latter layers. With this understanding, we can infer that the early stopping, which optimizes the network all at once, may fail to fully exploit the memorization effect and induce sub-optimal performance.
To address the above problem, we propose to optimize a DNN by considering it as a composition of several DNN parts and present novel progressive early stopping (PES) method. Specifically, we initially train former DNN layers by optimizing them with a relatively large number of epochs. Then, to alleviate the impact of noisy labels for latter layers, we reinitialize and progressively train latter DNN layers by using smaller numbers of epochs with preceding DNN layers fixed. Since different layers are progressively trained with different early stopping epochs, we term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with normal early stopping trick, PES can help to better exploit the memorization effect and obtain more promising and stable results. Moreover, since the model size and training epochs are gradually reduced during the optimization procedure, the training time of PES is only slightly greater than that of the normal early stopping. Finally, by combining PES with existing approaches noisy label training tasks, we establish new state-of-thh--ar (SOTA) results on CIFAR-10 and CIFAR-100 with synthetic noise. We also achieve competitive results on one dataset with real-world noise: Clothing- 1M (32).

---

The rest of the paper is organized as follows. In Section 2 we first introduce the proposed progressive early stopping and then present the details of the proposed algorithm by combining our method with existing approaches on noisy label training tasks. Section 3 shows the experimental results of our proposed method. Related works are briefly reviewed Section 4 Finally, concluding remarks are given in Section 5
2 Proposed Method
Let D be the distribution of a pair of random variables (X, Y) E X {I,.. where X indicates the variable of instances, Y is the variable labels, X denotes the feature space, and K the number of classes. In many real-world problems, examples independently drawn from the distribution D are unavailable. Before being observed, the clean labels are usually randomly corrupted into noisy labels. Let D be the distribution of the noisy example (x, Y). where Y indicates the variable of noisy labels. For label-noise learning, drawn from D. The aim is learn robust classifier from the noisy sample set that can classify test instances accurately.
only access a sample set {æ;, ū+))== independently
In the following, we first elaborate on the proposed progressive early stopping (PES). Then, based on PES, we provide a learning algorithm that learns with confident examples and semi-supervised learning techniques.
2.1 Progressive Early Stopping
When trained with noisy labels, if clean labels are of majority within each noisy class, deep networks tend to first clean labels during an early learning stage before eventually memorizing the wrong labels, which can be explained by the memorization effect. Many current methods utilize this property to counteract the influence of noisy labels by stopping the optimization at an early learning phase. Specifically, deep classifier can be obtained by optimizing the following objective function with relatively small epoch number T::
1 n
min Σ (), θ 7I
(I)
where f(::;) θ) is a deep classifier with model parameters t and C is the cross-entropy loss. When trained with noisy data, early learning regularization (ELR) [16] reveals that, for the most commonly used cross-entropy loss, the gradient is well correlated with the correct direction at the early learning phase. Therefore, with a properly defined small epoch number T, the classifier can have higher accuracy than at initialization. While, we continue to optimize the deep model after T epochs, the classifier will be able memorize more noise labels. Therefore, it is critical select a proper epoch number I' to utilize the memorization effect and alleviate the influence of noisy labels.
Current methods typically select the epoch number T by considering the network a whole. However, as Figure T makes clear, the impact of noisy labels on different DNN layers are different, which implies that the traditional early stopping trick, which optimizes the whole network all once, may make different DNN layers to be antagonistically affected by each other, thus degrading the final model performance. To this end, we propose to separate a DNN into different parts and progressively train layers in different parts with different training epochs. Specifically, assume that the whole network ((; Q) can be constituted with I. DNN parts
21 fi θ1), 21 = f((z-1;11),
(2)
1= 2.,,..,,
where fi(:: Or) is the (-th DNN part and æ is the corresponding output. The output of the last part # is the prediction. The network f(;; can also be represented as f(: Oy) To counteract the impact of noisy labels, We initially optimize the parameter O for the first part by training the whole network for Ti epochs with the following objective
n
min ep... YI
O,. ((.))).......)........
(3)
3

---

g
Test Scouracy
Test accuracy
Early stosping
Earyy Mooping
46
Ear 200ding
Spochs
Epochs
Epochs
(a) Symmetric 50%
(b) Pairflip 45%
(c) Instance 40%
Figure 2: Performance of the traditional early stopping trick and the proposed PES on CIFAR-I0 with different types of label noise. The lines present the of five runs.
Then, we keep the obtained parameter O fixed, reinitialize and progressively learn the 1-th ..... L) DNN part with the parameters for preceding DNN parts fixed. The training procedure is conducted with Ti epochs by optimizing the following objective
min O 71
Σ CC ();; 01 0r. Or.... OL).0(, 2. 1=1
(4)
We gradually optimize the ((+1)-th DNN part with the obtained parameter O₁ fixed, the optimization continued until all the parameters have been optimized. As elaborated above, latter DNN parts are more sensitive noisy labels than their former counterparts. Therefore, for the above initializing optimization in Eq. (3) and the following
steps of optimization in Eq. (4). we gradually reduce T better exploit the memorization effect. After
the training epochs (L.e Ti T2 2 optimization, we can obtain the final network as [(,, O) [(; .., Since this model is obtained by progressively exploiting the early stopping strategies for different DNN parts, we term the proposed method as progressive early stopping (PES).
To explicitly verify the effectiivnness of the proposed PES method, conduct several pilot experi- ments, which compare the traditional early stopping and PES with label noise from different types and different levels. The results illustrated in Figure 2. from which with models trained with traditional early stopping. models trained with PES can achieve superior classification accuracy with smaller variations in all the cases. Current state-of-the- methods [I5] usually adopt models with the traditional early stopping as base models to distill confident examples and then utilize semi-supervised learning techniques by considering confident examples labeled data and other noisy examples as unlabeled data to further improve the results. The final performance still heavily relies the base model trained with noisy labels. By improving the performance of the base model, our method combined with semi-supervised learning techniques is able to establish new state-of-te the-art results. In the following subsections, we will elaborate on how to utilize PES to distill confident examples and further combine it with semi-supervised learning techniques.
that, compared
2.2 Learning with Confident Examples
Based the deep network optimized with progressive early stopping, examples to facilitate the model training. Here, confident examples refer to examples that have high probabilities with clean labels. In this paper, given labels as confident examples. In addition, to make the results more robust, we generate two different augmentations for any given input and use the average prediction decide its predicted label. Formally, we can obtain the confident example set Dj as
select confident
examples whose predictions consistent with
Dj {x T.)lū. Ú., 1, I,....n), ( (Augment(x:))) (x)) ( (Augment((z)):)) K) 2
(5)
0: arg max
where Augment(-) indicates normal data augmentation operation including horizontal random flip and random crops, and [*(;;; θ) the predicted probability of belonging to class Note that Augment(-) is stochastic transformation, the two terms in Eq 5 are not identical. The average

---

Algorithm 1: Progressive Early Stopping with Semi-Supervised Learning
Input: Neural network with trainable parameters (O Ar).
Noisy training dataset
(ax, Dc) R= Number of training epochs for different part: Ti, Ti.... TL, and training epochs Te for refining with confident examples.
for i ,, do
L Optimize network parameter θ with Eq. 3
for = 2, do
Froze {6 1............................... 11- and re-initialize (0)..... O1):
for i = I...., do
L Optimize network parameter (Oi,..., with Eq. 4
Unfroze O;
for 1i l,... T. do
Extract confident example set Dr and unlabeled set D,, with classifier [(s 0) by Eq. 7
Training the elassifier ((: θ) with MixMatch loss Dj and D..
Evaluate the obtained classifier f(-, ().
prediction of augmented examples provides more stable prediction and is found empirically to improve performance. After obtaining the confident example set, one can easily train classifier by considering confident examples as clean data. However, since the number of confident examples for different classes can vary greatly, directly training the model with the obtained confident example set may introduce severe class imbalance problem. To this end, we adopt weighted classification loss
C. Wy. Cili f(x;; 0)),
where w; is the corresponding class weight. Assuming that σ₂ = K (x), B.) = k, (x,, ū D denotes the cardinality of the confident example set belonging the k-th class. Then, set dj) to indicate the corresponding class importance.
wi 0./ ( K
2.3 Combining with Semi-Supervised Learning
Training with only confident examples neglects the rest data and may suffer from insufficient training examples. To tackle this problem, we further resort to semi-supervised learning techniques by considering confident examples labeled data and other noisy examples unlabeled data. Specifically, the labeled data set and unlabeled data set can be obtained as
Di = {I,, Ti 154 0.. I...n
D. ax. 1i Di, I,...
(Augment(xx))) 0) + +5* (Augmen((z:;;;)) θ)],
arg mnx kE(1....)
where the labeled data set the same that in Eg 6 and D., the rest unlabeled data set. Similar to [15], we adopt MixMatch [ as the semi-supervised learning framework to train the final classification models. For more details about superviid learning, we refer [4]. The whole learning algorithm is summarized in Algorithm I
3 Experiments
3.1 Datasets and Implementation Details
We evaluate our method on two synthetic datasets, CIFAR- 10 and CIFAR- 100 [12] with different levels of symmetric, pairflip, and eestance-dependee label noise (abbreviated as instance label noise) and a real-world dataset Clothing- 1M [32 Both CIFAR-10 and CIFAR- 100 contain 50k training images and 10k test images of size x 32. Following previous works
Datasets:
31 [6 29),
symmetric noise is generated by uniformly flipping labels for percentage of the training dataset
5

---

Table Preliminary analysis of the performance and the quality of extracted confident examples on CIFAR-10. 10. The mean and standard deviation are computed over five runs.
Metrics
Methods
Sym-20%
Sym-50%
Pair-45%
Inst-20%
Inst-40%
Early Stopping
82.55 12.46
70.76.1
60.62.15.59
84.41
74.73.12.65
Test Accuracy
PES
85.87 1. 1.59
15.87.11.33
62.40.12.34
86.58 0..45
77.07 1.118
Early Stopping
98.81 1:0.15
94.65:10.19
5..23:15.26
0..40
90.77:11.88
Label Precision
PES
98.96:+0.09
0..46:+0.14
72.99:22.27
98.52.40.1 19
90.63-t0.92
Early Stopping
88.51112.26
75.18.1
67.84 5.006
90.37:11.01
82.15.1..17
Label Recall
PES
92.6711..43
$1.0311.88
71.06
93.24 0.60
85.91110.68
to all possible labels. Pairflip noise flips noisy labels into their adjacent class. And, instance noise is generated by image features. More details about the synthetic label noise are given in the supplementary material. For the flipping rate, it can include [9 31 or ex-include [5 16 true labels. We use the flipping rate including correct labels in Table 3 to compare with results [5], and use without correct labels in the rest of the experiments. Clothing- IM (32) is large-scale dataset with real-world noisy labels, whose images are clawed from the online shopping websites, and labels are generated based surrounding texts. It contains million training images, and 15k validation images, and 10k test images with clean labels.
Semi sumi--upervised learning may strongly boost the performance, we separately compare our method with approaches with or without semi-supervised learning. For the comparison with baselines with semi-supervised learning, we combine our proposed method with MixMatch used in
Baselines:
15 as indicated in Subsection 2.3 (1) Approaches without semi-supervien
learning: Co-teaching
99. Forward [24). Joint Optim 225 T-revision [II DMI 34, and CDR (2). (2) Methods with
semi-supervised learning: M-correction [1 DivideMix [5], and ELR+ [T6]. We also adopt standard
training with cross-entropy (CE) and MixUp 3 as baselines to show improvements.
Network structure and optimization: Our method is implemented by PyTorch v1.6. Baseline methods are implemented based on public codes with hyper-parameters set according to the original papers. For DivideMiix and ELR+, we evaluate the test accuracy with the first network. To better demonstrate the robustness of our algorithm, we keep the hyper-parae parameters fixed for different types of label noise. More technique details are given in the supplementary material.
For experiments without semi cemi-supereieed learning, we follow BI, and use ResNet-18 [0] for CIFAR-10 and R4 for C00. We split networks into three parts, the layers above block 4 as part 1, block of ResNet as part 2, and the final layer as part 3. T, is defined as 25 for CIFAR-10 and 30 for 100, T2 as and T3 5. The network is trained for 200 epochs and SGD with 0.9 momentum is used. The initial learning rate is set to 0.1 and decayed with a factor of 10 at the 100th and 150th epoch respectively, and weight decay is set 10-4 For T3 and T3, employ an Adam optimizer with learming rate of 10-
For experiments with semi-supervised learning, we follow the setting of [5] with PreAct Resnet- 18. We set the final layer as part 2, the rest as part 1. Ti is defined as 20 for CIFAR-10 10 and 35 for CIFAR-100 100, and T3 5. The network is trained for 300 epochs. For optimization, cycle of cosine annealing [19, and the learning rate begins from 2 x 10-2 and ends at x 10 with weight decay of 5 x 10-4, An Adam optimizer is adopted with a learning rate of 10 for Tz For hyper-parameters from MixMatch, we set them according to the original paper 4
single
For Clothing-IM (32), we follow the previous work (25, and employ a ResNet-50 10 pre-trained on ImageNet [3]. We set the final layer as part 2, the rest as part 1. Ti and T2 are defined as 20 and 7 respectively. The network is trained with CE loss for 50 epochs and SGD is used with 0.9 momentum and a weight decay of 10 3, The learning rate is 5 x 10-3 and decayed by a factor of 10 at the 20th and 30th epoch respectively. We employ Adam optimizer with learning rate of 5 x 10 for TT.
3.2 Preliminary Experiments
In Figure| we can observe that with the PES trick, the performance of classifiers is generally improved compared with that the traditional early stopping trick. In this section, we further carefully analyze the quality of extracted labels by examining them from three aspects, i.c., test accuracy,

---

Table Comparison with state-of-the methods without semi- cemi-supervised learning on CIFAR-10 and CIFAR-100. The mean and standard deviation computed over five runs are presented.
Symmetric
Pairflip
Instance
Dataset
Method
20%
50%
45%
20%
40%
CE
84.00.10.66
1..21
63.34.16.03
85.10.10.68
77.00
Co-teaching
87.16:t0.11
72.80=:0.45
70.11=1.16
86.54+0.11
80.98=0.39
Forward
85.63:00.52
77.92:10.66
60.15+1.97
85.29:10.38
74.72=3.24
CIFARIO
Joint Optim
89.7010.11
85.0011.117
82.63
89. 89.69:10.42
82.62.10.57
T-revision
89.63110.13
83,40110.55
77.0616.47
90.46.10.13
85.3713.36
DMI
88.18.1..66
78.28.10.48
17.60:14.556
89.14.10.36
84.78 1.97
CDR
89.72:10.38
8..6411.899
73.67:00.44
90.41110.34
8.30
Ours 92.38:10.40
87.45-10.35
8.,43
92.69:10.44
89.73=0.51
CE
51.43:10.58
37.6911.44
34.1012.04
52.19:11.45
42.26 1.29
Co-teaching
59.28:10.47
41.37:00.08
33.22:10.48
57.24:0.699
45.69=0099
Forward
57.75+0.37
44.66 1.01
27.88=0.80
58.76::0666
44.50=0.72
CO
Joint Optim
64.55.10.38
50.22.10.41
10.61
65.15.10.31
55.57
T-revision
65.40.11.07
50.24111.4
41.10.1..95
60.71.10.33
51.541.0.1
DMI
58.73:10.70
14.25::1.14
26.9010.45
58.05:10.20
47.36::0.68
CDR
66.52:+0.24
55.30++0.96
13.87=1135
67.33++0667
55.94=0.56
Ours
68.89:+0.45
58.90::2.72
57.18+1.44
70.49:10.79
65.68-1.41
Table 3: Comparison with state-of-the-art methods with semi-supervised learning on CIFAR-10 and CIFAR- 100 with symmetric label noise from different levels. Results with are token from 15 The mean and standard deviation are computed over three runs.
Dataset
CIFAR-10
CIFAR-
Methods Noise
Sym-20%
Sym-50%
Sym-80%
Sym-20%
Sym-50%
Sym-80%
CE
86.5
80.6.10.2
63.7.10.8
57.910.4
47.3
22.311.2
MixUp
93.2.10.3
88.2.10.3
73.3
69.510.2
57.1
34.1
M-correction*
94.0
92.0
86.8
73.9
66.1
48.2
DivideMix*
95.2
94.2
93.0
75.2
72.8
58.3
DivideMix
95.6=#0.1
94.6=0011
92.9+0.3
75.3+0011
72.7+0.66
56.4+00.3
ELR+
94.910.22
93.6.10.1
90.4.10.2
75.5
71.0.10.2
50,410.8
Ours (Semi)
95.9=10.1
95.1-10.2
93.1+10.2
77.4+00.3
74.3:10.6
61.6-+0.6
label precision, and label recall. Here, label precision indicates the ratio of the number of extracted confident examples with correct labels in the total confident example set, and label recall represents the ratio of the number of confident examples with correct labels among the total correctly labeled examples. Specifically, we train a neural network on CIFAR-10 10 with different kinds and levels of label noise for 25 epochs respectively and report the performance for each case before and after the proposed PES is applied.
Results in Table cllerly show that, compared with the traditional early stopping, PES help to obtain higher accuracies, precisions, and recalls for most cases. For instance-dependent label noise, PES can achieve higher recall values with comparable label precision values. Note that models with high recall values can help to collect more confident examples, which is critical for learning with confident examples and sumi-super learning. Therefore, by enhancing the performance of the initial model, PES can help to improve the final classification performance in all cases, which is also verified by the experiments in Section 3.3
3.3 Classification Accuracy Evaluation
Synthetic datasets. We first verify the effectiveness of our proposed method without semi supervisd learning techniques on two synthetic datasets: CIFAR- and CIFAR-100. For both of these two datasets, we leave 10% of data with noisy labels as noisy validation set. Results are presented in Table which shows that our proposed method can consistently outperform all other baselines across various settings by a large margin.
7

---

Table Comparison with satte-o--the-are methods with semi comi-supervised learning on CIFAR- 10 and CIFAR- 100 with mnstacceddepender and pairflip label noise from different levels. The mean and standard deviation are computed over three runs.
Dataset
CIFAR-10
CIFAR-100
Methods / Noise
Inst-20%
Inst-40%
Pair
Inst-20%
Inst-40%
Pair-45%
CE
87.
78.9=0.7
74.9=1.7
56.8=0.4
48.2=0.5
38.5+:0.6
MixUp
93.3+0.2
87.640.5
82.4:1.0
67.1+0.1
55.0=0.1
44.2:10.5
DivideMix
95.510.1
94.5:10.2
85,61117
75.2:0.2
70.9
48.2.11.0
ELR+
94.9+0.1
94.3++022
86.1#1.2
75.8=0.1
74.3=
65.3+1.3
Ours (Semi)
95.910.11
95.3.10.1
94.5.10.3
7..
76.1 0.4
73.6.11.7
Table 5: Compassion with state-of-the-art methods Clothing- Results of baseline methods are taken from the original papers. ours represent the results obtained by PES with a single network and ours indicate the results obtained by PES with ensemble model.
CE
Forward
Joint-Optim
DMI
T-revision
DivideMix
ELR+*
Ours
Ours*
69.21
69.84
72.16
72.46
74.18
74.76
74.81
74.64
74.99
GFARRIC JC Fartlip 45%
Toxk scuract
CIFAR-10 Painfip 45%
CIARR3C 50%
CIFAR, 10 50%
20
CIMR-IC
CIFAI-1CO
50%
CIFAR- 100 Sym- 50%
30
10
50
0
7
10
Epochs
Epocht
(a) T2
(b) T3
Figure 3: Sensitivity analysis for different training iteration numbers: T2 and T3-
Table and Table present the mean accuracy and standard deviation for our method and all baselines on CIFAR-10 and CIFAR-100, respectively. From the results, we can get that the proposed method can outperform all baselines in all cases. For pairflip label noise, the advantages of our proposed method become more apparent, and it significantly outperforms state-of-thh-an methods by 8% on both CIFAR-10 and CIFAR-100, These empirical results support our proposal that PÉS can improve the quality of selected confident examples, which helps improve performance and reduce the variance of the final classifier.
Real-world dataset. We evaluate the performance of the proposed method on real-world dataset with Clothing- 1M (32] and select methods such as CE, Forward, Joint- Shint-Optim, DMI, and T-revision, which use single network, and also methods such as DivideMix and ELR++ which adopt ensemble model with two different networks, as baselines. We also report the results for the proposed PES with single network as ours and the results for PES, which ensembles overall results are reported in Table 55, from which we can observe that the proposed PES with a single network can outperform all baselines using a single network. And with an ensemble model, which contains two different networks, our method can outperform all the adopted baselines. These results clearly demonstrate that, by improving the performance of the initial classification network, our method is more flexible to handle such real-world noise problems.
networks, ours". The
3.4 Sensitivity Analysis
In this section, we investigate the hyper-parameter sensitivity for the training iteration number T2 and T3. respectively. We firstly analyze the training epoch number for the second DNN part by varying Th from the range of [0, 10). The results are illustrated in Figure 3al from which can find that, with
8

---

Table Training time comparison for baselines on CIFAR-10 with 50% Symmetric label noise.
CF
Co-teaching
CDR
T-revision
F
DivideMix
Ours
Ours (Semi)
0.9h
1.5h
3.0h
3.5h 2.2h
5.5h
1.0h
3.
the increasing of T2, the performance of PES first increase and then decrease in all the cases except for 45% Pairflip noise on the CIFAR- 10 dataset. While the model achieves the best performance with T3 7 for all types of noisy labels. Then DNN part by varying T3 from the range of [o, 10]. The results are shown in Figure 3b) Although the performance variance for different T3 is smaller than that for T2, we can still observe that the best performance can be obtained when T33 is set as 5. More importantly, from these two figures, we can get that both T2 and T3 are robust to the different types of noisy labels.
fix T2 as 7, and analyze the impact of the third
3.5 Training Time Comparison
In this section, we compare the training time of our method and other sfate-of-the-art baselines. All the experiments are conducted on server with single Nvidia V100 GPU. The training times for all the methods are reported in 6 from which we can get that our algorithm with cross-entropy loss achieves the fastest speed across all baselines, only about 1 hour. Our method combining with MixMatch 4 is also fast, only little more than half of the training time of DivideMix. The time of ELR++ []6 shows superior, but ELR+ trains the network with fewer epochs, with 200 epochs compared with ours for 300 epochs.
Related work
Learning with noisy data has been well studied 17 6 21 27
Current works can be mainly categorized into two groups: model-based and model-free methods. this section, we briefly review some closely related works.
20
The first type models the relationship between clean labels and noisy labels by estimating the noise transition matrix and build a loss function to correct the loss 24 algorithms for estimating the noise rates and loss correction techniques together and introduces two alternative procedures for loss correction. It also proves that both of the two procedures enjoy formal robustness guarantees w.tf. the clean data distribution. DMI (34 proposes an citormmttionthhoreeee loss function, which utilizes Shannon's 's mutual information and is robustness to different kinds of label noise. T-revision 33 estimates the noise transition matrix without anchor points by adding fine-tuned slack variables. Although these methods have made certain progress, they are usually fragile to estimate the noise transition matrix for heavy noisy data and are also hard to handle large number of classes. Therefore, in this paper, we mainly focus on the model-free methods.
30 35
28
24
first combines
The second strand mainly counteracts noisy labels by exploiting the memorization effect that deep networks tend to first memorize and fit majority (clean) patterns and then overfit minority (noisy) patterns 22. To exploit this property, Co-teaching [9] employs two networks with different initializa- tion and uses small loss to select confident examples. M-correction [T] uses two Gaussian Mixture Models to identify confident examples, instead of using networks themselves. DivideMix [15] extends Co-teaching [9] and employs two Beta Mixture Model to select confident examples. MixMatch [ then adopted to leverage unconfident examples with a semi- supersseee learning framework. All the above methods exploit the memorization effect by considering the adopted network as whole. Recently, [1] shows that networks training with noisy labels can produce good representations, the structure of networks suits the targeted tasks. Our method further explains that noisy labels have different impacts for different layers DNN. And latter layers will receive earlier and more severe impact than their former counterparts. Therefore, by considering DNN as composition of several layers and training different layers with different epochs, our method is able to better exploit the memorization effect and achieve superior performance.
5 Conclusion
In this work, we provide progressive early stopping (PES) method to better exploit the memorization effect of deep neural networks (DNN) for noisy-label learning. We first find that the impact of noisy

---

labels for former layers in DNN is much less and later than that for latter DNN layers, and then build upon this insight propose the PES method, which separates DNN into different parts and progressively train each part to counteract the different impacts of noisy labels for different DNN layers. To show that PES can boost the performance of state-of-the-art methods, we conduct extensive experiments across multiple synthetic and real-world noisy datasets and demonstrate that the proposed PES can help to obtain substantial performance improvements compared to current state-of-the-ar baselines. The main limitation of our method lies in that, by splitting DNN into different parts, PES introduces several additional typer-parameters that need to be tuned carefully. In the future, we will extend the work in the following aspects. First, will study other mechanisms that distinguishing desired and undesired memorization rather than early stopping, e.g., the gradient ascent trick [8]. Second, interested in combining PES with interesting ideas from semi -uperssee learning and unsupervised learning.
Acknowledgments and Disclosure of Funding
YB was partially supported by Agriculture Consultant and Smart Management. BH was supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202 and HKBU CSD Departmental Incentive Grant. YY was partially supported by Key Research and Development Program of Shaanxi (ProgramNo. 202IZDLGY01-00
GN was supported by JST AIP Acceleration Research Grant Number JPMICR20U3, Japan. TL was partially supported by Australian Research Council Projects 19000131 and IC-190100031.
10

---

References
[1] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness, Unsupervised label noise modeling and loss correction. In ICML, pages 312-321, 2019.
[2] Devansh Arpit, Stanislaw Jastrzehski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. closer look memorization in deep networks, In ICMI, pages 233-242, 2017.
[3] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learinng practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 166332):1584-115844 2019.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, pages 5050-5060, 2019.
[5] Youngchul Cha and Junghoo Cho. Social-network analysis using topic models. In SIGIR, pages 565-574, 2012.
[6] Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
[7] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. The MIT Press, 2016.
[8] Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor W. Tsang, and Masashi Sugiyama. SIGUA: forgetting may make learning with noisy labels more robust. In ICMI, pages 4006-4016, 2020.
[9] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training deep neural networks with extremely noisy labels. In NeurlPS, pages 8527-8537, 2018.
[10] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[11] Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, Masashi Sugiyama. Do we need zero training loss after achieving zero training error? In ICMI,, pages 4604-4614, 2020.
[12] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. Technical report, 2009.
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS. pages 1097-1105, 2012.
[14] Jingling Li, Mozhi Zhang, Keyulu Xu, John P. Dickerson, and Jimmy Ba. Noisy labels can induce good representatious. arXiv preprint arXiv:2012.12896 2020.
[15] Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels semi-supervised learning. In ICLR, 2020.
[16] Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Fernandez-Grane Granda. Early-learning regular- ization prevents memorization of noisy labels. In NeurIPS, pages 20331-20342, 2020.
(17] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3)4447-61, 2016.
[18] Wei Liu, Yu-Gang Jiang, Jiebo Luo, and Shih-Fu Chang. Noise resistant graph ranking for improved web image search. In CVPR, pages 849-856, 2011.
[19] Tlya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In JCLR, 2017.
[20] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey, Normal- ized loss functions for deep learning with noisy labels. In /CML, pages 6543-6553, 2020.
[21] Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi N. R. Wijewickrema, and James Bailey. Dimensionality-drn learning with noisy labels. In ICML, pages 3361-330 1361- 2018.
[22] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. In ICLR, 2020.
11

---

[23] Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi- Thi-PPhung-NNung Ngo. Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. SELF: learning to filter noisy labels with self-ensembling. In ICLR. 2020.
[24] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. CVPR, pages 2233-2241, 2017.
[25] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. CVPR. pages 5552 5560, 2018.
[26] Yisen Wang. Weiyang Liu,, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. CVPR, pages 8688-8696, 2018
[27] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. ICCV, pages 222-330, 2019.
[28] Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In ICML, pages 11285-11295, 2021.
[29] Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong. Nannan Wang, Zongyuan Ge, and Chang. Robust early-leamning Hindering the memorization of noisy labels. In ICLR, 2021.
[30] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NewrIPS, pages 7597-76 7610, 2020.
[31] Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, pages 6835-6846, 2019.
[32] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pages 2691-2699, 2015.
[33] Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In NeurlPS, pages 6256-6268, 2020.
[34] Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. I__mi:: A novel information-theoretie loss function 2019.
for training deep nets robust to label noise. In NewrIPS, pages
[35] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual T: reducing estimation error for transition matrix in label-noise learning. In NeurlPS, pages 7260-7271, 2020.
[36] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2016.
[37] Hongyi Zhang. Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018
12


Reviews:
[{"review_id": "umyThMd0-za", "reviewer": "Reviewer_sdsb", "summary": "The paper tackles the problem of classification under a noisy data setting. The authors make the observation that noisy labels mainly affect the latter layers of deep neural networks, and propose a method to address this, termed progressive early stopping (PES). A network is trained by freezing the parameters layers of layers 1,2,...,L (numbered from input to output) at predefined epochs E1, E2...EL. After this step, the authors use the network thus trained to identify “confident” training examples (those for which the predicted label matches the true label). Then the paper proposes two ways of training a final model: (1) in a fully supervised fashion, using only the confident examples (while accounting for class imbalance), or (2) semi-supervised, where the examples that are not confident are considered unlabeled. The approach is tested on noisy versions of CIFAR10 and CIFAR100 and the Clothing-1M dataset, and compared with competing supervised and semi-supervised methods.  \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found this paper interesting and well written. It proposes some interesting ideas and provides thorough comparisons with competing methods. The authors also addressed several issues that could be considered weaknesses such as training run time. On the negative side, I did not find the proposed “progressive early stopping” (PES) strategy very original, as it bears a close resemblance to greedy layer-wise training [1,2], and the authors do not discuss this similarity. However, even if this part is not novel, the authors do use it in what I think is a novel way to handle noisy labels. One aspect that I am a bit unclear about is whether the PES part is necessary for the second part of the paper starting from section 2.2.  -- what if you skip that part and compute the confidences in (5) using a normally trained baseline and then apply the proposed semi-supervised approach? Aside from this, the experimental analysis looks pretty good.\n\nI will discuss this in more detail below.\n\n**Originality:**  As mentioned above, the PES component is very similar in spirit to [1] and [2] (there are probably other papers too, but these are pretty popular), and while the methods may not be identical, without a comparison in the paper is hard to understand what is new. Using the predicted noisy labels as unlabeled data in semi-supervised learning is also not new, and it was proposed in DivideMix (cited as [15] by the authors). So in fact the combination of PES and the idea in DivideMix is what in fact makes this method work so well. \n\n[1] Bengio, Yoshua, et al. \"Greedy layer-wise training of deep networks.\" Advances in neural information processing systems. 2007.\n\n[2] Belilovsky, Eugene et al. “Greedy Layerwise Learning Can Scale to ImageNet.” ICML 2019\n\n**Quality**: I really appreciated the experimental analysis in this paper, which considers many baselines, does a sensitivity analysis on the hyperparameters and reports run times. The paper contains a lot of details about hyperparameter configurations, although it’s not clear to me how these were set, especially the number of epochs T1, T2… which also differ per dataset. Could the authors please clarify this?\n\n**Significance**: The tackled problem is definitely of considerable practical importance, and the authors report significant improvements over existing methods. I am not though familiar with the most recent approaches for noisy labels to assess if the competing methods are state-of-the-art; perhaps the other reviewers can address this.\nThe method does have some limitations that hinder its use in practice, such as longer training time especially for the semi-supervised version, and the fact that there are multiple stages to it. However, this may be a trade-off that some users may be willing to pay for the improved performance.\n\n**Some clarification questions:**\n1. Line 161: how can we make sure we have enough labeled samples? For a bad classifier with low accuracy, many of the samples will be put in the unlabeled set, to the point where you may have too little data to train a good classifier using semi-supervised learning. What is a minimum accuracy necessary for the baseline classifier (the one used to compute the \\hat y) in order for this method to have enough labeled samples to perform better than the baseline itself?\n2. How were the hyper-parameters T1, T2… set in the various experiments? What about the rest of the training hyperparameters, such as learning rate, regularization etc.? \n3. Eq. (5) Do the authors use only 2 random augmentations? Are two enough to get a reliable estimate? How sensitive is it to how much you alter the original image? \n4. For methods “CE” in Tables 2 and 3? Is it standard training with cross-entropy on the noisy dataset? If so, how do you make this semi-supervised.\n5. In case this is not covered by the previous question, what happens if you compute \\hat y_i with early stopping in eq. (5), and then apply your semi-supervised approach on this? Is this reported in the table?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the major limitations that I could think of are the running time, number of hyper parameters and sensitivity to hyper-parameters, which the authors have addressed. The other potential issue, I have mentioned in question 3 above.\n\nI don’t think societal impact is discussed, but I don’t think this is a concern for this type of method and evaluation setting.\n", "review_text": "", "extra_scores": {"ethical_concerns": "Nothing I can think of.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "RppfOiyb6RS", "reviewer": "Reviewer_E13N", "summary": "This paper presents an interesting new method namely progressively early stopping to better exploit the memorization effect of DNN for learning with noisy labels. It first analyzes different effects of noisy labels to different DNN layers and finds that latter layers are much more sensitive to noisy labels. Based on this understanding, the authors then propose a progressively early stopping method, which progressively stops the training of different layers to counteract the influence of noisy labels. By combining some existing semi-supervised learning techniques, the proposed method achieves SoTA performance on several popular image classification benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n-\tStrong and clear motivation. The authors first analyze the influence of noisy labels on different DNN layers during different training iterations, which shows that latter layers are much more sensitive to noisy labels. Then, with this understanding, the authors propose to train different layers with different early stopping iterations to better exploit the memorization effect. The authors are the first to analyze the memorization effect for different DNN layers. \n-\tSolid technical contribution. The authors first split a DNN into different parts and optimize each part with different training iterations. Since latter layers are verified to be more sensitive to label noise, by applying smaller training iterations, the obtained model can be less overfitted to noisy labels. To exploit this phenomenon, the authors propose a new method dubbed progressively early stopping, which is also demonstrated to be easily combined with other semi-supervised learning techniques. \n-\tState of the art experimental performance. Their simple approach shows promising results and yields new SoTA performance on both synthetic and real-world datasets.\n-\tConvincing ablation experiments. The authors adopt a base CNN model and compare traditional early stopping with the proposed progressively early stopping. The results confirm that progressively early stopping is superior to the alternative. \n-\tPresentation is clear and easy to follow. \n\nCons: \n-\tThe authors claim that different DNN layers have different sensitivities to noisy labels. Fig.1 demonstrates this point by splitting the DNN into three parts, while in the experiments, the authors alternatively split the adopted DNN into two parts. According to the paper, how the network is split seems important to the proposed method but is not introduced in detail. \n-\tIn table 1, according to label precision, PES cannot surpass traditional early stopping, which seems not consistent with the claim of the method. And this issue is not explained in detail. Therefore, a better explanation is expected.\n-\tIn Table 2, The proposed method seems always performs even better than other baselines when the noise is heavier under all types of noises. Authors should elaborate on this phenomenon to better understand the PES method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concern was found. This paper should have positive social impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0e0TMaYylzN", "reviewer": "Reviewer_NpAD", "summary": "In this manuscript, the authors discussed the sensitivity of deeper and shallower layers in a DNN and improved the conventional early stop of the whole model to a layer-wise progressive early stop (PES) for label-noise learning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is interesting. The authors improved the conventional early stop method in the label noise learning task. However, I’m confused with the task setting. Furthermore, the proposed method is too naive and the authors did not provide key part of the proposed method (see the third point in Limitations And Societal Impact). The related work was adequately cited. The manuscript is well-written. The experimental results seem good on two small-scale datasets, but the authors did not evaluate their method on ImageNet, which is a common setting.\n\n1. In figure 1, the shallower layers are much better than the final layer with much higher accuracies in each case. How to understand this statement? Meanwhile, compared with figure 2, the test accuracies of the 9th layer are much higher than those of the PSE. Should we say that we can directly use the outputs of the 9th layer of the ResNet18, rather than the final outputs? This determines the significance of the label noise learning task.\n2. As in figure 1, the shallower layers converge later than the deeper layers, which means the shallower ones should train in more epochs. However, the authors first froze shallower layers in eq (4) and algorithm 1. They are the opposite. How to explain this?\n3. The authors did not discuss how to select T_1 to T_L in section 2.1, which is truly important. This makes the proposed method naive and hard to follow.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-hgu3wKT8eL", "reviewer": "Reviewer_9vjN", "summary": "The paper introduces a novel training scheme (progressive early stopping, PES) aimed at improving the performance of convolutional networks for image classification in presence of noisy labels. The strategy is motivated by the observation that the layers close to the output tend to overfit noisy targets more than early layers. PES thus proposes to differentiate the number of training epochs for different groups of layers, training early groups for more epochs than final ones. In detail PES divides the networks in N blocks and trains for T_1 epochs the whole network. It then freezes the learned weights of the first block and  reinitializes  and trains the layers of the remaining N-1 blocks for T_2 epochs. Crucially, T_2 < T_1. The same policy is then progressively repeated for all the remaining N-2 blocks with T_n <T_n-1 for n in  {1, …, N}. \nPES is also combined with a semi-supervised scheme (MixMatch) to reach SOTA performances on CIFAR10 and CIFAR100 datasets.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality:\n1)The training policy described in lines 192-196 is quite unclear. Here the number of epochs is set to 200 and the learning rate, starting from 0.1, is decreased by a factor 10 at epochs 100 and 150. Then T1 is set to 20/35, T2 to 7, T3 to 5 and the learning rate is set to 10**-4 at T2 and T3. What happens after T1+T2+T3 epochs? The value of the learning rate after the first T1 epochs is 0.1 or 10**-4?\n\n2) What is the motivation for introducing two different optimizers, i.e. SGD with a step learning rate scheduler and Adam in the same training run (see lines 192-209)?\n\n3) I have a concern regarding a comparison between the results reported in Table2 and the sensitivity analysis shown in Fig.3. If I understood correctly, PES with T2=0 and T3=0 should be equivalent to the standard cross entropy training since the whole network is trained for T1 epoch with noisy labels. From Table2  we can see that the accuracy gap between CE and PES for, say, cifar100 trained with symmetric noise 50% is between 16/21%. From Figure 3 instead we see that the accuracy gap between the optimal T2/T3 and T2=T3=0 is well below 10%.  A similar concern regards the other perturbations shown in the sensitivity analysis.\nWhy is this happening? \n\n4) As correctly recognized by the author(s) the approach requires fixing several metaparameters, in particular the number of epochs for each sub-block optimization. But why is this necessary? The common practice in early stop is finishing the optimization close to the maximum of the  accuracy estimated on a small dataset not used for the optimization (and different from the validation set). Why cannot this procedure be used in PES?\n\n5) The code used to train the network should be publicly shared.\n\nClarity:\n\nThe paper could be improved in the following parts:\n\nWhen the authors describe Fig. 1 they should say on which dataset they train the network used to produce the profiles, (only looking at table 3 of section 3.2 it is possible to guess that the dataset is Cifar10), and for how many epochs they trained the first block of the network. Moreover the meaning of ‘clean data’ line 53 is not very clear. Indeed if one compares the accuracy of a classifier trained only on a noisy  dataset with that obtained with the same classifier on noisy and noiseless data it seems quite natural that in the second case the accuracy improves;\n\nLine 172: it is not clear what ‘adiacent class’ means;\n\nThere are a few typos and grammar errors.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are rather briefly but rather clearly discussed at the end of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a variant of early stopping for learning with noisy labels. The method starts with the hypothesis that early layers are less prone to be adversely affected by label noise than later layers. Authors empirically validate this hypothesis and then propose a progressive layerwise training method for tackling label noise. This is then used to identify noisy examples which are treated as unlabeled data and existing semi-supervised methods are used to get further empirical gains. Reviewers have expressed some concerns on the heuristic components in the method (use of Adam for last layers vs SGD for earlier layers), several metaparameters (the number of epochs for each subblock), and the increased training time of the method due to progressive/multi-stage training, but they have found the motivation behind the paper convincing and have been overall positive about the paper. I believe the paper is above the acceptance threshold for the publication. 
Decision: Accept (Poster)