Prestopping: How Does Early Stopping Help Generalization Against Label Noise?
Authors: Hwanjun Song
Minseok Kim
Dongmin Park
Jae-Gil Lee
Venue:  (2020)
Keywords: noisy label
label noise
robustness
deep learning
early stopping

Abstract: Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we resume training the early stopped network using a "maximal safe set," which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point. Putting them all together, our novel two-phase training method, called Prestopping, realizes noise-free training under any type of label noise for practical use. Extensive experiments using four image benchmark data sets verify that our method significantly outperforms four state-of-the-art methods in test error by 0.4–8.2 percent points under existence of real-world noise.

Under review as a conference paper at ICLR 2020
PRESTOPPING: How DOES EARLY STOPPING HELP GENERALIZATION AGAINST LABEL NOISE?
Anonymous authors Paper under double-bling review
ABSTRACT
Noisy labels are very common in real-wort training data, which lead poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we resume training the early stopped network using a "maximal safe set," which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point. Putting them all together, our novel two-phase training method, called Prestopping. realizes noise-free training under any rype of label noise for practical use. Extensive experiments using four image benchmark data sets verify that our method significantly outperforms four state-of-the-an methods in test error by 0.4-8.2 percent points under existence of real-world noise.
INTRODUCTION
By virtue of massive labeled data, deep neural networks (DNNs) have achieved a remarkable success in numerous machine learning tasks, such image classification (Krizhevsky al. 2012) and object detection (Redmon et al. 2016). However, owing to their high capacity to memorize any label noise, the generalization performance of DNNs drastically falls down when noisy labels are contained the training data (Jiang et al. 2018 Han et al. 2018 Song et all 2019). In particular, /Zhang et al. (2017) have shown that standard convolutional neural network (CNN) can easily fit the entire training data with any ratio of noisy labels and eventually leads to very poor generalization on the test data. Thus, is challenging to train DNN robustly even when noisy labels exist in the training data.
A popular approach to dealing with noisy labels is "sample selection" that selects true-labeled samples from the noisy training data (Jiang et al., 2018 Ren et al.) 2018 Han et al. 2018 Yu et al. 2019: Song et al. 2019). Here, (1-7) x 100% of small-loss training samples are treated as true-labeled ones and then used to update a DNN robustly, where t E [0, 1] is a noise rate. This loss-based separation is well known to be justified by the memorization effect (Arpit et al. 2017) that DNNs tend to learn easy patterns first and then gradually memorize all samples. In practice, Han et al. (208)) empirically proved that training such small-loss samples yields much better generalization performance under artificial noise scenarios.
Despite its great success, [Snng et al. (2019) have recently argued that the performance of the loss- based separation becomes considerably worse depending on the type of label noise. For instance,
1.0
True labele sumple
0.8 True-labele sample
1.0 True- Etue-labeled sumple
0.8
False labeled semple
0.3 label-d sample
0.8
False- Jabeled sample
0.4
(1-r) 100% small-loss
0.2
(1-7) xx 00% small Toss
0.1
(1-r) x 100% small-ipts
0.2
0.2
0.0
0.0
6
6
0
2
22
-1
2
3
Loss (loge scale)
(log-
Loss (log- scale)
(a) Symmetric Noise.
(b) Pair Noise.
(c) Real World Noise.
Figure 1: Loss distributions at the training accuracy 50% using DenseNet (I=40, k=12): (a) and (b) show those on CIFAR-100 100 with two types of synthetic noises of 40%, where "symmetric noise" flips a true label into other labels with equal probability, and "pair noise" flips a true label into a specific false label; (c) shows those FOOD-101N (Lee al. 2018) with real-world noise of 18.4%.

---

Under review as a conference paper at ICLR 2020
True labeled sample
Faise labeled semple
Default
Prestopping
Manscstatioon Minoiizttion Rato
100%
Menotzzttoon Hebo
100%
60%
Emoe prone Pesiod
Symmetric Noise
75%
75%
Mis trawkiddded amplle
Tot Brer
50%
50%
40%
25%
Early Stop
0%
0%
30%
40 Epocts
120
80
120
10
80
120
Epochs
Epoctes
100%
Memortatton toto
100%
70%
Error- prone Penod
75%
73%
60%
Pair Noise
truwdlddeed anyple
50%
50%
50%
Geoeriltation Iepersormen
Toa
29%
Early Stop
40%
0%
30%
0
00
80
120
80
120
40
8D
120
Epocts
Epochs
Epocta
(a) Default.
(b) Prestopping.
(c) Test Error Convergence.
Figure 2: Key idea of Prestopping: (a) and (b) show how many true-labeled and false-labeled samples memorized when training DenseNet (L=40, k=12gpn CIFAR- 100 with two types of synthetic noises of 40%. "Default" is a standard training method, and "Prestopping" is our proposed one; (c) contrasts the convergence of test error between the two methods.
in symmetric noise (Figure I))) the loss-based approach well separates true-labeled samples from labse-labeled ones because many true-labeled ones exhibit smaller loss than false-labeled ones. On the other hand, in pair and real-world noises (Figures [b) and [[c), many false-labeled samples are misclassified as true-labeled ones because the two distributions overlap closely; this overlap confirms that the loss-based separation still accumulates severe label noise from many misclassified
cases, especially in real-world noise or pair noise which is regarded as more realistic than symmetric noise (Ren et al. 2018: Yu et al. 2019). This limitation definitely calls for new approach that supports any type of label noise for practical use.
In this regard, as shown in Figure aaa, a). we thoroughly investigated the memorization effect of a DNN on the two types of noises and found two interesting properties as follows:
noise type affects the memorization rate for false-labeled samples: The memorization rate for false-labeled samples is faster with pair noise than with symmetric noise. That is, the red portion in Figure 2 starts to appear carlier in pair noise than in symmetric noise. This observation supports the significant overlap of true-labeled and false-labeled samples in Figure [b). Thus, the bass- separation performs well only if the false-labeled samples are scarcely learned at an early stage of training, as in symmetric noise.
There is a period where the network accumulates the label noise severely: Regardless of the noise type, the memorization of false-labeled samples significantly increases at late stage of training. That is, the red portion in Figure a) increases rapidly after the dashed line, in which we call the error-prone period. (See Section 3.2. for the details of estimating the error-pront period). We note that the training in that period brings no benefit. The generalization performance of "Default" deteriorates sharply, as shown in Figure
Based on these findings, we contend that eliminating this error-prone period should make a profound impact on robust optimization.
In this paper, we propose a novel approach, called Prestopping, that achieves noise-free training based on the early stopping mechanism. Because there is no benefit from the error-prone period, Prestopping early stops training before that period begins. This early stopping effectively prevents network from overfitting to false-labeled samples, and the samples memorized until that point are added to a maximal safe set because they are true-labeled (i.e., blue in Figure a)) with high precision. Then, Prestopping resumes training the early stopped network only using the maximal safe set in support of noise-free training. Notably, our proposed merger of "carly stopping" and "learning from the maximal safe set" indeed climinates the pronee period from the training process, as shown in Figure(b) As a result, the generalization performance of a DNN remarkably improves in both noise types, shown in Figure c)..
'The learning rate, as usual, was decayed at 50% and 75% of the total numher of training epochs.
2

---

Under review as a conference paper at ICLR 2020
To validate the superiority of Prestopping, DenseNet (Huang et al. 2017) and VGG-19 19 Simonyan & Zisserman 2015) were trained on both simulated and real-world noisy data sets, including CIFAR-10 10, CIFAR- IO0, ANIMAL-1ON, and Food-101N. Compared with four state-of-the-ar methods, Prestopping significantly improved test error by up to 18. 18.1pp in a wide range of noise rates.
2 RELATED WORK
Numerous studies have been conducted to address the problem of learning from noisy labels. A typical method is using "loss correction" that estimates the label transition matrix and corrects the loss of the samples in the mini-batch. Bootstrap (Reed et al., 2015) updates the network based on their own reconstruction-baasd objective with the notion of perceptual consistency. F-correction (Patrini et al. 2017) reweights the forward or backward loss of the training samples based on the label transition matrix estimated by pre-trained normal network. R3: D2L (Ma et al. 2018) employs simple measure called local intrinsic dimensionality and then uses it to modity the forward loss in order reduce the effects of false-labeled samples in learning. Active Bias (Chang et al.) 2017) heuristically evaluates uncertain samples with high prediction variances and then gives higher weights to their backward losses. Ren et al. (2018) include small clean validation data into the training data and re-weight the backward loss of the mini-batch samples such that the updated gradient minimizes the loss of those validation data. However, this family of methods is known to accumulate severe noise from the false correction. especially when the number of classes or the number of lalse-labeled samples is large (Han et al. 2018 Yu et al. (019)).
To be free from the false correction, many recent researches have adopted "sample selection" that trains the network on selected samples. These methods attempt to select true-labeled samples from the noisy training data for use in updating the network. Decouple (Malach & maintains two networks simultancously and updates the models only using the samples that have different label predictions from these two networks. 8d: Wang et al. (0018) proposed an iterative learning framework that learns deep discriminative features from well-classifies noisy samples based on the local outlier factor algorithm (Breunig al., 2000 MentorNer (Tiang al., 2018) introduces a collaborative learning paradigm that a pre-trained mentor network guides the training of a student network. Based on the small-loss criteria, the mentor network provides the student network with the samples whose label is probably correct. Co-teaching (Han et al. 2018)) and Co-teaching+ (Yu et al., 2019) also maintain two networks, but each network selects a certain number of small-loss samples and feeds them to its peer network for further training. Compared with Co-teaching. Co-teaching+ further employs the disagreement strategy of Decouple. 81: ITLM (Shen & Sanghavi 2019) iteratively minimizes the trimmed loss by alternating between selecting fraction of small-loss samples at current moment and retraining the network using them. INCV (Chen et al.) 2019) randomly divides the noisy training data and then utilizes cross-validation to classity true-labelen led samples with removing large-loss samples at each iteration. However, their general philosophy of selecting small-loss samples works well only in some cases such as symmetric noise. RA: Differently to this family, exploit the maximal safe set initially derived from the memorized samples at the early stop point.
(017)
Most recently, a hybrid of "loss correction" and "sample selection" approaches was proposed by Song et al. (2019). Their algorithm called SELFIE trains the network on selectively refurbished false-labeled samples together with small-loss samples. SELFIE not only minimizes the number of falsely corrected samples but also exploits full exploration of the entire training data. Its component for sample selection can be replaced by our method to further improve the performance.
RA: For the completeness of the survey, we mention the work that provides an empirical or a theoretical analysis on why early stopping helps learn with the label noise. Oymak et al. (2019) and [Hendrycks
ct al. (2019) have argued that early stopping is a suitable strategy because the network eventually begins to memorize all noisy samples if is trained too long. [Li et al. al. 2019 have theoretically proved that the network memorizes false- samples at later stage of training, and thus claimed that the early stopped network is fairly robust to the label noise. However, they did not mention how to take advantage of the early stopped network for further training. Please note that Prestopping adopts early stopping to derive a seed for the maximal safe set, which is exploited to achieve noise-free training during the remaining learning period. Thus, our novelty lies in the "merger" of carly stopping and learning from the maximal safe set.
A pp is the abbreviation of a percentage point.
3

---

Under review as a conference paper at ICLR 2020
3 ROBUST TRAINING VIA Prestopping
3.1 PRELIMINARIES
A k-class classification problem requires the training data D = {2 yi / where x; is a sample and vf E (1,2..... }} is its true label. Following the label noise scenario, let's consider the noisy training data D {Ix; Dr ) where i E {1, .,.: k} is a noisy label which may be true. Then, in conventional training, when mini-batch B, { Dt )i=1 consists of b samples randomly drawn from the noisy training data D at time i, the network parameter O, is updated in the descent direction of the expected loss on the mini-batch Be as in Eq. ① where is a learning rate and L is loss function.
02+1 = 02 -aooa V
Σ FEB.
L(),
(1)
B
As for the notion of network memorization, sample i is defined be memorized by network if the majority of its recent predictions at time coincide with the given label, as in Definition 3.1
Definition 3.1. (Memorized Sample) Let ů (xx(0t)) be the predicted label of sample x at time t and (g) = (ūe, Dta..... St, be the history of the sample æ that stores the predicted labels of the recent times, where & is neural network. Next, P(y|æ, 3 t; q) is formulated such that it provides the probability of the label y E {1, k) k} estimated as the label of the sample x based on H as in Eq. (2). where [-] is the Iverson bracke
Egeny(a) 10 - y]
p(ylx,t:g)=
(2)
[H2(4)
Then, the sample x with its noisy label ÿ is memorized sample of the network with the parameter 0 at time if the condition in Eq. holds.
argmax, P(yla,, =
(3)
3.2 TWo MAIN COMPONENTS AND PHASES
The key idea of Prestopping is learning from a maximal safe set with an early stopped network. Thus, the two components of "early stopping" and "learning from the maximal safe set" respectively raise the questions about (Q1) when is the best point to early stop the training process? and (Q2) what the maximal safe set to enable noise-free training during the remaining period?
3.2.1 QUESTION 1: BEST POINT TO EARLY STOP
It is desirable stop the training process at the point when the network (i) not only accumulates litle noise from the false-labeled samples, (ii) but also acquires sufficient information from the true-labeled ones. Intuitively speaking, as indicated by the dashed line in Figure 3] a). the best stop point the moment when label precision and label recall in Definition 3.2 cross with each other because it is the best trade-off between the two metrics. The period beyond this point is what we call the error-prone period because label precision starts decreasing rapidly.
the ground truth y" of noisy label ÿ is known, the best stop point can be easily calculated by these metrics.
Definition 3.2. (Label Metrics) Let M, C D be set of memorized samples at time t. Then, label
precision (LP) and recall (LR) (Han et al. 2018) are formulated as Eq. 44).
K(x, Ø) M, : 9 v" H
1( () Me ū '²Hl
LP
LR
(4)
Mel
[[(x, () E D
H
However, it is not straightforward to find the exact best stop point without the ground-truit -truth labels. Hence, we present two practical heuristics of approximating the best stop point with minimal supervision. These two heuristics require either a small clean validation set or a noise rate r for the minimal supervision, where they widely regarded as available in many studies (Veit et al. 2017 Ren et al. 20188 Han et al. [2018] 3 Yu et al. 2019 Song et al. 2019).
'The Iverson bracket returns if P true; 0 otherwise.
4

---

Under review as a conference paper at ICLR 2020
-0- Labd Pusision
- Lakel Racall
Best Point
Appron Stop Point
"Symmetric Noise"
"Puir Noise"
"Symmetric Noise"
"Phir Nuise"
1.00
1.00
80%
80%
ane Lx
0.75
0.75
D
Validation Errur
60%
Man
00%
0.50
0.25
Best Paint
Best Point
40%
Approx Stops
40%
Approxx Strpl
Point.
0.00
0.00
20%
Point
20%
40
80
120
40
80
120
10
80
120
Epodas
Epochs
Epochs
Epochs
(a) Ideal Method.
(b) Validation Heuristic.
Figure 3: Early stop point estimated by ideal and heuristic methods when training DenseNet (L=40, k=12) on CIFAR-100 with two types of synthetic noises of 40%: (a) and (b) show the stop point derived by the ground-truth labels and the clean validation set, respectively.
Validation Heuristic: If a clean validation set is given, we stop training the network when the validation error is the lowest. is reasonable to expect that the lowest validation error is achieved near the cross point; after that point (i.e., in the error-prone period), the validation error likely increases because the network will be overfitted to the false-labeled samples. As shown in Figure B b). the estimated stop point is fairly close to the best stop point in both noise types.
noise rate T is known, we stop training the network when the training error reaches Tx100%. If we assume that all true-labeled samples of (1-T) are memorized before any false-labeled samples of 7 100% (Arpit et al., 2017). this point indicates the cross point of label precision and label recall with their values to be all T. This heuristic tends to perform worse than the validation heuristic because the assumption does not hold perfectly. We further discuss its quality in Appendix
Noise-Rate Heuristic: If
3.2.2 QUESTION 2: CRITERION OF A MAXIMAL SAFE SET
Because the network is early stopped at the (estimated) best point, the set of memorized samples at that time is quantitatively sufficient and qualitatively less noisy. That is, can be used as a safe and effective training set to resume the training of the early stopped network without accumulating the label noise. Based on this intuition, we define a maximal safe set in Definition 333] which is initially derived from the memorized samples at the early stop point and gradually increased as well as purified along with the network's 's learning progress. In each mini-batch B,, the network parameter 0, is updated using the current maximal safe set S, in Eq. 5)) and subsequently a more refined maximal safe set S,+1 is derived by the updated network.
Definition 3.3. (Maximal Safe Set) Let Estop be the early stop point. A maximal safe set S, at time t defined be the set of the memorized samples of the network (2x; 0.) when 2 stop The network & (z; 0.) at t = tstop is the early stopped network, i.e., St.eop = Me and the network & (x;; 0,) at > tstop obtained by Eq. (5).
0c+1 = 02 - v
Σ rEB
L(a, 0:0+))
B
(5)
B. = {ø|Φ E S. n B,}
Figure shows the main advantage of learning from the maximal safe set during the remaining period. Even when the noise rate quite high (c.g., 40%). this learning paradigm exploits most of true-labeled samples, in considering that the label recall of the maximal safe set was maintained very high both CIFAR data sets regardless of the noise type. The label recall was maintained over 0.81 in symmetric noise and over 0.84 in pair noise even in CIFAR- 100 with pair noise after the 80th epoch. Further, by excluding unsafe samples that might accumulate the label noise, the label precision of the maximal safe set tended to increase rapidly at the beginning of the remaining period; it was maintained over 0.98 in symmetric noise and over 0.94 even in CIFAR- 100 with pair noise after the 80th epoch. which could not be realized by the small-loss trick (Han et al. 2018 (ong et al.) 2019; Shen & Sanghavi 2019 Chen et al. 2019). 82: #22In Section 5.3 we have empirically verified that the maximal safe set superior to the selected samples based the small-loss trick.
5

---

Under review as a conference paper at ICLR 2020
"Symmetric Noise"
"Pair Noise"
1.0
1.0
LPandd LR
LPand LR
0.9
0.9
0.8
Agarcc Popp
0.8
Approc Stop
4
I abd Rraill
0.7
0.7
70
80
70
80
Epods
Epodhs
(a) R2: Label precision and label recall on CIFAR-10 (10 classes).
"Symmetric Noise"
"Pair Noise"
1.0
1.0
LPaddL
0.8
LPaad
088
Label Predsioon
0.6
Apprax. Stop Point
Appior. Stop Pormt
Labd Recal
0.4
0.4
64
20
60
90 Epochs
100
110
120
61
70
80
90 Epocus
100
110
120
(b) Label precision and label recall CIFAR- 100 (100 classes).
Figure 4: Label precision and label recall of the maximal safe set during the remaining epochs when training DenseNet (L=40, k=12) on CIFAR data sets with two types of synthetic noises of 40%.
Algorithm 1 Prestopping with Validation Heuristic
INPUT: D: data, V: clean validation data, epochs: total number of epochs, q: history length
OUTPUT: OG: network parameters
+ 1; 0, +- Initialize the network parameter;
2: Or Otstop t 0; 1* The parameter of the stopped network */
3: for i = 1 to epochs do /* Phase Learning from a noisy training data set */
4: for j
|B]/|B,| do
5: Draw a mini-batch B, from D:
6:
Bu+1 02 a |8, ExEB, C(x, Ú; 0.)); / Update by Eq. %/
7:
val -ererr-- Get_ Validation Error(V, 0,); 1* A validation error at time t */
8:
ifssMin(vallere then Oreap + 0, / Save the network when val_error is the /owest
9:
++
10: 0, < -Baeopp 1 Load the network stopped at tstop */
11: for = stop_epoch to epochs do 1 Phase II: Learning from a maximal safe set #/
for j
|D| [.| do
13:
Draw a mini-batch B, from D:
14:
S. + [z|argmaxx, P(y x t; g)
5}: 1* A maximal safe set in Definition 3.3 #/
l.+1 0. av S.BB ExES,nB, C(x, (:))); /* Update by Eq. 5 */
t + 1;
17: return 0, S₁;
ALGORITHM DESCRIPTION
4.1 MAIN ALGORITHM: Prestopping WITH VALIDATION HEURISTIC
Algorithm describes the overall procedure of Prestopping with the validation heuristic, which is self-explanatory. First, the network is trained on the noisy training data D in the default manner (Lines 3-6). During this first phase, the validation data V is used to evaluate the best point for the early stop, and the network parameter is saved at the time of the lowest validation error (Lines 7-8). Then, during the second phase, Prestopping continues to train the early stopped network for the remaining learning period (Lines 10-12). Here, the maximal safe set S, at the current moment is retrieved, and each sample x E S2 n 13 used to update the network parameter. The mini-batch samples not included in S, no longer used in pursuit of robust learning (Lines 14-15).
4.2 COLLABORATION WITH SAMPLE REFURBISHMENT
Prestopping+
To further improve the performance of Prestopping, we introduce Prestopping+ that employs the concept of selectively refurbishing labseed samples in SELFFE (Song al. 2019). In detail,

---

Under review as a conference paper at ICLR 2020
the final maximal safe set Stane is retrieved from the first run of Prestopping. and then it is used as the true-labeled set for the next run of SELFIE with initializing the network parameter. Following the update principle of SELFIE, the modified gradient update rule in Eq. (6) is used to train the network. For each sample i: in the mini-batch Br, the given label ÿ of æ is selectively refurbished into yre/umb if it can be corrected with high precision. However, the mini-batch samples included in St... omitted from the refurbishment because their label is already highly credible. Then, the mini-batch samples in the refurbished set R, are provided to update the network together with those in the final maximal safe set Stona. Refer to Song et al. (2019)'s work for more details.
0z-1 = av
Σ DE RERRUS, Fend
L reru, +.)+
Σ WES,
C(a, ((,, (6)
11 E BiI E R2 U Stond H
5 EVALUATION
Data Sets: To verify the superiority of Prestopping, we performed an image classification task on four benchmark data sets: CIFAR-10 (10 classes and CIFAR-100 (100 classes)', a subset of 80 million categorical images (Krizhessky et al. 2014); ANIMAL-ION (10 classes 5 real-world noisy data of labell online images for confusing animals (Song et al. 2019); FOOD-10IN (101 classes a real-world noisy data of crawled food images annotated by their search keywords in the FOOD-10T
taxonomy (Bossard et al. 2014 Lee et al. 2018). We did not apply any data augmentation.
Noise Injection: As all labels in the CIFAR data sets are clean, we artificially corrupted the labels in these data sets using typical methods for the evaluation of synthetic noises (Ren et al. 2018 Han et al.
2018 Yu et al. 2019). For & classes, we applied the label transition matrix T: (i) symmelric noise:
JoST vy = I-I and (ii) pair noise: TT = T A Vx T.x = O, where Tey is the probability of the true label i being flipped to the corrupted label j and T the noise rate. For the pair noise, the corrupted label was set to be the next label of the true label i following the recent work Yu et al., 2019; Song et al. 2019). To evaluate the robustness on varying noise rates from light noise to heavy noise, we tested five noise rates E {0.0, 0.1, 0.2, 0.3, 0.4). In contrast, we did not inject any label noise into ANIMAL-ION 1ON and FOOD 101N because they contain real label noise estimated at 8.0% and 18.4% respectively (Lee et al. 2018 Song et al. 2019).
Clean Validation Data: Recall that a clean validation set is needed for the validation heuristic in Section B.2.1] As for ANIMAL-100 and Food-101N.
did exploit their clean validation data; 5, 000 and 3, 824 images, respectively, were included in their validation set. However, as no validation data exists for the CIFAR data sets, we constructed a small clean validation set by randomly selecting 1, 000 images from the original training data of 50, 000 images. Please note that the noise injection process was applied only the rest 49, 000 training images.
Networks and Hyperparameters: For the classification task, we trained DenseNet (L=40, k=12) and VGG-19 with a momentum optimizer. Specifically, we used a momentum of 0.9, batch size of 128, dropout of 0.1, and batch normalization. Prestopping has only one unique hyperparameter, the history length q, and it was set to be 10, which was the best value found by the grid search (see Appendix B. for details). The hyperparameters used in the compared algorithms were favorably set to be the best values presented in the original papers. As for the training schedule, we trained the network for 120 epochs and used an initial learning rate 0.1, which was divided by 5 at 50% and 75% of the total number of epochs.
We compared Prestopping and Prestopping+ with not only baseline algorithm but also the four state-of-the- robust training algorithms: Default trains the network without any processing for the label noise; Active Bias re-weights the backward loss of training samples based on their prediction variance; Co-teaching selects a certain number of small-loss samples to train the network based the co-training (Blum& & Mitchell 1998); Co-teaching+ is similar Co-teaching, but its small-loss samples are selected from the disagreement set; SELFIE selectively refurbishes noisy samples and exploits them together with the small-loss samples. All the algorithms were implemented
Algorithms:
https://www.cs wwronto.ddu-krri//tei. html
https://dm.kee kaist. ac. kr//aatastt/animalllll
httpss//kangguei.ggini wit.ui.giiuub.nn.. io/ Foodoo 101N
In FOOD-10TN. we used subset of the entire training data marked with whether the lahel is correct or not.
7

---

Under review as a conference paper at ICLR 2020
Default
Co-beaching
Co-tecctingg
SELFIE
Prestopping (Valication)
18%
27%
47%
60%
Bex Test TTst.Trrr Enrer
DenseNVa
VGG
Best Test Testr
DenseNet
VGG
21%
52%
10%
13%
33%
44%
8%
9%
36%
0%
10% 20% 30% 40%
0%
10%
20% 30% 40%
0%
10%
20%
30% 10%
0%
10%
20%
30% 40%
Naise Rate
Naise Rate
Naise Rate
Naise Rate
(a) CIFAR 10.
(b) CIFAR 100.
Figure 5: Best
using two CNNS data with varying pair noise rates.
18%
47%
60%
Bust Tost Erto
DenseVVet
Best Test Trrur
DerseNVet
1/1
21%
40%
52%
6%
9%
2696
36%
0%
10% 20%
30% 40%
10% 20%
10%
20% 30% 40%
0%
10%
30% 40%
Naise Rate
Noiin Rate
Naise Rate
Notin Rato
(a) CIFAR-10.
(b) CIFAR-100.
Figure 6: Best test errors using two CNNs on two data sets with varying symmetric noise rates.
using TensorFlow 1.8.0 and executed using 16 NVIDIA Titan Volta GPUs. For reproducibility, provide the source code at altttps: //bitt... ly 21xxx In support of reliable evaluation, we repeated every task thrice and reported the average and standard error of the best test error, which was evaluated using the network at the time of the lowest validation error. Thus, the clean validation set used select the best model in all algorithms during the training procedure.
5.1 PERFORMANCE COMPARISON
the test error of the seven training methods using two CNNs on two data sets with varying pair and symmetric noise rates, respectively. The results of Prestopping and Prestopping in Section [5] were obtained using the validation heuristic. See Appendix A for the results of the noise-rate heuristic. In order to highlight the improvement of Prestopping and Prestopping+ over the other methods, their lines are dark colored. Figure 7 shows the test error on two real-world noisy data sets with different noise rates. The best test errors detailed in Tables [nd o Appendix C RA: In addition, more experimental results on Tiny-ImageNNi and Clothings (iaoo et al. 2015 data sets are discussed in Appendix B.2
Figures 5
B.3
5.1.1 RESULT WITH PAIR NOISE (FIGURE
The performance trend in the two network architectures was similar with each other. In general, either Prestopping or Prestopping+ achieved the lowest test error a wide range of noise rates on both CIFAR data sets. With help of the refurbished samples, Prestopping+ achieved slightly better performance than Prestopping in CIFAR-10. However, opposite trend was observed in CIFAR- 100; this phenomenon was due to large number of falsely corrected labels under the pair noise, especially when the number of classes is large (see Appendix [B.4 for details). Although SELFIE achieved relatively lower test error among the existing methods, the test error of SELFIE was still worse than that of Prestopping. Co-teuching did not work well because many false-labeled samples were misclas- sified as clean ones; Co-teaching+ + was shown to be even worse than Co-teaching despite it being an improvement of Co-teaching (see Section 5.4 4 for details). The test error of Active Bias not com- parable to that Prestopping. The performance improvement of ours over the others increased as the heavy noise rate of 40%, Prestopping Prestopping+
label noise became heavier. In partieular, significantly reduced the absolute test error by 2.2pp-18. 1 compared with the other robust methods.
5.1.2 RESULT WITH S SYMMETRIC NOISE (FIGURF
Similar to the pair noise, both Prestopping and Prestopping+ generally outperformed the other methods. In particular, the performance of Prestopping+
the best any noise rate all data sets, because the synergistic effect was higher in symmetric noise than in pair noise. Quantitatively, heavy noise rate of 40%, our methods showed significant reduction in the absolute test error
8

---

Under review as a conference paper at ICLR 2020
Delaltt
Aciive Bias
Co-tserinng
Co-tsechingg+
SELFIE
Prestopping (Validation)
18.3%
21.1%
57.7%
72.9%
DenseNet
VGG
DenseNatt
VGG
Bed Teel limur
17.3%
19.9%
56.47%
70.3%
Bed
16.3%
18.7%
55.19%
68.3%
15.3%
17.3%
33.8%
66.3%
Method
Metood
Method
Method
(a) ANIMAL- ION (r 8.0%).
(b) FOFOOIIOINN 18.4%).
Figure 7: Best test errors using two CNNs two data sets with real-world noises.
Froe promepptio
Froe proceprioo
47%
47%
Valodatton Rhoor
34%
34%
Co-teachngg
Co-teaching+
21%
21%
SFT FR
0%
8%
Prestogping
30
120
60
90
120
Prottogpnng+
Epoch
Epoch
(a) Pair Noise.
(b) Symmetric Noise.
Figure Convergence curves of DenseNet (L=40,, k=12) on CIFAR-10 with a noise rate of 40%.
by 0.3pp-17 1pp compared with the other robust methods. Unlike the pair noise, Co-teaching and SELFIE achieved low test error comparable to Prestopping because true-labeled samples could be well separated from false-labeled ones by their small-loss criteria; hence, the dominance between Co-teaching and Active Bias reversed that the former slightly outperformed the latter.
5.1.3 RESULT WITH REAL- WORLD NOISE (FIGURE 7
Both Prestopping and Prestopping+ maintained their dominance over the other methods under real-world label noise as well. Prestopping+ achieved the lowest test error when the number of classes is small (e.g.. ANIMAL-1ON), while Prestopping was the best when the number of classes aagge(e.g, FOOD- IOIN) owing to the difficulty in label correction of Prestopping+. (Thus, practitioners are recommended to choose between Prestopping and Prestopping+ depending the number classes in hand.) Specifically, they improved the absolute test error by 0.4pp-4.6pp and 0.5pp-8.2pp in ANIMAL-1ON and FOOD-101N respectively. Therefore, believe that the advantage of our methods unquestionable even in the real-world scenarios.
5.2 CONVERGENCE ANALYSIS ON CLEAN VALIDATION DATA (FIGURE
To verify that the prror-prone period is eliminated, we plot the convergence of validation error for the seven training methods. The validation error of Default, Active Bias, and Co-teaching+ first reached their lowest values before the error-prone period, and then increased rapidly the error-prone period. On the other hand, the remaining methods kept reducing the validation eтror even in that period. Notably, Prestopping+ achieved the lowest validation error at the end, owing to the successful merger of the advanced sample selection in Prestopping and the sample refurbishment in SELFIE. In addition, the performance of Prestopping was shown to be almost the same as and comparable to that of Prestopping+ in pair and symmetric noises, respectively. The validation error of SELFIE also converged faster than those of the other existing methods, but it was still inferior to Prestopping+. Thus, these results confirm that our two methods effectively overcome the error-prone period.
5.3 ACCURACY OF SAMPLE SELECTION (FIGURE
R.A: Figure g shows the average accuracy of sample selection for two different selection strategies using DenseNet. Average accuracy is defined as the average ratio of the true-labeled samples in selected mini-batch samples during the training process. As mentioned earlier Section [ because the loss distribution of false-labeled samples was overlapped closely with that of true-labeled ones in pair noise, the average accuracy of the small-loss trick was worse with pair noise than with symmetric noise; it decreased to 75.0% as the noise rate increased in CIFAR- 100 with pair noise. On the other hand, the maximal safe set maintained its dominance over the small-loss trick in all data sets regardless of the noise type. In particular, the average accuracy of the maximal safe set was 93.2%

---

Under review as a conference paper at ICLR 2020
teaching)
(Pretooppng)
100%
100%
100%
100%
Avengge Arrarsy
Avenge Accurcy
91%
91%
91%
91%
82%
82%
82%
82%
CIBAR-10
CIFAR 100
CHBAR--1 10
CIBAR: 100
73%
73%
73%
73%
0%
10%
20% 30%
40%
0%
10%
20% 30%
40%
0%
10%
20% 30%
10%
0%
10%
20% 30% 40%
Natan Rata
Naise Rate
Natan Rate
Naise Rata
(a) Pair Noise.
(b) Symmetric Noise.
Figure 9: Average accuracy of selecting true-labeled samples by the small-loss trick Co-teaching and the maximal safe set in Prestopping using DenseNet on two data sets with synthetic noises.
Disagreement
Agreemenn
tootzachhng
Co batcingg
Disegreenent ReDo
100%
100%
100%
Small Dzop
SubecionAcsung
73%
75%
ange Drop
75%
50%
50%
30%
23%
25%
spoch
0%
0%
25%
60 Epochs
90
120
0
90
120
30
60
90
120
Epochs
Epochs
(a) Simple Network.
(b) Complex Network.
(c) Selection Accuracy.
Figure 10: Anatomy of Co-teaching+ on CIFAR- 1-0AR-10 with 40% symmetric noise: (a) and (b) show the change in disagreement ratio for all true-labeled samples, when using Co-teaching+ to train two networks with different complexity. where "simple network" is network with seven layers used by Yu et al. (2019), and "complex network" is a DenseNet (L=40,, k=12) used for our evaluation; (c) shows the accuracy of selecting true-labeled samples on the DenseNet.
even in CIFAR-100 with pair noise of 40%, which led to the huge performance improvement of Prestopping over Co-teaching in Figure 5]b).
5.4 ANATOMY OF Co-teaching+ (FIGURE 10
Although Co-teaching+ is the latest method, its performance was worse than expected, as shown in Section 5. Thus, we looked into Co-teaching+ in more detail. poor performance of Co-teaching+ was attributed to the fast consensus of the label predictions for true-labeled samples, especially when training a complex network. other words, because two complex networks Co-teaching+ start making the same predictions for true-labeled samples too early, these samples are excluded too early. As shown in Figures 10 a) and [O),), the disagreement ratio with regard to the true-labeled samples dropped faster with a complex network than with simple network, in considering that the ratio drastically decreased to 49.8% during the first 5 epochs. Accordingly, it is evident that Co-teaching+ with complex network causes narrow exploration of the true-labeled samples, and the selection accuracy of Co-teaching+ naturally degraded from 60.4% to 44.8% for that reason, as shown in Figure ().c). Therefore, we conclude that Co-teaching+ may not suit complex network.
CONCLUSION
In this paper, we proposed a novel two-phase training strategy for the noisy training data, which we call Prestopping. The first phase, "early stopping," retrieves initial set of true-labeled samples as many as possible, and the second phase, "learning from a maximal safe set," completes the rest training process only using the true-labeled samples with high precision. Prestopping can be easily applied to many real-world cases because it additionally requires only either a small clean validation set noise rate. Furthermore, we combined this novel strategy with sample refurbishment to develop Prestopping+. Through extensive experiments using various real-worke and simulated noisy data sets, we verified that either Prestopping or Prestopping+ achieved the lowest test error among the seven compared methods, thus significantly improving the robustness to diverse types of label noise. Overall, we believe that our work of dividing the training process into two phases by early stopping is a new direction for robust training and can trigger lot of subsequent studies.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In ICML, pp. 233-244 242, 2017.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In COLT, pp. 92-100, 1998.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 10m-innn discriminative compo- nents with random forests. In ECCV, pp. 446-461, 2014.
Markus Breunig, Hans-Peter Kriegel, Raymond Ng. and Jörg Sander. LOF: identifying density- based local outliers. In ACM sigmod record, volume 29, pp. 93-104, ACM, 2000.
Haw-Shiuan Chang, Erik Learned Miller,, and Andrew McCallum. Active Bias: Training more accurate neural networks by emphasizing high variance samples. In NeurIPS, pp. 1002-1012, 2017.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning. pp. 1062-1070, 2019.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 8536-8546, 2018.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. International Conference on Machine Learning, pp. 2712-2721, 2019.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, In CVPR, pp. 4700-4708, 2017.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, pp. 2309-2318, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. In NeurlPS, pp. 1097-1105, 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 and CIFAR-100 datasets, 2014.
https://www. htttps//ww..s..ooroto.oo. wtt.ootto.ea//kiii///ietot.
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In CVPR, pp. 5447-5456, 2018
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is prov- ably robust label noise for overparameterized neural networks. arXiv preprint arXiv: 1903:1900..00 1680, 2019.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-die learning with noisy labels. ICML. pp. 3361-3370, 2018.
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In NeurlPS, pp. 960-970, 2017.
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint / 2019.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: loss correction approach. In CVPR, pp. 222-, 2017.
11

---

Under review as a conference paper at ICLR 2020
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: real-time object detection. In CVPR, pp. 779-788, 2016.
Unified,
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 2015.
Mengye Ren, Wenyuan Zeng. Bin Yang. and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4334 4343, 2018.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In ICML, pp. 5739-5748, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing unclean samples for robust deep learning. In ICML, pp. 5907-5915, 2019.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, pp. 839-847, 2017.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, pp. 8688 8696, 2018.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR. pp. 2691-26999 2015.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? ICML, pp. 7164-7173, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
12

---

Under review as a conference paper at ICLR 2020
A Prestopping WITH NOISE-RATE HEURISTIC
Algorithm 2 describes the overall procedure of Prestopping with the noise-rate heuristic, which is also self-explanatonn Compared with Algorithm| T in Lines 7-9 has changed.
only the way of determining the best stop point
Algorithm 2 Prestopping with Noise-Rate Rate Heuristic
INPUT: D: data, epochs: total number of epochs, qF history length, noise rate
OUTPUT: 0g: network parameters
t+
4- Initialize the network parameter;
2: O f 0; 1* The parameter of the stopped network */
3: for
epochs do /* Phase Learning from a noisy training data
4: for j -= | do
5: Draw a mini- natii-batch B, from D:
6:
D.+1 02 a |8,] EIEB, f(x, 0.)); 1a Update by Eq. 1
7:
train_eem prr= t Get Training ErrorrDD 0e); /* A training error at time */
8:
train_err&& err S r then Save the network when train_err S T*/
9:
Otytop t- 0ú; break;
10:
1 1-t+l
11: 8, + Descop 1* Load the network stopped at tstop */
12: for = stop_ec _ippeppcchhppppp to epochs do 1 Phase II: Learning from a maximal safe set */
13: for j = 1 to /DI/ [. do
14:
Draw a minimmittttt B, from D;
15:
S. + { [zlargmax, P(yle, (22, jayjara,,,Plyylttttg = ū}: 1 A maximal safe set in Definition 3.3 #/
0.41 0: av [S,na EmESins. (.), Update by Eq. 5
1 ++1+1;
18: return 9g. SG;
A.1 RESULT WITH SYNTHETIC NOISE (FIGURE
To verify the performance of Prestopping and Prestopping+ with the noise-rate heuristic in Section 3.2.1 we trained a VGG-19 network on two simulated noisy data sets with the same configuration as Section 5 Figure [[]sswss the test error of our two methods using the noise-rate heuristic as well as those of the other five training methods. Again, the test error of either Prestopping Prestopping was the lowest at most error rates with any noise type. The trend of the noise-rate heuristic here was almost the same as that of the validation heuristic in Section 5.1 Especially when the noise rate was 40%, Prestopping and Prestopping+ significantly improved the test error by 5.1pp-17 17.0pp in the pair noise (Figure (a)) and 1133p-177.pp in the symmetric noise (Figure I b)) compared with the other robust methods.
Delmut
Co-lsectings-
Prestopiin Rate)
27%
60%
27%
60%
Bet Ties Emtr
CIRAR-10
CIFAR: 100
Best Test liner
CIRAR-
CIFA C-FAR-100
21%
52%
21%
52%
15%
44%
13%
44%
9%
30%
9%
38%
0%
10% 20%
30% 40%
10%
20% 30% 40%
0%
10%
20%
30% 40%
0%
10% 20% 30% 40%
Natse Rata
Naise Rate
Naise Rate
Naise Rate
(a) Pair Noise.
(b) Symmetric Noise.
Figure 11: Best test errors using VGG-19 on two simulated noisy data sets with varying noise rates.
A.2 COMPARISON WITH VALIDATION HEURISTIC (FIGURE
Figure 12 shows the difference in test error caused by the two heuristics. Overall, the performance with the noise-rate heuristic was worse than that with validation heuristic, even though the worse one outperformed the other training methods shown in Figure 11 As the assumption of the noise-rate
13

---

Under review as a conference paper at ICLR 2020
heuristic does not hold perfectly, lower performance of the noise-rate heuristic looks reasonable. However, we expect that the performance with this heuristic can be improved by stopping a little earlier than the estimated point, and we leave this extension the future work.
Prostopping (Valication)
Prssogpingg (Validation
Prestopping (Noisn Roiri
Prodopping+ (Nrieei
16%
46%
19%
53%
Best Test Brvor
CTRAR-10
CIFAR-10 100
Dest Test Eroo
CIRAR 10
C. 1000
12%
40%
13%
41%
10%
39%
10%
10%.
10%
Nais: Potl:
Noix: Rete
Noise Rate
Naize Rate
(a) Pair Noise.
(b) Symmetric Noise.
Figure 12: Difference in test error between two heuristics on two simulated noisy data sets.
B SUPPLEMENTARY EVALUATION
B.I HYPERPARAMETER SELECTION (FIGURE 13
Prestopping requires one additional hyperparameter, the history length & in Definition 3.1 For hyperparameter tuning, we trained DenseNet (L=40, k=12) on CIFAR-10 and CIFAR-I00 with noise rate of 40%, and the history length was chosen in grid E {1,5, 10, 15, 20). Figure 13 shows the test error of Prestopping obtained by the grid search on two noisy CIFAR data sets. Regardless of the noise type, the lowest test eтror was typically achieved when the value of was 10 in both data sets. Therefore, the history length y was set be 10 in experiments.
15%
30%
13%
X
38%
Symmotric Noise
9%
32%
20
History Length
History Length q
(a) CIFAR-10.
(b) CIFAR-100.
Figure 13: Grid search on CIFAR-10 10 and CIFAR-100 with two types of noises of 40%.
B.2 EXPERIMENT USING TINY-IMAGENET DATA SET (FIGURE
R3: For a larger-scale experiment, we repeated the image classification task on Tiny-ImageNet (200 classes), subset of ImageNet (Krizhevsky al. 2012). with 100, 000 training and 10, 000 validation images. Because no test set exists. randomly selected 9, 000 images from the validation set were used the set, and the rest 1, 000 validation images were used as the clean validation for the validation heuristic. The experimental configurations were the same as those in Section 5
Figure 14 shows the test eгror of the seven training methods using VGG-19 19 on Tiny- ImageNet with varying noise rates. Similar to CIFAR data sets in Figures S and 6 both Prestopping and Prestopping+ outperformed the other robust methods in both noise types. The only difference was
Default
Co-texching
Co-teectingt
SELFIE
Prestopping (Valication)
Puestogoingt
Erear
67%
VGG
839
VGG
Noise Type
Pair Noise
Symmetric Noisc
60%
61%
Default
65.6=0.52
68.3+
Active Bias
62.9=0.31
66.7+0.87
Costenching
67.6-0022
64.0400.1
53%
53%
Co-teaching-
67.8=0.31
75.420059
467
45%
63.6=0.44
63.2:0.19
0% 10% 20% 30% 40%
0%
10% 20%
30% 40% Niiinn Rain
Prestopping
8..1=0.24
61.5:0031
Noise Rate
Prestopping+
56.4=0.37
62.8=0025
(a) Pair Noise.
(b) Symmetric Noise.
(c) Detailed Results Noise Rate 40%.
Figure 14: Best test errors using VGG-19 on -mny-Im with varying noise rates.
14

---

Under review as a conference paper at ICLR 2020
that there synergistic effect from collaboration with sample refurbishment because of the larger number of classes (i.e., 200). In particular, compared with other robust methods, Prestopping showed significant reduction in the absolute test error by 7.8pp-12.7pp at pair noise of 40% and 1.7p--3.9ppp at symmetric noise of 40%.
B.3 EXPERIMENT USING CLOTIING DATA SET (FIGURE 15
R2: We additionally evaluated the superiority of Prestopping(+) on another challenging real-world noisy data set Clothing70k (14 classes), a subse of Clothing (Xiao al. 2015), where its noise rate was estimated at 38.5%. For the Clothing70k data set, we randomly selected 70, 000 images from I million noisy training images in Clothing M and used them as its noisy training set; we did exploit the original clean validation and test sets consisting of 14, 313 and 10, 526 images, respectively. The experimental configurations were the same as those in Section
Figure 15 shows the test error of the seven training methods using two CNNs on Clothing with real-world noise of 38.5%. Overall, Prestopping and Prestopping+ significantly outperformed the other robust methods. In particular, Prestopping+ achieved the lowest test enror because the number of classes was not that large (i.e., 14 classes). Compared with other ofte--of-thh-ar methods, improved the absolute test error by 1.1pp333ppp using DenseNet and 0.7pp-5.8pp using VGG-19.
1: Defaiit
Artive Rlas
Cb-tarching
Co Co-brachingt++
SRR FIP 10 Prestopping (Validation)
Prrstoppingt (Valicbbion)))
39.0%
47.2%
Network
DenseNet
VGG-19
Best Test Ebor
DenaaN
VGG
Default
38.==010
46.920.31
38.0%
45.3%
Active Bias
37.7 3..7
0..5
Co-tecching
37.6-0.14
43.3
37.0%
43.4%
Co-teaching-
39.7=0.18
0.58
SELFIE
37.5
42.71.0.3
36.0%
41.5%
Prestopping
3..6--011
42.2
Mathod
Method
Prestopping+
36.4=0.07
4.10:
(a) DenseNet.
(b) VGG-19.
(c) Detailed Results.
Figure 15: Best test errors on Clothing Re 38.5%) along with the detailed result.
B.4 IMPACT OF NUMBER OF CLASSES ON Prestopping+ (FIGURE 16)
We investigate the impact of the number of classes on Prestopping+ under the pair noise of 40%. Figure 16 shows the ratio of refurbished samples used for training (hatched bar) with the refurbishing accuracy (solid line) on two data sets with 10 and 100 classes. When the number of classes is small (e.g.. 10), Prestopping+ refurbished most of false-labeled samples very accurately; the refur- bishing accuracy was consistently over 91.6% during the entire training period. On the other hand, when the number of classes is large (e.g.. 100), the refurbishing accuracy drastically dropped from 92.0% to 76.7% as more samples were refurbished. Thus, as shown in Figure 16) collaboration with sample refurbishment not always synergistic because such significant drop the refurbishing accuracy could induce many falsely corrected samples, especially when there are many classes
of Sumples
40%
100%
Refurbishing Ratio
20%
10%
Reforbishing ccumnry
0%
70%
Training Iteration
Training Lteration
(a) CIFAR-10 (10 classes).
(b) CIFAR- (100 classes).
Figure 16: Refurbishing ratio and accuracy of Prestopping+ when training DenseNet (I=40,, k=12) on CIFAR-10 and CIFAR-100 with pair noise of 40%.
We used subset of Clothing M finish this experiment within the ICI.R20 rebuttal period.
15

---

Under review as a conference paper at ICLR 2020
C COMPLETE RESULTS ON BEST TEST ERROR
Table shows the test error of seven training methods using two CNNs on two simulated noisy data sets with varying noise rates. T shows the test error of seven training methods using two CNNs on two real-world noisy data sets with different noise rates.
Table I:
The best test errors (%) of seven training methods on two types of synthetic noises with varying noise rates (0%, 10%, 20%, 30%, and 40%) in Figures 5 and 6
Noise Type
Pair Noise in Figure
Symmetric Noise in Figure O
Data Set
CIFAR-ID 1D
CIAAR.
CHFAR-I0
CITRR 100
Network
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
Defaub
7.2010.05
1144..14
27.3::..5
38.911.007
7.20:10005
11410.14
27.3410.55
38.9-0.07
Active Bias
745+0.36
01.40.08
26.7440.8
39.5+0.09
7.3500.36
114+0.08
26.740048
39.5=00.99
Co teaching
7.1610.01
11.11
27.4.10.87
37.7
7.16.10.01
11.1
27.410887
37.7-0004
Co-weahing
10.74:0.7
13.640028
30.14:024
43.6-10.09
10.74:0.2
13.640.28
30.1
43.6=0.09
SELFIE
720+0006
11.000.07
27.0++0.1
38.420.12
7.20:00.06
11.020.07
27.0+0.41
38.4=0.12 12
Prestopping
7.2.10.06
11.41..15
27.2.10.11
38.2.10.17
7.32.10.06
11.41..15
27.2110.11
38.2-2017
Prestopping*
7.0210.11
10.910.10
26.7:10.18
37.7
7.0210.11
10.910.10
26.71.0.18
0..5
(a) The best test errors under pair and symmetric noises of 0% (T 0.0).
Type
Pui Noise in Figure
Symmetric Higure
Data Set
CIFAR 10
CIA
100
CIAAR 10
CIARR
100
Network
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
Default
9.7910.02
1.8
32.1
41.410.15
11.910.74
16.8
32.71.0.6
44.9 0.24
Active Bias
9.77110.7
13.74.0.4
30.5:4063
42,24:.22
11.0110.4
14.1
30.9110.19
0..5=0221
Co-teaching
9.62:0.09
13.110133
30.2:10.88
41,51:0,00
9.79:10.18
14.520.12
29.2:10.15
42.5-0.30
Co-seaching+
12.2.10.14
14.0
33.4.10.14
43.6.0.52
13.2.10.34
16.710.09
33.610441
50.8 10224
SELFIE
R6240.00
13.0::0.07
28.7440.0
40.94.0.44
8.73.0.112
13.720.23
28.4::0.09
41.3=0032
Prestopping
1.
0.14
28.5:10.14
38,71:0.20
8.77:1.044
13.31.003
28.61:0.84
0.21
Prestopping+
7.72.10.01
1.00
28.3.10.12
38,5
8.23
12.2..0.33
27.91.0.33
38,3 0.16
(b) The best test errors under pair and symmetric noises of 10% = 0.1).
Noise Type
Pair Noise in Figure
Symmetric Noise Figure
Data Set
CIFAR-10
CIFAR-100
CIEAR-10
CIFAR-100
Network
DemeNet
VGG
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
Defawli
10.910.44
15.141.111
35.54:0.11
42,910221
13.4::0.77
18.520.31
35.4110.79
50.7=0022
Active Bias
10.800.77 17
14.2+0.13
34.1400.9
43.2+0.09
12.6+00.7
16.220.20
33.34:.25
50.0=0:25
Co-teaching
10.7
15.1
32,34095
43.9=0.31
10.9::027
13.5#0.08
30,4+0.20
4419=0.17
Co-tenching
13,910.38
15910031
36.610344
47.010.34
15.0:10.34
19.010.11
37.210.18
03.4=0.14
SELFIE
9,46+0 10
14.6+00.06
29.940:44
42,3=0.17
9.86-4-0225
0.
30.24:0377
43.9=0.09
Prestopping
8 8.3410.18
13.1+00.88
29.3:10.24
39,912042
9.73:1:0.88
14.920.23
30.8::0.70
42.5=0.20
Prestopping+
0.17
10.03
29.5:10:20
39.5
8.83:10.30
13.5 0.00
29.6:10.40
41.6 0.08
(c) The best test errors under pair and symmetric noises of 20% (T 0.2).
Noise Type
Pair Noise in Figure
Symmetric Noise in Figure
Data Set
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Network
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
Defawb
12.61198
18.641.11
41.1+:0.3
46.3+:0.25
15.9:1255
23.240.41
39.0+:0.88
54.4=0.25
Active Bias
12.240.00
15.620.1
38.42091
46.020.16
14.6:048
18772031
36.740.66
53.7=0.12
Co-tenching
118440.54
16.340.11
36.6:E1008
31.320.36
11.740.35
17.620.02
32.6:0.33
48.2=0.34
Co-seachin
15.7+0.19
16.640.12
41.8::0.99
50.2+0021
16.3+00.12
24.0=0.17
40.64:0.24
59.6=0.23
SELFIE
10.622034
16440.17
32.92013
45.12:0116
10.8:00009
16.020.24
31.340033
43.8=0.00
Prestopping
9.2440.4
13.620.14
31.1
41.3-0.14
11.2:0097
16.820.3
32.14:0.4
43.2=0.31
Prestopping*
8.6:10.29
13.0::0099
31.3::0199
41.3-:0.05
10.11:0.70
1..1
31.0-1.88
42.6=0.06
(d) The best test errors under pair and symmetric noises of 30% 0.3).
Type
Pair Noise in Figure
Symmetric Noise Figure h
Data Set
CIFAR 10
CIAR-
100
CIFAR 10
CIFAR 100
Network
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
DenseNet
VGG
Default
19.2100.3
24.110.97
41.210.4
54.110.82
1..0.1..66
25.010.19
46.110072
60.9_0.08
Acrive Bias
16.911093
24.41
49.6:10.66
49.6110.00
17.511.9
24.310.36
39.2-10.84
57.9=0.20
Co-teaching
13991.11
26.5422477
45.04-1.06
60.540.69
1388+041
19.120.07
34.5+-013
54.2=0.45
Co-teaching+
20.6.10.7
21.3100.00
50.7.10.65
56.7110.16
19.9110.28
27.10.24
44.611.04
66.0
SELFIE
12.3440.00
20.410.24
4..9
53.0-10.45
12.4400.07
18.020.14
33.54.0.00
49.5=00077
Prestopping
10.140020
15.240054
32.6:1..40
42.420.28
12.74-0.3
18.340.07
34.9--0.63
51.3=0.44
Prestopping+
9.60.10.13
14.610..17
34.4110.10
43.810..14
11.3100.3
0.08
32.9110.00
48.5 0.05
(c) The best test errors under pair and symmetric noises of 10% (T = 0.4).
16

---

Under review as a conference paper at ICLR 2020
Table 2: The best test errors (%) on real-world noises in Figure
Data Set
ANIMAL 1ON
FOOD-10IN
Definls
18.240.15
20.8=0.36
57.6:10.20
72. 72.14:0.5
Active Bias
17.622.13
20.0=0.30
55.4:0.11 10
69.24::88 84
Co-teuching
16.940.14
19.9=0.11
55.6=20.99
69,42:027
Co-senching
19.8-0.11
22.4=0.11
58.6:10.24
75.54:0.53
SELFIE
16420,21
18.5=0.14
55,4=0.17
68.6220.24
Prestopping
15.7=0.18
18.1=0.14
54.3-20.24
67.3:1:.21
Prestopping
15.5-0.12
54.9:10.17
D CASE STUDY: NOISY LABELS IN ORIGINAL CIFAR-100
One interesting observation is noticeable improvement of Prestopping+ even when the noise rate was 0%, as shown in Table a). It was turned out that Prestopping+ sometimes refurbished the labels of the false-labeled samples which were originally contained in the CIFAR data sets. Figure 17 shows a few successful refurbishment cases. For example, an image falsely annotated as a "Boy" was refurbished as "Baby" (Figure 17 a)), and image falsely annotated as "Mouse" was refurbished as a "Hamster" (Figure 17 c)). c)). Thus, this sophisticated label correction Prestopping+ helps overcome the residual label noise in well-known benchmark data sets, which are misconceived to be clean, and ultimately further improves the generalization performance network.
(a) Buy
(b) Doy Baby
(c Mouse Homsttr
(6) Shrew Mouse
Lion
(3) Worm Snake
(g) Spider
Trin
(1) Apple
(m) Sweet Pepper A pple
(c) Pear - Sweet Pepper
(6) Tulip Poppy
(p) Forest - Maple Tree
(g) House Road
(r1 Plain - Sea
[s] Sea Plain
18 Plain Clond
(u) Cloud
Can
(v) Can
Bowl
(w) Can
Bottle
(x) Skyssmper
Rocket
Figure 17: Refurbishing of false-labeled samples originally contained CIFAR-100. The subcaption represents "original label" + "refurbished label" recognized by Prestopping+.
17


Reviews:
[{"review_id": "r1xfP7FKFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to study how early stopping in optimization helps find confident examples. Overall, the paper is well-organized and easy to read. Although there is some parallel study regarding the theoretical aspect of how early stopping help finds confident examples (i.e., Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks, which has unfortunately not been cited), the paper focuses on the empirical perspective. A thorough empirical study illustrating how early stop works would interest the label noise community.\n\nThe authors claim that early stopping is efficient to find a maximal safe set. I think it would be necessary to illustrate the maximal safe set for all the datasets. The authors only did this for one case of CIFAR-100, which is not convincing. The small loss based learning has shown the confident examples extracted. It would be essential to compare the proposed method with those methods.\n\nThe authors are suggested to compare the proposed method with more baselines. There are lots of algorithms exploiting the transition matrix and with statistically consistent estimators. The authors have ignored all those methods. \n\nIt is unclear from the paper that if the baselines have used the clean validation sets. For fair comparison, those clean data should be used in the training procedures of the baselines.\n\nClothing1M is a more challenging dataset with real-world label noise. The dataset also includes some clean data for validation use.  The authors should verify the effectiveness of the proposed method on this dataset.\n\nThe authors are suggested to make it clear why noise rates are sometimes available for use.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1gL1Hq6Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a two-phase training method for learning with label noise. \n\nOn the positive side, this paper focuses on the idea of prestopping and proposes several relevant definitions to formalize their idea and come up with a heuristic algorithm. \n\nHowever, I believe the paper has missed several very relevant papers that provides very similar ideas. Both [2] & [3] provide theoretical analysis to why early stopping matters in learning with label noise for DNNs. Before these two papers, [1] also observed that the learning trajectories for clean and noisy samples are different in label noise problem, and they used early stopping in their experiments to address this issue. Given these existing literatures, the contribution of this paper should be considered more properly. \n\n[1] Learning with Bad Training Data via Iterative Trimmed Loss Minimization, Yanyao Shen, Sujay Sanghavi, ICML 2019.\n[2] Hu, Wei, Zhiyuan Li, and Dingli Yu. \"Understanding Generalization of Deep Neural Networks Trained with Noisy Labels.\" arXiv preprint arXiv:1905.11368 (2019).\n[3] Li, Mingchen, Mahdi Soltanolkotabi, and Samet Oymak. \"Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks.\" arXiv preprint arXiv:1903.11680 (2019).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syem3k3LqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a training approach on label noise datasets and outperforms state-of-art methods. It defines the samples whose average probability on assigned label in recent q iterations is largest among all labels as memorized samples, in the sense of the network memorize these samples. Then authors proposed two stage method which firstly early-stops at minimum validation error (or $\\tau$ memorized rate), and then trains on maximal safe set that gathers memorized samples. The experiments compared several state-of-art approaches and showed that the proposed method benefits from early-stopping and safe set. Authors also showed that the prestopping idea can also be used to improve other approaches.\n\nPros:\n\nThe proposed method achieves better performance than state-of-art methods.\n\nAuthors have good experiments which evaluate on multiple datasets and algorithms.\n\nAuthors also investigate the relation between model complexity and performance of co-teaching+\n\nCons:\n\nMany recent papers indicate the “error-prone period”, authors should include related works about early-stopping on label noise training.\nhttps://arxiv.org/pdf/1901.09960.pdf fig1\nhttps://arxiv.org/pdf/1903.11680.pdf fig5\nhttps://arxiv.org/pdf/1906.05392.pdf fig3\n\nAlthough the method achieves good performance, since the idea is a bit straightforward especially after exploring above papers, I am slightly worried about novelty of the ideas.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx-7fCU5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a training strategy for robustness against label noise. The training strategy is simple and straightforward. The neural network will first be trained on the entire dataset with all the noisy labels. After obtaining the network with lowest validation error, the network will be used to make a prediciton on the original training set and select a subset of it to construct a maximal safe set. Finally, the network will be findtuned on this maximal safe set. The training strategy is very similar to tradictional  self-training in semi-superivsed learning and co-training for domain adaptation ([Co-training for domain adaptation, NIPS 2011]), except that the proposed prestopping only iterate the procedure once.\n\nThe paper discusses two important questions for the method: (1) when to early stop the training; (2) how to constuct a maximal safe set. The authors' responses to these questions are very natual but less interesting. Using the lowest validation error to early stop the training could be suboptimal, since the small validation set can not fully capture the data distribution and could make the network empirically overfit to this validation set. The criterion to contruct a maxial safe set is also conventional, and is similar to what a number of papers are doing, for example,\n[1] Co-training for domain adaptation, NIPS 2011\n[2] Self-ensembling for visual domain adaptation, ICLR 2018\n[3] A dirt-t approach to unsupervised domain adaptation, ICLR 2018\n[4] Iterative learning with open-set noisy labels, CVPR 2018\n\nIn experiments, the results are not very surprising. There are some baselines that adopt a similar (iterative) pipeline (learning the network - selecting a subset of the training samples - re-learning the network):\n[1] Iterative Learning with Open-set Noisy Labels, CVPR 2018\n[2] Dimensionality-Driven Learning with Noisy Labels, ICML 2018\n[3] Symmetric Cross Entropy for Robust Learning with Noisy Labels, ICCV 2019\nThe authors can consider to compare to some of these baselines, especially [1] and [2]. The difference between the paper and [1,2] is basically the criterion to construct the maximal safe subset.\n\nBesides, I suggest the authors to conduct large-scale experiments on ImageNet or even a subset of ImageNet, since the difficulty of detecting label noise is much higher when the resolution of images become bigger. CIFAR-10 and CIFAR-100 only contain 32x32 images, which is far less challenging.\n\nOverall, I think the paper is well written, the idea is clearly presented, and the experiments also seem convinceing. However, the contribution of this paper is very incremental.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on avoiding overfitting in the presence of noisy labels. The authors develop a two phase method called pre-stopping based on a combination of early stopping and a maximal safe set. The reviewers raised some concern about illustrating maximal safe set for all data sets and suggest comparisons with more baselines. The reviewers also indicated that the paper is missing key relevant publications. In the response the authors have done a rather through job of addressing the reviewers comments. I thank them for this. However, given the limited time some of the reviewers comments regarding adding new baselines could not be addressed. As a result I can not recommend acceptance because I think this is key to making a proper assessment. That said, I think this is an interesting with good potential if it can outperform other baselines and would recommend that the authors revise and resubmit in a future venue.
Decision: Reject