ME-MOMENTUM:  EXTRACTING HARD CONFIDENT EXAMPLES FROM NOISILY LABELED DATA
Authors: Yingbin Bai
Tongliang Liu
Venue:  (2021)
Keywords: label noise
hard confident examples

Abstract: Examples that are close to the decision boundary—that we term hard examples, are essential to shaping accurate classifiers.   Extracting confident examples has been widely studied in the community of learning with noisy labels.  However, it remains elusive how to extract hard confident examples from the noisy training data.  In this paper, we propose a deep learning paradigm to solve this problem, which is built on the memorization effect of deep neural networks that they would first learn simple patterns,  i.e.,  which are defined by these shared by multiple training examples. To extract hard confident examples that contain non-simple patterns and are entangled with the inaccurately labeled examples, we borrow the idea of momentum from physics. Specifically, we alternately update the confident examples and refine the classifier.  Note that the extracted confident examples in the previous round can be exploited to learn a better classifier and that the better classifier will help identify better (and hard) confident examples.  We call the approach the “Momentum of Memorization” (Me-Momentum). Empirical results on benchmark-simulated and real-world label-noise data illustrate the effectiveness of Me-Momentum for extracting hard confident examples, leading to better classification performance.

Under review as a conference paper at ICLR 2021
MME-MOMENTMM EXTRACTING HARD CONFIDENT EXAMPLES FROM NOISILY LABELED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Examples that are close to the decision boundary that we term hard examples, are essential shaping accurate classifiers. Extracting confident examples has been widely studied in the community of learning with noisy labels. However, it remains elusive how to extract hard confident examples from the noisy training data. In this paper, we propose a deep learning paradigm to solve this problem, which is built on the memorization effect deep neural networks that they would first learn simple patterns, i.e., which are defined by these shared by multiple training examples. To extract hard confident examples that contain non-simple patterns and entangled with the inaccurately labeled examples, we borrow the idea of momentum from physics. Specifically, we alternately update the confident examples and refine the classifier. Note that the extracted confident examples in the previous round can be exploited to learn a better classifier and that the bet- ter classifier will help identify better (and hard) confident examples. We call the approach the "Momentun of Memorization" (Me-Momentum), Empirical results on manchmark-simulltte and real- world label-noise data illustrate the effective- ness of Me-Momentur for extracting hard confident examples, leading to better classification performance.
INTRODUCTION
As training datasets are growing big while accurately labeling them often expensive or sometimes even infeasible, cheap datasets with label noise are ubiquitous in many real-world applications. Without any care, label noise will degenerate the performance of learning algorithms, especially for these based on deep neural networks (Zhang et al., 2017). Learning with noisy labels (Angluin & Laird 1988) aims to reduce the side-effect of label noise and therefore has become an important topic in machine learning.
Existing methods for learning with noisy labels can divided into two categories: algorithms that result in statistically consistent inconsistent classifiers. Methods in the first category intent to design classifier-consisttr
algorithms (Zhang & Sabuneu 2018 Kremer et al., 2018; Liu Tao
2016 [Scott 2015; Natarajan et al.] [013]] Goldberger & Ben-Reuven] 2017 7| Patrini al., 2017 Thekumparampil et al.] 2018 Yu et al. 2018)[Liu & Guol (020) Xu et al. 2019). where classifiers
learned by using the noisy data will statistically converge the optimal classifiers defined by clean
data. However, these methods rely heavily on the noise transition matrix (Liu & Tao| [016 Patrini al. 2017 Xia et al. 2019). In real-world applications, is hard to learn the instance-independerr noise transition matrix (Cheng et al., 2020). To be free of estimating the noise transition matrix, methods the second category employ heuristics to reduce the side-effect of label noise (Yu al.)
2019
Han et al. 201 Malach & Shalev-Shward 2017 Ren et al. 2018 Jiang et al., 2018 Ma
al. 2018 Kremer et al.) 2018 Tanaka et al.] [00 Reed et al. 2015]] Han et al.) [008aa] Guo
ct al.
2018 Veit et al. 2017 Vahdat 2017: Li al. 2017 2020). These methods were reported to empirically work well, especially in the setting of instance instance-dependent label noise.
One promising direction in the second category is to extract examples with clean labels conflen
examples (Northcutt et al. 2017 0 Cheng et al. 2020 Chen et al.] 2019 Malach & Shalev- Shalev-Shwartz Shwartz,
2017
Han al. 2018b [b] Yu et al. 2019 Thulasidasan et al.] 2019 guyenn al. [220; Dredze
et al. 2008 Liu Tao. 2016; Wang et al. 2017 Ren et al., 2018 Jiang et al.j 2018). The idea is
that compared with the original noisy training data, the extracted examples are less noisy and thus

---

Under review as a conference paper at ICLR 2021
(a)
Clcatiication with clean data
(b) Classification with nolsy labels
Classification without hard (c) confident examples
Classification with hard confident examples
Figure 1: The illustration of the influence of hard (confident) examples in classification. Circles represent positive examples while triangles represent negative examples. Green and blue denote examples with accurate labels while red presents examples with incorrect labels. Blank circles and triangles represent unextracted data. (a) shows an example of classification with clean data. (b) shows noisy examples, especially those close to the decision boundary, will significantly degenerate the accuracy of classifier. (c) shows confident examples help learn fairly good classifier. (d) shows that hard confident examples are essential to train an accurate classifier.
will lead to classifier with better performance. Given only noisy data, state-of-the-an methods exploit the memorization effect (Zhang ct al., 2017, et al., 2017) to extract confident examples. The memorization effect will enable deep neural networks to first learn patterns that are shared by majority training examples. As clean labels are of majority in each noisy class (Natarajan et al.) 2013 Xiao et al. 2015), deep neural networks would therefore first fit training data with clean labels, and then gradually fit the examples with incorrect labels (Chen et al. (201)). Therefore, early stopping (Li et al., 2020; Song et al., 2019) and the small loss trick (Jiang et al. 2018 Han et al,, Yu et al. (219)) can be used to exploit confident examples.
2018b
Examples that are close to the decision boundary are called hard examples. As illustrated in Figure hard (confident) examples play an important role in shaping the decision boundary. It has also been widely studied in the traditional classification problem that hard examples are essential to train accurate classifiers [Vapnik 2013 [Bnniio et al. [2009] Huang et al. 2010; He et al. 2018). Notwith- standing the importance of hard confident examples, none of the existing methods studies how to extract hard confident examples from noisy data. Note that extracting hard confident examples is non non-effortless. Since hard examples are often of small proportion and contain less discriminative information compared with the easy ones (these that are far away from the decision boundary), they are often entangled with inaccurately labeled examples in the procedures of extraction.
In this paper, by alternately updating the confident examples and refining the classifier, we propose deep learning paradigm that is able to extract hard confident examples from the noisy training data, leading to better classification performance. Specifically, the idea similar to the usage of momentum from physics. As stated in the statistical learning theory, with better training data, bet- ter classifier can be obtained (Mohri et al., 2018). We can then think of the classifier as a particle traveling through the hypothesis space, getting acceleration from the confident data. Classifiers with better performance can be achieved by properly exploiting the previously extracted confident exam- ples. This is similar to the momentum trick used in optimization that previous gradient information can be used to escape local minimum and achieve fast convergence rates (Sutskever et al. 2013 [ At a high level, the proposed method is built on the memorization effect of deep neural networks and on the intuition that better confident examples will result in a better classifier and that better clas- sifier will identify better confident examples (and hard confident examples). The proposed method is therefore called the Momentum of Memorization (Me- -MM-Mmunnt
We conduct experiments to show the effectiveness of the proposed Me-Momentum on noisy versions of MNIST, CIFARIO, CIFARI00, and real-world label noise dataset ClothingIM, Specifically, on MNIST' and CIFAR, we generate class- and instance-dependent label noise and visualize the extracted hard confident examples, which justifies why Me-Momentum consistently outperforms the baseline methods. Codes will be available online.
ME-MOMEETUM
In this section, by specifying the proposed method of momentum of memorization (Me-Momentum; summarized in Algorithm T.. we would like to detail how to accomplish extracting hard confident
In optimisation, the parameter vector can be thought of as particle traveling through the parameter space, getting acceleration from the gradient of the loss. The momentum trick demonstrated that gradient in previous update can help escape local minimum and achieve fast convergence rates.
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Me-Momentum
Input: Noisy training data, noisy validation data, iteration number Ninnerr and Nouteri
Output: Extracted confident examples and the classifier; I: Initialize classifier fo by using the noisy training data and early toopinng://memorizzee effect for i ".....,ouus do
for j = j= inner do
2: Update the extracted confident examples;
//i.e., the training examples whose noisy labels are identical to the ones predicted by fz-
3: Obtain the classifier /j:
//imitialize the network by using the parameters of f-- and train it by employing con- fident examples; the classifier fy will be chosen with the highest noisy validation accuracy throughout the training procedure
4: Break and output f if the highest validation accuracy is non-increasing in the loop; end
5: Re-initialize a classifier fo:
//randomly initialize the network and train it by using confident examples; the classifier fo will be chosen with the highest noisy validation accuracy throughout the training procedure
6: Break and output f--1 if the highest validation accuracy is non-increent non-increasing in the loop;
end
examples and boosting the classification performances. At the high level, by alternately updating the confident examples and refining the classifier, Me-Momentum fulfils positive cycle that better confident examples will result better classifier and that a better classifier will identify better confident examples. Specifically, Me-Momentum has two loops, i.c., inner loop and an outer loop. In the inner loop. Me-Momentum alternates update of the confident examples and classifier (Steps 2 and 3). However, the inner loop continually refines classifier and thus depends heavily on the initialization of the classifier (Step 1). It may lead to the memorization of noisy labels and the inferiority of sample-selection bias. To handle this problem, the outer loop re-initialize the classifier (Step 5) while maintains the previously extracted confident examples.
There are some points to be clarified for the proposed Algorithm [
Q1. How to initialize a good classifier in Step 1?
Q2. How to extract confident examples in Step 2?
Q3. How to validate the learned classifiers in Steps and 5 without clean validation set?
Q4. Why the proposed method is called Me-Momentum?
Q5. Why hard confident examples can be extracted?
To answer the first question, we would like to mention that the aim of the initialization in Step is to initialize a good classifier for the positive cycle: a better classifier will identify better confident examples and better confident examples will result in a better classifier. A good candidate should have a fairly high classification accuracy, c.g., higher than random guessing. Otherwise, the positive cycle cannot be evoked. Fortunately, the initialization can be made by exploiting the memorization effect of deep neural networks that they would first fit clean data (Arpit et al.) [017]] /Zhang et al.) 2017). Note that this memorization effect is independent of training optimization or network back- bones (Arpit et al. 2017). Specifically, we use the early stopping trick. For easy understanding, we would like introduce definition of high-peak. called an i-th high-peak if it achieves the highest accuracy in the epoch range {1....,i}. Assume the i-th and j-th b. respectively. The training early stops if (b a)/(j i) < r, where T is hyper-parameter. In the experiments, we set = 0.1, which empirically works well across all datasets. Appendix A, compare the difference between the early stopping method and the traditional validation method; We also study the sensitivity of the
noisy validation accuracy at the i-th epoch is
occur next to each other, having noisy validation accuracies of a and
parameter.
The answer to the second question is closely related to the memorization effect. Note that the classifier initialized in Step would fit the clean data well but not the incorrectly labeled data because of the memorization effect and early stopping. Therefore, noisy labels are identical to the ones predicted by the classifier obtained in Step as confident examples. This also applies for the classifiers in Step 3 to extract confident examples, which are
can treat the training examples whose
3

---

Under review as a conference paper at ICLR 2021
iteratively trained by employing the updated confident data. Note that there some other feasible methods to extract confident examples, e.g., extract those who have a large class posterior.
In Step 3, aim to learn better classifier compared with the one in the previous round. This can be achieved because of two reasons: (1) we initialize the network with the parameters of the classifier learned in the previous round; (2) we have better set of confident examples the training sample. This starts the positive cycle that better confident examples will result in a better classifier and that a better classifier will identify better confident examples. The third question is essential for identifying the classifiers in the cycle. Note that the accurately labeled examples are always assumed to be dominant in each class in the community of learning with noisy labels (Natarajan et al. [013] Liu & Tao 2016: Han et al. 2018b). Otherwise, the true class label cannot be identified by only exploiting the noisy data. This assumption implies that the performance on the noisy validation set (split from the noisy training set) and these on test set are positívely correlated. The noisy validation set could be used as a surrogate to validate the classifiers if no clean validation set is available. We
therefore validate the classifiers in Steps 3 and 5 with the highest noisy validation accuracies during the training. Empirical results show that it works well.
To answer the fourth question, we would like to first mention that the proposed method heavily relies on the memorization effect of deep neural networks. Specifically, in Step 1, a classifier is initialized by exploiting the memorization effect via early stopping, which is used to identity confident exam- ples. Later, the classifier and confident examples are iteratively refined and updated, respectively, which is the positive cycle we have mentioned before. Note that this cycle also depends the memorization effect to update confident examples and refine classifiers. Our method is named as momentum of memorization (Me-Momentum) because it also uses the trick of momentum to better exploit the memorization effect. Specifically, we can think of the classifier as a particle traveling through the hypothesis space, getting acceleration from the updated extracted confident data. We exploit the previously extracted confident examples to help learn a better classifier, i.e., initializing the network by using the parameters of the classifier trained with the confident examples extracted in the previous round. The impact of the confident examples in a certain round will decay as we continue updating the classifier.
We intuitively answer the fifth question by employing Fig- ure 2 Hard confident examples will be progressively ex- tracted also due to the momentum trick, i.e., previously ex- tracted confident examples will help identify hard confident examples. Specifically, in Figure 2(a), we show the noisy training examples, where circles and triangles represent two classes respectively and the red ones denote the examples with incorrect labels. Let the dash line in Figure 2(b) de- note classifier learned by Me-Momentum. We can use it to extract confident examples, e.g., the green circles and blue triangles, according to Step 2 of Algorithm [ Compared with Figure ((a), we can see that hard confident examples close to the optimal boundary, i.e., the circle and triangles with red edges, will be extracted. Note that as better and better classifiers will be learned by Mc-Momentum, more and more hard confident examples will be extracted. An visualization is shown in Figure 4
(a) Noisy examples
(b) Confident examples
Figure 2: Illustrative of extracting hard confident examples.
The strategy of alternatively optimizing the classifier and updating the training examples is not new for dealing with label noise. For example, Joint Optim (Tanaka et al. 2018). Co-teaching (Han et al. [018b] Yu et al. 2019). and SELF (Nguyen et af.) (0220) are similar to ours. Specifically, Joint Optim and Co-teaching update the classifier with one step of stochastic gradient descent while SELF and the proposed method refine the classifier to be optimal with respect to the extracted confident examples. However, existing methods have not focused on extracting hard confident examples and thus are substantially different from this paper because they neglected the importance of avoiding the accumulated error caused by the single initialization of the classifier and the sample-selection bias. Experiments (e.g., Figures 3 and 4) show that Me-Momentum with the outer loop part (i.e., re-initialization of the classifier) contributes significantly to extracting hard confident examples and achieving high label precision. Note that without employing the ensemble technique, SELF is special case of Me-Momentum with only the inner loop part. We avoid to integrate the ensemble technique to the proposed method for simplicity and clarity.
Relation to existing work:
4

---

Under review as a conference paper at ICLR 2021
Mc-Momentum is similar to curriculum learning as it also leams from easy to difficult. How- ever, curriculum learning needs a predefined curriculum (sample weighting scheme), e.g., assigning big/small wights for confident/noisy data. If the curriculum is not available, some clean data is quired to learn a mentornet to provide a curriculum (Jiang et al. 2018) or a latent variable could be introduced by self-paced learning (Kumar al.] 2010) learn curriculum. Differently, Me- Momentum only based on noisy data and does not explicitly learn curriculum. Mc-Momentum also has similar flavor to active learning which tends to choose and label hard examples to learn from at each iteration. However, for active learning, no label information is available before the data chosen while Me-Momentum has noisy labels and have to consider the side-effect of label noise.
3 EXPERIMENTS
To verify the effectiveness of the proposed method, we do experiments on datasets with both synthetic and real-world label noise. Specifically, we manually corrupt MNIST (LeCun et al.) 1998). CIFARIO, and CIFARIOO (Krizhessky] 2009) with class-dependent label noise and instance- dependent label noise. We detail how to generate class-dependent and instance-dependent label noise in the supplementary material. We employ the real-world noisy dataset Clothing IM (Xiao et al. 2013). These datasets have been widely used in studies with noisy labels (Han et al. 2018b] Tanaka et al. 2018 Xia et al.) 2019).
Datasets:
For MNIST, CIFARIO, and CIFARIOO, we leave out 10% of the noisy training data noisy valida- tion data. Clothing IM contains one million noisy training images, which are crawled from shopping websites, labeling by surrounding text. Almost all existing work uses the 14k clean validation data in their experiments. To verify the robustness of the proposed method, we also employ noisy vali- dation data in our experiments. Specifically, 100k noisy data are randomly left as noisy validation data and the remaining 900k noisy data as the training data.
Baselines: Me-Momentum is compared against the following state-of- ofhee-----h---tt-- approaches. (1) Statis- tically consistent methods: Forward (Patrini et al. 2017), T-revision (Xia et al. (0119) and DMI (Xu
al. 2019): (2) Statistically inconsistent methods: Decoupling (Malach & Shalev-Shwart?
2017).
MentorNet (Jinn et al. 2018), Co-teaching (Han et al. (2018b). Joint Optim (Tanaka et al. 2018)
SELF (Nguyen et al., 2020) where MentorNet, Co-teaching and SELF use the idea of extracting
confident examples by employing the small loss trick.
Network structure and optimization: All the methods are implemented with default parameters by PyTorch v1.5. For the experiments on MNIST, CIFARIO. and CIFARI00, we set Nimer = 20, Nouter = 3, and follow the settings of T-revision (Xia et al., 2019). Specifically, LeNet-5, ResNet- 18 and ResNet-34 networks
used for MNIST, CIFARIO and CIFARIO0 respectively. We use SGD with momentum 0.9, weight decay 10-4 batch size 128, and an initial learning rate of 10-2 divided by 10 after the 40-th epoch and 80-th epoch respectively (we fix the learning rate of 10-2 for the early stopping method). Data augmentation is used with horizontal random flips and 32 x 32 random crops after padding pixels on each side. For Clothing MM, ResNet-50 is used. We set Niner = 6 and Nower 3. To show the effectiveness of the proposed method, we do experiments by randomly initializing the network and pre- training it by employing ImageNet, respectively. As the noisy training sample contains a large number of examples, we train the network for 5 epochs and choose the best one with the highest classification accuracy on the noisy validation set. We use SGD with momentum 0.9, weight decay 10-3, batch size 32, with a learning rate of 5 x 10-3, and divided it by 10 at the 3-rd and 5-th round in the inner loop. For each outer loop, the model will be randomly re-initialized (or replaced by pre-trained one). The learning rate will be reset to 5 x 10-3, For data augmentation, all images are resized to 256 x 256, horizontal random flipped, and cropped in the middle with size of 224 x 224.
Note that due to the page limit, some complementary experiments to Sections 3.1 and 3.2 and com- parison with SELF are put in the Appendix.
3.1 VERIFY MOMENTUM OF MEMORIZATION
In Section 2 we discussed that Me-Momnntum is implemented by fulfilling the positive cycle that better confident examples will result in a better classifier and that better classifier will identify
5

---

Under review as a conference paper at ICLR 2021
CIFARIO, Spmmetrio-419
(CIFARIE, Imataoo-4851
163 (CRFAXIO Stmmdrrk-445)
ICIFARION, Instancs-465
Labe Produee
72
15
(CIFRRIO, 4P55
Srands
20k
(CIFARIB, Inleees-445)
241
CCFFRRO0, S.tmndrii-40%
23.
(CIFARI00, Irsianee- 49%
Brends
Cuttina Tark Noater
Tur Nember
Neater
200
56 CIIARROO Symmetrio4855
(CIFARIS, Imtteo-4455
[CIFARIOO, Sommdrk--40)
(CIFARIOE, Instanes-445
20
Btennch
Rmands
Figure 3: We call one update of the classifier and the extracted confident examples as one round. We illustrate how the label precision of the extracted confident examples, the number of the extracted confident examples, and the classification accuracy of the classifier trained by using the extracted confident examples change during the training of Me-Momentum. We have three distinct peaks in these figures because we have set Nouter = 3 and the classifiers are re-initialized in the outer loop. The dash lines in the second row indicate the number of clean labels in the noisy training data.
better confident examples. In this subsection, we will empirically verify this positive cycle, which can be done the synthetic datasets as we have their ground-truth -ruth labels.
In Figure 3 we can see that in the inner loops (c.g.. rounds 0-5, rounds 6-9, and rounds 10-12 in the first column of figures represent three inner loops respectively), the classification accuracy generally increases (note that the figures are not smooth because the classifiers are tested on the unseen test data) and the number of extracted confident examples clearly increases (although their label precision decreases slightly). We can also see that in the outer loops (c.g., rounds O, 6, and 10 in the first column of the figures consist of an outer loop), the classification accuracy clearly increases and the label precision of the extracted confident examples clearly increases (although the number of extracted confident examples decreases slightly). This implies that compared with previous classifiers and extracted confident examples, better ones are obtained, which empirically justifies the positive cycle. Note that the classification accuracy in the outer loop is low because the models are re-initialized.
Figure 3 also shows the importance of the outer loop of Me-Momentum. We can see that the label precision of the extracted confident examples slightly decreases in the inner loops. This is because the deep model gradually memorizes the noisy labels as we continually refine This issue can be handled by re-initialize the deep model in the outer loop. Specifically, by re- re-initializing the model in the outer loop, the label precision of the extracted confident examples increases significantly. Although the number of the extracted confident examples decreases, the overall quality of the extracted confident examples is increasing as evidenced by the increase of the classification accuracy of the classifiers trained on the extracted confident data. Note that the low data quality in the first run of the inner loop also justifies that a single deep model initialization may lead to sample selection bias.
see from Figure that
There are interesting observations of the proposed method that the number of extracted confident examples is close to the number of accurately labeled data in the training set and that the label precision of the extracted confident examples is quite high, i.c., almost all are above 90%. This empirically proves that Me-Momentum is powerful in extracting confident examples. the next subsection, we will visualize that Me-Momentum is also good at extracting hard confident examples.
3.2 VISUALIZE HARD CONFIDENT EXAMPLES
To justify that Mc-Momentum is able to extract hard confident examples, we visualize the extracted confident examples by employing t-SNE (Maaten & Hinton| 2008). Specifically, we show how the

---

Under review as a conference paper at ICLR 2021
(MNIST, 5)mmetric-0%%
(MNIST, Symmetric-40%)
(MNIST, Instamce-40%)
(MNIST, Instance-40%)
(CIF LIFARIO, Sy5met..mmmerrr14555 (CIFARIO, Symmetric-40%)
(CIFARIO, Instance-40%)
(CIFARIO, Instance-40%
(CIF CIFARIO, Symmetric-4PP) (CIFARIOO, Symmetrie-40%
(CIFAR100, Instance-40%) 40%)
(CIF 1R100 Instance-40%)
Figure 4: Visualization of the extracted confident examples. The first and third columns are about the confident data extracted in the first run of the inner loop; while the second the fourth columns are about the confident data extracted in the outer loop. Specifically, green dots represent the data selected in the first round. Blue and red dots represent the new extracted data in the middle and the end rounds respectively. Large figures for CIFARI00 are provided in the supplementary material.
confident examples progressively extracted in the inner and outer loops. The results shown in Figure where green, blue, and red dots represent confident examples extracted at the beginning, middle, and end rounds of the loops, respectively. On the datasets of MNIST and CIFARIO, we can clearly see that the blue and red dots are mostly located at the boundaries of the clusters of green dots. Although the figures of CIFARI00 are small, we can also clearly see that there are lots of blue and red dots which are outside of the green clusters in the second and fourth figures. This supports and justifies our claim that Me-Momentum is able to extract hard confident examples (those are close to the decision boundary).
Comparing the extracted results of the first run of the inner loop (the first and third columns) with those of the outer loop (the second and fourth columns), we can find that the cluster boundaries in the latter are more clear. This further justifies that why better classification performance can be obtained by re-initialization in the outer loop. Comparing the confident examples extracted the class-deped dlass-de label noise datasets with those on the instance-dependent label noise datasets, we see that the proposed method is not very sensitive the type of label noise and can work well on the most general instance-dependent label noise cases.
3.3 CLASSIFICATION ACCURACY
Synthetic data To evaluate the classification performance of Me-Momenuum, we first conduct ex- periments on MNIST, CIFARIO, and CIFARI00 with class-dependent and instance-dependent label noise. Fach trial repeated five times. The results are presented in Tables I 2 and S respectively. Me-Momentum consistently outperforms the baselines. Specifically, CIFARIO0 is the most chal- lenging one among the three datasets. On CIFARI00, Mc-Momentum outperforms the basclines by clear margin across all the settings shown in Table 3 Note that the performance gain in Me-Momentum is caused by the improvement of the quality of the extracted confident examples.
In the baselines, Co-teaching, Joint Optim, and T-revision are the representative methods that learn robust classifiers by extracting confident examples, refining the noisy labels, and exploiting the noise transition matrix, respectively. Note that Co-teaching keeps updating constant number of confident

---

Under review as a conference paper at ICLR 2021
Table I: Means and standard deviations of classification accuracy on MNIST'
Flipping Rate
Decoupling
NentorNet
Co-teaching
Forward
Joint Optm
DMI
T-revision
Ours
Sym-20%
95.39%
96.37%
97.22%
22%
98.38%
98.92%
98.91%
98.94%
10.29%
20.18%
20.18%
20.0%%
20.15%
20.11%
1:0.04%
20.13%
Sym 40%
90.77%
96.16%
94.64%
96.71%
98.12%
98.63%
98.42%
98.66%
20.77%
20.49%
20.33%
20.16%
20.06%
10.11%
10047%%
20.07%
Inst 20%
96.945%
94.66%
95.37%
95.89%
98.10%
98.75%
97.12%
98.96%
10.07%
10.35%
20,08%
20.12%
20.14%
10.11%
1:0.09/
20,061
Inst 40%
54.98%
88.51%
90.06%
88.95%
92.00%
97.58%
98.11%
+0.27%
+00.66%
-0.81%
-2.7%
+1.39%
+0.822%
+-0.66%
+0.35%
Table 2: Means and standard deviations of classification accuracy on CIEARIO
Plipping-He Kate
Decoupling
NentorNet
Co-teaching
Forward
Joint Optmm
DMI
T-revision
Ours
Sym-20%
79.83%
80,49%
87.16%
83.63%
89.70%
88.18
89.63%
91,44%
+0.30%
+0.52%
=0.11%
=0.2%%
+0.11%
10.36%
10.133
20.33%
Sym-41%
69.47%
77.48%
83.59%
74.30%
87.79%
83.98%
86.81%
88.39%
+0.87%
+3445%
=0.28%
-0.26%
+0.20%
10.48%
1:0.21%
20.34%
Inst 20%
77.83%
79.12%
86.54%
85.29%
89.69%
89.14%
90.46%
90.86% %
+0.233%
+0.42%
-0.11%
-0.38%
+0.42%
+0.36%
+-0.13%
+0..21%
Inst-K0%
39.05%
70.27%
80.98%
74.72%
82.62%
84.78
85.37%
86.66 %
+0073%
+1.52%
-0.39%
3.24%
+0.57%
+1.97%
+3.366
+0.91%
Table 3: Means and standard deviations of classification accuracy on CIFAR/OO
Flipping-Rate
Decoupling
MentorNet
Co-teaching
Forward
Joint Optinn
DMI
T-revision
Outs
Sym 20%
42.75%
52.11%
59.28%
57.75%
64.35%
58.73%
65.40%
68.03%
=0.49%
+0.10%
#0.47%
#0.37%
+0.38%
=0.70%
+1.0%%
+0.53%
Sym-40%
37.13%
35.12%
51.60%
38.59%
57,97%
49.81%
57.71%
63.48%
=0.91%
+1.13%
#0.49%
+1.62%
#0.67%
=1.22%
#0.84%
+0.72%
Inst 20%%
48.33%
51.73%
57.24%
58.76%
65.15%
58.05%
60.71%
68.11%
-0.35%
4-0.17%
+0.699
+0.66%
4.0.31%
-0.20%
+0.73%
+.0.77%
Inst-40F%
34.20%
40,90%
45.60%
44.50%
55.57%
47.36%
51.48
58.38%
+0.59%
+-0.45%
+0.99%
+0..29%
4.0.41%
-0.68%
40,91%
1.28%%
examples from the mini-batches used in SGD. We therefore do not compare with its extracted con- fident examples in Section [3.2 as our method extracts confident examples from the whole training data at once. By comparing the classification performance, we can clearly see that the proposed method is much more powerful in extracting confident examples. Note that Joint Optim and T- revision employ all training data to train the classifiers; while our method only employs confident examples and discards the unconfident ones. The results further justify that Me-Momentum is able to extract high-quality confident examples. Note that the performance of Me-Momentum could be further improved by correcting the unconfident data with the idea of Joint Optim.
Real-world dataset We compare Me-Momentum with baseline methods on Clothing in Table 4 where "pre-trainedd and "scratch" mean the network was pre-trained by employing ImageNet and ini- tialized randomly, respectively; "clean" and "noisy" means the validation data is clean and noisy re- spectively. First, it is observed that Me-Momentum works well with noisy validation, even surpassing many baselines with clean validation. For fair comparison. we also use clean data to validate our method, which achieves the highest test accuracy of 75. 18%, better than T-revision by 1% and Joint Optim by 2.95%. Note that Forward and T-revision need the 50k clean data for estimating transition ma- trix, while Me-Momentum does not need any clean data for training. In addition, to show the robustness of ne-MMomentum, we conduct experiments with ResNet-50 from scratch, which achieves the second best accuracy.
Table 4: Classification accuracy on Cloth-
ing M.
Method
Validation
Accuracy
Cross Entropy
Clean
69.54%
Decoupling
Clean
53.98%
MentorNet
Clean
36.77%
Co-teaching
Clean
58.68%
Forward
Clean
69.84%
Joint Ophi
Clean
72.23%
DMI
Clean
72.46%
T-revision
Clean
74.18%
Ours (pre-trained)
Noisy
73.13%
Ours (scratch)
Clean
74.75%
Ours (pre-trained)
Clean
75.18%
4 CONCLUSION
In this paper, we propose method called Me-Momentum that is able to extract hard confident examples from noisily labeled data by exploiting the memorization effect of deep neural networks. In high level, it fulfils positive cycle that better confident examples will result in better classifier and that a better classifier will identify better confident examples. We have empirically verified its effectiveness by analyzing the statistics of the extracted examples, visualizing the hard confident examples, and comparing its classification performance with state-of-the-ar baselines. In future, we can extend our work by utilizing and exploiting the unconfident examples.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, and Yoshua Bengio. A closer look at memorization deep networks. In ICML, pp. 233- 242, 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning. In ICML, pp. 41-48, 2009.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, pp. 1062-1070, 2019.
Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with Mark Dredze, Koby Crammer, and Fernando Pereira. Confidence-weighted linear classification. In ICML, pp. 264-271, 2008.
bounded instance-and label-dependent label noise. In ICMI, 2020.
Xavier Gastaldi.
Shake-shake regularization. arXiv preprint arXiv: 2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In ECCV, pp. 135-150, 2018.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In NeurIPS, pp. 5836-5846, 2018a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, pp. 8527-8537, 2018b.
Warren He, Bo Li, and Dawn Song. Decision boundary analysis of adversarial examples. In ICLR, 2018.
Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. Active learning by querying informative and representative examples. In NeurIPS, pp. 892-900, 2010.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, pp. 2309-2318, 2018.
Jan Kremer, Fei Sha, and Christian lgel. Robust active label correction. In AISTATS, pp. 308-316, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NeurlPS, pp. 1189-1197, 2010.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The MNIST database of handwritten 1998.
digits. htt::/yaam.eecum.com/eelht
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparamettrrize neural networks. A/STATS, 2020.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Leaminng from noisy labels with distillation. In ICCV, pp. 1910-1918, 2017.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. JEEE Transactions pattern analysis and machine intelligence, 38(3):447-46 2016.

---

Under review as a conference paper at ICLR 2021
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In ICML, 2020.
Xingjun Ma, Yisen Wang. Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-dee -drvee learning with noisy labels. In ICML, pp.
2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 95Nov)2577--2655 2008.
Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". In NeurIPS, pp. 960-970, 2017.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, 2018.
Nagarajan Natarajan, Inderjit $ Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurlPS, pp. 1196-1204, 2013.
Tam Nguyen, Mummadi, T Ngo, L Beggel, and Thomas Brox. SELF: learning to filter noisy labels with selffensemblingg In ICLR, 2020.
Curtis G Northcutt, Tailin Wu, and Isaac L Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. UAI, 2017.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust label noise: loss correction approach. In CVPR, pp. 1952-1952, 2017.
Scott E Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR, 2015.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331-4340, 2018.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, pp. 838-846, 2015.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Prestopping: How does early stopping help generalization against label noise? arXiv preprint
2019.
lya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In ICML, pp. 1139-11411 2013
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In CVPR, pp. 5552-5560, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve seme-supe deep learning results. In NeurIPS, pp. 1194-1204, 2017.
Kiran K Thekumparampil, Ashish Khetan, Zinan Lin, and Sewoong Oh. Robustness of conditional gans noisy labels. In NeurlPS, pp. 10271-10220 2018.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In ICML, pp. 6234 6243, 2019.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In NewrlPS, pp. 5596-5605, 2017.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In CVPR, pp. 839-847, 2017.
10

---

Under review as a conference paper at ICLR 2021
Yixin Wang, Alp Kucukelbir, and David M Blei. Robust probabilistic modeling with bayesian data reweighting. In ICML, pp. 3646-3655. JMLR. org, 2017.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable label-noise learning? In NeurIPS, pp. 6835-6846, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Parts-dependent label noise: Towards instance-dependent label noise. arXiv preprint arXiv:2006.078.6 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. I_dmi: A novel information-thheoreiee loss function for training deep nets robust to label noise. In NeurIPS, pp. 6222-6233, 2019.
Xingrui Yu, Bo Han, Jiangehao Yao. Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement benefit co-teaching? In ICML, 2019.
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In ECCV, pp. 68 83, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NewrPPS, pp. 8778-8788, 2018.

---

Under review as a conference paper at ICLR 2021
A EARLY STOPPING
We discuss the early stopping trick used in Step 1 of Algorithm T We repeat explaining the trick here again. Specifically, we introduce a definition of high-peak. A noisy validation accuracy at the i- th epoch is called an i-th high-peak if it achieves the highest accuracy in the epoch range (1,..., .).. Assume the i-th and j-th high-peaks occur next to each other. having noisy validation accuracies of a and respectively. The training early stops if (6 aj/(j i) I т, where 7 a hyper-parameter. In the experiments, we set = 0.1, which empirically works well across all datasets. In Figure 5 we compare the difference between the early stopping method and the traditional validation method.
(CIFARIO, Symmetrie-20%%
(CIFAR OIFOR Symmetric-40%)
(CIFARIO, Instance-20%)
(CIFARIO, Instance-40%)
10
70
train acc
val acc
Accrracy
70
60
highest
Accerscy
50
Accurncy 100
80
Acerray
50
50
80
40
80
-3
30
200
Epoch
Epoch
Epoch
Epoch
Figure 5: Comparing the difference between the carly stopping method in Step and the traditional validation method where the classifier with the highest validation accuracy during the whole training procedure will be output. The green dash line indicates the epoch at which early stopping happens; while the orange dash line indicates the epoch at which the highest validation accuracy is achieved during the whole training procedure. In the third plot, the two dash lines are identical to each other.
CIFAR10, Early stop t
10%
Iss 20%
Sym 40%
Inx 40%
52
test accuracy
50
0.5
Figure 6: Illustrative of extracting hard confident examples.
We also study the sensitivity of the hyper-parameter. Specifically, we study its sensitivity on CI- FARIO by setting T to be the values in the range (0.05, 0.07, 0.1, 0.3, 0.5). Other settings are the same as those in this paper. The results are presented in Figure 0 We can see that the classification performance of Mc-Momentum is robust and not sensitive to the change of the value of
B GENERATING LABEL NOISE
Transition matrix The transition matrix T(x) was proposed to explicitly model the generation process of label noise, where Tiy(2) Pr( = = X = x). Pr(A) denotes as the probability of the event A, X as the random variable for the instance, as the noisy label, and Y as the latent clean label. In high level, the ij entry of the transition matrix denote the probability that the instance will flip from the clean class j to the noisy class i.
Symmetric class-dependent label noise The label noise is generated according to symmetric class- dependent and instance-inde indeeeedeendeee noise transition matrices. If the flip rate is a, the diagonal entries of a symmetric transition matrix are and the off-diagonal entries are a/ (c 1).
12

---

Under review as a conference paper at ICLR 2021
Instance-dependent label noise We generate the instance instance-dependent label noise according to the following Algorithm 2 Xia et al. (2020). More details about this algorithm can be found in Xia et al. 2020
Algorithm Instance-dependent Label Noise Generation
Input: Clean samples { 2/) j 14 Noise rate
1: Sample instance flip rates E R" from the truncated normal distribution (T, 0.1?. [0, 1));
2: Independently sample w1,, W2, ...,, We from the standard normal distribution (0, 12);
3: For i = 1,2,..., do
4: p 0; W'ys
// generate instance-dependent flip rates
5: Py: =o;
I/ control the diagonal entry of the instance-dependen transition matrix // make the sum of the off-diagonal entries of the y:-th row to be q: // set the diagonal entry to be 1 - 9i 8: Randomly choose a label from the label space according to possibilities p as noisy label y::
6: p = q: softmar(p):
7: Py. = =1 q:i
9: End for.
Output: Noisy samples {( 11)
C COMPARE RESULTS WITH SELF
The performance of our re-implemented SELF (Tarvainen & Valpola 2017
good that in the original paper. For a fair comparison, we change our backbone consistently with SELF and com- pare with the results from the original paper directly (without Mcan Teacher (Tarvainen Valpola 2017 ). Specifically, the network is changed to ResNet26 with Shake-shake regularization (Gastaldi 2017 and the learning rate is changed 0.05 and weight decay 2c-4.
Table shows that Me-Momentun outperforms SELF by a large margin in CIFARIO and CI- FAR100 with Symmetric 40% and Symmetric 60% noise. The gap between the performance of Me-Momentum and SELF becomes larger in Symmetric 60% in both datasets compared with Sym- metric 40% because hard confident examples play more important role in more noisy datasets. Specifically, both of the methods are based on the same backbone with Cross-Entrops loss, so the improvement of Me-Momentum can only be as result the quality of extracted confident exam- ples. Therefore, Me-Momentum is able to extract better hard confident examples than SELF.
Table 5: Means and standard deviations of classification accuracy compared with SELF
CIFAR 10 Sym-40
CIFAR 10 Sym-60
CIFARI00 Sym-40
CIFARI00 Sym-60
SELF
87.35%
75.47%
61. 11.40%
50.60%
Ours
92.31 %
87.88 %
68.25%
59.51 %
Furthermore, results in Table 5 shows that the performance of Me-Momentum can be improved by changing a better backbone. However, to show the effectiveness of the proposed method and avoid complexity, we simply choose in the paper the standard CNN network.
D EXPERIMENTS COMPLEMENTARY TO SECTION 3.1
In Section 3.1, discuss the statistics of the extracted confident examples. However, due the space limit, in the paper, we only provide parts the empirical results. In this supplementary material, provide the statisties all the employed datasets and settings.
13

---

Under review as a conference paper at ICLR 2021
(MNIST; Symmetric-0%%)
(MNIST, Symeerrie20%%
(MNIST, Symmetrie-0%%
100
Canfadent Data Number
99
04k
94
Labh Prochion
98
42K
98
07
60k
97
96
38k
96
90
10
12
304
95
10
12
Rounds
Rounds
Rounds
(MNIST, I25t))ce-20%)
(MNIST, Instance-20%)
(MNIST, Instance-20%)
Caalideat Dain Number
Prodiion
97
40k
96
28k
96
0.0
2.5
5.0
7.5
20.0
12.5
3.00
2.5
5.0
7.5
10.0
12.5
0.0
2.5
5.0
7.5
10.0
12.5
Rounds
Rounds
Rounds
(MNIST, Symmetric40%%)
(MNIST, 5ymeetri-40%%
(MNIST, 3ymmetric440%
100
34k
100
99
Conlidet Datn Dain Namber
33k
Precision
32k
23k
36k
29k
8.00
280
8.0
Instance-40%)
(MNIST, Instance-40%)
Instance-40%)
100
34k
100
98
Canidtar Dats Nazber
335
99
Text Accuracy
32k
96
31k
1abel
92
88
5.0
7.5
12.5
28.
7.5 Rounds
12.5
%6.0
2.5
10.0
12.5
Rounds
Rounds
Figure 7: Statistics of the extracted confident examples on MNIST by Me-Momentum. We call one update of the classifier and the extracted confident examples one round. We illustrate how the label precision the extracted confident examples, the number the extracted confident examples, and the classification accuracy of the classifier trained by using the extracted confident examples change during the training of Me-Momentum. The dash lines in the middle column indicate the number of clean labels in the noisy training data.
14

---

Under review as a conference paper at ICLR 2021
(CIFARIO, Symmetric-20%)
(CIFARIO, S04.1000-mmmttrr2225955
(CIFARIO, Symmetric-20%)
37k
100
36k
80
35k
98
088
34k
97
88
33k
96
84
20
32%
15
20
95
1.0 Rounds
20
Rounds
Rounds
(CIFARIO, Insfance-20%)
(CIFARIO, Instance-20%)
(CIFARIO, Insfance-20%)
Number
80
088
33k
96
84
10
20
32%
10
20
95
10
20
Rounds
Rounds
Rounds
(CIFARIO, Symmetric-0%%)
(CIFARIO, Symmetric-40%)
(CIFARIO, Symmetrie-00%)
90
28k
Confifent Daits Namber
27k
86
Tabel Prediion Predioon
14
82
80
78
Rounds
(CIFARIO, Instance-40%
(CIFARIO,
(CIFAR10, Instance-40%
90
30k
98
Confolent Data Namber
29k
96
Test Accurscy
86
Label Preciios
84
82
92
90
78
76 Rounds
15
20
24k
10
20
88
1.0
15
20
Rounds
Rounds
Figure 8: Statistics of the extracted confident examples on CIFARIO by Me-Momentum. We call one update of the classifier and the extracted confident examples one round. We illustrate how the label precision the extracted confident examples, the number the extracted confident examples, and the classification accuracy of the classifier trained using the extracted confident examples change during the training Mc-Mommntum. We have three distinct peaks in these figures because have set Notter and the classifiers re-initialized in the outer loop. The dash lines in the middle column indicate the number of clean labels in the noisy training data.
15

---

Under review as a conference paper at ICLR 2021
(CIFARI00, Symmetrii-200)
(CIFARI00, Symmelric-20%
(CIFARI00, Symmetrie-20%.
38kk
100
W
66
LaciiPecaiann
64
98
62
97
88
96
56
15
20
22%
15
20
95
10
15
20
Rounds
Rounds
Rounds
(CIFARI00, Instance-20%)
(CIFARI0O, Instance-20%)
(CIFARI00, Instance-20%)
68
66
96
64
30k
95
60
28k
94
58
24k
26k
93
10
3.5
20
25
22k
10
15
20
25
92
10
3.5
20
25
Rounds
Rounds
Rounds
(CIFARI00, Symmetric-40%
(CIFARI00, Symmeetrc-40%%)
(CIFARI00, Symmetric-40%
28k
100
VV
Confolent Datrs Dain Namber
26k
60
24k
Preddton
66
22k
54
20k
(CIFARIO0,
Instance-40%%
(CIFARI00,
(CIFARI00, Instance-40%)
62
28k
98
6.0
Confilent Data Number
26k
Test Accuracy
Label Prediion
45g
155
166
15
82g
15
Rounds
Rounds
Rounds
Figure 9: Statistics of the extracted confident examples on CIFARIO0 by Me-Momentum. We call one update of the classifier and the extracted confident examples one round. We illustrate how the label precision of the extracted confident examples, the number the extracted confident examples, and the classification accuracy of the classifier trained by using the extracted confident examples change during the training of Me-Moomnnum. We have three distinct peaks in these figures because have set N and the classifiers re-initialized in the outer loop. The dash lines in the middle column indicate the number of clean labels in the noisy training data.
E EXPERIMENTS COMPLEMENTARY TO SECTION 3.2
In Section 3.2, we have visualize the extracted confident examples by using t-SNE. It verifies that the proposed Me-Momentum is effective to extract hard confident examples. However, due the space limit, in the paper, only provide parts of the empirical results. In this supplementary material, we provide the visualization on all the employed datascts and settings.
Specifically, we show how the confident examples progressively extracted in the inner and outer loops. In the figures, green, blue, and red dots represent confident examples extracted at the begin- ning, middle, and end rounds of the loops, respectively.
On the datasets of MNIST and CIFARIO, we can clearly see that the blue and red dots are mostly located at the boundaries of the clusters of green dots. On CIFARI00, also clearly that
16

---

Under review as a conference paper at ICLR 2021
there are lots of blue and red dots which are outside of the green clusters in the second and fourth figures. This supports and justifies our claim that Mc-Momentum is able to extract hard confident examples (those are close to the decision boundary).
The figures are presented in the next and the following pages.
17

---

Under review as a conference paper at ICLR 2021
(MNIST, Symmetric-20%)
(MNNIST,Smmeetrie,, Symmetric-20%%
(MNIST, Instance-20%)
(MNIST, Instance-20%
(MNIST, Symmetric-40%
(MNIST, Symmetric-40%
(MNIST, Instance-40%)
(MNIST, Instance-40%)
Figure 10: Visualization of the extracted confident examples on MNIST. The first column is about the confident data extracted in the first run of the inner loop; while the second column is about the confident data extracted in the outer loop.
18

---

Under review as a conference paper at ICLR 2021
(CIFARIO, Symmetric-20%)
(CIFARIO, Symmetric-00%
(CIFAR10, Instance-20% 20%
(CIFARIO, Instance-20%)
(CIFARIO, Symmetric-40%
(CIFARI0, Symmetric-40%
(CIFAR10, Instance-40%
(CIFAR10, Instance-40%)
Figure 11: Visualization of the extracted confident examples CIFARIO. The first column is about the confident data extracted in the first run of the inner loop; while the second column is about the confident data extracted in the outer loop.
19

---

Under review as a conference paper at ICLR 2021
(CIFAR100, Symmetric-20% (CIFAR100, Sy0m)tric-20%)
(CIFAR100, fetaacce22%%
(CIFAR100, mastance-20%
Visualization of the extracted confident examples CIFAR100. The first column is about the confident data extracted in the first run of the inner loop; while the second column is about the confident data extracted in the outer loop.
Figure 12:
20

---

Under review as a conference paper at ICLR 2021
(CIFAR100, Symmetric-40% (CIFAR100, Symmetric-40%
(CIFAR100, Instance-4N9 -00%
(CIFAR100, Instance-40%)
Visualization of the extracted confident examples CIFAR100. The first column is about the confident data extracted in the first run of the inner loop; while the second column is about the confident data extracted in the outer loop.
Figure 13:
21


Reviews:
[{"review_id": "svTd45rxxj1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*quality*\nThe organization of this paper is barely satisfactory. The technique proposed to extract hard confident examples in this paper is not convincing, even though the experimental results seem promising.\n\n*clarity*\nIt is not difficult to understand the proposed method, however, the Figures in this paper are somewhat confusing to readers.\n\n*originality*\nIn this paper, the authors focus on extracting hard confident examples from the noisy training data for learning with noisy labels. There is no method to extract hard confident examples before, the idea is novel. However, the technical contribution is limited.\n\n*significance*\nThe idea of extracting hard confident examples is good, however, I do not think this paper solves this issue properly. How to extract hard confident examples correctly still remains a challenging problem.\n\n*pros and cons*\nPros:\n(1). Authors provide sufficient experiments which evaluate the methods on multiple datasets.\n(2). The proposed method achieves better performance than state-of-the-art methods. \n\nCons:\n(1). As for the question “How to validate the learned classifiers in Steps 3 and 5 without a clean validation set?”, the authors claim that the noisy validation set could be used as a surrogate to validate the classifiers if no clean validation set is available. I do not think it is reasonable here.\n(2). The confident examples can be extracted based on the memorization effect, it seems reasonable. But why hard confident examples can be extracted? The authors explained as previously extracted confident examples will help identify hard confident examples, it is so empirical and maybe another explanation is needed here. Moreover, in the corresponding paragraph, there might be some error with “Figure 6” (should be “Figure 2”?).\n(3). The intuition “better confident examples will result in a better classifier and a better classifier will identify better confident examples” have been mentioned many times, I wonder will the method proposed in this paper stuck in the local optimum when the two loops in Algorithm 1 execute.\n(4). The proposed method performs better than the state-of-the-art with 20% and 40% noisy labels. It would be better if the experiments with lager noise rate (e.g., >=50%) are also conducted.\n(5). Some minor issues, for example, “extract” should be “extracted” in the step 2 of Algorithm 1; “$\\times$” and “*” are abused in Section 3; and the resolution of some Figures in this paper is not satisfactory.\n\nGenerally, I feel that the method proposed in this paper is somehow too empirical, and more theoretical study is needed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nFczKfgmm9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes momentum of memorization as a way to distinguish hard examples needed for efficient learning from noisy examples which decrease classification accuracy. The method finds confident, hard examples and updates them dynamically during model training. This is done by iteratively selecting examples with labels that agree with model predictions and then training on only the confident data. Results show improved accuracy on standard image classification datasets with both synthetic and real world label noise.\n\n\nNovelty\n- In addition to the methods cited in \"Relation to existing work,\" the authors should cite and discuss self-training methods which have become quite popular over the past decade [1]\n- The paper also describes Algorithm 1 as adding an outer loop to the recent SELF algorithm [2], which considerably limits its novelty.\n\nCorrectness/Experiments:\n- Results show consistent improvement over baselines\n- A more thorough evaluation would include ablation on the parameters $N_{outer}$ and $N_{inner}$, as well as the modified early stopping \"trick\"\n- Figure 4 shows qualitative t-SNE visualizations of identified confident examples. qualitatively, these look well separated. However, t-SNE is known to preserve local structure better than global structure, making it a poor way to visualize how close confident examples are to decision boundaries. It's also impossible to tell whether any of the dots (in particular blue and red) are incorrectly labeled wrt ground truth class labels (i.e. \"noisy\" vs \"hard\")\n- Many claims are made without theoretical or empirical justification. For example, showing how training/validation accuracy varied with $N_{outer}$ and $N_{inner}$ would support the claim that Me-Momentum is leveraging the memorization effect.\n\nClarity\n- Example figures and problem formulation are quite nice\n- I believe this naming may cause confusion with the momentum update widely used in optimization. Additionally, I am not sure how this paper fits the notion of momentum more than any number of other training procedures. For example, increasing the learning rate at each epoch would fit the colloquial/physics meaning of \"traveling through hypothesis space\" just as well as this approach which uses repeated fine tuning.\n- Algorithm 1 and Section 2 would benefit from additional notation in addition to the pseudocode and text explanation\n- Unclear what the reader should take away from Figure 4 and 10-13 (see above)\n- typos: \"an surrogate\" should be \"a surrogate\", and \"clothing1M\" should be \"Clothing1M\"\n\nPro\n- Nice illustrations and problem description\n- Results perform well against baselines\n\nCons\n- Limited novelty compared to SELF and self-training methods more generally\n- Concerns about methodology and clarity, especially wrt visualizations\n- Empirical methodology/analysis should be improved\n\nQuestions:\n- How does the running time compare to normal neural network training, and to other baseline methods?\n- Do Tables 1-3 show a similar improvement to Table 4 when switching to clean validation set?\n- How sensitive is Me-Momentum to hyperparameters $N_{outer}$ and $N_{inner}$?\n- Is the recall of confident examples wrt accurately labeled data as high as the precision? Will identifying additional confident samples lead to even higher accuracy?\n\n\n[1] Huang and Harper. Self-Training PCFG Grammars with Latent Annotations\nAcross Languages, EMNLP 2009 https://www.aclweb.org/anthology/D09-1087.pdf\n\n[2] Nguyen et al. SELF: learning to filter noisy labels with self-ensembling, ICLR 2020. https://arxiv.org/abs/1910.01842\n\nEDIT: The author response addressed some of my concerns. In particular, it confirms that the experimental results are impressive compared to many baselines. However, I would appreciate the distinction between easy and hard confident examples much more if the authors went beyond illustrative figures and defined this concept more precisely. Without a precise definition, it's difficult to verify the paper's claims about why the method performs well. Based on t-SNE visualizations, the author response offers an alternate definition of \"far away from the cluster centroids.\" The submission would be much stronger if it developed this idea further and analyzed it quantitatively.\n\nNext, the authors suggest that methods cannot distinguish hard confident examples from mislabeled examples using the \"small loss trick\" alone, and that their \"momentum trick\" is necessary. However, they do not present a principled argument or strong evidence to support the claim. In fact, some recent methods do show a separation between these types of training data using measurable quantities, see Figure 1 of [3] and Figures 1-2 of [4].\n\nFinally, the authors claim that reinitialization helps escape bad local optima. However, I do not see how low standard deviation supports this claim.\n\n[3] Pleiss et al. Identifying Mislabeled Data using the Area Under the Margin Ranking, Neurips 2020. https://arxiv.org/abs/2001.10528\n\n[4] Swayamdipta et al. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics, EMNLP 2020. https://arxiv.org/abs/2009.10795]", "extra_scores": {}, "extra_text": {}}, {"review_id": "XLb9bkVW5V0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce an interesting approach to handling hard \"confident\" samples in learning with label noises. At the heart of the proposed approach is an interactive method that jointly refines the classifier and the samples. The confident samples are initialized by utilizing the memorization effect of deep networks. Then, a classifier is learned from such samples. \n\nDuring the learning process, hard confident data are selected progressively by looking at the classification results, which further better select confident samples. \n\nExperimental results show that the proposed method achieves state of the art.\n\nPros:\n1. The task studied here could be of interest to a large number of readers in the community.\n2. The overall idea is quite interesting and intuitively makes sense. I like the fact that the core idea of the proposed approach finds its root in physics. \n3. The results are very promising, in spite of the simple nature.\n4. The manuscript is overall well-written and easy to follow. \n\nCons:\n1. The iterative nature is good and bad. It seems to me that sometimes it requires many rounds in the inner loop, as shown in Fig. 3, for example, the number is up to 20 to 30. Please show some numbers in terms of running time and compare them with the baselines. \n2. I might be missing something here but, what about the not-so-confident samples? Any scheme to take care of them?\n\nMinor issues\n1. The fonts in Fig. 3 should be enlarged.\n2. It would be better if the author could elaborate more on its connection with the momentum in physics.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGZl9qGdqhH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose a novel and effective method called Me-Momentum to cope with noisy labels. The algorithm borrows the idea of momentum from physics and tries to identify hard examples. The authors alternately update the hard examples and improve the classifier to achieve the robustness to noisy labels. Experiments and comparisons with recent state-of-art methods are provided to verify the effectiveness of Me-Momentum. \n\nPros:\n\n1. Different from the existing methods, which aim to identify simple clean examples, this paper analyzes the importance of hard examples and provide a way to identify them. The method is interesting and effective. It gives a new perspective and makes clear contributions for learning with noisy labels. \n\n2. The design of inner loop and outer loop is interesting and insightful, and is proved to be very effective. \n\n3. The experimental results are promising. The proposed method achieves great performance (75.18%) on Clothing1M. In addition, compared with SOTA SELF,  Me-Momentum outperforms it by a large margin. The authors also provide an ablation study to analyze the sensitivity of the hyperparameter $tau$. Thus, the significance of the proposed method with respect to experimental results may be high in the community. \n\n4. The paper is well written. The description of the technical details is very clear. It is easy to reproduce this method.\n\nSpecific comments and questions:\n\n1. Insufficient analysis of visualization results and analysis of hyperparameter sensitivity. The authors should add them to improve this paper. In addition, the baselines are reimplemented with default parameters? I hope the author can emphasize or add some descriptions of the implementation details of the comparison methods. \n\n2. Adding more baselines will be better. Some recent methods achieve great classification performance for learning with noisy labels, such as [1]. The authors can compare the proposed method with them to make the results more convincing. \n\n3. The proposed method uses a noisy validation set to choose classifiers, and then identify confident examples with robust classifier. The authors should explain why a noisy validation dataset can be used to choose classifiers which perform well on clean datasets. This choice may not be accurate?\n\n4. There are some grammatical errors and typos. The author should proofread this paper. For example, “The results are presented in Figure XX”, the authors miss the figure number. Please check it carefully. \n\n[1] NLNL: Negative Learning for Noisy Labels, Youngdong Kim et al, ICCV2019. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a process to leverage the memorization effect of deep learning models to filter out examples at the boundary (hard) that the models are confident on, and argue that identifying those hard confident examples help improve the accuracy when learning under noisy data. The process essentially alternates between confident example selection and classifier updating, where the two parts are expected to help each other to form a positive cycle. Experiments demonstrate superior results over other self-purifying approaches.

The reviewers have a very diverse opinion about the paper. On the positive side, everyone agrees that the superior experimental results to be very impressive. The authors have addressed some concerns well, such as the running time. One reviewer pointed out that the current study has not been combined with semi-supervised learning yet, but during the discussion, most agreed that it is not a crucial negative point of the current paper. On the actual negative points, there are issues that were not cleared even after the rebuttal, such as whether re-initialization in the process helps escape local optimal, and the key difference between the "small-loss trick" and "memory-momentum trick." While the authors argued the novelty with respect to SELF, more illustrations and experiments are needed to highlight the novelty aspect.

Given the diverse opinions, the AC read the paper in detail, and assessed the reviewer's opinions and the authors rebuttal. Overall a serious concern is the leap of faith that the proposed process is indeed (a) "leveraging the memorization effect" to (b) "extract hard confident examples" to (c) "improve accuracy in noisy learning". For (a), it is mentioned that deep learning models "learns simple patterns on majority of data" first, where the majority in this work is argued is the clean ones. But there is no validation of this claim in the experiments. For instance, there is no figure/discussion that shows how much "clean data" has been correctly captured/memorized by the earlier deep learning models. For (b), the terminology of "hard but confident" is ill-defined. If examples being hard means them to be around the boundary, one can argue that they could never be "confident" as one measure of the confidence is the margin. The authors may want to mean "hard but clean", but then more illustrations are needed to analyze whether the extracted examples are really the clean ones, or if there are noisy ones being "confident" from the proposed process as well. For (c), it is then unclear whether the improved performance is caused by noise removal (as the authors hope to argue), or by just zooming in to the boundary (regardless of whether the extracted examples are clean or noisy). The authors are encouraged to not just look at the superior performance on accuracy, but analyze more on what actually happened behind the scenes to understand the proposed process better.

Some more comments from the AC that could help the authors (1) Are the examples kept by the proposed approach similar to the ones kept by SELF? Why or why not? Are there empirical studies on this? (2) For the competitors' approaches, what would their Figure 4 look like? In particular, for approaches that use "small-loss trick", what would Figure 4 look like? How would Figure 4 be different for the proposed process (i.e extracting hard "confident" examples) and others (like extracting just hard examples without considering confidence, or just confident examples without considering hardness)? (3) Are there studies that deliberately initialize the process with noisy data, to see how sensitive the process is before the "positive cycle" begins?

Decision: Reject