SELF: Learning to Filter Noisy Labels with Self-Ensembling
Authors: Duc Tam Nguyen
Chaithanya Kumar Mummadi
Thi Phuong Nhung Ngo
Thi Hoai Phuong Nguyen
Laura Beggel
Thomas Brox
Venue:  (2020)
Keywords: Ensemble Learning
Robust Learning
Noisy Labels
Labels Filtering

Abstract: Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.

Published as a conference paper at ICLR 2020
SELF: LEARNING TO FILTER NOISY LABELS WITH SELF-ENSEMBLING
Duc Tam Nguyen' ; Chaithanya Kumar Mummadi : Thi Phuong Nhung Ngo i Thi Hoai Phuong Nguyen 1, Laura Beggel Thomas Brox
ABSTRACT
Deep neural networks (DNNs) have been shown to over-fit dataset when be- ing trained with noisy labels for a long enough time. To overcome this problem. we present simple and effective method self-ensemble label label filtering (SELF) progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these en- semble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on var- ious image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise- learning across different datasets and can be applied to a broad set of network architectures.
INTRODUCTION
The acquisition of large quantities of high-quality human annotation is frequent bottleneck in applying DNNs. There are two cheap but imperfect alternatives to collect annotation at large scale: crowdsourcing from non-experts and web annotations, particularly for image data where the tags and online query keywords are treated as valid labels. Both these alternatives typically introduce noisy (wrong) labels. While Rolnick al. (2017) empirically demonstrated that DNNs can be surprisingly robust to label noise under certain conditions, Zhang et al. (2017) has shown that DNNs have the capacity to memorize the data and will do so eventually when being confronted with too many noisy labels. Consequently, training DNNs with traditional learning procedures noisy data strongly deteriorates their ability to generalize noise is of great practical importance.
severe problem. Hence, limiting the influence of label
common approach to mitigate the negative influence of noisy labels to eliminate them from the training data and train deep learning models just with the clean labels (Frénay & Verleysen 2013). Employing semi-supervised learning can even counteract the noisy labels (Laine & Aila [016 Luo et al. 2018). However, the decision which labels are noisy and which are not is decisive for learning robust models. Otherwise, unfiltered noisy labels still influence the (supervised) loss and affect the task performance as in these previous works. They use the entire label set to compute the loss and severely lack a mechanism to identify and filter out the erroneous labels from the labels set.
In this paper, we propose a self-ensemble label filtering (SELF) framework that identifies potentially noisy labels during training and keeps the network from receiving supervision from the filtered noisy labels. This allows DNNs to gradually focus learning from undoubtedly correct samples even with an extreme level of noise in the labels (c.g.. 80% noise ratio) and leads to improved performance as the supervision become less noisy. The key contribution of our work is progressive filtering, i.c.,
" "Compueer Vision Group, University of Freiburg, Germany I Bossh Center for AL. Bosch GmbHl Germany I Karlsruhe Institute of Technology, Germany

---

Published as a conference paper at ICLR 2020
90
85
60
Accuracy
50
75
SELF (ours)
70
SELF (ours)
Lq
Trunc b
Trunc Lg
Forward 7
60
Forward
20
Teaching
55
Co Teaching
10
20
30
40
50
60
70
80
20
30
40
50
60
70
80
Noise ratio in %
Noise ratio in %
(a) Evaluation on CIFAR-10
(b) Evaluation on CIFAR-100
Figure Comparing the performance SELF with previous works for learning under different (symmetric) label noise ratios on the (a) CIFAR-I0 & (b) CIFAR-100 datasets. SELF retains higher robust classification accuracy all noise levels.
leverage the knowledge provided in the network's ' output over different training iterations to form a consensus of predictions self-ensembll predictions) to progressively identify and filter out the noisy labels from the labeled data.
When learning under label noise, the network receives noisy updates and hence fluctuates strongly- Such conduct of training would impede to learn stable neural representations and further mislead the consensus of the predictions. Therefore, it is essential to incorporate model with stable training behavior to obtain better estimates from the consensus. Concretely, we employ the semi-supervised technique as a backbone to our framework stabilize the learning process of the model. Correctly, we maintain the running average model, such as proposed by Tarvainen & Valpola (2017), a.k.a. the Mean Teacher model. This model ensemble learning provides more stable supervisory signal than the noisy model snapshots and provides stable ground for progressive filtering to filter out potential noisy labels. Note that this is different from just mere combination of semi-supervised techniques with noisy label filtering method.
We call our approach self-ensemble label filtering (SELF) - that establishes model ensemble learning as backbone to form solid consensus of the self-ensemble predictions to filter out the noisy labels progressively. Our framework allows to compute supervised loss on cleaner subsets rather than the entire noisy labeled data in previous works. It further leverages the entire dataset, including the filtered out erroneous samples in the unsupervised loss. To best of our knowledge, we are the first to identify and propose self-ensemble as a principled technique against learning under noisy labels.
Our motivation stems from the observation that DNNs start to learn from easy samples initial phases and gradually adapt hard during training. When trained wrongly labeled data, DNNs learn from clean labels at ease and receive inconsistent error signals from the noisy labels before over-fitting the dataset. The network's 's prediction likely be consistent on clean samples and inconsistent or oscillates strongly on wrongly labeled samples over different training iterations. Based on this observation, we record the outputs of single network made on different training epochs and treat them as ensemble of predictions obtained from different individual networks. We call these ensembles that are evolved from single network self-ensemble predictions. Subsequently, we identify the correctly labeled samples via the agreement between the provided label set and our running average self-ensemble predictions. The samples of ensemble predictions that agree with the provided labels are likely to be consistent and treated as clean samples.
summary, our SELF framework stabilizes the training process and improves the generalization ability of DNNs. We evaluate the proposed technique image classification tasks using CI- FARIO, CIFAR100 & ImageNet. We demonstrate that SELF consistently outperforms the existing approaches asymmetric and symmetric noise at all noise levels, shown in Fig. T Besides, SELF remains robust towards the choice of the network architecture. Our work transferable to other tasks without the need modify the architecture or the primary learning objective.
2

---

Published as a conference paper at ICLR 2020
Training at iteration i
Training dataset
Remove noisy labels from La!
Identify noisy labels
Initial label set Lo
Filtered label set
Start
clean labels
noisy labels
Model Ensemble Learning
forms
Prediction Ensemble
Images
Model Training
Model Ensemble
matching loss
Validation dataset
YES
Clean and noisy labels
Found better model
Model Evaluation
Images
NO
Stop
Figure 2: Overview the self-ensemble label filtering (SELF) framework. The model starts in iteration 0 with training from the noisy label set. During training, the model maintains self- ensemble, a running average of itself (Tarvainen & Valpola 2017) to provide a stable learning signal. Also, the model collects a self-ensemble prediction (moving-average) for the subsequent filtering. Once the best model is found, these predictions identify and filter out noisy labels using the original label Lg. The model performs this progressive filtering until there is no more better model. For details see Algorithm
2
SELF-ENSEMBLE LABEL FILTERING
2.1 OVERVIEW
Fig. shows overview of our proposed approach. In the beginning, assume that the labels of the training set are noisy. The model attempts to identify correct labels progressively using self- forming ensembles of models and predictions. Since wrong labels cause strong fluctuations in the model's predictions, using ensembles is a natural way to counteract noisy labels.
Concretely, in each iteration, the model learns from detected set of potentially correct labels and maintains running average of model snapshots (realized by the Mean Teacher model Tarvainen & Valpola (2017). This ensemble model is evaluated on the entire dataset and provides an additional learning signal for training the single models. Additionally, our framework maintains the running- average of the model's predictions for the filtering process. The model is trained until we find the best model w.r.t. the performance on the validation set (e.g., by carly-stopping). The set of correct labels detected based on the strategy defined Sec. 2.2 In the next iteration, we again use all data and the new filtered label set as input for the model training. The iterative training procedure stops when no better model can be found. In the following, we give more details about the combination of this training and filtering procedure.
2.2 PROGRESSIVE LABEL FILTERING
Progressive detection of correctly labeled samples Our framework Self-Ensemble Label Filter- ing (Algorithm focuses on the detection of certainly correct labels from the provided label set To- In each iteration i, the model is trained using the label set of potentially correct labels 14. the end of each iteration, the model determines the next correct label set L using the filtering strategy described in 2.2 model stops learning when no improvement was achieved after training on the refined label set L,+1-
In other words, in each iteration, the model attempts to learn from the easy, in some sense, obviously correct labels. However, learning from easy samples also affects similar but harder samples from the same classes. Therefore, by learning from these easy samples, the network can gradually distinguish between hard and wrbeel samples.
3

---

Published as a conference paper at ICLR 2020
Algorithm 1 SELF: Self-Ensenne Label Filtering pseudocode
Require: Dtrain noisy labeled training
Require: Doat noisy labeled validation set
Require: (a. y) = training stimuli and label
Require: a ensembling momentum, 0 S a <
counter to track iterations
M; traint Dirain, Dval)
initial Mean-Teacher ensemble model training D set initial model as best model
Mhest f- M;
2, 4-0
b initialize ensemble predictions of all samples
(ignored sample index for simplicity)
while acc(M, Deal) 2 acc Mocsst ) do Muest M.
iterate until no best model is found on Doul
save the best model
D Pitter t Dirain
& set filtered dataset as initial label set
3 1 1+1 for (3r, y) in Dritter do
2 Mbest (2)
evaluate model output 2;
2, 5- až + (1 a)&
D accumulate ensemble predictions Σ; b verify agreement of ensemble predictions & label
if y * argmaau(Z.) then
y +- 0 in D Daater
> identify it as noisy label & remove from label set
end if
end for
M. t train Ditter, Dvat)
> train Mcan-Teacher model on filtered label set
end while return Mbest
Our framework does not focus on repairing all noisy labels. Although the detection of wrong labels is sometimes easy, finding their correct hidden label might be extremely challenging in case of having many classes. If the noise is sufficiently random, the set of correct labels will be representative to achieve high model performance. Further, in our framework, the label filtering is performed on the original label set Lo from iteration O. Clean labels erroneously removed in an earlier iteration (c.g., labels of hard to classify samples) can be reconsidered for model training again later iterations.
Filtering strategy The model can determine the set of potentially correct labels L; based on agree- ment between the label y and its maximal likelihood prediction 5/ with Le {(y.x) üz y: Vy, E Lo)- Lo the label set provided the beginning, (y, x) are the samples and their respective noisy labels in the iteration i. In other words, the labels are only used for supervised training if in the current epoch, the model predicts the respective label to be the correct class with the highest likelihood. In practice, our framework does not use ((x) of model snapshots for filtering but a moving-average of the ensemble models and predictions to improve the filtering decision.
2.3 SELF-ENSEMBLE LEARNING
The model's predictions for noisy samples tend to fluctuate. For example, take a cat wrongly labeled as a tiger. Other cat samples would encourage the model to predict the given cat image as a cat. Contrary, the wrong label tiger regularly pulls the model back to predict the cat as a tiger. Hence, using the model's predictions gathered one single training epoch for filtering is sub-optimal. Therefore, in our framework SELF, our model relies ensembles of models and predictions.
Model ensemble with Mean Teacher A natural way to form model ensemble is by using an exponential running average of model snapshots (Fig. [ap. This idea was proposed in Tarvainen & &Valpola (207)) for semi-supervised learning and is known as the Mean Teacher model. In our framework, both the mean teacher model and the normal model are evaluated on all data to preserve the consistency between both models. The consistency loss between student and teacher output distribution can be realized with Lean-Square--rree loss or DullbaakkLeiibbeeddiveeeeee More details for training with the model ensemble can be found in Appendix A..
Prediction ensemble Additionally, we propose to collect the sample predictions over multiple training epochs: 3, = = (1-a) whereby Σ₁ depicts the moving-averagg prediction of sample at epoch j. a is a momentum, 2, is the model prediction for sample in epoch j. This scheme is displayed in Fig. 3b For each sample, we store the moving-average average predictions, accumulated over the past iterations. Besides having a more stable basis for the filtering step. our proposed procedure also leads negligible memory and computation overhead.
4

---

Published as a conference paper at ICLR 2020
O0 iter. 1-2
nl in
Iter. 1-1
iter.
ensemble
epoch 1-2
epoch i-1
epoch
ensemble
(a) Model ensemble (Mean teacher)
(b) Predictions ensemble
Figure 3: Maintaining the (a) model and (b) predictions ensembles is very effective against noisy model updates. These ensembles are self-forming during the training process as a moving-average of (a) model snapshots or (b) class predictions from previous training steps.
Further, due to continuous training of the best model from the previous model, computation time can be significantly reduced, compared to re-training the model from scratch. On the new filtered dataset, the model must only slowly adapt to the new noise ratio contained in the training set. Depending on the computation budget, a maximal number of iterations for filtering can be set to save time.
3 RELATED WORKS
Reed et al. (2014): Azadi et al. (2015) performed early works on learning robustly under label noise for deep neural networks. Recently, Rolnick et al. (2017) have shown for classification that deep neural networks come with natural robustness to label noise following a particular random distribution. No modification of the network or the training procedure is required to achieve this robustness. Following this insight, our framework SELF relies on this natural robustness to kickstart the self-ensemble filtering process extend the robust behavior more challenging scenarios.
Laine & Aila (016): Luo et al. 2018) proposed to apply semi-supervised techniques on the data to counteract noise. These and other semi-supervised learning techniques learn from static, initial set of noisy labels and have no mechanisms to repair labels. Therefore, the supervised losses in their learning objective are typically high until the model strongly overfits to the label noise. Compared to these works, our framework performs a variant of self-supervised label corrections. The network learns from dynamic, variable set of labels, which is determined by the network itself. Progressive filtering allows the network to (1) focus repair wrong decisions made by itself in an earlier iteration.
label set with a significantly lower noise ratio and (2)
Other works assign weights to potentially wrong labels to reduce the learning signal (Jiang et al.) 2017 Ren et al. 2018 Jenni & Favaro 2018). These approaches tend to assign less extreme weights or hyperparameters that are hard to set. Since the typical classification loss is highly non-linear, lower weight might still lead to learning from wrong labels. Compared to these works, the samples in SELF only receive extreme weights: either they are self-detecting the correct samples, instead of repairing the wrong labels. Typically, the set of correct samples are much easier to detect and are sufficiently representative to achieve high performance.
one. Further, SELF focuses only
Han et al. 2018b); Jiang et al. (2017) employ two collaborating and simultaneously learning net- works determine which samples to learn from and which not. However, the second network is free in its predictions and hence hard to tune. Compared to these works, we use ensemble learning as a principled approach to counteract model fluctuations. In SELF, the second network extremely restricted and is only composed of running averages of the first network. To realize the second network, we use the meachee model (Tarvainen Valpola, (017)) as backbone. Compared to their work, our self-ensemble label filtering gradually detects the correct labels and learns from them, so the label set is variable. Further, we do use not only model ensembles but also an ensemble of predictions to detect correct labels.
Other works modify the primary loss function of the classification tasks. Patrini al. (2017) es- timates the noise transition matrix correct the loss, Han et al. (2018a) uses human-in-the-loog Zhang& & Sabuncu 2): Thulasidasan et al. 2019)) propose other forms of cross-entropy losses. The loss modification impedes the transfer of these ideas to other tasks than classification. Compared to these works, our framework SELF does not modify the primary loss. However, many tasks rely on the presence of clean labels such anomaly detection (Nguyen et al. 2019a) self-supervised and unsupervised learning (Nguyen et al. (0195). The progressive filtering procedure and self-ensemble learning proposed are also applicable these tasks counteract noise effectively.
5

---

Published as a conference paper at ICLR 2020
Table Comparison of classification accuracy when learning under uniform label noise on CIFAR- 10 and CIFAR-IO8 Following previous works, we compare two evaluation scenarios: with noisy validation set (top) and with 1000 clean validation samples (bottom). The best model is marked in bold. Having small clean validation set improves the model but is not necessary.
CIFAR-10
CIFAR- 1-0
NOISE RATIO
40%
60%
80%
40%
60%
80 %
USING NOISY VALIDATION SET
REED ET AL.
2014)
69.66
51.34
S-MODEL GOLDBERGER & BEN-REUVER
2016)
70.64
49.10
OPEN-SET WANG ET AL. (2018)
78.15
RAND. WEIGHTS (REN
2018
86.06
58.01
BI-LEVEL-MODEE JJNNI FAVARO
2018
89.00
20.00
61.00
13.00
MENTORNET (JANG ET AL. 2017)
89.00
49.00
68.00
35.00
La (ZHANG & SABUNCU 2018)
87.13
82.54
64.07
61.77
53.16
29.16
TRUNC La ZHANG & SABUNCU
2018
87.62
82.70
67.92
62.64
54.04
29.60
FORWARD PATRINI ET AL..
2017
83.25
74.96
54.64
31.05
19.12
08.90
CO-TEACHING HAN AL.
2018B
81.85
74.04
29.22
55.95
47.98
23.22
D2L MA AL. 2018)
83.36
72.84
52.01
42.27
SL WANGET AL. 2019)
85.34
80.07
53.81
$3.69
41.47
15.00
JOINTOPT (TANAKA ET AL.
2018)
83.27
74.39
40.09
52.88
42.64
18.46
SELF (OURS)
93.70
93.15
69.91
71.98
66.21
42.09
USING CLEAN VALIDATION SET (1000 IMAGES)
DAC (THULASIDASAN ET AL.
2019
90.93
87.58
70.80
68.20
59.44
34.06
MENTORNET (JANG ET AL. 2017)
78.00
59.00
RAND. WEIGHTS (REN ET AL. 2018
86.55
58.34
REN (REN ET 2018
86.92
61.31
SELF* (OURS)
95.10
93.77
79.93
74.76
68.35
46.43
EVALUATION
4.1 EXPERIMENTS DESCRIPTIONS
4.1.1
STRUCTURE OF THE ANALYSTS
We evaluate our approach on CIFAR- 10, CIFAR- 100, an TrageNet-ILSSRE on different noise sce- narios. For CIFAR-10, 10, CIFAR-100, and ImageNet, we consider the typical situation with symmetric and asymmetric label noise. In the case of the symmetric noise, a label randomly flipped to another class with probability p. Following previous works. we also consider label flips of semantically sim- ilar classes on CIFAR-10, and pair-wise label flips on CIFAR- the choice of the network architecture and the ablation of the components in framework. Tab. (Appendix) shows the in-deep analysis semi-supervised learning strategies combined with recent works. Overall, the proposed framework SELF outperforms all these combinations.
Finally, we perform studies on
4.1.2 COMPARISONS TO PREVIOUS WORKS
We compare work to previous methods from Reed-Hard (Reed et al.
2014). S-model (Gold-)
berger & Ben-Reuven, 2016). Wang et al. 2018), 3 Rand. weights (Ren et al.
2018), ) Bi-level-model
(Jenni & Favaro 2018). D2L (Ma et al.
2018 SL (Wang et al. 2019). Lg (Zhang & Sabuneu,
2018 Trunc Lg [Zhang & Sabuncu 2018). Forward T' (Patrini et al. 2017). DAC (Thulasidasan
et al. 2019), Random reweighting (Ren al. 2018), and Learning to reweight (Ren et al. 2018). For co-teaching (Han et al., 2018b), MentorNet (Jiang et al. (017). JointOpt (Tanaka et al. 2018). the source codes available and hence used for evaluation.
(Ren et al. 2018 and DAC (Thulasidasan et al. 2019) considered the setting of having small clean validation set of 1000 and 5000 images respectively. For comparison purposes, we also experiment with a small clean set 1000 images additionally. Further, we abandon oracle experiments or methods using additional information keep the evaluation comparable. For instance, Forward T (Patrini et al. 2017) uses the true underlying confusion matrix to correct the loss. This information neither known in typical scenarios used by other methods.

---

Published as a conference paper at ICLR 2020
Table Asymmetric noise on CIFAR-IO, CIFAR-100. All methods use Resnet34. CIFAR-I0: flip TRUCK - AUTOMOBILE, BIRD AIRPLANE, DEER - HORSE, CAT+DOG with prob. p. CIFAR-100: flip class (i + 1)%100 with prob. p. SELF retains high performances across all noise ratios and outperforms all previous works.
CIFAR- 10
CIFAR-
NOISE RATIO
10%
20%
30%
40%
10%
20%
30%
40%
CCE
90.69
88.59
86.14
80.11
66.54
59.20
51.40
42.74
MAE
82.61
52.93
50.36
45.52
13.38
11.50
08.91
08.20
FORWARD T
90.52
89.09
86.79
83.55
45.96
42.46
38.13
34.44
Lu
90.91
89.33
85,45
76.74
68.36
66.59
61.45
47.22
TRUNC Lg
90.43
89.45
87.10
82.28
68.86
66.59
61.87
47.66
88.24
85.36
80.64
65.58
65.14
63.10
JOINTOPT
90.12
89.45
87.18
87.97
69.61
68.94
63.99
53.71
SELF (OUrS)
93.75
92.76
92.42
89.07
72.45
70.53
65.09
53.83
Whenever possible, we adopt the reported performance from the corresponding publications. The testing scenarios are kept as similar as possible to enable fair comparison. All tested scenarios use noisy validation set with the same noise distribution as the training set unless stated otherwise. All model performances are reported on the clean test set.
Table 3: Effect of the choice network architecture on classification accuracy on CIFAR-10 & -100 with uniform label noise. SELF is compatible with all tested architectures. Here represents baseline accuracy of the architectures that are trained on fully supervised setting at 0% label noise.
CIFAR-10
CIFAR
CIFAR-10
CIFAR-
RESNET101 101
93.89*
84.
RESNET34
93.5*
76.76*
NOISE
40%
80%
40%
80%
NOISE
40%
80%
40%
80%
MENTORNET
89.00
49.00
68.00
35.00
Lg
87.13
64.07
61.77
29.16
CO-T.
62.58
21.79
39.58
16.79
TRUNC Lg
87.62
67.92
62.64
29.60
SELF
92.77
64.52
69.00
39.73
FORWARD T
83.25
54.64
31.05
8.90
SELF
91.13
63.59
66.71
35.56
WRN 28-10
96.21
81.02*
RESNET26
96.37*
81.20
NOISE
80%
40%
80%
NoISE
40%
80%
40%
80%
MENTORNET
88.7
46.30
67.50
30.10
REWEIGHT
86.02
58.01
Co-T.
81.85
29.22
55.95
23.22
SELF
93.34
67.41
72.48
42.06
SELF
93.70
69.91
71.98
42.09
4.1.3 NETWORKS CONFIGURATION AND TRAINING
For the basic training of self-ensemble model, we use the Mean Teacher model (Tarvainen & Valpola 2017) available on GitHub The students and teacher networks are residual networks (He al. 2016) with 26 layers with Shake-Shake-en Shake-regularizztion (Gastaldi) 2017). We use the Py- Torch Paszke et al. 2017) implementation of the network and keep the training settings close to (Taavainen & Valpola 2017). The network is trained with Stochastic Gradient Descent. In each filtering iteration, the model is trained for maximum of 300 epochs, with patience of 50 epochs. For more training details, see the appendix.
4.2 EXPERIMENTS RESULTS
4.2.1 SYMMETRIC LABEL NOISE
CIFAR-10 and 100 Results for typical uniform noise scenarios with noise ratios on CIFAR-10 and CIFAR-100 are shown in Tab. More results are visualized in Fig. Ta (CIFAR- 10) and Fig. Tb (CIFAR-ION) 100). Our approach SELF performs robustly in the case of lower noise ratios with up to 60% and outperforms previous works. Although strong performance loss occurs at 80% label noise,

---

Published as a conference paper at ICLR 2020
Table Ablation study on CIFAR-I0 and CIFAR- 100. The Resnet baseline was trained on the full noisy label set. Adding progressive filtering im- proves over this baseline. The Mean Teacher main- tains an ensemble of model snapshots, which helps counteract noise. Having progressive filtering and model ensembles (-MVA-pred.) makes the model more robust but still fails at 80% noise. The full SELF framework additionally uses the prediction ensemble for detection of correct labels.
Table 4: Classification accuracy on clean ImageNet validation dataset. The mod- els are trained at 40% label noise and the best model is picked based on the evalu- ation on noisy validation data. Mentornet shows the best previously reported results. Mentornet* is based on Resnet-101. We chose the smaller Resnex150 model to re- duce the run-time.
Resnext18 P@1 P@5
Resnex150 P@I P@5
CIFAR- 10
CIFAR- 100
Accurraey
NOISE RATIO
40%
80%
40%
80%
Mentorne**
65.10 85.90
RESNET26
83.20
41.37
53.18
19.92
ResNext
50.6 75.99
56.25 80.90
FILTERING
87.35
49.58
61.40
23.42
Mean-T.
58.04 81.82
62.96 85.72
MEAN-T.
93.70
52.50
65.85
26.31
SELF (Ours)
66.92 86.65
71.31 89.92
MVA-PRED.
93.77
57.40
71.69
38,61
SELF (OURS)
93.70
69.91
71.98
42.09
SELF still outperforms most of the previous approaches. The experiment SELF* using a 1000 clean validation images shows that the performance loss mostly originates from the progressive filtering relying too strongly on the extremely noisy validation set.
ISVC Tab. shows the precision@) 1 and @5 of various models, given 40% label noise in the training set. Our networks are based on ResNext18 and Resnex150. Note that MentorNet (Jing et al., 2017) uses Resnet101 (P@ I: 78.25) (Goyal et al. 2017]. which has higher performance compared to Resnext50 (P@ 1: 77.8) (Xie et al. 2017) the standard ImageNet validation set.
Despite the weaker model, SELF (ResNext50) surpasses the best previously reported results by more than 5% absolute improvement. Even the significantly weaker model ResNext18 outperforms MentorNet, which is based on a more powerful ResNet101 network.
4.2.2 ASYMMETRIC LABEL NOISE
Tab. 2 shows more challenging noise scenarios when the noise is not class-symmetric and uniform. Concretely, labels are flipped among semantically similar classes such as CAT and DOG on CIFAR- 10. On C00,R-100, each label is flipped to the next class with probability p. In these scenarios, our framework SELF also retains high performance and only shows small performance drop at 40% noise. The high label noise resistance of our framework indicates that the proposed self-ensemble filtering process helps the network identify correct samples, even under extreme noise ratios.
4.2.3 EFFECTS OF DIFFERENT ARCHITECTURES
Previous works utilize a various set of different architectures, which hinders a fair comparison. Tab. 3 shows the performance of our framework SELF compared to previous approaches. SELF outperforms other works in all scenarios except for CIFAR-10 with 80% noise. Typical robust learning approaches lead to significant accuracy losses at 40% noise, while SELF still retains high performance. Further, note that SELF allows the network's performance to remain consistent across the different underlying architectures.
4.2.4
ABLATION STUDY
Tab. shows the importance of each component in our framework. See Fig. 4a Fig. 4b for experi- ments on more noise ratios. As expected, the fesnet-baseline rapidly breaks down with increasing noise ratios. Adding supervueeee filtering increases the performance slightly in lower noise ratios. However, the model has to rely on extremely noisy snapshots. Contrary, using model ensemble alone such in Mcan-Teacher can counteract noise on the noisy dataset CIFAR-10. On the more challenging CIFAR- 100, however, the performance decreases strongly. With self-supervised filter- ing and model ensembles, SELF (without MVA-pred) is more robust and only impairs performance at 80% noise. The last performance boost is given by using moving-average predictions so that the network can reliably detect correctly labeled samples gradually.
Fig, shows the ablation experiments on more noise ratios. The analyses shows that each component in SELF is essential for the model to learn robustly.
8

---

Published as a conference paper at ICLR 2020
100
80
60
Accuracy
50
70
SELF (ours)
SELF (ours)
40
Resnet28
Resnet26
30
Filtering
Filtering
Mean-Teecher
50
Mean-Teacher
20
w.o. pred.
SCLT mvv
10
40
40
60
Noise ratio in
Noise
(a) Ablation exps. CIFAR-10
Ablation exps. on CIFAR-100
Figure 4: Ablation study on the importance of the components our framework SELF, evaluated on (a) Cifar-10 and (b) Cifar-100 with uniform noise. Please refer Tab. 5 for details of components.
Table 6: Analysis of semi-supervised learning (SSL) strategies: entropy learning, mean-teacher combined with recent works. Our progressive filtering strategy is shown to be effective and per- forms well regardless of the choice of the semi-supervised learning backbone. Overall, the proposed method SELF outperforms all these combinations. Best model in each SSL-category is marked bold. Running teah-tt co-teaching using the same configuration is not possible due to memory constraints.
CIFAR-
CIFAR-
NOISE RATIO
40%
60%
80%
40%
60%
80%
BASELINE MODELS
RESNET26 GASTALDI 2017
83.20
72.35
41.37
53.18
44.31
19.92
Co- TO-THENHING (HAN ET AL.
2018B
81.85
74.04
29.22
55.95
47.98
23.22
JOINTOPT (TANAKA ET AL.
2018)
83.27
74.39
40.09
52.88
42.64
18.46
PROGRESSIVE FILTERING (OURS)
87.35
75.47
49.58
61.40
50.60
23.42
SEMI-SUPERVISED LEARNING WITH ENTROPY LEARNING
ENTROPY
79.13
85.98
46.93
54.65
41.34
21.29
ENTROPY CO-TEACHING
84.94
74.28
35.16
55.68
43.52
20.5
ENTROPY JOINT-OPT
84.44
75.86
39.16
56.73
43.27
17.24
ENTROPY+FILTERING (OURS)
90.04
83.88
52.46
59.97
46.45
23.53
SEMI-SUPERVISED LEARNING WITH MEAN-TEACHES
MEAN TEACHER
93.70
90.40
52.5
65.85
54.48
26.31
MEAN-TEACHER JOINTOPT
91.40
83.62
45.12
60.09
45.92
23.54
MEAN-TEACHER TEACHER FILTERING SELF (OURS)
93.70
92.85
69.91
71.98
66.21
42.58
4.2.5 SEMI-SUPERVISED LEARNING FOR PROGRESSIVE FILTERING
Tab. d shows different semi-supervised learning strategies: entropy learning, mean-teacher com- bined with recent works. Note that Co-Teaching+MMan--TTeee Teacher cannot be implemented and run in same configuration as other experiments, due to memory constraints.
The analysis indicates the semi-supervised losses mostly stabilize the baselines, compared to the model without cemi-supervised learning. However, Co-teaching and JointOpt sometimes perform worse than the purely semi-supervised model. This result indicates that their proposed frameworks are not always compatible with cemi-supervised losses.
The progressive filtering technique is seamlessly compatible with different semi-supervised losses. The filtering outperforms its counterparts when combined with Entropy Learning or Mean-teacher model. Overall, SELF outperforms all considered combinations.

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
We propose a simple and easy to implement framework to train robust deep learning models under incorrect or noisy labels. We filter out the training samples that are hard to learn (possibly noisy labeled samples) by leveraging ensemble of predictions of the single network's output over different training epochs. Subsequently, we allow clean supervision from the non-hard samples and further leverage additional unsupervised loss from the entire dataset. We show that framework results in DNN models with superior generalization performance on CIFAR-10, CIFAR-100 & ImageNet and outperforms all previous works under symmetric (uniform) and asymmetric noises. Furthermore, our models remain robust despite the increasing noise ratio and change in network architectures.
REFERENCES
Samanch Azadi, Jiashi Feng, Stefanie Jegelka, and Trevor Darrell. Auxiliary image regularization for deep cnns with noisy labels. arXiv preprint arXiv: 2015.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE transactions on neural networks and learning systems, 25(5):845-869, 2013.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint
2017.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. pp. 9.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint a 706.02667, 2017.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In Advances in Neural Information Processing Systems, pp. 5836-5846, 201 8a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pp. 5535-85455 2018b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Simon Jenni and Paolo Favaro. Deep bilevel learning. In ECCV, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Fei-Fei. MentorNet: Learning Data- Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. arXiv:1712.0500 [cs), December 2017. URL
www.orr///bb////aa 05055 arXiv: 1712.05055.
Samuli Laine and Timo Aila. Temporal ensembling for cemi-supervised learning. arXiv preprint arXiv:1610.02244) 2016.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0300 2016.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8896-8905, 2018.
Xingjun Ma, Yisen Wang. Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudan- thi Wijewickrema, and James Bailey. Dimensionnlity-diivea learning with noisy labels. arXiv preprint 1806: /20.000/2. 2018.
10

---

Published as a conference paper at ICLR 2020
Duc Tam Nguyen, Zhongyu Lou, Michael Klar, and Thomas Brox. Anomaly detection with multiple-hypothesss predictions. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 00.480044899 Long Beach, California, USA, 09-15 Jun 2019a. ml prwss / /v7 htt http//////////nnnn//e///// ngwyen 9b. html
PMLR.
Tam Nguyen, Maximilian Dax. Chaithanya Kumar Mummadi, Nhung Ngo. Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox. Deepusps: Deep robust unsupervised saliency predic- tion via self-supervision. In Advances in Neural Information Processing Systems, pp. 204-214, 2019b.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to Reweight Examless for Robust Deep Learning. arXiv: 09050 [cs, stat], March 2018. utt.http://rxxi..oo arXiv: 1803.09050.
abs//803..0m 09050
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv: 10694, 2017.
Ilya Sutskever, James Martens, George E Dahl, and Geoffrey E Hinton. On the importance of initialization and momentum in deep learning. ICML (3), 13(1139-11477:5,2011
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averagee consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. arXiv preprint a905: 2019.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. arXiv preprint arX1v:1804.00000 2018
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross en- tropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vision, pp. 322-330, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1492-1500, 2017.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Rep- resentations, 2017. URL http:::/
net/ /twttfouum//.. www.oooomuomaooomooooo forum?1-=SSyggB53
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, pp. 8778-8788 2018.
11

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1
MEAN TEACHER MODEL FOR ITERATIVE FILTERING
We apply the Mean Teacher algorithm in each iteration i in the train (D filter, Dvat) procedure as follows.
Input: examples with potentially clean labels Ditter from the filtering procedure. In the beginning (i = 0), here fitte refers entire labeled dataset.
Initialize a supervised neural network as the student model M
Initialize the Mean Teacher model Mf as a copy of the student model with all weights detached.
Let the loss function the sum of normal classification loss of M2 and the consistency loss between the outputs of Mf and Mf
Select an optimizer
In each training iteration:
Update the weights of M using the selected optimizer
Update the weights of M an exponential moving-average of the student weights Evaluate performance of MC and Mi over Det to verify the early stopping criteria.
Return the best M
A.2 ASSUMPTIONS DICUSSIONS
Our method performs best when the following assumptions are hold.
Natural robustness assumption deep networks Rolnick et al. 2017): The networks attempt to learn the easiest way to explain most of the data. SELF uses this assumption to kickstart the learning process.
Correct samples dominate over wrongly labeled samples At 80% noise on CIFAR-10, the cor- rectly labeled cats (20% out of all cat images) still dominates over samples wrongly labeled as cat (8.8% for each class).
Independence results less overfitting SELF performs best the noises on the validation set and training set are i.i.d. SELF uses the validation data for early stopping. Hence, high correlation of label noise between train and valid increases the chance of model overfitting.
Sufficient label randomness assumption The subset of all correctly labeled samples capture all samples clusters. In fact, many works from the active learning literature show that less than 100 % of the labeled samples required achieve the highest model performance. SELF performs progressive expansion of the correct labels sets. At larger noise ratios, not all clusters are covered by the identified samples. Therefore task containing many classes, e.g., CIFAR-100, the model performance decreases faster than on CIFAR-10.
The model performance reduces when these assumptions are strongly violated. Each assumption should have its own "critical" threshold for violation. A future in-depth analysis to challenge the assumptions is interesting future research direction.
A.3 TRAINING DETAILS
A.3.1
CIFAR-10 AND 100AR-100
Dataset Tab. 7 shows the details of CIFAR-10 10 and 100 datasets in our evaluation pipeline. The
validation set contaminated with the same noise ratio the training data unless stated otherwise.
12

---

Published as a conference paper at ICLR 2020
Network training For the training our model SELF, we use the standard configuration provided by Tarvainen Valpola 2017 Concretely, we use the SGD-optimizer with Nesterov Sutskever et al. (2013) momentum, learning rate of 0.05 with cosine learning rate annealing Loshchilov & Hutter 2016), weight decay of 2e-4, max iteration per filtering step of 300, patience 50 epochs, total epochs count of 600.
Table 7: Dataset description. Classification tasks on CIFAR-10 10 and CIFAR-100 with uniform noise.
Note that the noise on the training and validation set is not correlated. Hence, maximizing the accuracy on the noisy set provides a useful (but noisy) estimate for the generalization ability on unseen test data.
TYPE
CIFAR-10
CIFAR 100
TASK
CLASSIFICATION
10-WAY
100-WAY
RESOLUTION
32x32
TRAIN (NOISY)
45000
45000
DATA
VALID (NOISY)
5000
5000
TEST (CLEAN)
10000
10000
For basic training of baselines models without semi-supervised learning, we had to set the learning rate 0.01. In the case of higher learning rates, the loss typically explodes. Every other option kept the same.
Semi-supervised learning For the mean teacher training, additional hyperparameters are required. In both cases CIFAR-10 and C10, we again take the standard configuration with the con- sistency loss to mean-squared-erro and consistency weight: 100.0, logit distance cost: 0.01, consistency- caapttenccy-rm-uu..S.... The total batch-size is 512, with 124 samples being reserved for labeled samples, 388 for unlabeled data. Each epoch is defined complete processing of all unlabeled data. When training without semi-superviset-earinng the entire batch is used for labeled data.
Data augmentation The data are normalized to zero-mean and standard-variance of one. Further, real-time data augmentation with random translation and reflection, subsequently random horizontal flip. The standard Py TToch-library provides these transformations.
A.3.2 IMAGENET-ILSVRC.
Network Training The network used for evaluation were ResNet al. (2016) and Resnext Xie et al. (0177) for training. All ResNext variants use a cardinality of 32 and base width of 4 (32x4d). ResNext models follow the same structure their Resnet counterparts, except for the cardinality and base width.
All other configurations are kept as close as possible to Tarvainen & Valpola (2017). The initial learning rate to handle large batches Goyal et al. (2017) is set to 0.1; the base learning rate is 0.025 with a single cycle of cosine annealing.
Semi-supervised learning Due to the large images, the batch size set to 40 in total with 20/20 for labeled and unlabeled samples, respectively. We found the Kullback-divergene leads to no meaningful network training. Hence, we set the consistency loss to mean-squared-errorr with weight of 1000. We consistency ramp up of 5 epochs to give the mean teacher more time in the beginning. Weight decay set to 5e-5; patience is four epochs to stop training in the current filtering iteration.
Filtering We filter noisy samples with the topk=5 strategy, instead of taking the maximum- likelihood (ML) prediction as on CIFAR-10 and CIFAR-100. That means the samples are kept for supervised training if their provided label lies within the top predictions of the model. The main
13

---

Published as a conference paper at ICLR 2020
min P'aanee (x, D)
with max NI..I D)
Predictions
Gradient ascent with negative weights
(a)
(b)
Figure 5: Simple training losses to counter lahel noise. (a) shows the prediction of a sample given model. The red bar indicates the noisy label, blue the correct one. Arrows depict the magnitude of the gradients (b) Typical losses reweighting schemes are not wrong but suffer from the gradient vanishing problem. Non-linear losses such seattive-oo-likelhhooee are not designed for gradient ascent.
reason that each image of ImageNet might contain multiple objects. Filtering with MI-pprddicions too strict and would lead small recall of the detection of the correct sample.
Data Augmentation For all data, normalize the RGB-images by the mean: (0.485, 0.456, 0.406) and the standard variance (0.229, 0.224, 0.225). For training data, we perform random rotation of up to 10 degrees, randomly resize images to 224x224, apply random horizontal flip and random color jittering. This noise is needed in regular mean-teache training. The jittering setting are: brightness=O..4 contrast=0.4, saturation=0.4, hue=0.1. The validation data are resized to 256x256 and randomly cropped to 224x224
A.3.3 SEMI- SEMI-SUPERVISED LOSSES
For the learning of wrongly labeled samples, Fig. shows the relationship between the typical reweighting scheme and our baseline push-away-loss. Typically, reweighting is applied directly to the losses with samples weights (kk for each sample k as shown in Eq. 4
min w (k) NLL( (x)) D)
D is the dataset, (K) and Viohet the samples is and its noisy label. w (k) is the samples weight for
(k)
the sample & at step i. Negative samples weights w!" (k) are often assigned to push the network away
from the wrong labels. Let wi(h = -c, with ci > 0, then we have:
(k)
min o (N) (R) (k), .D)
NL
(2)
Which results in:
(k) (NL D)
max C
(3)
In other words, we perform gradient ascent for wrongly labeled samples. However, the Negative- logelikeldoood not designed for gradient ascent. Hence the gradients of wrongly labeled samples vanish if the prediction is too close to the noisy label. This effect is similar to the training of Generative Adversarial Network (GAN) Goodfellow et al.] In the GAN-framework. the generator loss is not simply set to the negated version of the discriminator's loss for the same reason.
Therefore, to provide a fair comparison with our framework, we suggest the push-away-loss (yia) L (Blabel (K), D) with improved gradients as follows:
(K) NT. D))
min
(4)
Whereby Y the set of all classes in the training set. This loss has improved gradients to push the model away from the potentially wrong labels.
14

---

Published as a conference paper at ICLR 2020
(b)
Figure 6: The entropy loss for ceni-supervised learning. (a) Extreme predictions such as [0, 1] are encouraged by minimizing the entropy each prediction. (b) Additionally, maximizing the entropy the mean prediction on the entire dataset or large batch forces the model to balance its predictions over multiple samples.
Table 8: Accuracy of the complete removal of samples during iterative filtering on CIFAR-10 and CIFAR-100. The underlying model the MeanTeacher based on Resnet26. When samples are com- pletely removed from the training set, they are no longer used for either supervised-or -rr-oouuuerviiedd learning. This common strategy from previous works leads rapid performance breakdown.
CIFAR-10
CIFAR-
NOISE RATIO
40%
80 %
40%
80 %
USING NOISY DATA ONLY
DATA REMOVAL SELF (OURS)
59.98 69.91
68.99 71.98
35.53
93.7
42.09
WITH CLEAN VALIDATION SET
COMPL. REMOVAL
94.39 95.1
70.93 79.93
71.86
36.61
SELF (OURS)
74.76
46.43
Entropy minimization The typical entropy loss for semi-supervised learning is shown in Fig, 6 encourages the model provide extreme predictions (such 0 or 1) for each sample. Over large number of samples, the model should balance its predictions over all classes.
The entropy loss can easily be applied all samples to express the uncertainty about the provided labels. Alternatively, the loss can be combined with a strict filtering strategy, as in our work, which removes the labels potentially wrongly labeled samples.
For large noise ratio, predictions of wrongly labeled samples fluctuate strongly over previous training iterations. Amplifying these network decisions could lead to even noisier models model. Combined with iterative filtering, the framework will have rely single noisy model snapshot. In the case an unsuitable snapshot, the filtering step will make many wrong decisions.
A.4 MORE EXPERIMENTS RESULTS
A.4.1 COMPLETE REMOVAL SAMPLES
Tab. shows the results of deleting samples from the training set. It leads to significant performances gaps compared to our strategy (SELF), which considers the removed samples as unlabeled data. In case of a considerable label noise of 80%, the gap close to 9%.
Continuously using the filtered samples lead to significantly better results. The unsupervised-loss
provides meaningful learning signals, which should be used for better model training.
15

---

Published as a conference paper at ICLR 2020
Train SSuueet
100
Stin_Sttdent
100
Valid
Test. Studentt
80
Test Studdnt
80
Valid Teacher
Valid, Teacher
Test_ Teacher
Accuracy
Test
Accuracy
40
20
20
100
200
300
400
500
600
0
100
200
300
400
500
000
Epoch
Epoch
(a)
(b)
Figure 7: Sample training curves of our approach SELF on CIFAR-100 with (a) 60% and (b) 80% noise, using noisy validation data. Note that with approach, the training loss remains close to O. Further, note that the mean-teacher continously outperforms the noisy student models. This shows the positive effect of temporal emsembling counter label noise.
A.4.2 SAMPLEF TRAINING PROCESS
Fig. 7 shows the sample training processes of SELF under 60% and 80% noise CIFAR-100. The mean-teacher always outperform the student models. Further, note that regular training leads to rapid over-fitting label noise.
Contrary, with our effective filtering strategy, both models slowly increase their performance while the training accuracy approaches 100%. Hence, by using progressive filtering, our model could erase the inconsistency in the provided labels set.
16


Reviews:
[{"review_id": "rkek1NdX5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposed a self-ensemble label filtering (SELF) method to deal with the noisy label learning problem. They progressively filter out the wrong labels during training, i.e.,  filtered samples are removed entirely from the supervised\ntraining loss, and are leveraged via semi-supervised learning in the unsupervised loss. The filtering is based on identification of inconsistent predictions throughout training. \n\nStrengths:\n1. The motivation of the paper is very clear. \n2. Experiments are conducted on various dataset CIFAR10, CIFAR-100 and ImageNet. \n\nWeakness:\n1. The contribution of SELF is not clear. Just a combining of several previously proposed components?   \n2. For the experimental comparisons, the authors at least should report the acc on clean test set, which is useful for understanding the ideal case performance. \n3. The organization of the tables and figures are somehow hard to read. \n3. The comparisons are not fair. SELF incorporate semi-supervised techniques while baselines are not. \n4. The author missed some important baselines here. \n     1) Symmetric cross entropy for robust learning with noisy labels, ICCV2019 \n     2) Joint Optimization Framework for Learning with Noisy Labels, CVPR2018 \n     3) Dimensionality-driven learning with noisy labels, ICML2018", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SylY2HN4tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed \"self-ensemble label filtering\" for learning with noisy labels where the label noise is instance-independent (in fact, the noise model is the class-conditional noise). Among the existing directions in this area, it falls into the sample selection direction, but it also takes semi-supervised learning based on the likely noisy data into account.\n\nNovelty: borderline. As other sample selection methods, the proposed one would like to identify the training data with correct labels. What's new is that the authors \"form running averages of predictions over the entire training dataset using the network output at different training epochs\" and show that \"these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch\". This is the major contribution of the paper. Furthermore, the data likely to have incorrect labels are not thrown away but used in a semi-supervised manner. This is a minor contribution, because semi-supervised learning is orthogonal to label-noise learning and everybody in this area knows the combination of them can work better in practice. Note that this is an academic/scientific paper, not an industrial product, so you don't need to combine all things that might work.\n\nSignificance: high. The proposed method significantly outperformed all baseline methods. However, it's not completely fair to compare a label-noise + semi-supervised method with other label-noise only methods... As a matter of fact, you don't need to apply perturbation consistency (or other semi-supervised) regularization after identifying the training data with incorrect labels. Semi-supervised regularization such as virtual adversarial training can even improve supervised learning.\n\nIssues: It's known under class-conditional noise model, the backward loss correction is the unique way to estimate the classification risk (or equivalently, the classification accuracy) given noisy validation data. So how can the validation (i.e., hyperparameter tuning) be performed for the proposed and baseline methods in Table 1 given noisy validation data? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SyeJgpOAtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Overall ---\n\nThis paper proposes an algorithm for learning from data with noisy labels which alternates between updating the model and removing samples that look like they have noisy labels, thereby allowing the training procedure to focus on clean samples. Overall, I found the paper very well-written, the proposed approach reasonable, and the experiments convincing. I have some questions about what assumptions are required for such a procedure to work, but in general, I think this is a strong paper.\n\n--- Major comments ---\n\n1. I found it somewhat unclear how large the methodological contribution was. In particular, has the approach of filtering out samples based on disagreement with predictions from the model been tried before (i.e. the primary contribution is self-ensembling)? When the proposed method includes multiple pieces (Mean teacher + iteratively creating a filtered dataset + self-ensembling), I recommend being *very* explicit about which parts are new contributions. With that said, I greatly appreciated the ablation experiments which really highlight the importance of each piece.\n\n2. I don't feel like I have a good sense for what assumptions need to be satisfied for the proposed method to work. For example, in section 2.2 the authors say \"If the noise is sufficiently random, the set of correct labels will be representative to achieve high model performance\". What is meant by \"sufficiently random\" here? Is there a formal version of this assumption? Do any independence or positivity assumptions need to be satisfied? Most importantly, what happens when the label noise does not look like what you expect? I would love to see some experiments examining the failure modes of the algorithm. For example, what happens when label errors are concentrated in a particular region of the feature space (or just generally depend on the features)? In this case, even if the filtering procedure work perfectly, the filtered dataset will have a different feature distribution than the data distribution leading to potential covariate shift problems. If I understood the experiments correctly, the method was only tested on label-depended noise models.\n\n3. Along the same lines: what are the necessary conditions to guarantee that this procedure converges? While the authors suggest that self-ensembling prevents samples from oscillating in and out of training set, is this a guarantee or an empirical observation? More broadly, it is not totally clear what the filtering does to the objective function or whether this procedure is even formally optimizing a well specified objective function (potentially some temperature limit of a soft-weighted objective?). \n\n--- Minor comments ---\n\n1. Figures 1 and 4 are not readable in black and grey-scale.\n\n2. I would front-load the justification for using self-ensembling. In particular, I think the two sentences starting with \"When learning under label noise,...\" on page 2 could be moved much earlier. \n\n3. I'll be interested to see what the other reviewers say, but I found Figure 2 hard to follow.\n\n4. The formatting of Section 4.2.4 makes it a bit hard to figure out where the text starts (as opposed to the table captions).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors addressed the issues raised by the reviewers; I suggest to accept this paper.
Decision: Accept (Poster)