Robust Temporal Ensembling
Authors: Abel Brown
Benedikt Schifferer
Robert DiPietro
Venue:  (2021)
Keywords: learning with noise
robust task loss
consistency regularization

Abstract: Successful training of deep neural networks with noisy labels is an essential capability as most real-world datasets contain some amount of mislabeled data.  Left unmitigated, label noise can sharply degrade typical supervised learning approaches.  In this paper, we present robust temporal ensembling (RTE), a simple supervised learning approach which combines robust task loss, temporal pseudo-labeling, and a new ensemble consistency regularization term to achieve noise-robust learning.  We demonstrate that RTE achieves state-of-the-art performance across the CIFAR-10, CIFAR-100, and ImageNet datasets, while forgoing the recent trend of label filtering/fixing. In particular, RTE achieves 93.64% accuracy on CIFAR-10 and 66.43% accuracy on CIFAR-100 under 80% label corruption, and achieves 74.79% accuracy on ImageNet under 40% corruption. These are substantial gains over previous state-of-the-art accuracies of 86.6%, 60.2%, and 71.31%, respectively, achieved using three distinct methods. Finally, we show that RTE retains competitive corruption robustness to unforeseen input noise using CIFAR-10-C, obtaining a mean corruption error (mCE) of 13.50% even in the presence of an 80% noise ratio, versus 26.9% mCE with standard methods on clean data.

Under review as a conference paper at ICLR 2021
ROBUST TEMPORAL ENSEMBLING
Anonymous authors
Paper under double-blind review
ABSTRACT
Successful training of deep neural networks with noisy labels is an essential ca- pability most real-world datasets contain some amount of mislabeled data. Left unmitigated, label noise can sharply degrade typical supervised learning ap- proaches. In this paper. we present robust temporal ensembling (RTE), simple supervised learning approach which combines robust task loss, temporal pseudo- labeling, and ensemble consistency regularization term to achieve noise-robust learning. We demonstrate that RTE achieves state-of-the-an performance across the CIFAR-10, CIFAR- and ImageNet datasets, while forgoing the recent trend of label filtering/fxingg. In particular, RTE achieves 93.64% accuracy on CIFAR-I0 and 66.43% accuracy on CIFAR-I00 100 under 80% label corruption, and achieves 74.79% accuracy on ImageNet under 40% corruption. These substan- tial gains over previous statc-of-the- accuracies 86.6%, 60.2%, and 71.31 1.3%% %. respectively, achieved using three distinct methods. Finally, we show that RTE retains competitive corruption robustness to unforeseen input noise using CIFAR- 10-C, obtaining mean corruption error (mCE) of 13.50% even in the presence of an 80% noise ratio, versus 26.9% mCE with standard methods on clean data.
INTRODUCTION
Deep neural networks have enjoyed considerable success across a variety of domains, and in partic- ular computer vision, where the common theme is that more labeled training data yields improved model performance (Hestness et al. 2017 Mahajan et al. 2018 Xie et al. [019b] Kolesnikov et al.) 2019). However, performance depends on the quality of the training data, which is expensive to collect and inevitably imperfect. For example, ImageNet (Deng et al. 2009) is one of the most widely-used datasets the field of deep learning and despite over 2 years of labor from more than 49,000 human annotators across 167 countries, it still contains erroneous and ambiguous labels (Fei- Fei & Deng, |017]] Karpathy| 20144). It is therefore essential that learning algorithms in production workflows leverage noise robust methods.
Noise robust learning has a long history and takes many forms NNaaraaann et al., 2013 Frenay & Verleysen 2014 Song et al.] (2220). Common strategies include loss correction and reweighting (Patrini et al. 2016 Zhang & Sabuncu] 22018 Menon et al. 2020). label refurbishment (Reed et al. 2014 Song et al. 2019), abstention (Thulasidasan al. 2019). and relying on carefully constructed
trusted subsets of human-verified labeled data (Li et al. 2017 Hendrycks et al. 2018 Zhang et al.
2020 Additionally, recent methods such SELF (N Nguyeen al.] 20220) and DivideMix (Li al.
2020 convert the problem of learning with noise into a semi-supervised learning approach by split- ting the corrupted training set into clean labeled data and noisy unlabeled data at which point semi- supervised learning methods such as Mean Teacher (Tarvainen & Valpola (017)) and Mix Match (Berthelot al. 2019) can be applied directly. In essence, these methods effectively discard a ma- jority of the label information so as to side-step having to learning with noise at all. The problem here is that noisy label filtering tactics are imperfect resulting in corrupted data in the small labeled partition and valuable clean samples lost to the large pool of unlabeled data. Moreover, caution is needed when applying semi-supervised methods where the labeled data is not sampled i.i.d. from the pool of unlabeled data (Oliver et al.). Indeed, filtering tactics can be biased and irregular, driven by specification eтror and the underlying noise process of the label corruption. Recognizing the success of semi-supervised approaches, semi-supervised learning such as entropy regularization for learning with noise without discarding most valuable asset, the labels?
ask: can we leverage the underlying mechanisms of

---

Under review as a conference paper at ICLR 2021
2 ROBUST TEMPORAL ENSEMBLING
2.1 PRELIMINARIES
Adopting the notation of Zhang & Sabuncu (2018), we consider the problem of classification where X' C. I2d is the feature space and y = {I....,c} cJ is the label space where the classifier function deep neural network with a softmax output layer that maps input features to distributions over labels f : X + R The dataset of training examples containing in in-sample noise is defined as D (z,, 9. 12=1 where (z,, Ds) E (X x )) and y; is the noisy version of the true label y such that p(ū = kly: = j,m;) Thjk- We do not consider open-set noise (Wang et al.) 2018). in which there is particular type of noise that occurs on inputs, Ø. rather than labels. Following most prior work, make the simplifying assumption that the noise is conditionally independent of the input, Ti, given the true labels. In this setting, we can write Tkikk = p(ū = y) = j) 7/Kk which is, in general, considered to be class dependent noise P
To aid in a simple and precise corruption procedure, we now depart from traditional notation and further decompose 775 as Py Cyk, where Pj E [0, is the probability of corruption of the j-th class and Cj.: [0, 1] is the relative probability that corrupted samples of class j are labeled as class k with Gitj 2 0, Cjj = 0 and Σx Gjk- = 1. A noisy dataset with m classes can then be described as transition probabilities specified by
F diag(P) : C diag(1- - P) II
where C' E Rmxm defines the system confusion or noise structure, P E R defines the noise intensity or ratio for each class, and I is the identity matrix. When Cjk = caj the noise is said to be symmetric and is considered asymmetric otherwise. If ratio of noise is the same for all classes then Py = p and the dataset is said to exhibit uniform noise. For the case of uniform noise, equation interestingly takes the familiar form of the Google matrix equation
F = p (1-p)))
Note that, by this definition, M)i - p- C3o 0 which prohibits d: = y:- This ensures a true effective noise ratio of p. For example, suppose there are m = 10 classes and we wish to corrupt labels with 80% probability. Then if corrupted labels are sampled from ) rather than ) \ {y), TO 0.8 = 8% of the corrupted samples will not actually be corrupted, leading to a true corruption rate of 72%. Therefore, despite prescribing p 0.8, the true effective noise ratio would be 0.72, which in turn
yields a 0.08
0.8 40% increase in clean labels, and this is indeed the case in many studies (Zhang &
Sabuncu 2018 Nguyen al. 2020 Li et al. 2020 Zhang et al. 2020).
2.2
METHODS
At very high level, RTE is the combination of noisc-robust task loss, augmentation, and pseudo- labeling for consistency regularization. We unify generalized cross entropy (Zhang & Sabuncu 2018). Aug stochastic augmentation strategy (Hendryeks et al. 2020). an exponential moving average of model weights for generating pseudo-labels (Tarvainen & Valpola 2017), and an aug- mentation anchoring-like approach (Berthelot et al. 2020) to form robust approach for learning with noisy labels.
2.2.1 NOISE-ROBUSE TASK LOss
Generalized cross entropy (GCE) (Zhang & Sabuncu 2018) is theoretically grounded noise-robust loss function that can be seen generalization of mcan absolute error (MAE) and categorical cross entropy (CCE). The main idea is that CCE learns quickly, but more emphasis is put on difficult samples which is prone to overfit noisy labels, while MAE treats all samples equally, providing noise-robustness but learning slowly. To exploit the benefits of both MAE and CCE, a negative Box-Cox transformation (Box & Cox 1964) is used the loss function
C,(f(zt). = (1 (x,)))
(3)
SSee Lee ct al. (019)) for treatment of conditionally dependent semantic noise such that Tyyk # nik- NNote that Patrini al. 2016) define the noise transition matrix T such that TyK Tjkk-
2

---

Under review as a conference paper at ICLR 2021
where q (0, 1], and f, denotes the j-th element of f. Note that GCE becomes CCE for lim -0 Cg and becomes MAE/unhinged loss when 9 =
2.2.2 ENSEMBLE CONSISTENCY REGULARIZATION
Consistency regularization works under the assumption that a model should output similar predic- tions given augmented versions of the same input. This regularization strategy is common com- ponent of semi-supervised learning algorithms with the general form of [Ipo(y (aug)) Po (yl Fauge) ) where (y]z is the predicted class distribution produced by the model having parameters 0 for input x (Zheng et al., 2016;; (aajadii et al., 2016). We build upon numerous variations from semi-
supervised learning (Laine & Alia 2017] Tarvainen & Valpola 2017, [erttelltt et al.
2019 2020
and leverage an ensemble consistency regularization (ECR) strategy as
ECR
1 DIN
Σ
(plo po (o(y)A(())
where x is the training example, is stochastic augmentation function reevaluated for each term in the summation, 8/ = at911 + (1 - a)); is a temporal moving average of model weights used to generate pseudo-label targets, and inputs are pre-processee with standard random horizontal flip and crop. In practice, this consists of initializing copy of the initial model and maintaining an exponential moving average as training progresses. Some methods directly average multiple label predictions together at each optimization step to form single pseudo-label target (Berthelot et al., 2019 Li et al. 2020) but we find pseudo-label target distributions generated by to be better suited for the learning with noise problem due to the intrinsic ensemble nature of the weight averaging process over many optimization steps (Tavaaienn & Valpola, 2017). In semi-supervised learning techniques, it is common to leverage large batch-size of unlabeled data for consistency regular- ization. However, we found that modulating N rather than the hatch size of the consistency term, yields a monotonic increase in model performance consistent with related works (Berthelot al., 2020). Moreover, in semi-supervised learning, different batches are used for between supervised and unsupervised loss terms but we find (see section 4.3) that for the case of learning with noise, batches synchronized with GCE task loss term yields superior performance.
2.2.3 AUGMENTATION
AugMix (Hendrycks et al. 2020)> is data augmentation technique which utilizes stochasticity, di- verse augmentations, a Jensen-Shannon divergence consistency loss, and a formulation to mix mul- tiple augmented inputs. Other augmentation strategies such as RandAugment (Cubuk et al., 2020). augmentations are applied sequentially with fixed intensity which can degrade input quickly. In AugMix, to mitigate input degradation but retain augmentation diversity, several stochastically sam- pled augmentation chains are layered together in a convex combination to generate highly diverse transformations. These mixing coefficients randomly sampled from Dirichlet distribution with shared concentration parameters, and the resulting augmented version of the input is combined with the original input through second random convex combination sampled from beta distribution, again with shared parameters.
2.2.4 LENSEN-SHANNEL DIVERGENCE
The Jensen-Shannon consistency loss is used to enforce a flat response of the classifier by incen- tivizing the model be stable, consistent, and insensitive across diverse range of inputs (Zheng et al. 2016). The Jensen-Shannon divergence (ISD) is minimized across distributions Porig Paugl and Paug2 of the original sample Fori and its augmented variants TaugI and Taug2 which can be under-
stood to measure the average information that the sample reveals about the identity of its originating
distribution (Hendrycks et al. 2020). This JSD term is computed with M = (Porig + Paugl + Pang2)
and is then
1 (p || M) + (L || M) + KL-(Panga M)) JSD
(5)
3
where KL(p |I q) is Kullback-Lein the stability of training in the presence of noisy labels and heavy data augmentation with a modest contribution to final classifier test accuracy as shown in Table 5
divergence from q to p. In this way, the JSD term improves
3

---

Under review as a conference paper at ICLR 2021
2.3 PUTTING IT ALL TOGETHER
We unify the various components defined in sections 2.2 together under a single parsimonious loss function at training defined
(6) which is essentially composed of robust task loss and consistency regularization. Here the JSD) term is synchronized with ECR by computing the clean distribution using par. Final performance is reported using 0'.
LRTE = C. + AISD JSD +XECR ECR
To understand the synergy of GCE and ECR it is helpful to first point out that because GCE leverages Box-Cox power transform to stabilize loss variance, can be shown in this case to be a form of maximum likelihood estimation (Ferrari & Yang 2010). The ECR term itself is based on pseudo- labeling and pseudo-labeling can be shown to be form of entropy regularization (Grandvalet & Bengio 2004) which in the framework of maximum posterior (MAP) estimation encourages low- density separation between classes by minimizing the conditional entropy of the class probabilities of the noisy data (Lee) 2013). That is, by minimizing entropy, the overlap of class probability distribution can be reduced. The implicit assumption here is that classes are, in fact, well separated (Chapelle & Zien 2005). Moreover, MAP estimation itself acts as regularization of MLE by incorporating priori knowledge of related training examples in order to solve the ill- ill-posed noisy leamnigg objective and further prevent overfitting. Indeed, entropy regularization is favorable in situations for which the joint distribution, p(z, v). is mis-specified (Grandvalet & Bengio (004) which further underpins the motivation of pseudo-labeling as apt basis for regularization of the GCE loss.
Pseudo-labeling and data augmentation often go hand-in-hand, Data augmentation serves dual pur- pose as a generic regularizer to mitigate over-fitting of noisy labels (Zhang et al.) (018)) as well as provides additional information about the vicinity or neighborhood of the training examples which is formalized by Vicinal Risk Minimization (Chapelle et al. (001). These augmented examples can be seen as drawn from a vicinity distribution of the training examples to enlarge support of the training distribution such that samples in the vicinity share the same class but does not model the relation across examples of different classes (Zhang et al., (018). Therefore, data augmentations ap- proximate samples of nearby elements of the data manifold where the difference, {(x) = A(z)) I, approximates elements of its tangent space (Athiwaratkun et al. 2019)). In this way, ECR can loosely of stochastic differential constraints at each optimization step of the clas-
be seen as generating sification task loss. While stronger augmentation can enrich the vicinity distribution, augmentation methods such as MixUp ZZaag et al. 2018)) and RandAugment (Cubuk et al. (020) can overly degrade training examples and drift off the data manifold (Hendrycks et al. 20220). When learning with noise, it is therefore essential to leverage an augmentation process rich variety but which also preserve the image semantics and local statistics such as AugMix (Hendrycks et al. 2020) so as to minimize the additional strain on an already ill ill-posed noisy learning objective. Consistent with this understanding. although Rand/Agment has been successfully leveraged in semi-supervised learning (Berthelot et al., 2020; Kurakin et al.) 20200] Xie et al., 2019a), our experiments with RandAugment proved unsuccessful for extreme levels of label noise. Moreover, AugMix augmentation used to- gether with the Lensen-Shannon consistency loss endows trained models with far superior model robustness to corrupted data in deployment as shown in Table 7
RELATED WORK
Some methods for learning with noise attempt to improve noisy learning performance head-on by leveraging augmentation as a strong regularizer to mitigate memorization of corrupted labels (Zhang af. 2018) while others attempt to refurbish corrupted labels to control the accumulation noise from mislabeled data (Song et al. 2019). A recent theme in learning with noisy labels has been to transform the learning with noise problem into a semi-supervised one by removing the labels of training data determined to be corrupted to form the requisite dichotomy of clean labeled data and a pool of unlabeled data (Nguyen et al. 2020 Li et al. 2020); then directly applying semi- supervised approaches such as MixMatch (Berthelot et al.) 2019) and Mean Teacher (Tarvainen & Valpola 2017). Other methods go so far as to require trusted human verified data and combine re- weigghttnn with re-labeling into meta optimization approach (Zhang et al. 2020).
4

---

Under review as a conference paper at ICLR 2021
Semi-supervised learning algorithms have advanced considerably in recent years, making heavy use of both data augmentation and consistency regularization. MixMatch (Berthelot et al.) 2019) proposed low-entropy label-guessing approach for augmented unlabeled data and mixes la- beled and unlabeled data using MixUp. In Mitchthh, pseudo-label targets are formed by aver- aging label distributions produce by the model on samples drawn from the vicinity distribution [ Σx Pe (v))) However, this averaging requires artificial sharpening to generate low-entropy pseudo-labels. From the MAP estimation perspective, sharpening does not add auxiliary priori knowledge for the optimization step but rather prescribes desirable property of the model gener- ated label distribution. Indeed, our experiments with the use of artificial label sharpening in RTE resulted in failed training at high levels of label noise and subsequent related work recognized that stronger augmentations can result in disparate predictions so their average may not generate mean- ingful targets (Berthelot et al., (020). ReMixMatch (Berthelot et al., 2020) introduced augmentation anchoring which aims to minimize the entropy between label distributions produced by multiple weak and strong data augmentations of unlabeled data using control theory augmentation ap- proach. While pseudo-label guessing and augmentation anchoring motivate the utility of multiple augmentations of the same data, our proposed ECR for learning with noise differs in the following important ways: ECR does not use distribution alignment for "fairess"", distribution averaging, or label-sharpening: ECR forms pseudo-label targets using an exponential average of model weights and is batch-synchronizee with the GCE task loss term. Finally, the recent work, FixMatch (Kurakin al., 2020), proposes a simplified remi-supervised approach where the consistency regularization term uses hard pscudo-labeling for low-entropy targets together with a filtering step to remove low- confidence unlabeled examples but does not leverage multiple strong augmentations.
4 EXPERIMENTS
In this section we analyze the performance of RTE against various uniform noise configurations for both symmetric and asymmetric settings. For asymmetric noise, we test both the traditional con- figuration (Patrini et al. 2016), typically reported by related works, and an additional configuration defined by ( which is in the spirit of (Lee et al., 2019), where define the asymmetric noise structure using the confusion matrix of a trained shallow network. In all of these experiments, RTE outperforms existing methods. Finally, we perform additional ablation studies to better understand the contribution and synergy of the terms in equation (6) as well as to probe the efficacy of ECR.
In our experiments we consider the standard CIFAR-10, CIFAR-100, and ImageNet datasets (Krizhevsky] 2009; Deng et al. 2009). CIFAR-I0 and CIFAR- 100 each contain 50,000 training and 10,000 test images of 10 and 100 classes, respectively: and ImageNet contains approximately 1,000,000 training images and 50,000 validation images of 1000 classes. Additionally, we test net- works trained with noisy labels against unforeseen input corruptions using CIFAR-10-C (Hendrycks) & Dietterich] 2019) which was constructed by corrupting the original CIFAR-10 test set with a to- tal of 15 noise, blur, weather, and digital corruptions under different severity levels and intensities. Classifier performance is averaged across these corrppiion types and severity levels to yield mean corruption error (mCE). Since CIFAR- 10-C is used to measure network behavior under data shift, not included during the training procedure. Here, CIFAR- helps tablish a rigorous benchmark for image classifier robustness to better understand how models trained with noisy data might perform in safety-critical applications.
these 15 corruptions
To mitigate the sensitivity of experimental results to empirical, and perhaps arbitrary, choices of hy- perparameters, we present additional results that leverage Population Based Training (PBT) (Jader- berg et al. 2017 Li et al.) 2019) which is a simple asynchronous optimisation algorithm that jointly optimize a population of models and their hyperparameters.
particular, PBT discovers a per-epoch schedule of hyperparameter settings rather than a static fixed configuration used over the entirety of training. These PBT schedules, for example, allow task loss f. to vary between CE and MAE loss dynamically during training and similarly the number of ECR terms N' can be modulated to realize a form of curriculum learning. Moreover, for our purposes, PBT schedules also provide a form of quasi-ablation study, as optimization of the task-loss parameter g, the number of ECR terms N and the ECR weight λECR allows for the realization of a simplified loss which forgos these components if determined maximally beneficial. We find, in other studies, that this joint optimization of hyper- parameter schedules typically results in faster wall-clock convergence and higher final performance.
(Ho et al. 2019 Li et al. 2019).
5

---

Under review as a conference paper at ICLR 2021
Clean Corrupt
Clean
25000
5000
Corrupt
20000
4000
Count
Count
15000
3000
10000
2000
5000
1000
8
0
3
Cross Entropy Loss
Cross Entropy Loss
Loss distributions for clean labels versus corrupt labels on CIFAR-10 with 40% label noise (left) and 80% label noise (right). All losses are computed with respect to labels used during training, which mimics realistic setting (no access to clean labels).
Figure 1:
4.1 UNIFORM SYMMETRIC NOISE
Training Setup. Please see Section C the Appendix.
Baselines. To best interpret the effectiveness of RTE, we compare our results to many techniques for learning with noise (Table I description of each baseline method can be found in Appendix B Only two of these references provide ImageNet results trained with label noise (Table 2
Results. Experimental results with uniform symmetric noise for both CIFAR- 10 and CIFAR- are presented in Table [ with comparisons to related work, including current oate-of-tthe-ar methods. RTE establishes new state-of-the-ar performance at all noise levels and exhibits especially large performance gaps at high noise levels. At 80% noise, previous state-of-the-art was achieved by (Arazo et al. 2019) in the case of CIFAR-10 and by (Li al. 2020) the case of CIFAR- 100. RTE improves performance over these methods by 7.0 absolute percentage points and 6.2 absolute percentage points, respectively. Of all of these works, only two report results on ImageNet training that performance in terms of both top-1 accuracy and also include loss distributions
with noisy labels. These are included alongside RTE results in Table 2 where once again
RTE performs favorably, improving top-5 accuracy. As in (Arazo et al. 2019) and (Li et al. 2020). over clean and corrupt labels in Figure I Here we can see that RTE prevents rote memorization of noisy labels. Moreover, Table 7 shows that RTE retains strong corruption robustness with an mCE of 12.05% and 13.50% at noise ratios of 40% and 80% respectively. as measured using CIFAR- 10-C. Put in context, experiments summarized in Table indicaae that even with extreme levels of mislabeled training data, RTE trained models have lower corruption error than models trained using standard methods using clean data.
4.2 UNIFORM ASYMMETRIC NOISE
Training Setup. For consistency, uniform asymmetric noise experiments use the same hyperpa- rameter configurations outlined for uniform symmetric noise. Here we RTE performance using both the traditional asymmetric noise configuration (Patrini et al. (0166) typically reported by related Section of the Appendix well as an additional configuration in the spirit of (Lee et al. 2019) where we define the asymmetric noise structure using the confusion matrix of a trained shallow network defined by Equation
works defined by Equation
Section of the Appendix.
The asymmetric noise defined by Patrini et al. (2016) in equation ( does not corrupt all classes but rather attempts to capture noise process whereby labelers confuse specific pairs of classes which by some is argued to be more realistic in practice (Han et al.) 2018 Ren et al., 2018). We additionally consider a rich noise structure by training shallow classifier (ResNet-10) on clean CIFAR-10 10 and use the resulting confusion matrix of this model to define the noise structure in equation 7 For example, this asymmetric noise process readily captures the phenomenon that objects blue backgrounds
often confused (e.g. birds, ships, and airplanes) and its natural
asymmetry where p(J; smp yy aIRPlane) = 0.2772 while p(ū; = aIRPLaNe|;; = sunr) = 0.4603 (locations [1, 9] and [9. in Eq. 7 Dataset statistics provided for instance of CIFAR-10 with

---

Under review as a conference paper at ICLR 2021
Table 1: Test accuracy CIFAR-IO and CIFAR-100 under uniform symmetric label noise. Results in parentheses are upper bounds since they were computed using lower noise levels. The results for Reed-Hard, S-Model (Goldberger & Ben-Reuven 2016), Forward and Co- Co-Teaching are from Nguyen et al. (2020) and the results for Mix and Meta-Learning are from Li et al. (2020). RTE provides better robustness to label noise than all other methods. Higher is better.
Method
CIFAR-I0
CIFAR-100
# Params
Noise Ratio
Noise Ratio
40%
80%
40%
80%
(Prior Work)
Reed-Hard RRed al.
2014)
69.66
51.34
S-Model
70.64
49.10
MentorNet PD
(Jiang et al.
2018
84M
77
56
14
Forward T
Patrini et al. 2016
83.25
54.64
31.05
8.90
Open Set Wang et al.
2018
78.15
Rand Weights Ren al.
2018
36.4M
86.06
58.01
Bi-Level Jenni & Favaro
2018
11.2M
89
61.6
GCE (Zhagg Sabuncu
2018
21.8M
(87.12)
(64.07)
(61.77)
(29.16)
Co- To-ahching (Han et al.
2018
81.85
29.22
55.95
23.22
MixUp Zhang et al.
2018
(71.6)
(30.8)
SELFIE (Song et
2019
86.5
62.9
RoG Lee et al. 2019
81.83
55.68
M-DYR-H Arazo et al.
2019
11.2M
86.6
48.2
PENCIL Yi & Wu
2019)
21.8M
69.12
"fail"
Meta-Learning (Li et al. 2019
(77.4)
(42.4)
SELF (Nguyen al. 2020
25.0M
93.70
69.91
71.98
42.09
DivideMix Lii et al. 2020
11.2M
94.9
79.8
75.2
60.2
(Our Work)
RTE (Manual)
13.1M
94.84
93.09
76.70
64.02
RTE (PBT)
13.1M
95.52
93.64
77.44
66.43
Table 2: Validation accuracy on ImageNet with 40% uniform symmetric label noise.
MentorNet
SELF
RTE
# Params
59M
25.0M
25.6M
Top-1 Acc
65.1
71.31
74.79
Top-5 Acc
85.9
89.92
91.26
asymmetric label noise prescribed according to equation with uniform noise ratio of 60% in
Table of Appendix
Baselines. In the case of asymmetric noise as defined in Patrini et al. 2016). by equation we compare the performance of RTE against existing work. A brief description of each baseline method can be found in Appendix[B [ In the case of asymmetric noise structure as defined in equation (7). to knowledge, prior work does exist, and report RTE performance varied noise levels.

---

Under review as a conference paper at ICLR 2021
Table 3: Test accuracy CIFAR-10 with asymmetric noise as defined in Patrini al. (2016) by
equation (8. Higher is better.
Noise Ratio: 40%
GCE
SELF
PENCIL
DivideMix
RTE
Test Acc
64.79
89.07
91.16
93.4
94.49
Table 4: RTE test performance on CIFAR-I0 for different ratios of uniform asymmetric noise de- fined according equation (7). Sharp declines in accuracy begin to occur at 65% noise due to more AUTOMOBILE images labeled as TRUCK, than actual TRUCK images labeled as TRUCK, and so on.
Noise Ratio
20%
40%
60%
65%
70%
72%
Test Acc
95.34
94.82
93.99
80.55
72.12
59.70
mCE
11.22
11.89
13.73
25.44
33.61
44.87
Results. The results for asymmetric noise presented related works defined in Patrini et al. (2016) by equation (8) with a uniform noise ratio of 40% are shown in Table Table side the perfor- mance of related methods. Again, RTE improves the ootte--f-the-rrr performance in this category, with 1.1 absolute percentage point increase over (Li et al. (020).
Test accuracy for different level of asymmetric noise using C' defined by (7) are shown in Table Even with 60% noise ratio, RTE achieves 93.99% test accuracy. The first significant decline accuracy occurs around a 65% asymmetric noise ratio, when the majority labels in class are corrupted labels from another class. That is, for 0. with C' defined by (7). there are more AUTOMOBILE images labeled as TRUCKs, than actual TRUCK images labeled as TRUCK.
4.3 ABLATION STUDIES
We perform various ablation studies better understand the contribution of each term in equation ( probe the efficacy of ECR, and compare with alternative regularization approaches. Our abla- tion results are presented in Table These ablation studies use the training configurations defined in section unless otherwise stated. We perform component analysis where we remove one com- ponent at a time from equation 6 to better understand the performance contributions of each term. Removal of any term degrades performance. We also test alternative consistency regularization ap- proaches using label guessing as proposed in MixMatch (Berthelot et al. 2019) and augmentation anchoring from ReMixMatch (Berthelot al. 2020) which both underperform by significant mar- gins compared to ECR. Moreover, our results show significant benefits in the use of EMA whereas performance degrades with the augmentation anchoring approach consistent with prior work (Berth-) elot et al.] 2019). Additionally, we test if label sharpening could benefit ECR, but we find that the artificial sharpening process amplifies noisy pseudo-labels early in training and learning collapses for high noise ratios. Similarly, we find the strong linear chains of augmentations performed by RandAugment lead training instabilities. Figure 2 summarizes the comparison of ECR tra- ditional semi-supervised approach where a larger batch size is used for unsupervised regularization terms. This comparison indicates improved noisy learning performance with batch synchronization and repeated augmentation over larger batch sizes with single augmentations, validating the use of ECR for learning with noise.
5 CONCLUSION
We introduced robust temporal ensembling (RTE), which unifies semi-supervised consistency regu- larization and noise robust task loss effective approach for learning with noisy labels. Rather than discarding noisy labels and applying semi-supervised methods, we successfully demonstrated a new approach for learning with noise which leverages all the data together without the need filter,
8

---

Under review as a conference paper at ICLR 2021
Table 5: Ablation study. Test accuracy reported from CIFAR-10 with 80% noisy labels. Label guessing [Berthelot et al. 2019) and augmentation anchoring (Berthelot et 2020 use sharpen- ing temperature of 7' 0.5 as recommended in the associated related works.
Ablation
Test Acc
Ablation
Test Acc
RTE
93.09
Label Guessing, K = 2
79.09
No ECR (AECR =
61.91
Aug. Anchoring, K
83.59
No GCE (q 0)
76.08
Aug. Anchoring, K 4
83.24
No JSD (AJSD 0)
90.37
Aug. Anchoring, K = 6
83.20
with ECR, N 2, EMA
67.23
Aug. Anchoring, K 2, EMA
77.38
with ECR, N 2, no batch-syne
88.46
ECR with Label Sharpening
fail
with ECR, N 2, batch-sync
91.90
ECR with RandAugment
fail
90
90
85
Test
80
75
2
3
5
32
64
128
256
512
1024
varlable
batch-size
Figure 2: RTE ablation study using CIFAR-10 with uniform symmetric noise ratio of 80%. Left: the ECR batch entries shared with the task loss and the batch size is fixed 128, while the number of ECR terms (N*) is varied. Right: batch entries that are distinct from the task loss (analogous approach). The dashed red line on the right is the ECR baseline established using N = 8.
ECR term is used with varying ECR batch size, using more traditional semi-supervised
refurbish, abstain from noisy training examples. Through various experiments, we showed that RTE performs quite well in practice, advancing state-of-the-art performance across the CIFAR-10, CIFAR-104 100, and ImageNet datasets by 7.0, 6.2, and 3.5 absolute percentage points, respectively. Moreover, experiments summarized in Tables[aane show that despite significant label noise, RTE trained models retain lower corruption error on unforeseen data shifts than models trained using standard methods on clean data. Finally, the results of numerous ablations summarized in section 4.3 validate the composition of loss terms and their combined efficacy over alternative methods. In future work, we are interested in the application of RTE for different modalities such as natural language processing and speech where label noise can more pervasive and subjective.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. arXiv preprint arXiv:1904. 1238, 2019.
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many sistent explanations of unlabeled data: Why you should average. In International Conference on Learning Representations, 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raf- fel. Mixmatch: A holistic approach to semi smii-supervseed learning. In NewrIPS, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In International Conference on Learning Representations, 2020.
G. E. P. Box and D. R. Cox. An analysis of transformations. Journal of the Royal Statistical Society. Series B (Methodological), 26(2):211 1964. ISSN 00359246.
Olivier Chapelle and Alexander Zien. Semi- semi-super classification by low density separation. Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, 57-64 (2005), 01 2005.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In T. Leen, Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13, pp. 416-422. MIT Press, 2001.
Ekin D. Cubuk, Barret Zoph. Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In 2020 IEEE/CV Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp- 3008-3017, IEEE, 2020. doi: 10.1 1091009R/955009990 2020.0039..
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, 2009.
Li Fei-Fei and Jia Deng. Imagenet: Where have we been? where are we going? http:// image-net net org/ http/hhallenngs/takks.. 2017:magee.et_iissrr2211 11011111vvv_._00 vl.0.pdl. 2017.
i1svrc2017
Davide Ferrari and Yuhong Yang. Maximum 1 likelihhod estimation. Ann. Statist., 38(2):753-783, 04 2010. doi: 012214/09-AOS661
B. Frenay and M. Verleysen. Classification in the presence of label noise: A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 25(5):845-86) 2014.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. 2016
Yves Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. volume 17, 01 2004.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gameet (eds.), Advances in Neural Information Processing Systems 31, pp. 8527-8537. Curran Associates, Inc., 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. Proceedings of the International Conference on Learning Represen- fations, 2019.
10

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10456-10465. Curran Associates, Inc., 2018.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference Learning Representations, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep Learning Scaling is Predictable, Empirically. arXiv e-prints, art.
December 2017.
Daniel Ho. Eric Liang, Ion Stoica, Pieter Abbeel, and Xi Chen. Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules. arXiv e-prints, art. 1rXiv:1905.05399 May 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population Based Training of Neural Networks. arXiv e-prints, art. arXiv:1711.09846 November 2017.
Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 618-633, 2018
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven currieulum for very deep neural networks corrupted labels. In International Conference on Machine Learning, pp. 2304-2313, 2018.
Andrej vnet on imagenet.
leamed from competing against http://karpathy. 101p:///t//....a..../////////// 02/ what learned- from competing aaaiggaaannnstti.n convnet on imagenet/
Karpathy.
What
2014.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. arXiv e-prints, art. arXiv:1912.1 11370, December 2019.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Kurakin, Chun-Liang Li, Colin Raffel, David Berthelot, Ekin Dogus Cubuk, Han Zhang, Ki- hyuk Sohn, Nicholas Carlini, and Zizhao Zhang. Fixmatch: Simplifying sumi-supervised learning with consistency and confidence. In NeurIPS, 2020.
Samuli Laine and Timo Alia. Temporal ensembling for supi-super learning. In International Conference Learning Representations, 2017.
Dong-Hyue Lee. Pseudo-label: : The simple and efficient cuni-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. volume 97 of Proceedings of Machine Learning Research, pp. 3763-3772, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A Generalized Framework for Population Based Training, arXiv e-prints, art, 1902:1192..189. February 2019.
Junnan Li, Yongkang Wong, Qi Zhao. and Mohan S Kankanhalli. Learning leam from noisy la- beled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5051-5059, 2019.
11

---

Under review as a conference paper at ICLR 2021
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. pp. 1928-1936, 10 2017. doi: 10.1 10.110//1CV22211..111
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- lional Conference on Learning Representations, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens der Maaten. Exploring the Limits of Weakly Supervised Pretraining. arXiv e-prints, art. arXiv: 1800.00000000 00092, May 2018.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 1196-1204 Curran Asso- ciates, Inc., 2013.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling. In International Conference on Learning Representations, 2020.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems, pp. 3235-3246. Curran Associates, Inc.
Giorgio Patrini, Alessandro Rozza, Aditya Menon, Richard Nock, and Lizhen Qu Making Deep Neural Networks Robust Label Noise: a Loss Correction Approach. arXiv e-prints, art. arXiv:1609.03683. September 2016.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training Deep Neural Networks Noisy Labels with Bootstrapping. arXiv e-prints, art. arXiv:1412.6566 December 2014.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. arXiv preprint arXiv: arXiv::803.00000 10080000000 2018.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep superviie learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems Curran Associates, Inc., 2016.
29, pp. 1163-1177 1171.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing unclean samples for robust deep learning. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5907-5915, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from Noisy Labels with Deep Neural Networks: A Survey. arXiv e-prints, art. arXiv:2007. 08199, July 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 1296)119 2014.
I. Sutskever. J. Martens, G. Dahl. and G. Hinton. On the importance of initialization and momentum in deep learning. 30th International Conference on Machine Learning, ICML 2013, pp. 1139- 1147, 01 2013.
12

---

Under review as a conference paper at ICLR 2021
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged con- sistency targets improve semi--uuprrvied deep learning results. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamnet (eds.), Advances in Neural Information Processing Systems 30, pp- 1195-1204. Curran Associates, Inc., 2017.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention, arXiv preprint arXiv: 10964, 2019.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thame Luong, and Quoe V. Le. Unsupervised Data Augmentation for Consistency Training, arXiv e-prints, art. arXiv:1904, 12848, April 2019a.
Qizhe Xie, Minh- Minh-Thang Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with Noisy Student improves ImageNet classification, arXiv e-prints, art. arXiv:1911.0422) November 2019b.
Kun and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7017- 7025, 2019.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, May 2016.
Wide Residual Networks. arXiv e-prints, art.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Pazz Paz. mixup: Beyond empiri- cal risk minimization. In International Conference on Learning Representations, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gareet (eds.), Advances in Neural Information Processing Systems 31, pp. 8778-8788. Curran Associates, Inc., 2018.
Zizhao Zhang, Han Zhang, Sercan O. Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
S. Zheng, Y. Song, T. Leung, and I. Goodfellow. Improving the robustness of deep neural networks via stability training. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 4480-4488, 2016.
13

---

Under review as a conference paper at ICLR 2021
A APPENDIX: HYPERPARAMETERS
Table 6: Manual hyperparameter configurations for uniform symmetric noise experiments. The two values are AJSD and AECR.
Wt. Decay
Dropout
BS
LR
A
q
N
a
CIFAR-10
.001
.01
128
cos Zark
12,
sin(
13mk IGK
10
.99
CIFAR-100
.0005
.01
128
0.04
5,
0.3
.99
ImageNet
.001
.9
.00
256
.1, 10, .001
12, 10
0.3
3
.99
B APPENDIX: BASELINES FOR TABLE
1
In this section we provide a brief summary of the baseline methods in the main text:
Reed et al. 2014) introduce two methods for achieving prediction consistency, based on construction and one based on bootstrapping, and demonstrated empirically that bootstrapping leads better robustness to label noise. Goldberger & Ben-Reuven (2016) model the correct label as latent and having gone through a parameterized corruption process. Expectation maximization is used estimate both the parameters of the corruption process and the underlying latent label. Jiang curriculum-eeaminn
et al. (2018) introduce the idea of learning train student model to be robust to label noise. Patrini et al. (2016) estimate the noise transi- tion matrix (under the assumption of feature independent noise) and show that, given the true noise transition matrix, optimizing for the true underlying labels possible. Waang et al. (2018) intro- duce an iterative scheme that combines 1. outlier detection in feature space (acting noisy-label detection), 2. a Siamese network (taking either a clean, clean pair or a clean, noisy pair) to encourage separation, and 3. sample reweighting based on clean order to effectively filter out noisy labels during training. They focus primarily on open-ser noise, but they also report performance of their system when used in the closed-set set setting. Ren et al. (2018) use a meta-learning approach to dynamically weight examples to minimize loss using a set of validation examples with clean labels, however they also report competitive baseline using randomized weighting scheme which requires no clean validation set. Jenni & Favaro (2018) for- mulate example weighting as bilevel-optimization problem, in which performance on a validation set is maximized with respect to example weights, subject to the constraint that the model maximizes performance on the training set; and they argue that this approach should lead better generaliza- tion when label noise is present. ZZhang & Sabuncu (2018) introduce loss function that is generalization of cross-entropy loss and mean absolute error, which is beneficial since each exhibits distinct desirable properties: cross-entropy exhibits better gradient properties for learning, while mean absolute error exhibits better
strategy with a mentor model to
proxy to
noisy confidence levels in
seretiicall--ooonndee robustness to noisy labels. Han et al. (2018) leverage co-teaching such that two networks trained together, in which each network I. ieettiiie high-confidence examples, 2. passes this information in message to its peer, and 3. lever- ages the incoming message optimize using the examples selected by its peer. Zhang et al.) (2018) train using convex combinations of both input images and their labels, arguing that this approach makes it more difficult for the network memorize corrupt labels. [Sng al. (2019) measure label consistency throughout training in order to determine which samples are 'refurbishable' and these samples are then 'corrected' by replacing their ground -ruth label with the most trequentlypreddicte label. Lee et al. (2019) do not modify the training process of the underlying neural network but in- stead form a generative model over the final (pre-softmax) features of the neural network, and this generative distribution along with Bayes rule is then used to estimate more robust conditional dis- tribution over the label. Arazo et al. (2019) a beta mixture model over the loss using two mixture components, representing clean and noisy labels, and each sample's underlying component proba- bilities are used to weight each sample's 's contribution during training. They combine this approach with MixUp (Zhang al. 2018). Yi & Wu (2019) maintain a direct estimate of distribution true underlying labels during training, and train the parameters neural network by minimizing reverse KI. divergence (from the model's predicted distribution to these true-label estimates). Mean-
14

---

Under review as a conference paper at ICLR 2021
while 'compatibility loss' is introduced to ensure that the estimated label distribution stays close to the noisy labels provided with the training set. [i et al. (2019) subject a student model to artificial label noise during training and take alternating gradient steps and maintain teacher model that is not subjected to such noise. Here, alternating gradient steps are taken to 1. minimize classification loss and 2. minimize the KI. divergence from the student's predicted distributions to the teacher's predicted distributions. Nguyen et al. (2020) use discrepancy between an ensemble-based teacher model and labels to identify and filter out incorrect labels, and continue leverage these samples without the labels. This is done in semi-supervised fashion by maintaining consistency between the teacher's predictions and the student's 's predictions. Li et al. (020) maintain two networks and for each network models loss using mixture of Gaussians with two components (clean and noisy). Each network estimates which samples belong to each component, and the other network then uses the clean samples in supervised manner along with the noisy labels in semi-supervised manner.
C APPENDIX: UNIFORM SYMMETRIC NOISE EXPERIMENTAL SETUP
For CIFAR-10, 10, leverage equation with C and employ a 28-layer residual network (He et al., 2016) with widening factor of 6 (WNR 28x6) (Zagoruyko & Komodakis 2016). dropout rate of 0.01 (Srivastava et al.) 2014). a = 0.99, AugMix with a mixture width and severity of 3, batch size of 128, and 300 epochs of training. We optimize using SGD with Nesterov momentum of 0.9 (Sutskever et al. 2013 a weight decay of 0.001, and a cosine learning rate Loshchilov & Hutter 2017) of 0.03 cos (mmk/16 where & is the current training step and K is the total number of training steps. The RTE loss function (6) is configured with static ÄISD- λECR and N of 12, 1, and 10, respectively, whereas q is scheduled according to 0.6 sin(13 1 (which assigns small g-values in early training epochs, reaches a maximum of = 0.6 after 180 epochs, and decreases q = 0.33 over the remaining 120 epochs). For CIFAR- 100, the setup similar, but different hyperparameters are used; details are included in the Appendix in Table 6 In addition to manual configurations, we consider PBT with a population size of 35 to optimize learning rate, weight decay. g. ÄISD, λECR and N Fastidious readers will find the complete PBT configuration defined in Appendix [F) For ImageNet, ResNet50 is used and trained with SGD for 300 epochs with stepped learning rate of 1, 0.01 and 0.001 which begin at epochs 0, 100 and 200 respectively. ImageNet hyperparameters are also included in the Appendix Table
D APPENDIX: CONFUSION MATRIX FOR UNIFORM ASYMMETRIC NOISE
.0000
.0396
2475
.0594
.0594
.0396
.0495
.0693
.2772
.1584
.1765
.0000
.0294
.0000
.0000
.0000
.0294
.0000
.1665
.5822
.1745
.0000
.0000
.1544
.1879
.1074
.2617
.0872
.0268
.0000
.0388
.0116
.1473
.0000
.1240
.3682
.1899
.0853
.0155
.0194
.0303
.0000
.2197
.1667
.0000
.0606
.2879
.2121
.0227
.0000
C=
.0324
.0000
.1435
.4676
.1019
.0000
.1204
.1157
.0093
.0093
.0536
.0179
.3571
.3036
.1071
.0714
.0000
.0536
.0179
.0179
.0704
.0000
.0986
.1268
.3803
.1831
.0986
.0000
.0000
.0423
.4603
.0952
.0794
.0476
.0317
.0000
.0476
.0317
.0000
.2063
.1711
5132
.0263
.0526
.0263
.0132
.0658
.0395
.0921
.0000
E APPENDIX: PERFORMANCE DATA ON CIFAR-10-C
Table 7: RTE mean corruption error on CIFAR-10-C for models trained at various uniform sym- metric noise ratios. Baseline reference values for 'Standard' and 'AugMix' mCE are reported from Hendrycks et al. (2020) using WRN 40x2 on clean data. Lower is better.
Noise Ratio
Standard
AugMix
0%
40%
80%
mCE
26.9
11.2
11.5
12.05
13.50
15

---

Under review as a conference paper at ICLR 2021
F APPENDIX: PBT EXPERIMENTS
Table 8: PBT sampling configuration for CIFAR-10 and CIFAR-I00. We used a population size of 35, and permutation interval of 2 epochs. In the each hyperparameter resampled from its distribution with p = 0.25 or is multiplied with Uniform(O.8, 1.2) within its parameter range with p 0.75. In the case of N the previous/next hyperparameter from the ordered list is selected.
member inherits another checkpoint,
Parameter
Sample distribution
Batch size
128
Dropout
0.01
B
0.9
0.99
LR
Uniform((.00000) 0.1)
weight decay
Uniform(O.00000)) 0.002)
Uniform(0.0, 1.0)
ASSD
Uniform(O.0. 20.0)
AECR
Uniform(0.0, 5.0)
N
Uniform (3, ... 10)
G APPENDIX: UNIFORM ASYMMETRIC NOISE ON CIFAR-10
The matrix defines the noise structure for uniform asymmetric noise on CIFAR-10 with following labels: AIRPLANE, AUTOMOBILE, BIRD, CAT, DEER, DOG, FROG, HORSE, SHIP, TRUCK.
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
C
(8)
0
0
0
0
0
0
0
0
0
0
0
0
0
0
H APPENDIX: EXTENDED DATA AND ANALYSIS
16

---

Under review as a conference paper at ICLR 2021
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Escets
Epechs
Foochs
Frochs
00020
00220
0:230
0.0000
0.0000
0.005
0.000
0.0000
0,0005
310
305
Spocts
Epecha
Epechs
20
50 10ggs
5
100
300
350
200
200
100
200
2000
0
1000
200
302
1.00
200
300
1.00
1.0
1.0
1.0
0.8
0.5
0.0
0.0
0.2
320
300
200
200
200
200
300
Spects
Epechs
Epects
Epecha
10-
10-
10
10
Number EDR DETTE
300
200
200
100
Spocha
Epocio
Exoclo
Epoclo
Figure 3: Parameter schedules for ir, weight decay. JSD weight ÄJSD, ECR weight AECR, q and N for CIFAR-10 and 1IFAR-100 with 40% and 80% uniform symmetric noise rates.
17

---

Under review as a conference paper at ICLR 2021
Table 9: Overview of class distribution of total and correct labels after sampling noisy CIFAR-10 training labels with asymmetric noise defined by equation with uniform 60% noise ratio.
# samples
% samples
# correct labels
% correct labels
AIRPLANE
5578
11%
1958
35%
AUTOMOBILE
4069
8%
2003
49%
BIRD
6023
12%
2017
33%
CAT
6205
12%
2038
33%
DEER
5056
10%
1986
39%
DOG
4480
9%
1977
44%
FROG
5476
11%
2019
37%
HORSE
4130
8%
2028
49%
SHIP
3896
8%
2024
52%
TRUCK
5087
10%
1950
38%
TOTAL
50000
100%
20000
40%
Table 10: RTE test accuracy and mean corruption emoor (mCE) CIFAR-I0 and CIFAR-IO-C, respectively. this experiment, fixed batch size bs 128 used and the number of ECR terms, N is varied. Training configuration of these data is described in section 4.1 Test accuracy is presented in Figure (left).
CIFAR-10
Fixed batch-size: 128
Uniform Symmetric Noise: 80%
Vary the number of ECR terms: N
2
3
4
5
8
1 Test Ace
91.51
91.90
92.57
92.65
92.77
93.14
93.09
93.21
mCE
15.32
14.87
13.74
13.90
13.84
13.48
13.67
13.66
Accuracy
Accuracy
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Confidence
Confidence
Figure 4: Reliability diagrams for RTE training models CIFAR-10 with 40% uniform label noise (left) and 80% label noise (right). Perfectly calibrated models follow the black line, whereas over- confident models lie below and under-confident models lie above. This figure indicates RTE trained model is well calibrated when trained with 40% label noise, while (perhaps justifiably) con- servative when trained with more extreme level of 80% label noise.
18

---

Under review as a conference paper at ICLR 2021
Table 11: RTE test accuracy and mean corruption error (mCE) on CIFAR-10 and CIFAR-IO-C. respectively. In this experiment single consistency loss term used and vary the batch size of that term. This experiment with varying batch size analogous to a more traditional semi-supervised approach where large batch size used for unsupervised loss terms. Training configuration for these Test accuracy is presented in Figure (right).
data is described in section 4.1
CIFAR-10
Fixed ECR terms: N 1
Uniform Symmetric Noise: 80%
Vary the batch size:
32
64
128
256
512
1024
Test Acc
86.54
88.95
90.32
88.46
85.87
78.13
mCC
19.77
17.78
16.41
18.20
20.42
28.57
19


Reviews:
[{"review_id": "y2oZ7J94cY4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: this paper proposes a method for learning with label noise. The proposed method combines three techniques: GCE from robust loss literature, AugMix from data augmentation literature, and Mean Teacher from semi-supervised learning literature. This paper shows that the combination of these methods are effective for label noise learning.\n\nStrength: this paper studies an important problem; the combination of existing methods is intuitive, and each method plays an important role; the paper is mostly well-written and easy to follow.\n\nWeakness:\n1. The paper claims to \"introduce a new ensemble-based form of consistency regularization which leverages multiple augmentations of the same images\". However, this strategy has already been used by methods such as MixMatch (MixMatch: A Holistic Approach to Semi-Supervised Learning).\n2. The proposed method seems to be a rather ad-hoc combination of several existing methods (GCE, AugMix, Mean Teacher), hence the technical novelty is limited. It is fine as long as the experimental results are strong, which I am not fully convinced.\n3. The comparison with some previous methods on CIFAR seem to be unfair. For example, DivideMix uses a 18-layer PreAct ResNet, whereas this paper uses a 28- layer Wide ResNet. It is important to make sure that all previous methods are compared under a fair setting.\n4. How important is AugMix to the model's performance? What if a different augmentation is used?\n5. The results in Table 5 (robustness to data shift under label noise) is expected because AugMix does not consider label noise. Hence it is hard to justify the value of this experiment.\n6. The propose method is not validated on real-world noisy datasets, such as the widely used WebVision, Food-101, or Clothing1M. Experiments on synthetic noisy datasets alone cannot fully justify the effectiveness of the method, because real-world noise can be more complicated.", "extra_scores": {}, "extra_text": {}}, {"review_id": "xtvBxANnUBB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary: \n\nReal-world data contains noise in the annotated labels. To mitigate, the authors propose a supervised learning approach, Robust Temporal Ensembling (RTE). RTE combines 1) task loss correction, which is a generalized cross entropy loss, 2) different augmentations resulting from AugMix technique and the Jensen-Shannon divergence (JSD), 3) the ensemble consistency regularization and pseudo labeling.\n\n##########################################################################\n\nReasons for score: \n \nOverall, I vote for accepting. The idea of improving the robustness predictions in noisy labeled data is interesting. My major concerns are on the motivation and the clarity of the idea at some places. Hopefully the authors will address these concerns in the rebuttal period. \n##########################################################################\n\n\nPros:\n+ Overall, the paper is well written with minimal to no grammatical errors, easy to follow and understandable. \n\n+ The approach is well motivated, clearly describes the use of loss functions and the ensemble consistency, etc. Positions the proposed approach on how the existing techniques are combined and further improved.\n\n+ The paper is strong in terms of empirical evidence when combined with multiple existing techniques.\n\n+ Ensuring the true effective noise ratio is mathematically interesting, as opposed to the state-of-the-art practices. \n\n+ The idea of forgoing the identification and filtering/fixing noisy labels is an encouraging piece of contribution which further strengths this line of research.\n\n+ Solid experimental results, although having results on other benchmarks does not hurt. However the ImageNet results dominate.\n\nCons:\n- Motivation for this work can be further improved (In the Introduction in Paragraph 1). I see the citation for Fe-Fei Li work on Imagenet.\n\n“Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, 2009.”\n\n- Maybe this can be used early on and you can highlight the problem of label noise in the following way, if not exactly, “Amazon Mechanical Turk spent 49k people spread across 167 countries over 2.5 of years. Yet, Imagent has label noise ….”\nThe long history of noise robust learning is missing the approaches of abstention, falling under identifying and filtering or fixing the incorrect labels.\n\n“Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff A. Bilmes, Gopinath Chennupati, Jamal Mohd-Yusof: Combating Label Noise in Deep Learning using Abstention. ICML 2019: 6234-6243”\n \n“Lihong Li, Michael L. Littman, and Thomas J. Walsh. 2008. Knows what it knows: a framework for self-aware learning. In Proceedings of the 25th international conference on Machine learning (ICML '08).”\n \n- Novelty of the method is suboptimal. The proposed technique is a combination of methods in literature.\nAlthough true effective noise ratio is interesting, it does not really make much difference in ImageNet kind of large number (1000) of classes (1/1000 * 0.8) .\n\n- “In semi-supervised learning techniques it is typical to leverage a larger ...” the above sentence does not flow well, maybe worth re-written.\n\n- Too many hyper-parameters\n\n- “... while q is prescribed an ad-hoc schedule ...” does not make sense, modify please\n\n- Is this effective noise ratio kept exactly 40/80% for the baselines unlike the common practice? From the results, it does not appear that way, i) please clarify, ii) if not setting those noise ratios might benefit the paper even more because the baselines might perform even worse.\n\n- The temporal ensembling part is little vague in the current form of the paper, probably hiding/missing from section 3.2.3. Can you be more specific with the details for this?\n\n- Also, is it possible to stud the certainty of predictions, in terms of calibration error or some other metric? If they can be squeezed in the paper somehow, that will further strengthen the paper. Moreover, one can actually understand the effect of ensembles on the robustness of predictions in noisy environments.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "uOxW4Kj5Td5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall impression\nThis submission deals with robust supervised learning in the presence of noisy labels. The label noise is modeled using a probabilistic (and conditionally independent) transition matrix that changes the label of one class to another one. In order to classify with noise, the network is trained with a mixture of three known losses including: 1) generalized cross entropy (GCE) rejects the outlier labels, 2) JSD divergence to assure the soft-max distribution matches the augmented data distributions, and 3) an ensemble consistency regularization (ECR) that penalizes the inconsistencies of the augmented data based on the mean teachers. Experiments with CIFAR-10, CIFAR-100, and ImageNet classification indicate substantial gains compared with state-of-the-art alternatives. \n\nStrong points:\n- The empirical study of combining three different metrics is extensive and useful; the gains are also significant; ablation study is also useful\n\nWeak points:\n- The contribution is rather incremental, combining three known loss metrics in outlier detection and semi-supervised learning\n- The motivation behind eq.5 is not very clear. Why is it needed to use augmented data in eq. 5? why not simply using mean-teachers (ensemble of previous network weights) to penalize the consistency between the predicted labels and the noisy ones, in the same spirit as mean teachers? \n\n\nSuggestions:\n- The paper would improve if Table 1 and the ablation study can include the results for ImageNet as a more realistic dataset as well .\n- What is index i running over in eq (5)?\n- A lot of details about experiments are provided in the main paper. For more clarify, those could be moved to appendix, and more intuition and explanation about the reason for adopting the three loss components would be more useful. perhaps toy examples with diagrams could be helpful.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4TwqNmTmOIk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\nThis work aims to train models with noisy training labels.  \n1. This work introduces Robust Temporal Ensembling, which is composed of ideas introduced in prior work:\n - Noise-robust task loss (Generalized cross entropy from (Zhang & Sabuncu, 2018))\n - Augmentation (JSD loss from AugMix (Hendrycks* et al., 2020))  Minimizes the KL divergence between each of the three model outputs (original image + two augmentations) and the average of those three outputs.\n - Ensemble consistency regularization (ECR) combines (Tarvainen & Valpola, 2017) which uses a moving average of model parameters and the consistency regularization formalized in (Laine & Alia, 2017) to compare the difference in outputs of the teacher and a set of augmentations from AugMix.\n2. These three factors contribute to the final loss.  The remainder of the work demonstrates the advantage of this combined loss through a barrage of experiments. \n - They show that on CIFAR-10, CIFAR-100 (with 40% and 80% corruption) and ImageNet (with 40%) corruption, their approach outperforms all baselines, often by a considerable margin.\n - They also explore the impact of different types of noise (class-symmetric, asymmetric, and varying degrees of noise)\n - Ablation experiments show that the main novelty (ECR) plays the largest role in the performance of the model, but GCE is also needed to get state of the art performance on CIFAR-10 (80%)\n - Finally, they show that using multiple augmentations in ECR in a single batch can considerably improve performance.\n\n**Positives:**\n - This approach clearly outperforms all the compared baselines (to my knowledge, they aren’t missing any comparisons)\n - The experiments are quite extensive.\n - The exposition clearly explains how this work relates to prior work and how it composes those ideas into the final model.\n**Negatives:**\n - This work leans heavily on prior work.  See summary above for how it relates to prior work.\n**Recommendation:**\nThe main contribution of this work is the ECR term, which uses an idea from semi-supervised learning to match the prediction to a “mean teacher”.  By extending it to use multiple augmentations, they improve results over using a single augmentation (without mean teacher Laine & Alia, 2017) or without augmentations (vs. a mean teacher, Tarvainen & Valpola, 2017).\nThe question is whether this combination of loss terms is a significant enough contribution.  I think it’s borderline, but since the performance gain is so significant and since I can’t think of any way to improve the work, I’ll lean toward accepting.\n\n\n\n\n**Minor comments:**\nEq (5) is missing a  )\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reading the paper and the reviews themselves, I found myself conflicted about this work:

- Multiple reviewers commented that this is a rather incremental piece of work, given that it's a rather straightforward combination of existing losses/models.
- On the other hand, there is admittedly value in (1) realizing that this combination is meaningful (2) understanding the meaningful ways in which these work or do not work with ablation studies.
- I am not quite satisfied that the datasets and experiments in this work represent in any meaningful way real world noise. However, it does appear that the authors ran experiments on common benchmarks using common protocols so there's only so much that they themselves can be blamed for.
- Tangentially, I am somewhat surprised about the relatively good ImageNet performance of this method. I suspect the combination of this being done with uniform noise rather than structured noise is helping quite a bit.

All in all, this work is certainly interesting enough, but the results are just not quite compelling enough to pass the bar.
Decision: Reject