Robust Curriculum Learning: from clean label detection to noisy label self-correction
Authors: Tianyi Zhou
Shengjie Wang
Jeff Bilmes
Venue: ICLR 2021 Poster (2021)
Keywords: curriculum learning
noisy label
robust learning
training dynamics
neural networks

Abstract: Neural network training can easily overfit noisy labels resulting in poor generalization performance. Existing methods address this problem by (1) filtering out the noisy data and only using the clean data for training or (2) relabeling the noisy data by the model during training or by another model trained only on a clean dataset. However, the former does not leverage the features' information of wrongly-labeled data, while the latter may produce wrong pseudo-labels for some data and introduce extra noises. In this paper, we propose a smooth transition and interplay between these two strategies as a curriculum that selects training samples dynamically. In particular, we start with learning from clean data and then gradually move to learn noisy-labeled data with pseudo labels produced by a time-ensemble of the model and data augmentations. Instead of using the instantaneous loss computed at the current step, our data selection is based on the dynamics of both the loss and output consistency for each sample across historical steps and different data augmentations, resulting in more precise detection of both clean labels and correct pseudo labels. On multiple benchmarks of noisy labels, we show that our curriculum learning strategy can significantly improve the test accuracy without any auxiliary model or extra clean data.

Published as a conference paper at ICLR 2021
ROBUST CURRICULUM LEARNING: FROM CLEAN LA- BEL DETECTION TO NOISY LABEL SELF-CORRECTION
Tianyi Zhou", Shenggic Wang", Jeff A. Bilmes University of Washington, Seattle (t (tianyi zh, wangaj, bi lmes| @uw.wwdd edu
ABSTRACT
Neural network training can easily overfit noisy labels resulting in poor gener- alization performance. Existing methods address this problem by (1) filtering out the noisy data and only using the clean data for training or (2) relabeling the noisy data by the model during training by another model trained only a clean dataset. However, the former does not leverage the features' information of wrongly-labeled data, while the latter may produce wrong pseudo-labels for some data and introduce extra noises. this paper, we propose a smooth transition and interplay between these two strategies as a curriculum that selects training samples dynamically.
particular, we start with learning from clean data and then gradually move to learn noisy-labeled data with pscudo labels produced by a time-ensemble of the model and data augmentations. Instead of using the instantaneous loss computed at the current step, data selection is based the dynamics of both the loss and output consistency for each sample across historical steps and different data augmentations, resulting in more precise detection of both clean labels and correct pseudo labels. On multiple benchmarks of noisy labels, we show that our curriculum learning strategy can significantly improve the test accuracy without any auxiliary model or extra clean data.
INTRODUCTION
The expressive power and high capacity of deep neural networks (DNNs) result in accurate modeling and promising generalization if provided with sufficient data and clean(correct) labels. However, recent studies show that the training process is fragile and can easily overfit on noisy labels (Zhang (et 2017), which commonly appear in real-world data since precise annotation is not always available or affordable. Hence, it is important to study the training dynamics affected by imperfect labels and develop robust learning strategies that ideally eliminate the negative impact of noisy labels while fully exploiting the information from all the available data.
Numerous approaches have been developed address this challenge from various perspectives, e.g..
loss correction (Xiao et al. 2015 Vahdat 2017 Lee et al.) 2018 Veit et al.
[017]] Li et al. 2017b),
robust loss functions (Ghosh al. 2017 Zhang& & Sabuncu, 2018 Wang al.
2019 Ma et al. (0200)
with provable noise tolerance, sample re- weighting (Patrini et al.
2017). 7, curriculum learning (Kumar
let al. 2010:) Jiang al.] 2018 Guo et al. (0188), model co-teaching (Han al. 2018). etc. A principal methodology behind a variety of methods is to detect clean labels while discard/downweigh the
data with wrong labels, so the model mainly learns from correct labels. A broadly applied criterion is to select the samples with small losses and treat them as clean data. It is inspired by empirical observations that DNNs learn simple patterns first before overfitting on the noisy labels (Zhang et al. 2017 Arrpit et al. (017). Several curriculum learning methods utilize this criterion (Kumar et al.) 20100) Tiang et al. 2014), ) and in each step, select/upweigh samples with small losses. Robust loss functions also suppress the large losses associated with the possibly wrong labels. More recent approaches use mixture models (Arazo et al. 2019 estimate the distribution of losses for clean and noisy data.
However, the instantaneous loss (i.e., the loss evaluated at the current step) of an individual sample is an unstable signal that can rapidly fluctuate due to DNN training's randomness. The error generated by such an unstable metric accumulates when the selected samples are used to train the model producing the losses. Co-teaching methods alleviate this problem by training two DNNs and using the loss computed on one model to guild the other. Also, as the model changes during training, each sample's loss needs to be re-evaluated even when it is not selected, which requires extra inference

---

Published as a conference paper at ICLR 2021
cost. MentorNet (Jiang al.) 2018) and Data Parameters (Saxena al. 2019) train extra model to produce the sample weights or selection results without computing the loss. Furthermore, it may not be efficient to repeatedly train the model only on clean data that consistently have small losses, since the model have already learned, well memorized or overfitted to them.
A primary drawback of training only on clean labels detected is that discarding the whole data pairs (z.z) with wrong labels y removes potentially useful information about the data distribution p(z) (Arazo et al. 2019). Hence, there has been growing interest in leveraging noisy data. Loss correction methods aim to correct the predicted class probabilities based on an estimated mislabeling probability between classes. Some other methods seek to relabel them by using the model itself (e.g., bottstaatg loss (Reed et al. 2014)) or another model/mechanism (e.g., directed graphical models, conditional random fields, CNNs) trained however, is not always available. Self-training and unsupervised learning techniques (Rasmus et al. 2015; Berthelot et al., (219)) have also been employed to generate pseudo labels to replace noisy labels (Arazo et al. 2019). The pscudo labels are optimized together with the model or generated by the model with data augmentations to encourage the output consistency on the same sample's s augmentations. Unfortunately, the pseudo labels' quality may vary across different samples and significantly degenerate when the noise ratio is high, or the model fails to produce stable and correct predictions. In such a case, the relabeling error on some samples can be accumulated during training.
additional set of clean data, which,
In this paper, we address the aforementioned problems of noise-label learning by developing curriculum learning strategy called Robust Curriculum Learning (RoCL) that smoothly transitions between two phases: (1) detection and supervised training on clean data; and (2) relabeling and self-superrision on noisy data. Specifically, we train the model for multiple episodes, each starting from plase(I) and gradually moving to phase(2). Unlike existing approaches, we only select samples with accurate given/pscudo labels that are most informative to the current model training. Our data selection criterion takes both the dynamics of per-sample loss and output consistency (across multiple data augmentations) into account. Using an exponential moving average of the loss and consistency over training history, it overcomes the instability of instantaneous losses and does not incur any additional inference cost. In addition, by adjusting a temperature parameter, the criterion can interpolate between the two phases and keep the training focusing on the data that the model mostly needs to improve on, e.g., clean data with unsatisfying output consistency or wrongly- labele data with accurate pseudo labels. Thus, we can fully exploit both clean and noisy data more efficiently with less risk of introducing extra noise or error accumulation. We further show that our data selection can be derived from novel optimization formulation for robust curriculum learning. We evaluate our method on multiple noisy learning benchmarks and show that our method outperforms diverse set of recent noisy-label learning approaches.
1.1 RELATED WORK
Early curriculum learning (CL) Khan et al. 2011 Basu & Christensen 2013 Spitkovsky et al. 2009
Zhou et al.) 2021) seeks optimized sequence of training samples (i.e., curriculum,, which be designed by human experts) to improve model performance. Self-paced learning (SPL))Kumar et al., 2010; Tang et al. 2012a Supancic III & Ramanan 2013] Tang et al.) 2012b) selects easy samples with smaller losses. starts with selecting a few samples of small loss and gradually increases the selection size to cover all the training data. Self-paced curriculum learning (Jiang et al., 2015) combines the human expert in CL and loss-adaptation in SPL. SPL with diversity (SPLD) (Jiang jet al. 2014) applies a negative group sparse regularization to SPL to promote the diversity of selected samples. Minimax curriculum learning Zhou & Bilmes (0118) promotes the diversity of samples during early learning encourage exploration and focus on hard samples in later stages.
In the context of robust learning with noisy labels, label correction methods aim to identify the wrong labels and possibly correct them get more consistent labels for training. Previous work often apply an extra noise model (directed graphical model (Xiao et al. 2015), conditional random fields (Vahdat) 2017), neural network (Lee et al.) 2018: Veit et al. (017), knowledge graph (Li et al.) (017b) to cor- rect the noisy labels, which often require extra clean data and as well as training/Inferene of the noise model. Another line of research focuses on loss correction, which modifies the loss prediction prob- abilities during training to correct the misinformation from the noisy labels. Patrini et al. (2017) uses two noise transition (backward and forward) matrices to correct the prediction probabilities. Label Smoothing Regularization (Szegedy et al.] 2016; Pereyra et al. (017) alleviates the overfitting to noisy labels by using soft labels instead of hot labels. Reed et al. (2014) augments the loss with a notion
2

---

Published as a conference paper at ICLR 2021
of perceptual consistency. Jiang et al. (2018) trains mentor network to reweigh samples duri 'ig the training of a student network. Guo et al. (2018) 8 designs a curriculum by ranking the complexity of data using its distribution density in feature space. Ren al. (2018) proposes
meta-leariig algo- rithm that Iearns to assign weights to samples based on their gradients in training compared to those of validation data, which requires extra clean data. Co-teaching (Han et al. 2018) feeds in the network with the most confident samples of another network to reduce confirmation bias. [miid et al. (2019) generalizes the logistic loss and the exponents in the softmax by applying temperature to each of them and makes the training more robust to noise. Hu et al. (019)) trains a network on noisy labels in the weakly supervised setting and uses it regularization term improve the training on clean data.
Some approaches focus on designing loss functions that have robust behaviors and provable tolerance to label noise. Ghosh et al. (2017) theoretically proves that the Mcan Absolute (MAE) is robust loss. The Generalized Cross Entropy (Zhang & Sabuncu, 2018 uses a negative Box-Cop transformation to obtain loss function that generalizes MAE and Cross Entropy loss. Wang et al. (2019) proposes a Symmetric Cross Entropy that combines Cross Entropy loss and Reverse Cross Entropy loss. Ma et al. (2020) proposes loss normalization method and shows that any loss can be made robust to noisy labels.
RoCL shares similar ideas with some CL methods in that RoCL starts with learning easy and clean samples and gradually moves to hard and noisy ones. RoCL is more related to the loss correction approach in noisy-label learning literature RoCL generates curriculum dynamically assigning weight (probability) to each sample. RoCL differs from existing methods in: (1) it only selects a subset of informative and reliable labels for training in each epoch; (2) it is smooth transition not only from clean data to noisy data but also from supervised learning to sul--sueerision: (3) it runs multiple episodes of the curriculum to avoid getting in local minimum dominated by small set of clean/noisy data or a specific type of loss; (4) it does not assume the availability of an extra set of clean data; (5) it does not require extra computation or any modification to the model.
2
DYNAMIC PATTERNS OF CLEAN/NOISY LABELS IN TRAINING
2.1 LOss DYNAMICS AND CLEAN LABEL DETECTION
A key challenge for most noise-label 24 learning methods is to design a reliable criterion to select/reweigh clean data and distinguish them from the noisy data, all the clean data can be fully exploited while most noisy labels are filtered out of the training process. Loss computed at an instantaneous step have been widely used for this purpose according to the observation that the loss on clean data is usually smaller than noisy data. One important reason is that the clean labels are mutually consistent with each other in producing gradient updates, and therefore, the model can fit them better and faster. On the other hand, the noisy labels may contain mutually inconsistent information, creating a form of long-lasting "tug of war" amongst themselves. For example, it can be hard for the model to find consistent vi- sual patterns from images with noisy labels to make the desired predictions. However, instantaneous loss suffers from high vari- across training epochs (as shown in the first plot of Figure [ and is inaccurate for clean data detection under high noise ratio (i.e., the proportion of wrong labels is high) and the randomness of DNN training, e.g., random initialization, random data augmentation, etc. Moreover, it needs to evaluate the instantaneous loss for all samples in each step, resulting in extra inference cost unselected samples.
Indari tots liss cina with ave clice sne's
165
330
Dit Inrel dets coeett Dea lurel des pier loss
15 x
10
1V1
Dynamic patterns (meandist)) of instantaneous metries (top) and exponential moving average (EMA) metries (bottom) when applying Alg. thht alternates between super- vised learning on given labels and self-supervision on pseudo labels. Larger gup between curves in each plot is better. Symmetrie noise is defined in the beginning of Section 4 We use cross entropy supervised loss E(-,-) in Eq. and 0-1 loss for E(:,-) in consistency loss Eq. (2) with m
Figure 1:
3

---

Published as a conference paper at ICLR 2021
The dynamic patterns of losses (Zhou et al. 2020b) over the course of training give a new insight for better clean data detection even when the noise ratio (proportion of wrong labels) is high. In particular, we hypothesize that sample's ' label is more likely to be correct if its losses persistently retain low values over training steps. Given a sample (x),34) with v, being the features and y being the label, we describe its loss dynamics using simple exponential moving average (EMA) of the instantancous loss ff fc), 36) (where f 0e) denotes the model output and 0g is the model parameters step t along the training history, which is defined and computed recursively
( x ((f((z 0,), vi) (1 7) x (1) (1(1))
if E S,
(1)
else.
Where 7 E [0, 1] is discounting factor, V is the set of all n training samples, and S, S V is the set of samples selected (by a certain curriculum) for training at epoch L. We only update the EMA loss for selected samples using the byproduct (i) of training without requiring extra inference.
we show how the losses and EMA losses associated with clean/noisy data change throughout the training process. Specifically, we train ResNet34 (He) al. 2016) model on CIFARTÓ with 60% of the original labels randomly changed to wrong class. To avoid quick overfitting to the noise, we train the model for multiple episodes (each composed of several epochs over all data with a cosine annealing learning rate) and alternate between the supervised learning episode that minimizes the cross-entropy loss against the given noisy labels and the self-supervision episode that minimizes the consistency loss Eq. against the pseudo labels. Comparing the shaded areas (std) of instantaneous loss and EMA loss and the gap between curves in the two plots, we see that EMA leads to smaller variance within each group and larger gap between the clean and noisy groups, demonstrating the effectiveness of EMA loss for clean data detection.
In the first and the third plots of Figure [
2.2 CONSISTENCY DYNAMICS AND PSEUDO LABEL SELECTION
Simply removing noisy data (z. y) with wrong label y discards important information about the data distribution p(x) (Arazo et al. 2019). Including all the noisy data for self-supervision, bootstrapping or relabeling can also harm the training since the pseudo labels' quality not equal across samples and much depends on the model generating them, where the model's 's predictions may contain errors that can accumulate if adopted for training. Hence, a careful selection of noisy data is necessary. However, without access to purely clean dataset or a reliable pre-trained model, it is nontrivial to evaluate pseudo labels' correctness. By analyzing the training dynamics of model outputs in the above experiment, we discover that the model output for sample tends to be an accurate pseudo label if the output remains consistent over training steps and across different augmentations of the sample.
We first define the instantancous consistency loss of sample at step t as the discrepancy of the model output fz,; 6) between step t and t 1 on 2; and its m data augmentations (,,"))p² y.m where the discrepancy can be measured by any loss function E(.,)). However, the discrepancy can be small the models of epoch t. and t are too similar and make the same errors. Therefore, we use an exponential moving average of the model parameters (according to mean teacher (Tarvainen & Valpola 2017) and compute the prediction at step t- 1 by averaging over multiple data augmentations
(according MixMatch (Berthelot et al. 2019)): ): (z.) 1/m f f(x,"); B.), 0, _ 70,-1
(1 - (L1-)88, -)),,-1 Computing pseudo labels on augmented data and a time averaging ensemble of models is commonly-adopted for semi-supervised learning (Sajjadi al. [2016] Laine & Aila 2016) /Zhou et al. 2020a). The instantaneous consistency loss (e(z)) of sample x; at step I is then defined as
(1)
((((g))).. j=1
(2)
Which can also be minimized as consistency loss for v; in self-supervised learning when no label is given. Similar to the EMA loss in Eq. ( we define the EMA consistency loss over training history
(1) = 7 x (e(1) + (1-7) x (i) if Se
(3)
(1)
else.
Note we use the same y value as in Eq. (I. The EMA consistency loss (i) measures both the time-consistency (Zhou et al. 2020a) over multiple training steps and the spatial consistency over different augmentations of sample zy. If the output prediction is wrong and contradicts other samples'
'Fach step does not strictly refer to a mini-batch: when the subset Se for step t is larger than the mini-batch size (which is common in practice). we run SGD steps over all mini-batches of the subset for one pass. 'This name maybe misleading since it actually measures the "inconsistency". but is commonly called "consistency" in previous works (Zhu et al. 2017 Berthelot et al. 2019). so we decide to use the same name.

---

Published as a conference paper at ICLR 2021
labels, it will be inconsistent over time and across augmentations since it can easily change or flip after the next training step. In the last plot of Figure I we report the mean and standard deviation (the middle line and the shaded area) of the EMA consistency loss for two groups of data at each epoch, i..., the ones with correct pseudo labels and the ones with incorrect pseudo labels. Comparing to the instantaneous consistency loss in the second plot, EMA consistency loss is more reliable criterion for allocating correct pseudo labels. Thus, we can safely learn the noisy data by using their pseudo labels as training targets and avoid introducing harmful noises.
ROBUST CURRICULUM LEARNING
In this section, we first introduce the selection criterion for both the clean label detection and pseudo label selection. By adjusting two temperature parameters T1 and T2, it can smoothly interpolate between the two criteria and control their trade-off. We then show that the criterion is derived from a novel optimization formulation for robust curriculum learning. We finally present the RoCL algorithm.
3.1 DATA SELECTION CRITERION
To combine of clean label detection and pseudo label selection, we use the two criteria from the previous sections and apply temperature parameter to control the preference for small/large loss or consistency loss and their trade-off. We sample 2; at training step t with probability:
Pi(i) À (i) + (1 A) (:),,
(4)
where pt( and go (i) are defined as softmax probabilities, i.e.,
exp[n1((1)]
exp/T2c((1)
(i) A
(o(()) Σj= exp[T2c((i)]
(5)
Σラ exp[T14(0)]
and À E [0, 1] controls trade-off between them. Here, (i) is a softmax probability computed from EMA losses for clean data detection: samples with smaller (larger) EMA losses and thus clean (noisy) labels tend to have high probability (i) when Ti is negative (positive). Similarly, q:(i) is computed from EMA consistency loss for pseudo label selection: samples with smaller (larger) EMA consistency loss and thus correct (wrong) pscudo labels tend to have high probability q:(1) when 72 is negative (positive). When TI, 72 = O, the probabilities are uniform, and when TI, T2 + +oo/ -00, the probabilities approximate the max (min) operator.
Fither (i) or (i) can be independently employed to select or reweigh samples for noise-label learning. However, selecting samples with high probabilities (i) when < 0 tends to make the training focus on clean data that the model has already learned, which carries limited new information and prevents exploration. Similarly, selecting samples with high probabilities q:(i) when 72 & 0 is not informative since the model outputs are consistently correct for those data, and little progress can be made. We can encourage exploration by manipulating the temperature parameters. By setting TI and 72 close to zero, we move towards uniform exploration of all data, A more effective strategy is to couple the values of Ti and 72. For example, negative Ti and positive 72 strengthen the preference for clean data that have not been fully exploited and learned by the model. Alternatively, positive TI with negative 72 emphasizes the noisy data with correct pseudo labels, so relabeling them provides new information in addition to the clean data.
We apply the data selection criterion of Eq. (4) in each step and gradually change its parameters (T1, 72, A) over the course of training according to the properties discussed above. We start from negative T1 associated with positive 72 and a large X (i) dominates), then gradually increase Ti while decrease T2 and X, and end with a positive T1, a negative T2 and a small In this way, we get a curriculum with smooth transition between supervised learning of clean data using correct given labels and self-supervision of noisy data using reliable pseudo labels (i.e., minimizing Eq, (2). Moreover, the coupling strategy on Ti and encourages selecting informative samples that the model mostly needs to improve on, i.e., clean data with inconsistent model outputs or noisy data with correct pseudo labels. In our experiments, we can further reduce the hyperparameters: (1) given the sequence for 71 in the curriculum as TI.T, we can reverse as the sequence for 72, i.e., T7:1: (2) we can make À monotone increase with TI, e.g., setting the initial value λ aT + b and ending value Xx = aTT + b, solving this linear system for and b, which generate >1.r = 0T1. + b.
3.2 ROBUST CURRICULUM LEARNING AS AN OPTIMIZATION
The data selection criterion is derived from an optimization formulation of our robust curricu- lum learning (RoCL.), in which we aim to minmmize combination of supervised loss and
5

---

Published as a conference paper at ICLR 2021
consiseencylsel--ssueeviiee loss in the following form.
À
Σ
1
1 n Σ expl (x
min F(0) A
log
(xp(n +
log
(6)
TI
T2
1=1
where the consistency loss ((t) is defined in Eq. 2 (with t removed). For the first term of Eq. ( unlike the conventional empirical risk minimization (ERM) with arithmetic average loss, I.c.,
f/n En {(f(z,; 0), yt), we use LogSumExp loss with an additional temperature parameter (i.e.. T log ( En= exp(T]((fx: f), y:)l)), ), so it can approximate both the min-loss (when 71 + -00), max-loss (when TI + +00), and any interpolation between them, where the min-loss focuses on the easiest samples with the smallest losses and the max-loss focuses on the hardest ones. Note it reduces the arithmetic average loss when T - O. It is called "tilted loss" in a recent work (Li et al. 2020). which shows several intriguing properties in different learning settings. The second term of Eq. (6) focuses on the consistency loss (()). We use control the trade-off between the supervised loss and the consistency loss. By simple algebra, the gradient of F F()) at step is
11
n
VoF(O,) = > V(( 0,) v1 + +(1 A) Σ Voet()
(pp(()
(o 0.).0:) + (1 - 1)(((1)
Σ
P(())
VoGi(t) EE-P)(()G
Ge(1), (7)
Pi(
P((()
where
Gi(1)= (i)(1) Vol(s(Z; 0.). (1 ())
(i).
(8)
(i())
P((
Here, (i) and (1(1) are similar to (i) and ze (i) defined in Eq. except that the EMA loss and
EMA consistency loss are replaced by their instantaneous counterparts respectively, i.e., pi(1) & explr (( ((f(, (.9e))
(i) 2 expl72ς+67] Similarly, we also denote (t(())
exp(raCi(()
pf(()
E expltié (f (ar,
(1 x (i). Note in Section 2 we already discussed that the EMA metrics are better alternatives
to the instantaneous metries when used for data selection in noisy-label learning. In our experiments, we use the EMA metrics (i), ge (i), (.) instead of the instantancous ones (p)(1), (i()).
For every training step, an unbiased estimator of the gradient in Eq. 7 can be achieved by drawing subset of samples S₁ according to Pt(1) and averaging their gradients (i) in Eq. (8)
3.3 ROBUST CURRICULUM LEARNING ALGORITIM
We describe our curriculum learning algorithm RoCL in Alg. I based on the data selection criterion in Sec- tion 3. and the optimization formu- lation in Section 3.2 We denote the update of 0, produced by the adopted optimizer as ((),), 7),, where contains all hyperparameters of the op- timizer at step t, e.g., the learning rate.
Algorithm Robust Curriculum Learning (RoCL)
I: input: { (i),, 3:) h(::)), ((,,)) :), f(; 0), TokK <
0. Fr > 0; \1, Ay E [0, 7, [0,
2: initialize: 8,, bu E (0, (u(1) co(i) = - E [n]
3: for k E (0,,.....
do
4: Schedule TI.T, and A by Eq. -0)
5:
for t' E (1.-.. TTke do t t t' + T
6:
We denote (t)
shorthand nota-
7:
if k then
tion for fff(2;; 8 1) 81). We apply a warm starting episode of few epochs (c.g.. 5-10) over all the data and given labels with label smoothing obtain stable EMA metrics. After that, we apply multiple episodes of curriculum learning, each including a sequence of steps following the curriculum at the end of Section 3.J for data selection per step (Line 6- 14). We repeat the transition between clean data learning
8:
S. + [n]:
9:
0, + 0, + ha (V ET 4(0:n):
10:
else
11:
Draw a subset S. C [n] of by samples according
to probability P. in Eq. (4) with 7i Tr, 72
Th--'' x Apr
12:
0, + 0 + h Eies, G((():)) (Eq. (8 (8);
13:
end if
14:
Update 4++ (1) and c (f) by Eq. and Eq.
15:
end for
16: by+1 <- (1 + +7) ) x byi
noisy data learning for K episodes
to avoid getting trapped in a local
17:
end for
minimum dominated by small set of clean/noisy data specific type of loss. It also reinforces
6

---

Published as a conference paper at ICLR 2021
the memorization of clean labels and correct pseudo labels learned in previous episodes. Moreover, under the coupling strategy of T1 and T2, each episode is encouraged to explore the clean/noisy data that the previous episode fails to learn. Considering the undertrained model (producing inaccurate pscudo labels) and the relatively high variance of the EMA metrics at the earlier episodes, we start from small budget for the selected subset size and gradually increase in later episodes.
To generate the whole schedule of TI:T in each episode, we can apply any monotone interpolation between 7i and Tr whose values are predefined. Let g R + [-a, a be invertible monotone continuous function. We define the interpolation between Vi and TT as follows, Vi E [T],
Tt TT Tm x g((o)) (00) + g(o) 0
(o), Tm = TI + TT
+im or - (-
2
2
Note the g( produces interpolation values between |-o,, o that correspond to T evenly spaced In our curriculum for each episode, we need to keep high quality of
input o E 1g 1 (-o), ((r)) (or)]. the selected clean (pseudo) labels in earlier (later) stages and make the exploration stages in between shorter since their selected labels contain more noise. Therefore, we choose "s"-shaped functions such as tanh or the logistic function for the interpolation. In this paper, we use g(-) = tanh()) and pick a = 0.95. We illustrate Eq. ( and visualize our choice of g(+) and the resulting 7. in Figure (Appendix). The corresponding schedule for À can then be defined as an affine transformation of TTT
AT A1
11 E (T], == =x( - 11) XI,
(10)
TT TI
4 EXPERIMENTS
We evaluate RoCL with other approaches for noisy-
Table 1: Accuracy (%) evaluated on WehVision and ILSVRC2012 validation sets for DNNs trained by noisy-label learning methods on mini-Web Web training set (first 50 classes). which contains real-world web-label noises.
label learning on three widely used benchmarks, i.e.,
CIFAR10/100 with two types of synthetic noises (i.e..
symmetric and asymmetrie), and mini-Web Vision (Li
itt al. 2017a) (the first 50 classes) containing unknown noises from web labels. Symmetric noise flips each label randomly to an incorrect class with probabil- ity p (i.e., noise rate), and our experiments cover p {0.4, 0.6, 0.8}. Asymmetric noise flips the labels within a specific set of classes. For CIFARIO, flipping TRUCK-AUTOMOBIE DEER-HORSE, DDO. In CIFAR100, the 100 classes are grouped into 20 clpereclasees with each has 5 sub-classes, we then flip each class within the same super-class to the next in a circular fashion with prob- ability p. Our experiments cover p = {0.2, 0.3, 0.4).
Val. set
WebVision
ILSVRC2012
Accuracy
Top-1
Top-5|
Top-1
Top-5
F-correct
61.12
82.68
57.36
82.36
Decoupling
62.54
84.74
58.26
82.26
MIRD-AIRPLANNE
Co-teaching
63.58
85.20
61.48
84.70
MentorNet
63.00
81.40
57.80
79.92
MentorMix *t+
76.00
90.20
72.90
91.10
D2L
62.68
84.00
57.80
81.36
INCV
65.24
85.34
61.60
84.98
RoCL (ours) 1**:
80.04
92.
81
92.28
Practical Modifications In the experiments, we follow previous work and apply the techniques below. We will present an ablation study of their effectiveness in Table 5
We apply the dlass-balance regularization used in (Tanaka et al. 2018), which prevents the model from predicting the same class for all the samples within a mini-batch B. We add (1/B) Exe {([(x;; 1/c 1) (where C' is the number of classes) to the objective for a mini- batch B with regularization weight of 1.
We apply label smoothing whose effectiveness in noisy-label learning has been studied in (Lukasik et al. 2020). We modify each one-hot label yı to be ÿJ < (1 ax)ya + a/c (e.g., we use a = 0.5).
We apply Mix-Up (Zhang et al., 2018) to all the selected data. However, Mix-Up of two (soft) pseudo labels can significantly increase the entropy of the mixed label if both pseudo labels are under-confident Hence, we apply a curriculum to the beta distribution 's parameter a of Mix-Up and gradually reduce it (e.g., from 8.0 to 0.2 in our experiments) within each episode.
Hyperparameeer Setting We apply RoCL to train ResNet34 CIFAR10/100 and ResNet50 on Web Vision, which are the most widely used models in other baseline papers. We apply SGD with momentum of 0.9, weight decay of 10-4 and cosine annealing learning rate in each training episode. The initial learning rate is set to 0.1 for CIFAR 10/100 and 1.0 for Visiin. In all RoCL experi- ments, apply To 10 warm starting epochs followed by K = 10 episodes of curriculum learning,
7

---

Published as a conference paper at ICLR 2021
whose lengths start from Ti = 10 and increase by 10 for every episode afterwards. We initialize the subset size by = 0.2n and set 7 = T% = 0.1, which are common choices for discoumting/ugmmetiii factors. We use Cubuk et al. (2200) for data augmentations, We did not heavily tune λ1. Xr and T1, Ty and followed a principle that the resulting curriculum should have a transition from supervised learning clean data to seeff--ueeriieed learning noisy data with correct pseudo labels.
For A, we start from Å₁ close to 1 and end with AT close because our curriculum a transition from supervised learning (A = 1) to self-supervised learning (A 0).
As explained in Section 3.1 our curriculum requires Ti for (i( changing from negative to positive values and an inverse sequence for 72 in qe (i) to gain the above transition and encourage learning on more informative samples. Hence, we set the starting value Tι to be negative and 7r to be positive for the sequence T1:7-
We set their exact values based on observations in Figure [ clean data detection is casier but the detection of correct pscudo-labels is harder. So we can be more confident on the former than the latter and set the starting Tı larger than 7y in magnitude. For the same reason, we set the starting value W to be closer to I than the ending value Ar to O. In experiments, we tried since this choice performs consistently
{-4, -3)
and Tr = {1, 2} and finally chose 71 = -4 and Ty well on all experiments, though it might not be the best choice for all; set λ 0.9 and did not try other values; we tried xy = {0.1, 0.2, 0.3} and on some experiments the first two choices lead to slightly worse performance,
chose Är 0.3.
We compare RoCL the following baselines: F-
Table 2:
Test accuracy (%) of RoCL applied with different loss functions on CIFARIO corrupted by (60%, 80%} } sym- metric(uniform) noises (CE-cross entropy).
correct (Patrini et al.
[017)), Decoupling (Malach &
Shalev-Shwartz (217)) Co-tcaching (Han al.) (2018).
D2L (Ma et al., 2018). INCV (Chen et al. 2019). MD-
DYR-SH (Arazo et al., 2019). MentorNet (Jiang et al.
Noise Rate
60%
80%
2018). MentorMix (Jiang et al. 2020). O2U-net (Huang ct al. 2019) RoG+D2L (Lee 2019)), PENCIL (Yi Wu 2019), GCE (Zhang & Sabuncu 2018). SCE (Wang al. 2019), NFL/NCE variants (Ma et al.] 2020),
CE
90.22 + 0.24
77.47 + 0.67
GCE
89.30 + 0.68
79.84 1.12
SCE
92.06 1 0.23
74.25 1 0.86
NFL+MAE
88.73 + 0.47
85.76 0.26
and Bootstrap (Reed et al. 2014). To better compare and categorize different baseline methods, we use the following symbols to denote the techniques used: + for additional clean training data; for training additional auxiliary models; for using mixup; * for using data augmentations; for class
NFL+RCE
87.68 + 0.35
80.09 0.41
NCE+MAE
90.37 + 0.43
82.16 0.93
NCE+RCE
88.03 + 0.39
80.33 + 0.80
regularization; for label-smoothing We report the results and comparisons to baselines in three tables: real-world noise in Table. symmetric noise in Table. 3 and asymmetric noise in Table. 4 RoCL. achieves the best performance in every setting, and for most of the cases, improves upon the existing methods by large margins. The closest rival to RoCL is MentorMix, which utilizes MentorNet and Mix-Up to assign weights to each sample. We note that MentorMix requires training of an extra mentor network to generate the sample weights, while RoCL is more flexible and only makes changes to the training process without modifying the model. Table. 2 reports RoCL's performance when applied with different loss functions on
Table 3: Test accuracy (%) of noisy-label learning methods on CIFAR 10/100 corrupted by
wymmetric(unfooree label noises of different levels. All the baselines' results from the original papers the following-up works. There are two formats of these reported results: "mean variance" of 5 trials and single-trial accuracy.
Dataset
CIFARIO
CIFARI00
Noise Rate
40%
60%
80%
40%
60%
80%
MD-DYR-SIT SH **
92.3
86.1
74.1
70.1
59.5
39.5
MentorNet
91.2
74.2
60.0
68.5
61.2
35.5
MentorMix +I+
94.2
91.3
81.0
71.3
64.6
41.2
O2U-net
90.3
43.4
69.2
39.4
RoG+D2L.
87.0
78.0
64.9
40.6
PENCIL
69.12 + 0.62
57.79
3.86
fail
GCE
87.62 + 0.26
82.70
0.23
67.92
+ 0.60
62.64 + 0.33
54.04
0.56
29.60 + 0.51
SCE
85.34 + 0.07
80.07
0.02
53.81
0.27
53.69 0.07
41.47
15.00
0.04
NFL+MAE
83.81 1 0.06
76.36
1.52
58.18 1.08 0.08
46.10
24.78
NCE+RCE
86.02 1 0.09
0.50
52.71
1.90
59.48
1: 0.56
47.12
25.80 1.12
RoCL (ours) txt1
94.55 02.0222222222200....
0
85.
85.76
76.766222
74.
66.79
0.58
53.89 0.62
8

---

Published as a conference paper at ICLR 2021
Table 4: Test accuracy (%) of noisy-label learning methods on CIFARIO/100 comppted by asymmetric(class- dependent) noises of 3 levels. All the baselines' results are from the original papers or the following-up works.
Dataset
CIFARIO
CIFARI00
Noise Rate
20%
30%
40%
20%
30%
40%
PENCII.*
92.43
91.84
91.01
74.70 + 0.56
72.52 + 0.38
63.61
+ 0.23
Bootstrap
86.57 + 0.08
84.86 + 0.05
79.76
63.44 0.35
63.18 + 0.35
62.08
+ 0.22
F-correct +*
89.09 1 0.47
86.79 :
83.55
1..55:0.55 0.58
38.13 + 2.97 22.16
34,44 2.97
1.93
GCE
86.07 + 0.31
8.27
74.98
0.32
53.99 0.89
0.29
41.49
+ 0.79
SCE
83.92 + 0.07
0.27
78.20
+ 0.03
58.22
49.85
0.91
42.19
+ 0.19
NEL+MAE
86.81 1 0.32
83.91
4.33
77.16
1.16
63.10
56.19
0.61
43.51
1 0.42
NCE+RCE" *
88.56 1 0.17
85.58
1 0.44
79.59
1 0.40
62.68 0.79
57.82
0.41
46.79 0.96
RoCL (ours) 95.38 1 0.21 94.19 1 0.28
92.31
1 0.35
80.03
77.59 1 0.45
73 73.28 0.83
CIFARIO under high noise rates, i.e., 60% and 80%. We observe significant improvements over
their performance without using RoCL in Table. 3 It indicates that RoCL is compatible with any loss function and can further enhance their performance.
To analyze the effect of each component in RoCL, we conduct a thorough ablation study of 10 variants of RoCL, each removing/changing one component of the original RoCL. In Table 5 we report their test accuracies on CIFARI0/100 with noise rates of (60%, in Appendix, we report how their test accuracies change during the training to study their learning efficiency and conver- gence. Among them, "no ClassBalance" removes the class- regularization; "no RandAugment" replaces the strong data augmentation RandAugment CCbuk eet al.] (2020) with random crop and random horizontal flip: "no RandSampling" replaces the weighted sampling in Line 11 of Algorithm| by selecting the top-b; samples with the Targest (i)::: "no EMA metrics" replaces EMA loss and EMA
Table 5: Ablation study: Test accuracy (%) of RoCL variants with one part removed/changed when applied to CIFAR10/100 corrupted by symmetric(uniformt label noise.
Dataset
CIFARIO
CIFARI00
Noise Rate
60%
80%
60%
80%
In Figure 7 I0
RoCL: no MixUp
92.98
88.18
69.72
58.72
RoCL: no LabelSmooth
91.94
85.05
62.92
42.95
RoCL: no ClassBalance
93.08
74.91
62.66
43.94
RoCL: no RandAugment
86.59
72.35
64.84
44.06
RoCL: no RandSampling
92.31
85.99
64.09
57.00
RoCL: no EMA metrics
92.84
87.79
65.99
53.10
RoCI: (t) = 1/n
92.42
86.05
62.69
44.35
RoCL: q: (1) = 1/n
92.59
86.93
64.71
50.79
RoCL: (i) = (1) = 1/n
92.07
85.77
64.18
47.88
RoCLBase: curriculum
87.83
66.93
61.84
41.92
RoCL: original version
92.82
88.00
66.79
54.22
MentorMix: +RandAugment
85.45
20.68
52.70
8.02
MentorMix: -RanjAugmem MixUp
84.31
38.22 21
58.31
8.18
MentorMix: original version
91.30
81.00
64.60
41.20
consistency loss with their instantaneous counterparts; (i) = 1/n" samples the clean data using uniform probabilities; (i) = 1/n" samples the correct pseudo-labels using uniform probabilities; (P)(1) = (+) = 1/n" uses uniform probabilities for both. Note for the final three variants, we still have the curriculum of X. We keep the same hyperparameter settings the original RoCL. We give brief conclusions here and leave a detailed analysis to Appendix: (1) Except RoCL in Algorithm 2 "no RandAugment" and "no ClassBallnce"., most variants perform similarly as the original RoCL and outperform the previous SOTA achieved by MentorMix. The removed components are more important under higher noise rates. (2) RoCL removes our proposed curriculum and preserves all other techniques but shows significant degradation on accuracies, indicating that the curriculum is essential RoCL's appealing performance. (3) A strong data augmentation is critical to effective self-supervision and accurate EMA consistency loss estimation in RoCL, while a weak one may lead to error accumulation. However, applying RandAuggent in MentorMix degrades its original performance. (4) Class-balane regularization is only important under very high noise rates. (5) Removing Mix-Up can improve RoCL's performance since it damages information when mixing soft pseudo labels. (6) Compared to other variants, "no RandSampling" or "no EMA metrics" causes less degeneration
the final accuracies but can slow down the convergence and learning speed in the early stages when exploration is insufficient.
(7) Changing Pr(i), qe or both to uniform probabilities reduces the final accuracies in all cases and significantly slows down the learning process.
5 CONCLUSION
We propose a novel curriculum learning method RoCL for robust learning under label noises. RoCL features smooth transition from learning with clean data to noisy data, and from learning with supervised loss to self-supervised loss. Based on observations of training dynamics, RoCL can select samples with reliable labels/pseudo labels and most informative to training. RoCL does not require availability of extra clean data or training of extra auxiliary models. On multiple benchmarks of noisy label learning, RoCL significantly improves upon existing baselines.
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This research is based upon work supported by the National Science Foundation under Grant No. IIS- 1 1162606, the National Institutes Health under award R01GM103544, and by a Google, a Microsoft, and an Intel research award. It is also supported by the CONIX Research Center, one of six centers in JUMP, Semiconductor Research Corporation (SRC) program sponsored by DARPA. Some GPUs used to produce the experimental results are donated by NVIDIA. We would like to thank ICLR area chairs and anonymous reviewers for their efforts in reviewing this paper and their constructive comments! We also thank all the MELODI lab members for their helpful discussions and feedback.
REFERENCES
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. In Advances Neural Information Processing Systems, pp- 15013-15022, 2019.
Eric Arazo, Diego Ortego, Paul Albert, Noel E OoConoo,, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. arXiv preprint arXiv:1904.1 1123. 2019.
Devansh Arpit, Stanistaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 233-242, 2017.
Sumit Basu and Janara Christensen. Teaching classification boundaries to humans. In AAAI, pp. 109-115, 2013.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems 32 (NeurIPS), pp. 5050-5060. 2019.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. arXiv preprint arXiv: urXiv:190..05040 2019.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. arXiv preprint arXiv: 1et...122..9.... 2017.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135 150, 2018.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Mengying Hu, Hu Han, Shiguang Shan, and Xilin Chen. Weakly supervised image classification through noise regularization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11517-11525, 2019.
Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. O2u-net: A simple noisy label detection approach for deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3326-3334, 2019.
Lu Jiang, Deyu Meng, Shoou-1 Yu, Zhenzhong Lan, Shiguang Shan, and Alexander G. Hauptmann. Self-paced learning with diversity. In NeurlPS, pp. 2078-2086, 2014.
10

---

Published as a conference paper at ICLR 2021
Lu Jiang. Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G. Hauptmann. Self- Self-paced curriculum learning. AAAI, pp. 2694-2700, 2015.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels, In International Conference on Machine Learning, pp. 2304-2313, 2018.
Lu Jiang. Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. ICML, 2020.
Faisal Khan, Xiaojin (Jerry) Zhu, and Bilge Mutlu. How do humans teach: On curriculum learning and teaching dimension. NeurIPS, pp. 1449-1457, 2011.
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self- learning for latent variable models. In NeurlPS, pp. 1189-1197, 2010.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.0222., 2016.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. arXiv preprint 1rXiv:1901.11.000 2019.
Kuang- Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5447-5456, 2018.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. arXiv preprint arXiv:2007.0/1 J62, 2020.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708. 02862, 2017a.
Yuncheng Li. Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1910-1918, 2017b.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint arXiv:2003.02819 2020.
Xingjun Ma, Yisen Wang, Michael Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-drivee learning with noisy labels. arXiv preprint /88.v:/880..0262,, 2018
Xingjun Ma, Hanxun Huang, Yisen Wang. Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. arXiv preprint arXiv:2006. 13554, 2020.
Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". In Advances in Neural Information Processing Systems. pp. 960 970, 2017.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1944-1952. 2017.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint (.0iv:7701.65.40 2017.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems 28, pp. 3546-3554 2015.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint 1412 2014.
11

---

Published as a conference paper at ICLR 2021
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. arXiv preprint arXiv:1803.00000 09000, 2018.
Mehdi Sujjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep sumi-super learning. Advances in neural information processing systems, 29:1 111-11631111 2016.
Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning differentiable curriculum. In Advances in Neural Information Processing Systems, pp. 11095-11105, 2019.
Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. Baby Steps: How "Less is More" in unsu- pervised dependency parsing. In NeurlPS 2009 Workshop on Grammar Induction, Representation of Language and Language Learning, 2009.
James Steven Supancic III and Deva Ramanan. Self-paced learning for long-term tracking. In CVPR, pp. 2379-2386, 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-22888 2826, 2016.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5552-5560, 2018.
Kevin Tang, Vignesh Ramanathan, Li Fei-fei, and Daphne Koller. Shifting weights: Adapting object detectors from image to video. In NewrlPS, pp. 638 646, 2012a.
Ye Tang, Yu-Bin Yang, and Yang Gao. Self-paced dictionary learning for image classification. In MM, pp. 833-836, 2012b.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems 30 (NeurIPS), pp. 1195-1204. 2017.
Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In Advances in Neural Information Processing Systems, pp. 5596-56055 2017.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 839-847, 2017.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vision pp. 322-330, 2019.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699, 2015.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7017-7025, 2019.
Chiyuan Zhang, Samy Bengio, Moritz. Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin. and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 8778-8788, 2018.
12

---

Published as a conference paper at ICLR 2021
Tianyi Zhou and Jeff Bilmes. Minimax curriculum learning: Machine teaching with desirable difficulties and scheduled diversity. ICLR, 2018.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consistent self-supervision for semi-supervised learning. In International Conference Machine Learning (ICML), 2020a.
Tianyi Zhou, Shengjie Wang, and Jeff A. Bilmes. Curriculum learning by dynamic instance hardness. In NeurIPS, 2020b.
Tianyi Zhou, Shengjie Wang, and Jeff A. Bilmes. Curriculum learning by optimizing learning dynamics. In AISTATS, 2021.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-ing to-iaag translation using cycle-con adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
13

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 RoCL (NO CURRICULUM) IN SECTION 2 AND FIGURE 2 10
Algorithm 2 RoCL (no curriculum)
I: input: [1z, y/) y1 1- n): (). f(:0). To:ki [0,
2: initialize: 0. (a(1) co(1) = - [n].T_1=1
3: for & (0, (0..... } do
4: for (1, ,Tk do
5: (<( 4-1'TTT-11
6:
S. < [n];
7:
if 19 then
8:
0: - - + h Vo
= n))::)): (supervised learning using given labels)
9:
Update 442 by Eq.
{update EMA loss)
10:
else
11:
- 8 + h (Vo n
= (e()): n): (self-supervised learning using pseudo labels)
Update c-1 (1) by Eq.
(update EMA consistency loss)
13: end if
end for
15:
end for
Note the EMA metrics in line and line 12 are not used training in RoCL Bose. They have been updated and recorded for the purpose of empirical study presented in Section 2
A.2 ADDITIONAL. EXPERIMENTS
symmetric
90
accuracy
50
RcCL instant model
40
ReCL EMA model
30-00
Baseline (no currculum) instant model
Baseline curricluum EMA model
20
10000
20000
30000
40000
50000
60000
70000
80000
training batches (batch 128)
Figure 2: RoCL (Algorithm) 1 vs. RoCL without any curriculum (Algorithm) 2 in Appendix) during the training of ResNet34 CIFAR1O containing 60% symmetric noises labels.
CIFARD 80% symmetric naise
80
socuracy 186)
70
60
50
40
RoCL instant model
RoCL EMA model
Baseline (no currcclumm instant model
Baseline (no currculum) EMA model
10000
20000
30000 number of training batches (batch size 128)
40000
50000
60000
70000
80000
Figure 3: RoCL (Algorithm) I vs. RoCL without any curriculum (Algorithm) 2 Appendix) during the training of ResNet34 on CIFAR1O containing 80% symmetric noises on labels.
14

---

Published as a conference paper at ICLR 2021
CIFAR100 60% symmetric noise
50
test accurary
30
RoCL EMA model
20
Baseline (no currculuum) instant model
10
curriclumm
number training batches (batch size 128)
Figure 4: RoCL (Algorithm) I vs. RoCLBase without any curriculum (Algorithm) Appendix) during training ResNet34 on CIFAR100 containing 60% symmetric noises on labels.
CIFARI0O 80% symmetric noise
40
accuracy
30
20
ReCL irstant model
RoCL EMA model
10
currccluum model
Baseline (no curriculum) EMA model
10000
20000
50000 number training batches (batch 128)
30000
40000
60000
70000
80000
Figure 5: RoCL (Algorithm) 1 vs. RoCI Base without any curriculum (Algorithm) 2 in Appendix) during the training of R4sNet34 CIFAR100 containing 80% symmetric noises labels.
Table Extended version Bahle two more haselines: NFL+RCE NCE+MAE.
Dataset
CIFARIO
CIFARI00
Noise Rate
40%
60%
80%
40%
60%
80%
MD-DYR-SH
92.3
86.1
74.1
70.1
59.5
39.5
MentorNet
91.2
74.2
60.0
68.5
61.2
35.5
MentorMix
94.2
91.3
81.0
71.3
64.6
41.2
O2U-net
90.3
43.4
69.2
39.4
RoG+D2L
87.0
78.0
64.9
40.6
PENCIL
0.62
57.79
3.86
GCE
87.62 + 0.26
82.70 + 0.23
67.92 0.60
62.64 + 0.33
54.04 0.56
29.60
0.51
SCE
85.34 + 0.07
80.07 + 0.02
53.69 + 0.07
41.47 0.04
15.00 + 0.04
NFL+MAE
83.81 + 0.06
76.36 1 0.31
45.23 0.52
58.18 1 0.08
46.10 0.50
24.78 0.82
NFL+RCE
86.05 = 0.12
79.78
55.06 1.08
58.20
46.30
25.16 0.55
NCE+MAE
84.19 0.43
77.61
49.62
59.22 0.36
48.06
25.50 0.76
NCE+RCE
86.02 - 0.09
79.78 + 0.50
52.71 1.90
59.48
0.56
47.12
25.80
1.12
RoCL (ours) s+1t 94.55 1 0.12
92.06 1 0.23
75.76 1 0.26
74.64 1 0.43
66.79 0.58
53.89 1 0.62
Table Extended version of Table with two more baselines: NFL+RCE and NCE+MAE.
Dataset
CIFAR10
CIFAR100
Noise Rate
20%
30%
40%
20%
30%
40%
PENCIL
92.43
91.84
91.01
74.70 + 0.56
72.52
0.38
63.61
0.23
Bootstrap
86.57 1 0.08
84.86 1 0.05
79.76
0.07
63.44 0.35
63.18 1.18
62.08
1.222
F-correct
89.09 1 0.47
86.79
0.36
13.55-1..58
42.46 2.16
38.13
1.94
GCE
86.07 = 0.31
0.21
74.98
$9.99+0.58
41.49 5..99
SCE
83.92 + 0.07
7..70
7.020
58.22
49.85
42.19 0.19
NFLAMAE
86.81 + 0.32
83.91
7..16
0.22
0.61
43.51
0.12
NEL+RCE
88.73 0.29 0.29
0.22
79.27
0.41
0.332
42.97 = 1.03
NCE+MAE
86.44
83.98
78.23
0.42
0.60
58.02
47.22 7.32
NCE+RCE
88.56 0.17
85.58
79.59
0.40
02.68+0.77
0..42
46.79 + 0.96
RoCL (ours)
95.38 1 0.21
94.19
92.31 + 0.35
80.03 0.34
77.59 0.45
73.28 0.83
15

---

Published as a conference paper at ICLR 2021
1.00
G: 1,
T peints between (T,, trl
0.50
0.25
gix) (g1x El-o,
between [g [ol]
o(x) 0.95
-0.25
o(x) -== - 0.95
-2
0.50
-3
-1.00
-4
-2.0
-1.
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
0 10 20 30
40
50
Figure 6: Illustration of Eq. 99 and visualization our choice for 9(-) and resulted n. when T = 50. use g(.) = t-nh(-) (which can be other "S" "S"pp functions) and σ 0.95 in our experiments, Here, we map the points on the black curve in the left plot to the points on the red curve in the right plot. Each gray point on the bottom of the left plot is from the T' evenly spaced x-coordinates between the x-interval [g ((-ar. (a)).
We scale them the I' t-coordinates in the bottom of the right plot (i.e., t ,, 50), which associates with I' 7: values represented by the red points between [Ti, TT] on the red curve.
CIFAR10 60% symmetric naise
85
test accuracy (9))
RoCL: original
80
RoCL:
RoCL: no label smoothing
RoCL: balancing regularization
75
RoCL: RandAugment
RoCL: no random sampling ng (using tel-kselection)
70
RoCL: EMA metrics (using Instantaneous metrics)
RoCL: clean
detection p:()) 1/n
RoCL: ne nolsy label correction qx10 1/n
RoCL: uniform curriculum pr00 qull) 1/n
RoClasse cunfculum
10000
20000
30000
40000
50000
60000
70000
number of training batches (batch size 128)
Figure Ablation study: RoCL. vs. its variants during the training of ResNet34 on CIFAR10 containing 60% % symmetric noises on labels.
symmetric naise
80
70
test accuracy (%)
RoCL: original
60
MixUp
RoCL: label smoothing
RoCL: ne class balanding regularization
RandAugment
RoCL: no random sampling (using top-k selection)
40
RoCL: ne EMA metrics (using instantaneous metrics)
detection p.0))
ReCL: label correction a:0 1/n
30
RoCL: unifarm curriculum pil q//) 1/n
RoC no curricluu
20
10000
20000
30000
40000
50000
60000
70000
number of training batches (batch size 128)
Figure 8: Ablation study: RoCL vs. its variants during the training of ResNet34 on CIFAR10 containing
80 % symmetric noises on labels.
16

---

Published as a conference paper at ICLR 2021
CIFAR100 60% symmetric noise
60
50
RoCL: original
RoCL: no MixUp
test accuracy
40
RoCL: label smoothing
RoCL: no class balancing reqularization
RoCL: no RandAugment
30
RoCL: random sampling (using top-k selectinn
ReCL: ne EMA metrics (using instantaneous metrics)
20
RoCL: no cleean data detection p:()] = 1in
RoCL: noisy
correction a:!0
10
RoCL: uniform curriculum pilo q,/) 1/n
RoCilasss curriculum
10000
20000
30000
40000
50000
60000
70000
number of training batches (batch size 128)
Figure 9: Ablation study: RoCL vs. its variants during the training of ResNet34 on CIFAR100 containing
60% % symmetric noises on labels.
CIFARI 80 80% symmetric noise
60
50
40
RoCL: original
test accuracy
RoCL: ne MixUp
label smoothing
30
RoCL: no class balancing reguarizztion
RoCL: no RandAugrient
ROCL: random sampling (using top-k selection)
20
ReCL: ne EMA metrics (using instantaneous metrics)
RoCL: no clean data detection p:()) = 1/n
RoCL: neisy
correction ad1
RoCL: unifarm curriculum pito q,/) 1/n
RoCiause curricuuum
10000
20000
30000
40000
50000
60000
70000
number of training batches (batch size 128)
Figure 10: Ablation study: RoCL vs. its variants during the training of ResNet34 on CIFAR100 containing 80% symmetric noises labels.
We present a more detailed analysis of the ablation study results with explanations of the observed phenomenons below.
Most variants (except RoCL no RandAugment, and no ClassBalance) have similar perfor- mance the original RoCL and perform better than competitive with the SoTA results achieved by MentorMix. The differences compared to original RoCL become smaller under the lower noise rate setting (60%). RoCL-Base uses all data for training in each step without applying any curriculum, showing that our proposed curriculum is the most critical component of RoCL in achieving the appealing improvements. Note RoCI-Base already outperforms most methods in T which verifies the effectiveness of multi-episode training that alternates between supervised learning with the given labels and self- with the pseudo labels.
Removing RandAugment degrades the performance, especially when the noise rate is very high (e.g., 80%) because strong data augmentations required by the self-supervision and the EMA consistency loss in RoCL, while trivial data augmentations can result in error accumulation or over-confidence in pseudo labels and inaccurate EMA consistency loss. The self-supervision aims to encourage the model output consistency over different augmentations of the same sample. Without augmentations with sufficient variations, self-supervision reduces to reinforeing the same outputs on similar samples and thus carries little information and can even magnify/accummllte errors (if any) in the original outputs. Also, the EMA consistency loss cannot generate meaningful consistency measures if computed on the same data or its trivial augmentations. Note a strong data augmentation
always beneficial in all noisy label learning methods since it can increase the uncertainty in the presence of wrong labels, making the detection of clean data and noise correction more challenging. For example, tried applying RandAugment to MentorMix (using
17

---

Published as a conference paper at ICLR 2021
the official implementations of both) but observed inferior performance compared to the results using its original data augmentations.
Class balance regularization is useful for the very high noise rate setting (80%), in which wrong label may dominate the learning on a mini-batch by a large chance. However, when the noise rate is not that high (e.g., 60% on CIFARIO) 10), removing it results in better performance.
Although Mix-Up has been proved effective in previous methods, and for this reason, we followed MentorMix by starting with a relatively strong Mix-Up (alpha = 8.0) and then gradually reducing it to a 0.2. the ablation study, we find that completely removing Mix-Up significantly improves performance. Mix-Up is helpful when applied to mix clean label with noisy label since the latter can be mediated with the former and thus softened. However, this is rarely the case for RoCL since RoCL either mainly learns from clean data wrongly- pseudo labels, and the transition between the two phases is short. When applied to two correct labels/pseudo labels, Mix-Up weakens each label's confidence, and may lose information from the inter-class probabilities in the soft pscudo labels.
data with correct
Replacing weighted sampling with top-k selection ("no RandSampling") or replacing EMA metrics with instantaneous metrics ("no EMA metrics") causes less degeneration on the final test accuracies. However, they are important to the carly-stage exploration and accurate estimation of EMA metrics on less-visited samples. Figure 7 10 these two variants usually suffer from low accuracy and convergence speed during early stages. The only exception is "no RandSampling" in Figure 10 which performs better than the original RoCL. A possible reason is that the randomness brought by high uniform label noises already bring sufficient randomness for exploration.
Replacing (t)... (i) or both with uniform probabilities over all samples reduces the final test CIFAR with 80% noise. In accuracies in all cases, c.g., the degradation is significant
Figure 7 10 we can see that by setting 4: (i) = 1 1/n results in less degradation than the other two. This is due to the more accurate pseudo labels generated for more data (even the ones with larger EMA consistency loss) as training proceeds. Moreover, since we are conservative in setting Ay and TT, the performance is not very sensitive to wrong pseudo labels.
18


Reviews:
[{"review_id": "yFlMyxBLG9w", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThe paper proposes a joint pseudo-labelling and curriculum learning strategy. It addresses the problem of training with noisy labels, especially robustly correcting noisy labels.\n\n# Recommendation\nBorderline paper, with a lot of strong points, as given below. My major concern, is that the improvement is e.g. due to more advanced data augmentation (Cubuk et al. (2020)). I think it should be easy for the authors to provide more details here, i.e. an ablation study, that could help in deciding. So far I am unfortunately not conviced, that the progress is due to the proposed method alone.\n\n# Strong/Weak points\n## Pros\nCoupling the acquisition of pseudo-labels and selection of clean labels and especially a smooth curriculum using both is an interesting idea.\nThe proposed usage of the exponential moving average is reasonably motivated by the oscillating patterns of the instantaneous loss values.\nThe proposed algorithm is well embedded in recent publications and well motivated.\nMultiple noise rates and multiple datasets are considered, proving the methods applicability.\n\n## Cons\nThe origin of Equation 6 is not as clear as it should be. The description of the terms in eq. 6 could be improved, especially the fact, of \"abuse notation\".\nWhile the coupling of $p_t(i)$ and $q_t(i)$ seems to be a good idea, the theoretical justification is not as convincing to me. Further theoretical background or experimental verification would be good to support the claim of this coupling being meaningful.\nThis and the fact, that a ablation study is missing makes it hard to judge the methods contribution.\n\n# Questions to the authors\nA more detailed comparison e.g. experimental between using $p_t(i)$ and $q_t(i)$ separatly, vs. using the jointly would be very interesting.\n\nWhat data augmentation was used for all other methods, except RoCL? If all other results are taken from other papers, the aim should be to excatly reproduce the setting used there, or at least prove via an ablation study, the impact of each part of your training.\n\nWhat happens if you do not use data augmentation at all?\n\nCould you provide more details on $\\tau_{1,2}$ and $\\lambda$, so far do not see a justification for their respective values. Simple hyperparameter search? Is there a theoretical interpretation, or limits you could derive?\n\n# Detailed comments\nEq. 2 is referenced before it is stated, consider rearranging.\n\nEq. 6 please clarify the description, e.g. what is \"abuse notation\", why are you replacing by instantaneous counterparts here?\n\nMinor comment, the text in Figure 1 could be larger\n\n\nMinor typo \"Simply removing noisy data from training discards important information about data distribution.\" --> \"the data distribution\", some more typos in that same paragraph, please take some time to correct them.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pwq898rG6G1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a curriculum learning method to handle noisily labeled data. The idea is to introduce a consistency measure instead of directly apply a loss function for the typical supervised learning, where the specific consistency is measure for both temporal dimension along neighboring steps and spatial dimension over different data augmentation samples for a given real sample. The consistency measure is applied for self-supervision while the loss function is applied for supervised learning. The final optimization is managed between the two components through weighting parameters, such that the training is made through a migration from a supervised learning to self-supervision by gradually adapting the weighting parameters. Evaluations are reported on Cifar10/100, WebVision, and ILSVRC2012.\n\nThe motivation and rationale of this work makes sense, and the proposed method appears to be correct, at least at the conceptual level though I did not check in detail. After the discussions and the interactions with the authors, it came to my attention that one of my comments was wrong - I looked at a different paper that led me to have the conclusion that the authors cited a wrong performance number from a competitor in the literature; it turns out that I was wrong and I apologize to the authors.\n\nThe paper at the initial submission version read very rough, with a lot of grammatical errors, typos, or misleading/incorrect statements. In the experiment section, it is stated that three datasets were used for evaluations but in Table 1 it appears there is the fourth dataset ILSVRC2012 used, which never mentioned in the text, nor is mentioned in the paper on how that dataset is used (a portion like WebVision or the full). The whole paper ended up with no conclusion or discussion. After the discussions with the authors, it became clearer that the mentioned three datasets in the text were for training and ILSVRC2012 was used for evaluation. But still it would be a lot clearer to have such a statement in the text.\n\nI had the comment that the title of the paper was misleading. The title reads: ROBUST CURRICULUM LEARNING: FROM CLEAN LAEL DETECTION TO NOISY LABEL SELF-CORRECTION. However, the proposed method, together with all the reported evaluations, focuses on learning the noise (in the labels); neither clean label detection nor noisy label correction is addressed. The authors disagreed with me but I was still not convinced by their argument. For noisy label self-correction, it may be relevant; but for clear label detection, I don’t think so.\n\nI had a comment regarding the scale of the datasets used in the evaluations. But after having read the competitors’ work such as MentorMix, I now took it back and agreed with the authors. On the other hand, I agree with the comments raised by the other reviewers on lacking the ablation studies. I appreciated the authors’ efforts to report back the ablation studies, though only in part, and the results appeared to be convincing to me.\n\nSo overall, after the discussions and the revision provided by the authors, I am convinced that the paper is above the acceptance threshold. The paper does have presentation issue, and lacks extensive ablation studies. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "l45XWARYdb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#Summary\n\nThis paper proposes a robust curriculum learning method that interpolates a regular loss and a consistency loss, aiming at a smooth transition from learning from clean data and then to noisy data with pseudo labels.\n\n#Pros\n- The paper is well-written, and the results over several benchmark datasets seem to be strong.\n- Some of the insights provided in this paper, seem rather interesting, like by transitioning from supervised learning, to self-supervised learning of noisy data, can better benefit the learning process.\n\n#Cons\n- The authors should perform an ablation study of the RoCL method. Currently the final proposed method mixes too many components, and it is hard to disentangle the true contribution of each component. For example, in section 3.4, the authors mentioned additional techniques were added, like class-balance regularization, label-smoothing, and mix-up, a further analysis is required to understand the true contribution for each individual part.\n\n- Similarly, for all the baselines used, the authors should do a better categorization of each baseline method, to ensure a fairer comparison. For example, does any of the baselines use model averaging, mix-up, label-smoothing, or data augmentations? How is RoCL without mix-up compared to baselines that don't use mix-up? \n\n- If I understand correctly, one of the key contributions is the interplay between the regular loss and the consistency loss, but the scheduling part is not super principled and seems to involve a lot of ad-hoc tuning of the balancing parameter $\\lambda$ and the temperatures. Is there a principled way to balance the two losses?\n\n- The RoCL algorithm seems to involve a lot of parameters. The averaging parameter $\\gamma$ for EMA, $\\lambda$ for the trade-off between loss and consistency, and tempature $\\tau_1, \\tau_2$ (and the additional params required for scheduling the temperatures properly). In practice, how are those parameters picked? Is there a lot of careful tuning required?\n\n- The paper seems to have combined a lot of existing techniques. The paper would be stronger if the authors can provide further analysis to better understand how/why RoCL works. E.g., how important is EMA/data augmentation/data sampling respectively?\n\n\n#Overall recommendation\n\nOverall I'm on the fence but tend to reject. I think this paper can benefit a lot from better organizing of the methods and results, with a clearer focus on its major contributions. Currently the experimental result is a mixture of multiple existing approaches and the proposed RoCL method, it is hard to know what the role of each approach is and whether the proposed RoCL method indeed improves curriculum learning. I think this paper has the potential of providing some great insights, but the current set of results are rather noisy.\n\n#Minor comments\n- Figure 1 is really small and hard to read.\n- The paper needs to be better organized. Currently it seems like the authors run out of space and rushed through the experimental results.\n- Can you clarify if the $\\gamma$ is the same for Eq. 1, Eq. 3, and for computing $\\bar{\\theta}_t$? If they are different please use different notations.\n- I don't quite get Eq. 9, why is the temperature defined in this way?\n- How important is the sampling part in Algorithm 1? It involves another parameter $b_k$ and how sensitive is RoCL to the choice of that parameter?\n- The consistency loss over augmented examples is also a commonly-adopted technique in semi-supervised learning, citations to the use of those methods in existing literature are missing, e.g., [1] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. NeuIPS 2016.\n[2] Samuli Laine and Timo Aila. Temporal ensembling for semisupervised learning. ICLR 2017.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5-6rg8tmc8Z", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article is concerned with the problem of training models under noisy data. The authors first adopt the loss and output consistency for data selection. EMA method is used for smoothing to obtain more accurate clean label detection. Meanwhile, through the introduction of temperature hyperparameters, the model gradually completes the transition from supervised learning using clean labels to self-supervised learning using noisy labels. \n\nStrength:\n1.\tThe authors use both loss function and prediction invariance for sample selection. The proposed model adjusts the sample selection strategy for different training stages to obtain a more informative training sample.\n2.\tThe authors used the EMA algorithm to smooth the sample selection metrics, resulting in better clean sample detection performance.\n3.\tThe change of hyperparameters allows the model to accomplish the change of sample selection strategy and transform the training strategy of the model from fully supervised to self-supervised.\n4.\tThe authors provide a detailed theoretical analysis and experimental demonstration of the proposed method, which achieves SOTA performance in both the CIFAR10/100 and WebVision datasets\n\nWeakness:\n1.\tAs several modifications mentioned in Section 3.4 were used, it would be better to provide some ablation experiments of these tricks to validate the model performance further. \n2.\tThe model involves many hyperparameters. Thus, the selection of the hyperparameters in the paper needs further explanation. \n3.\tA brief conclusion of the article and a summary of this paper's contributions need to be provided.\n4.     Approaches that leveraging noisy label noise label regularization and multi-label co-regularization were not reviewed or compared in this paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been thoroughly evaluated by four expert reviewers and it had received one public comment. The authors provided extensive explanations and added technical updates to the contents of their submission in response to constructive critiques from the reviewers. Even though some minor issues have not been fully resolved in the discussion between the authors and the reviewers, I consider this paper worthy of inclusion in the program of ICLR 2021 since, albeit marginally, the apparent strengths outweigh its outstanding limitations.  
Decision: Accept (Poster)