Curriculum Loss: Robust Learning and Generalization  against Label Corruption
Authors: Yueming Lyu
Ivor W. Tsang
Venue:  (2020)
Keywords: Curriculum Learning
deep learning

Abstract: Deep neural networks (DNNs) have great expressive power, which can even memorize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper studies the 0-1 loss, which has a monotonic relationship between empirical adversary (reweighted) risk (Hu et al. 2018). Although the 0-1 loss is robust to outliers, it is also difficult to optimize.   To efficiently optimize the 0-1 loss while keeping its robust properties, we propose a very simple and efficient loss, i.e. curriculum loss (CL). Our CL  is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses.  Moreover, CL can adaptively select samples for stagewise training. As a result, our loss can be deemed as a novel perspective of curriculum sample selection strategy, which bridges a connection between curriculum learning and robust learning.     Experimental results on noisy MNIST, CIFAR10 and CIFAR100 dataset validate the robustness of the proposed loss.

mem- Published as a conference paper at ICLR 2020
CURRICULUM Loss: ROBUST LEARNING AND GEN- ERALIZATION AGAINST LABEL CORRUPTION
Yueming Lyu & Ivor W. Tsang Centre for Artificial Intelligence. University of Technology Sydney yueming ng yu@ 1, com, Tvor. TsangLuts. edu.
ABSTRACT
Deep neural networks (DNNSs) have great expressive power, which orize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper stud- ies the 0-1 loss, which has a monotonic relationship with empirical adversary (reweighted) risk (Hu et al. 2018). Although the 0-1 loss has some robust proper- ties, it is difficult to optimize. To efficiently optimize the 0-1 loss while keeping its robust properties, we propose very simple and efficient loss, i.e. curriculum loss (CL). Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses. Moreover, CL can adaptively select samples for model training. As result, loss can be deemed as novel perspective curriculum sample selection strategy, which bridges a connection between cur- riculum learning and robust learning. Experimental results on benchmark datasets validate the robustness of the proposed loss.
INTRODUCTION
Noise corruption is a common phenomenon in our daily life. For instance, noisy corrupted (wrong) labels may be resulted from annotating for similar objects etu al. 2012 Yuan et al.] (019)) crawl- ing images and labels from websites (Hu al., [017]] Tanaka et al.) 2018) and creating training sets by program (Ratner et al. 2016: Khetan et al.) 2018). Learning with noisy labels is thus an promising area.
Deep neural networks (DNNs) have great expressive power (model complexity) to learn challenging tasks. However, DNNs also undertake higher risk of overfitting the data. Although many regu- larization techniques, such adding regularization terms, data augmentation, weight decay, dropout and batch normalization, have been proposed, generalization is still vitally important for deep learn- ing to fully exploit the euper-expressive power. Zhang al. (2017) show that DNNs even fully memorize samples with incorrectly corrupted labels. Such label corruption significantly degenerates the generalization performance of deep models. This calls lot of attention on robustness in deep learning with noisy labels.
Robustness of 0-1 loss: The problem resulted from data corruption label corruption is that test distribution is different from training distribution. Hu et al. (2018) analyzed the adversarial risk that the test distribution density is adversarially changed within limited /-divergence (e.g. KL- divergence) from the training distribution densíty. They show that there is a monotonic relationship between the (empirical) risk and the (empirical) adversarial risk when the 0-1 loss function is used. This suggests that minimizing the empirical risk with the 0-1 loss function is equivalent to minimize the empirical adversarial risk (worst-case risk). When we train a model based on the corrupted training distribution, we want our model to perform well on the clean distribution. Since we do not know the clean distribution, we want our model to perform well for the worst case estimate of the clean distribution in some constrained set. It is thus natural to employ the worst-case classification risk of the estimated clean distribution as the objective. Note that the worst-case classification risk is an upper bound of the classification risk of the true clean distribution, minimizing the worst-case risk can usually decrease the true risk. When we employ the 0-1 loss, because of the equivalence between the classification risk and the worst-case classification risk, directly minimize the

---

Published as a conference paper at ICLR 2020
classification risk under the coruupted training distribution instead of minimizing the worst-case classification risk.
From the learning perspective, the 0-1 loss is more robust to outliers compared with unbounded (convex) loss (c.g. hinge loss) (Masnadi-Shirazi & Vasconcelos) 2009). This is due to unbounded convex losses putting much weight on the outliers (with a large loss value) when minimizing the losses (Masnadi-Shirazii & Vasconcelos) 2009). If the unbounded (convex) loss is employed in deep network models, this becomes more prominent. Since training loss of deep networks can often be minimized to zero, outlier with a large loss has large impact on the model. On the other hand, the 0-1 loss treats each training sample equally. Thus, each sample does not have too much influence on the model. Therefore, the model is tolerant of a small number of outliers.
Although the 0-1 loss has many robust properties, its non-differentiabbiity and zero gradients make it difficult to optimize. One possible way to alleviate this problem is to seek an upper bound of the 0-1 loss that still efficient to optimize but tighter than conventional (convex) losses. Such a tighter upper bound of the 0-1 loss can reduce the influence of the noisy outliers compared with conventional (convex) losses. At the same time, it is easier to optimize compared with the 0-1 loss. When minimizing the upper bound surrogate, we expect that the 0-1 loss objective is also minimized.
Learnability under large noise rate: The 0-1 loss cannot deal with large noise rate. When the noise rate becomes large, the systematic error (due to label corruption) grows up and becomes not negligible. As a result, the model's generalization performance will degenerate due to this systematic error. To reduce the systematic error produced by training with noisy labels, several methods have been proposed. They can be categorized into three kinds: transition matrix based method (Sukhbaatar et al.] [2014] Patrini al. 2017 Goldberger & Ren-Reuven] 2017). regular- ization based method (Miyato et al. 2016) and sample selection based method (Jiang et al., 2018
Han et al. 2018b). Among them, sample selection based method is one promising direction that se- lects samples to reduce noisy ratio for training. These methods are based on the idea of curriculum learning (Bengio al.) 2009) which one successful method that trains the model gradually with samples ordered in meaningful sequence. Although they achieve success to some extents, most of these methods are heuristic based.
To efficiently minimize the 0-1 loss while keeping the robust properties, we propose a novel loss that is a tighter upper bound of the 0-1 loss compared with conventional surrogate losses. Specifically, giving any base loss function ((u) 2 1 1(u < 0). u E R, our loss Q(u) satisfies [u1,... U.,] with u, being the classifica- tion margin of 7th sample, and 1-)) an indicator function. We name it as Curriculum Loss (CL) because our loss automatically and adaptively selects samples for training, which can be deemed as a curriculum learning paradigm.
E, 1(u O) S Q(u) I ",_,,((5)) where
Our contributions are listed as follows:
We propose a novel loss (i.e. curriculum loss) for robust learning against label corruption. We prove that our CL is tighter upper bound of 0-1 loss compared with conventional summation based surrogate loss. Moreover, CL can adaptively select samples for stagewise training, which bridges connection between curriculum learning and robust learning.
We prove that CL can be performed by a simple and fast selection algorithm with On log n) time complexity. Moreover, our CL supports mini-batch update, which is con- venient to be used as a plug-in in many deep models.
We further propose a Noise Pruned Curriculum Loss (NPCL) to address label corruption problem by extending CL to a more general form. Our NPCL automatically prune the estimated noisy samples during training. Moreover, NPCL also very simple and efficient, which can be used as plug-in in deep models as well.
2 CURRICULUM LOSS
In this section, we present the framework of our proposed Curriculum Loss (CL). We begin with discussion about robustness of the 0-1 loss in Section 2.1 We then show that our CL is tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses Sec- tion 2.2 A tighter bound of the 0-1 loss means that it is less sensitive the noisy outliers, and it
2

---

Published as a conference paper at ICLR 2020
better preserves the robustness of the 0-1 loss with small rate of label corruption. For large rate of label corruption, we extend our CL to a Noise Pruned Curriculum Loss (NPCL) to address this issue in Section 2.3 A simple multi-class extension and novel soft multi-hinge loss included in the Appendix. All the detailed proofs can be found in the Appendix as well.
2.1 ROBUSTNESS OF 0-1 LOSS AGAINST LABEL CORRUPTION
We rephrase Theorem in (Hu al., 2018) from a different perspective, which motivates us to employ the 0-1 loss for training against label corruption.
Theorem 1, (Monotonic Relationship) (Hu et al. (Hu et al. 2018 Ler p(x, y) and q(x, y) be the training and test density.
Define r(x, U) q(. p(x. y) and T; r((z, Vs). Let 10. v) = 1 (s) + y) and I@d v) = (argmaar (Tk) + v) be 0-1 loss for binary classification and multi-class classification, respectively. Let f(-) be convex with f(1) = 0. Define risk R()), empirical risk R0), adversarial risk Rodo (0) and empirical adversarial risk Red as
R(0) = pl (x)g(()),
(1)
(0) -
i= (g0( vc)
(2)
Rend (0) sup Ep(e,)) [r( ())(())) rElt,
Rado (0) sup rEtt n2
rel r: (2))
(4)
Uy = (r(z, y) ((..)) [1 (r(z, y))l S 6, Bp(a [r(or, [() = r(, v) 2 0, 0,4(2,
E X x y}
and {r| +. E (r;)) 5 Ei= ri = 1,r 2 }}. Then we have that
If Rudv (01) < 1, then R(0,) < R(22) = Rudo (01) < Rudu
(5)
If Radv (01) = 1, then R(01)= - R(02) = Rade (02) =
(6)
The same monotonic relationship holds between their empirical approximation: R(()) and Rade-
Theorem (uu et al., 2018) shows that the monotonic relationship between the (empirical) risk and the (empirical) adversarial risk (worst-case risk) when 0-1 loss function is used. It means that minimizing (empirical) risk is equivalent to minimize the (empirical) adversarial risk (worst-case risk) for 0-1 loss. When we train a model based on the corrupted training distribution p(z, v), we want our model to perform well on the clean distribution q(I, y). Since we do not know the clean distribution g, we want our model to perform well for the worst-case estimate of the clean distribution, with the assumption that the f-divergence between the corrupted distribution and the clean distribution q is bounded by 5. Note that the underlying clean distribution is fixed but unknown, given the corrupted training distribution, the smallest 8 that bounds the divergence between the corrupted distribution and clean distribution measures the intrinsic difficulty of the corruption, and it is also fixed and unknown. The corresponding worst-case distribution w.r.t the smallest ó is an estimate of the true clean distribution, and this worst-case risk upper bounds the risk of the true clean distribution. In addition, this bound is tighter than the other worst-case risks w.r.t larger 5. It is natural to use this upper bound as the objective for robust learning. When we use 0-1 loss (that is commonly employed for evaluation), because of the equivalence of the risk and the worst-case risk, we can directly minimize risk under training distribution instead of directly minimizing the worst-case risk (i.e., the upper bound). Moreover, this enables us to minimize the upper bound without knowing the true beforehand. When the true 5 is small, i.e., the corruption of the training data is not heavy, the upper bound is not too pessimistic. Usually, minimizing the upper bound can decrease the true risk under clean distribution. Particularly, when the clean distribution coincides with the worst-case estimate w.r.t the smallest 5, minimizing the risk under the corrupted training distribution leads to the same minimizer minimizing the risk under the clean distribution.
2.2 TIGHTER UPPER BOUNDS OF THE 0-1 LOSS
Unlike commonly used loss functions in machine learning, the non-differentiabiiiy and zero gradi- ents of the 0-1 loss make it difficult to optimize. We thus propose tighter upper bound surrogate
3

---

Published as a conference paper at ICLR 2020
loss. We use the classification margin to define the 0-1 loss. For binary classification, classification margin is u = vy. where y and y E [+1,-1} denotes the prediction and ground truth, respectively. (A simple multi-class extension is discussed in the Appendix.) Let u, R be the classification margin of the 1th sample for E {1,-..,) n.. Denote u = [M1,-.,,,,, Un] The 0-1 loss objective can be
defined as follows:
((u) = Ex= 1( < 0).
(7)
Given a base upper bound function 1(u) 2 1 1 < 0), u E R, the conventional surrogate of the 0-1 loss can be defined as
J(u) - E/= ((u))
(8)
Our curriculum loss (u) can be defined as 1q. (u)) is tighter upper bound of 0-1 loss J()) compared with the conventional surrogate loss (u). which is summarized in Theorem 2
Theorem 2. (Tighter Bound) Suppose that base loss function ((u) 2 1(u < 0), R is an
upper bound of the 0-1 loss function. Let us I be the classification margin of the 8th sample for i E [1....n]. Denote max(",)) )) as the maximum between two inputs. Let u = [u1, un]- Define
Q (u) as follows:
(u)) min max (E(1)))"
vil(us), n2- Σx 04 + Σ.
1(m < x)).
i=
Then J(u) S Q (u) S J (u) holds true.
For any fixed u, we can obtain an optimum solution v of the partial optimization. The index indicator v can naturally select samples as a curriculum paradigm for training models. The partial optimization w.r.t index indicator v can be solved by a very simple and efficient algorithm (Algorithm I in On log Thus, the loss is very efficient to compute. Moreover, since Q (u) is tighter than conventional surrogate loss Ju), it is less sensitive to outliers compared with J()). Furthermore, it better preserves the robust property of the 0-1 loss against label corruption.
Remark:
The difficulty of optimizing the 0-1 loss is that the 0-1 loss has zero gradients in almost everywhere (except at the breaking point). This issue prevents us from using first-order methods to optimize the 0-1 loss. Eq. (9) provides surrogate of the 0-1 loss with non-zero subgradient for optimization, while preserving robust properties of the 0-1 loss. Note that our goal is to construct tight upper bound of the 0-1 loss while maintaining informative (sub)gradients. Eq.(9) balances the 0-1 loss and conventional surrogate by selecting (the trust) samples (index) for training progressively.
Updating with all the samples at once is not efficient for deep models, while training with mini-batch more efficient and well supported for many deep learning tools. We thus propose batch based curriculum loss Q(u) given as Eq.( We show that Quu) is also a tighter upper bound of 0-1 loss objective (u) compared with conventional loss (u). This property is summarized in Corollary
Corollary 1. (Mini-batch Update) Suppose that base loss function I(u) 2 1(u < O), u E R is an upper bound of the 0-1 loss function. Let b, m be the number of batches and batch size, respectively.
Let Uiy E R be the classification margin of the ith sample in batch j for i E {1,..., m) and j e (1,...b). b). Denote u
[111: .... Umi)) Let =bb. Define Q (u) as follows:
(())= 1- ve m
min max E vigl(ur)), m Σm Dig > 1uuy < <)). (10)
Then J(u) S (Q( S (u) (u) & J (u) holds true.
Remark: Corollary T shows that batch-based curriculum loss is also tighter upper bound of 0-1 loss Ju) compared with the conventional surrogate loss J(u). This enables us to train deep models with mini-batch update. Note that random shuffle in different epoch results in different batch- based curriculum loss. Nevertheless, we at least know that all the induced losses are upper bounds of 0-1 loss objective and are tighter than J(u). Moreover, all these losses are induced by the same base loss function i(-). Note that, our goal is to minimize the 0-1 loss. Random shuffle leads to a multiple surrogate training scheme. In addition, training deep models without shuffle does not have this issue.
We now present another curriculum loss E (u) which is tighter than Q(u). E (u) is an (scaled) upper bound of 0-1 loss. This property is summarized as Theorem
4

---

Published as a conference paper at ICLR 2020
Algorithm Partial Optimization
Input:
u; for E {1, ..., n., the selection threshold C: Output: Index set (V1, U2..... Un);
Compute the losses 4 = I(u,) for = 1,.., n2;
Sort samples (index) w.r.. the losses {4 J in a non-decreasing order; Initialize Ln = 0: for i to do
// /ett 1 ... S hn
I = [,-1 + 1;
if L, - (C (C++1 i) then
Set 1/ = 1;
else
Set U/ 0;
end if
end for
Theorem 3. (Scaled Bound) Suppose that base loss function I(u) 1lu < 0) u R is an upper bound of the 0-1 loss function. Let us E be the classification margin of the ith sample for i E n.. Denote [01y.-.;p)] Un Define E (u) as follows:
E (uu)= min max |__, vellu))) Em vz). vE(O,1)"
(11)
Then J(u) 2F & 2 (u) holds true.
Remark: E(u) has similar properties to Quu) discussed above. Moreover, it is tighter than Q(u), ie. E(u) Q(u). Thus, it is less sensitive to outliers compared with Quu). However, Q(u) can construct more adaptive curriculum by taking 0-1 loss into consideration during the training process.
Directly optimizing (u)) is not efficient that optimizing Quu). We now present batch loss objective E(u) given as 12). Elu) is also tighter upper bound of 0-1 loss objective (()) compared with conventional surrogate loss J(u).
Corollary 2. (Mini-batch Update for Scaled Bound) Suppose that base loss function I(u) Z 1uu < 0), E R is an upper bound of the 0-1 loss function. Let b, m be the number of batches and batch size, respectively. Let Uij E R be the classification margin of the 8th sample in batch j for i (1,...,m) m] and j {1,...,b) Denote
(u11,...; Umbb) ummbl. Let mb. Define E(u) follows:
(u)= (u)) 2= (0.11)
min max En myl(uss), m Σm1 1=1 vt)).
(12)
Then J(u) 2E < <2 (u) 2 (u) holds true.
All the curriculum losses defined above rely on minimizing partial optimization problem (Eq to find the selection index set v*. We now show that the optimization of v with given classification margin u, E R. E [l....n} can be done in On logn).
Theorem 4. (Partial Optimization) Suppose that base loss function I(u) 2 1(u < 0), E E is an upper bound of the 0-1 loss function. For fixed u, E R, E (1..... the minimization problem in Eq. 11 can be achieved by Algorithm
...., n.. an minimum solution v* of
min max
(E v,l(u,), C C u))
(13)
vê (0,1)
where C. the threshold parameter such that 0.C $022n. 2n.
Remark:
The time complexity of Algorithm Dis On Moreover, it does not involve complex operations, and is very simple and efficient to compute.
Algorithm D can adaptively select samples for training. has some useful properties to help us better understand the objective after partial minimization, present them in Proposition
5

---

Published as a conference paper at ICLR 2020
Proposition 1. (Optimum of Partial Optimization) Suppose that base loss function I(u) 2 1( < 0), E R is upper bound of the 0-1 loss function. Let E IR. for i E (1....,n be fixed values. Without loss of generality, assume ((+)) I ((2)) S ((un). Let v be an optimum
solution of the partial optimization problem in Eq. 13 Let T Ei= oz and LT Σi= 1(4)) Then we have
LT- --1-1
(14)
LT > C T
(15)
Tr > max(IsT=,CC C' C-Tm) TT))
(16)
min max Σ En njl(us), C Σn 8.) m C- vE(U,1]
(17)
Remark: When C' cn++E [u < <)), Eq.01 17) tighter than the conventional loss Au). When 2 n, Eq. (17) is a scaled upper bound of 0-1 loss J(u). From Eq. we know the optimum of the partial optimization problem 13) (i.e. our objective) is max(Lrr- C'-T"), When
LT- 2 C T*, we can directly optimize LT- with the selected samples for training. When LT- < C-T", note that LT+41 > max LTr,, C T*) from (16). we can optimize LT+ for training. Note that when T < n, we have that LT-+1 < Ln = Di=1 I(u,), which still tighter than the conventional loss J(u). When T = for the parameter C' n +n++i= I(u < 0), we have that max(Lr-, LT. = J(u)= 2 J(() 2 C - n = C T". Thus we can optimize max (LT , - T*) = Ju). In practice, when training with random mini-batch, we find that optimizing Ly. in both cases instead of 41 does not make much influence.
2.3 NOISE PRUNED CURRICULUM Loss
The curriculum loss in Eq. 9 and Eq. (1 expect to minimize the upper bound of the 0-1 loss for all the training samples. When model capability (complexity) is high, (deep network) model will still attain small (zero) training loss and overfit to the noisy samples.
The ideal model is that it correctly classifies the clean training samples and misclassifies the noisy samples with wrong labels. Suppose that the rate of noisy samples (by label corruption) is e [0, 1]. The ideal model is correctly classify the (1 <)n. clean training samples, and misclassify the noisy training samples. This is because the label corrupted. Correctly classify the training samples with corrupted (wrong) label means that the model has already overfitted to noisy samples. This will harm the generalization to the unseen data.
Considering all the above reasons, thus propose the Noise Pruned Curriculum Loss (NPCL) as
L. (u) min max E esl(uu), C- -- v:):: v0,01))
(18)
where (1 c)n (1 or C (1- 1(4 < <))
When we know there are noisy samples in the training set, (The impact of misspecification of the prior is included in the supplement.) When C = (1-e)m
leverage this our prior.
(assume C, en are integers for simplicity), from the selection procedure in Algorithm T know
en samples with largest losses ((u) will be pruned. This is because C - -" U, + < 0 when
21=1 02 2 (1 e)n + 1. Without loss of generality, assume I(u1) I ((u2) ... S I(un). After
pruning, we have U(1-c)at1
Un the pruned loss becomes
[())= min max (ELL (1-e)m vel(u)), (1-e)n-
- (1-e))
(=1 v.).
(19)
ve(0.13
It is the basic CL for (1 c)n samples and it is the upper bound of E((-e)n 1(u, < 0). If we prune more noisy samples than clean samples, it will reduce the noise ratio. Then the basic CL can handle. Fortunately, this assumption is supported by the "memorization" effect in deep networks (Arpit et al.) 2017). i.c. deep networks tend to learn clean and easy pattern first. Thus, the loss of noisy or hard
Ei=1
When (1-e)++1 I(M)) en samples will be pruned. Otherwise, en samples will he pruned.

---

Published as a conference paper at ICLR 2020
Algorithm 2 Training with Batch Noise Pruned Curriculum Loss
Input: Number of epochs N, batch size m, noise ratio E; Output: The model parameter w; Initialize model parameter w. for k = to N do
Shuffle training set D;
while Not fetch all the data from D do
Fetch a mini-batch D from D;
Compute losses [4: for data D:
Compute the selection threshold C' according Eq. 21
Compute selection index v by Algorithm |
Update a Vl (Dv-) w.r.t the subset Dv- of D selected by v";
end while end for
data tend remain high for period (before being overfitted). Therefore, the pruned samples with largest loss are more likely to be the noisy samples. After the rough pruning, the problem becomes optimizing basic CL for the remaining samples in Eq. (7)). Note that Cs tight upper bound approximation to the 0-1 loss, it preserves the robust property to some extent. Thus, it can handle case with small noise rate. Specifically, our CL(Eq (9) further select samples from the remaining samples for training adaptively according to the state of training process. This generally will further reduce the noise ratio. Thus, we may expect our NPCL be robust noisy samples. Note that, all the above can be done by the simple and efficient Algorithm without explicit pruning samples in a separated step. Namely, our loss can do all these automatically under unified objective form in (8.
When C (1 c)n, the NPCI. in Eq. 18 reduces basic CL E(u) in Eq. 11 with e = When C' -==(1ee) (1-e)nn((1e)
1 (uu < 0), for an ideal target model (that misclassifies noisy samples
only), we know that [E[]] (1 - e) e) 1 1wu < 0)] = (1 - -)) (1 -een e)en (1 - c)n. Moreover,
(1 c)n. has similar properties as choosing
more adaptive by
considering 0-1 loss during training at different stages. In this case, the NPCL in Eq. (T8) reduces to
the CL (u) in Eq. when = Note that is a prior, users can defined based on their domain knowledge.
To leverage the benefit of deep learning, we present the batched NPCL as
Í(u) = Σ
min max Σ vigl(urs), - Wis),
(20)
j=1
ve
where Õ -1= -e)m or as in Eq. 21
(1- (1- E. I(u < <0).
(21)
Similar to Corollary 1, we know that L (u) S <((). Thus, optimizing the batched NPCL is indeed minimizing the upper bound of NPCL. This enables us to train the model with mini-batch update, which very efficient for modern deep learning tools. The training procedure is summarized in Algorithm 2 It uses Algorithm I select subset of samples from every mini-batch. Then, it uses the selected samples to perform gradient update.
EMPIRICAL STUDY
3.1
EVALUATION OF ROBUSTNESS AGAINST LABEL CORRUPTION
We evaluate NPCL by comparing Generalized Cross-Entrope (GCE) loss Zhang & Sabuncu
2018). Co-teaching (Han et al. 2018b), Co-teaching+ (Yu et al. 2019). MentorNet (Jiang et al,
2018 and standard network training on MNIST, CIFAR IO and CIFAR100 dataset in (Han al.
20186 Patrini et al.) 2017 Goldberger & Ben-Reuven 2017). Two types of random label corrup-
tion, Symmetry flipping (Van Rooyen al. 2015) and Pair flipping (Han et al. 2018a),

---

Published as a conference paper at ICLR 2020
Accuracy
Tesst Accuracy
Testt Accuracy
211
(a)
(b)
(c)
TW:L2WEEEOKK. 22
Label Prectsion
Lebel Preecsson
Label Precision
Exoch
Eboch
Eboch
(d)
Pa.ttlp-35%%
(e)
Symmetry-505%
(f) Symmetry-20%
Figure 1: Test accuracy and label precision vs. number of epochs on MNIST dataset.
considered in this work. Symmetry flipping is that the corrupted label is uniformly assign to one of K-1 incorrect classes. Pair flipping is that the corrupted label is assign to specific class similar to the ground truth. The noise rate e of label flipping is chosen from {20%, 50%, 35%} as representative. As robust loss function, we further compare NPCL with GCE loss in detail with noise rate in {0%, 10%, 20%, 30%, 40%, 50%}. We employ same network architecture and network hyperparameters in Co-teaching (Han al., [2018b) for all the methods in comparison. Specif- ically, the batch size and the number of epochs is set to m 128 and N = 200, respectively. The Adam optimizer with the same parameter as (Han et al. 2018b) is employed. The architecture of neural network is presented in Appendix L For NPCL, we employ hinge loss the base upper bound function of 0-1 loss. In the first few epochs, we train model using full batch with soft hinge loss (in the supplement) as a burn-in period suggested in (Jiang et al., 2018). Specifically, we start NPCL at 5th epoch on MNIST and 10th epoch on CIFARTO and CIFARTOO, respectively. For Co- teaching (Han et al., 201 1b) and MentorNet in (Jiang et al. (2018). we employ the open sourced code of Co-teaching (Han al. 2018b). For Co-teaching+ Yuu et al. 2019), we employ the code pro- vided by the authors. We implement NPCL by Pytorch. For NPCL, Co-teaching and Co-teaching+, we employ the true noise rate as parameter. Experiments are performed five independent runs. The error bar for STD is shaded.
For performance measurements, we employ both test accuracy and label precision as in (Han et al, 2018b). Label precision is defined as number clean samples/ / number of selected samples, which measures the selection accuracy for sample selection based methods. A higher label precision the mini-batch after sample selection can lead to a update with less noisy samples, which means that model suffers less influence of noisy samples and thus preforms more robustly to label corruption.
The pictures of test accuracy and label precision vs. number of epochs MNIST are presented in Figure The results on CIFAR10 and CIFAR1OR are shown in Figure 5 and Figure in Appendix, respectively. shows that NCPL achieves superior performance compared with GCE loss in terms of test accuracy. Particularly, NPCL obtains significant better performance compared with GCE loss in hard cases: Symmetry-50%% and Pair-flip-35%% which shows that NPCL. is more robust to label corruption compared with GCE loss. Moreover, NPCL obtains better performance on MNIST, and competitive performance on CIFARIO and CIFARI00 compared with Co-teaching. Furthermore, NPCL achieves better performance than Co-teaching+ on CIFARIO and two cases MNIST. In addition, we find that Co-teaching+ is not stable on CIFAR with 50% symmetric noise. Note that NPCL is simple plug-in for single network, while Co-teade Do-teaching/Co-teaclee networks to train the model concurrently. Thus, both the space complexity and time complexity of Co-teachinn/CCoteachnngg is doubled compared with our NPCI.
employs two
8

---

Published as a conference paper at ICLR 2020
Both our NPCL and Generalized Cross Entropy (GCE) loss robust loss functions plug-in for single network. Thus, we provide a more detailed comparison between our NPCL and GCE loss with noise rate in {0%, 10%, 20%, 30%, 40%, 50% J. The experimental results CIFAR10 presented in Figure [ The experimental results on CIFAR 100 and MNIST are provided Figure and Figure in Appendix. From Figure FiguregJande Figure 7
observe that NPCL obtains similar and higher test accuracy in all the cases. Moreover, from Figure 3 and Figure 7 we can that NPCL achieves similar test accuracy compared with the GCE loss when the noise rate is small. The improvement increases with the increase of the noise rate. Particularly, NPCL obtains remarkable improvement compared with the GCE loss on CIFARIO with noise rate 50%. shows that NPCL is more robust compared with GCE loss against label corruption. GCE loss employs all samples for training, while NPCL prunes the noisy samples adaptively. As result, GCE loss still employs samples with wrong labels for training, which misleads the model. Thus, NPCL. obtains better performance when the noise rate becomes large.
3.2 MORE EXPERIMENTS WITH DIFFERENT NETWORK ARCHITECTURES
We follow the experiments setup in [ee et al. 2019). We use the online code of (Lee al. 2019) and only change the loss for comparison. We cite the numbers of Softmax, RoG and D2L (Ma et al. 2018 in (Lee et al. 2019) for comparison.
The test accuracy results on uniform noise, semantic noise and open-set noise are shown in Table I Tad Tableß] respectively. From Table Softmax (cross-entropy) and RoG (cross-entropy) on five cases for uniform noise. Note that RoG is an ensemble method, while CL/NPCL is single loss for network training, one can combine them to boost the performance. From Table 2| we can see that CL obtains consistently better performance than cross-entropy and D2L (Ma et al. 2018) for the semantic noise. Table shows that NPCL achieves competitive performance compared with RoG for open-set noise.
observe that both NPCL and CL outperforms
Table 1: Test accuracy(%) of DenseNet on CIFARIO and CIFARI00.
CIFARTO
CIFAR 100
Noise type
NPCL.
CL
Softmax
RoG
NPCI.
CL
Softmax
RoG
uniform (20%)
89.49
89.32
81.01
87.41
64.88
67.92
61.72
64.29
uniform (40%)
83,24
85.57
72.34
81.83
56,34
58.63
50.89
55.68
uniform (60%)
66.2
68.52
55.42
75.45
44.49
46.65
38.33
44.12
Table 2: Test accuracy(%)) of DenseNet on CIFARIO and CIFARI00 with semantic noise.
Dataset
Label generator (noise rate)
NPCL
CL
Cross-entropy
D2L
DenseNet(32%)
66.5
67.45
67.24
66.91
CIFARI0
ResNet(38%)
61.88
62.88
62.26
59.10
VGG(34%)
68.37
69.61
68.77
57.97
DenseNet(34%B)
57.59
55.14
50.72
5.00
CIFARI00
ResNet(37%)
54.49
53.20
50.68
23.71
VGG(37%)
55.41
52.71
51.08
40.97
Table 3: Test accuracy( (%) of DenseNet on CIFARIO with open-set noise.
Open-set Data
NPCL
Softmax
RoG
CIFAR 100
82,85
79.01
83.37
ImageNet
87.95
86.88
87.05
CIFAR PETAR10--mmae 10m-mmeeeett
84.28
81.58
84.35
further evaluate the performance CL/NPCL on the Tiny-ImageNet dataset. We use the ResNet18 network the test-bed. For GCE loss, we employ the default hyper-parameter q 0.7 in all cases. All the methods are performed five runs with seeds {1, 2, 3,3,,,55 5... The curve of mean test accuracy (shaded in std) provided in Figure 2 We can see that NPCL and CL obtain higher test accuracy than generalized cross-entrope loss and stand cross-entropy loss on both cases. Note that CL. does not have parameters, is much convenient to use.

---

Published as a conference paper at ICLR 2020
Test Accuracy
Accuracy
Generatteerrsttrrna
Grasdistrppy
Croestramppy
EDoch
EDoch
(a) Symmetrie-20%
(b) Symmetric-50P
Figure Test accuracy (%) Tiny-ImageNet dataset with symmetric noise
Accurnty
20
10
(a) Symmetry-0%
(b) Symmetry-1000
Symmetry-20%
diaro comeetre 0A
diara coweetrr as
Test Accuraty
Test Accuracy
Test Accuracy
300
150
15
150
11
Enoch
Epoch
Epoch
(d) Symmetry- 30%
(c)
Symmetry 40%
(1) Symmetry 50%
Figure 3: Test accuracy vs. number of epochs on CIFARIO dataset.
CONCLUSION AND FURTHER WORK
In this work, we proposed a curriculum loss (CL) for robust learning. Theoretically, we analyzed the properties of CL and proved that it tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses. We extended our CL to more general form (NPCL) to handle large rate of label corruption. Empirically, experimental results benchmark datasets show the robustness of the proposed loss. As a further work, we may improve our CL to handle imbalanced distribution by considering diversity for each class. Moreover, it is interesting to investigate the influence of different base loss functions in CL and NPCL.
ACKNOWLEDGEMENT
We sincerely thank the reviewers for their insightful comments and suggestions. This paper was supported by Australian Research Council grants DP1801001 and DP200101328.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, closer look at memorization in deep networks. In ICML, pp. 233-242--2222 233-242,77 2017.
Peter L Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, (1(477))138-1560 2006.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pp. 41-48. ACM, 2009.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In Advances Neural Information Processing Systems, pp. 3836,5846, 201
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, pp. 8527-8537, 2018b.
Mengqiu Hu, Yang Yang, Fumin Shen, Luming Zhang, Heng Tao Shen, and Xuelong Li. Robust web image annotation via exploring multi- factt and structural knowledge. IEEE Transactions on Image Processing, 26(10):4871 1-8844, 2017.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? 2018.
Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, and Alexander Hauptmann. Self-paced learning with diversity. In Advances in Neural Information Processing Systems, pp. 2078-2086, 2086, 2014.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced cur- riculum learning. In Twenty-Nnthh AAAI Conference on Artificial Intelligence, 2015.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. 2018.
Ashish Khetan, Zachary Lipton, and Anima Anandkumar. Learning from noisy singly-labeled data. ICLR, 2018.
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pp. 1189-1197 2010.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, 2019.
Xingjun Ma, Yisen Wang Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionallty-driee learning with noisy labels. In Interna- tional Conference on Machine Learning, 2018
Hamed Masnadi-Shinn and Nuno Vasconcelos. On the design of loss functions for classification: theory, robustness to outliers, and savageboost. In Advances in neural information processing systems, pp. 1049-1056, 2009.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Virtual adversarial training for semi-supervised text classification. In ICLR, 2016.
Robert Moore and John DeNero. LI and 12 regularization for multiclass hinge loss models. In Symposium Machine Learning in Speech and Language Processing, 2011.
11

---

Published as a conference paper at ICLR 2020
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1944-1952. 2017.
Alexander Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data
programming: Creating large training sets, quickly. In Advances in neural information processing systems. pp. 3567-3575, 2016.
Hao Su, Jia Deng, and Li Fei-Fei. Crowdsoureing annotations for visual object detection. Work- shops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv preprint arXiv: 2014.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5552-5560, 2018.
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In Advances Neural Information Processing Systems, pp. 10-18, 2015.
Yichao Wu and Yufeng Liu. Robust truncated hinge loss support vector machines. Journal of the American Statistical Association, 102(479):774-983 2007.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning. pp. 7164-7173, 2019.
Yuan Yuan, Yueming Lyu, Xi Shen, Ivor W. Tsang, and Dit-Yar Yan Yeung. Marginalized average attentional network for weakly-supervised learning. ICLR, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 8778-8788 2018.
12

---

Published as a conference paper at ICLR 2020
A EXPLANATION OF THEOREM 1 FOR ROBUST LEARNING
Theorem. (Monotonic Relationship) ((Hu et al. 2018 Let p(a, v) and q(2.)) y) be the train- ing and test dexsstv.rexpecctielly Define r(æ, y) q(E,y) v)/pl)ppppp //pp(x,, (., y) and r; = r( Vi)- Let 1(0.y) (sign(() + v) and ((. (arggmars(0 # u) be 0-1 loss for binary classifi- cation and multi-class classification, respectively. Let [(-) be convex with f(1) 0. Define risk (0), empirical risk R(0), adversarial risk Rodv (f) and empirical adversarial risk Rado (f)
R(0) = Eptx.) (1g(())
(22)
R0) = n1 Σ ==1 ((go(z)); yc)
(23)
Rade (0) sup Ep(an) [r(2, (x) retly
(24)
Rade (0) = sup rett, n
(x,))
(25)
Uy {r(a,, v) (pt.,) 1f (r(a, w))l S 6, Bp(a..) (r(2,, r2, v) 2 0, V(x,)) E X x 3
and Ūr = {r| 1 n E f(r:) s 6, E = 1, 2 0}. Then we have that
If Rade (01) < 1, then R(01) < R(02) = Rado < Rado (02).
(26)
If Rado (02) 1, then R(0,) 3 R(22) = Radv (02) 1.
(27)
The same monotonic relationship holds between their empirical approximation: R(0) and Radv-
Hu et (0018) show that minimizing (empirical) risk is equivalent to minimize the (empirical) adversarial risk (worst-case risk) for 0-1 loss. Thus, we can directly optimize the risk instead of the worst-case risk. Specifically, suppose we have an observable training distribution p(x, y). The observable distribution p(z, y) may be corrupted from an underlying clean distribution q(2, y). We train a model based on the training distribution p(a, y), and we want our model to perform well on the clean distribution (xx, y). Since we do not know the clean distribution g(r, y). we want our model to perform well for the worst-case estimate of the clean distribution, with the assumption that the f-divergence between the corrupted distribution p and the clean distribution is bounded by 8. Note that the underlying clean distribution is fixed but unknown, given the corrupted training distribution, the smallest 8 that bounds the divergence between the corrupted distribution and clean distribution measures the intrinsic difficulty of the corruption, and it is also fixed and unknown. The corresponding worst-case distribution w.r.t the smallest ō is an estimate of the true clean distribution, and this worst-case risk upper bounds the risk of the true clean distribution. In addition, this bound tighter than the other worst-case risks w.r.t larger 8. Formally, the upper bound w.r.t the smallest 8 is given as
G(0) = -sup Ey(za)) [[go(z), v)l
(28)
gEûY,
where Uy is an equivalent constrainted set w.r.t for q(x, y). Then, we have
G(O)= = sup Ey(x,g) [10(()) v)] = sup Ep(z.w [r(z, (x)) getty
(29)
rEMy
When ((-) is 0-1 loss, from Theorem I, we know that minimize G(O) is equivalent to minimize G()). Thus, we can minimize (()) instead of G(Ø).
((0)) := Ep(z,) [[go(1)) v)]
(30)
Minimize the 30. enables us to minimize the Eq (8) without knowing the true divergence pa- rameter 5 beforehand. Usually, minimizing the upper bound can decrease the true risk under clean distribution. Particularly, when the clean distribution coincides with the worst-case estimate w.r.t the smallest 5, minimizing the risk under the corrupted training distribution leads to the same minimizer as minimizing the risk under the clean distribution.
Relationship between label corruption and general corruption
13

---

Published as a conference paper at ICLR 2020
Label corruption is special case of general corruption. Label corruption restricts the corruption in the space Y/ instead of the space X' V. That to say, the training distribution p(z) same as the clean distribution ((x) over X. Then, have the robust risk for label corruption as
Gy(0)===
sup By(r,)) [[g(()) w) oEūfnn
(31)
where H == ta(a, ((.)) EX } The supremum in (, is taken over
Uy n H, while the supremum in G(0) is taken over Uy. Due the additional constrain g(x) p(r), V(x,y) y) E X' x V. we thus know that the robust risk Gy) is bounded by G(Ø), i.e. Gy(0) - G(O). Moreover, it more piratical and important to be robust for both label corruption and feature corruption.
B PROOF OF THEOREM 2
Proof. Because 1(u < 0) S ((u), we have E (ui) 2 Eis 1u; < 0). Then
(u) min v pn
max Σn vel(u)) n2 Σ "
iml D/ + Σ "
1(u < 0))
(32)
[u.)), 70- E 1+ Σ.= 1lu: < <0)
S max
(33)
[uus) E, ;= 1(u; < <0))
max
(34)
I(u))
(35)
Since loss J(n) = Σm= (u:), we obtain (u) S J(u. On the other hand, we have that
(u) = min max
Σ i= ul((u)) 12 En + ",1,1(00
1lu < 0))
vE(O.1)"
min
Σ + Σ 1 (uu <0)
(36)
ve(0,1)"
> i=1
1 [[u <)
(37)
Since J(u) Σn 1(u: <),, we obtain Q 2 (u)
C PROOF OF COROLLARY
Proof. Since n = mb, similar to the proof of Q (u) < J (u), we have
min max (Em 1=1 (u) #=1 (E m
Vigl(uss), m Σ 1=1 Uij + Em
1 Ilug < 0))
max ((1ig)) Σ i=1
1 2iz Ilug < <))
(38)
max
m (uig)) Σ T7
i-1 I(mij < <0)
(39)
i=1 I(wig) = J (u)
(40)
14

---

Published as a conference paper at ICLR 2020
On the other hand, since the group (batch) separable sum structure, have that
@ (u) = -
b
min max
1=1 Usy((ut)); m
m
m
Vig
+ Σ i=1
m
1 (uiy < 0))
j= vE (0,1))
i=
min
Σ
max
(eyl(ui)):
Vij + Σ i==
I( < x0)
(41)
v
j=1
b
b
b
min ve(
max
(us((
Vij
1
[wss < <0)
(42)
j=1i=1 1=1
j=1 i=1
j=1
Q (u) 2 J (u)
(43)
D PROOF OF PARTIAL OPTIMIZATION THEOREM (THEOREM 4
Proof. For simplicity, let = I(us), i E (1,...,?) Without loss of generality, assume I < 1.....
.....
S
b. Let v be the solution obtained by Algorithm 1 Assume there exits V such that
a max Σ
Meli C
Σ vi) max Σ 4 C Σ r
(44)
Let T = Σ Us and T = Σ D.
1=1
Case 1: If T then there exists an 1%k and ot
O. From Algorithm
I we know k > T
(vg 0 0 k T and la 2 2j.jj E T". Then we know Σ
<
0eli Thus, we can
achieve that
max Σ
444 C
Σ
uj) max
044, C-
11)
(45)
t=1
i=1
1=1
< max
Dilis C- Σ
11).
(46)
1=1
This contradicts the assumption in Eq. (4)
T Σ 11. Since 4 2 0, we
Case 2: If > T", then there exists an 2k and 0i 0. Let LT-
have Ly- + 4/ 2 LT-. From Algorithm
we know that LT- + 4 > =C - T", Thus we obtain that
max
Σ uil, C- -
0.) > LT- + 4x
(47)
2 max (IT=, C-T*)
(48)
= max Euihi
Σ iml
(49)
This contradicts the assumption in Eq. 44
Case 3: T < T we obtain
C T T + Then we can achieve that
n
max Σ uf C- Σ
ot) max CLT-,
(50)
SC+1-T T*
(51)
SC-T
(52)
= Σn
1i
(53)
< max
udlis C Σ v)).
(54)
1=1
15

---

Published as a conference paper at ICLR 2020
This contradicts the assumption in Eq. (4).
Finally, we conclude that v obtained by Algorithm is the minimum the optimization problem
given in 13
E PROOF OF PROPOSITION 1
Proof. Note that 7 = > f from the condition of o3² = in Algorithm we know that LT S
in Algorithm [ we know that LT* > C-T" Because C+1-T". From the condition of v n..,,,,,, have LT-+1 Lr- + [(Tr++1) 2 LT. Thus, I(u,) 2 1(4 <0) 0 for i [I,....
obtain LT max(LT., C - T*). By substitute the optimum v² into the optimization function, we obtain that
min max E" vsl(us), C -E",. 11) (0,1))
(55)
max ( 1=1 v7((),), C
-- i=1 2!))
(56)
Cax(br,,C-1"
(57)
F PROOF OF THEOREM 3
Proof. We first prove that objective is tighter than the loss objective Ju) in Eq 8 we prove that objective [I an upper bound the 0/1 loss defined equation
After this,
For simplicity, let 4 = l(u,). we obtain that
(u) = min max ve(0,1)
uul( n
Σ
m2)
(58)
1=1
max
luuz), (n Σ 1))
(59)
lu;))
(60)
12 Note that Juu) Σ I(u,), thus, we have E (u) s J (u).
i
Without loss of generality, assume 4 S l2.... S wn Let L, = -Σ4 1 T = Σ 2j
where
j=1
1=1
v'=[uf, ez e. is the optimum of v for fixed u. Let k that the 0/1 loss (u) is as follows:
Σ
1(u 2 0).
Then we achieve
1=1
((u)= = Σ 1(4 < <0==n k
(61)
From Algorithm with C = n, we achieve that I.T Sn - T and L.r11 > n - T.
Case 1: If & T, can achieve that
2E (u) j(u) = =2max(L n - T) (n k)
(62)
2 2(n - T) (n k)
(63)
=n+ k: 2T 2 0.
16

---

Published as a conference paper at ICLR 2020
Case 2: kf T,n - T Lr, can obtain that
2F (u) (u) 2(n - 7) (n - k) = n 27.
(64)
Since lt < T, if follows that
T Σ
T 2 Lb+ Σ 1
=
(65)
j=k+1
j= 1
= L* T - k 2 T k.
(66)
(67)
Together with n - 2 Lr. we can obtain that
n - T LT 2 T -
20. 0.
(68)
Thus, can achieve that
2E (u) - J(u) = = 2T 20. 0.
(69)
Case 3: kf < n, < Lr, we can obtain that
2E (u) j(u) = 2 1max(LT, - n
(n - k)
(70)
2LT (n -kk k)
(71)
> (n T) + LT - n+
(72)
From 67 we have Lr 2 T - k. Together with (72). it follows that
2F (u) - J(u) > (n T) + (T k) n+ 2 0.
(73)
Finally, we can achieve that J(u) 2F (u) - 2J)
G PROOF OF COROLLARY
Proof. Since n = mb, similar to the proof of Q (u) S J (u),
we have
È Eu)
min max (E(0. Tm
E i=1 Ui vizl(uis), Em 0it)
max ΣΣ E. (uiz); m Em 1)
(74)
Σ j=1 max (Σ I(ii)) 0)
(75)
= Σn i kuny J (u)
(76)
On the other hand, since the group (batch) separable sum structure, we have that
Euu = -
(E 1=1 Uiyl(ur));
Σ i=1
min max 1
vig)
j=1 =1
mm
min
Σ j=1
max Σ i=1
Usyl(uig), m
Uzg))
(77)
v R
min max
(ygl(1s))
Vig
(78)
1=1==1 1=
==1==1 i=1
E (u)
(79)
Together with Theorem
we obtain that J(u) < 2E (u) S 2 (u) 2T
17

---

Published as a conference paper at ICLR 2020
OFAR100 Adam
CIFR100 SGD
Softttigg Trai
Fredhtige Train
Softttinge Test
Haedftinge Test
Accuracy
Accuracy
Softhinge
Hardhinge
Softtinge Test
0
Harthingg Text
100
Epoch
Epoch
(a) Training with Adam optimizer
(b) Training with SGD optimizer
Figure 4: Training/Test accuracy for soft and hard hinge loss with different optimizer on CIFAR100
H MULTI-CLASS EXTENSION
For multi-class classification, denote the groudtruth label as y (1, K)). Denote the classifica- tion prediction (the last layer output of networks before loss function) as 4, E (1,..,, K}. Then, the classification margin for multi-class classification can be defined follows
u by max L.- i
(80)
We can that 1 (u 0) 1(t
maxt, 0) is indeed the 0-1 loss for multi-class classification. 178
With the classification margin u, we employ the hinge loss. As we need the upper bound of 0-1 loss, the multi-class hard hinge loss
compute the base loss ((u) 1 1( 0). In this work,
function Moore & DeNero 2011 be defined
H(t, v) = max(1 u, 0) = max(1 ty + max f4 0).
(81)
The multi-class hard hinge loss in Eq. 81 is not easy to optimize for deep networks. We propose novel soft multi-class hinge loss function follows:
max(1 + max 4, 0)
ty max 4; 2 0 i
S(t, u) max(1 by + L)gSumExp(t)..) by max 4 0.
(82)
The soft hinge loss employs the LogSumExp function to approximate the max function when the classification margin is less than zero, i.., misclassification case. Intuitively, when the sample misclassificd, is far away from being correctly separate by positive margin (e.g. margin 2 1). In this situation, smooth loss function can help speed up gradient update. Because LogSumExp(t)) > max (L.... is we know that the soft hinge loss is an upper bound the hard hinge loss, i.e., S(t, 3) 2 (t,, y) Moreover, we can obtain a new weighted loss F(t, yı B) = 3S(t, 9) (1 (t, y), E [0, 1] that is also upper bound of 0-1 loss.
I EVALUATION OF EFFICIENCY OF THE PROPOSED SOFT-HINGE LOss
We compare our soft multi-class hinge loss with hard multi-class hinge loss Moore & DeNero (011) on CIFAR 100 dataset training with Adam and SGD optimizer, respectively. We keep both net- work architecture and hyperparameters same. We employ the default learning rate and momentums of Adam optimizer in Py Torch toolbox, i.c. Ir = 10 3 0.9, Br = 0.999. For SGD optimizer, the learning rate (Ir) and momentum (p) are set Ir 10 and p 0.9 respectively.
The pictures of training/test accuracy v.s number of epochs are presented in Figure We can observe that both the training accuracy and the test accuracy of our soft hinge loss increase greatly fast the number epochs increase. contrast, the training and test accuracy of hard hinge loss grow very slowly. The training accuracy of soft hinge loss arrive 100% trained with both optimizers. Both
18

---

Published as a conference paper at ICLR 2020
training and test aceuracy of soft hinge loss are consistently better than hard hinge loss. In addition, training accuracy of hard hinge loss can also reach 100% when SGD optimizer is used. However, its test accuracy is lowever than that of soft hinge loss.
J IMPACT OF MISSPECIFIED ESTIMATION OF NOISE RATE
E
We empirically analyze the impact of misspecified prior for the noise rate c. The average test accu- racy over last ten epochs MNIST for different priors are reported in Table We can observe that NPCL robust to misspecified prior for small noise cases (Symmetry-20%% Moreover, it becomes bit more sensitive on large noise case (Symmetry-505% and on the pair flipping case (Pair-35%)
Table 4: Average test accuracy of NPCL with different on MNIST over last ten epochs
Flipping Rate
0.5c
0.75c
1,25c
1.3c
Symmetry-2200 20%
96.31% + 0.17%
97.72% = 0.09%
99.41% + 0.01%
99.55% + 0.02%
99.10% = 0.04%
Symmetry-500P -50%
78.67% + 0.36%
87.36%
0.29%
98.53% + 0.02%
97.92% + 0.06%
67.61% + 0.06%
Pair-35%
80.59% 1 0.400%
87.86% 0.48%
97.90% + 0.04%
99.33% + 0.02%
86.66% = 0.08%
Table 5: Average test accuracy on MNIST over the last ten epochs.
Flipping-Rate
Standard
MentorNet
Co-teaching
Co-teaching-
GCE
NPCL
Symmetry-20%
93.78% = 0.04%
96.68%
0.05%
97.14% = 0.03%
99.1%% = 0.01%
99.40 = 0.01%
99.41%
0.01%
Symmerry 50%
65.81% 0.14%
90.53%
0.07%
91.35%
D.0PS
97.79%
0.03%
92.48
0.13%
98.53%
0.02%
P5r--3%
70.30% = 0.16%
89.62%
0.13%
90.961 = 0.18%
93.81% = 0.20%
72.26 = 0.06%
97.90% = 0.04%
Table 6: Average test accuracy on CIFAR 10 over the last ten epochs.
Hlipiing-Kee Rate
Standard
MentorNer
Co-traching
Co-caching+
GCE
NPCL
Symmetry- 20%
76.62% 0.07%
81.20% 0.09%
82.13%
0.08%
80.64%
84.68%
84.30% 0.07%
Symmary-30%% 300%
49.9256 : 0.09%
72.09% = 0.00%
74.28% = 0.11%
58.43% = 0.30%
61.80% = 0.11%
77.66% = 0.09%
35i
62.26% 0.09%
71.52% 0.06%
77.77% 0.14%
62.72%
0.05%
76.52% 0.11%
K RELATED LITERATURE
Curriculum Learning: Curriculum learning is a general learning methodology that achieves suc- cess in many area. The very beginning work of curriculum learning (Bengio et al.] (009) trains model gradually with samples ordered in a meaningful sequence, which has improved performance on many problems. Since the curriculum in (Bengio et al. (0099) is predetermined by prior knowl- edge and remained fixed later, which ignores the feedback of learners, Kumar et al. (Kumar et al.) 2010) further propose Self-paced learning that selects samples by alternative minimization of an augmented objective. Jiang et al. (Jiang et al., 2014) propose a self-paced learning method to select samples with diversity. After that, Jiang et al. (Jiang et al. 2015)) propose a self- celf-paced curricu- lum strategy that takes different priors into consideration. Although these methods achieve success, the relation between the augmented objective of self-paced learning and the original objective (e.g. cross entropy loss for classification) is not clear. In addition, as stated in (Jiang et al. 2018). the alternative update in self-paced learning is not efficient for training deep networks.
Learning with Noisy Labels: The most related works are the sample selection based methods for robust learning. This kind of works are inspired by currieulum learning (Bengio et al. 2009). Among them, Jiang et al. (Jiag et al., 2018) propose to learn the curriculum from data by a mentor net. They use the mentor net select samples for training with noisy labels. Co-teaching (Han al.) 2018b) employs two networks to select samples to train each other and achieve good generalization performance against large rate of label corruption. Co-teaching+ (Yu et al. 2019) extends Co- teaching by selecting samples with disagreement of prediction of two networks. Compared with Do-reaching/Co-teachiin
our CL is a simple plugin for a single network. Thus both space and time complexity of CL are half of Co-teaching's. Recently, Zhang & Sabuneu 2018) propose generalized Cross-entropy loss for robust learning.
Construction of tighter bounds of 0-1 loss: the 0-1 loss, many methods have been proposed. To name a few, Massadii-Shiaal et al. (Masnadi-
Along the line of construction of tighter bounds of
Shirazi & Vasconcelos 2009) propose Savage loss, which is non-convex upper bound of the (-1
19

---

Published as a conference paper at ICLR 2020
Table 7: Average test accuracy CIFARIO0 the last ten epochs.
Flppinng Rate
Stamdard
MentorNel
Co-teaching
Co-teaching
GCE
NPCL
Symmetry 20%
47.05% 4- 0.11%
51.58% 4 0.15%
53.89% a 0.09%
56.15%
0.09%
51.86%
0.09%
55.30% 4: 0.09%
Symunetry
25.47%
0.07%%
39.65%
0.10%
41.08%
37.88%
0.053
42.56%
1 0.066%
335
39.91%
1 0.115
40.42%
0.07%
43,30% 0.08%
40.88% 1 0.16%
36.64%
0.07%
44.45% 1 0.15%
Test Rccuracy
Test Rccurecy
Test Accuracy
LP1
(a)
(b)
(84C3 Semeetre 05
(71:0 sartic 0:15
Frecciion
Irecision
Lebbl Frecisionn
135
155
130
100
taoch
Epoch
tooch
(d) Symmetry 20%
(c) Symmetry 50%
(1)
Pairflip 355%
Figure Test accuracy and label precision vs. number of epochs on CIFARIO dataset.
loss function. Bartlett et al. (Bartlttt et al.] 2006) analyze the properties of the truncated loss for conventional convex loss. Wu et al. (Wu Liu 2007) study the truncated hinge loss for SVM. Although the results are fruitful, these works are mainly focus on loss function at individual data point, they do not have sample selection property. In contrast, our curriculum loss can automatically select samples for training. Moreover, it can be constructed in tighter way than these individual losses by employing them as the base loss function.
L ARCHITECTURE OF NEURAL NETWORKS
CNN on MNIST
CNN on CIFAR-10
CNN on CIFAR-100
28x28 Gray Image
32x32 RGB Image
32x32 RGB Image
3x3 conv, 128 LReLU
3x3 conv, 128 LReLU
3x3 conv, 128 LReLU
2x2 max-pool, stride 2 dropout, p= 0.25
3x3 conv, 256 LReLU
3x3 conv, 256 LReLU
3x3 conv, 256 LReLU
2x2 max-pool, stride 2 dropout, p 0.25
3x3 conv, 512 LReLU
3x3 conv, 256 LReLU
3x3 conv, 128 LReLU avg-pool dense 128->10
dense 128 128->10
dense 128 228++100
20

---

Published as a conference paper at ICLR 2020
Text Accurscy
Text
Ebcch
Ebcch
Ebcch
(a)
(b)
Label Frestsion
10
Lebel Precisioon
Label Prectsion
Eboch
Epoch
Epoch
(d)
Symmetry 20%
Symmetry- 50%
(f) Pairflip355%
Figure 6: Test accuracy and label precision number of epochs on CIFAR100 dataset.
Accurary
Test Accuraty
Test Accuraty
Ebcco
Ebcch
Ebcch
(a) Symmetry-OSt
(b) Symmetry-
Symmetry-20%
Accurscy
tauch
tsocc
tsocch
(d) Symmetry-30%%
(e)
Symmetty-4400
Symmeetr--500
Figure 7: Test accuracy vs. number of epochs MNIST dataset.
21

---

Published as a conference paper at ICLR 2020
Ebcch
Ebcch
Eboch
(a) Symmetry-U%
Symmetry-10%
Symmetrr-20%
Test Accuracy
Test Accuracy
Test Accuracy
Eoach
Eboch
Eboch
(d) Symmetry-309%
Symmetry-409%
Symmerry-505%
Figure 8: Test accuracy vs. number of epochs CIFARI00 dataset.
22


Reviews:
[{"review_id": "SkxvXnoatH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal,\n\nI think the authors made a valid argument to address my concerns on evaluation. So, I'd like to increase my score as weak accept! \n\n=====\n\n\nSummary:\n\nTo handle noisy labels, this paper proposed a curriculum loss that corresponds to the upper bound of 0-1 loss. Using synthetic noisy labels on MNIST and CIFAR, the authors verified that the proposed method can significantly improve the robustness against noisy labels.\n\nDetailed comments:\n\nOverall, the paper is well-written and the ideas are novel. However, experiments are a little weak due to weak baselines and experimental setups (see suggestions for more details). I will consider raising my score according to the rebuttal.\n\nSuggestions:\n\n1. Could the authors consider more baselines like D2L [Ma' 18] and Reweight [Ren' 18] \n\n2. Similar to [Lee' 19], could the authors evaluate the performance of the proposed methods on more realistic noisy labels such as semantic noisy labels and open-set noisy labels? \n\n[Lee' 19] robust inference via generative classifiers for handling noisy labels, In ICML, 2019.\n\n[Ma' 18] Dimensionality-Driven Learning with Noisy Labels, In ICML, 2018.\n\n[Ren' 18] Learning to Reweight Examples for Robust Deep Learning, In ICML, 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1efYqIiKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a new loss function: curriculum loss, which is a meta-loss function that we can still specify an existing surrogate loss to use this loss function. This meta-loss function guarantees to be tighter than using a traditional pointwise-sum loss function as used in the empirical risk minimization framework. Intuitively, the proposed CL loss embed the sample selection process in the objective function. The authors suggest that it is robust against label corruption because it is tighter and provided promising experimental results.\n\n========================================================\nClarity:\nThe paper is well-written and easy to follow. \n\n========================================================\nSignificance:\nThe proposed paradigm is interesting and I am convinced that it can be useful under label noise. The experiments look promising. Future work about the analysis of NPCL/CL is also interesting to consider (e.g., which surrogate loss to use, rigorous theoretical guarantee, etc.). I think the proposed method is impactful. \n\n========================================================\nComments:\nThe proposed method is interesting and can give a tighter bound for any surrogate loss by using this method (CL). Moreover, the author suggested a simple extension of CL for label corruption (NPCL) and the performance is impressive. I would like to vote accept for this paper but the following point highly concerns me and I am not sure about the correctness (see the concern below). It is about the motivation not the proposed method.\n\nConcerns about motivation:\n\nI disagree with the original motivation of this paper. The authors used the result of Hu et al. 2018 to motivate the use of CL. To my knowledge, the main point raised by Hu et al. is as follows:\n\nIn classification, minimizing the adversarial risk yields the same solution as using the standard empirical risk. This suggests that minimizing the adversarial risk may not enhance the robustness of a classifier. Yet, it may still be useful when we consider regression (other settings but not classification). As a result, in classification, we should try other methods to make a robust classifier. Then, Hu et al. considered to utilize some kind of structural assumption to make a robust classifier. From their title: \"Does Distributionally Robust Supervised Learning Give Robust Classifiers?\", I think they suggested \"No\" as an answer and the discussion about 0-1 loss in the curriculum loss paper will be contradicted to them from the motivation perspective. \n\nFurthermore, regarding the adversarial risk, it is not focusing on the label noise but rather the noise of the feature-label pair, i.e, perturb (x,y) adversarially within an f-divergence ball. However, in my opinion, if we randomly flip the label of the data regardless of x (as the authors and existing work did in experiments when considering label corruption: symmetric, partial, etc.), we cannot be confident to state that the f-divergence between test distribution and corrupted training distribution is small under label noise. \n\nAnother point to motivate the use of 0-1 loss that the author mentioned is when we have outliers (Masnadi-Shirazi & Vasconcelos, 2009). This makes sense and this is a famous argument to discourage the use of too steep loss functions, e.g., exponential loss. I think this motivation is fine but it is not directly related to label corruption because we do not add out-of-distribution data but rather the label noise. Furthermore, the authors did not inject any outliers in the experiments in my understanding. I think this is totally no problem because we are focusing on label noise here, but this makes the motivation about outliers less important when we are talking about label noise.\n\nI think the most important direction both in theory and experiments about the robustness to label noise of the 0-1 loss is that 0-1 loss satisfies a \"symmetric property\", i.e., \\ell(z)+\\ell(-z) = Constant for a margin-based loss function in binary classification. Under symmetric label noise, \"the minimizer of the expected symmetric noise risk (a risk that the label is corrupted by coin flipping noise) is identical to the minimizer of the clean risk (normal risk)\". Although it is not empirically but the expected version, it gives a good insight about the advantage of directly minimizing 0-1 loss under label noise. This is first pointed out by \n\n[1] Manwani et al.: Noise tolerance under risk minimization, IEEE Transactions on Cybernetics 43 (2013) \n[2] Ghosh et al.: Making risk minimization tolerant to label noise Neurocomputing 160 (2015): 93-107. \n\n([1] focused on the 0-1 loss while [2] extended it to symmetric losses.)\n\nThen, it was extended to the multiclass loss by the following paper:\n\n[3] Ghosh et al.: Robust loss functions under label noise for deep neural networks. AAAI2017. \n\nThe advantage of symmetric losses is also discussed in this paper that the authors already cited in the symmetric noise experiment section. \n\n[4] van Rooyen et al.: Learning with symmetric label noise: The importance of being unhinged, NeurIPS2015\n\nThe advantage of the symmetric condition and 0-1 loss is also discussed in a more general noise scenario and more evaluation metrics:\n\n[5] van Rooyen et. al: An average classification algorithm. arXiv:1506.01520, 2015\n[6] Charoenphakdee et al.: On symmetric losses for learning from corrupted labels, ICML2019\n\nAnd the following paper that was also cited in the submitted work and compared:\n\n[7] Zhang and Sabuncu: Generalized cross-entropy loss for training deep neural networks with noisy labels, NeurIPS2018\n\nis also inspired by the robustness of the symmetric losses (including 0-1 loss). They argued that although the symmetric loss (MAE) for multiclass proposed by Ghosh AAAI2017 is robust, it is hard to train for challenging datasets, and they try to relevate this condition while making it easier to train.  This paper outperformed [7] and I think it is clearer and better to build a story along this line.\n\nIn short, here is the key message why I think the current motivation does not feel right. When we have noisy labeled data, instead of motivating the use of 0-1 loss by suggesting that \n\n\"If we have clean labeled data, minimizing the \"adversarial\" ERM risk using \"clean\" labeled data yields the same minimizer as minimizing the \"standard\" ERM risk using \"clean\" labeled data\",\n\nI believe the story to motivate the robustness of 0-1 loss under label noise should be \n\n\"If we have noisy labeled data, minimizing the \"standard\" or \"modified\" risk using \"noisy\" labeled data yields the same minimizer as minimizing the \"standard\" ERM risk using \"clean\" label data\"\n\nThe latter statement corresponds to the literature I suggested. \n\nApart from the motivation raised by the authors, as we can see from this curriculum loss paper, NPCL nicely outperformed generalized cross entropy loss in [7], which is impressive.\n\n========================================================\nDecision.\nI strongly feel that motivating the noise robustness of 0-1 loss by discussing about the adversarial risk (Hu et al.) is misleading. Nevertheless, I feel the proposed method itself makes a lot of sense and I am impressed by the results. If the author can convince me that using the current motivation of the paper is suitable, I am happy to improve the score. Another way is to agree to modify the motivation part. Given the experiments were done, it is not to difficult to change the motivation of the paper. At this point, I have decided to give a weak reject. \n\n========================================================\nQuestions:\n1. Is it straightforward to combine NPCL with Co-teaching/Mentornet/Co-teaching+?\n2. Does the traditional theory about classification-calibration (Zhang, 2004, Bartlett+, 2006) can guarantee the Bayes-optimal solution if we use NPCL?\n\n========================================================\nMinor comments:\n1. Page 9: Both our NPCL and Generalized Cross Entropy(GCE) << space missing between Entropy and (\n\nUpdate: I have read the rebuttal. Although I am still not fully convinced with the motivation of the paper and still doubting whether NPCL works well because of the given motivation, I still believe that the proposed NPCL should give a new perspective to deal with noisy labels. I like the idea of the paper. Thus, I change the score to Weak Accept.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJghRHVAKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of learning with noisy labels and proposes a novel cost function that integrates the idea of curriculum learning with the robustness of 0/1 loss. The resulting cost function is theoretically justified and experimentally validated. \n\nPros:\n(1) The proposed cost function is novel in its design, especially the aspect of curriculum learning with a computationally efficient implementation, as in Algorithm 1.\n(2) The new cost function could be treated as a simple-to-implement add-on to make learning more robust to noisy labels. \n(3) The introduction is well formulated and organized with focused motivation.\n\nCons:\n(1) Equ. 9 requires more explanation of the intuition of using a combination of conventional surrogate loss and 0/1 loss, and furthermore the role of the index indicator in balancing the above two parts.\n(2) Curriculum learning focuses on easy example followed by hard ones. Yet noisy examples are mixed with difficult ones in your formulation of sample selection mechanism (index indicator). The pruned examples are therefore more likely to have a high proportion of hard examples, which is undesirable. To illustrates the effectiveness of the proposed algorithm against such scenarios , one would like to see experiments on more difficult datasets such as Tiny-ImageNet. \n(3) It is not clear if the quantitative results in Table 2 and 3 are produced with the pre-defined \\epsilon beforehand or with grid search as done in Table 4. Knowing \\epsilon would render comparison unfair for baselines.\n \nOther remarks:\n(1) E(u) threshold parameter changes from “n” in equation 11 to “C” in equation 13 (probably considering equation 9). In Equ 13, C is given as \"n+0/1 loss\", its transition to the other alternative forms in Equ 18 is not fully explained. \n(2) The purpose of proposition 1 is unclear and may be at least shortened.\n(3) Should have used some uncertainty metric instead.\n(4) Incremental improvement over SOTA. SOTA was actually better in some cases.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies learning with noisy labels by integrating the idea of curriculum learning.

All reviewers and AC are happy with novelty, clear write-up and experimental results.

I recommend acceptance. 

Decision: Accept (Poster)